Learning to Defer with an Uncertain Rejector via Conformal Prediction
NeurIPS 2024 Workshop on Bayesian Decision-making and Uncertainty · 2024
Abstract
Learning to defer (L2D) allows prediction tasks to be allocated to a human or machine decision maker, thus getting the best of both’s abilities. Yet this allocation decision depends on a ‘rejector’ function, which could be poorly fit or otherwise misspecified. In this work, we perform uncertainty quantification for the rejector sub-component of the L2D framework. We use conformal prediction to allow the reject to output sets, instead of just the binary outcome of ‘defer’ or not. On tasks ranging from object to hate speech detection, we demonstrate that the uncertainty in the rejector translates to safer decisions via two forms of selective prediction.
Abstract source: NeurIPS 2024 Workshop on Bayesian Decision-making and Uncertainty paper, page 1.
Research question
Can uncertainty about the rejector improve decision making in learning-to-defer systems, where each input is assigned to either a model or a human expert?
Approach and evidence
This workshop paper uses conformal prediction to make the rejector return sets rather than a single defer-or-predict decision. It evaluates two responses to ambiguous routing: withholding the prediction and checking agreement between the human and model. The reported experiments cover tasks ranging from object to hate-speech detection. The workshop PDF contains the original method and results.
Scope and versions
The paper focuses on multiclass learning to defer with one expert. Its selective workflows can abstain or consult both decision makers, so their behavior includes decisions that are withheld as well as those returned. This is the NeurIPS 2024 Workshop on Bayesian Decision-making and Uncertainty version. The subsequent TMLR 2026 article has a separate abstract, review record, and citation.