Investigating Data Usage for Inductive Conformal Predictors

Tue, 18 Jun 2024 00:00:00 +0000

How should limited development data be divided between training and calibration for inductive conformal prediction? This study examines the tradeoff between coverage validity and predictive efficiency, measured through prediction-set size, when calibration data are scarce or overlap with training data.

The experiments wrap an artificial neural network with an inductive conformal predictor on the multiclass Covtype dataset. They vary training/calibration allocation, development-set size, and overlap, repeating randomized splits to examine variability. Small calibration sets can increase variability, while training/calibration overlap can produce smaller prediction sets at the cost of undercoverage. These are empirical findings from one classification dataset; they do not establish a universally optimal split or validate overlapping calibration data in general.

Marginal Coverage | Yizirui Fang

Investigating Data Usage for Inductive Conformal Predictors