# Investigating Data Usage for Inductive Conformal Predictors Canonical page: https://yiziruifang.com/publication/fang-2024-investigating/ Website owner: Yizirui Fang How training/calibration splits and overlap affect coverage validity and prediction-set size in inductive conformal classification. Authors: Yizirui Fang, Anthony Bellotti Publication date: 2024 Venue / version: arXiv preprint arXiv:2406.12262 DOI: https://doi.org/10.48550/arXiv.2406.12262 ## Abstract Inductive conformal predictors (ICPs) are algorithms that are able to generate prediction sets, instead of point predictions, which are valid at a user-defined confidence level, only assuming exchangeability. These algorithms are useful for reliable machine learning and are increasing in popularity. The ICP development process involves dividing development data into three parts: training, calibration and test. With access to limited or expensive development data, it is an open question regarding the most efficient way to divide the data. This study provides several experiments to explore this question and consider the case for allowing overlap of examples between training and calibration sets. Conclusions are drawn that will be of value to academics and practitioners planning to use ICPs. Abstract source: [arXiv preprint 2406.12262v1, June 18, 2024](https://arxiv.org/abs/2406.12262v1). Abstract license: [CC BY-NC-ND 4.0](https://creativecommons.org/licenses/by-nc-nd/4.0/). ## Sources and artifacts - [Paper PDF](https://yiziruifang.com/publication/fang-2024-investigating/paper.pdf) - [Original PDF source](https://arxiv.org/pdf/2406.12262v1) - [arXiv](https://arxiv.org/abs/2406.12262) - [BibTeX citation](https://yiziruifang.com/publication/fang-2024-investigating/cite.bib) How should limited development data be divided between training and calibration for inductive conformal prediction? This study examines the tradeoff between coverage validity and predictive efficiency, measured through prediction-set size, when calibration data are scarce or overlap with training data. The experiments wrap an artificial neural network with an inductive conformal predictor on the multiclass Covtype dataset. They vary training/calibration allocation, development-set size, and overlap, repeating randomized splits to examine variability. Small calibration sets can increase variability, while training/calibration overlap can produce smaller prediction sets at the cost of undercoverage. These are empirical findings from one classification dataset; they do not establish a universally optimal split or validate overlapping calibration data in general. --- [All papers](https://yiziruifang.com/publication/) ยท [Research](https://yiziruifang.com/research/)