**FQ1: “I think tunning the $g$ on the calibration data breaks the basic assumption of CP, i.e., exchangeability. Tunning the hyper-parameters on fresh/hold-out data is reasonable.”**
FA1: Thank you for your feedback. In the following response, we
(1) clarify that tuning the $g$ will not violate the exchangeability assumption.
(2) elaborate our experiment setting and show additional experimental results, where RC3P significantly outperforms the best baseline with $30.627\\%$ (four datasets) or $40.8\\%$ (excluding CIFAR-10) reduction in APSS ($\downarrow$ better). It is similar to the improvements of RC3P that we report in our main paper ($26.25\\%$ (four datasets) or $35\\%$ (excluding CIFAR-10) reduction).
(3) explain that limited calibration samples may result in inaccurate quantile and thus inaccurate coverage/efficiency measures.
(4) elaborate the experiment where RC3P still outperforms baselines without tuning $g$ in Table 7 of rebuttal PDF. Conditioned on similar APSS of all methods, RC3P significantly outperforms the best baselines with $35.18\\%$ reduction in UCG on average.
**(1) Adding inflation on calibration dataset will not violate the exchangeability assumption**:
The exchangeability is defined [fr1] by the invariance of the joint distribution of the variables $\\{ Z_1, \cdots, Z_n\\}$ to every permutation $\pi$ for $\\{1,…,n\\}$, i.e., $P(Z_1, …, Z_n) \\overset{d}{=} P(Z_{\pi(1)}, …, Z_{\pi(n)})$.
The exchangeability is the property of distribution. In our experiments, adding inflation on nominal coverage is an algorithm design and will not change the data distribution, where calibration and testing datasets still consist of i.i.d. samples form the same distribution. The same algorithm design (i.e., inflating the coverage during calibration) is applied in previous CP papers, such as ARCP (see Theorem 2, Equation 8 in [fr2]) that uses an inflated nominal coverage.
**(2) New experiment on hold-out data for tuning $g$: RC3P achieves the same level of improvement in the reduction of APSS ($30.627\\% \downarrow$ from the best baseline).**
Below we report the experiments where $g$ is tuned on hold-out samples on four datasets with the APS scoring function and imbalance EXP, 0.1. We first split calibration dataset into calibration and hold-out samples (50-50 split), where hold-out samples are used to tune $g$ and calibration samples are used to compute class-wise quantiles. UCR is controlled to $\leq 0.10$ on CIFAR-10 and $0.08$ on other datasets.
|Dataset|CIFAR-10|CIFAR-100| mini-ImageNet|FOOD-101|
|-|-|-|-|-|
|\# hold-out samples per class| 250 |25 |75|62 |
|CCP| 1.940 $\pm$ 0.020 | 38.937 $\pm$ 0.335 | 32.249 $\pm$ 0.419 |33.253 $\pm$ 0.367|
|Cluster-CP| 2.212$\pm$ 0.019 |34.125 $\pm$ 0.990 | 36.372 $\pm$0.270 | 40.762 $\pm$ 0.343|
|RC3P| 1.940$\pm$ 0.020 |19.668 $\pm$ 0.005 |17.736 $\pm$ 0.002 |21.565 $\pm$ 0.007|
These results shows that RC3P significantly outperforms the best baseline with $30.627\\%$ (four datasets) or $40.8\\%$ (excluding CIFAR-10) reduction in APSS ($\downarrow$ better). Similar order of improvements of RC3P can be found in our main paper ($26.25\\%$ on four datasets or $35\\%$ excluding CIFAR-10 APSS reduction from the best baseline, Table 1).
**(3) Limited calibration samples may result in inaccurate quantile and thus inaccurate coverage/efficiency measures**:
There are few training data samples for the tail classes and a limited number of calibration samples (e.g., 50 calibration samples of each class in CIFAR-100, see A5 for Q5) in our setting.
If we split the calibration datasets into calibration and hold-out samples, the limited samples may cause inaccurate class-wise quantiles compared to the true quantiles, and thus inaccurate coverage and efficiency measures.
In practice, our experiment in the main paper used as many samples as possible for calibration.
For instance, comparing the results in the above table with Table 1 in our main paper, there are minor perturbations in APSS measures.
**(4) Without tuning $g$, RC3P still outperforms than baselines ($35.18\\%$ reduction in UCG on average)**: We add the experiments without controlling UCR in Experiment (1) of GR1 (Table 7 in PDF) under the same setting with the main paper.
The model is trained from scratch using LDAM [fr3]. UCR is not controlled. We then use the total under coverage gap (UCG, $\downarrow$ better) between class conditional coverage and target coverage $1-\alpha$ of all under-covered classes.
We choose UCG as the fine-grained metric to differentiate the coverage performance in our experiment setting. Conditioned on similar APSS of all methods, RC3P significantly outperforms the best baselines with $35.18\\%$ reduction in UCG on average.
[fr1] Shafer, G. and Vovk, V., 2008. A tutorial on conformal prediction
[fr2] Gendler et al., 2021. Adversarially robust conformal prediction
[fr3] Cao et al, 2019. Learning imbalanced datasets with label-distribution-aware margin loss