Thank you for your response.
> As far as I understand, in a practical sense, the main difference is that one method is designed for online settings while the other is designed for offline settings. Is that true? I did not understand if your method is designed to detect harmful shifts of X. Is that the case?
Yes, the major difference is that our method is designed for online use, while the other is for offline. Regarding whether our method is designed to detect harmful "shifts of X" (i.e. covariate shifts), our central assumption is Assumption 4.1, i.e. that the selector function generalizes in terms of FDP from calibration to production. This is more likely to hold in the case of pure covariate shift, where the relationship Y | X is invariant. However, it may still be effective if this relationship changes, provided the model tends to have high error in the same regions of the input space before and after the shift. Note that in our empirical experiments, we study shifts in natural datasets, where the Y | X relationship is very likely to be at least somewhat non-invariant. Our method is shown be effective in these cases.
> Do you know why the Detectron has such a bad FDP in your experiment?
Detectron’s main idea is to learn a Disagreement Classifier that performs as well as the original model on the training distribution while disagreeing with the original/base model's predictions on the production data. This method is highly sensitive to the performance of the base classifier, the function class used, and the size and nature of the production data. While it effectively detects harmful shifts (as shown by its high power in our experiments), it may fail when the shift is benign.
Below is a simple code example that generates a plot illustrating a failure case. In this example, the training data points is represented in red and green, with the ground truth shaded in corresponding colors. The base model is shown as a solid black line. For simplicity, we assume the model is a perfect classifier. The data has shifted to the right, creating unlabeled production data, all correctly classified by the base classifier.
We’ve also depicted the potential learnable classifier as a dashed line, representing the boundary of all possible functions, which depends on the model type, complexity used for the disagreement classifier, and the nature and size of the data. We have shown a potential disagreement classifier in orange that performs similarly to the original model on training data but disagrees on the predictions of the base classifier in the production data. As shown, even with a benign shift, we can still find a disagreement classifier that performs well on training data but disagrees significantly in production, raising a false alarm.
```{python}
import matplotlib.pyplot as plt
import numpy as np
# Generating sample data
x = np.linspace(-3, 3, 500)
base_classifier = 2 * np.sin(x)
shaded_area = 2 * np.sin(x)
boundary_upper = 5 * np.sin(x)
boundary_lower = 0.5 * np.sin(x)
disagreement_classifier = 3.5 * np.sin(x) * (x <= 0) + 4.5 * np.sin(x) * (x > 0)
np.random.seed(0)
x_class1 = np.random.uniform(-2, -1, 10)
y_class1 = np.random.uniform(-2, 1, 10) + 3
x_class2 = np.random.uniform(-2, -1, 10)
y_class2 = np.random.uniform(-5, -2, 10) - 3
x_class1_shifted = np.random.uniform(-2, -1, 20) + 3
y_class1_shifted = np.random.uniform(-2, 2, 20) + 4.5
x_class2_shifted = np.random.uniform(-2, -1, 10) + 3
y_class2_shifted = np.random.uniform(-5, -2, 10) + 2
x_unlabelled = np.concatenate([x_class1_shifted, x_class2_shifted])
y_unlabelled = np.concatenate([y_class1_shifted, y_class2_shifted])
# Plotting
plt.figure(dpi=200, figsize=(8, 5))
plt.fill_between(x, shaded_area, 12, color='green', alpha=0.1, edgecolor='none', label='Ground Truth (Class 1)')
plt.fill_between(x, -12, shaded_area, color='red', alpha=0.1, edgecolor='none', label='Ground Truth (Class 2)')
plt.scatter(x_class1, y_class1, c='green', marker='+', label='Training Data (Class 1)')
plt.scatter(x_class2, y_class2, c='red', marker='+', label='Training Data (Class 2)')
plt.scatter(x_unlabelled, y_unlabelled, facecolors='none', edgecolors='black', label='Unlabelled Production Data')
plt.plot(x, boundary_upper, 'gray', linestyle='--', label='Set of Potential Learnable Classifiers')
plt.plot(x, boundary_lower, 'gray', linestyle='--')
plt.plot(x, base_classifier, 'k-', label='Base Classifier')
plt.plot(x, disagreement_classifier, 'tab:orange', linestyle='-', label='Disagreement Classifier')
plt.legend(loc='upper left', bbox_to_anchor=(1, 1), fontsize='small', frameon=False)
plt.grid(False)
plt.tight_layout()
plt.axis('off')
plt.show()
```