Multi-model Ensemble Conformal Prediction in Dynamic Environments

Summary

This paper considers conformal prediction in the online setting with multiple fitted models, which is very interesting and in line with real applications. As it remains an active question regarding how to aggregate multiple models, the proposed approach adopts importance sampling to choose the 'best' model at each step. The performance is justified with comprehensive simulations and comparisons with baseline approaches are presented.

Strengths

The problem studied in this paper is well-defined and is of interest in the literature. The theoretical guarantee and numerical performance of the proposed approach are well presented.

Weaknesses

Please see questions in the following section.

Questions

1. [Choice of weights] The weights defined in (6) follow exponential decay. It would be interesting to explore other choices of the weights: (1) in terms of theoretical guarantee, how will the coverage/accuracy guide the choice of w? (2) choices of w can also be compared via numerical simulations. 2. [Thm 1] Average regret is considered in Thm 1. It remains unknown how will the MOCP approach outperform FACI in terms of the coverage guarantee. 3. [MOCP vs SAMOCP] MOCP would also be capable of the tasks presented in the simulation section, then it could be more informative to compare SAMOCP with MOCP to validate the advantage of SAMOCP. 4. [From adversarial setting to stochastic setting] It would be interesting to study the theoretical performance of the proposed approach under the stochastic setting where observations are i.i.d. to understand the gains of model aggregation. Please see references e.g. (1) Duchi et al. (Adaptive Subgradient Methods for Online Learning and Stochastic Optimization); (2) Hazan et al. (Adaptive Online Gradient Descent); etc.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Limitations are discussed in the paper.

Summary

This paper proposes an online multimodal conformal prediction method named SAMOCP, developed to address data distribution shifts in dynamic environments. Specifically, the method selects the best model from multiple candidates to create prediction sets via strongly adaptive online learning.

Strengths

1. The method aims to expand single-modal online conformal prediction to multimodal. 2. The method achieves better efficiency through the selection from multiple models.

Weaknesses

1. This paper appears to be a variant of SAOCP [1], utilizing many techniques from [1] to construct candidate sets with multiple models. By optimizing the weight of each model, the final prediction set is obtained from the best model. However, there is insufficient evidence to validate the superiority of SAMOCP compared to SAOCP, both theoretically and empirically. 2. The authors claim that the method is devised for a multimodal setting. However, both the methodology and experiments seem to involve multiple experts for a unimodal dataset using different deep neural networks, which contradicts the introduction of the method. 3. There are several concerns regarding the theoretical analyses. What is the meaning of Theorems 1 and 3? Are they intended to demonstrate that the pinball loss will increase with the addition of the optimal single model? What is the strength of these theoretical analyses for decision-making in an online multimodal setting? 4. It is suggested to provide a Complexity Analysis of the overall model.

Questions

See in the Weaknesses.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The limitations of the authors have been pointed out in the Conclusion.

Summary

This paper investigates online conformal prediction within dynamic environments. The authors introduce a novel adaptive conformal prediction framework that leverages multiple candidate models. The proposed algorithm achieves sublinear regret, and its effectiveness is demonstrated through both real and synthetic datasets.

Strengths

The proposed method is novel, with a clear presentation and comprehensive literature review. It demonstrates good performance compared to benchmark methods.

Weaknesses

The intuition behind the proposed algorithm in the dynamic environment (Section 3.2) is not sufficiently developed.

Questions

1. In Algorithm 1, how should \(\epsilon\) be chosen? 2. How does this multi-model approach perform compared to retraining the non-conformal score after each update? 3. How quickly can this method adapt to distribution shifts compared to benchmarks in real data studies? Rapidly capturing distribution shifts and making valid prediction sets is crucial in practical applications. 4. Could the authors provide intuition behind the design of the algorithm in a dynamic environment? Specifically, why are many experts retained when \(t\) is large, despite having a total of \( M\) models?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

NA

Summary

This paper proposes the Strongly Adaptive Multimodal Online Conformal Prediction (SAMOCP) methodology, which constructs adaptive conformal prediction sets by integrating information from multiple learning models in dynamic environments. This is accomplished by creating multiple experts at each time step, where each expert is an online learning algorithm that dynamically updates the weights of each model and has a finite active time interval. The appropriate miscoverage probability at time $t$ for constructing the prediction set in the subsequent step is determined by the miscoverage probability associated with the selected expert among the many active experts.

Strengths

1. The concept of developing a Conformal Prediction framework for dynamic environments using information from multiple learning models is novel. 2. The paper is well-written and well-structured. 3. The performance of the proposed method is analyzed with theoretical justifications and has demonstrated advantages through extensive experiments compared to existing benchmarks.

Weaknesses

1. As already pointed out in this paper, this new method is much slower compared to other benchmarks. Therefore, it would be beneficial if the authors could conduct a comprehensive computational complexity analysis, in addition to reporting the empirical runtime in the experiments. This would help practitioners understand how well this method can scale to more complex datasets with a larger number of learning models.

Questions

1. Did you try to implement the method on time series data where consecutive data can potentially exhibit continuous distributional shifts? For example, on stock price data, with the response output being 1 if the price goes up and 0 if it goes down. 2. Can you explain how exactly to ‘select one of the miscoverage probabilities ${\alpha_t^m\}$ according to the pmf …’? Is it simply selecting the $\alpha_t^m$ with the highest normalized weight? 3. In the numerical experiments, if my understanding is correct, the sudden shifts setting involves having intact data before a certain threshold and fully corrupted data after that threshold. How exactly does it work for the gradual shifts? For instance, if there are five distinct levels of severity, do you assign all images with level 0 in the first 1/5 of the horizon, level 1 in 1/5-2/5 of the horizon, and so on? 4. How is the number of experts chosen? Minor: There seems to be a typo in line 134: the miscoverage probability $\alpha_{t+1}^m$ can be updated via ..., instead of $\alpha_{t}^m$. For completeness and better clarity, I would suggest adding a line in the algorithm between ‘select one miscoverage probability…’ and ‘Observe true $Y_{t+1}$' that says ‘construct prediction set for $Y_{t+1}$ using the selected miscoverage probability.'

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors addressed the limitations of their work.

Follow-up comments

Thank you for the references and additional simulation studies! I'll maintain my scores at this moment and keep the authors' responses in mind for my evaluation through the discussion period.