Conformal PID Control for Time Series Prediction

Summary

The paper proposes an adaptation to existing adaptive conformal prediction in two novel ways: (i) by tracking the quantile via online regression over the running sum of the errors; and (ii) by incorporating a second model to anticipate the quantile in the next instant ("scorecasting"). The authors include theoretical guarantees for the resulting method, as well as comparisons against a benchmark in the field (ACI, which is shown to be a a special case of the proposed algorithm) over many different datasets.

Strengths

- Clear: the paper is well-written and very clear regarding its contributions. - Limitations: the discussions on the method's limitations are very helpful. - Theoretical results: the mathematical proofs are easy to follow and the notation is clear. - Empirical results: the empirical studies are extensive and cover several different datasets. The experiments in the supplementary material are also very thorough. - Code: the code is well-organized and readable (but please include the `environments.yml` file, all the necessary datasets, and instructions on how to reproduce figures in the paper! See below).

Weaknesses

- Theoretical guarantees: the guarantees of the proposed method are only asymptotic. Unfortunately, no finite-sample results are included. None of the theoretical developments in the paper showcase advantages that the proposed method has over leading alternatives. - Baselines: there are many recent proposals in the field (even in your related works section), but the only baseline is ACI. SPCI (Xu and Xie, 2022) also obtained asymptotic longitudinal coverage of time series without creating infinitely-wide intervals; can you underline in what ways your proposal is better? - Experimental results: Can you include more details on how the hyperparameters were chosen for the examples in the main paper (for example, $r_t$)? Having too many hyper parameters can be overwhelming to the end-user (I appreciate the discussion in appendices C and D!). It would be great to include at least one example of adaptive learning in the proposed algorithm. - Scalability: all the time series considered, both in the main text and in the supplementary material (SM), are small, with the largest ones comprising around 3000 data points. In scenarios where new data arrives every second, for example, it is not clear if the method would work as-is or if any modifications would be needed. A discussion about this and an experiment with a very long time series would be great. - References: the authors point out the field of online calibration, and mention connections to game theory and online learning, but never explicitly discuss the connection to their work. It would be nice to better understand what tools from the field you are using or building upon. - Relationship to control theory: seems slightly out-of-place. Either explain the connection better (perhaps in the SM) or develop it in another paper? As it is, it's hard for the CP audience understand what the PID is, or why this connection is relevant. - Reproducibility: adding the proposed method's pseudocode would help a lot, either to the main text or to the SM if space is lacking. Trying to reproduce the results following the README.md, I was instructed to install the dependencies listed on `environment.yml`, but no such file was provided and I had to track them down one by one. There also seems to be a dataset missing (`datasets/preds_cases.csv`) and not all plots from the paper were generated. - Minor writing suggestion: on one hand, the relation to control theory seems currently very incipient and more of a curiosity, but it is presented prominently in a section of its own. On the other hand, the interesting Proposition 3 is cramped in the final section. You might consider adding an independent section for extensions and incorporate the control theory analogy into the final discussion section. - Other minor suggestions: I don't see the point of including the zoom in the figures; it oftentimes hides parts of the figure and the underlying patterns are clear from the whole graph. Besides the many plots, adding a table with a summary of the results among all datasets could help a reader to draw conclusions faster. The meaning of sublinear function is given in line 164, but the term is used before in lines 55 and 67, so we suggest bringing the definition up. Display (5) has a floating \cdot in the last inequality and inconsistent usage (c \cdot h(t) vs -ch(t)). Figure 1 caption says prediction sets are in purple, but it is actually in orange. - Typos: line 115: $(1-\alpha)$ should be $\tau$; line 139: the em-dash is confusing.; line 151: "an particular" should be "a particular; proof of proposition 2: $C$ should be $c$; display under line 192: the second $\text{err}_i$ should be $\text{err}_t$; line 201: "can made" should be "can be made"; line 295: "saturation function as in 2" should be "saturation function that satisfies (5)" or "saturation function as in Proposition 2"line 301: missing \mathrm for err_i.; the bibliography is not printed in the SM and the references do not match the main text, so it ca be confusing.

Questions

- Why is asymptotic coverage the best one can hope for in the adversarial setting? Is this a formal result? It would be helpful to develop this idea a bit more in the text. - What are advantages of using the scorecaster in the conformal prediction setup vs adapting the model instead? - Any guidance on how should one pick $r_t$ and the learning rate? How was this decided for the figures in the main paper? The paper seems to imply trial and error. The impact on figures in the appendix is extremely large! - When do you expect ACI to be better than Quantile+Integrator+Scorecaster? - What are advantages and disadvantages of your proposal vs SPCI, given both achieve coverage of time series without creating infinitely-wide intervals? How does the empirical performance compares? In what ways is your proposal better? - Can you explain what you mean by "Our algorithms may not be universally useful on time series different from those we used for evaluation" (line 283)? When should it be useful? - There is a method called "Trail" in the appendix plots, but no explanation. What is it?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Yes.

Summary

The paper tackles the problem of conformal UQ under distribution shifts. This is a very realistic setting where the upstream prediction model cannot be frequently updated, and we still want to certify some level of safety. The paper cleverly fames the problem as a PID control problem, and provides a practical algorithm that they show is effective on synthetic and real datasets.

Strengths

- The setting addressed is impactful and practical. A major hindrance of using CP in ML systems deployment is that the exchangeability assumption is often violated. This work provides a practical adaptation while still maintaining some guarantees of the method. - The PID framing of the problem & solution is clever and insightful. (very Big Brain, if i may use gen-z lingo.) The two Gibbs & Candes paper from 2021 and 2022 for CP under distribution shifts was able to circumvent the exchangeability assumption and achieve some guarantees by transforming the distribution shift into a online optimization problem for a single parameter $\alpha^*$, which they prove is similar to a P controller. This work takes the idea a step further, and introduces the integrater (I) and forecasting (D) algorithms, completing the picture (fig 2 <- such a nice figure). - The method can utilize time-series modeling approaches to adapt to seasonality. - The experiment section is well-presented. It walks you through part-by-part of the PID controller and shows the impact of each. The examples chosen are clear and convincing.

Weaknesses

1. The paper did not define some concepts it used in its writing. For example: - line 29, exchangeability - line 25, sharpest - line 83, burn-in period - line 137 (not really a definition just... what are you trying to say here?) "the quantile tracker proceeds agnostically and performs the same updates in any case." I think explaining them, even just in the appendix, will help with clarity. 2. The paper only compared to ACI & ablations of its own method. A common choice of algorithm for the scenario introduced by the authors would be Enbpi and SPCI from Xu & Xie (references 29 and 30 in the paper).

Questions

- Can you elaborate on the reason why Enbpi and SPCI from Xu & Xie are not good baselines for comparing your model? Many have used previous timesteps as covariate X, similar to your scorecasting approach.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors provided a good address of limitations in their discussion section.

Summary

This paper examines the issue of how to parameterize conformal prediction in the time series setting. It argues that standard conformal inference methods would not provide valid inferences in the sequential setting, which lacks exchangeability. To resolve this, the paper suggests using online quantile tracking, integrating errors for stabilizing coverage, and utilizing scorecasting in the presence of systematic trends. The authors establish the corresponding theoretical properties, such as asymptotic coverage and conduct several experiments to justify their approach.

Strengths

1. A new method for conformal prediction in the in the time series setting with a clear presentation of the details and background. 2. The authors show the proposed method has a major practical benefit, that unlike ACI (Gibbs and Candes, 2021), no infinite sets are produced. 3. The authors demonstrate the practical utility of their methods with an extensive series of experiments on a wide range of real-time series data sets.

Weaknesses

1. Although this paper present a general theorem that their method achieves asymptotic coverage, there is insufficient discussion on the coverage guarantees such as the lower bounds and upper bounds on coverage. 2. The authors also explicitly avoid addressing the choice of automatic algorithms for the two parameter $C_{sat}$ and $K_I$ in the nonlinear saturation function. 3. In the sequential setting (7), the authors didn’t show how to select the learning rate $\eta$, what factors might affect the selection of the learning rate $\eta$, and what impact this selection have on the performance of the method.

Questions

1. How the specific selection of the function $r_t$ and $\hat{q}_t$ influence the overall performance of the proposed method in conformal prediction for time series? 2. In the context of time series conformal prediction, how does this new method compare with other popular methods in terms of robustness, and computational efficiency? 3. How does a non-constant threshold function $h$ help with tolerating a greater degree of coverage error throughout the sequence?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

1. Despite demonstrating strong results across various experiments, the authors acknowledge potential limitations, including the challenge of proving asymptotic coverage, potential inapplicability to different time series, and limitations in scorecasting. 2. The methods they develop are specific to cases where response $y_t$ is revealed at each time point. However, there are many settings in which we receive the response in a delayed fashion or in large batches.

Summary

This paper aims to provide asymptotically valid conformal prediction (CP) regions for the time series prediction problem. Similar to the adaptive conformal inference (ACI) framework [11], the main idea is to choose the appropriate quantile of the non-conformity scores (or equivalently tuning the miscoverage rate $\alpha$) at each time step to fulfill the required error rate. The framework performs an online quantile update while taking into account the sum of coverage errors overtime to stabilize the coverage better. They provided theoretical analysis that guarantees the asymptotic coverage of the predictions made by their framework. Numerical experiments on real datasets show that their method brings more stable and efficient prediction sets than ACI.

Strengths

In general, I liked the idea of designing a PID-like framework. Thanks to this structure, the prediction regions turn out to be more stable and efficient. Moreover, it ensures that their framework never outputs infinite size prediction sets, which is a big advantage over ACI. Also, they provided many numerical experiments in the paper and the Appendix to illustrate the power of their proposed framework highlighting the effect of each block and different choices of hyperparameters.

Weaknesses

1. The related work section is not well-written in general. Also, it does not represent the works in the intersection of conformal prediction and time series prediction in a fair manner. Indeed many good papers are missing, such as - Foygel Barber, R., Candes, E. J., Ramdas, A., and Tibshirani, R. J. Conformal prediction beyond exchangeability. arXiv preprint arXiv:2202.13415, 2022, - Sun, S. and Yu, R. Copula conformal prediction for multi-step time series forecasting. ArXiv, abs/2212.03281, 2022, to mention a few. 2. The comparisons are limited to ACI. Though it is a natural baseline to compare with, many other CP frameworks are designed for the time series prediction problem. It is interesting to see how they behave compared to the methods proposed in this paper. 3. The scorecasting block seems unnecessary and, to some extent, redundant. Accounting for a wrong choice of learning model (e.g., a model that does not account for the seasonality effect) can not be a part of the design of a CP framework. This point is also mentioned in the paper in lines 286-288. However, it seems that the authors tried to put this block to force the similarity of their framework and the (well-known) PID controller. 4. There are numerous design choices (such as the saturation function) and hyperparameters in the framework, and it's worth noting that several of them may not be immediately intuitive for people in the Machine Learning or Conformal Prediction community when it comes to making appropriate choices. 5. Generally speaking, I am still unsure to what extent it is worth exploring frameworks with an asymptotic coverage property, while the popularity of conformal prediction lies in its ability to provide a finite-sample guarantee. 6. Minor Comments: - There are some typos in Equation 5. - Line115: There is a typo in the definition of the pinball loss (\tau should be replaced with \alpha). - Line 178: The equation reference should be (10) instead of (11). - In the proof of Proposition 2., the capital C has to be replaced with the small c. - Bringing the extensions of the proposed method (e.g., conformal risk control and Proposition 3. or conditional coverage) into the discussion section is unappealing. - In order to facilitate the comparisons for the readers, it is a good idea to report the Avg. size of the prediction sets and Avg. coverage in the figures like what you have in the Appendix figures.

Questions

Isn't it a good idea to provide a concrete explanation of why the proposed framework never returns infinite sets while this can happen in ACI?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors fairly mention the limitations of the proposed framework in the discussion section.