Conformal Prediction for Time Series with Modern Hopfield Networks

Summary

This paper proposes a novel method for estimating uncertainty in prediction of time series. Modern Hopfield Networks is used as a method of measuring uncertainty based on the existence of similar historical data. This paper presents its construction method and shows that it works for multiple data and forecasting methods

Strengths

The authors have clarified the issues with CP for time series and proposed a method to improve this point by applying MHNs, and experiments have shown very good results.

Weaknesses

My four points of concern are as follows: - This paper consists only of a method proposal and an experimental evaluation, and does not include theoretical effects. In this structure, is an evaluation on four data sets sufficient to demonstrate the effectiveness of the proposed method? - The task setting of this paper seems to be related to Gaussian processes and Bayesian optimization as well. This point seems to be a natural one for many people, and while it may not be directly comparable, would it be appropriate to make no mention of it? - The proposed method incorporates MHNs, a function that calculates the reliability of forecasts from past information, into CP. The results shown in the experiment indicate the validity of the CP issue and the effectiveness of the improvement direction. However, the experiments did not provide justification for introducing HMNs to solve the problem. - Identification of issues and presentation of solutions is an important contribution, but it is a combination of existing methods, and I feel it is weak in terms of novelty.

Questions

- Regarding the first point of Weakness, can you state why these data are sufficient, although they are a limit to the number of experiments? Or can they be increased? - For the second point, for example, can you compare it to or give reasons why it is not comparable to the following advanced methods of Gaussian processes - X. Zhu et. al, BayesianTransformedGaussianProcesses, TMLR (4/2023) - G. Corani et. al, Time Series Forecasting with Gaussian Processes Needs Priors, ECML PKDD 2021 - X. Sun et. al, Recurrent neural network-induced Gaussian process, Neurocomputing 509(2022) - For the third point, can you specify why HMNs is more effective than other methods that can assume similar effects?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The authors do not explicitly address Limitation. On the other hand, the study is oriented toward reducing the social impact on the limitations of conventional methods and does not promote adverse effects.

Summary

The work presents HopCPT, a conformal prediction algorithm for Modern Hopfield Networks in the setting of time series forecasting. The work uses a novel (attention-like) weighted quantile approach to leverage similar time steps from the past, improving both coverage and sharpness compared to SOTA CP methods. The proof mainly extends from (Foygel Barber et al., 2022) and (Xu and Xie 2022). The method is shown to perform well on both synthetic and real world datasets.

Strengths

- Novel and sound CP algorithm on a widely-used architecture for forecasting. - Excellent efficiency performance on time series datasets, with through comparisons to existing methods. - I also appreciate the use of proper scoring rules for evaluation (Winkler score).

Weaknesses

- From the paper alone, it's difficult to tell what exactly are the theoretical guarantees of your algorithm. While the authors did refer to Appendix B, I think it is important to briefly describe the assumptions and state the main bound (B.8 and B.9) of your theoretical analysis in the paper.

Questions

1. Does your method assumes stationarity of the time series? because the $a_i$s are fixed, 2. Equation 7. Would searching for a $\beta$ as in SPCI (Xu & Xie 2023) instead of using $\alpha/2$ using further improve efficiency? 3. I'm not sure point 4 in your main contributions is worded in the best way - your guarantees are not specific to hydrological prediction. Maybe a better way to say it is that HopCPT is the first algorithm with marginal coverage guarantees that can be applied to hydrological prediction applications? 4. Given the current soft-selection algorithm, since the $a_{t+1}$ s are static, would HopCPT still perform well when the seasonality in data is not regular? What if there are drifts in the distribution?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- It might be good to stress that the guarantee is marginal and asymptotic, which can be achieved by rather simple algorithms (see Gibbs & Candes 2021, 2022). This is of course due to the properties of time series data, not a weakness of the approach.

Summary

This paper deals with providing conformal prediction (CP) sets for the time series prediction problem or, more generally, the cases where the exchangeability assumption is violated. Similar to the prior work of Foygel Barber et al. (2022), they also try to assign weights to the points in the calibration set with higher (lower) values for those who are coming from a more (less) similar region (aka regimes) as the test point. For such a weighting mechanism, they use Modern Hopfield Networks (MHN) that are capable of finding regions similar to the point for which a prediction set is sought. Using numerical experiments, they showed the superiority of their proposed framework.

Strengths

The paper is indeed well-structured, nicely written, and easy to comprehend. The extensive amount of numerical experiments supports their claim about the (empirical) superiority of their proposed framework over the vast range of competitors.

Weaknesses

The contribution of this paper seems to be relatively marginal. Indeed, the authors introduced a new way of weighting the non-conformity scores (i.e., reforming the distribution of non-conformity scores) on top of the NexSCP framework proposed by Foygel Barber et al. (2022) while adopting the structure proposed by Xu & Xie (2022a) for interval constructions. While the numerical results show their framework's (empirical) superiority in terms of efficiency and coverage, one should keep in mind that it adds relatively high computational complexity to the model. This is, of course, also reported in Appendix A.6.

Questions

1. What is the justification behind the choice of encoding network (section 2.3)? Apparently, for the current experiments, it is working adequately. However, why should one stick to such a structure for every other dataset? 2. I still do not understand the role of $W_q$ and $W_k$. How are they learned? What is the difference between them?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors mention a limitation of the proposed framework: the need for large data for HopCPT to perform well. However, there is still room for elaboration on other limitations.

Summary

The authors propose a conformal prediction approach for black box time series forecasting models. The method is based on learning to predict the model’s errors using a small neural network structured similarly to the softmax computation in an attention mechanism. The prediction interval of the proposed method utilizes the softmax scores in the error sampling procedure, drawing each error with the probability given by the softmax score. Both in synthetic and benchmark experiments, the proposed method outperforms other CP approaches in terms of PI-width and coverage gap.

Strengths

* The manuscript is extremely well structured and very easy to follow. The authors provided an excellent overview of the conformal prediction task and relevant work. * The experiments provide a thorough insight into how the proposed method functions in a synthetic and more realistic setting. Figure 2 contrasts the proposed with competing methods in a comprehensible and straightforward way. * In addition to a solid theoretical motivation, the proposed method performs outstandingly and convincingly on a wide range of prediction models and several data sets.

Weaknesses

* Without a background on Hopfield networks, the method seems an application of the attention idea to conformal prediction. It would be helpful to lay out the similarities and differences between MHNs and the attention mechanism. * I am missing a section on the limitations of the proposed method. For instance, how does the method scale to very long time series?

Questions

* How is the actual marginal miscoverage $\alpha^{*}$ computed? * Could you elaborate on the meaning of $\delta_{+\infty}$ in Eq. 3 and how are weights $w_i$ computed? Is it model-dependent? * Does the bound in Equation 5 apply to your method? * Is it possible to make Figure 3 less dense and make the connection to the prediction interval more comprehensible? * I am slightly confused about the notation in Eq. 3. The quantile should be computed on a set, but I am not sure from which values it is computed. Can you rewrite this in set-notation like Eq. 2?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

While there is no dedicated limitation section, the authors provide “negative results” in the appendix. A concise paragraph about the method’s limitations would be appreciated.

Thank you for your clarification.

Thank you for your clarification. I received clear answers to the first three questions. I cannot confirm the revised version due to the rules at this time, but I hope it will be revised so that the entire required content can be captured in the main body. Regarding the fourth point, I recognize the novelty of the idea of introducing HMNs in CP. Please clarify whether the idea is the bulk of the contribution, or whether it is non-trivial to come up with the idea and then build a specific algorithm, and whether there is a contribution in that regard as well.