Follow up answers
> What type of details are missing?
> The ones that lead to the questions above.
We will add details regarding the tokens mixing, as described in the rebuttal, as well as precisions regarding RoPE and many other details that were asked by the other reviewers. We hope that will result in a final version with significantly improved clarity with respect to the original draft.
> The Multi Quantile version[...] is not the main contribution of the work, so we had to keep the results and comparison light in this regard.
> Multi Quantile is either a contribution, or it is not. Since you put it second in the list of contributions, it appears to be one. It would help if you placed this work with respect to the relevant literature and compared its performance with relevant approaches. If no existing work applies, explain why.
We agree with the reviewer. We therefore added a literature review on uncertainty extraction in general, and related to regression and forecasting tasks in the manuscript (unfortunately due to space issues, and because we cannot update the rebuttal pdf, we cannot include it entirely but you will be able to observe it in the next revision) Only very few approaches actually exist to attach prediction intervals to regression estimates, as it will be mentioned in the additional literature review. Ensemble and bootstrap methods would really not be the most efficient here as our model is still a bit heavier than model typically used with such methods (regression trees etc), and the approach suggested by Metnet (Sønderby, Casper Kaae, et al. "Metnet: A neural weather model for precipitation forecasting." arXiv preprint arXiv:2003.12140 (2020).), meaning separating the prediction into multiple value bins, and predicting the probability of each of the bins for each time step, might work, but would be in our opinion less expressive, as one map head would predict multiple bins instead of having multiple specialized heads. It is in a way a generalization of the same concept.
> We do evaluate its median prediction [...] and its ability to include the observed true values.
> None of that measures the uncertainty estimation quality, the module's main goal.
When estimating prediction intervals, it is common practice to evaluate the quality of the estimation by examining the fraction of test points that fall inside the corresponding prediction intervals, which is precisely what we do. Note that most works, including the ones you mention in your next comment, tackle the evaluation of uncertainty regarding classification tasks (meaning taking into account the uncertainty of probabilistic outputs when converting them into classes) and only a handful tackle its regression counterpart. Note also that the evaluation of such prediction intervals attached to regression estimates is surprisingly challenging and a topic of research on itself, as pointed out by Sluijterman et al, 2023 (« How to Evaluate Uncertainty Estimates in Machine Learning for Regression? »), in a very recent work. Sluijterman et al acknowledges that is one of the most common ways of doing so, yet advocates for simulation-based approaches. We will definitely explore these new ways of evaluation in the future, but for the time being for time concerns, we shall keep our evaluation scheme as it is if it is ok with the reviewer.
> Do you have suggestions for more things we can include?
>Here is a non-exhaustive list of relevant work[...]
Thanks a lot for these suggestions. We included them in the new related works paragraph regarding uncertainty extraction, along with many other references closely related to our problem.
> CrossViViT without RoPE
> Reading the other rebuttal, I see that you replaced RoPE it with "a learned PE". Can you give details about this baseline? [...]
I would be inclined to increase my rating if the RoPE experiment turned out to be valid, and if the authors can correctly situate and evaluate their uncertainty prediction module (or explain convincingly why they cannot).
Indeed, RoPE is ablated against a standard learnable positional encoding as was done in the Video VIT and VIT papers. The learnable positional encoding is a learnable parameter $\mathbf{p}\in\mathbb{R}^{N\times d}$ where $N$ is the number of tokens in the sequence and $d$, the embedding dimension. We chose the learnable positional encoding scheme instead of the fourier-based one because the former is a standard choice and can be more felxible (if correctly learned) than a static embedding.