While diffusion models can successfully generate data and make predictions, they are predominantly designed for static images. We propose an approach for efficiently training diffusion models for probabilistic spatiotemporal forecasting, where generating stable and accurate rollout forecasts remains challenging, Our method, DYffusion, leverages the temporal dynamics in the data, directly coupling it with the diffusion steps in the model. We train a stochastic, time-conditioned interpolator and a forecaster network that mimic the forward and reverse processes of standard diffusion models, respectively. DYffusion naturally facilitates multi-step and long-range forecasting, allowing for highly flexible, continuous-time sampling trajectories and the ability to trade-off performance with accelerated sampling at inference time. In addition, the dynamics-informed diffusion process in DYffusion imposes a strong inductive bias and significantly improves computational efficiency compared to traditional Gaussian noise-based diffusion models. Our approach performs competitively on probabilistic forecasting of complex dynamics in sea surface temperatures, Navier-Stokes flows, and spring mesh systems.
Paper
Similar papers
Peer review
Summary
This paper proposes DYffusion, an approach that mimics the diffusion models for spatiotemporal forecasting. It treats the noising process as interpolation (parameterized by $I_\phi$) and the denoising process as forecasting (parameterized by $F_\theta$), i.e., it reimages the noising step $T$ in the original diffusion models as the temporal step in forecasting. DYffusion parameterizes the probabilistic transitions in forecasting continuous-time trajectories, which is similar to parameterizing the diffusion of SDEs. It performs competitively on spatiotemporal forecasting tasks, including sea surface temperatures, Navier-Stokes flows, and spring mesh systems, in terms of probabilistic skill score metrics.
Strengths
1. DYffusion reimagines the continuous-time probabilistic forecasting problem as a diffusion process. It benefits from existing diffusion algorithms and methods to accelerate the inference sampling. 2. DYffusion achieves the best or competitive scores while significantly reducing the time cost compared to diffusion approaches.
Weaknesses
1. Eq. (6) is incorrect. There should be a differential instead of a derivative in the integral. 2. DYffusion's method is closely related to neural ODE and SDE. However, empirical studies lack corresponding baselines. For example, one possible baseline could involve extrapolating $dF_\theta/ds$ using an ODE solver. 3. It is unfair to use $F_\theta$ trained with Algorithm 1 in the baseline method Dropout. The forecaster used in Dropout should be trained with an objective such as $||F_\theta(x_{t+i}, i)-x_{t+h}||^2$ instead. 4. One of the main contributions mentioned is that DYffusion reduces complexity. However, a complexity analysis is missing. 5. Neural SDE [1,2,3,4] parameterizes the stochastic dynamics for modeling continuous-time processes and is therefore inherently suitable for probabilistic continuous-time forecasting. However, there is no discussion on a range of related works on neural SDE. [1] Deng, Ruizhi, et al. "Modeling continuous stochastic processes with dynamic normalizing flows." Advances in Neural Information Processing Systems 33 (2020): 7805-7815. [2] Liu, Xuanqing, et al. "Neural sde: Stabilizing neural ode networks with stochastic noise." arXiv preprint arXiv:1906.02355 (2019). [3] Jia, Junteng, and Austin R. Benson. "Neural jump stochastic differential equations." Advances in Neural Information Processing Systems 32 (2019). [4] Kidger, Patrick, et al. "Efficient and accurate gradients for neural SDEs." Advances in Neural Information Processing Systems 34 (2021): 18747-18761.
Questions
Why does Dropout perform so poorly at $t=2$ in Figure 3? Does this observation indicate the mentioned Weakness 3?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
This paper includes a separate section for discussing limitations. There is no potential negative societal impact.
Summary
In this paper, the authors propose to build on diffusion model for modelling spatio-temporal data. In particular, they first train a time-dependent interpolation network which learn to interpolate the temporal dynamics given a frame at the horizon time $x_{t+h}$, a frame $x_t$, and an index $i$ interpolating between the two distributions. Then, the idea is to frame a forecast model as a denoising model between $x_t$ and $x_{t+h}$, that is given an interpoled frame to be able to recover the original 'denoised' $x_{t+h}$ frame. This approach allows for memory efficient multi-step predictions, as opposed to existing diffusion models for videos. They apply the developed model to the forecasting of sea surface temperatures, Navier-Stokes flows, and spring mesh systems.
Strengths
- The main advantage of the proposed approach seems to be the small memory footprint as it is constant w.r.t. the horizon, in contrast with standard dynamics forecasting (if I understand correctly). - It also allows for 'upscaling' the resolution, or for making prediction at any time value, thus is not constrained to a regular discretisation of the time axis. - The experiment on the sea surface temperatures dataset is quite promising, as the proposed method is able to accurately forecast not only the predicted mean but also the uncertainty with a lower computational cost than the video diffusion model MCVD. It would be valuable to explore whether one could obtain better performance at the expense of more computational cost.
Weaknesses
- I found the writing to be confusing at times, I think that Section 3 could be better conveyed, in particular the choice of notations, but also the relation with diffusion models, and the 'cold posterior' sampling scheme.
Questions
- Equation 3: How is the architecture handling the two inputs $x_t$ and $x_{t+h}$? By doubling the number of input channels? - line 119: Why do we need this? This deserves more explanation. - Equation 4: Why do we need the interpolator network here? Can't we directly predict $s^{(n+1)}$ with $F_\theta(s^{(n)}, i_{n+1})$? Aren't we actually extrapolating with the interpolator here? - Algorithm 2: What is happening line 4? How does this iteratively refine the prediction of $x_{t+h}$? This is likely worthy of some more explaination. - Equation: Isn't this ODE implying that the dynamics is Markovian? - line 249: What is the Continuous Ranked Probability Score (CRPS)? Worth at least giving a high level idea I think. - line 268-269: Why not training the DDPM with more diffusion steps? Since the number of steps is an hyperparameter for all methods it would be easier to compare them with the same number of steps, albeit could be varied to see how performance vs computation evolves. - Table 1: Would you know why the performance of the models is widely different betweent the SST and the Navier-Stokes datasets?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
1 poor
Contribution
3 good
Limitations
N/A
Summary
This paper proposes a new forecasting model for spatiotemporal data. The idea is based on separately training an interpolator and a forecaster network and applying them in an alternating fashion at inference time to iteratively refine the forward prediction. The inference procedure loosely resembles the denoising process in a diffusion model, where one computes a new denoising target and moves slightly towards it at every step. Numerical experiments are conducted on several datasets to validate its skills.
Strengths
* The idea of iteratively refining forward predictions is interesting and quite novel to my knowledge. * The paper is generally well written and easy to follow. * The numerical experiments cover multiple non-trivial tasks and show moderate improvement in the metrics against benchmarked diffusion-based and ensemble prediction models.
Weaknesses
* Although advertised as a diffusion model, the actual connection (at least to diffusion-based generative models) is handwavy at best. It is not clear that the underlying process arising from the defined noising and denoising operations form a well-defined generative model (for example see how DDPM is derived with clearly defined conditional and marginal distributions at each step). Only the variational posterior is provided in equation (4). * The model is mostly deterministic (except for the last step that injects some Gaussian noise) so I think it’s important to show comparison against deterministic forecast models. The high memory footprint for multi-step training is somewhat fair, but there exist ways to improve single-step models (e.g. noise injection, curriculum training, etc.) which should be considered when validating the proposed model. This is necessary to justify doing the additional steps of interpolation. Overall I feel in its current state given the strengths and weaknesses I vote for borderline reject tentatively.
Questions
* Equation (4), the second line is not a probability. Do you mean a $\delta$ distribution at the interpolator output? * Also equation (4), adding Gaussian noise only at the last step looks a bit cryptic to me. What is this trying to achieve besides adding randomness to the prediction and how is the noise level $\sigma$ determined?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
The authors mention that input and output space must be the same and that inference costs are higher than predicting directly.
Summary
In this paper, authors tackle the long-term forecasting problem applied to dynamics system. To solve this problem, they propose to use diffusion principle along with interpolating and forecaster mechanisms. The former interpolates timestep data in between lookback and target windows (therefore, at a lower resolution). Authors evaluate their proposal on three datasets, analyze and interpret the results as well as further discuss directions for improvements.
Strengths
* Interesting approach * Extensive ablation study, with summary in the main paper section * Interesting discussion of the results
Weaknesses
* Reproducibility (no code, Navier-Stokes and Spring-Mesh experiment set-ups are presented in the table caption, but I wonder if they are complete for anyone who would like to re-run the same experiments) * Comparison with the main reference [43] is in my opinion very subjective and should be improved (as authors build-up on [43] for two datasets, it seems to me that [43] should be included as a baseline) * Require some proof-reading
Questions
The proposal is very interesting and sound promising. However, in my opinion, we are missing a representation of the proposed architecture to better understand the paper and how each module interacts with each other. ## Reproducibility The details for training /validation and testing for Navier-Stokes (NSFlow) and Spring-Mesh (SM) are not provided in the dataset description, but in the experiments caption. And yet we can wonder if they are complete, which could make more difficult to reproduce results. ## Performance comparison I think we are missing the time column in table 2 and the one for NSFlow experiment in Table 1. `For the out-of-distribution test set of the Navier-Stokes benchmark, the results are almost identical to the one in Tab. 1, so we do not show them.` Why not include them in the appendix and let readers be the judge and make their own opinion? ## Comparison with reference [43] As authors build up on ref. [43] models for NSFlow and SM, why are they not comparing with the model (unet and CNN) from such a reference. Especially looking at results from [43], for NSFlow from Figure 5, unet seems to achieve MSE between 0.05 and 0.007. These variations depend on the number of obstacles and there might be some difference in the set-up, but it would be good to see how DYffusion do compared to these baselines. As is it does not look like to me that the proposed model is doing better contrary to what authors are claiming. Other readers might feel the same and wonder what is the advantages of DYffusion. As a consequence, it would be good to define precisely the condition of NSFlow and SM and compare with original model. `It is worth noting that our reported MSE scores are significantly better than the ones reported in [43]` I have difficulty agreeing with this claim. First, results in [43] are not presented in a table format so it is more difficult to compare. Authors should either provide their results with the same representation (stepMSE vs. OoD MSE) for instance in the appendix, for reader to better judge and compare both solutions. Or auhtors should include [43] models as mentioned above as a baseline in their table 2 (or both). While clearly setting the configuration of the forecast to make sure that we are dealing with similar set-up. `our MSE results significantly surpass the ones reported in Fig. 8 of [43]`: Again, if authors wish to compare their results with model from [43], why not running forecast with CNN from reference [43] in the same condition? As Authors, in this paper, are re-using CNN from ref. [43], it should then be possible and prevent readers to go check paper [43] to make sure the settings are the same and try to grasp the similitude between papers as each paper use different representations of the results. In addition, if CNN performed so poorly in [43], why choosing it as the base model for SM dataset? Why not choose an MLP or nn kernel that seems to perform better on the figure mentioned by authors? ## Model evaluation `resulting forecast to be visibly pleasing and temporally consistent` Not really a scientific judgement in my opinion…But it is indeed quite similar. Nevertheless, at t=3.70, the velocity of sample 2 to 5 are not really “visibly pleasing”, how should we interpret this? ## Ablation study In my understanding, the proposal relies on the forecaster and interpolator. Therefore, I would have expected to see an ablation study, without these modules to judge of their importance. Is it impossible to do so? Is it not possible also to do ablation study with different frame number of the interpolator (k-1)? BTW, unless I missed it, we don’t know what the value of k for each dataset is. Does it have an impact on the performance? ## Proof-read I found some points to be corrected and authors should pay some more attention on possible other typos or issues: * Line 25. and and require -> and require * Line 92, should not it be x_{t+1}:{t+h} * Line 196, Sentence is a bit long, consider breaking it * Line 271, In the spring mesh dataset -> In the Spring Mesh dataset * Line 288, whether it is actually able -> whether it is actually possible * Line 302/303, For the Navier Stokes and spring mesh dataset it was sufficient -> For the Navier Stokes and Spring Mesh dataset, it was sufficient * Line 327, could be the reason for why -> could be the reason why
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
3 good
Limitations
Authors discuss their results and limitations as well as future directions.
Thank you for these clarifications. I am satisfied with the answers, except with the comparison to [43]. Even if models in 43 are deterministic, if your goal is to have the best single- and multi-step predictions, your probabilistic model should beat probabilistic SOTA and deterministic SOTA. If you consider that deterministic and probabilistic are different domains, then don't mention that your model is doing better than 43 in the first place. However, I would like to emphasis that all the details and additional explanations provided here should be included in the new revision of the paper (either in the main paper or appendixes). Authors should give extra attention that all their equations, experiment set-ups and results be easily understandable for other to reproduce them if necessary. Authors should also be careful of the typos (in the rebuttal document, in the caption of figure 7, the reference should be [43] not [44], right? Same for Table 12? Maybe you added a new reference which change the order and now 43 is 44, but you should double check it and adapt it in the Figure 7 legend).
Thank you for your response and taking the time to carefully read our rebuttal. We are glad to hear that all answers except the following point satisfied you. > I am satisfied with the answers, except with the comparison to [43] It is unrealistic to expect a probabilistic model to ``beat’’ both probabilistic SOTA and deterministic SOTA because they are fundamentally different classes of models with very different objectives and evaluation criteria. A probabilistic SOTA is not optimized for point estimates, typically evaluated by MSE. Instead, probabilistic models emphasize on calibration and coveraged, as reflected in CRPS. Therefore, they are not comparable to a deterministic SOTA. **However, we now see the benefit of including a direct comparison to the models from [43] in terms of MSE, which is why we will include the new figure 7 of the joint rebuttal PDF, plus discussion, into our revised paper**. *Please let us know if there is any other benchmarking against deterministic models or [43] that you think is important to include.* > if your goal is to have the best single- and multi-step predictions We would like to stress that our goal is NOT to "have the best single-step predictions", but rather we focus on multi-step probabilistic forecasting, and especially long rollouts in this paper. This is an important distinction, because while the models from [43] are appropriate for single-step predictive skill comparison, these models lack stability for long-term forecasts (the focus of this work). We demonstrate this in figures 7a and 7b in our joint rebuttal PDF, where the models from [43] diverge or significantly underperform compared with our method AND baselines in terms of MSE. This challenge is why we focus on multi-step/long-term probabilistic forecasting. > However, I would like to emphasis that all the details and additional explanations provided here should be included in the new revision of the paper (either in the main paper or appendixes). Rest assured that we will include all these details to our revised paper or appendix. > Authors should also be careful of the typos (in the rebuttal document, in the caption of figure 7, the reference should be [43] not [44], right? Same for Table 12? Maybe you added a new reference which change the order and now 43 is 44, but you should double check it and adapt it in the Figure 7 legend). Thank you for pointing this out, you are correct that [44] in our joint rebuttal PDF is [43] from our submitted main paper. We apologize for this inconsistency that arises because we strived to integrate the reviewers feedback directly into our revised paper. In the final version this will not be a problem.
Thank you for these additional responses. > It is unrealistic to expect a probabilistic model to ``beat’’ both probabilistic SOTA and deterministic SOTA Perhaps, but that very important to have the global picture of where your proposal is beating SOTA (no matter the SOTA deterministic or probabilistic) and where it fails. Where it fails might be room for improvements. Especially, in your rebuttal figure 7, even though your proposal is not beating ref. 43 on short-term horizon, it seems very close to it. Therefore, your proposal might be a good trade-off when dealing with both short- and long-term forecast. These types of results and comparison emphasis what you mentioned: > we focus on multi-step probabilistic forecasting, and especially long rollouts in this paper. This is an important distinction, because while the models from [43] are appropriate for single-step predictive skill comparison, these models lack stability for long-term forecasts For anyone in the community not clear with difference between deterministic and probabilistic models, this kind of information is gold and on top of that it is back up by results. And Figure 7 makes this very clear. In my opinion, with such details you provide the full picture by doing a fair comparison and a clear conclusion: as you expected, your proposal excel in long-term horizon, while being competitive on short-term horizon. Considering that Authors will attach importance to include all these details to their manuscript, I increased my score from 5 to 7.
We deeply appreciate your feedback and careful consideration of our rebuttal, resulting in a higher score. We believe that your comments have significantly improved our paper and allowed us to more effectively convey its significance to the community. As mentioned before, this will all be reflected in our final version of our paper.
Thank you for the clarification on the differences between DYffusion and neural ODE/SDE that I had overlooked. Based on your helpful explanation, I have some additional questions to better understand your work: Given that the horizon $h$ remains fixed once chosen, and considering that the training of the interpolator $\mathcal{I}$ is independent of the forecaster $\mathcal{F}$, does this imply that alternative forecasting models capable of predicting $x_{t+h}$ from $x_t$ can also be employed to achieve continuous forecasting in conjunction with the trained interpolator $\mathcal{I}$? If such is the case, it would be beneficial to conduct further ablation studies to elucidate the **necessity and advantages of the forecaster** $\mathcal{F}$. These studies could shed light on how the designed diffusion training and inference contribute to the efficacy of the forecaster. For example, **exploring diffusion settings in common practice** could be insightful, i.e., investigating a forecaster that generates $x_{t+h}$ by denosing from Gaussian noise conditioned on $x_t$ might be worth considering, especially given the lack of evidence indicating that the intermediate $\hat{x}_{t+i_n}$s are utilized. It would be valuable to address why such a configuration was not selected and how it contrasts with the chosen approach. As for neural SDE on high-dimensional data,I would like to suggest considering Neural-SPDE [5], which applies neural SDE to Navier-Stokes equations in a 64x64 grid setting. Its codebase provides implementations of Neural-SPDE as well as various baselines, including NCDE and NRDE. I recommend leveraging these baselines to showcase the effectiveness of DYffusion against neural ODE/SDE based approaches, if time permits for an empirical comparison. [5] Neural Stochastic PDEs: Resolution-Invariant Learning of Continuous Spatiotemporal Dynamics, NeurIPS 2022.
We genuinely thank you for carefully reading through our rebuttal and acknowledging the significance and novelty of our approach, especially compared to neural ODE/SDE methods. > Given that the horizon ℎ remains fixed once chosen, and considering that the training of the interpolator $\mathcal{I}$ is independent of the forecaster ${F}$, does this imply that alternative forecasting models capable of predicting $x_{t+h}$ from $x_t$ can also be employed to achieve continuous forecasting in conjunction with the trained interpolator $\mathcal{I}$? Yes, in principle, this would be possible. > If such is the case, it would be beneficial to conduct further ablation studies to elucidate the necessity and advantages of the forecaster ${F}$. Please note the new figure 5 that we will include to our revised appendix (anonymous gdrive link: https://drive.google.com/file/d/1jFYdEn1tAkJ--HsndoXRndjJ2X_2qPTk/view?usp=sharing), where we ablate exactly this. Here, we show that the forecasts of $x_{t+h}$ of the forecaster $F$ gain more skill as the reverse diffusion process progresses (i.e. as the corresponding time of the diffusion step comes close from $t$ to $t+h$). > For example, exploring diffusion settings in common practice could be insightful, i.e., investigating a forecaster that generates $x_{t+h}$ by denosing from Gaussian noise conditioned on $x_t$ might be worth considering We thank the reviewer for this idea, and agree that this is an interesting method to consider. Honestly, we did not think of this method since it has not been proposed or used before. We believe that future work can explore if such a method will actually perform well. In this paper, we focus on our own proposed method, common multi-step forecasting approaches, and video diffusion models as baselines. Notably, while video diffusion is the most straight-forward approach for applying conventional diffusion models to multi-step forecasting, it had not been applied before to this problem. > (..) given the lack of evidence indicating that the intermediate $\hat{x_{t+i_n}}$ s are utilized This is wrong. When the optional line 6 of Alg. 2 is disabled, the intermediate $\hat{x_{t+i_n}}$ are used as forecasts for the intermediate timesteps between $t$ and $t+h$. This is what is actually being done in our SST dataset experiments. In addition, you can see in our ablations table (*No ref.* row in Table 5 in the appendix) that utilizing these intermediate $\hat{x_{t+i_n}}$ works well (but can sometimes be outperformed by enabling line 6 in Alg. 2). > As for neural SDE on high-dimensional data,I would like to suggest considering Neural-SPDE [5], which applies neural SDE to Navier-Stokes equations in a 64x64 grid setting. Its codebase provides implementations of Neural-SPDE as well as various baselines, including NCDE and NRDE. I recommend leveraging these baselines to showcase the effectiveness of DYffusion against neural ODE/SDE based approaches, if time permits for an empirical comparison. We thank you for the pointer to this baseline. In the limited time frame we had, we have done our best to run neural SPDE as a baseline, focusing on the spring mesh dataset since it has the lowest dimensionality amongst all our datasets. Please see the resulting figure in the following anonymous gdrive link: https://drive.google.com/file/d/1IoQuNvNKAphrLbBX7zUHTPsd2V5S96Po/view?usp=sharing This figure is the same as Fig 7b) of our joint rebuttal PDF, but extended by your proposed neural SPDE baseline. As you can see, our baselines and DYffusion outperform it. To attain this result, we run neural SPDE with ``n_iter=1, modes1=100, modes2=100, hidden_channels=32, solver=’fixed_point’ ``and a training horizon of ``h=100``. We note that increasing ``n_iter`` leads to significantly worse performance, and similarly for reducing the number of modes. We also would like to note that to our understanding neural SPDE models the noise W to be a sample path from a Wiener process, but does not really model the distribution of the forecast. Thus, the resulting forecasts are deterministic. Lastly, neural SPDE shines when the noise forcing is known (in their paper results $u_0 \rightarrow u$, where the noise $\xi$ is unknown, neural SPDE does not perform significantly better than baselines), which is not the case for our datasets or can be expected in real-world applications.
Thanks for your clarifications and I apologize for not registering the obvious stochasticity introduced by inference dropout in the interpolator. That said, I do not fully agree that deterministic models are irrelevant here as (a) comparison is done against MSE which is a deterministic metric and (b) probabilistic predictions may be obtained from deterministic models using ensemble with perturbed initial conditions. Uncertainties in probabilistic predictions inevitably contain both model errors and internal variability of the system and the latter is reflected in a setup like (b) to the very least. I have another follow up question regarding inference speed: how does the cost of the interpolator compare to that of the forecaster? The reason I am asking is that it is more fair to compare performance under the same constraint. For example, one might argue that the fair comparison would be to run an autoregressive model with half the step size (assuming that the autoregressive stepper is similar to the forecaster and interpolator alone). Superior metrics in this benchmark would be much stronger evidence that the guessing long into the future and iteratively refining it is indeed better than looking short-term only.
Thanks for acknowledging the misunderstanding. We hope that you find our manuscript, and especially its originality and significance to the community, more convincing now. > That said, I do not fully agree that deterministic models are irrelevant here as (a) comparison is done against MSE which is a deterministic metric and (b) probabilistic predictions may be obtained from deterministic models using ensemble with perturbed initial conditions. Uncertainties in probabilistic predictions inevitably contain both model errors and internal variability of the system and the latter is reflected in a setup like (b) to the very least. We agree. For point (a), please see our new comparison against the deterministic models from [43] in Fig. 7 of our joint rebuttal PDF. For point (b), please note that we explicitly include this baseline in our Table 1, see the ``Perturbation`` row. > I have another follow up question regarding inference speed: how does the cost of the interpolator compare to that of the forecaster? This depends on the choice of the corresponding network architectures. Usually, the cost of the interpolator will be at most that of the forecaster. This is because interpolation is an easier task than forecasting, so we can expect to do fine by using an architecture with the same or lower complexity as the forecaster. In our experiments and for simplicity, all interpolator and forecaster networks share the same architecture for the respective datasets. As noted in the appendix B.5.1, we do halve the hidden dimensions of the interpolator relative to the forecaster network on the SST dataset. > The reason I am asking is that it is more fair to compare performance under the same constraint. For example, one might argue that the fair comparison would be to run an autoregressive model with half the step size (assuming that the autoregressive stepper is similar to the forecaster and interpolator alone). Superior metrics in this benchmark would be much stronger evidence that the guessing long into the future and iteratively refining it is indeed better than looking short-term only. We understand your concern. Unfortunately, we already train/evaluate on the highest possible temporal resolution, so it is not possible to half the step size. That being said, this proposed baseline would be effectively training on a horizon of half the length of before, and requiring twice the number of autoregressive steps for our full rollout evaluations on Navier-Stokes and spring mesh. The potential enhancement of rollout metrics through such an approach remains uncertain. It is worth noting that we have duly addressed and demonstrated that DYffusion operates at a slower pace compared to certain single-forward pass baselines, as substantiated by Table 1 and our discussion within the limitations section. Notably, the reassuring aspect lies in the fact that DYffusion's heightened computational demands during inference, though still more efficient than conventional diffusion models, consistently correlate with improved predictive performance in comparison to the established baselines.
Thanks for your response. Despite still having my own reservation about the last point, I will raise my score to 5.
Final summary on the originality and significance of your work
Dear authors, Thank you for submitting your work to NeurIPS, responding to the initial reviews provided by the reviewers, and engaging in subsequent discussions with them during this author-reviewer discussion phase. As you can see, even though we are approaching the end of this discussion phase after much deliberation, there are still divided opinions among the reviewers regarding the acceptance decision. As such, may I invite you to give a final summary in light of the further discussions with the reviewers? In particular, please address specifically the originality and significance of this work to justify its acceptance for presentation to the NeurIPS community. Originality and significance, among others, are major review criteria for acceptance, as you can see from the NeurIPS reviewer guidelines: https://nips.cc/Conferences/2023/ReviewerGuidelines. We look forward to your summary (or “pitch”) soon. Best, AC
Pitch
## Originality: Our proposed method, DYffusion, is a completely new approach for performing probabilistic multi-step spatiotemporal forecasting. This stands in a refreshing contrast to the most common approaches for spatiotemporal forecasting, which tend to focus on deterministic models and single-step training. These approaches have limitations: - Deterministic models are ineffective for uncertainty quantification. Perturbing the initial conditions can generate multiple forecasts from a deterministic model, but this underperforms in our results (see Perturbation baseline). Further, they are commonly trained on MSE loss, which leads to mode collapses, especially in long rollouts, and physically unrealistic or blurry forecasts. - Single-step forecasting training leads to a deviation between training and inference, since dynamics forecasting models are usually used for long rollouts and long-term forecasting. As a result, single-step forecasting models tend to collapse or underperform when used for long rollouts (see our Fig. 7 in the global rebuttal PDF). DYffusion presents a natural solution for both these issues, by designing a temporal diffusion model (leads to naturally training to forecast multiple steps) and embedding it into the “generalized diffusion model” framework so that by taking inspiration from existing diffusion models we can build a strong probabilistic forecasting model. In addition, DYffusion is the **first** diffusion model that relies on a task-informed forward/reverse process (here, we propose to use temporal interpolation and forecasting). All other existing diffusion models, albeit more general, use data corruption processes. As a result, our work gives a new perspective on designing a capable diffusion model, and may lead to a whole family of task-informed diffusion models. ## Significance: Accurate high-dimensional forecasts with calibrated uncertainty are of high practical relevance but challenging. DYffusion is the first diffusion model to employ dynamics-informed forward and reverse processes, effectively addressing both probabilistic and multi-step forecasting—often neglected by existing deep learning approaches. This pioneering solution yields strong empirical results and opens exciting avenues for further improvements. Our method shows significant improvement over the state-of-the-art video diffusion model in speed, memory footprint, and forecasting performance. DYffusion is the best or second-best model on all three benchmark datasets. It lowers the barrier for training/using diffusion models for dynamics forecasting, and performs strongly on challenging datasets where video diffusion models underperform or fail (here, Navier Stokes and spring mesh). ## Central concerns by the reviewers ### Misunderstanding by Reviewer 7msc The main criticism from Reviewer 7msc is our model being `` mostly deterministic’’ whereas our model is designed for **probabilistic forecasting**. They have acknowledged this misunderstanding in the comment. We also provided detailed answers to their follow-up questions on (1) comparison with deterministic models (2) inference speed. ### Baselines by Reviewer 4TEk The central concern raised by Reviewer 4TEk is the lack of "baselines in neural ODE and neural SDE’’ because of DYffusion being "closely related to neural ODE and SDE”. The reviewer has acknowledged that significant differences between these approaches exist. To accommodate their request, we have run their proposed baseline, Neural SPDE, and shown that it underperforms both our method AND our baselines in terms of deterministic MSE. We would like to emphasize that this is to be expected because Neural SPDE assumes the noise underlying the data generating process to be a known and observed Wiener process. In our datasets and for real-world data this is a limiting assumption. It also assumes the noise be one sample path from the stochastic process, and hence models a deterministic function instead of a distribution. Thus, it is another inappropriate baseline for the probabilistic evaluation that we conduct in our paper.
Thanks for responding to my request and addressing the concerns from the reviews which includes conducting more experiments. AC
response
Thanks for the response to my comments! > Could you let us know which specific parts were confusing? We would like to address these. I think it's partly due to the various indices $h$, $i_n$, $N$ etc. What's more it took me a 2nd read to eventually understand properly the interplay between the _forecaster_ and the _interpolator_. Thus I'd focus on this. > 'cold posterior' sampling scheme I had a closer read at the related paper and things are now clarified. The additional results from Table 10 also brings additional empirical evidence for using this scheme. I still believe that it's important to introduce this scheme a bit more than in the submitted manuscript. I've updated my score to reflect the clarifications and the fact that part of my concerns have been addressed. The main one remaining being clarity and presentation, yet this is tricky to address without being able to update the manuscript. I hope that the authors would improve on this so that most can benefit from this submission.
We genuinely thank you for your feedback and taking the time to read through both our clarifications and the related paper introducing 'cold sampling'. We will introduce this scheme more in our revised paper, and use the easier to understand 'naive sampling' counterpart as a starting point for introducing the better performing cold sampling. > What's more it took me a 2nd read to eventually understand properly the interplay between the forecaster and the interpolator. Thus I'd focus on this. Thank you for your feedback! Do you think that Figure 6a) in our appendix makes the interplay more clear, potentially extended by explicitly adding the interpolator and forecaster symbols to their respective arrows? Would be beneficial to add it to the main text?
Decision
Accept (poster)