Follow-up answer to Reviewer FREq's comments
**AQ1-3**: Thank you for your feedback. We will carefully take it into account when preparing the final version.
We welcome any comment on our response to **Q4** on the solution quality of the solver.
**AQ5: Dropping indices without prior mention.**
Please note that we specify that on Line 195 in [paper], which reads "Prior to that [i.e., defining RandNet for Parareal], we define how RandNets work in a general setting with input $\boldsymbol{U}$". Indices are introduced later, upon combining RandNets with Parareal, in line 237. Should you find this still unclear to the reader, we could change it to "Prior to that, we define how RandNets work in a general setting with input $\boldsymbol{U}$, *before going back to the input of interest $\boldsymbol{U}_i^k$ within the Parareal framework*".
**AW2 and Further Q: On the potential impact of RandNet-Parareal on ML/AI, and, specifically, on Diffusion Models (DMs).**
Thank you for engaging in further conversation about this matter. Properly communicating the relevance and impact of our method is very important for us.
Our work exemplifies an instance when ML methods allow the advancement of various fields of science and engineering where solving (O/P)DEs is needed, see also our response to W2 above, and W1 of Ref awZW. We mentioned these applications as our submission belongs to the Primary Area of *ML for Physical Sciences*.
However, AI is also one of the fields that could greatly benefit from progresses regarding efficient ODE solvers, and thus our method. This is why we gladly provide examples of how our approach can also assist ML and AI, adding these points in the camera-ready version of the paper.
ODEs are a crucial building block of some relevant techniques, such as *DMs*, *Neural ODEs*, *Optimal control and reinforcement learning*, and *Optimization for ML models*, to name a few.
- _Solvers for DMs_: In the context of DMs, mentioned in your answer, the continuous time reversal process can be described by a probability flow ODE, defined by the score function, usually approximated with deep (convolutional) NNs (note that the ODE formulation offers significant advantages over SDEs in high dimensions). Faster ODE solvers can improve sampling speed, yielding faster image synthesis. The main existing directions in the literature involve using classical sequential solvers [30], developing faster and dedicated ones such as DDIM [24], DDPM [29], DPM-Solver [25], Heun [26], and parallelization of the autoregressive sampling process of DMs (by using Picard-Lindelöf iterations for ODE/SDE) [27,28]. The latter emerged mainly due to the need for other solutions to avoid the bottlenecks of sequential solvers. To the best of our knowledge, modulo this particular parallelized sampling, PinT schemes have not yet been used in this context. **This is the gap this paper could fill, as the existing successful sequential dedicated solvers could be embedded into our proposed RandNet-Parareal**. We expect this to be straightforward, since the goal is to collect samples via solving the corresponding (diffusion) ODEs at $[0,T]$, where RandNet-Parareal immediately finds its purpose.
- _GPU Implementation_: There are two increasingly sophisticated approaches for implementing RandNets-Parareal on GPUs: (i) parallel implementation of the solvers (model parallelism); (ii) parallel implementation of Parareal (in-time parallelism). For (i), both the solver and the RandNet computation can be carried out on the GPU, see e.g.[31]. In-time parallelism would then be implemented across GPUs. Otherwise, under (ii), both model and time parallelism can be implemented concurrently on the same hardware [32]. The latter guarantees a better use of resources at the cost of increased implementation complexity.
We hope these clarifications provide you with evidence of the expected impact/implications of our method in the context of generative AI models and beyond.
**References used above:**
[24] J. Song, C. Meng, S. Ermon. Denoising diffusion implicit models. 2020.
[25] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, J. Zhu. DPM-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. 2022.
[26] T. Karras, M. Aittala, T. Aila, S. Laine. Elucidating the design space of diffusion-based generative models. 2022.
[27] A. Shih, S. Belkhale, S. Ermon, D. Sadigh, N. Anari. Parallel sampling of diffusion models. 2023.
[28] Z. Tang, J. Tang, H. Luo, F. Wang, T.-H. Chang. Accelerating parallel sampling of diffusion models. 2024.
[29] J. Ho, A. Jain, P. Abbeel. Denoising diffusion probabilistic models. 2020.
[30] Y. Song, J. Sohl-Dickstein, D.P. Kingma, A. Kumar, S. Ermon, B. Poole. Score-based generative modeling through stochastic differential equations. 2021.
[31] A.N.Budko, M. Möller, C. W. J. Lemmens. Time integration parallel in time. 2020.
[32] A.Q. Ibrahim, S. Götschel, D. Ruprecht. Parareal with a physics-informed neural network as coarse propagator.