Generalizing Weather Forecast to Fine-grained Temporal Scales via Physics-AI Hybrid Modeling

Summary

The paper proposes a physics-AI hybrid modeling framework for fine-grained weather forecast. They propose to adaptively tune a PDE kernel together with a neural network as the encoder. Following the Euler time stepping, the PDE kernel can perform a fine-grained temporal forecast, which act as a physics-guided modeling part.

Strengths

1. The combination of AI and physics is crucial and novel for weather forecast. 2. The fine-grained weather forecast is of interest to nowcasting and temporal downscaling.

Weaknesses

Some opportunities to improve: 1. The experimental details are way from sufficient. What are the hyperparameters you use, except for learning rate? How do you divide validation and test set? How do you divide input and label? What are the inputs and outputs to the model? Their sizes? What are the datasets' statistics, as there is none introduced? What is the time cost or number of parameters for your model compared to other models? How do you obtain the results in the tables, since there are no error bars? Are these all based on a one-time run? Are they statistically significant, and what are the p values, etc.? Table 3 does not demonstrate that the proposed model is the best model. Is there a specific reason why 120-min forecast is especially good? Any pseudo algorithm for understanding? Is there code for understanding and checking? 2. The experimental results are both not good enough and not analyzed well enough. It looks strange from Figure 4 that FourcastNet is much worse than ECMWF-IFS even though FourcastNet should have been better as reported in the original paper. There needs discussion to justify, or maybe it is due to experimental setting. It is hard to know due to lack of experimental details pointed out above. The nowcast results are basically suggesting that the proposed model is no better than previous models, except for 120 min that is a less realistic setting for real life. Figure 6 is revealing very minimum information about the comparison between models. All the errors look alike, and it is hard to know which model's prediction is better without ground truth. I am not convinced by the explanation why the physics weight is decaying within each hour: unless your neural network counterpart's prediction does not change much, the ratio between physics and AI does not directly reflect how much they change. It could be that the AI part is predicting/contributing less, although the weight is surging. It does not mean anything. 3. I want to highlight this problem using a separate paragraph. The ablation study is strange: it seems the physics part is not always helping. This might relate to how you use the PDE kernel and derive the PDEs. Your appendix A needs to cite references and name the equation names you are using. For example, how do you determine the coefficient for those constants in the equations? Shouldn't that be learned, since you cannot tell how big is the friction coefficient, let alone the term, for example? Moreover, Eq. 14 seems incorrect. I cannot recall such an equation in fluid dynamics. It is the continuity equation if you change p to z. However, pressure level slash geopotential slash height are not strictly the same thing. I feel concerned that they are assumed to be the same without any justification.

Questions

I encourage the authors to address my concerns listed in the weaknesses.

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

Limitations are discussed.

Summary

In this paper, a hybrid model (a model that combines machine learning with physics) is demonstrated for nowcasting and medium-range weather forecasting. WeatherGFT uses machine-learned weightings that combine two successful methods for weather forecasting (machine learning and the traditional numerical, PDE-based method). The combination of both methods allows for significantly increased temporal resolution compared to purely data-driven methods (6-hourly vs 15 minutes). This approach also provides a new framework for combining physics-based methods with data-driven methods, which allows for the ability to differentiate through the hybrid model.

Strengths

Combining machine learning with physics-based approaches (e.g. hybrid modeling) is an exciting and highly researched topic for weather and climate. As mentioned in the paper, purely data-driven models are black-box systems even if they do perform well. Existing data-driven models also are too temporal coarse for some operational weather forecasting applications. This paper does a good job address both of these existing problems in the field.

Weaknesses

The novelty of the paper is not as strong as claimed in the paper. Other hybrid models that combine machine learning with physic-based approaches exist (see Arcomano et. al. 2022 and 2023, Clark et. al. 2022, and others). The Arcomano et. al. 2022/2023 even includes a machine-learned weighting for the combination of the ML-based model and the physics-based model. The time-embedding and multi-lead times in one model have also been demonstrated before, see Stormer (https://arxiv.org/abs/2312.03876) and MetNet (https://arxiv.org/abs/2306.06079).

Questions

In section 4.3, I don’t understand how FourCastNet, ClimODE, or Keisler was used for nowcasting. They all have a time step of 6 hours, how is it possible to get 30-minute forecasts using frame interpolation methods? Are you using interpolation between ERA5 (e.g. the initial conditions) and a 6-hour forecast from these models? In section 4.3, how are the forecasts including WeatherGFT initialized? Do they all use ERA5? I would like to see the effects of the PDE Kernel on other metrics. Does this inclusion of a physics kernel improve spectral bias or allow stability? What about the conservation of energy or momentum? What is the computational cost during inference of having a PDE kernel compared to just the transformer by itself?

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Overall the limitations of the paper are well laid out, however, some claims are not supported by the paper. Claims about being the first to move away from fixed lead times for data-driven weather forecasting are not true. Stormer (https://arxiv.org/abs/2312.03876) and MetNet (https://arxiv.org/abs/2306.06079) have demonstrated this previously. Claim “In addition, the prediction error of our model at the lead time of 6-hour is significantly smaller than that of the physical dynamic model ECMWF-IFS”. For Z500 this is not supported by Figure 4. For Figure 5 the prediction of the subtropical high (it should be a subtropical ridge) isn’t convincing. That seems to be a function of the contouring in Matplotlib as the difference plots show similar magnitude of errors. Overall, if some of these claims are toned down and the addition of the appropriate citations the manuscript will be improved.

Summary

The paper proposes WeatherGFT, a physics-AI hybrid model designed to generalize weather forecasts to finer temporal scales beyond the training dataset. By integrating PDE kernels for physical simulation and neural networks for adaptive bias correction, the model aims to provide accurate 30-minute forecasts using an hourly dataset. The lead time-aware training framework enhances the model's ability to generalize across multiple lead times, achieving state-of-the-art performance in both medium-range and nowcasting tasks.

Strengths

1. The framework that fuses AI and PDE is innovative and improves the model's generalibility. 2. The model demonstrates generalization capabilities across time scales, achieving finer temporal resolutions (e.g., 30-minute forecasts) from coarser data. 3. Extensive experiments validate the model's state-of-the-art performance across various forecasting tasks and lead times.

Weaknesses

1. The paper is well-structured but could benefit from clearer explanations and analyses of the novel modules and specific contributions. 2. The physics model relies on a limited set of PDEs for simulation, which may not fully capture the intricacies of real-world atmospheric dynamics. 3. The baseline comparisons are limited, as many AI models available for weather forecasting have not been included.

Questions

1. Can you further explain Equation 5 and how you formulate $K_x$ and $M_x$ 2. How does the convolution layer work to align neural network features with physical features? 3. How do you update the learnable factor $r$? How do you compute the weight to draw Figure 1 since the weight is a vector? 4. Does tp (hourly precipitation) have a related PDE? How do you deal with variables that are not related to any PDE? 5. How fine-grained can the model achieve? Doing 30-minute forecasts with hourly data is impressive, but can the model achieve finer resolutions like 15 minutes or even 5 minutes?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Discussion period instructions

Dear Reviewers, The authors have provided comprehensive rebuttals and tried to address the concerns raised in your reviews. Please take the time to review their responses carefully. If you have any further questions or require additional clarification, please engage in a discussion with the authors. Thank you for your continued efforts. AC

Results are far short of SOTA for medium-range forecasting

This paper claims state-of-the-art performance for medium-range weather forecasting, which is manifestly untrue and misrepresents the state of the art. The WeatherBench2 [paper](https://arxiv.org/abs/2308.15560v2) and [website](https://sites.research.google/weatherbench/) lists a number of AI models with significantly higher skill than ECMWF IFS and WeatherGFT, including GraphCast, FuXi and NeuralGCM. Over a dozen AI models with such performance have been [reported in the literature](https://docs.google.com/spreadsheets/d/1n30zDDjEzlXl5nAGF8uD_dbZWJAamqImQGCZjfOMuDg/edit?usp=sharing). The authors cannot credibly claim ignorance of this prior work: - NeuralGCM is cited here, although it is incorrectly reported to be "primarily designed for medium-range forecasting" and the fact that it makes sub-hourly predictions is not mentioned. - Three of the authors of this paper were also authors of [FengWu](https://arxiv.org/abs/2304.02948), another model that reported clearly significantly higher skill than any of the baselines from this paper.