Fourier Amplitude and Correlation Loss: Beyond Using L2 Loss for Skillful Precipitation Nowcasting

Summary

The paper introduces novel loss functions - the Fourier Amplitude Loss (FAL), Fourier Correlation Loss (FCL) and a Regional Histogram Divergence (RHD) - to improve the realism of the predictions of precipitation nowcasting models without the use of generative models. The loss is applied to established precipitation nowcasting methods and is seen to significantly improve the sharpness and overall realism of the predictions.

Strengths

The paper presents a few interesting loss functions, and in particular the RTD is an interesting approach to improving the representation of precipitation quantiles while not overly penalizing small spatial shifts. The paper also shows comparisons of various key metrics with standard model architectures trained with MSE and with the new losses, showing the advantages of the new losses with

Weaknesses

The authors sensibly use MSE loss as a baseline when evaluating their model. However, there is quite a long history of developing new losses for making precipitation nowcasting produce sharper images and improve the representation of extreme precipitation. As such, I would have wanted to see a paper introducing a new loss function do some comparisons to earlier attempts to improve on the MSE loss if it wants to claim state-of-the-art performance. See e.g.: https://proceedings.neurips.cc/paper/2017/hash/a6db4ed04f1621a119799fd3d7545d3d-Abstract.html https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2019EA000812 https://www.mdpi.com/2073-4433/10/5/244

Questions

You mention that you introduce FCL because the FAL doesn't fully constrain the spatial positions in the image. A seemingly simpler solution would be to use the FAL in combination with MSE. Have you tried this approach and if so, do you see advantages with using FCL instead? Also, you are using a loss where you randomize between FAL and FCL and the probability of using FAL increases over time. You motivate this (lines 210-214) as it being tricky to find the correct weighting of FAL and FCL for a linear combination, but I don't quite understand, how is this not functionally equivalent to using a linear combination with the weight of FAL increasing over time?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper discusses some limitations of the approach, but one point that is not mentioned is that unlike generative approaches to producing realistic nowcasting predictions, the model trained with the new loss function cannot produce multiple outputs for the same input, and is thus unable to provide stochastic predictions where multiple outputs from the model are used to characterize the probability distribution of precipitation.

Summary

This paper propose a new Fourier Amplitude and Correlation Loss (FACL) to replace the traditional L_2 losses in precipitation nowcasting task. They evaluated the FACL on one synthetic dataset and three radar echo datasets, which demonstrates their method improve perceptual metrics and meteorology skill scores. Besides, they propose Regional Histogram Divergence (RHD) to improve the error margin in meteorological skill scores.

Strengths

1. The FACL increases the quality of predictions compared to MSE. The model trained with FACL has more small patterns than MSE model, especially for longer lead time. Besides, FACL doesn’t import additional time cost according to Appendix G. 2. Regional Histogram Divergence (RHD), a variation of FSS, replaces the “hit” and “miss” of FSS with regional pixel distribution. It cancels the need of choosing threshold for FSS, alleviating the effect of threshold when evaluating models.

Weaknesses

1. The evaluation is incomplete and unconvincing. The csi-m of EarthFormer is much lower than the results in [1] (0.3982/0.44). Besides, it could include more methods, such as NowcastNet[1] and DiffCast[2]. 2. In figure9, compared to generative methods such as LDCast and MCVD, the FACL methods still lack predictions for extreme value and the distribution of predictions given by FACL is slightly different to the ground truth. 3. The intuition of FACL has no strong connection with precipitation nowcasting. 4. The coefficients of FAL and FCL needs to be selected manually, which may hinder the applications of FACL in other datasets and tasks. Minor: Models adopt the “prediction and refine” may help improve the related work section. [1] Skillful nowcasting of extreme precipitation with NowcastNet [2] Cascast: Skillful high-resolution precipitation nowcasting via cascaded modelling

Questions

Why does the author measure the CSI of SEVIR and MeteoNet with the same threshold? The data in SEVIR are VIL images, while the ones in MeteoNet are radar reflectivity (dbz).

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The evaluation is incomplete and unconvincing;lack predictions for extreme value;The intuition of FACL has no strong connection with precipitation nowcasting

Summary

This paper proposes the FACL loss function and provides theoretical and empiral proofs on how it boosts clarity and structure for images. The paper also shows how the loss behaves with generative setups and additionally proposes a new metric that is tolerant to deformations.

Strengths

- Clearly demonstrates why a naive fourier loss has no benefit over MSE - Clearly demonstrates how FAL is translation invariant - Clearly demonstrates why FCL provides global information to predicted pixels instead of pixel-level local information

Weaknesses

- Instead of the stochastic modification to Moving-MNIST, [1] already introduced a chaotic yet deterministic N-Body MNIST to mimic the complexity of Earth system interactions, would have been interesting to see the performance on this dataset - FVD and LPIPS calculated on the basis of models trained on natural images, difficult to see why it would extend to scientific data [1] Gao, Z., Shi, X., Wang, H., Zhu, Y., Wang, Y. B., Li, M., & Yeung, D. Y. (2022). Earthformer: Exploring space-time transformers for earth system forecasting. Advances in Neural Information Processing Systems, 35, 25390-25403.

Questions

- Does the alpha parameter swing wildly between datasets? Any good strategy to figure this out? - What are the benefits of RHD over a simple wasserstein distance between predicted and true distributions or their quantile error?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes