Identifying Spatio-Temporal Drivers of Extreme Events

Summary

In this paper, the author investigates a novel, significant, and practical problem: how to efficiently identify extreme anomaly events from climate data. To address the temporal delays between anomalies and extremes and the spatially uneven response of anomaly events, the author first innovatively constructs three comprehensive datasets, including synthetic and real-world datasets. Next, the author proposes an end-to-end spatio-temporal anomaly detection network model, with key concepts including independent spatio-temporal encoders and a compact quantization layer adapted for anomaly detection. Finally, the author conducts detailed experiments on extreme anomaly event detection tasks using two types of datasets to demonstrate the effectiveness of the proposed datasets and methods.

Strengths

+ 1. This paper addresses an important but overlooked issue in practical scenarios: identifying extreme anomaly events from climate data. The motivation for this research is intuitive and substantial, as the task of recognizing extreme anomaly events is crucial for understanding climate patterns and can be applied in significant areas such as agricultural production and social activities. --- + 2. Unlike traditional anomaly detection, anomaly events typically exhibit temporal delay characteristics and spatially uneven responses. Therefore, the author first constructs an anomaly event detection dataset approximating the historical Earth system based on reanalysis and remote sensing datasets. However, there are reliability issues with extreme drought events obtained from remote sensing data as ground truth. Consequently, the author further develops multiple synthetic datasets to facilitate more convenient and reasonable experimental analysis and guidance. I greatly admire and appreciate the immense effort behind this work. --- + 3. Furthermore, the author presents an end-to-end neural network solution. Despite the limited technical innovations, the performance achieved is still impressive. In particular, the thorough experimental comparisons and analyses are commendable.

Weaknesses

+ 1. This paper involves extensive preprocessing and adaptation of datasets. Although the author has not released the related code and data details, I believe a thorough introduction to the datasets or code is necessary for future use, which is crucial for subsequent work. This raises another issue: I find the contributions from the problem and dataset introduction to be overwhelming compared to those from the method itself. Therefore, it seems more suitable for the datasets and benchmarks track rather than the research track. --- + 2. This paper appears to remain limited to regional-level anomaly event detection. However, we know that the entire Earth system is dynamically interconnected and evolving; changes in one corner of the Earth can significantly impact distant regions (e.g., the butterfly effect). To model the spatiotemporal dynamics at the global level, the current backbone network seems inadequate for such scalability. --- + 3. Although the paper is structurally clear, there is still much room for improvement in writing and presentation. Particularly for interdisciplinary papers, the author should not assume that readers have backgrounds in multiple disciplines. Necessary background knowledge and related work should be supplemented as thoroughly as possible. --- + 4. I have also listed several confusions and suggestions in the paper. If the author can address my concerns, I would be willing to support its publication here and recommend an increase in its score, even though it is more suitable for the DB track.

Questions

+ 1. The second contribution point in Section 1 should be swapped with the first to reflect the paper's primary contribution more logically. The main contribution appears to be the introduction of a novel anomaly detection problem in Earth sciences, supported by detailed benchmark experiments, while the methodological innovation is limited and should be de-emphasized. --- + 2. The anomaly detection part in Section 2 is overly redundant. It should focus on the works most relevant to the new anomaly detection approach proposed in this paper, emphasizing the connections and main differences with existing work, as well as the motivations. --- + 3. In line 113, the phrase "but in spatio-temporal configurations of variables that potentially cause an extreme event with some time delay and at a potentially different location" lacks clarity about how these variables differ from standard anomaly variables. I suggest the author provide a simple illustrative example figure to help readers understand. --- + 4. Considering the large parameters of the video Swin-Transformer, is it suitable as a backbone network for global-level spatiotemporal anomaly detection? There are many lightweight alternative spatiotemporal prediction backbone networks. The author could consider adding experiments with different backbone networks. Although this is not mandatory, it would be commendable. --- + 5. Regarding the dataset, what is the spatial resolution? Specifically, what is the actual area of one pixel? How do different resolutions impact the final detection results? Are there any experiments addressing this? --- + 6. In Table 2(a), the comparison between the first and second rows shows that while the $\mathcal{L}_{(extreme)}$ loss term improves extreme detection, it harms anomaly prediction. Ideally, it should benefit both. Why is this the case? Can you provide an explanation? Similarly, adding the $\mathcal{L}(anomal)$ loss term (comparing the third and first rows) shows it helps both types of detection, but the comparison between the fourth and second rows indicates it harms extreme detection, which seems contradictory. I hope the author can provide reasonable supplementary experiments and explanations. --- + 7. The text in the figures is too small and not reader-friendly. It should be just one size smaller than the main text for better readability.

Rating

6

Confidence

4

Soundness

3

Presentation

1

Contribution

3

Limitations

I have already listed them in the question section.

Summary

This work aims to identify the atmospheric drivers of extreme droughts. For this, they assume that for every impact of extreme droughts measurable with remote sensing, there is a precursor signal in assimilated land surface and meteorological data. The work proposes to identify these precursor signals with inherently interpretable machine learning: a computer vision model is trained to map input data into a binary latent space. These binary encodings are subsequently used to predict the future occurance of drought impacts. In that way, the binary encodings are assumed to be interpretable as "anomaly in atmospheric data" and "normal atmospheric data". Models trained in this way achieve good prediction skill (F1 score ~0.9) of drought impacts on synthetic data. Also, the binary encodings match anomalies in the synthetic input data reasonably well (F1 score ~0.8). The prediction skill on real world data is rather low: F1 Score ~0.2 for drought impacts.

Strengths

1. This work introduces a potentially novel variant of anomaly detection: Detecting only those Anomalies that are predictive for correlated impacts. This variant is relevant for studying the drivers of extreme drought impacts. 2. The work compares a wide array of baselines and performs many ablation studies. 3. Synthetic experiments are conducted to study the proposed method before shifting to real-world data 4. The main text of the paper is reasonably concisely written, with many additional details supplied in the appendix.

Weaknesses

Major points: 1. Confusing terminology: The authors speak about "anomalies" and "extreme events" without properly defining what is meant with which term. Furthermore I believe the used terminology is non-standard in the field, and propose the authors to instead use: - Land surface Impacts of extreme events: These are what you call "extreme events", i.e. the VHI below a certain threshold. I would argue what you mean is the impact on state variables representing the land surface state (the ecosystem health) of extreme events. In your case droughts, but this could be any type of extreme events. - Atmospheric drivers of extreme events: These are what you call "anomalies", but both the VHI below a certain threshold or the surface temperature above a certain level could be considered anomalous. Hence I recommend you rather focus on drivers here, these could be atmospheric or hydrological state variables (e.g. temperature or soil moisture) or land-atmosphere fluxes (e.g. evaporation). - Luckily, you should be able to resolve this issue through simply rewriting your article. 2. Scientific validity of experiment design: I find a few choices of the authors a bit odd in the experiment design: - Albedo "fal" / "al" & Soil temperature "stl1" are state variables of the land surface that should be very related to reflectance. In fact, I believe it is not unlikely that remotely sensed brightness and brightness temperature has been assimilated to obtain these variables. This is not too different from VHI, which is created from similare remote sensing products. Thus I would say it should be if anything an Output of your approach. And even if no satellite products have been used to assimilate these variables, this would not solve the issue, but rather raise another issue: then the variables would entirely depend on prescribed schemes in the land surface model of IFS, which means your whole approach is limited by how well IFS reproduces these variables, which I assume is pretty poorly, so any "anomalies" you detect in these variables could be considered spurious. - Soil moisture "swvl1" / "vsw" is a state variable of the hydrological cycle, and thus should be highly correlated with VHI. However, as far as I know, its representation in ECMWFs land surface reanalysis is relatively poor (e.g. https://ieeexplore.ieee.org/document/9957057) - The anomalies in precipitation that would drive drought are **no precipitation for many weeks**. So if you produce a binary encoding for every single time step, this should not be very predictive. In fact, precipitation is somewhat exponentially distributed: in most regions many days observe 0 precipitation, even under no drought conditions. One way you could potentially circumvent this issue is by using accumulated precipitation over many weeks, e.g. through an exponential moving average. Another way could be to actually implement a simple water balance model. Than again, if you use Soil moisture as inputs, it is essentially coming from such a water balance model... - In other words: I think scientifically most interesting would be if you could connect anomalies in atmospheric variables like temperature, humidity and precipitation to the anomalies in land surface states (VHI). Then you could find the primary drivers for the impacts on vegetation and their time lag, and their spatio-temporal variability, which could be super interesting to study. 3. Low predictive skill: The performance on the real-world data is pretty bad if I understand correctly (F1 of 0.2 - 0.3, Table 14). In addition the synthetic experiments revealed there seemingly is typically a lower skill on the latent binary variables compared to the outputs, so this makes we wonder if the predicted anomalies for extreme events mean anything at all? 4. Many baselines for anomaly detection, but none for interpretable forecasting. You compare with a lot of baselines, which is generally great. But all of these perform some sort of anomaly detection on the inputs, which you then assume to be predictive features for your VHI labels. To me what would be a more interesting baseline is one that directly predicts the VHI label from the inputs, and then uses some post-hoc method to try to explain the predictions (e.g. shap, integrated gradients, ...). Because then you are comparing predictions of the drivers directly, and not just general anomalies. Minor Points: 1. This work seems to focus a lot on the spatial aspect of things. However, arguably what matters most for drought at a particular pixel is the water balance at that pixel. And that is primarily driven by precipitation and evapotranspiration at that pixel, with only runoff introducing some sort of spatial component. 2. VHI < 26 may be the result of not just drought. A heatwave could have a similar effect. Also, VHI is a general vegetation condition index, not just for agricultural areas, but also for natural land cover. You may wish to rephrase your framing of this work studying the "drivers of impacts of agricultural droughts" into the "drivers of impacts of extreme events on vegetation". 3. An alternative approach could be to not just predict a binary label, but rather the exact value of VHI. This would be similar to vegetation forecasting (e.g. https://www.sciencedirect.com/science/article/abs/pii/S003442572030256X , https://openaccess.thecvf.com/content/CVPR2024/html/Benson_Multi-modal_Learning_for_Geospatial_Vegetation_Forecasting_CVPR_2024_paper.html , https://gmd.copernicus.org/articles/17/2987/2024/ ). Probably it would be relevant to mention this related stream of literature also in the related works section. Also you may want to consider adding a comment on why you directly predict the (VHI < 26) label instead of the raw VHI, and afterwards apply detection. 4. Quite some typos, e.g. l.22 "very" instead of "vary"

Questions

Climate data has extremely high spatio-temporal autocorrelation. How do you ensure your models are not overfitting?

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors mention a variety of limitations in section 6. However, I believe as is the work has more fundamental flaws that I mention in the Weaknesses section, which, unless fixed, should definitely be named as limitations.

Summary

The paper proposes a novel approach to identifying spatio-temporal anomalies correlated to extremes such as drought. Using neural network, to predict extreme events by learning spatio-temporal binary masks of anomalies identified in climate data. The network is trained end-to-end to predict both anomalies and extremes from physical input variables, focusing on the spatio-temporal relations between them.

Strengths

Introduces a new method for identifying spatio-temporal anomalies that are correlated with extreme events.

Weaknesses

The model is dependent on temporal resolution, which might not be well documented in all parts of the world. The method only shows results on droughts. Binary masks tend to oversimplify real-world events. The method seems to bluntly connect anomalies with extremes without specific theoretical reasoning.

Questions

Have you tried the method on other extreme events other than droughts? And what if there is discontinuity in terms of temporal data?

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

If possible, please add the reasoning behind feature represetations and extremes.

Summary

This paper proposes an approach to learning the spatio-temporal relationships between events with spatial differences and temporal delays. Specifically, they propose a method that identifies spatial-temporal anomalies in multivariate climate data that are correlated with extremes. The authors conduct experiments on both synthetic data and climate reanalysis data.

Strengths

1. The problem of anomaly detection and learning their relations is crucial. 2. The summary of the relevant literature is relatively complete. 3. The authors conduct experiments on both synthetic data and real-world data.

Weaknesses

1. The motivation of model design is not clear. For example, why do you need to detect the anomaly and then classify the extreme events instead of detecting extreme events directly? Such a designed pipeline will lead to more accumulated errors. 2. The writing can be improved. For example, there are typos, such as 'MIL Is a weakly ...'. And the difference between anomaly and extreme in this paper is not clear. 3. The title is somehow misleading, actually what the paper does is more about extreme event prediction instead of learning spatial-temporal relations.

Questions

1. Anamly detection is a classification problem with severe class imbalance problems, how did the authors tackle this problem? What is the ratio of extreme events to ordinary events? 2. What is the difference between anomaly and extreme in this paper? Could you provide some examples to illustrate this? 3. When I read spatio-temporal relations, I thought this paper would build spatio-temporal graphs to describe the relations. Have you considered using graphs to tackle this problem?

Rating

4

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors adequately addressed the limitations.