LITE: Modeling Environmental Ecosystems with Multimodal Large Language Models

Summary

This work proposed a multi-model LLM framework for modeling time series data. Two different representations 1) text description 2) trend image and a domain description are combined to represent each instance for prediction. In order to deal with various challenges in the environmental ecosystem data 1) incomplete features 2) distribution shifts, mixture of experts and multi-granularity features are used. Experiment on three datasets (water temp, stream flow, agricultural N2O) shows significant improvement over various SOTA approaches. Ablation study shows the importance of each component and especially the text description, the trend image and multi-granularity features.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

This work shows the importance of combining image and text representation for modeling time series data. It can seem as an extension to the ViTST approach which only leverage the trend image representation.

Reasons to reject

This is a fairly complex system. Although an ablation study is provided, I still feel that the paper can be stronger if alternative solution for each challenge can be discussed and compared to.

Summary

For handling incomplete features and distribution shifts, this paper proposes Multimodal Large Language Model for EnvIronmenTal Ecosystem Modeling (LITE) to better capture and process ST dynamics and correlations from a multimodal perspective. The experiments demonstrate that LITE significantly enhances performance in environmental spatial-temporal prediction.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

1. The proposed method is effective;

Reasons to reject

1. The latest baselin is in 2021, which is relatively old. Why are there no updated baselines? 2. In ablation Studies, because the difference between the parameters of a linear layer and Llama-2-7b is too great, the effect of LLM, the role of LLM is not prominent enough. If the LLM decoder is replaced with small LM, it can better reflect the role of LLM. 3. In Multi-granularity information, The contribution of the two types of data, week data and year data, is unclear.

Questions to authors

1. Why dose transform the linearized data point to natural language? 2. How to find missing variables after transforming the linearized data point to natural language? 3. What is the role of imputation loss？What if delete it？

Summary

This paper proposes a new hybrid arch to model environmental ecosystems, predicting physical variables such as water temperatures in a given region at a specific time step. The authors first prompt LMs to transform input features into natural language descriptions, then employ a semantic time-series encoder to encode these descriptions. These text features are fused with image features as well. The arch allows for imputation of missing features and the authors further integrate multi-granularity information to mitigate distribution shift issues. Finally, all these features are fed into a frozen llama LLM to make the final prediction. Comprehensive experiments and ablation analysis demonstrate the effectiveness of the approach.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

1. Applying LLMs to spatial-time series data prediction in the environmental context is very interesting and potentially impactful for society. The authors fuse both text and image features that are novel. 2. The main results are quite good, greatly improving previous baselines that are kind of traditional. 3. While many designs zoom into specific details, the authors try to conduct comprehensive ablation analysis to justify the effectiveness of each component. For example, I appreciate the ablation experiments that focus on the settings with missing features and distribution shifts.

Reasons to reject

I don’t have any significant concerns for this paper.

Summary

The authors present LITE, a multimodal Large Language Model for environmental Ecosystem Modeling. LITE uses spatial-temporal data and transforms these into two data modalities: semantic time-series and temporal trend images. These transformed data are then encoded to learn multimodal representations of environmental data from pre-trained foundation models. The work addresses gaps frequently arising in real-world data: by imputing the incomplete observations; and incorporating multi-granularity information to handle data distribution shifts in environmental data. A decoder, in the form of a frozen LLM, is used to fuse these multimodal representations. The authors demonstrate the feasibility and generalizability of LITE on real-world environmental data. The motivation, ideas and previous literature are well discussed, but there are points that need further clarification.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

The paper is well written, but does need tidying up. Please see comments.

Reasons to reject

None

Questions to authors

The motivation, ideas and previous literature are well discussed, but there are points that need further clarification. In general; a) the functional forms of Equations are missing; and b) a table with variables and their usage/function in the Appendix will be a good reference since there are a handful of variables that are not defined. Please note these points below. 1. Page 2: 2nd paragraph. Please provide examples of ‘variables’, and therefore ‘inter-variable correlations’. One could use the same environmental application to provide this. 2. Eqns 3, 5, 9; what are these functions? Provide a functional form or a reference. 3. Comparing Eqns 3 and 2: Is \x_{r,t} the same as a data point or feature value or physical driver or an observation? Can these terms be used interchangeably? Provide additional clarification on these definitions. 4. Another term that caused confusion is ‘missing variables’. On page 4, the first line mentions that missing variables are replaced. After Eqn 5, there is mention that missing variables are imputed using SMoE, as well as in the Algorithm 1. If both procedures are done, then update the text and Algorithm accordingly. Please clarify. 5. Eqn 6. What is v (parameter to TopK)? 6. Eqn 7. What is E? 7. Figure 2: needs additional details. Is this a single i_{r,t} or a group of them shown? * What variable is being plotted * Label what each subplot is * what the x-axis and y-axis are * are the axes the same for each subplot * is {\beta} = 9 which is why there are 9 subplots * are these z-normalized values? 8. What is the difference between the LLMs in Eqn 4 and 10 9. The algorithm is still incomplete, for example, it does not mention Eqn 10 or steps needing Eqn 10. 10. Line immediately after Eqn 10: ‘we employ a linear layer on the top of output representation….”. What is the functional form of this layer, and why was it needed? 11. What is the role of \eta_1 and \eta_2 in the Loss function? * What does it mean to ‘balance loss terms’? * Do these have constraints? * where there any bounds to the loss function? 12. Legend in Fig 3 is not clear. Which curves relate to the fixed versus random sensors? 13. Is the average RMSE increase in ratio ‘223%’?