Plots Unlock Time-Series Understanding in Multimodal Models

Response to reviewer comments

Thanks very much for your thorough reading and helpful and positive feedback! **Regarding your specific questions:** > How are multiple time series handled? Please add more details. For example, for the fall detection task on IMU data, is the input {img} a scatter plot containing six signals? Does a scatter plot containing multiple points from various variables lead to errors in the results? Including descriptions or figures about the input plots would enhance understanding. We have expanded the details in the manuscript to add clarity around the shape of the input plots. Specifically, we included plotting code snippets for the tasks in Supplementary Section A.1 (Detailed Task Descriptions) starting on page 13, and examples of IMU data being plotted as done for the fall detection (with all 6 IMU dimensions plotted on one figure) in Supplementary Figure S6 (page 19) activity recognition tasks (with 3 accelerometer and 3 gyroscope axes respectively plotted on separate figures) in Supplementary Figure S7 (page 21). For Activity Recognition we did find that if we split the IMU data into two separate plots, with the 3 accelerometer axes and 3 gyroscope axes separately, there was a marginal performance gain vs plotting all 6 axes on the same plot; see new Supplementary Figure S8 on page 22. Regarding the number of points used, we subsampled down to 10 Hz for both datasets, using avgpool, due to GPT-4o context window constraints. > Regarding the ablation experiments on visual elements in the matplotlib figure (including size, plot style, color palettes, markers, and plot components), which setting combination is reasonable or optimal? Is there a relationship between plot settings and the number of shown points? Since this paper focuses on plot representation, such a discussion is noteworthy and would be helpful for the follow-up work. Thanks for the question. We include the ablations described in Supplementary Information Section A.3 starting on page 28 to show that the plotting performance is generally robust to aesthetic choices, rather than as a rigorous investigation of the impacts, or optimisation, of these choices. The exact appearance of the time-series plots that lead to optimal performance would likely depend intimately on the exact downstream use or user requirement, so a priori cannot be determined. As such, we did not focus on carefully optimising the specific plotting approach, which could in theory be automated and forms the basis for future work – thank you for bringing this important point and we have highlighted it in the penultimate paragraph of the revised Section 5 (Conclusions and Future Work) on page 10. > From the supplementary tables, it appears that results on cubic and periodic functions are worse on the plot-based approach. Some discussions on this finding are appreciated. This is a good question – we would like to point to Supplementary Figure S9(c) on page 24 which illustrates the performances across models and modalities for different function classes for the function classification task. Here we see that for all classes except periodic (including cubic), plots outperform or are within error bars of the text method, even if the absolute performances are lower than for other classes. For the periodic functions, we found that the text models were biased towards this answer choice, so this class was a low precision (but high recall) answer. As such, we believe the text models are not necessarily better at identifying periodic functions. > It might be beneficial to consider elaborating on the scalability of the approach for large-scale datasets. Thank you for raising this good point, which we have expanded on in the manuscript in the second paragraph of Section 4.4 (Cost) starting on page 9. Assuming large-scale in the context of longer time-series, the visual representation approach we study here will scale better than a textual representation of the same time-series data. This is because the number of tokens required to represent the textual data scales roughly with the number of data points n, while for a fixed plot size the number of tokens required to represent an image is constant with respect to n. **To your other points of feedback:** > Lack of insights from a large number of experiments and ablation studies. As discussed above, our ablations (especially the plotting ones) are mainly included to show general robustness of the trend that visual representations outperform textual representations, rather than an attempt to optimise any one parameter to get the best absolute task performance out of the models. As such, we felt that focussing too much discussion on the results of the ablation studies would distract from the main point at the cost of space better spent on what we feel are the more relevant details findings. We appreciate your comment though, and would welcome any additional feedback on specific ablation areas you feel might remain unclear.

Response to reviewer comments [1 / 2]

Thanks very much for your thorough reading and helpful feedback! **Regarding your specific questions:** > Could you provide references or empirical evidence to support the claim that the model's approach mirrors the human approach? Thank you for pointing out the gap in evidence related to this claim. We have added the following citations to the first paragraph of Section 1 (Introduction) on page 1 to justify the claim that visual approaches can help humans ingest complex data: - S. K. Card, J. D. Mackinlay, and B. Shneiderman, “Readings in Information Visualization: Using Vision to Think,” San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1999. - M. Adil Yalçin, Niklas Elmqvist, and Benjamin B. Bederson. 2016. Cognitive Stages in Visual Data Exploration. In Proceedings of the Sixth Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization (BELIV '16). Association for Computing Machinery, New York, NY, USA, 86–95. https://doi.org/10.1145/2993901.2993902 To be clear, our claim is that models benefit from using visual ingestion of complex data relative to textual representations similarly to how humans benefit from visual representations of data, not that the reasoning mechanism in which models parse visual data is the same as the cognitive mechanism with which humans comprehend visual stimuli. We have included this important point in the first paragraph of Section 5 (Conclusions and Future Work) on page 10. > Could you consider grouping the related work into subsections based on different approaches to make it clearer? Thank you for this helpful suggestion – we have re-organised Section 2 (Related Work) on pages 2-3 into subsections based on approaches including “Forecasting”, “Time-series models”, “Vision models and visual representations” and “Measuring understanding". > Could you add a clear mathematical formulation to define the problem and solution, which would make it more rigorous? Our work is an empirical contribution, in which we seek to probe the capabilities of existing models without additional training, investigation of the internal machineries or novel encoding/decoding mechanisms. As such, we don’t believe that there are relevant mathematical models to quote. Furthermore, the black-box, closed-source nature of the models we study here preclude a closed-form expression of exactly how various inputs are tokenized, encoded, processed and decoded. We also assume that the quality metrics (accuracy, mean absolute error, and F1-score) we employ are sufficiently familiar to the readership of ICLR as to warrant omission of definitions. > Could you consider how to enhance the depth of analysis to meet the standards of high-quality research for ICLR? We believe the analysis we employ here to be of the appropriate quality. We note that other reviewers (e.g. reviewer LLVU) have listed our analytical approach as a strength of our work, but would welcome any constructive feedback on the methodology as described in Section 3 on pages 3 and 4.

Response to reviewer comments [1 / 2]

Thanks very much for your thorough reading and helpful feedback! **Regarding your specific questions:** > “What is the prevalence of plot vs. numerical text data in the typical available data? The benefit of this method seems dependent on currently having substantial, unused plotted time-series data. Using the vision encoder to interpret visual input improves performance over text-only input on only some of the pairs of real-world example task and foundation model.” As a point of clarification, our focus is not on ingesting previously unused plots of time-series data, but rather investigating for any time-series data whether a model can more easily reason about either the numeric or a (de novo) plotted representation of that same data. With that in mind, we do not believe that the relative prevalence of plot versus text data is relevant, as we believe our argument holds for any time-series data. We acknowledge the second point that the visual performance is not strictly better than the text performance in all cases, but as shown in Table 2 on page 5 it is non-inferior in the vast majority of cases (39 out of 42 results), and comes with efficiency benefits over the text representation as described in Section 4.4 on page 9. Furthermore, in cases where the text approach performs better than the visual approach (e.g., the readiness task on Gemini Flash as described in Section 4.2 on page 9) the difference is small and within distributional overlap, so that the benefit of using the text approach is not strong. > “Why does the method perform well or poorly on various tasks, and why is there so much variation in the performance of the method between different foundation models (e.g., Gemini Flash 1.5 vs GPT4o in Figure 3)?” This is an excellent question, but unfortunately beyond the scope of our work as our focus is not on explainability. As the models we use are closed-source, we can speculate that the reason must be some mix of (unreleased) details of training data, model size, numerical tokenization mechanism and vision encoding mechanism. This question would form the basis of very interesting further work, which we have included in the final paragraph of the expanded Section 5 (Conclusions and Future Work) on page 10. **To your other points of feedback:** > “It seems like the actual task of recognizing the types of visual patterns achieved by this method are not inherently novel – Section 2 shows that models are already able to parse plots and tables. So the contribution of the paper would be more clear if the authors provided (1) comparisons to actual timeseries baselines to show an improvement over prior work or (2) further justification on why one would ever feel constrained to using a foundation model to interpret time-series data (i.e., how often or why text interpretation would be the only alternative), which is currently unclear to me.” The novelty of our work lies in the systematic investigation of the relative performances of visual versus textual encodings on these tasks across foundational models. To be clear, we are not claiming that our approach replaces or obviates training task-specific models, which we expect will always achieve the highest absolute performance, but rather quantifies the differential ability of models to ingest visual versus textual information. As such, the appropriate baseline is the performance of a foundational model on the text representation of the data. To our knowledge, this differential performance has not been rigorously studied in the literature, and is an important contribution to the field in the context of understanding and optimizing the use of foundation models. Beyond the theoretical contribution of understanding model capabilities, our work also has important real-world implications. Namely, as modern foundational models continue maturing and increasing their real-world adoption, users will expect the ability to interact with data-intensive products such as wellness trackers (Fitbit, Whoop, Apple Watch etc) via a natural language interface, which will necessitate a mechanism for foundation models likely to be incorporated into these products to best make sense of time-series data. Current models such as Gemini Pro and GPT4o are already showing great progress in their multimodal capabilities, and this trend will likely continue such that text representations are no longer the default or optimum, especially for complex data types such as time-series as shown in our work. Thanks for raising these important points – we have revised the manuscript to include parts of our response here. The point on the appropriate baseline has been added to the second paragraph of Section 3 (Methodology) starting on page 3, and the relevance has been added to the final paragraph of Section 1 (Introduction) on page 2 and the first bullet point of Section 5 (Conclusions and Future Work) on page 10.

Response to reviewer comments [1 / 3]

Thanks very much for your thorough reading and helpful feedback! **Regarding your specific questions:** > "How are synthetic time series generated?" We have added key information about the synthetic tasks that was previously in the Supplementary Information into Section 4.1 (Synthetic Data Tasks) of the main text starting on page 6, and have expanded the details for how the tasks were implemented in Supplementary Section A.1.1 (Synthetic Tasks) starting on page 13. > "How are the options of the time series understanding questions generated?" Also detailed in Supplementary Section A.1.1 (Synthetic Tasks) starting on page 13, all the synthetic time series understanding tasks are generated from a fixed range of values based on the task: - one of five functional forms for 'functional form identification' - a set of slope coefficients for each line for 'correlation of two lines' - a range of the number of distinct clusters for '2D cluster counting' - the derivative of one of the multiple choice options for 'derivative identification' and 'quadratic derivative identification' > "What kinds of correlation between two lines are you measuring? Cross-correlation?" For the correlation task we are taking the sign of the Pearson correlation coefficient between the two lines as the ground truth for whether the lines were positively or negatively correlated. In Supplementary Section A.4.2 (Correlation of two lines) we include the prompts showing how this was phrased to the model for the task. We have expanded the details in Section 4.1 (Synthetic Data Tasks) starting on page 6 to clarify this. **Regarding your further points and questions raised:** > "For example, there's a lot of forward references to the appendix and supplementary material with key information. This not only makes reading the manuscript harder, but the leaves the reader to wonder about some basic questions, for example how is the synthetic time series generated" We empathize with you on this issue, and found we had to carefully balance the amount of details on the methods in the main, page-limited manuscript with the presentation of primary results. As the focus of the paper is the relative performance differences in time series understanding between plot and text based methods, and we explored this over a range of synthetic and real-world tasks we felt that including the high level elements of each task and covering the range of findings to demonstrate the higher performance of plot methods was the most important. The specific details of the synthetic time series tasks and the generation of their datasets formed the first 3 pages of the Supplementary Information. We agree they form an important part of the research, and wanted to ensure they were covered in necessary detail, but felt that promoting all of that detail to the main manuscript would result in other key elements needing to be removed. However, as above, we expanded the details in Section 4.1 (Synthetic Data Tasks) starting on page 6 to include more of the key information in the main text. Similarly the details on the range of ablation experiments performed were also important, but on balance the bulk of this was also placed in the Supplementary Information. > "The captions should be written so they communicate a story, rather than just the mechanics of the plot." Thank you for the suggestion; we have adjusted figure and table captions throughout to better convey the narrative of the paper. > "The rationale behind some design decisions are not explained, for example the use of structured prompting" For structured prompting specifically, the rationale was to more easily scale across different model APIs by using the response structuring layer as a mechanism to enable general code and enforce model response types / values. We chose LangFun as our structured prompting library as it is open-source, and we include exact prompt templates in Supplementary Information Section A.4 starting on page 43 in the spirit of reproducibility. If there are further design decisions you feel need further clarification, we'll be happy to do so. > "some of the tables and figures can be made smaller, some figures and long descriptions of statistical tests can also be moved to the appendix" We have made Table 2 and Figures 2-4 smaller, which allowed us to expand on the text while remaining in the page limit; thank you for the suggestion. > "more thorough related work section, for example covering how this work is positioned in the context of time series foundation models [1--4], multimodal time series and text models [5]" We appreciate the feedback and have expanded the Related Work section in pages 2-3 to better position our work in the context of existing work on time series foundation models and multimodal time series and text models. In particular, the references you very helpfully provided have been included in the “Time-series models” subsection on page 2.