Characterizing Multimodal Long-form Summarization: A Case Study on Financial Reports

Summary

The paper presents a case study on summarization capabilities of LLMs. Specifically, the authors focus on summarization of financial reports with several current top commercial LLMs (GPT, Claude, Command). The study focuses a lot on processing numbers, which are known to pose various difficulties to LLMs. The authors perform various analyses, including a small-scale manual annotation of number hallucinations. In general, the authors find that number processing is very strong in some of the models, especially Claude. The study also analyzes positional bias with LLMs tending to mostly cover information from the beginning of the document, and find interesting differences between Claude and GPT.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

- The study presents useful findings about the behavior of LLMs, partially in line with previous research, partially providing different or more complex observations than previous research, and partially adding new observations that had not been previously reported. - The manual analysis of number hallucinations is particularly interesting, delving quite deep and showing various types of number hallucinations (with some not actually being hallucinations but rather new numbers correctly derived by the LLM from the text). - The observed differences in the position bias are new and quite intriguing. This gives us a new finding that probably GPT4 is not great at summarizing documents where the important information is not at the beginning but elsewhere. - Although the study focuses on one specific document type, I believe the findings are probably generalizable to a great extent and are thus of general interest and usefulness. - The findings are not interpreted in detail, which provides nice future work avenues for trying to explain the observations presented.

Reasons to reject

- The study is only descriptive and observational, it covers in great detail the observed behavior and capabilities of the models, but does not attempt at all at interpreting the findings or hypothesizing about the reasons that lead to the findings. Of course this is somewhat hard since all of the investigated systems are closed-source with only limited information available about them, but still some information is provided by the models' authors and other information have been reported by independent researchers, so it would be good to at least try to link the known differences in the models' design and training to the observed differences in behavior on the examined task. - The study mostly does not attempt to evaluate the quality of the reports, claiming there is no gold standard summary for their inputs. This is sad, as although we observe some differences in the models, in many cases we do not know for sure whether the differences mean that one of the models produces better summaries than the other, or whether these are just two possible approaches to the summarization task which are both good. I understand the difficulty in not having the gold standard data, but this should at least be discussed more explicitly and in more detail. For example, the fact that a model produces less numbers in the report, does this automatically mean the report is worse, or may it be that it is a highly abstractive report that interprets the numbers textually without mentioning them and thus this might also be a great report? I am not knowledgeable in the domain of financial reports and their summaries, so I do not know whether a good summary absolutely must contain a lot of numbers (which is somehow indirectly implied by the paper but not explicitly said or discussed). So this is a clear limitation of the paper. Still, I appreciate the manual evaluation of number hallucinations, which goes beyond this and actually does evaluate what is good and what is bad. - For models with a short input context window, the authors simply truncate the input. This does not seem right, why not use a sliding window approach?

Questions to authors

These are something between questions and tentative suggestions (which might or might not be good suggestions, so this is just for consideration). - Is text+tabular data generally called "multimodal" data? I can understand that, but I am more accustomed to "multimodal" referring to other modalities than only text, while tables are still text. But maybe my terminology is wrong and yours is correct, I have not specifically focused on working with tabular data so my terminology is not strong there. - In table 1, would it make sense to report A B C D as percentage instead of absolute numbers, probably a sort of recall of what proportion of the numbers from the report appear in the summary? - Also, table 1 is quite far from it description, could it maybe be moved to a lter part of the paper? - Also, A B C D are not very reader-friendly numbers, could they be maybe e.g. X B XB N (teXt taBle teXt+taBle None) or something more interpretable?

Summary

The paper conducts a comparison of various LLM models regarding their ability to summarize sections of financial reports (10-K), which encompass not only textual content but also tabular and numeric data. It highlights two key findings. Firstly, it identifies a position bias among all models, indicating a tendency to prioritize the initial sentences. Upon further analysis, the paper demonstrates that Claude exhibits less position bias compared to GPT-4 when assessed on randomly-shuffled texts. Secondly, the paper reveals that Claude demonstrates a more sophisticated utilization of numbers, including basic computation, compared to other models. Additionally, it categorizes different instances of hallucination by LLMs in handling numerical data. The paper introduces intriguing phenomenon, such as Claude exhibiting reduced position bias and superior utilizing of numeric data. However, merely reporting these observations may not contribute significantly to the scientific values. The paper would be further strengthened if it could elucidate the underlying reason for the phenomenon. Another drawback is that it has been evaluated only on a single document type -- 10-K. This raises questions about the generalizability of the findings to the other document types.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

- The paper presents several evaluation results that may be of interest to the research community. Specifically, it demonstrates that Claude outperforms other LLMs in summarization tasks by considering actual content rather than blindly favoring beginning sentences. Moreover, Claude demonstrates a sophisticated utilization of numeric data, including the ability to perform numeric computations. - The research on financial document summarization remains under-explored. Financial documents present a unique challenge due to their inclusion of tabular and numeric data. Therefore, evaluation results in this under-explored domain could offer valuable insights to the research community.

Reasons to reject

- The scientific contribution of the paper appears to be limited. Its primary contribution lies in contrasting the behavior of Claude2, GPT-4 and other LLMs. However, the significance of these contributions is unclear, as even acknowledged by the authors themselves, who state in the paper, "...While the position bias of GPT-4 is probably sub-optimal, it is unclear what makes a good summary, especially for such long-form inputs…" - The generalizability of the paper's claims is also uncertain. The evaluation is based solely on one type of financial document - 10-K reports, which are publicly available and likely used during the pre-training of the LLM models. It's questionable whether the findings can be applied to other financial documents or to any document containing tables and numbers. - Strengthening the paper would involve providing hypotheses to explain its findings. Specifically, the paper reveals that larger models tend to utilize numbers, while smaller models do not. Additionally, it demonstrates that Claude employs numbers in a more sophisticated manner than GPT-4. Offering potential explanations for these phenomena would enhance the paper's impact and interest.

Questions to authors

- It looks like each different model received truncated texts due to the limited context window. It would be great if the paper adds the average amount of the truncated contents. - In Section 3.1, the paper says “Specifically, we find that the GPT-4’s summary shows a high degree of condensation such as summarizing net income, net sales, and gross profit in a single sentence (Table 2)”. What is the evidence backing this claim? - How were the tables represented? Were the HTMLs used as an input to LLM?

Summary

This paper analyzes the use of LLMs for summarizing financial reports with a specific focus on how LLMs handle numbers since financial reports are often dense with numerical figures and a good summary needs to condense not just text but also numerical data correctly. They also do human labelled case studies on abstractive sentences and certain numerical figures found in the summary to check if these are correct or if they are hallucinations.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

1. They analyze the summaries from both an abstractive and an extractive point of view 2. They break down numerical figures into 4 types, one of those being "not found in the report". This 4th type is analyzed further to see if it was obtained by the LLM doing simple arithmetic or if it a made up hallucinated number. They report hallucinations to be minimal 3. They also compare Claude 2 and GPT 4 summaries in terms of position bias i.e. are the LLMs able to find the truly important information or are they just focussing in the beginning/end 4. They compare 4 leading LLMs including Claude 2 and GPT 4, with good conclusions and comparisons

Reasons to reject

1. While this work is a good effort in this direction, we are a long way away from ensuring that the summaries are factually correct. 2. Some more prompting methodologies such as chain of thought or speculative prompting could have been used to see if hallucinations can be fixed through LLM introspecting it's own output. 3. Most analysis is done via human labelled case studies and will be hard to reproduce

Questions to authors

--

Summary

This paper presented a case study of long-form summarization in the finance domain in order to study LLMs’ behaviors when it comes to summarizing long input texts and numerical operations. Specifically, the authors first proposed a computational framework to characterize multimodal long-form summarization in summarizing financial reports and investigated how the information in a given financial report is utilized. Then the authors conducted a thorough inspection and evaluation of how selected LLMs handled numbers in financial reports, and proposed a taxonomy of categorizing errors related to numeric operations. The findings of the experimental results show that LLMs have strong position biases and can be mitigated by shuffling operations.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

This paper is well-written and well-organized, making it very easy to follow. The problem studied in this paper is well-motivated and took on a rather challenging and urgent task in the age of LLMs. The presented study asked important questions with regard to how LLMs use information in long input text in a specific domain that contains many numbers that are important to the interpretation of the original text. I also like that the authors showed concrete examples when detailedly explaining the four types of numeric value utilization. In addition, the authors also called for new approaches or perspectives of evaluating long-form summarization when the gold standard isn’t necessarily the sole ground truth and when such ground truth is getting more and more difficult to get when the source text gets longer and longer.

Reasons to reject

I don’t have much to say here, but I have some comments and questions that I would like to bring up in the following section.

Questions to authors

1. Regarding the authors’ comment and references about LLM-generated summaries overwhelmingly preferred by humans: I think this is only applied to single document one-sentence new summarization [1]. As some previous work showed , this is definitely not the case for non-news summaries ([2]---[4]). In other words, single document summarization of news is solved != summarization is solved. 2. The authors may want to cite some prior work on chart-to-summary generation ([5]--[7]). [1] https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00632/119276/Benchmarking-Large-Language-Models-for-News [2] https://dl.acm.org/doi/abs/10.1145/3477495.3531802?casa_token=g-Uw8_87GzwAAAAA:gnrbj6AxvcswhwB7Mjzumbzynm0eG6aB-A1WYPlsKhyGjn-9p12bn71uCp8rde7fxMRgHRwHTMNswg [3] https://aclanthology.org/2023.findings-acl.593/ [4] NewSumm Keynote 1 by Kathleen McKeown [5] https://aclanthology.org/2022.acl-long.277/ [6] https://aclanthology.org/2021.ranlp-1.183/ [7] https://link.springer.com/chapter/10.1007/978-3-319-46478-7_41

[Area Chair comment] Author response

Dear reviewer , Please take a look at the author's rebuttal. If the rebuttal addressed your concerns, please let the authors know about this and update your review. If not, please continue to engage with the authors and the other reviewers in the discussion forum. Are your concerns about the descriptive vs prescriptive nature of the results addressed? (Reviewer Lb8C also had similar issues). Other reviewers should also feel free to comment on these points! Thanks.