SPEER: Sentence-Level Planning of Long Clinical Summaries via Embedded Entity Retrieval

Summary

This paper presents a LLM-based method, named SPEER (Sentence-level Planning via Entity Retrieval), to generate hospital-course summaries from clinical notes (admission to discharge notes). The main innovation is to identify entity synonym groups from from the notes, identify the salient entities using classification among them to perform sentence-level guiding of the LLM to generate sentences that include the salient entities. They fine-tune open-source LLMs (Mistral-7B-Instruct and Zephyr-7B) for the task. They show that SPEER improves the coverage of salient entities and faithfulness over non-guided and guided LLM baselines.

Rating

7

Confidence

5

Ethics flag

1

Reasons to accept

- A well-reasoned approach and a well-executed study for a difficult clinical NLP task, demonstrating good performance. - Extensive evaluation and ablations.

Reasons to reject

- The main contribution is the sentence-level planning aspect, which may be considered to be of limited novelty to this community. - It would have been interesting to assess the effect of the method on several biomedical summarization tasks to show generalizability.

Questions to authors

- The ground truth references are extracted from discharge summaries. How noisy are these references in terms of content? - They cluster related terms using embeddings. What would be the effect of using simple entity linking instead of identifying entity synonym groups? -The number of entity spans in reference summaries seems too low (30; Table 1). Is this correct?

Summary

This paper introduces SPEER (Sentence-Level Planning via Embedded Entity Retrieval), a method for improving the coverage and faithfulness of long clinical summaries generated by fine-tuned large language models (LLMs). I view this work as a unique and important contribution to real-world clinical research and practice. The method and insights can potentially be generalized to other long-form summarization tasks.

Rating

8

Confidence

3

Ethics flag

1

Reasons to accept

1. Fine-tuning open-source LLMs (Mistral-7B and Zephyr-7B) on a large-scale dataset for long-form clinical summarization and evaluating on three diverse test sets. This demonstrates the challenges LLMs face in generating complete and faithful summaries. 2. Showing that content selection for this task should be treated separately from content realization. A dedicated, smaller content selection classifier outperforms the implicit content selection of auto-regressive LLM decoding. 3. Proposing SPEER, a method that marks salient entities with special tags in the source text and instructs the LLM to explicitly retrieve these entities before generating each summary sentence. This sentence-level planning improves coverage of salient content and faithfulness compared to non-guided and prompt-guided LLM baselines. 4. The code and data release will be a valuable contribution to the entire community.

Reasons to reject

1. It would benefit from adding a section for related works, including state-of-the-art LLM summarization methods. 2. It would benefit from an efficiency analysis (e.g., time efficiency, scale-up analysis).

Questions to authors

1. The current Hospital-Course Summarization seems to belong to a related work section. 2. A human/expert evaluation should be considered, especially regarding faithfulness analysis.

Ethics concerns details

N/A

Summary

This paper presents an approach for generating clinical note summaries (in particular, discharge reports). To overcome challenges in the lengthy source text, the authors first identify important entities and then retrieve the relevance context during the generation process on the basis of these entities. Automated assessments show this process is stronger than non-guided generation using existing LLMs.

Rating

5

Confidence

2

Ethics flag

1

Reasons to accept

- The healthcare domain is important and an area that could benefit a lot from LLMs - From what I can tell, the method is new and somewhat interesting. A similar approach could potentially be beneficial in other domains. - The evaluation focused on achieving high recall and a low hallucination rate -- both of which are critical to get right in this domain - The empirical results are strong - The work clearly fits the remit of COLM

Reasons to reject

- The main algorithm was described imprecisely (only an English-language description and a figure). I am not certain that I would be able to reproduce the process without a clearer description. - The evaluation was limited to automatic assessment of the reports. It would be stronger if (at least a sample) was assessed by professionals to ensure the suitability of the reports (even if the human assessment focused on the recall and hallucination, as the automated process did) - The relationship between this work and prior art was not sufficiently described, especially WRT other approaches that focus on entity guidance (more below).

Questions to authors

- There's prior work in guiding clinical note generation with salient entities (e.g., https://www.semanticscholar.org/paper/Attend-to-Medical-Ontologies%3A-Content-Selection-for-Sotudeh-Goharian/4c3b5f8db4f44ed4d24e15227a0da30f7c20a665?utm_source=direct_link). This work seems substantially different than those due to the focus on planning, but they probably ought to be discussed as prior related work. - Consider using an algorithm block to describe the algorithm more clearly and precisely.

Summary

The authors address the challenge of generating comprehensive and accurate hospital discharge summaries using LLMs. Fine-tuning open-source LLMs (Mistral-7B-Instruct and Zephyr-7B-β) on nearly 170k patient records revealed issues with incomplete and unfaithful summaries. To improve performance, the authors developed SPEER (Sentence-level Planning via Embedded Entity Retrieval), which uses a smaller, encoder-only model to predict salient entities and guide the LLMs. This method marks entity spans with special tags, enhancing content coverage and faithfulness. The proposed method demonstrated improved performance over non-guided and guided baselines on three diverse datasets.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

1. The paper focuses on a critical topic in the medical domain and the proposed method is helpful in guiding future work in generating more faithful and human-quality level long-document summaries. 2. The paper clearly pointed out the challenge of selecting salient information from long clinical documents.

Reasons to reject

1. Since the final summary is the concatenation of generated summary sentences, the author provided method might affect the coherence of the final summary. However, the authors do not evaluate the coherence of the summary or mention the importance of coherence in this specific application. 2. Given that the reference summary may contain information that cannot be supported by the source document (as mentioned by the author, especially on MIMIC), it is not clear that an increase in ROUGE score by author's proposed methods mean better performance. 3. For the remaining auto-evaluation metrics, it looks like a simple baseline may also achieve a high performance. For example, suppose a model simply extracts a sentence (can extract short sentences on purpose to make the final summary short or close to the length of other model-generated summaries) that has entity mentions from each ESG based on the source document and then concatenates these extracted sentences together. This final summary would get a Source-Grounded Recall (SGR) of 1 and a Hallucination Rate (HR) of 0. Similarly, this summary would get a BertScore and ClinDistill close to 1 since all sentences are directly extracted from the source document. Would this final summary be a good one? (assuming ROUGE do not reflect the quality of LLM-generated summaries (https://arxiv.org/pdf/2209.12356, https://arxiv.org/pdf/2301.13848)). 4. The authors claim that BSP and ClinDistill are SOTA metrics for faithfulness evaluation. However, BSP do not correlate well with human judgements and it is not clear whether ClinDistill is suitable to evaluate summaries generated by LLMs as the model is initially being evaluated over non-LLM generated summaries and errors from LLM-generated summaries might be harder to detect and hence lower correlation. 5. It is not clear regarding the accuracy of forming entity synonym groups (ESG). This makes the evaluation results related to ESG not convincing enough. 6. There is a lack of significant tests in the main tables. It is not clear whether some improved results are significant. 7. There is no human evaluation of the quality of model generated summaries. Are summaries generated by the author's proposed method more coherent, readable, and factual? Do humans prefer summaries generated by author proposed methods? 8. There are no example outputs of generated summaries from different models so the reviewer cannot develop an overall feeling about the quality of those generated summaries.

Questions to authors

1. The reviewer does not find the average total length of source documents for each example. 2. The mathematical formula for HR misses “|” on the right side.

Thank you for your rebuttal

I acknowledge that I have read the responses by the authors and I will keep my positive score.

Positive

The authors' responses make sense, and I keep my overall positive score.

Follow up on the rebuttal

Hi - Thank you for taking the time to review SPEER. We wanted to follow up and see if you had any lingering doubts after reading our rebuttal. Thanks again!