Towards Semi-Structured Automatic ICD Coding via Tree-based Contrastive Learning

Summary

This paper addresses automating diagnosis coding via tree-based contrastive learning. It uses an established benchmarking dataset for this task, and achieves insightful results from its comparative performance evaluations and ablation studies.

Strengths

This paper addresses automating diagnosis coding via tree-based contrastive learning. It uses an established benchmarking dataset for this task, and achieves insightful results from its comparative performance evaluations and ablation studies. It has a well chosen range of machine learning methods to compare in the coding task, and its methods are very clearly and thoroughly described.

Weaknesses

The paper is somewhat lacking in its qualitative analysis. It would be helpful to extend Section 5 with a mixed method approach to analyse and evaluate the experimental methods also using qualitative approaches. The paper is also calling for a discussion section to deepen the lessons learnt part of the study. For example, what are the limitations of the study? What are the envisioned pros and cons of the studied methods in their broader context in health and medicine? What are the ethical considerations related to using the MIMIC -III dataset? It would have been helpful to connect this natural language processing paper to prior shared tasks and their shared datasets to allow seeing trends in methods. E.g., Computational Medicine Center's 2007 Medical NLP Challenge, followed by those by I2B2, N2C2, and CLEF eHealth would be worth of briefly surveying to assure that methods are compared to more traditional ones as well.

Questions

See above

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See my broader impact and ethical consideration comments above for minor comments

Summary

This paper describes a novel method of ICD coding that explicitly model clinical note sections. Instead of treating a clinical note as a long sequence of tokens, the authors propose to segment a clinical note into sections and then use contrasive learning to pre-train the model. Experiment results on MIMIC-III show that the proposed components can be used to improve the effectiveness of several existing CNN, RNN, Transformer-based models, especially when the training data is limited.

Strengths

* A simple yet effective contrastive learning variant based on label tree * The proposed (section segmentation and contrastive learning) components are used with several existing models and are shown to be effective, especially with limited training data.

Weaknesses

* The used baselines are mainly CNN, RNN based. The only used transformer-based baseline PLM-ICD seems not to be a strong one. See suggestion 3

Questions

* Suggest swapping equations (1) and (2) * Section 4.3: can you explain what is the role of perm operation (or saying, why it is necessary)? * Suggest to consider stronger transformer-based baselines [1] [1] Xiang Dai, Ilias Chalkidis, Sune Darkner, Desmond Elliott, "Revisiting Transformer-based Models for Long Document Classification", in Findings of EMNLP, 2022.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

The paper proposed a semi-structured automatic ICD coding algorithm with a contrastive pre-training and masked section training and evaluate the algorithm using MIMIC-III dataset.

Strengths

The paper is well structured with clear explanation in research motivation, related work, experiment configuration and results. Empirical results were obtained with multiple baseline models on benchmark dataset MIMIC-III. Code and data are also provided.

Weaknesses

1. In the related work part, the paper misses the weak supervision approach applied on ICD coding. 2. Result tables (Table 1 and Table 2) and result plots (Figure 4) miss confidence intervals (CIs). Standard deviations are not intuitive to illustrate the variance for model comparison, especially only 5 repetitions are performed. Pls compute the CIs for each results, which should be straight forward. 3. The comparison of w/C and w/M in Figure 4 is interesting to illustrate the utility and necessity of section identification in your proposed framework. Better to provide a table with results and confidence intervals in main text or supplement. Otherwise, the performance difference looks very trivial. In general, the model evaluation is weak in the current representation, which might eliminate the technical soundness.

Questions

1. Are the results cross validated? It is not specified in the main text or supplement. Pls justify the significance and generalizability of the proposed model.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

Yes. Limitations are addressed in the paper.

Summary

The paper tackles automatic ICD coding. It lists challenges and proposes solutions to them: 1. For ignoring structural information, it proposes a content-based algorithm that automatically segments clinical notes into sections. 2. For limited availability of data and variability of clinical notes, it proposes a contrastive learning framework based on a soft multi-label similarity with tree edit distance and a masked section training strategy. The authors conduct extensive experiments on MIMIC-III and demonstrate that their proposed methods can enhance the performance of existing ICD coding models.

Strengths

- The paper is well-written and easy to follow. The introduction, related work and preliminaries gives a clear background and motivation to the problem. - Code is provided in the supplementary material. - DF-IAPF is neat, easy to implement, and fast to run. - DF-IAPF has two assumptions: 1) section titles have high document frequency; 2) section titles have low average phrase frequency. While there are corner uses where a section title only appears in some notes, or it appears multiple times in a note, the paper provides both qualitative and quantitive analysis that compare DF-IAPT with a rule-based algorithm and demonstrates the effectiveness of DF-IAPT. - Soft multi-label similarity based on tree edit distance captures the hierarchy difference between two ICD label sets. - The choices of tasks cover frequent, rare, and the entire ICD codes in MIMIC-III. - Experiments verify the effectiveness of proposed methods on a diverse set of backbones. - Ablation studies clearly show the improvement of each method proposed in this paper.

Weaknesses

An expert selection process is needed after DF-IAPF proposes a candidate set. How important is the expert selection process? There should be a comparison between DF-IAPF fully automatic and DF-IAPF followed by an expert selection.

Questions

- Line 238: Is h_section more accurate than h_note? - Typo in line 12 of Algorithm 1 in Appendix A: TF(t) -> PF(t)?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

This paper includes a section in Appendix, discussing broader impacts. The conclusion section also briefly states the limitations of this work. It would be better to have a separate section to discuss these limitations.

Rebuttal response

Based on the clear and convincing response by the authors, I have revised my review scoring.

Thank you

Thank you for providing the answers to the questions as well as confidence intervals for the experiments. However, it seems that a two-side t-test is performed to check the difference in distributions, instead of a one-side t-test to prove that the performance is improved from one to the other. It's not enough to indicate the proposed method is better than baseline. Pls justify. Thank you!