Embedding Trajectory for Out-of-Distribution Detection in Mathematical Reasoning

Summary

This work studied the ODD detection in mathematical reasoning, which presents a new measurement, TV score, based on the observed pattern collapse property and early stabilization of GLMs. It seems the first discussion on the OOD detection in mathematical reasoning.

Strengths

This appears to be the first discussion on OOD detection in mathematical reasoning. The authors clearly explain why the trajectory works in an understandable and empirical manner. The manuscript is well-organized, and the methodology is both simple and effective.

Weaknesses

1. Additional metrics commonly used for ODD should be included for evaluation. 2. The writing requires improvement. 3. Further insight into the potential impact of over-smoothing on the setting of the critical parameter $k$ is necessary.

Questions

1. Is it true that the embedding dimension should be fixed across layers to compute the embedding difference between neighboring layers? 2. To better support the claim of choosing $k \leq 5$, it would be beneficial to provide a quantitative analysis of over-smoothing occurrence when $k$ is set to a larger value. 3. For testing the ODD performance, could you report the evaluation results in terms of AUPR and F1? 4. Which datasets were used for the analysis in Figure 3? 5. There are some grammatical mistakes that need fixing. Here are a few examples: (a) "we need ..., then computer ..." in Line 150. (b) The sentence in Lines 161-162. (c) "Outliers ..., then lead ..." in Lines 164-165.

Rating

7

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors adequately claimed the limitations of this work.

Summary

This paper presents a trajectory-based method for OOD detection in the mathematical reasoning setting. OOD detection is extensively studied in the text setting and image setting. The main motivation of this work is claimed to that mathematical reasoning poses significant challenges to embedding-based methods due to its high-density feature of output spaces, but this feature causes larger discrepancies in the embedding shift trajectory between different samples in latent spaces. The proposed method uses trajectory volatility for OOD detection in mathematical reasoning. Experiments are conducted to validate the performance of the proposed method.

Strengths

This paper studies OOD detection in the mathematical reasoning setting, which is less studied compared to text and image setting. This paper uses examples to illustrate the motivation and key idea. This improves the accessibility of this paper.

Weaknesses

The motivation is not strong and convincing. Figure 1 illustrates two challenges with respect to the input space and out space respectively. It is not convincing, and more evidence or data analysis should be provided to make these two challenges solid. I do not buy that pattern collapse is generally hold for all types of mathematical reasoning tasks. The example is just a special case. The output can take any value in the real line, and the collapse probability not that high. The idea of the proposed method is not convincing. I am not convinced that early stabilization is generally held for mathematical reasoning problems. More evidence should be provided to justify it. The writing is also not precise. For example, in Equation (1), the domain of f is not specified. The notation \phi is not defined. Also, the experiment is not solid. There is no experiment to justify that the performance improvement is due to addressing the challenges in the input space and output space posed in the Introduction.

Questions

See weakness.

Rating

3

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

See weakness.

Summary

This paper studies the OOD problem in GLMs under mathematical reasoning and found that the the patter collapse phenomena in the output space. The trajectory violation score is proposed to distinguish the ID and OOD samples. A thorough evaluation shows that the proposal can outperform traditional algorithms under offline detection, online detection, and quality estimation.

Strengths

(+) This paper is well structured and well-written. (+) The authors conduct a thorough evaluation of the proposed methods

Weaknesses

(-) The input data for the empirical study in Figure 3 is not introduced. Thus, whether it can reflect the scenario of mathematical reasoning is not clear. It lacks theoretical analysis. (-) The relationship of the pattern collapse (in Figure 1) and the early stabilization (in Figure 3) is not clearly illustrated. (-) The datasets for experiments are quite limited.

Questions

1. What is the internal relationship between the pattern collapse (in Figure 1) and the early stabilization (in Figure 3)?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations are adequately addressed.

Summary

This work discusses a novel method for out-of-distribution (OOD) detection in generative language models (GLMs), particularly in the context of mathematical reasoning tasks. The key insights are: 1) The high-density output space in mathematical reasoning leads to a "pattern collapse" that causes larger discrepancies in the embedding shift trajectory between different samples in latent spaces, 2) GLMs exhibit early stabilization for in-distribution (ID) samples in mathematical reasoning, while OOD samples do not show this behavior.

Strengths

The authors propose a novel trajectory-based method called "TV score" that leverages the unique characteristics of the high-density output space in mathematical reasoning tasks to effectively detect OOD samples. This approach goes beyond the traditional OOD detection methods focused on uncertainty estimation and embedding distance measurement, which struggle in the challenging mathematical reasoning domain. Robust OOD detection is crucial for the real-world deployment of generative language models, as these models are susceptible to performance degradation when faced with out-of-distribution inputs. The authors' work addresses a practical and important problem in the field, as mathematical reasoning tasks are increasingly incorporated into language models with high-stakes applications. The authors' analysis of the unique characteristics of the input and output spaces in mathematical reasoning tasks provides valuable theoretical insights into the challenges posed by this domain for OOD detection.

Weaknesses

The underlying mechanism of the TV score method and its relationship to the observed "pattern collapse" in the output space is not fully explained. Providing a more detailed analysis and visualization of the trajectory dynamics, as well as the intuition behind the choice of trajectory volatility as the detection metric, would enhance the interpretability of the approach. The computational complexity of the TV score method is not discussed, which is an important consideration for real-world deployment, especially in resource-constrained environments. Investigating the computational efficiency of the method and exploring potential optimizations or approximations would be valuable for improving its practical applicability. The paper does not address the robustness of the TV score method to adversarial attacks, which is a critical consideration for the security of generative language models. Evaluating the method's performance under different types of adversarial perturbations and developing strategies to improve its robustness would be a valuable extension of this work.

Questions

What is the underlying mechanism of the TV score method, and how does it relate to the observed "pattern collapse" in the output space? How can providing a more detailed analysis and visualization of the trajectory dynamics, as well as the intuition behind the choice of trajectory volatility as the detection metric, enhance the interpretability of the approach? How does the computational complexity of the TV score method impact its real-world deployment, especially in resource-constrained environments? What steps can be taken to investigate the computational efficiency of the method and explore potential optimizations or approximations to improve its practical applicability? How robust is the TV score method to adversarial attacks, and what are the critical considerations for the security of generative language models? What steps can be taken to evaluate the method's performance under different types of adversarial perturbations and develop strategies to improve its robustness?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

NA

Thanks for reviewing & Invite feedback

Dear Reviewers, We want to express our sincere appreciation for your efforts and time spent reviewing our work and the insightful and constructive comments. During the rebuttal period, we have provided a detailed response to address all the concerns mentioned by all reviewers. With the reviewer-author discussion period half over, we kindly invite you to give some feedback on our responses, and we really hope our responses adequately address your concerns. Best, Authors