Timo: Towards Better Temporal Reasoning for Language Models

Summary

This paper presents a framework to improve the general temporal reasoning capabilities of LLMs. The authors empirically verify the intuition that temporal reasoning is closely related to math reasoning, which motivates the proposed approach that adopts the LLM-as-Judge scheme using a math LLM and temporal reasoning instructions to produce temporal-related pairwise preference dataset, which is later adopted for training a DPO model called TIMO. The authors conduct experiments on TRAM (a temporal reasoning benchmark consisting of 10 categories) and show that TIMO outperforms the math LLM under most cases.

Rating

5

Confidence

4

Ethics flag

1

Reasons to accept

1. This work empirically shows that temporal reasoning benefits from better math reasoning abilities. Such an assumption has been indirectly studied before, but no prior work has really proved it. 2. The produced TIMO checkpoint, upon being publicly released, could be a good option for direct usage on temporal reasoning tasks.

Reasons to reject

1. The final DPO step uses the data distilled and evaluated by the math-LLM through the LLM-as-Judge framework. However, there is no comparison with adopting a non-math LLM (e.g., vanilla LLaMA or generally-SFT LLaMA) for generating such data. Existing work on weak-to-strong generalization suggests that distilling data from a weaker or equivalent LLM benefits a stronger LLM. Thus, it is possible that the improvements shown by TIMO results from the LLM-as-Judge framework rather than the math LLM (which is considered as the main contribution of this work). 2. Among all models that are compared against TIMO, TIMO is the only one that has been trained on the instructions from TRAM. So one concern is whether the improvements of TIMO over math-LLM result from better general temporal reasoning or specialized adaptation on TRAM. Some evaluations on other held-out temporal reasoning datasets could be much more convincing.

Summary

Comments after Author Discussion Reasoning about time is essential for Large Language Models (LLMs) to understand the world. In this paper, the authors propose a simple but effective self-critic temporal optimization method (TIMO) to enhance the model’s temporal reasoning capabilities without sacrificing general task abilities. It’s designed to excel in temporal reasoning at the 7B and 13B scales and outperforms the LLaMA2 by 10 and 7.6 in average accuracy scores on 38 Temporal reasoning tasks.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

1. In this paper, the authors discover the inner correlation between time and mathematics, where temporal reasoning could benefit from math instructions by systematically studying diverse temporal reasoning tasks. 2. The authors make the first attempt to build a unified framework to address 38 temporal tasks. They propose a simple but effective self-critic temporal optimization method to strengthen the temporal reasoning capabilities comprehensively upon mastering mathematical reasoning capabilities.

Reasons to reject

1.In section ‘Main Results’, there is lack of analysis on comparison with the other models. 2. In section ‘Ablation Analysis of Framework’, there is lack of analysis on comparison with the GPT4 model. Why do the performance differences between TIMO and gpt4 differ on different tasks? The author should make appropriate analysis . 3. In table 1, the result ‘83.0’ of column ‘Pure-time tasks-AMB.’ is not the best result. Suggest the authors to highlight the second results in the table as well. 4. In case study, there is a certain ambiguity in the first case (Math-CoT lacks prerequisites ‘1652 Jan 1st is Monday’). Suggest the authors to replace other case.

Questions to authors

None

Summary

This paper proposes a model, Timo, achieving new SOTA at 7B and 13B scales, which leverages the mathematical datasets to set a solid foundation model for temporal reasoning and enhance with self-critic temporal optimization method.

Rating

7

Confidence

5

Ethics flag

1

Reasons to accept

1. For temporal reasoning, it leverages the mathematical dataset as the foundational SFT data and uses DPO with temporal reasoning datasets for further improvement. 2. Experimental results demonstrate the performances of Timo are even better than GPT-4 in some areas. The analysis of the framework is insightful, and can deeply inspire other reasoning tasks.

Reasons to reject

1. More generalization of the framework should be validated in other reasoning tasks.

Looking forward to further discussion

Dear Reviewer Q4nZ, Thanks for your review! Considering the response deadline is approaching, we hope to have more discussions with you. If you have any further inquiries or suggestions, please do not hesitate to reach out to us! Best, Authors of TIMO

Looking forward to further discussion

Dear Reviewer QQVe, To address your concerns, we have provided additional model comparisons, improved case studies, and conducted new experiments on LLaMA2-8B to verify the effectiveness of TIMO. With the rebuttal period closing soon, could you please check the response to see whether it mitigates your concerns? If you have any further inquiries or suggestions, please do not hesitate to reach out to us! Best, Authors of TIMO

Looking forward to further discussion

Dear Reviewer P1tK, To address your concerns, we have conducted experiments on vanilla LLaMA-7B and LLaMA2-7B-chat to validate the importance of the math LLM in our framework. Additionally, we have evaluated TIMO on MCTACO and TempReason to demonstrate its effectiveness. Since the rebuttal period is closing very soon, could you please check our response to see whether it mitigates your concerns? Feel free to reach out with any further questions or suggestions! Best, Authors of TIMO