RECKONING: Reasoning through Dynamic Knowledge Encoding

Summary

The paper introduces a two-level learning algorithm called RECKONING, which enhances the in-context reasoning performance by addressing the issue of distractor facts. The algorithm consists of two learning steps, an inner and an outer loop, where the inner loop trains the model to encode contextual knowledge into its parameters through BP, and the outer loop teaches the model to use the updated parameters to answer questions. The experimental results on two multi-hop reasoning datasets indicate that RECKONING outperforms the ICR baseline by up to 4.5%. Additionally, RECKONING generalizes better to longer reasoning chains unseen during training, is more robust to distractors in the context.

Strengths

Overall, the research topic presented in this work is of interest and importance, as it enhances transformer-based language models by incorporating in-context knowledge into parameters to improve reasoning ability. The proposed RECKONING consistently outperforms the conventional ICR-based methods. The proposed method's superiority is further confirmed through comprehensive experiments and ablation studies, which demonstrate its superiority in generalization to longer reasoning chains.

Weaknesses

While the research topic is interesting and the proposed algorithm outperforms conventional in-context reasoning methods, there are some weaknesses in the paper. The proposed inner-loop of encoding knowledge into model parameters to improve reasoning abilities is limited in its incremental nature, which may lead to issues when facts from different contexts contradict each other. For example, the fact that "Peter is a son to Kyle" may be changed to "Kyle is a son to Peter" in another question's context. Additionally, the experiment part compares two baselines "No-Facts" and "Randome-Facts," which are too weak, and no other publicly available baselines have been compared.

Questions

1. In Figure 3, it would be helpful to include more experimental results with training on both 4- and 6-hop questions, or both 2- and 4-hop questions to further validate the proposed algorithm's performance. 2. It would also be beneficial to include other subtasks like proof generation in the ProofWriter dataset to further explore the algorithm's capabilities. 3. Conducting more experiments on public pre-trained language models like LLaMA would provide additional validation for the proposed method's superiority.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

I can not find any potential negative societal impact.

Summary

This work aims to solve reasoning tasks where models need to rely on knowledge provided as part of the task input. Motivated by the fact that pre-trained models encode a lot of irrelevant facts and they are good at retrieving the relevant ones for solving a downstream task, this work proposes an algorithm, RECKONING, that updates a model’s parameters to memorize the facts on the fly and then answer the question without having those facts as input explicitly. Experiments show that the proposed method can improve the model performance compared with the in-context reasoning baseline. The resulting model also generalizes better to examples requiring longer reasoning chains, is more robust to distractors and even more efficient in a certain setting.

Strengths

- Originality: Unlike prior works that propose different ways to filter irrelevant facts explicitly, this work makes use of the pre-trained LM to subsume all the facts and filter the irrelevant facts on its own. The idea is interesting and makes sense. - Quality: To validate the effectiveness, the work conducts sufficient experiments. The results demonstrate the superiority of the method in scenarios where a longer reasoning chain is needed or there exist a lot of distractors, which well motivate this study. An analysis is also provided to address the concern about the computation efficiency of the method. - Clarity: The algorithm is well explained despite requiring some effort to understand it.

Weaknesses

- In the standard setting where there are no added distractors, the improvement brought by the method is actually minimal (Table 1). - Also, the need to update the parameters on the fly makes the algorithm hard to be applied on large language models, which might have a greater capability in filtering irrelevant facts in-context already. I would suggest extending the idea to these large LMs by updating a subset of parameters (like adapters do). - One potential disadvantage of the method could be that the knowledge facts contain a lot of noise or toxic content that may undermine the model’s basic ability for reasoning. In this regard, methods that first filter irrelevant facts and then provide the remaining facts in-context do not have such an issue.

Questions

1. For the Random-Facts baseline, do you mean the random facts are provided during inference or also during training? Do you train it with the multi-task objective?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

No limitation is discussed.

Summary

This paper introduces a novel method for addressing logical reasoning in natural language question answering, specifically when the required knowledge is part of the context. One major challenge highlighted in this paper is that the included knowledge often contains irrelevant information, which can mislead the reasoning process and negatively impact question-answering performance. The proposed solution involves a bi-level optimization technique during inference. The technical key is utilizing the knowledge to quickly fine-tune the model at inference time, using a causal language modeling objective. This eliminates the need for the knowledge to be explicitly included in the context, as it is encoded within the model parameters. It also helps the model to focus on the relevant parts of the knowledge. To evaluate the effectiveness of the approach, multiple baselines incorporating variations of in-context learning are compared. The experimental results demonstrate improved performance using the proposed method, particularly in terms of generalization over longer reasoning chains.

Strengths

Pros: The proposed approach is sound and interesting. The paper is well-written. They show that their inference-time optimization makes the model more robust to the redundant information. It becomes more generalizable to longer chains of reasoning. It is more efficient when multiple questions are asked based on the same context.

Weaknesses

The run-time analysis is to some extent misleading. For the mentioned problem setting, in general, we assume each example comes with its own context that is the context is not often shared. Therefore, I guess we expect this model to be less efficient in general for reasoning as it needs the bi-level optimization at inference time. I think highlighting the efficiency in the multi-question setting looks a bit far fetch.

Questions

—While it is expected that the model learns to focus on the relevant parts of the knowledge, it is not very clear to me why the approach should make the model more generalizable to longer chains of reasoning? Any intuition? —Is there any SOTA results better than what you report in this paper in your baseline variations? -- As it is explained in the paper, for the multiple question setting some parts of the knowledge are relevant for one question while irrelevant for others. Since the model is tuned once and the same model is used for answering all questions then the performance should be lower in this case, right?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitation section is not included in the paper.

Summary

Summary: The paper introduces RECKONING, a bi-level learning algorithm designed to improve reasoning in transformer-based language models. RECKONING encodes contextual knowledge into the model's parameters using gradient updates, allowing the model to answer questions based on its updated parameters. The authors demonstrate, through experiments on two multi-hop reasoning datasets, that RECKONING outperforms an in-context reasoning baseline and is more robust to distractions, generalizes better to longer reasoning chains, and is more computationally efficient under certain conditions. Contributions: 1. Propose a bi-level learning algorithm, RECKONING, to teach language models to reason by updating their parametric knowledge through back-propagation. 2. Conduct experiments on two multi-hop reasoning datasets, ProofWriter, and CLUTRR-SG, showing that RECKONING outperforms the in-context reasoning baseline and provides several other benefits. 3. Provide analyses on the ability of RECKONING to memorize knowledge, measure its performance under distractor conditions, and analyze its run-time efficiency, etc.

Strengths

1. the idea of encoding knowledge into the model's parameters through gradient updates is an interesting and novel idea in the field of natural language reasoning 2. RECKONING demonstrates better performance than the baseline model in several aspects, including better generalization to longer reasoning chains and its robustness to distractions 3. The paper is well-presented

Weaknesses

The main weakness I see for this paper is the scope of its conducted experiments: 1. The experiments are conducted on two synthetic multi-hop reasoning datasets, ProofWriter and CLUTRR-SG. While these analyses provide valuable insights, further evaluation on a broader range of real-world datasets would strengthen the generalizability of RECKONING's performance. 2. The experiments are conducted only using the base GPT-2 model, which is really behind the state of the art. It's hard to tell if on the best performing models we can still see this impovement.

Questions

1. How would RECKONING perform with more complex and diverse real-world reasoning tasks beyond the synthetic multi-hop reasoning datasets used in the experiments (ProofWriter and CLUTTR-SG)? E.g., HotpotQA? 2. Is this improvement agnostic to model architectures? How does RECKONING do on seq2seq models? 3. In the multi-task training setting, does the choice of weighting different terms affect the performance of RECKONING? (from the paper is 1:1?)

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

NA

Summary

The paper presents a two-phase learning algorithm that 1) encodes background knowledge in the parameters of an LM through fine-tuning (instead of providing it in context), and 2) learns how to use and reason the encoded knowledge for a given question (?). The paper shows better performance than in-context learning, and better generalization to longer reasoning chains.

Strengths

- The paper is easy to read for the most part. - The generalization experiments results support Reckoning as a useful approach.

Weaknesses

- Most of my concerns / questions have to do with Phase 2 of the model updates. More details given below in questions.

Questions

- Does the order in which the questions are seen effect the results? - Which task from the Proofwriter dataset was actually used in this paper? - Is the outer-loop of updates actually needed? Finetuning on knowledge and then trying to answer the question directly (without any more parameter updates) is a natural baseline to compare against. I believe this is slightly different from the FT-ICR setup. - In any case, the performance difference between FT-ICR and Reckoning is only somewhat apparent on the CLUTTRR dataset. Can you conjecture why that might be the case? - I think the bigger performance gaps are noticeable in the OOD generalization setup. Can you provide an intuition / analysis of why generalization gets better? - At multiple places, the paper says phase 1 is to memorize the knowledge and phase 2 is to quickly memorize the given knowledge and perform reasoning. e.g. lines 108 to 111. Can you clarify why one needs 2 steps to memorize the said knowledge. Is the knowledge being memorize different in these two phases? Moreover, it seems memorize and "learn" are being used interchangeably, but I believe they are different. Overall, phase 2 application and its impact are confusing. - If the knowledge from a question is being encoded in the model, how does the model handle contradictory information? For example: it is possible "A is B's son" in instance 1, but "B is A's son" in another instance. Is there an assumption that this is not possible?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

-

Author response addressed my questions

Thanks for taking the time to run additional experiments, and the experiment on FOLIO looks great.

Rebuttal Response

Thank you for the response and for clarifying my questions. The new results are also interesting to learn about. I've also read the other reviews and responses. I was already positive about the paper and I am happy to raise it a bit in light of the strong new results that've been reported during the rebuttal phase. Thanks!

Thank you for improving your rating!

Many thanks to the reviewer for their helpful and constructive suggestions! We are grateful for the reviewer raising their score from 4 to 5.