RA-DIT: Retrieval-Augmented Dual Instruction Tuning

Evaluation Setting and Performance Comparison Clarification

Thanks for your time and feedback. We would like to clarify misunderstandings regarding our approach and result comparison based on the weaknesses brought up. 1. **In our paper, evaluations on NQ and TQA were conducted using the original data released by Kwiatkowski et al. (2019) and Joshi et al. (2017). Correspondingly we quoted the 64-shot results from Table 8 in the ATLAS paper for comparison.** The NQ and TQA results in Table 10 of the ATLAS paper are based on the customized evaluation sets released by the KILT Benchmark (Petroni et al. 2021), which are not comparable to ours. (Note that for the other 6 KILT tasks, we quote the numbers from Table 10 of ATLAS paper for comparison.) 2. Thanks for bringing Hofstätter et al. (2022) to our attention; we will cite the paper in our related work section. However, it is important to note that **their evaluation setting differs significantly from ours**. **Our paper primarily evaluates the fine-tuned RALM in the few-shot setting, whereas Hofstätter et al. (2022) employ a multi-task supervised-learning setting**, leading to their higher reported performance. Specifically, the fine-tuning datasets we used, as detailed in Table 1, do not overlap with our evaluation datasets; and our main experiments are conducted in the in-context few-shot learning setting (Brown et al. 2020). In contrast, Hofstätter et al. (2022) train their model using a re-balanced combination of training sets from the KILT task (which comprises approximately 808K examples) and evaluate the model on the same set of tasks.

Re: Architecture Novelty Concerns

Thank you for your response and revising the rating accordingly. We would like to respectfully follow up and see if there are any remaining concerns we can address for a further improved rating. > The paper's architecture lacks significant novelty as it primarily relies on existing models and introduces minor modifications. This may limit its impact and originality in the field. Regarding the weakness raised on architecture novelty, we highlight that fusing independently optimized LMs and retrievers through lightweight fine-tuning is a strength of our approach **with potentially greater practical value and impact**, as also noted by Reviewer 68qv. This strategy extensively leverages existing pre-training efforts in both the LM and retriever components, which are typically resource-intensive and feasible to carry out in only a limited number of organizations. Additionally, we demonstrate that **the implementation complexity and computation costs for fine-tuning in-context RALM architectures can be further reduced through the dual fine-tuning framework proposed**, which does not involve end-to-end gradient back-propagation across both components **while still maintaining effectiveness**.

Response to Review Questions

Thanks for your time and feedback. Please find our responses to the questions below: 1. When early stopping is used, the total amount of data seen by the model is equal to the product of batch size and the number of steps taken, and the model may not be seeing a full epoch of data before fine-tuning stops. As outlined in Appendix B, our implementation does not apply a hard cutoff of 7500 examples per dataset. Instead, we recompute the dataset distribution given this cap and sample from all datasets based on the newly adjusted distribution. We chose the cap of 7500 examples after observing the fine-tuned model substantially outperformed the baseline model on our development sets. However, we avoided carefully tuning this hyperparameter to prevent overfitting to the development sets. **We have updated Appendix B to reflect the decision-making process**. 2. Prior work (Chung et al. 2022; Iyer et al. 2022) have demonstrated that fine-tuning the language model on a diverse collection of instruction-based datasets leads to improved model generalization for unseen instructions. We hence adopt a similar strategy by combining five categories of fine-tuning tasks to enhance the language model's knowledge utilization (dialogue, open-domain QA, chain-of-thought reasoning) and to improve its contextual awareness for prediction generation (reading comprehension, summarization). These categories were selected given their representativeness of practical knowledge-intensive language tasks. **We have updated Appendix B to explain this choice**. 3. Please refer to our ablation study in Table 7, where we evaluate our model's performance using various retrievers, including DRAGON+ (our default choice), Contriever, and a version of Contriever fine-tuned with the MS-MARCO dataset (Izacard et al. 2022). We found the model performance is indeed sensitive to the effectiveness of the retriever. Notably, using DRAGON+ results in significant performance improvement compared to using both versions of Contriever.

Response to Review Questions

Thanks for your time and feedback. Please find our responses to the questions below: 1. Our retrieved chunk has a maximum length of 200 words (sec 3.1). We use up to 10 chunks in the experiment. For 5-shot in-context learning, we have to adopt the parallel augmentation approach (Shi et al. 2023) instead of concatenating all retrieved chunks to stay within the 2k context window of Llama. For 0-shot experiments, we adopt the same setting for consistency. **We have added the comparison of parallel retrieval-augmentation vs. concatenation using top-3 chunks in the 0-shot setting in Appendix E.2.** 2. We agree this hypothesis might be valid; however, other methods including the base LM would likewise be ineffective when condition i) and ii) simultaneously hold. Nevertheless, as highlighted in footnote 2 of our paper, by incorporating the SQuAD 2.0 dataset – a reading comprehension dataset that includes questions that cannot be answered with the given support documents – we encouraged the model to respond with “I don’t know” when it lacks sufficient information to provide an answer instead of hallucinating one. This approach could actually reduce hallucination but we leave a careful study to future research, where the focus could be on enabling the model to abstain from responding or to request additional information when needed. 3. Yes, the IT 65B baseline is instruction-tuned with the same blended dataset without retrieval augmentation. We hypothesize that the competitiveness of the IT 65B model in the top-1 setting is due to the inclusion of reading comprehension and summarization tasks in our fine-tuning dataset blend. Such tasks likely improve the model's capacity for contextual interpretation, which is helpful for generating accurate answers. However, **without retrieval-augmentation, the IT 65B model is not exposed to the broader and potentially noisier content returned by the retriever. This limits its ability to generalize, especially when more retrieved chunks are included.** 4. **We have added the top-3 results in Table 4.** For both Llama 65B and RA-IT 65B, we observe monotone performance increase as the number of retrieved chunks increase. **The performance of IT 65B slightly dropped going from top-3 to top-10, further validated that this baseline is less effective in processing noisier context.** 5. In Table 2 main results, the NQ results of Llama 65B are obtained using the close-book setting. Llama 65B performs poorly on NQ in the 0-shot setting because it struggles to generate answers that match the short format of the ground truth responses. 6. In Section 5.2, we presented an ablation on the dual instruction tuning approach where we analyze the individual impact of LM-ft and R-ft. As shown in Table 6, R-ft consistently improves the performance of both the base Llama 65B and the RA-IT-ed Llama 65B, Our final approach combines LM-ft and R-ft since it yields further gains over using either LM-ft or R-ft alone. 7. InstructRetro (Wang et al. 2023) is a very interesting concurrent work which applies instruction tuning to the Retro architecture (Borgeaud et al. 2021). **The paper was released on ArXiv post ICLR submission**, which reinforces that enabling LMs to better integrate external knowledge in downstream tasks is an important direction. This approach involves encoding and integrating retrieved content into the language model (LM) using separate encoders and cross-attention mechanism. In comparison, RA-DIT adopts a more streamlined architecture and training procedure by separating the fine-tuning processes of the LM and the retriever. Although the differences in base LM, fine-tuning datasets and inference settings make direct comparisons between the two models challenging, RA-DIT 65B compares favorably to InstructRetro 48B in zero-shot settings on shared evaluation datasets: TriviaQA (RA-DIT 75.4 vs. InstructRetro 65.6) and NQ (RA-DIT 35.2 vs. InstructRetro 38.9). We will add this discussion to our related work.

Fine-tuning Process Clarification

Thanks for your time and feedback. In our paper, we fine-tune the retriever using the base LM (as opposed to the already fine-tuned LM); and similarly, we fine-tune the LM using the retrieval results from the base retriever. The term “independent” in our paper is used to indicate that the LM fine-tuning step and retriever fine-tuning step can be carried out concurrently. We hope this helps to clarify the depiction in Figure 1, which illustrates the non-sequential nature of these two steps. As shown in Table 6, we found combining the separately fine-tuned LM and retriever yields further gains. While we leave sequential fine-tuning out of the scope of this work, we recognize it as a conceptually compelling alternative method that merits exploration in future research.