Thank you for your follow-up comments and for acknowledging the clarifications provided in our response.
### Response to W1:
> W1. regarding effectiveness (the RougeL score): In comparison with Tk-Instruct in Table 2, there was little improvement. Although TAGI outperformed T0 in Table 3, it may be due to the knowledge distillation from Tk-instruct because TAGI used Tk-instruct, which is a refined version of T0, as a teacher model. That is, TAGI would not be effective in comparison with the standard instruction tuning.
In Table 2, our method outperforms similar hypernetwork-based approaches, surpassing the previous best method HINT by more than 1.5. Although the improvement in the 'Def' case for the traditional full fine-tuning method Tk-Instruct is modest, our approach uses fewer fine-tuning parameters, training resources, and less training time. In the 'Def + 2 Pos.' case, our method significantly outperforms Tk-Instruct, except for the 11B model.
The results in Table 3 do not pertain to Tk-Instruct. Instead, we fully meta-trained a model, replacing 'metatrain,' which incorporates versions of two different models: T5-lm and T0. From Table 3, it is evident that our method surpasses existing results for meta-training, including multi-task standard instruction tuning and all TAGI results. Additionally, our training parameters and overhead are considerably reduced, leading to a 10% reduction in computational requirements for inference.
Additionally, in the ablation study shown in Table 4, the T5-XL model achieves a score of 57.3 even without knowledge distillation (w/o L_kl), which surpasses the metatrain result of 53.1.
### Response to W2:
> W2. regarding efficiency (inference cost): I understood that there is an advantage when there are multiple samples. On the other hand, even with standard self-attention, it is possible to decode multiple samples efficiently by using cache KV if the model is decoder-type. L237 states `We limit our scope to encoder-decoder models for our experiment'. However, it is misleading to state that the scope of the experiment is limited. I consider that the scope of the research (or, the usefulness of the proposed method) is limited to the encoder-decoder-type and this should be discussed as the limitation.
Thank you for highlighting this issue. We acknowledge it as a limitation and have discussed it in Appendix B.1. Our approach aimed to compare with prior work, and therefore, we followed the settings of those studies. However, contemporary decoder-only models such as LLaMA3, QWen2, and other advanced models have already undergone extensive QA for Supervised Fine-Tuning (SFT), serving as robust baselines. Additionally, the continued improvement in model performance using efficient parameter fine-tuning methods like LoRA on decoder-only models suggests the feasibility of generating efficient fine-tuning modules via hypernetworks. Even though the use of KV-cache can reduce computation, it does not undermine our method's initial goal of encoding instructions only once to minimize computational load from the input side. In Table 2, we used the results from the HINT paper for GPT-2 XL (1.5B) and OPT (13B), which perform lower than encoder-decoder models of the same caliber. However, these results do not represent the most advanced decoder-only models available today.
**Next, let's analyze the KV cache.**
Here, we will let $l$ be the number of layers, $d$ the model hidden dimension, $h$ the number of heads, $k$ the size of the keys/values, and $s$ be the length of the sequence we want to save. We ignore biased terms for simplicity.
For decoder-only models, if we want to cache the key/value pairs for a given sequence, we will store $2lhks$ values - a key and value for every head in every layer, for each item in the sequence. Assuming \($kh = d$\), the memory cost of decoder-only models is proportional to \($lsd$\).
In the default TAGI settings, we store two main components: the processed instruction sequence, which consists of \(ds\) values (one vector per token), and the LoRA weights, totaling \($2 * 32 * l * h * k * 2$\) values (with a 32-rank LoRA for both key and value per layer per head). The total memory cost is therefore \($ds + 128 lhk$\). Note that the default LoRA rank is set to 32, but this parameter can be adjusted to manage memory costs. Assuming \($kh = d$\), the memory cost of TAGI is proportional to \($ds + ld$\).
In conclusion, although decoder-only models can save computations through KV caching, our method only computes the instruction once, directly reducing the processing length at the input end. Therefore, it has better scalability in terms of sequence length (larger $s$) and model size (larger $d$, $l$).