Improving Few-Shot Generalization by Exploring and Exploiting Auxiliary Data

Summary

The paper presents a new approach to Few-shot Learning with Auxiliary Data (FLAD), a method that uses additional data to improve model generalization. Traditional methods for integrating auxiliary and target data have limitations, particularly in terms of computational scale with an increasing number of auxiliary datasets. This study introduces two new algorithms, EXP3-FLAD and UCB1-FLAD, that relate FLAD to the explore-exploit problem in multi-armed bandit scenarios, thereby achieving computational complexity that is independent of the quantity of auxiliary datasets. This paper shows that a combined exploration and exploitation strategy is vital, with the new methods surpassing all existing FLAD methods by 4%. The research also yielded the first 3 billion parameter language models that outperform the 175 billion parameters GPT-3.

Strengths

* This study introduces a novel approach to enhance few-shot learning in language models by leveraging auxiliary data. The innovative use of additional datasets significantly improves the generalization capabilities of the models in the few-shot learning scenario. * The incorporation of an exploration/exploitation framework, often seen in multi-armed bandit scenarios, adds an interesting aspect to the methodology. It ensures the computational complexity is independent of the number of auxiliary datasets, thus addressing the significant issue of scalability often seen in previous methodologies. * The experimental results displayed in Table 1 and Figure 2 are good and offer strong support for the efficacy of this approach. * One of the most exciting features of this methodology is its universal applicability; it can be effectively utilized with any language model. It provides a path to significantly elevate the performance of relatively smaller models, leveraging them to outperform even the more sizable models, as demonstrated by the 3 billion parameter language model surpassing the 175 billion parameters GPT-3.

Weaknesses

* Figure 2 presents an interesting picture of the model's performance, revealing an interesting dependency on the auxiliary data for some tasks, while seeming almost indifferent between using T0 or T5 for others. The interpretation of these results is not straightforward and questions about the behavior exhibited by the model across different tasks. * In terms of its responsiveness to auxiliary data and the use of T0 or T5, the performance differs significantly from task to task. This variance potentially suggests that certain tasks might be more intrinsically aligned with the structure or features of the auxiliary data, or that some tasks may respond more favorably to the model architectures embodied by T0 and T5, but there is no ablation or discussion about this aspect.

Questions

* Why is the policy configured to select a singular specific auxiliary dataset? Could there not be more value in a more flexible approach, allowing the policy to identify and utilize the most appropriate samples across the entirety of the auxiliary data? Especially considering the successes achieved with cross-task transfer for T5 and T0, it seems that an exploration of this approach could be beneficial. * Figure 2 the HellaSwag task, where all methods show inferior performance, with the exception of GPT-3. Could there be certain features inherent to this task that align better with the GPT-3 model, or are there particular aspects of the task that the other models struggle with? Do you have some intuition? * Do you have updated results against GPT-4?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

* I do not have much practical experience with RL and it is difficult for me to judge the limitations of this method.

Summary

The paper addresses the problem of using auxiliary data for the few-shot learning. The main novelty of the proposed approach is to formulate the selection of the auxiliary dataset from a pool of datasets as a multi-arm bandit problem (balancing exploration and exploitation) and adapting EXP3 and UCB1 to that goal. This allows to scale the number axillary datasets and improve generalization in the few-shot learning. The method is evaluated in application to NLP.

Strengths

Important problem Improves the result with a smaller model Shows nice empirical results

Weaknesses

The novelty of the paper is incremental There are some problems in the presentation that cause confusion as explained below.

Questions

One of the main claims in the paper is that the proposed approach almost doesn't add to computational cost. The description in line 172-174: " This is done for each auxiliary dataset by calculating the gradient ∇a = ∇θL(fθ, x, y), where the number of samples in {x, y} is significantly larger than a mini-batch, and can be up to the size of the full dataset." doesn't seem to go along with that clain, as this computation is large and it has to be done in every round. Do I miss something here? A more minor thing that needs to be taken care of is in line 219. Exploit-only is suppose to be an opposite to Explore-only. So it's confusing how Exploit-only is an extension of Explore-Only. The text later explains the idea, but it needs amendment.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

I did not really see any clear discussion of limitations. While the selection of the datasets are done using the proposed algorithms of balancing exploration and exploitation, there is still an implicit assumption that the datasets are useful. I believe that in language tasks this is less of a problem but in general it could be the case.

Summary

In the context of Language Models, this paper studies the use of Auxiliary Data during training to gain in generalization. The Few Shot settings is also considered here where proposed tasks only have limited set of labeled data. Compared to previously proposed approaches using auxiliary data, here it is consider to exploit more auxiliary dataset than before. In this context, then it is crucial to select what are the good auxiliary datasets to be used (otherwise performance could degrade) with a reasonable complexity. A multi-armed bandit (MAB) approach is proposed here to iteratively refine a selection policy among these auxiliary datasets. Two MAB algorithms have been studied (EXP3 and UCB1) in conjunction with different reward functions (based on gradient alignment, gradient magnitude alignment of a combination of both). Experiments are reported on specific training of T5-XL and T0 model. Significant gains with respect to previously proposed approaches are reported. Moreover the complexity associated to the proposed method allows to use high number of auxiliary datasets which helps to boost the performance further. Discussions on the various aspect of the method are also provided.

Strengths

The paper is clearly written. The adaptation of MAB here allows to derive a performant policy to select auxiliary datasets for training. Specifically since model updates and policy updates are intrinsically performed, the overall complexity is not armed by the number of auxiliary dataset. Significant gain are reported with respect to previous works, and relevant discussions are provided upon the importance of rewarding functions used and associated algorithms (e.g. benefits of having exploration and exploitation phases thanks to the bandit).

Weaknesses

As mentioned by the authors, the proposed training has to be performed from scratch for every domain.

Questions

1. Author mentioned that complexity is independent of the number of auxiliary datasets (l. 370-371). Is it really true? When considering even higher number of auxiliary datasets, it could need more iterations to reach a good policy $\pi$ among datasets (need more exploration). So aren't there really an independence there? Or practically since rewards distribution tend to converge quite fast (see fig 4 in annex), impact should be limited?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Authors adequately addressed associated limitations.

Summary

This paper focuses on the problem of few-shot learning with auxiliary data (FLAD) problem. The authors formulate FLAD as a multi-armed bandit problem, and employ exploration-exploitation algorithms to tackle it. Experiment results show that the proposed method outperforms exploration or exploitation-only approaches by significant improvement in accuracy.

Strengths

- This paper models the FLAD problem with the multi-armed bandit framework, which makes a difference from previous works. The proposed method scales well to the number of auxiliary datasets, and adds a limited amount of computational and memory overhead. - The proposed approach shows a significant accuracy improvement over other exploration-only and exploitation-only baselines.

Weaknesses

-The novelty is limited since the proposed multi-armed bandit method is a existing technique. - The authors should clarify the difference in definitions and settings between FLAD and other similar problems such as general few-shot learning and multi-task learning. - More experiment results had better be provided. For example, in order to demonstrate the scalability of the proposed method, it would be better to provide what the changes of performance be like with different values of |\mathcal{A}| or |\mathcal{D}_T|.

Questions

Please address the concerns in the Weaknessese.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Yes.