Theoretical Investigations and Practical Enhancements on Tail Task Risk Minimization in Meta Learning

Summary

This work investigates the tail risk minimization in meta-learning from theoretical and practical perspectives. Overall, this work is well-written, novel and theoretically enriches TR-MAML[1]/DR-MAML[2]. In the realm of large models, meta-learning plays a crucial role due to the pressing concern of distributional robustness across various tasks, particularly in risk-sensitive applications. Here, I express my positive attitude based on this work's completeness, novelty, and workload. [1] Collins, L., Mokhtari, A., & Shakkottai, S. (2020). Task-robust model-agnostic meta-learning. Advances in Neural Information Processing Systems, 33, 18860-18871. [2] Wang, Q., Lv, Y., Xie, Z., & Huang, J. (2024). A simple yet effective strategy to robustify the meta learning paradigm. Advances in Neural Information Processing Systems, 36.

Strengths

The novelty of this work lies in two aspects. (1) This work reduces the distributionally robust meta-learning to a max-min optimization and performs analysis on tail risk generalization, convergence rate estimation, and the role of quantile estimation; (2) This work enhances DR-MAML through more accurate quantile estimates with theoretical support. Overall, this work clarifies its contribution in Table3 and completes all claims together with detailed proofs. Extensive experiments further verify theoretical insights.

Weaknesses

No concrete weakness.

Questions

With complete proofs and empirical evaluations, I do not have too many concrete questions except for some discussions. (1) Both group DRO and tail risk minimization handle the distribution shift in robust optimization; what are the advantages or disadvantages between them in meta-learning? (2) What difficulties will we encounter if we adopt the developed strategy in the optimization of large models?

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

None

Summary

This paper proposes an enhancement to the previous work termed DR-MAML by reformulating it as a Stackelberg game. Theoretical investigations regarding its solution concept, convergence rate, and generalization bound are provided. Numerical experiments demonstrate the improved robustness of the proposed method.

Strengths

1. Both theoretical guarantees and numerical validations are provided to justify the sought robustness. 2. Reformulating DR-MAML's objective as a Stackelberg game enables analyzing the problem from the viewpoint of game theory.

Weaknesses

1. The contribution of this work is incremental and exclusive to DR-MAML, which limits its scope and broader applicability. 2. The writing is hard to parse, and several notions are pretty vague. For instance, in lines 95-96, $F_{\ell}^{\alpha}$, $\Omega_{\alpha,\tau}$, and $p_{\alpha}$ are defined through illustration in a figure instead of mathematical expressions. 3. Experiments are limited and lack comparison to SOTA methods. It is recommended comparing DR-MAML+ with popular meta-learning methods (such as MetaCurvature and MetaOptNet) on open-source benchmark datasets including tieredImageNet, CUB-200-2011, and MetaDatasets. In addition, the improvement in Table 1 and Table 2 are rather marginal (no more than 0.6%). 4. There is no comparison of time and space complexity, and the scalability to large NNs like ResNet is unknown. 5. The last statement of Theorem 4.1 is imprecise. According to Eq. (22), this convergence rate solely holds true when $t$ approaches infinity. 6. The paper contains numerous typos. For instance, in line 4, the citation is not compiled. In line 95, "the resulling" should be corrected to "resulting." Additionally, in lines 96, 130, 240, 241, 244, 246, 254, 265, 283, 295, 303, 319, 328, 332, and 335-337, the cross-references are in the equation style instead of the correct format.

Questions

See above.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

No explicit discussion on limitations are provided.

Summary

The paper provides theoretical investigations for better understanding of an existing method in literature that focuses on minimizing expected tail risk. Equivalence of the algorithm is shown to a Stackerlberg game, which allows to study its convergence rate and asymptotic bounds on performance and generalization are provided. Finally, a practical heuristic is suggested to obtain performance improvements over the base method.

Strengths

- The presentation and structure of the paper is good. - The theoretical results, while limited in mathematical novelty, seem comprehensive and are clear to follow. - Sufficient experiments and ablations are provided.

Weaknesses

Clarity: - It is not clear how the expression for the expected tail risk minimization is obtained in (3); the reviewer had to refer to the corresponding reference to understand it and thus, including it would help make the paper self-contained. - Figure 1 is not clear; the font size needs to be increased.

Questions

- What is the overhead induced by searching for the appropriate value of hyperparameter $\alpha$, compared to other baselines?

Rating

6

Confidence

2

Soundness

3

Presentation

2

Contribution

3

Limitations

N/A

Thanks for the rebuttal and update the review

After reading the rebuttal and other reviewers' comments, I add further reviews. (1) Overall, this is a little niche but comprehensive theoretical paper about the tail risk minimization for meta learning. (2) The extra effort made by the authors is impressive in the era of large models. These new results on tail risk minimization for MaPLe are inspiring and well answer my lasting question about the role of CLIP like models in few-shot prediction. I am happy to see the benefits of tail risk minimization in large-model augmented meta-learning and looking forward to the open-sourced code. For the above, I thank the authors for the rebuttal and tend to raise my score as the bonus.