Summary
The paper conducts a comprehensive examination of the effectiveness of pre-trained (PT) models in comparison to Model Agnostic Meta-Learning (MAML) within the realm of few-shot learning. Despite prevailing beliefs suggesting the dominance of PT models, this study provides an unbiased comparison by employing identical architecture and ensuring complete model training. The research uses a robust statistical methodology, specifically the effect size (Cohen’s d), to discern the practical differences between PT and MAML models. A "diversity coefficient" metric is used to define the formal diversity of datasets. Key findings highlight that for datasets with low formal diversity, PT models slightly outperform MAML. Conversely, in scenarios with high formal diversity, MAML tends to be more effective. However, the magnitude of these differences is marginal, with an effect size less than 0.2. When evaluating the overall performance across various dataset diversities, neither method exhibits a clear advantage. The authors conclude that the inherent diversity of datasets is an important factor in determining the efficacy of PT versus MAML models.
Weaknesses
- The paper is solid from the empirical point of view, presenting a large variety of results. However, it is somehow limited in terms of novelty as it does not introduce any new techniques or analyses.
- A significant limitation of the paper lies in its reliance on two methodologies, MAML and PT with fine-tuning restricted to the head, which may not represent the current best practices in the field. Firstly, while MAML is undoubtedly foundational in meta-learning, its relevance has waned over time. Contemporary advancements have introduced more efficient derivatives, such as MAML++ (Antoniou et al., 2018). Incorporating comparisons with these modern variants could have enriched the paper's insights. Secondly, the paper's approach to fine-tuning is notably narrow, focusing only on the head's parameter adjustments. Contrarily, cutting-edge methods today, like BiT (Kolesnikov et al., 20202), fine-tune the entirety of both body and head parameters, while others, like FiT (Shysheya et al., 2022), selectively adjust a subset of body parameters. A juxtaposition against these state-of-the-art techniques would have been insightful. These oversights are critical as the paper's primary conclusions might shift when evaluated against more contemporary, optimized methods.
- The presentation of data exclusively in tabular format, though beneficial for transparency, hinders a quick understanding of the trends. I recommend the authors to enhance data representation by incorporating visual aids, such as scatter plots. This would facilitate a more intuitive grasp of the data patterns. The tables could be conveniently relocated to the appendix to maintain thoroughness without overwhelming the main content.
- There are some formatting issues, e.g. (i) Table 6 should be within the body of the paper, (ii) in the text "Meta-Data set" should be replaced with "Meta-Dataset"
References
----------
Antoniou, A., Edwards, H., & Storkey, A. (2018). How to train your MAML. arXiv preprint arXiv:1810.09502.
Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., & Houlsby, N. (2020). Big transfer (bit): General visual representation learning. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16 (pp. 491-507). Springer International Publishing.
Shysheya, A., Bronskill, J., Patacchiola, M., Nowozin, S., & Turner, R. E. (2022). Fit: Parameter efficient few-shot transfer learning for personalized and federated image classification. arXiv preprint arXiv:2206.08671.
Questions
Do the authors believe that a comparison between various methods, such as MAML++, BiT, or FiT, is feasible? Additionally, would such comparisons yield consistent conclusions with the current findings?
Please refer to the "Weaknesses" session for other potential points of discussion.