Meta-Learning Neural Procedural Biases

Thank you for taking the time to review our paper, and for highlighting the originality, quality, clarity, and significance. In what follows we address your questions: **RE: Method Novelty** Prior works has explored extending MAML to meta-learning additional components [1-9]. Many of these methods are meta-learning components that have previously been meta-learned in the literature — however, this does not imply that they are not novel, as the strategies for integrating the components and how they are meta-learned are vital for the downstream performance (e.g., WarpGrad [10] extending T-Net’s [11]), often yielding very different performance. One of the key novel contributions of this work (Section 4) was to show that by meta-learning three key components, the parameter initialization, optimizer, and loss function, you can implicitly meta-learn other components such as the scalar and parameter-wise learning rate, batch size, weight regularizer and more. Due to this implicit behavior, many existing prior meta-learning algorithms become special case of our proposed method (Appendix B). **RE: Computational and Memory Complexity** Compared to the baseline, MAML, our proposed method requires less runtime — on MiniImageNet 5way-5shot using the 4-CONV model the runtime is 11.4 hours compared to MAML’s 15.5 hours (on single A6000 GPU). This reduced runtime is achieved by utilizing a pre-trained backbone to initialize the encoder weights (Appendix A3), thus requiring only half the meta-gradient steps. In addition, MAML’s computational graph can be reused to avoid recomputing trajectory information, thus there is minimal storage of computational overhead when going from MAML to our proposed method. In the final manuscript we will add an additional section in the appendix to further discuss the proposed methods runtime in contrast to the baseline. **RE: State-Of-The-Art Performance** The performance of our proposed algorithm is to the best of our knowledge SOTA, or near SOTA on optimization-based few-shot learning methods (under the constraints of the same data resources and models). There are methods such as [12] which have superior performance, however, this is due to them using additional data resources, as well as higher capacity network architectures. We would like to emphasize to the reviewer that the central goal of this work was not primarily to achieve SOTA performance, it was to show that there are three key components that should be meta-learned in order to learn the procedural biases of a deep neural network. **RE: Theoretical Results** Thank you for the suggestion. What additional theoretical results (in addition to those discussed above) does the review recommend to further enhance the contribution of the paper? 
 — We hope we have managed to clarify the points of confusion and address your concerns. We kindly, ask you to consider updating your score, if you feel we have answered your questions adequately. If there are any further questions, please do not hesitate to reach out during the short rebuttal period. Best regards, The Authors — [1] Behl, HS, et al. "Alpha MAML: Adaptive Model-Agnostic Meta-Learning." ICML2019. [2] Baik, S, et al. "Meta-Learning with Adaptive Hyperparameters." NeurIPS2020. [3] Li, Z., et al. Meta-SGD: Learning to Learn Quickly for Few-Shot Learning. arXiv2017. [4] Lee, Y., et al. Gradient-Based Meta-Learning with Learned Layerwise Metric and Subspace. ICML2018. [5] Antoniou, A, et al. How to Train your MAML. ICLR2019. [6] Antoniou, A. et al. Learning to Learn by Self-Critique. NeurIPS2019. [7] Park, E., et al. Meta-Curvature. NeurIPS2019. [8] Simon, C., et al. On Modulating the Gradient for Meta-Learning. ECCV2020. [9] Baik, S., et al. Meta-Learning with Task-Adaptive Loss Function for Few-Shot Learning. ICCV2021. [10] Flennerhag, S., et al. Meta-learning with Warped Gradient Descent. ICLR2020. [11] Lee, Y., et al. Gradient-Based Meta-Learning with Learned Layerwise Metric and Subspace. ICML2018. [12] Hu, S, et al. "Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning make a Difference." CVPR2022.

Thank you for reviewing our paper, and for the kind words regarding the presentation and clarity. Below we address your questions: **RE: Comparison to Non-Bi-level Optimization Methods** Regarding methods such as [1, 2] these methods utilize significantly larger more powerful models (as you noted), as well as they use additional data resources which we (and other optimization-based methods) do not; hence, it is not fair to directly compare their performance. In regard to your question, there is an inherent tradeoff between these two competing paradigms. Bilevel optimization methods are more computationally expensive at meta-training time; however, since the models are more compact, the models require less memory and compute at inference time when deployed, relative to pre-training methods which rely on scaling up the model and use significantly more data (which is not always available). Therefore, depending on the requirements of the problem and the domain, one technique may be preferable to another; however, pre-training methods do not Pareto dominate optimization-based methods. Thanks for bringing this up, we will include a discussion on this topic in the updated version of the manuscript. **RE: Performance at Initialization** To further contextualize the performance before and after meta-training, we ran some experiments on mini-ImageNet using no meta-training and the results are shown below. The results clearly show that meta-training is vital for obtaining competitive few-shot learning performance. We will include these additional results in the appendix of the manuscript. | Method | 5way-1shot (4-CONV) | 5way-5shot (4-CONV) | 5way-1shot (ResNet-12) | 5way-5shot (Resnet-12) | | :-------------------- | :------: | ----: | :------: | ----: | | NPBML (no meta-training) | 49.44% | 63.44% | 54.89% | 70.75% | | NPBML (with meta-training) | **57.49%** | **75.01%** | **61.59%** | **78.18%**| **RE: Comparison Methods Numbers** The results reported in Table 1 and 2 are taken from the respective papers. The only exception to this was MAML, where the tiered-ImageNet, CIFAR-FS, and FC-100 results were inherited directly from [3], as the results did not exist in the original MAML paper but were important to include as a point of reference. — We hope we have managed to clarify the points of confusion and address your concerns. We kindly, ask you to consider updating your score, if you feel we have answered your questions adequately. If there are any further questions, please do not hesitate to reach out during the short rebuttal period. Best regards, The Authors — [1] Hu, S, et al. "Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning make a Difference." CVPR2022. [2] Fifty, C, et al. "Context-Aware Meta-Learning." ICLR2024. [3] Baik, S, et al. "Meta-Learning with Task-Adaptive Loss Function for Few-Shot Learning." ICCV2021.

Thank you for taking the time to review our paper, and for all the feedback and suggestions you have given. We address your questions below: **RE: Meta-Overfitting and Regularization** Thank you for raising the topic of meta-overfitting. As shown in Table 1 and 2, our empirical results show that our proposed method attains superior *out-of-sample testing performance* on four well established few-shot learning datasets compared to 11 other few shot learning methods. Analogous to conventional supervised learning settings, a more powerful and expressive meta-learning technique is exposed to a higher risk of meta-overfitting, however, this can be mitigated through regularization techniques. In meta-learning, meta-regularization techniques such as proximal regularization [12] could be leveraged to avoid over adaptation to new tasks in the inner optimization. In the final manuscript we will include some additional discussion on meta-regularization. **RE: Inner Loop Hyperparameters** All hyperparameters where possible were taken from the established literature. Regarding the inner loop gradient steps, this was taken from [1], in order to ensure a fair comparison between methods. As our works primary contribution was not about meta-optimization, simple unrolled differentiation was used identical to MAML. Unrolled differentiation scales linearly in memory with respect to the number of inner gradient steps; however, this can be obviated by using techniques such as implicit differentiation [12] or trajectory agnostic meta-optimization techniques such as [7, 13, 14]. **RE: Larger Models** The models selected for our experiments were chosen following the experimental protocol from established literature [1-11]. As we do not have access to larger compute resources we cannot evaluate on larger convolutional or vision transformers models. However, the results in Table 1 and 2, indicate that our methods performance scales in a comparable manner to existing methods on both the few-shot learning CIFAR-100 and ImageNet partitions. **RE: Additional Applications** Regarding the applications domains explored, our paper performs extensive experiments on a diverse range of few-shot image classification tasks — these being, mini-ImageNet, tiered-ImageNet, CIFAR-FS, and FC-100, which are the most established and recognized datasets in the area. While it would be valuable to explore further applications domains such as: segmentation, detection, super-resolution, translation, text summarization etc., due to space limitations as well as ICLR not being an application focused conference, we leave this for future work to explore. — We hope we have managed to clarify the points of confusion and address your concerns. We kindly, ask you to consider updating your score, if you feel we have answered your questions adequately. If there are any further questions, please do not hesitate to reach out during the short rebuttal period. Best regards, The Authors — [1] Finn, C., et al. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. ICML2017. [2] Li, Z., et al. Meta-SGD: Learning to Learn Quickly for Few-Shot Learning. arXiv2017. [3] Lee, Y., et al. Gradient-Based Meta-Learning with Learned Layerwise Metric and Subspace. ICML2018. [4] Antoniou, A, et al. How to Train your MAML. ICLR2019. [5] Antoniou, A. et al. Learning to Learn by Self-Critique. NeurIPS2019. [6] Park, E., et al. Meta-Curvature. NeurIPS2019. [7] Flennerhag, S., et al. Meta-learning with Warped Gradient Descent. ICLR2020. [8] Simon, C., et al. On Modulating the Gradient for Meta-Learning. ECCV2020. [9] Baik, S., et al. Meta-Learning with Task-Adaptive Loss Function for Few-Shot Learning. ICCV2021. [10] Baik, S., et al. Learning to Learn Task-Adaptive Hyperparameters for Few-Shot Learning. TPAMI2023. [11] Kang, S., et al. Meta-Learning with a Geometry-Adaptive Preconditioner. CVPR2023. [12] Rajeswaran, A., et al. Meta-Learning with Implicit Gradients. NeurIPS2019. [13] Flennerhag, S., et al. Transferring Knowledge Across Learning Processes. ICLR2019. [14] Flennerhag, S., et al. Bootstrapped meta-learning. ICLR2022.

Thank you for taking the time to review our paper and for highlighting our strong few-shot learning performance and soundness. Below we address your questions: **RE: Meta-Overfitting** Meta-overfitting and meta-regularization are important topics of research, similar to conventional learning paradigms. However, we would like to emphasize to the reviewer that it is not the central topic under study in this paper. This paper aims to propose a method for meta-learning task-specific procedural biases which are specifically designed to attain strong learning performance. The experimental results in Table 1 and 2, confirm that our proposed method can attain superior *out-of-sample testing performance* on four well established few-shot learning datasets compared to 11 other few shot learning methods. If our method is found to overfit on other datasets, popular meta-regularization techniques from the literature could be leveraged such as proximal regularization [1]. **RE: Method Novelty** Prior works has explored extending MAML to meta-learning additional components [2-11]. Many of these methods are meta-learning components that have previously been meta-learned in the literature — however, this does not imply that they are not novel, as the strategies for integrating the components and how they are meta-learned are vital for the downstream performance (e.g. WarpGrad [11] extending T-Net’s [5]), often yielding very different performance. One of the key novel contributions of this work (Section 4) was to show that by meta-learning three key components, the parameter initialization, optimizer, and loss function, you can implicitly meta-learn other components such as the scalar and parameter-wise learning rate, batch size, weight regularizer and more. Due to this implicit behavior, many existing prior meta-learning algorithms become special case of our proposed method (Appendix B). Regarding MT-Nets, they do not implicitly learn a loss function as the last layer of the model does not use transformation (T) layers, only the last layer of the encoder does (see https://github.com/yoonholee/MT-net). Furthermore, meta-learning a loss function is not equivalent to meta-learning a T-Layer in the classifier, as this would result in the classification head being frozen in the inner loop resulting in no adaptation of the classifier at meta-testing time. Finally, as shown in Table 1 and the ablation in Table 3, our method shows substantial improvements over MT-Nets, e.g. 51.7% accuracy vs 57.49% on mini-ImageNet 5-way 1-shot. **RE: Implementation and Computational Complexity** Concerning the implementation difficulty of our proposed method, NPBML, is not significantly more difficult to implement then MAML, and in many cases it would just be a drop in replacement of a few lines of code (see our attached code). As for the computational complexity, much of the computational burden is obviated by reusing the optimization trajectory store by MAML. Furthermore, as we utilize pre-trained backbones to initialize the encoder weights (Appendix A3), following recent SOTA methods [12-16], our method only requires half the number of meta-training gradient steps. On mini-ImageNet 5way-5shot using the 4CONV model the runtime is 11.4 hours compared to MAML’s 15.5 hours. In the final manuscript we will add an additional section in the appendix to further discuss the runtime of our algorithm. **RE: Benchmark Datasets** Thank you for raising this topic. While we are agree that the CIFAR-100 and ImageNet partitions are not the perfect benchmark, they are by a significant margin, the most popular and most well-established dataset used in this area. Therefore, to aid in cross comparison, we performed experiments using these dataset. **RE: ProtoNet and SimpleShot** Regarding the benchmarks, we chose the 11 most closely related optimization-based meta-learning methods to compare against in Table 1. In the updated manuscript we will include further experiments comparing to ProtoNet [17] and SimpleShot [18]. As we use a standardized experimental setting we can directly compare the results as follows: | Method | 5way-1shot (4-CONV) | 5way-5shot (4-CONV) | | :-------------------- | :------: | ----: | | ProtoNet [17] | 49.42% | 68.20% | | SimpleShot [18] | 33.17% | 63.25% | | NPBML (Ours) | **57.49%** | **75.01%** | As shown, our method performs significantly better than both ProtoNet and SimpleShot. Note, although the “inference-time adaptation” may be faster then optimization-based approaches, these methods often rely on significantly larger network architectures in order to get better performance; consequently, this greatly increase their inference costs when making predictions when deployed.

— Apologies for the lengthy response. We hope we have managed to clarify the points of confusion and address your concerns. We kindly, ask you to consider updating your score, if you feel we have answered your questions adequately. If there are any further questions, please do not hesitate to reach out during the short rebuttal period. Best regards, The Authors — [1] Rajeswaran, A, et al. "Meta-Learning with Implicit Gradients." NeurIPS2019. [2] Behl, HS, et al. "Alpha MAML: Adaptive Model-Agnostic Meta-Learning." ICML2019. [3] Baik, S, et al. "Meta-Learning with Adaptive Hyperparameters." NeurIPS2020. [4] Li, Z., et al. Meta-SGD: Learning to Learn Quickly for Few-Shot Learning. arXiv2017. [5] Lee, Y., et al. Gradient-Based Meta-Learning with Learned Layerwise Metric and Subspace. ICML2018. [6] Antoniou, A, et al. How to Train your MAML. ICLR2019. [7] Antoniou, A. et al. Learning to Learn by Self-Critique. NeurIPS2019. [8] Park, E., et al. Meta-Curvature. NeurIPS2019. [9] Simon, C., et al. On Modulating the Gradient for Meta-Learning. ECCV2020 [10] Baik, S., et al. Meta-Learning with Task-Adaptive Loss Function for Few-Shot Learning. ICCV2021 [11] Flennerhag, S., et al. Meta-learning with Warped Gradient Descent. ICLR2020. [12] Rusu, A., et al. Meta-Learning with Latent Embedding Optimization. ICLR2019. [13] Qiao, Siyuan, et al. "Few-Shot Image Recognition by Predicting Parameters from Activations”. CVPR2018. [14] Requeima, J., et al. "Fast and Flexible Multi-Task Classification using Conditional Neural Adaptive Processes”. NeurIPS2019. [15] Ye, H., et al. “Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions”. CVPR2020. [16] Ye, H. J., et al. “How to Train Your MAML to Excel in Few-Shot Classification”. ICLR2022. [17] Snell, J., et al. Prototypical Networks for Few-Shot Learning. NeurIPS2017. [18] Wang, Y., et al. SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning. arXiv2019.