ProtoDiff: Learning to Learn Prototypical Networks by Task-Guided Diffusion

Summary

The paper introduces ProtoDiff, a novel addition to the family of Prototypical Network methods that uses a diffusion model to obtain better class prototypes at inference time. Prototypical Networks perform classification with metric-based comparisons between examples and “class prototypes”, variables computed based on support sets of class examples. In ProtoDiff, a “vanilla” prototype is first computed based on the class support set. This vanilla prototype is then used as conditional input of a diffusion model, computing over T steps a residual between the vanilla prototype and a better, “diffused” prototype for the class. This diffused prototype is then used to perform traditional prototypical network classification. To train the diffusion model at meta-training time, one has to produce “overfitted” prototypes for tasks, based on both the query set and support set for the task. The authors then demonstrate the effectiveness of their approach with a series of experiments for within-domain few-shot learning, cross-domain few-shot learning, and few-task few-shot learning.

Strengths

The combination of prototypical networks and diffusion models is interesting: applying insights from recent developments in diffusion models to the field of meta-learning is still under-explored in the field. The paper is clearly presented, the procedure itself is a novel combination of existing ideas, and is clearly motivated and easily argued for. The experiments seem to support the claim that the diffusion model specifically is what is needed to edge out any remaining performance missing from the original prototypical network approach, in the form of a residual.

Weaknesses

Overall, the paper doesn’t have any glaring flaw, if not for the fact that experimental gains seems marginal. In practice, the idea behind the paper seems to be about squeezing any additional percentage point of performance out of the existing technique of prototypical networks. This is especially apparent with the fact that the diffusion model is used to produce a residual, and is also itself conditioned on the “vanilla” prototype. As for related work, it would be interesting to explore the connection with *Nava et al. (2022). Meta-Learning via Classifier (-free) Diffusion Guidance.* (https://arxiv.org/abs/2210.08942). They also apply diffusion models in the context of meta-learning, using a similar metric classification technique for the downstream tasks, even though they approach the problem with the goal of conditionally diffusing a model’s parameters, instead of class prototypes.

Questions

I would ask the authors to further elaborate on the positioning of their work within the meta-learning literature, arguing for the choice of conditioning the diffusion model on the existing “vanilla” prototype, and comparing it with possible alternative choices, such as specifically learnt class embeddings.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors satisfactorily discuss the limitations of their work in the main paper text.

Summary

This paper proposes using the diffusion model to learn a prototypical network. During the meta-train stage, an overfitted prototype (by fine-tuning the classifier until it gets a probability of 1 for the task) is obtained first, then given a vanilla prototype (by simply averaging the features) as input, the diffusion model is to denoise a random prototype into the overfitted prototype. This is because the overfitted prototype is unknown during the meta-test stage, therefore the diffusion model is to obtain this overfitted prototype by using the vanilla prototype as conditional information.

Strengths

- the whole design of leveraging the diffusion model to predict the prototype is quite interesting and novel. That is, overfitted prototype indeed is better than the vanilla prototype (can be seen as an "optimal" prototype for a given task), but obtaining it is not possible during the meta-test stage because the query set is not available. Hence, using the diffusion model to capture the distribution of how a vanilla prototype can be turned into an overfitted prototype is quite making sense.

Weaknesses

- as the diffusion model is data-hungry, I believe when the number of available tasks is low and not diverse, this proposed model may not perform better than a conventional model. The author should also include this analysis to make the paper more complete. - the proposed model has increased training time due to overfitting the prototype and a high number of timesteps to predict the prototype, as mentioned by the author. The author should also include this analysis to let readers know how much increased time over conventional methods.

Questions

- I believe Eq.8 has a typo. - What is the ratio of the increased training time over the conventional model? - What is the case when the number of available tasks is low and not diverse? Will the conventional model perform better or not? - How can this model be applied in production compared to conventional model?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

- the author mentioned two limitations, one is the proposed model requires a substantial number of timesteps to sample the prototype during the meta-test stage. - secondly, obtaining the overfitted prototype requires increased training time.

Summary

This paper proposes a prototype-based meta-learning method called ProtoDiff. During meta-learning stage, ProtoDiff introduces a task-guided diffusion process, learning a generative process conditioned on vanilla prototype for more robust prototype representation. ProtoDiff is extensively validated on within-domain, cross-domain, and few-task few-shot classification.

Strengths

(1) The diffusion models have been very successful generative models, attracting great interests very recently. It is interesting to see the authors introduce the diffusion models into few-shot learning regime, for addressing the limitation of unrobust prototype estimation given a limited number of training examples. (2) The proposed ProtoDiff is extensively evaluated on a variety of few-shot learning problems, including within-domain, cross-domain, and few-task few-shot classification. The ablation study is carefully and thoroughly designed for validating the effectiveness.

Weaknesses

(1) The diffusion model for prototype learning is the key in the proposed ProtoDiff; however, some technical details on the diffusion model are missing. About the architecture, how many layers are used in the encoder and what about dimension of linear transform and the number of heads in the attention module and what about the hidden dimensionality in MLP? What the proposed ProtoDiff will perform if the design parameters vary? About the diffusion process, how do you select T? How does different T affect the performance and speed of the proposed ProtoDiff? (2) Some other concerns. In ProtoDiff, the ground truth prototypes, called overfitting prototypes, are obtained by finetuning the network for several steps. Besides that strategy, have you ever considered, during meta-training, using more support images (e.g. 8 or 16) by which one may get more accurate prototypes? As suggested in ProtoNet, it is beneficial to train with a higher way that will be used at meta-test. In table 3, the baseline of ProtoNet is SetFeat; therefore, I suggest the authors change “This paper’’ to “SetFeat+ProtoNet”. How do you implement ProtoNet in Tables 4 and 5? Is it based on SetFeat or ProtoNet? Please clarify. In equation (10), $\mathbf{z}$ is not defined; the subscript $t$ of $\sum$ is missing.

Questions

See "Weaknesses" section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Though the authors discuss the limitations of the proposed ProtoDiff, i.e., heavy computational cost, it is not clear how ProtoDiff is expensive compared to the baselines. I suggest the authors provide wall-clock time of meta-training/meta-testing of ProtoDiff against the ProtoNet.

Summary

This paper tries to improve Prototypical Network in few-shot learning. Specifically, the authors use a diffusion model to predict optimal prototypes from initial prototypes. In each iteration of the meta-training phase, the optimal prototypes are obtained by several inner loops on the current task, then a diffusion model takes the initial prototypes as input, and predicts the difference between the optimal and initial prototypes. The final loss is used to meta-learn the feature extractor and the diffusion model. The experiments show that the proposed method can improve ProtoNet to some degree.

Strengths

- Introducing diffusion models into few-shot learning and using it to translate features is interesting. - The use of residual prototype learning is clever.

Weaknesses

- In essence, the key idea is to learn a module to predict optimal prototypes from initial prototypes. Thus this module is a set-to-set function, and does not need to be stochastic, e.g., a MLP or transformer. From this point, there is no motivation of using generative models like diffusion models, considering that no probabilistic inference is used during training or testing. The motivation of this paper is thus very weak. - In fact, back in 2019, FEAT algorithm [1] already exploited such an idea, using a shallow transformer to transform initial prototypes into near-optimal prototypes without using any probabilistic models. While being much more efficient (no diffusion operations, no inner gradient loops, and much fewer parameters used) and being less tuned for hyperparameters, FEAT can achieve performance comparable to ProtoDiff. Similarly, CrossTransformer [2] also uses a shallow transformer to transform initial prototypes into near-optimal prototypes, but in addition, considers spatial information. Thus the novelty and the value of the paper are quite limited. - The writing of the method section needs to be significantly improved. Before looking at the algorithm 1-2 in the appendix, I struggle to understand the pipeline of the method, and some notations are confusing. For example, there are clear mistakes in the notations in lines 112-116. - In Table 1, the result of Classifier-Baseline is borrowed from Meta-Baseline. Instead, the authors need to show the results of their own pre-trained model (trained with CE loss on the whole training set) before the use of ProtoDiff to show how much improvement of their method brings. - In Table 2, only generative models are considered. As stated above, more deterministic feature transformation methods should be compared. [1] Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions. CVPR 2020. [2] CrossTransformers: spatially-aware few-shot transfer. NeurIPS 2020. ======Post-rebuttal======= The authors have addressed most of the concerns during the rebuttal. Thus I raise my score from 3->7.

Questions

See Weakness above.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors have stated some of the limitations.

Response to Authors Rebuttal

Thanks for the rebuttal. I totally agree that uncertainty is vital when dealing with limited samples. However, there is no "useful" uncertainty in this paper due to design choices. Let me detail it below. When uncertainty is taken into account, only when there are multiple samples sampled from the posterior probability which are used for inference, the uncertainty makes sense (as what is done in variational inference). However, in this paper, during inference, only one prototype is produced for each class, thus there is no probabilistic inference, i.e., the distribution built for the prototype makes no use at all and, the uncertainly is not taken into account; furthermore, it can be easily inferred that the optimal prototype becomes the best choice if only one prototype is produced. So in principle, in the authors' method, the optimal posterior prototype distribution should be a point estimate of the optimal one, so there is, in fact, no uncertainty in your method. The strong performance only comes from the auxiliary deep architecture (used for the diffusion process) for estimating the optimal prototypes, which does not need uncertainty at all. Besides, your baseline using transformer is far below that reported by FEAT [1], the method identical to adding a transformer layer for estimating the optimal prototype. I have re-implemented FEAT, and it can exactly match the reported performance, which is very similar to ProtoDiff. This indicates that generative modeling used in this paper is not needed, at least when used without probabilistic inference. I know that different from FEAT, Protodiff learns to fit the overfitted prototype explicitly. However, due to the mechanism of meta-learning, FEAT implicitly does such a process (meta-learn to map prototypes to optimal ones, the "supervision/guidance" in your rebuttal), and its performance matches Protodiff closely. I think it is interesting to take this mapping process out explicitly, but I still do not think there is anything new here. [1] Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions. CVPR 2020

Reply to Rebuttal by Authors

I thank the authors for their careful responses, which addressed most of my concerns. I hope the authors include, in the modified version, the discussion and analysis herein about technical details and ablation on diffusion models, more accurate estimation of prototypes with more support images, and wall-clock time of meta-training/test. Despite the fact that the proposed ProtoDiff has heavy computational cost, I think it is interesting and inspiring to introduce diffusion model for robust prototype estimation in the few-shot learning. I opt to accept this paper.

Reply to the Authors

We thank the Authors for their detailed response, which addresses the concerns I raise. I look forward to a broader comparison with the research landscape in the final version. Meanwhile, I am willing to raise my score to a 7.