Revisiting Logistic-softmax Likelihood in Bayesian Meta-Learning for Few-Shot Classification

Summary

This paper tackles the problem that classification tasks in Bayesian meta-learning meet mathematical problems with the softmax function. The previously proposed logistic-softmax function as an alternative can be optimized, but tends to exhibit inherent lack of confidence in prediction. To solve this problem, the authors propose to add a temperature scaling parameter in the function. This simple change not only solves the confidence problem when setting the temperature to less than 1, but also enjoys both theoretical and empirical advantages, as verified in the paper. For efficient and intractable optimization, the authors additionally use the data augmentation technique to derive a fully analytical mean-field inference method for the Bayesian meta-learning model.

Strengths

+ While being simple, the revised logistic-softmax function indeed enjoys better theoretical and empirical advantages, and these are all verified in the paper with proofs or experiments. + The proposed logistic-softmax function with temperature has the potential to be applied to broader research areas such as multi-class classification. + The writing is clear, making the paper easy to follow.

Weaknesses

- When we just focus on few-shot classification, I have to say that all bayesian meta-learning algorithms are not comparable to even very simple algorithms like ProtoNet or CE Baseline (I mean, if well-tuned, not the performance in Table 1) with the same architecture Conv-4. Also, as standard algorithms now use ResNet-12 as the backbone, all bayesian meta-learning algorithms still use Convnet, making the performance not comparable to those in most recent reported papers. This leads to doubt that do these bayesian meta-learning algorithms still perform well if we use ResNet-12 as the backbone? If the reason not to use ResNet-12 is the computation burden, then the biggest problem may be how to make bayesian inference more tractable. Also, another problem of interest is to figure out why bayesian meta-learning with complicated training steps cannot perform well on few-shot learning, even if equipped with nice theoretical properties. Since this paper does not answer these questions, I think the value of this paper is restricted.

Questions

- In theorem 3.3, $a$ is said to be the mean function with no restriction, so the mean of different sample points can be different. However, in line 444 in the appendix, all different samples have the same mean, can the author explain this? - How does your method compare with non-Gaussian processes like neural process [1]? It seems that neural process is more flexible and efficient than Gaussian process. [1] Garnelo et al. Neural processes.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have discussed the limitations which I found reasonable. I suggest the authors think more about the role of Gaussian process in bayesian meta-learning, as well as the role of bayesian meta-learning in few-shot classification.

Summary

The paper revisits the design of logistic-softmax function in the context of classification in Bayesian machine learning. In particular, the paper shows that a logistic-softmax function with a temporature could be more expressive than the conventional softmax function. However, due to the intrinsic nature of the logistic-softmax function, the inference might be non-conjugate, resulting in an intractable solution. To mitigate such an issue, the paper adopts the data augmentation technique proposed in the original logistic-softmax paper and propose a mean-field variational inference as an approximation to make the inference more efficient. The paper demonstrates the capability of the logistic-softmax with temporature in the context of few-shot meta-learning and shows that the approaches that integrate such method could achieve comparable performance in the two classificatio benchmarks: CUB-200-2011 and mini-ImageNet.

Strengths

- The paper has carried out a thorough investigation of the properties of the logisitic-softmax function with a temporature parameter. In particular, the paper shows that the logistic-softmax function with a temporature can easily model a one-hot vector or a uniform one (Theorem 3.1) or it can converge to a softmax function under certain conditions (Theorem 3.2). In addition, the logistic-softmax function of interest could model a richer family of distribution functions (Theorem 3.3). - The explanation of the paper is clear with easy-to-follow intuitition. The formulations are also clear to increase its clarity.

Weaknesses

The weakness of the paper might be at the applications of the logistic-softmax function. Currently, the paper targets to meta-learning, but to me, the paper is about the logistic-softmax function and meta-learning is just a mere application to demonstrate. Since the authors specify explicitly meta-learning, I cannot ask for other exploration. However, the paper might be strengthen more if it could include other applications using such a function for classification with GP.

Questions

Could the authors discuss further some potential applications using the logistic-softmax function of interest (beside meta-learning mentioned in the paper)?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- This is an increamental improvement from the original paper of the logistic-softmax function. - As mentioned in section 7, the current analysis is carried out in the context of Bayesian meta-learning. It might worth to investigate in other settings.

Summary

This paper proposes to modify the logistic Softmax likelihood by including a temperature coefficient in GP-based meta learning for few-shot classification. This is motivated by the observation that prediction made by logistic Softmax likelihood do not have confidence. Theoretically, they demonstrate that using the temperature, one could control for the confidence in logistic softmax. Furthermore, it is proved that softmax likelihood is a special case of logistic softmax. They also propose to use mean-field variational inference to approximate the posterior which is computationally more efficient than the typically adapted Gibbs sampler. It is demonstrated that their method achieves superior performance in uncertainty quantification. However, the performance improvements measured by accuracy are not as impressive as the uncertainty quantification.

Strengths

The problem is well motivated, and all the required theoretical derivations have been included. There are new theoretical findings presented.

Weaknesses

The idea of including temperature is not novel but its application to a new domain is novel. Including one more recent method for few-shot classification, referenced in my questions, can improve the benchmark.

Questions

Q1. I understand that the typical benchmark for few-shot classification uses 5 classes. Have you seen improvements with your method when there are more than 5 classes? Like 10 or 20? (Answering this question won’t impact my decision negatively.) Q2. Can you also include https://arxiv.org/pdf/2101.06395.pdf method in your benchmark? Q3. I understand that the temperature has been tuned for your experiments, but it would be nice to see the variation in the accuracy and uncertainty estimation as the temperature changes. Can you demonstrate this relationship for a subset of your experiments?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

NA

Summary

In this work, the logistic-softmax likelihood is redesigned, allowing control of the a priori confidence level through a temperature parameter. The modified logistic-softmax is shown to encompass softmax as a special case and induces a larger family of data distributions. By integrating this modified likelihood into a deep kernel-based Gaussian process meta-learning framework with data augmentation, well-calibrated uncertainty estimates are achieved in experiments, and competitive results are obtained on standard benchmark datasets. Post rebuttal: I have read the authors' rebuttal and I appreciate the authors' effort in addressing my concerns.

Strengths

+The logistic-softmax function with temperature is a nice idea and has potential to be used in multiple domains. +The theoretical analysis of the logistic-softmax likelihood is solid. +Some promising results are presented. +The paper is well written and easy to follow.

Weaknesses

-The proposed modified logistic-softmax function with temperature can be fundamental for different machine learning problems, it is not clear why it is specifically used for Bayesian meta-learning for few-shot classification. It is suggested to be tested on simpler tasks first before applying it to tasks like Bayesian meta-learning. -The proposed method performs marginally worse than state-of-the-art in maximum calibration error on a few benchmark datasets. It is not clear why how it happens. Theoretically, an adaptive temperature is expected to give better performance for all the tasks.

Questions

Q1: Have you considered other forms of the temperature in the modified logistic-softmax likelihood, for example having it as the power of the functions? Q2: Can you give a bit more detail of how the modified logistic-softmax function in multi-label classification?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations have been discussed in the paper: the performance of the proposed modified logistic-softmax is only evaluated in Bayesian meta-learning.

Thank you for your quick response

Thank you for your quick response. With all your concerns now resolved, would you be willing to consider increasing the score?

Discussion

Thank you, the authors, for addressing my concerns.

Response acknowledged

I thank the authors for providing the response