Beyond Unimodal: Generalising Neural Processes for Multimodal Uncertainty Estimation

Summary

The paper tackles the problem of uncertainty estimation specifically for multi-modal data, i.e., inputs consisting of different sources. Specifically, it improves the popular Neural Process (NP) in three aspects: dynamic context update, multi-modal Bayesian aggregation, and a novel attention mechanism based on the RBF kernel. The paper demonstrates superior test accuracy, uncertainty estimation (calibration and OOD detection), and robustness on multiple benchmarks compared to prior works.

Strengths

* The paper studies the less well-studied area of uncertainty estimation for multi-modal data. This has practical implications as many real-world applications can be multi-modal. * The paper presents multiple metrics including accuracy, calibration, OOD detection, and robustness, which provide a holistic evaluation of the proposed method. * The idea of dynamic context update is most interesting to me. Specifically, the method replaces uninformative context samples with informative ones where "uninformative-ness" is measured by attention weights and "informative-ness" is measured by difficulty of classification.

Weaknesses

* While the paper presents three innovations and claims that they are all tailored to multi-modal data, only the Bayesian aggregation is inherently related to multi-modal inputs. The dynamic context update and the RBF context mechanism do not utilize the multi-modal characteristic. This makes the contributions and claims less coherent. * It seems the Bayesian aggregation mechanism is not ablated. It's not clear how much improvement it brings to the overall performance as it is the main mechanism responsible for tackling multi-modal inputs. Does the improved robustness come from the aggregation algorithm or the other two components?

Questions

* Can the authors comment on the importance of the Bayesian aggregation component? * It would be good to consolidate a single table (for one dataset) to directly show the effectiveness of each component in terms of accuracy, uncertainty, and robustness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The work does not have an apparent negative societal impact.

Summary

This work proposes a confidence calibration algorithm for multimodal classification problems. The algorithm includes three key components: 1) dynamic context memory, 2) multimodal Bayesian aggregation, and 3) adaptive radial basis function. The algorithm is evaluated on multiple benchmarks using classification accuracy, calibration, and OOD performance. On multiple benchmarks, the proposed algorithm is able to obtain SOTA results.

Strengths

1, The targeted problem and challenges are explicitly stated. For each challenge, a technically reasonable solution is proposed. 2, The proposed algorithm is able to obtain strong empirical results, obtaining SOTA results on multiple benchmarks and different metrics. 3, Comprehensive ablation studies have been conducted to demonstrate each components. 4, As illustrated in Figure 2, the proposed adaptive RBF algorithm outperformed other attention mechanisms.

Weaknesses

1, Some notation definitions are confusing. Please refer to section "Questions" for details. 2, Because of the above notation section 3.2 is a bit hard to follow. 3, Ablation study is conducted for individual components. It would be interesting to design ablation studies and see how each component affects the end-to-end performance. 4, The component directly related to multimodality learning, i.e., multimodal Bayesian aggregation, is based on previously work [1]. It would be good if the authors can explain how the proposed MBA component is different from previous work, either in the revision or rebuttal. [1] M. Volpp, F. Flürenbrock, L. Grossberger, C. Daniel, and G. Neumann. Bayesian context aggregation for neural processes. In International Conference on Learning Representations, 2021.

Questions

1, **Line 71**, from my understanding, the context set C and target set T contains both training and test datasets. But $N_C +N_T=N_{\text{train}}$. Why is the summation of $N_C$ and $N_T$ only equal to the number of training points? 2, **Line 148 and 150**, what are $r^m$ and $s^m$? How are they connected/different from $r^m_{\*}$ and $s^m_{\*}$? 3, **Line 160**, how are encoders parameterized by $\theta$ and $\omega$ connected/different from those by $\phi$ and $\psi$?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors explicitly discussed the limitations.

Summary

This paper proposes a multimodal neural processes (neural network generaliation of Gaussian processes) model. The overall approach has several novel elements: * A way to maintain a dynamic context set throughout training (e.g a support set for few-shot learning, these context sets are needed for neural processes) * A Bayesian aggregation scheme for combining multiple modalities * An adaptive RBF attention mechanism as an alternative to (vanilla) dot product attention which the authors argue is overconfident on OOD samples. Experiments on a suite of datasets show that the proposed method is faster, more accurate and better calibrated, and better at detecting OOD samples than prior work.

Strengths

This reads like a polished piece of work with clear writing, extensive ablations (in the supplementary) and novel technical contributions. Experimental results seem fairly convincing (that the proposed method is indeed better than prior similar works). My caveat is that my familiarity with this part of the literature on Neural Processes is passing at best.

Weaknesses

I have no major complaints about this paper, but some things to point out are: It is pretty unclear what are the actual multiple modalities in the datasets that are used for experiments (which makes these experiments not particularly compelling perhaps unless you are already familiar with this line of work). I had to trace through a series of cited works to figure out that (likely) these different modalities are different features extracted e.g using different networks. Section 3.2 could be written more clearly — I couldn’t figure out why there were multiple encoders until encountering Lemma 3.1 — it’d be much better to explain this up front. A nit: it’s also not clear when reading section 3.1 how often the dynamic context memory is meant to be updated until I read the pseudocode much later (I’d recommend saying that it’s a per-minibatch update somewhere).

Questions

See above.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Summary

This paper proposes a new method for multimodal uncertainty estimation by extending neural processes. The authors summarize three challenges to do that and give solutions correspondingly. Experimental results show that the proposed method is more robust and outperforms existing baselines. --- Thanks for the clarification! My main concerns are addressed.

Strengths

The proposed method (MNPs) can achieve good performance empirically.

Weaknesses

1. The motivation is not clear. Indeed, there is little discussion about neural processes with multimodal data, and extending unimodal neural processes to multimodal scenarios could be challenging. However, existing works discussed the multi-view data [16,21]. The reason why extending the neural processes for multimodal data (instead of extending/improving [16,21]) remains unclear. 2. Lack of understanding. I would like to suggest that the authors add some ablation studies. From current experiments, we can only see that proposed MNPs outperform baselines. However, why it can do that is not clear, e.g., why the unimodal method (DE) can achieve the best test accuracy.

Questions

(Line 185): Maybe I missed something, but why using RBF can address the overconfident issue?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Yes

Summary

This paper extends one of the promising uncertainty estimation method: Neural Process from unimodal to multimodal. This is motivated by the fact that: current techniques are predominantly designed for unimodal data, and directly applying them to multimodal information is ineffective. However, this extension poses several challenges: 1) how fuse information from different modalities effectively and efficiently; 2) the context memory of the original design grows in proportion to M - the number of modality, which is memory-consuming, and sets the question of how to maintain it small yet informative; 3) how to find an appropriate length-scale of the Radial Basis Function (RBF) that ensures a tight boundary between in-distribution and out-of-distribution data. To tackle these challenges, the paper introduces three solutions: 1) Dynamic Context Memory to choose the most informative samples in the context memory, 2) Multimodal Bayesian Aggregation, and 3) Adaptive RBF Attention. Empirical experiments show that the method can outperform other unimodal and multimodal baselines in terms of maintaining good accuracy and calibration at the presence of noise and in terms of out-of-distribution detection.

Strengths

1. originality: the first to extend Neural Process from unimodal to multimodal and effectively combine existing techniques to solve some of the challenges incurred. 2. clarity: in general, the paper articulates its ideas clearly, even though some parts remain difficult to understand or unclear in terms of details (see the weakness part) 3. significance: the proposed method shows promising results in terms of outperform most unimodal and multimodal baselines.

Weaknesses

**Clarity**: the clarity of the technical introduction (Sec. 3) and the experiment (Sec. 4) parts can be largely improved. 1. The method introduction part could be **challenging to comprehend for readers unfamiliar with the Neural Process and its technical details**. It lacks adequate contextual knowledge, such as an explanation for why the context memory is required, why it is used to store training samples rather than using a context feature. I think the Sec. 2 should be improved to be more comprehensive and general in introducing the Neural Process, to avoid the readers to revisit several referenced papers repeatedly. 1. The **math details can be overwhelming and at times confusin**g. For instance, when reviewing Sec 3.3, it necessitates frequent referral back to Section 3.2. It would be helpful if the semantic meaning of symbols is reintroduced when they appear, preventing the need for readers to revisit previous sections for notation understanding. 1. **Derivation details**: The logic in some derivations is not easy to understand, e.g. Line 221 introduces the function $f(T_X^M)$ without any prior explanation. Also, there is no explanation on how equation (12) is derived, and why softmax is applied on a density function. These issues impede reading and understanding. The formula (4) and formula (6) are not consistent as their parameter notation is not the same on $\omega$ and $\psi$. 2. The MNPs **pseudocode** lacks a high-level introduction and has heavy reference links, failing to provide clarifying information. 3. **Captions**: Some table captions lack sufficient information. For instance, Table 4/5 does not clearly describe the dataset and settings used. 4. **Experimental Setup**: The presentation of the experimental settings, such as how datasets e.g. Caltech101 were modified into multi-view datasets, is not clear. **Soundness**: Despite the multimodal setting, the experiments use only multi-view image data. The lack of experiments with multimodal combinations such as image + text impedes the proof of the model's effectiveness in various settings.

Questions

see questions listed on the weakness part

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

The author have listed one limitation that the updating mechanism is not theoretically guaranteed to obtain the optimal context memory.

Response to Rebuttal

Thanks to the authors for adding the ablation study, this addressed my second question. But still, I think that the dynamic update and RBF context mechanism are not inherent progress towards multi-modality even though they do contribute to more accurate uncertainty estimation. Therefore, I keep my original score.