Summary
This paper proposes prototype-based aleatoric uncertainty quantification (PAU) to represent aleatoric uncertainty (i.e., data uncertainty) for cross-modal retrieval tasks. The proposed method uses the CLIP backbone, where the initial similarity matrix M is computed by the output of the CLIP encoders. Then, "evidence" $e$ is computed by the similarity between each modality embedding output and the prototype vectors. Using $e$, the belief $b$ and the uncertainty $u$ is computed. As far as the reviewer understood, the computed $u$ represents how the current embedding is *different from* the prototype vectors, i.e., if the current embedding is specifically different (in terms of the cosine similarity) from the prototype vectors, then its uncertainty becomes large. Using the learned uncertainty $u$, the proposed method re-rank the retrieved items by cosine similarity of the output of the CLIP encoders. Two losses are introduces to make the prototype vectors better, the uct loss (minimize the distance between average embedding vectors and prototypes) and the diversity loss (orthogonal regularization for prototype vectors). The effect of the proposed method is evaluated on four datasets.
Weaknesses
My concerns can be summarized as:
- It is not clear that replacing fixed class information of SL with learnable prototype embeddings will ensure a proper uncertainty estimation.
- This paper misses the important previous work on the aleatoric uncertainty estimation method for cross-modal retrieval [A].
- The comparisons of MS-COCO 5K would not be fair (only the proposed method uses the pre-trained CLIP model), and many recent state-of-the-art methods are missing.
### Can the proposed method provide a proper uncertainty estimation?
The proposed method highly depends on SL [27], where SL uses fixed class information; hence, there is no learnable prototype. As this paper introduces new learnable prototypes, the proposed method requires two additional objective functions, Eq (5) and Eq (6). I am not sure how Eq (5) and Eq (6) will guarantee the quality of the proper uncertainty measure.
Eq (6) looks okay, but it enforces the orthogonality of the prototype embeddings, not the mutual exclusion among the embeddings. The latter one would be derived from independence, not orthogonality. Note that independence and orthogonality are highly related and somewhat exchangeable in many cases, I am not sure that smaller orthogonality regularization (i.e., Eq (6)) can ensure the weaker dependency between the prototype vectors. Moreover, I am not sure that Eq (6) is specifically better than widely-used orthogonality regularization, such as $\| I - Z^\top Z \|_F^2$, where $I$ denotes the identity matrix, and $Z$ denotes the full prototype vectors. Another possible alternative is using Maximum Mean Discrepancy (MMD) or Hilbert-Schmidt Independence Criterion (HSIC), whose values directly induce the strength of the dependency between two probabilistic densities. In my opinion, Eq (6) could be better, but Eq (6) itself is okay and can be somewhat sound regularization.
To me, the meaning of Eq (5) is ambiguous. It seems that Eq (5) behaves as minimizing the distances between prototypes and the embeddings; thereby, it enforces to make the value of $e_K$ larger (i.e., make the uncertainty value $u$ smaller). I am not sure how Eq (5) helps the proper uncertainty estimation.
I think both Eq (5) and (6) can be explained with some in-depth discussions with minimal changes in the manuscript, but as of now, I am not fully confident whether the proposed method can represent proper uncertainty estimations despite of using learnable prototypes rather than fixed class information.
### Missing related works on uncertainty estimation
This paper misses many directly related works on uncertainty estimation and cross-modal retrieval. Especially this paper did not consider PCME [A], a probabilistic embedding approach to represent aleatoric uncertainty (i.e., the quality of the data) for cross-modal retrieval.
- [A] Chun, Sanghyuk, et al. "Probabilistic embeddings for cross-modal retrieval." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
I think [A] should be discussed in the related work section and in the experiment section if possible.
There are also more aleatoric uncertainty estimation methods based on probabilistic embeddings, such as:
- Shi, Yichun, and Anil K. Jain. "Probabilistic face embeddings." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
- Park, Jungin, et al. "Probabilistic representations for video contrastive learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
- Neculai, Andrei, Yanbei Chen, and Zeynep Akata. "Probabilistic compositional embeddings for multimodal image retrieval." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
- Sun, Jennifer J., et al. "View-invariant probabilistic embedding for human pose." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16. Springer International Publishing, 2020.
I would like to suggest adding discussions of the comparisons of the SL-based aleatoric uncertainty estimation and the probabilistic embedding-based aleatoric uncertainty estimation.
### Missing comparison methods for MS-COCO 5K
It could be a minor issue, but I think Table 4 misses many relevant and state-of-the-art methods for cross-modal retrieval. For example, as my previous comment, PCME [A] is a highly related work to this paper. In terms of using multiple diverse embeddings, [B] could be another relevant work to the proposed method, which is more relevant to PVSE [49]. I also would like to suggest adding more recent state-of-the-art cross-modal retrieval works, such as VSE$\infty$ [C].
- [B] Kim, Dongwon, Namyup Kim, and Suha Kwak. "Improving Cross-Modal Retrieval with Set of Diverse Embeddings." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
- [C] Chen, Jiacheng, et al. "Learning the best pooling strategy for visual semantic embedding." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.
Especially, Table 4 can be unfair because all other comparison methods do not use the pre-trained CLIP backbone; hence their performances are much worse than a simple "baseline" of this paper. Therefore, in my opinion, Table 4 should contain the comparisons with other methods using the pre-trained CLIP backbone.
However, I fully understand that it is not easy to compare with other cross-modal retrieval methods, especially when they apply different backbones, pre-extracted features, and optimization techniques. After a quick Google search, I found a very recent work, that uses the CLIP ViT-B/32 backbone for cross-modal retrieval:
- [D] Chun, Sanghyuk. "Improved Probabilistic Image-Text Representations." arXiv preprint arXiv:2305.18171 (2023).
I think the experimental results of [D] can be directly comparable with Table 4 of this paper, including VSE $\infty$, PCME, and InfoNCE. I would like to suggest adding the results of [D] to the main paper if the authors are willing to update their table to contain more recent papers. *Note that [D] is not a direct comparison method of the proposed method because it arxived after the NeurIPS submission deadline*; therefore, I don't argue that [D] is a significant related work that should be compared with the proposed method, but I bring [D] because [D] uses the same CLIP ViT-B/32 backbone with the experiments in this paper, and it reported [A] and [C] with the ViT-B/32 backbone, listed in my review.
### Minor comments
- The implementation details of the overall framework are somewhat hard to catch at first glance. I suggest adding the detailed process for computing $e$ and $u$ in Figure 2, and adding the details of $e$ (Table 7 in Appendix) to the main paper.