Response to Reviewer pnPG (1/2)
Thank you very much for your appreciation of our work and your valuable suggestions on the presentation of our manuscript. In our newly submitted PDF, we have carefully revised the presentation of most paragraphs to improve its clarity and readability. We hope the new submission can solve your concerns about writing.
### **W1: statements about ULIP**
ULIP aligns a 3D encoder to an image-text model called SLIP [1] (not CLIP) through 3D-image-text data. Ex-MCR only uses the aligned 3D-image representation of ULIP and extends it to a different image-text model (i.e., CLIP) via the paired-data-free way. So we are not reproducing or refining the 3D-text alignment of ULIP, but building a new alignment between the 3D representation and new image-text pre-training.
[1] Slip: Self-supervision meets language image pre-training. Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. ECCV 2022
### **W2: repetition of acronyms**
Thanks for your suggestions, in the newly submitted paper we carefully refine the usage of acronyms to improve readability. The modifications of acronyms can be summarized as:
1. Avoiding confusion due to many similar acronyms. Replace “MCR” with "space", "intra-MCR alignment" with "intra-space alignment", and "inter-MCR alignment" with"inter-space alignment". Remove unnecessary acronyms, like “MLP”, “InfoNCE Loss” and “L2 Loss”
2. Use "audio-text space" "image-text space" and "image-3D space" in the introduction instead of using "CLAP", "CLIP", and "ULIP" to directly explain the aligned modalities in each space.
### **W3 & W4: Statements in the “modality-centric consistency” part**
We reorganize and rewrite the entire "modality-centric consistency" part in the new submission.
In the introduction, we avoid directly giving many self-defined terms without explicit explanation, such as: “modality-centric”, “semantic bias” and “reflect an MCR space comprehensively”. The so-called modality $\mathcal{A}$-centric pseudo data means using only data of modality $\mathcal{A}$ to retrieve semantically similar data in other modalities, while "semantic bias" means that one modality cannot be fully represented by another modality (giving examples of audio and images to support), and "various modality-centric data" denotes combining the pseudo pairs retrieved by different modalities.
In the first and second paragraphs of the Section 3.2.1, we define the "single modality-centric data" and "various modality-centric data", and provide examples of image of “mushroom” and audio of “wind noise” to illustrate that single modality-centric data cannot comprehensively reflect representations of different modalities in different MCR spaces.
### **W5 & Q1 & Q2: unclear notation**
The $softmax(\cdot)$ in Equation 1 and Equation 2 is over all the samples in used datasets.
The InfoNCE function in Equation 6 is calculated on all the samples in a training batch.
The tilde symbols mean the features are processed to be semantically consistent.
### **W6: The metrics in Table 3,4,5,6**
At the caption of Table 3 and the beginning of Section 4.5, we add descriptions about the metrics used in Tables 3, 4, 5, and 6. In ablation experiments, we report the mAP metrics on audio-image retrieval (AVE) and audio-text retrieval (AudioCaps). In the Appendix, we provide detailed ablation experiment results on all the datasets and metrics.
### **W7: Too crowded Figure 1**
We redesign Figure 1 and rewrite its caption. Figure 1 is divided into two sub-figures (a) (b).
**To improve the simplicity of the figure, we made the following changes to Figure 1 (a):**
1). Remove unnecessary text such as "pull close" and "pull close & push away" and replace them with a gray bold dash line.
2). Replace text such as "audio features" "CLAP Text features" with corresponding vector symbols. On the one hand, keeping there are only vector symbols, modules, and lines in the pipeline figure, can display the calculation process more clearly. On the other hand, it echoes the notation definitions in Section 3.2 and helps the reader to quickly understand the meaning of the defined notations.
3). Frame the base-MCR and two leaf-MCRs into three boxes, which not only improves the aesthetics and simplicity, but also emphasizes that the three pre-trained spaces are separate.
4). Add explanations about some symbols at the bottom to help understand the illustrations.
**For Figure 1 (b), the subfigure about various modality-centric data**
we cancel the repeated "text-centric consistency", "audio-centric consistency", and "image-centric consistency" which are difficult to understand. More emphasis is placed on combining pseudo-data pairs from different sources to form a data pool.
For captions, we do not broadly explain the entire picture, but provide relevant explanations for each sub-picture. And use more space to explain the more complex Figure 1 (b).