Further response to Reviewer 3nbF (1/2)
### C1. Labeling audio-text-image triplets using CLIP and CLAP to obtain CLIAP
The compared baseline AudioCLIP（for audio-visual）and ULIP 2 (for 3D-text) actually use the native or constructed audio-text-image and 3D-image-text triplet data to expand existing vision-language MCRs for more modalities. Compared to these models trained on native triplet data, our method achieves better results.
Although we do not think that constructing pseudo data is the key to our method, we still construct audio-text-image triplets to extend the CLAP audio representation to CLIP space, and the retrieval results on AVE and FlickrNet are (3.04, 2.71, 3.16, 3.06), which are much lower than our method (4.11, 4.13, 4.57, 4.92) and slightly better than using "argmax" to label image-audio (2.47, 2.41, 3.14, 2.97). These results prove again that labeling pseudo data is an inefficient representation connection learning method.
In addition, we discussed in our paper and rebuttal that our work is an efficient and flexible contrastive learning scheme, and the idea of constructing pseudo-datasets and using datasets to extend MCR conflicts with exploring Parameter Efficient Learning (PEL) on contrastive representation field.
### C2.1、"the proposed model benefits not only from audio-visual pairs but also from aligning the embedding spaces"
This statement is inaccurate. In the 3D-text experiment, we found that aligning pseudo 3D-text data pairs brings a negative impact on learning connection. And in audio-visual, the benefit of pseudo audio-visual for learning connection is negligible. These new results in more modality settings, showcase pseudo data pairs are dispensable or even harmful to our method.
### C2.2 "It is not a fair comparison when baseline methods exclude audio-visual pairs. The audio-visual information may not be enough for the proposed method, but it could be helpful for the baselines."
The mainly compared baselines, AudioCLIP and ULIP 2, use large-scale audio-visual (3D-text) data for their training. We further discuss the fairness of the comparison between our method and baseline methods as follows:
**Usage of data.** All the data we use are unsupervised unimodal data. For the audio data, our data is the unimodal part of the triplets dataset used by AudioCLIP. That means, the audio data we use are just a subset of datasets used by baseline models by removing all the pair images and texts. The overall data size is approximately equal, while the data quality and annotation we use are much lower than that of baseline models.
**Training costs.** The trainable parameters in our method are 60x times less than that of AudioCLIP, and since all the representations in our method can be pre-extracted and stored, the GPU memory and training time required for training are also far less than the compared baselines.
**Usage of pre-trained models.** The baseline methods AudioCLIP and WAV2CLIP use CLIP, while our method uses the same CLIP and extra CLAP model.
To sum up, our method can be seen as using lower-resource unsupervised data and much less training cost to achieve better contrastive learning results on different modality settings (especially for the modalities without high-quality and large-scale pairs datasets) by connecting and mining the knowledge in multiple existing representation models.
Besides, we also provide the experiment of using the constructed pseudo audio-image and audio-image-text triplet to expand CLIP to ‘CLIAP’. This method of constructing pseudo datasets and then extending MCR, uses the same data, the same pre-training model, and a much more expensive training cost compared to our methods, but achieves is far inferior to our method.
If you mean that the compared baselines, in addition to being trained on their original native paired datasets, also need to be further trained on the reconstructed pseudo-datasets. This far exceeds our method in terms of data, training cost, and pre-trained models. This is a unfair comparison on all aspects.