In real-world scenarios, achieving domain generalization (DG) presents significant challenges as models are required to generalize to unknown target distributions. Generalizing to unseen multi-modal distributions poses even greater difficulties due to the distinct properties exhibited by different modalities. To overcome the challenges of achieving domain generalization in multi-modal scenarios, we propose SimMMDG, a simple yet effective multi-modal DG framework. We argue that mapping features from different modalities into the same embedding space impedes model generalization. To address this, we propose splitting the features within each modality into modality-specific and modality-shared components. We employ supervised contrastive learning on the modality-shared features to ensure they possess joint properties and impose distance constraints on modality-specific features to promote diversity. In addition, we introduce a cross-modal translation module to regularize the learned features, which can also be used for missing-modality generalization. We demonstrate that our framework is theoretically well-supported and achieves strong performance in multi-modal DG on the EPIC-Kitchens dataset and the novel Human-Animal-Cartoon (HAC) dataset introduced in this paper. Our source code and HAC dataset are available at https://github.com/donghao51/SimMMDG.
Paper
Similar papers
Peer review
Summary
This work focuses on addressing a special case of multimodal learning, i.e., multi-modal domain generalization, where one or more modalities will be absent during test phase. This work propose a approach to solve multimodal DG from a disentanglement perspective. Modality-specific features contain modality-specific information, while Modality-shared feature contains information that is shared across domain such as class labels. Supervised contrastive learning is employed to learn better modality-shared features while a distance-based regularize is used to enlarge the discrepancy between modality-specific and modality-shared features. In addition, a cross-modal translation module is proposed to enhance the modality-shared features. This work also release a dataset named Human-Animal-Cartoon (HAC).
Strengths
This work has following strengths: - This paper released a dataset, HAC, which is helpful for multimodal domain generalization community. - This paper is easy to follow. - Multi-modal DG is interesting and has research potential since real-world task is complicated that often involves multiple modalities.
Weaknesses
This work has following weaknesses: - Is the name of the setting appropriate? Typically, domain generalization refer to a setting that testing domain is unseen during training. However, the setting in this work is that the testing domain/modality exists in training data but some training modalities might be missing. - Lack of proof that the network truly learn modality-specific and modality-shared features. - It's not clear that why Cross-modal Translation can enhance modality-specific features, since the MLP use both features for translation. How to guarantee the two types features are well disentangled. - The technique contribution of this work is somehow not strong. Supervised contrastive loss is a mature technique. What's the essence to use supervised contrastive loss here? Can CE loss obtain similar performance? Translation idea is also appeared in [1]. [1] Ge, Yunhao, et al. "Zero-shot synthesis with group-supervised learning." arXiv preprint arXiv:2009.06586 (2020).
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Summary
This paper addresses the challenge of domain generalization in a multi-modal setting. The core idea is that aligning features from multiple modalities in a shared embedding space can hinder generalization. To overcome this limitation, the paper proposes a more effective approach: it separates the features within each modality into modality-specific and modality-shared components and then aligns only the modality-specific components. This alignment is achieved through supervised contrastive learning and cross-modal feature translation techniques. To validate the proposed approach, the paper introduces a new dataset called HAC, which consists of video, audio, and optical flow data from three domains: Human, Animal, and Cartoon. Extensive experiments conducted on both the HAC dataset and the EPIC-Kitchens dataset demonstrate the impressive performance of the proposed method when compared to existing approaches.
Strengths
1. Multi-modal domain generalization is an important practical problem and the proposed method and dataset introduced in this paper will help in advancing research in this direction. 2. From a methodological standpoint, the idea of splitting the features into modality-specific and modality-shared and aligning them separately seems new. Interestingly, this goes against the strategy followed by popular multi-modal models such as CLIP. 3. The paper showcases a comprehensive range of experiments to validate the effectiveness of the proposed method. It explores various combinations of modalities (e.g., Audio+Video, Audio+Video+OpticalFlow), different numbers of training domains, and even evaluates performance in scenarios with missing modalities during inference. This thorough evaluation provides a robust understanding of the method's capabilities and limitations. 4. The paper is very well written and easy to follow. The diagram in Figure 2 is especially well done and effectively illustrates the framework in a concise manner.
Weaknesses
1. The paper highlights an apparent contradiction in its argument regarding modality-specific information. On one hand, it acknowledges that certain modalities, such as video and flow, possess distinct and unalignable features, such as lighting and motion information. However, the paper introduces a cross-modal transformation module that seemingly enables seamless transformation between these features. This raises a critical question: Does this transformation not contradict the initial premise that modality-specific information is inherently unalignable? 2. The paper notably overlooks the utilization of domain labels or any explicit feature alignment techniques across domains. This omission raises concerns about the effectiveness of the proposed method in achieving robust domain generalization.
Questions
1. Can the usage of domain labels and explicit domain alignment strategies improve the generalization performance? 2. Can you clarify the contradiction pointed out in Weakness-1?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The authors adequately addressed the limitations in the paper.
Summary
This paper presents a framework for domain generalization for models trained with multi-modal/ multi-source data: SimMMDG. The approach is motivated by the observation that each modality contains features shared with other modalities as well as features specific to that modality. The paper proposes to split per-modality features such that modality-shared features should be similar while modality-specific features should be complementary. A multi-loss strategy is adopted to implement this idea. The author evaluates the approach's performance on EPIC-Kitchen and a proposed Human-Animal-Cartoon dataset.
Strengths
- The proposed approach, SimMMDG seems simple yet novel. It relies on a simple intuition that modalities should have modality-specific features, which had been overlooked in previous literature. The approach seems very general and has the potential to become a popular framework used by any researcher working with multi-modal representation learning. Its simplicity also enables combining this framework with previous approaches. - On the empirical results, the method seems to provide non-trivial gain over baselines and previous approaches targeting multi-modal supervised classification. - The paper is clearly written; it feels one can directly implement the proposed idea without too much trouble.
Weaknesses
- Despite how much I appreciate the idea proposed by the paper, some important experiments are missing, which leads me to question the technical soundness of the work. For example, the learning rate of CL (supervised contrastive loss) is set to 3.0; this essentially increased the effective learning rate of the model significantly. I wonder if authors have tuned the baseline learning rate accordingly to ensure that the performance gain is not influenced by the learning rate change. In Table5, it seems all results adds CL; what if we remove CL, can CT effectively learn similarly as CL? In addition, the work is often motivated with the recent use of text + vision, but the evaluation does not include any text+vision task. - Impact of the work. The proposed framework seems very general, but the author eventually chose to test only on Domain Generalization. This seems to undersell the work, since it seems feasible to implement this idea to perform multi-modal classification tasks in a generic fashion. I wonder if authors have tried more evaluation apart from domain generalization, and if the methods provide improvement there. - Lack of discussion on HAC. One of the contribution of the paper is the HAC dataset. This is a new evaluation benchmark, yet studies/ analysis/ details on this dataset is missing. I also checked the supplementary of the paper, but found very limited information on this benchmark, such as dataset statistics, how are videos selected, how is the annotation performed, is there any QA adopted. Without details and studies (e.g. comprehensive baseline evaluation) on HAC, it will be hard to count HAC as a sound contribution and trust evaluation on this new benchmark. - SimMMDG with other baseline methods. A major contribution author claims is on the simplicity. This makes me wonder if the pipeline is actually compatible with many of the baseline adopted, such as Gradient Blending [53] and Non-Local [54]. I wonder if authors have tried their approach with these methods to show if the proposed framework is complementary to existing techniques. In general, although I really enjoy reading this paper and likes the intuitions, it seems the work needs more polish and empirical evaluation to prove its soundness.
Questions
See weaknesses
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Author should discuss the potential impact regarding their newly proposed dataset HAC.
Summary
This paper proposes a method for multi-model domain generalization. Rather than mapping features of different modality into the same embedding space, it splits features within each modality into modality-specific feature and modality-shared feature. Then it applies different learning strategies to different split features. Besides, a new dataset called Human-Animal-Cartoon is proposed in this paper. Experiments a re conducted on two datasets and shows good performance.
Strengths
The paper proposes a novel method for multi-model domain generalization and a new dataset for this task. The motivation and method are clear to me. The results are convincing to me.
Weaknesses
I am confused to the section 3.2.2 Cross-model Translation. The design is translating the i-th modality feature to j-th modality feature. However, like the paper states that, different modality features contains share-features and unique features. Does the translation will undermine the unique features in different modalities?
Questions
See the weaknesses.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The cross-modal translation does not make sense to me. Will rise my rating if rebuttal can address my concern.
Summary
The paper tackles the problem of multimodal domain generalization in which the goal is to learn a model from multi-modal multi/single-source domain data that can generalize to unseen multimodal distributions. The main motivation is that mapping data from multiple modalities to a shared embedding space only harnesses the common features among different modalities while overlooks the modality-specific information. The paper proposes three main technical contributions to tackle the aforementioned problem and address the multimodal domain generalization problem. At first, each modality features are split into modality-shared and modality-specific components and then a supervised contrastive loss is used for alignment. Next, a distance loss is leveraged to repel the modality-specific features from each modality. Also, a cross-modal translation module is proposed to regularize the learned features and facilitate the missing modality issue. Finally, the paper also introduces a new multimodal DG dataset, namely Human-Animal-Cartoon (HAC). Results on EPIC-kitchens and HAC datasets claim improved DG performance under both multi-source and single-source settings.
Strengths
1) The problem of multimodal domain generalization is relatively new and mostly underxplored and carries practical relevance because in many real-world scenarios the data is often observed via multiple modalities. 2) The cross-modal translation module is capable of regularizing the learned features and handle missing modalities issue. 3) The paper proposes a new Human-Animal-Cartoon (HAC) dataset, which consists of different action classes and three different domains, to facilitate research in multimodal DG. 4) Results on two different datasets and under both multi-source and single-source DG setting show that the proposed framework provides notable gains over the baselines and existing methods.
Weaknesses
1) The paper seem to be missing the domain generalization aspect, like which pertinent DG problem in the context of multimodal data the paper is trying to solve, because all the technical contributions mostly aim at either bringing the modality-shared component or repel the modality-specific component which seems effective compared to the only relevant baseline [44]. 2) The two important technical contributions i.e. learning shared embedding space and pushing modality-specific feature is not very new in the context of DG and even broader scope [A],[B]. This is also because the supervised contrastive loss [29] as well as the distance based loss are the existing components taken from literature. 3) The unimodal DG comparison is made with rather old approaches RSC [23] and MixUp [55] which weakens the claim of significant performance gain. There are several new unimodal DG approaches which have shown state-of-the-art performances such as [C], [D] and [E]. 4) Fig. 2 that displays the overall architecture of the proposed method is not very helpful towards grasping the mid-level idea. For instance, one thing is to improve the patterns and colors. Also, the some symbols from the text in methodology should be used in the diagram to better connect the two. 5) Details on how the HAC datsets is being developed is missing. For instance, how many volunteers were involved? what were the quality assurance mechanisms in place? I was expecting these details at least in the supplementary material, but the supp material only provides some high-level statistics of the dataset such as number of videos or to a little bit class-imbalance issue. [A] Bui, M.H., Tran, T., Tran, A. and Phung, D., 2021. Exploiting domain-specific features to enhance domain generalization. Advances in Neural Information Processing Systems, 34, pp.21189-21201. [B] Chattopadhyay, P., Balaji, Y. and Hoffman, J., 2020. Learning to balance specificity and invariance for in and out of domain generalization. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16 (pp. 301-318). Springer International Publishing. [C] Cha, J., Chun, S., Lee, K., Cho, H.C., Park, S., Lee, Y. and Park, S., 2021. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34, pp.22405-22418. [D] Rame, A., Kirchmeyer, M., Rahier, T., Rakotomamonjy, A., Gallinari, P. and Cord, M., 2022. Diverse weight averaging for out-of-distribution generalization. Advances in Neural Information Processing Systems, 35, pp.10821-10836. [E] Rame, A., Dancette, C. and Cord, M., 2022, June. Fishr: Invariant gradient variances for out-of-distribution generalization. In International Conference on Machine Learning (pp. 18347-18377). PMLR.
Questions
The paper tackles a challenging and a relatively new problem of how to learn a cross-domain generalizable model under multi-modal settings. Overall, the paper presents an effective framework comprised of three components and alongside a new multimodal dataset, however, there are some important questions/concerns as listed in weaknesses (1-5) due to which my initial rating for the paper is ‘weak accept’. Also: It would be interesting to know the performance of the method under: a) different types of domain shifts, such as texture, background etc. b) increasing domain shifts. Are the results reported as the average of different trials and if yes, what are the standard deviations?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The paper mentions one limitation regarding the increasing complexity of cross-modal translation module with increasing number of modalities.
I thank authors for providing a thorough response to my comments in the rebuttal and I have gone over other reviews. Most of the rebuttal responses are satisfactory, and I'm leaning towards accepting the paper, however, I would like more clarification on the following: Q1: I think this response is bit weak. Because what do you mean by solving the general multi-modal DG problem? It would be good to know the specific challenge(s) that arise when extending traditional DG to multimodal case and this paper is trying to solve? Q5: Thanks for providing these details. Is it possible to provide any statistics on the dataset development process, for instance, discarded unqualified data, noisy/wrong classes etc. and accuracy of the dataset?
We are glad to hear that we have addressed most of your concerns and that you are leaning toward accepting the paper! Thanks for spending a significant amount of time on our submission and giving lots of valuable suggestions, which make our paper even stronger! We will include all added experiments and the details on our HAC dataset in the final paper. We address your further concerns and questions as follows: ___ **A1**: There are two main challenges that arise when extending traditional DG to multimodal cases. 1. The first challenge is **how to better exploit complementary information from different modalities.** Different modalities consist of both shared information that is consistent across modalities and unique information that is specific to each modality. It is essential to extract both types of information in order to make better predictions, as motivated in our introduction part and Figure 1 (a) in the main paper. Traditional multi-modal contrastive learning frameworks [29, 45] project the features of different modalities into a common embedding space, which will only preserve modality-shared information and overlook modality-specific information. In order to address this challenge, we propose to split the feature embedding of each modality into modality-specific and modality-shared components and use supervised contrastive learning and distance loss to extract meaningful separated features. 2. The second challenge is **how to effectively facilitate modality interactions.** Modality interactions referred to as different modality elements interact to give rise to new information when integrated together for task inference [1]. Several works [2,3] have already demonstrated that modalities interaction can help improve the performance of multi-modal tasks. For example, the proposed approach in [2] learns a coordinated similarity space between image and text to improve image classification. The approach proposed in [3] translates language into video and audio for language sentiment analysis. In this work, we propose to use a cross-modal translation module for modality interaction to regularize the learned features. We also address the missing-modality generalization problem with our proposed cross-modal translation module, which is robust even in scenarios where multiple modalities are missing. Solving the general multi-modal DG problem means we propose a simple and universal framework, that can be used for multi-modal multi-source DG, multi-modal single-source DG, and missing-modality DG problems. Our framework can also be easily combined with other DG strategies to achieve further performance improvement (SimMMDG+Gradient Blending and SimMMDG+DANN as shown in global response). ___ **A5:** The discarded unqualified data referred to those videos with no audio or with very large background noise, videos with very low resolution and quality, or duplicate videos. Since we introduced the basic requirements for the data to each volunteer beforehand and split the action classes and cartoon series for each person, there is very little such data and we discarded about 10 such unqualified samples. The noisy/wrong classes referred to those videos with ambiguous classes, such as "a person is watching TV and at the same time eating/drinking something", or "a person is running while opening the door". We discarded about 30 such unqualified samples. After these quality assurance mechanisms, we make sure that all data samples are unique and of good quality in our dataset. ___ If you have any other questions or concerns, we would really appreciate the opportunity to discuss them with you further. Thank you again! ___ [1] Paul Pu Liang, et al. Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions, arXiv preprint arXiv:2209.03430, 2022. [2] Andrea Frome, et al. Devise: A deep visual-semantic embedding model. In NeurIPS, 2013. [3] Hai Pham, et al. Found in translation: Learning robust joint representations by cyclic translations between modalities. In AAAI, 2019.
I thank authors for providing further clarifications on A1 and A5. I believe authors have satisfactorily addressed my concerns and I would like to keep my pre-rebuttal rating. I would strongly encourage authors to include the points from rebuttal, including the Q3 experimental results in the main paper.
Thanks for recognizing our efforts and keeping your positive rating!
We are glad to hear that we have addressed your concerns and that you keep your positive rating! Thanks for spending a significant amount of time on our submission and giving lots of valuable and insightful suggestions, which make our paper even stronger! We will for sure include all added experiments and points in the final paper for better clarification.
Further question regarding Q2. Figure 4 (d) and (e) show that the modality-shared features are aligned and the modality-specific features are aligned. 1. How to make sure that the modality-shared feature is modality-shared, and the modality-specific feature is modality-specific?
Thanks for proposing your further questions. Based on our assumptions in the paper, modality-shared features reflect shared information between all modalities, like all modalities that describe the same people, objects, actions, or gestures. Modality-specific features are specific pieces of information that are unique to each modality, like texture, depth, and visual appearance in images, syntactic structure, vocabulary, and morphology in language. Our goal is to align the modality-shared features of different modalities from different source domains with the same label to be as close as possible in the embedding space, while pushing away features with different labels, to make the embedding space more distinctive. At the same time, we want the modality-specific features to be as far as possible from the modality-shared features, such that they carry different types of information. From the methodology view, we make sure of this by using supervised contrastive loss on modality-shared features and adding distance loss between modality-specific and modality-shared features. From the experimental results view, we further calculate two correlation metrics to validate our assumption. Let's assume $\mathbf{E}^{v}_s$ to be the modality-specific feature of video and $\mathbf{E}^{v}_c$ to be the modality-shared feature of video. We define similarly $\mathbf{E}^{a}_s$ and $\mathbf{E}^{a}_c$ for audio. We first calculate the Pearson correlation coefficient between modality-specific and modality-shared features. Pearson correlation coefficient is the most common way of measuring a linear correlation between two sets of data. It is a number between –1 and 1 that measures the strength and direction of the relationship between two variables. From our results, the Pearson correlation coefficient between $\mathbf{E}^{v}_c$ and $\mathbf{E}^{a}_c$ is 0.70, which means the modality-shared features of video and audio have a strong positive correlation. The Pearson correlation coefficient between $\mathbf{E}^{v}_s$ and $\mathbf{E}^{a}_s$ is -0.44, which means the modality-specific features of video and audio have a weak negative correlation. This is consistent with our assumption that modality-specific features reflect shared information and should have a high correlation, while modality-specific features reflect unique information in each modality and should have a weak correlation. We also calculate the Cosine similarity between modality-specific and modality-shared features. Cosine similarity is a measure of similarity between two non-zero vectors defined in an inner product space, which is also related to correlation. Cosine similarity is also a number between –1 and 1 that measures the strength and direction of the relationship between two variables. From our results, the Cosine similarity between modality-shared features $\mathbf{E}^{v}_c$ and $\mathbf{E}^{a}_c$ is 0.82 and the Cosine similarity between modality-specific features $\mathbf{E}^{v}_s$ and $\mathbf{E}^{a}_s$ is -0.10. This also indicates that the modality-shared features of video and audio have a strong positive correlation and the modality-specific features of video and audio have a weak negative correlation. From the results of the above two correlation metrics, we can conclude that our modality-shared features are indeed modality-shared, as they have a strong positive correlation across modalities. And our modality-specific features are indeed modality-specific, as they have a very weak correlation across modalities. If you have any other questions or concerns, we would really appreciate the opportunity to discuss them with you further. Thank you again!
Thanks for the authors' reply. Authors use numbers to validate that "modality-shared" features are close across domains while far away from "modality-specific" features. However, this can not lead to those "modality-shared" features representing shapes, actions, or gestures. What I intend to state is that it's not guaranteed that "modality-shared feature" in this paper is truly meaningful and sharable. For example, stylegan-like works use example [content A] + [style B] = [image A with style B] to validate [content] feature has extract content information.
Thanks for proposing your further insightful questions. Following your suggestions, we further analyzed the meaningfulness and the ability to share features of modality-shared features by adding another experiment on cross-modal retrieval (similar to Table 13 in CLIP [1]). The ability of cross-modal retrieval is highly related to the shareable of features, as only features that are meaningful, shareable, and highly connective can give a good recall rate. We use modality-shared features of videos to retrieve from the modality-shared features of audios and calculate the recall, and vice versa for audios. We report the R@1, R@5, and R@10 values for Video to Audio Retrieval and Audio to Video Retrieval. For example, R@5 for Video to Audio Retrieval means we retrieve 5 candidates from audio, if at least one of them has the same label as the query video, we consider the retrieval success and then we calculate the average success rate for all query videos. We also do the same thing on modality-specific features: using modality-specific features of videos to retrieve from the modality-specific features of audios, and vice versa for audios. As shown below, we have a very high recall rate when we use modality-shared features, while the recall rate is biased towards random when we use modality-specific features. This further indicates that the modality-shared features are truly shareable and meaningful and there are very strong relations and connections across modalities. The modality-specific features are instead with more information that is private to each single modality. **Video to Audio Retrieval:** | | R@1 | R@5 | R@10 | | :--- | :--- | :--- | :--- | Modality-specific Features | 11.80 | 45.37 | 49.78 | | Modality-shared Features | $\mathbf{76.58}$ | $\mathbf{90.13}$ | $\mathbf{92.83}$ | **Audio to Video Retrieval:** | | R@1 | R@5 | R@10 | | :--- | :--- | :--- | :--- | | Modality-specific Features | 10.24 | 33.01 | 47.10 | | Modality-shared Features | $\mathbf{67.59}$ | $\mathbf{89.60}$ | $\mathbf{93.99}$ | To further verify the meaningfulness of modality-shared features, we also train a classifier only on the concatenation of modality-shared features of video and audio, and discard the modality-specific features. As shown below, without modality-specific features, the performances drop slightly compared to the whole SimMMDG framework, but still competitive. This indicates that modality-shared features truly have some meaningful information that can be used for prediction tasks. | Method | D2,D3→D1 | D1,D3→D2 | D1,D2→D3 | mean | |---------|----------|----------|----------|------| | SimMMDG (modality-shared features only) |54.71|64.67|62.83|60.74| | SimMMDG |57.93|65.47|66.32|63.24| In this paper, we assume modality-shared features reflect all shared/mutual information between different modalities, rather than specific attributes like identity, pose, or background as in [2]. This is reasonable, as different modalities describing the same action/class/object should have some mutual information, as motivated in our introduction part and Figure 1 (a) in the main paper. We only disentangle features at the modality level, i.e. modality-shared and modality-specific features, to keep the whole framework simple and extendable. We don't further disentangle modality-shared or modality-specific features into different domain-specific attributes, like identity, pose, and background, as in [2]. Furthermore, it is important to note that our work primarily focuses on the realm of classification rather than generation. These are the reasons why it is not possible to validate the features in a similar way to StyleGAN. However, we effectively validate that modality-shared features are truly meaningful and shareable and modality-specific features are specific pieces of information that are unique to each modality, by the carefully designed experiments above, as well as the Pearson correlation coefficient and Cosine similarity calculated in the last comment. In future work, it would be also interesting to combine [2] with our proposed framework to learn disentangled features at both domain and modality levels. Thanks for your insightful questions! We will put all added experiments in the main paper for better clarity. If you have any other questions or concerns, we would really appreciate the opportunity to discuss them with you further. Thank you again! [1] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." ICML, 2021. [2] Ge, Yunhao, et al. "Zero-shot synthesis with group-supervised learning." arXiv preprint arXiv:2009.06586 (2020).
I appreciate the authors' effort in response. Modality-shared feature shows an absolute advantage against the modality-specific feature on cross-modal retrieval tasks. I believe these experiments can prove that the modality-shared features are truly shared across modalities cause it benefits the cross-modal tasks. And this is more convincing than the feature distance of different feature groups (shared-shared, shared-specific, specific-specific feature). I highly recommend including those cross-modal experiments in the main paper. I believe the authors have addressed my concerns and I will raise my score to positive.
Thanks for recognizing our efforts and raising your score to positive!
We are glad to hear that we have addressed your concerns and that you raised your score to positive! Thanks for spending a significant amount of time on our submission and giving lots of valuable and insightful suggestions, which make our paper even stronger! We will for sure include all added experiments in the final paper for better clarification.
Thanks for your response
After reading authors' rebuttal and other reviews, I think the main concerns are addressed very well with the new experiments provided. I think authors should consider including them in the main draft of the paper. These results improve the soundness of the proposed approach. I am inclined to raise my rating for acceptance. Would like to hear the post-rebuttal feedback from other reviewers. Minor: For Q1, I meant setting weight 3.0 means the learning rate is changed by 3x. But since you did parameter search already, this is not a concern anymore.
Thanks for being inclined to raise the rating for acceptance
We are glad to hear that we have addressed most of your concerns and that you are inclined to raise the rating for acceptance! Thanks for spending a significant amount of time on our submission and giving lots of valuable suggestions, which make our paper even stronger! We will for sure include all added experiments and the details on our HAC dataset in the final paper.
A kind reminder to update the final rating in the system
Dear Reviewer kbvb, We sincerely appreciate your insightful comments and willingness to raise your rating for acceptance! Your feedback has been immensely beneficial in enhancing the quality of our paper. Considering the author-reviewer discussion period is ending soon, we are afraid that the rating cannot be changed anymore after this period. Could you please spend some time updating the rating in the system if you are satisfied with our rebuttal? If you have any other questions or concerns, we would really appreciate the opportunity to discuss them with you further. Thank you once again for your time and engagement throughout this process!
Thanks for the response
Thank authors for the detailed clarification. It make sense to me. After reading all other reviewer's opinion, I will change to my rating to accept.
Thanks for recognizing our work and being willing to change your rating to accept!
We are glad to hear that we have addressed your concerns and that you will change your rating to accept! Thanks for spending a significant amount of time on our submission and giving lots of valuable and insightful suggestions, which make our paper even stronger! We will also include all added experiments and points in the final paper for better clarification.
A kind reminder to update the final rating in the system
Dear Reviewer Zavq, We sincerely appreciate your insightful comments and willingness to change your rating to accept! Your feedback has been immensely beneficial in enhancing the quality of our paper. Considering the author-reviewer discussion period is ending soon, we are afraid that the rating cannot be changed anymore after this period. Could you please spend some time updating the rating in the system if you are satisfied with our rebuttal? If you have any other questions or concerns, we would really appreciate the opportunity to discuss them with you further. Thank you once again for your time and engagement throughout this process!
Thank you for the response. The authors have adequately addressed my concerns. The explanation for the domain translation makes it more apparent to me. I would encourage the authors to include this intuitive reasoning in the main paper and explicitly point out that this translation is lossy, as explained above. Also, the experiments with domain labels strengthen the method in my opinion. Thus, I would like to raise my score to weak accept.
Thanks for recognizing our work and being willing to raise your score!
We are glad to hear that we have addressed your concerns and that you will raise your score! Thanks for spending a significant amount of time on our submission and giving lots of valuable and insightful suggestions, which make our paper even stronger! We will include all added experiments and intuitive reasoning for the cross-modal translation module in the final paper for better clarification.
A kind reminder to update the final rating in the system
Dear Reviewer c1hR, We sincerely appreciate your insightful comments and willingness to raise your score to weak accept! Your feedback has been immensely beneficial in enhancing the quality of our paper. Considering the author-reviewer discussion period is ending soon, we are afraid that the rating cannot be changed anymore after this period. Could you please spend some time updating the rating in the system if you are satisfied with our rebuttal? If you have any other questions or concerns, we would really appreciate the opportunity to discuss them with you further. Thank you once again for your time and engagement throughout this process!
Decision
Accept (poster)