Classifier-guided Gradient Modulation for Enhanced Multimodal Learning

Summary

This paper proposes a balanced multimodal learning method. Compared to existing methods that only consider the gradient size, it also considers the direction of the gradient.

Strengths

The experiment includes multiple data sets and multiple tasks.

Weaknesses

There is less visualization analysis of the experiment.

Questions

1. It is interesting that the author calculates the improvement of each modality (such as the change in accuracy) instead of the current performance. However, it is hoped that more convincing theoretical proofs can be added. 2. It is recommended to visualize changes in indicators before and after modulation (such as utilization rate, etc.), including only adjusting the gradient size, gradient direction, and both. It is best to also add the performance change process for each modality to avoid "using the indicators proposed by yourself to measure your performance." 3. Can it be implemented using only one classifier, instead of one classifier for each modality? 4. Please add a brief introduction to the experimental comparison method, such as AGM, PMR, etc. 5. The method in this paper is similar to the OGM method, but the original OGM also includes a part that uses Gaussian noise to improve generalization. Is this part of the method used in this paper? Can you add relevant discussions?

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

Yes, the authors have adequately addressed the limitations

Summary

This paper proposed a balanced multi-modal learning method with Classifier-Guided Gradient Modulation (CGGM), considering both the magnitude and directions of the gradients, with no limitations on the type of tasks, optimizers, the number of modalities.

Strengths

1. Balanced multi-modal learning considering both the magnitude and directions of the gradients is a reasonable idea. 3. The proposed method is easy to follow.

Weaknesses

1. Balanced multi-modal learning considering the directions of the gradients is not novel. A previous work [1] have already analyze the issue of modality dominance caused by gradient conflicts. The difference and the comparison with this method should be considered in detail. Besides, the approach to controlling gradient magnitude is similar to the ideas of OGM[2] and PMR[3]. 2. This framework still does not explore the imbalance issue of multi-modal learning in more flexible task formats, such as the potential imbalance in tasks like AVQA and multi-modal generation. Expanding task formats to regression and segmentation tasks is only a minor improvement. Existing work can also be extended to these tasks with minor adjustments. [1] Wang, H., Luo, S., Hu, G. and Zhang, J., 2024, March. Gradient-Guided Modality Decoupling for Missing-Modality Robustness. In *Proceedings of the AAAI Conference on Artificial Intelligence* (Vol. 38, No. 14, pp. 15483-15491). [2] Peng, X., Wei, Y., Deng, A., Wang, D. and Hu, D., 2022. Balanced multimodal learning via on-the-fly gradient modulation. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition* (pp. 8238-8247). [3] Fan, Y., Xu, W., Wang, H., Wang, J. and Guo, S., 2023. Pmr: Prototypical modal rebalance for multimodal learning. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (pp. 20029-20038).

Questions

The weaknesses above should be carefully considered.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors have addressed the limitations.

Summary

This paper focuses on the notorious modal imbalance problem in multi-modal learning. To alleviate the modality imbalance, the proposed method modulates gradient magnitude and the directions of the gradient simultaneously. Experiments on various multi-modal datasets demonstrate the efficiency.

Strengths

- This paper explores an interesting problem. In the current joint learning paradigm, the dominant modality overpowers the learning process and the resulting gradient prohibits further exploration of the features in the weak modality. - This paper is well-written and easy to follow.

Weaknesses

- **The motivation is not stated clearly.** From Eq(3) and Eq(4), we can observe that the gradient magnitude can affect the update of a specific modality. However, the explanation of how the gradient direction between the specific modality and their fusion influences the modality update is unconvincing. - **The experiments are not convincing.** The authors ignore recent state-of-the-art methods, such as UMT/UME [1], QMF [2], and ReconBoost [3]. It is recommended that the authors compare these methods. Additionally, the authors should plot the gradient direction, accuracy curve, and gradient profile after using their method and compare these to Fig. 2 to better highlight the effectiveness of their approach. - **The related work section lacks discussion on recent research.** UMT [1] distills well-trained uni-modal features to assist multi-modal learning. QMF [2] provides a quality-aware multimodal fusion framework to mitigate the influence of low-quality multimodal data. ReconBoost [3] finds that the major issue arises from the current joint learning paradigm. They propose an alternating learning paradigm to fully harness the power of multi-modal learning. - Some notations are confusing. Please see the questions below. For now, I recommend a borderline for this paper, leaning to reject. If the concerns in weakness can be addressed in the rebuttal phase, I am willing to raise my concern and accept this paper. [1] On Uni-Modal Feature Learning in Supervised Multi-Modal Learning. ICML 2023. [2] Provable Dynamic Fusion for Low-Quality Multimodal Data. ICML2023 [3] ReconBoost: Boosting Can Achieve Modality Reconcilement. ICML 2024.

Questions

In section 3.2, $\mathcal{L}$ denotes both the overall empirical loss and the loss of individual samples. It is recommended to denote the loss of individual samples as $\ell$. Equations (3) and (4) are incorrect. The chain rule of differentiation for a scalar with respect to multiple vectors should be applied, rather than the chain rule for a scalar with respect to another scalar. $$ \frac{\partial z}{\partial x} = \left(\frac{\partial y}{\partial x}\right)^{T}\cdot \frac{\partial z}{\partial y} $$

Rating

7

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

None.

Summary

This paper proposes CGGM, a novel strategy to balance the multimodal training process. Compared with existing methods, it can deal with the unbalanced multimodal learning problem with different optimizers, takes, and more than two modalities.

Strengths

The motivation is sufficient and the experiments on different tasks and datasets prove that the proposed method solves the problem well. CGGM stands out by considering both the magnitude and direction of gradients for balancing multimodal learning. This combined approach effectively addresses the modality competition problem and ensures that all modalities contribute equally to the model’s performance.

Weaknesses

1. The reviewer is curious about the computational complexity of the additional classifier or decoder. Is there any experimental result? 2. As present in Line 149, the classifier fi consists of 1-2 multi-head self-attention (MSA) layers and a fully connected layer for classification and regression tasks. Does this apply to all models or classification tasks? Why is it set up like this? Why not just set it to the same classifier structure as the multimodal head? 3. What's the light decoder used for segmentation tasks? 4. The introduction of unimodal classification may limit the learning of multimodal tasks, such as gradient conflicts. How to deal with this problem? 5. PMR has also discussed the problem of gradient direction and introduced unimodal loss to assist multimodal learning. What's the difference between CGGM and PMR? 6. The reviewer is concerned about the accuracy of using the difference between the two consecutive ε to denote the modality-specific improvement for each iteration. According to my experience, the loss of the dominant modality will quickly drop to the magnitude of 1e-2 to 1e-3, while the magnitude of the weak modality is around 1e-1. At this time, the loss change of the weak modality will be larger, and according to the author, it will be regarded as the dominant modality. I 7. What's the performance of the proposed method on CRAME-D and AVE datasets? They are also widely used in previous studies.

Questions

see weakness

Rating

4

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

No