Response by authors
Thanks for your valuable feedback and insightful comments! We respond to some concerns below:
1. Transformer adopted training from scratch and did not consider any **pretraining strategies**.
Thank you for your advice on our work. To validate it, we conduct experiments using the ImageNet-pretrained Transformer on the Kinetic-Sounds dataset. Experimental results demonstrate that our method can also perform well with pretraining.
| Transformer | clean | FGM | PGD-$\ell_2$ |
|:-----------:|:-------:|:-------:|:-------:|
| Baseline | 0.6788 | 0.1366 | 0.0865 |
| CRMT (ours) | 0.7406 | 0.3198 | 0.2078 |
2. How does the method **generalize to other fusion mechanisms**, besides late fusion.
Thank you for your comments. In this manuscript, our analysis mainly focuses on the representative fusion strategy, late fusion, which is widely used in multi-modal research [1,2].
Meanwhile, our method are adaptable to other fusion mechanisms, where the modality-specific representations could interact earlier in the process. Previously, we defined the margin as $\zeta^{(m)}_j(\boldsymbol{x}^{(m)})$, where $j$ is the nearest class to calculate the margin, and the margin is only determined by uni-modal input $\boldsymbol{x}^{(m)}$. To adapt our method for these scenarios including intermediate fusion, we can redefine the representation margin as $\zeta^{(m)}_j(\boldsymbol{x}^{(1)}, \boldsymbol{x}^{(2)})$, indicating that both modalities' input influence the margin. This modification allows us to extend the framework to measure multi-modal perturbations in a more integrated manner. Additionally, we can adapt the definition of the Lipschitz constant in our theoretical analysis here to measure how the multi-modal perturbation influences the margin:
\begin{equation}
\begin{aligned}
|\zeta_j^{(m)}(\boldsymbol{x}^{(1)}, \boldsymbol{x}^{(2)}) - \zeta_j^{(m)}(\boldsymbol{x}'^{(1)}, \boldsymbol{x}'^{(2)})| \le \tau_j^{(m1)}\left\|\left\| \boldsymbol{x}^{(1)} - \boldsymbol{x}'^{(1)} \right\|\right\|_2 + \tau_j^{(m2)}\left\|\left\| \boldsymbol{x}^{(2)} - \boldsymbol{x}'^{(2)} \right\| \right\|_2
\end{aligned}
\end{equation}
where $\tau_j^{(m1)}$ represents the Lipschitz constant of modality $m$ from modality $1$. This constant can reflect how the alteration in modality $1$ influences the margin in modality $m$. Then following the proof in the manuscript, we can observe that:
\begin{equation}
\begin{aligned}
c_j^{(1)} |\zeta_{j}^{(1)}(\boldsymbol{x}^{(1)}, \boldsymbol{x}^{(2)}) - \zeta_{j}^{(1)}(\boldsymbol{x}'^{(1)}, \boldsymbol{x}'^{(2)})|+ c_j^{(2)} |\zeta_{j}^{(2)}(\boldsymbol{x}^{(1)}, \boldsymbol{x}^{(2)}) - \zeta_{j}^{(2)}(\boldsymbol{x}'^{(1)}, \boldsymbol{x}'^{(2)})| \\\\ \leq (c_j^{(1)} \tau_j^{(11)} + c_j^{(2)} \tau_j^{(21)})\left\|\left\|\boldsymbol{x}^{(1)} - \boldsymbol{x}'^{(1)}\right\|\right\|_2 + (c_j^{(1)} \tau_j^{(12)} + c_j^{(2)} \tau_j^{(22)})\left\|\left\|\boldsymbol{x}^{(2)} - \boldsymbol{x}'^{(2)}\right\|\right\|_2 .
\end{aligned}
\end{equation}
Thus, we can obtain the perturbation bound in this setting:
\begin{equation}
\begin{aligned}
& \min_{\boldsymbol{x}'} \left\|\left\|\boldsymbol{x} - \boldsymbol{x}'\right\|\right\|_2
\geq \frac{c_j^{(1)} \zeta_j^{(1)}(\boldsymbol{x}^{(1)}, \boldsymbol{x}^{(2)}) + c_j^{(2)} \zeta_j^{(2)}(\boldsymbol{x}^{(1)}, \boldsymbol{x}^{(2)})+ \beta_j}{\sqrt{(c_j^{(1)} \tau_j^{(11)} + c_j^{(2)} \tau_j^{(21)})^2 +(c_j^{(1)} \tau_j^{(12)} + c_j^{(2)} \tau_j^{(22)})^2 }} \\\\
whe&re \quad j \neq y \quad s.t. \quad c_j^{(1)} \zeta_j^{(1)}(\boldsymbol{x}'^{(1)}, \boldsymbol{x}'^{(2)}) + c_j^{(2)} \zeta_j^{(2)}(\boldsymbol{x}'^{(1)}, \boldsymbol{x}'^{(2)}) + \beta_j = 0.
\end{aligned}
\end{equation}
The idea of the proof is similar to the one in Section A.1 in the Appendix.
Additionally, we present an experiment where our method is applied to intermediate fusion using the widely adopted MultiModal Transfer Module (MMTM) [3] on the Kinetic-Sounds dataset. The results demonstrate that our approach can enhance the robustness of intermediate fusion mechanisms.
| Kinetic-Sounds | Clean | Missing #v | Missing #a | FGM | PGD-$\ell_2$ |
|:--------:|:-------:|:---------:|:---------:|:-------:|:-------:|
| MMTM | 0.6693 | 0.4542 | 0.2028 | 0.3438 | 0.3205 |
| CRMT-MMTM (ours) | 0.6737 | 0.5211 | 0.3169 | 0.3445 | 0.3358 |
[1] Y. Huang, J. Lin, C. Zhou, H. Yang, and L. Huang, “Modality competition: What makes joint training of multi-modal network fail in deep learning?(provably),” *arXiv preprint arXiv:2203.12221*, 2022.
[2] Y. Huang, C. Du, Z. Xue, X. Chen, H. Zhao, and L. Huang, “What makes multi-modal learning better than single (provably),” *Advances in Neural Information Processing Systems*, vol. 34, 2021.
[3] H. R. Vaezi Joze, A. Shaban, M. L. Iuzzolino, and K. Koishida, “Mmtm: Multimodal transfer module for cnn fusion,” in *Conference on Computer Vision and Pattern Recognition* (CVPR), 2020.