Summary
The paper presents a novel regularization approach tailored for multimodal/heterogenous data fusion, which can be applied off-the-shelf to modern neural network architectures. The approach, namely RegBN, targets the reduction/removal of confounding factors and partial dependencies that are usually present among features extracted from different modalities. To do so, RegBN is implemented as a regularization technique that promotes feature independence across data modalities. The regularization is iteratively optimized over mini-batches using a separate solver not relying on back propagation and gradient descent. This separation is critical as otherwise the network could simply learn to use the confounding factors instead of removing them. Experiments are performed on several multimodal datasets based on different modalities and spanning across different applications, such as multimedia, affective computing, healthcare diagnosis, and robotics. Moreover, a synthetic dataset is designed to further highlight the impact of RegBN. The results showcase a positive impact in all the test cases.
Strengths
1. The approach is novel and grounded with theoretical motivation
2. The method can be plugged off-the-shelf to most multimodal architectures, possibly leading to vaste adoption
3. Wide breadth of experiments that explore several multimodal scenarios
4. Positive results in all scenarios
Weaknesses
1. No integration with recent widely adopted multimodal foundation models, such as CLIP, ImageBind, and many others. Despite I understand that RegBN purpose contrasts with the multimodal alignment objective driving those models, I believe it would be valuable to bring them to the discussion. It would also be beneficial for the positioning of the paper.
2. The boost is sometimes minor, e.g see Table 2, 5, I.3, and in some settings even slightly negative, e.g., see Table 5 and I.4 (video fusion rows). Also, it is unclear the high difference between AV-MNIST results reported in the main paper (~ 99% accuracy) and the ones reported in table I.3 (~71% accuracy). Furthermore, why for the experiments on LLP, training metrics are reported in Table 2 and validation metrics in the supplementary?
3. As RegBN requires a separate optimization, its integration affects the time complexity of the model training. The Authors provides multiple examples on how RegBN could/should be integrated in existing architectures, see Figure I.1, however, in most cases a single RegBN layer is added to the network. It is unclear whether this choice is suboptimal for performance in favor efficiency, or whether it is useless to have multiple RegBN as in I.1c.
4. Minor:
1. The writing is for sure enjoyable, however there are some parts that could have better explained / could be less ambiguous. For instance, in section 3: _"Ideally, the residual layer ... RegBN minimizes the linear relationship between these layers via: equation (2)"_, but then equation (2) does only contain $f$ and $g$, and thus it is completely tacit the key insight on why solving equation(2) would provide $f_r$ orthogonal to $g$, as well as it is not explicit that $f_r$ corresponds to the normalization of $f$. Moreover, Figure 1 provides a general idea of the impact of RegBN, but it could have been more aligned with the formulation, e.g., by adding $f$, $f_r$, and $g$; and in general the caption could be expanded to better describe it — what does the grey band represent?. Finally the positioning could have been more clear by stating why recent multimodal foundation models like CLIP are out of the scope.
2. “multimodAl” often mistyped as “multimodEl”; Table I.5, 3rd column, video fusion, (70.6) vs (81.4) seems a bit suspicious; Color gradient is not explained in Figure I.9; wrong x axis label in Figure I.10;
Questions
1. Do the Authors envision any use/application of RegBN for large foundation models, e.g., is it meant for whenever they are employed for downstream tasks?
2. What could be the reason why RegBN affects negatively performance? Is it due to the fact that when evaluating some dataset in-distribution learning spurious correlation might provide a boost? Please comment also on the last two sentences of weakness # 2.
3. Is it enough to apply RegBN once as shown in I.1a,b,d or might be worth to apply it to several layers?
4. Consider slightly modifying sec. 3 and Figure 1, as well as adding somewhere (introduction or related work) a few positioning sentences wrt foundation models. Fix other typos/mistakes.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
Societal impact is not discussed. I believe that RegBN could provide a benefit in reducing stereotypical biases that currently affect multimodal models.