Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perception

Summary

This work proposes a solution to train multimodal multi-task training and demonstrate the model trained via this strategy outperforms other approaches in zero-short learning problems. The proposed solution mainly relies on current solutions including AGD, JAX library, DropToken, etc. it seems that this work can be helpful in multimodal related works.

Strengths

1) The proposed training strategy and solution can help related work in multimodal research area. 2) The training strategy demonstrated better performance on zero-short learning approach over other compared works.

Weaknesses

1) If this work focuses on the training strategy, more studies on downstream tasks are needed. 2) Although the work focuses on the training solution, more analysis of multimodal experimental results analysis are needed. For example, the analysis of the improvement over other approaches, especially what kind of cases the new approach help improve the performance?

Questions

1) Could the author demonstrate the major difference compared to AGD? In line 119, the author mentioned that it’s a more generic solution based on AGD, however, not clear about the innovation and difference. 2) How can the proposed approach be applied on these cases where one or two modalities are missing? 3) Will the proposed approach handle the situation like training with several different datasets together? And how to deal with the difference among them? Is there any other additional steps needed?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

N/A

Summary

This paper proposes Integrated Multimodal Perception, which can use image, video, and audio datasets for multimodal and multitask training. The method is scalable, benefiting from the alternating gradient descent as it can alternate diverse modalities, loss functions, and datasets to perform gradient descent. Moreover, using a mixture of experts can help handle different modalities with only a modality-agnostic encoder. The multi-resolution training accelerates the training by using different spatial resolutions, sequence lengths, and batch sizes.

Strengths

1. This paper provides a generic multimodal training solution involving input, model, and objective and successfully trained models with a combination of image, video, and audio datasets. 2. It achieves SOTA zero-shot results on video action recognition. 3. The paper is well-written and easy to follow.

Weaknesses

1. Ablation studies show that alternating between the objectives on each step is better than combining them by summing them. Is this true for all cases or only for large-scale pretraining? Many papers on various topics sum up multiple losses for training. For example, in semi-supervised learning, the loss general consists of two parts: supervised loss and unsupervised loss. We may need to give a proper context for this conclusion. 2. The method's two technical components, alternating gradient descent and mixture of experts, mainly follow previous works. This paper integrates them and applies them to a larger scale of pretraining.

Questions

1. Algorithm 1 shows the sampling uses step t and model state S_t, but Lines 142-143 say that sampling is directly proportional to the number of examples in each dataset. That is, the more samples a task has, the more likely the dataset gets sampled. It would be better to make the algorithm consistent with the descriptions. Moreover, in Algorithm 1, should there be another mini-batch sampling given a chosen task? Should the sampling function f be only for task/dataset sampling? 2. Line 227 mentions filtering examples close to the downstream tasks. How is the filtering conducted? 3. Why the ablation study in Section 4.3 uses spatial resolution 224 while the main results in Table 1 use resolution 256 during training? 4. The paper mentions using JAX in several places, like jax.jit compiling graph-of-interest at runtime, jax.checkpoint saving memory by not checkpointing any of the 152 intermediate model states, and jax.lax.scan accelerating the graph compilation. I'd like to know whether Pytorch can also do so. I know Pytorch also has checkpointing functionality, and Pytorch 2.0 provides torch.compile() API. 5. In Figure 3, a 224x224 image can result in 196 tokens, while a 320x320 image corresponds to 400 tokens. After dropping 50% tokens, it has 200 tokens longers than 196 tokens. Do you use padding to make them have the same length in experiments? 6. Line 237 mentions a patch size of 4x16x16. What does the 4 represent? An image generally has 3 RGB channels. Shouldn't it be 3x16x16? Line 238 says image resolution is 4x256x256, which also confuses me.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The experiments seem not to show whether pretraining with one modality can help improve another modality's downstream performance. For example, whether the image pretraining datasets help improve the downstream video action recognition results and whether using the video pretraining datasets can boost the downstream performance on ImageNet-1K.

Summary

This paper proposes a scalable multimodal multitasking approach. It combines alternating gradient descent and mixture-of-experts to train a unified model. The extensive experiments verify the effectiveness of the proposed method. By scaling up the model, this method sets up a new state-of-the-art in zero-shot video classification.

Strengths

- The proposed method achieves excellent performance in zero-shot video classification. It further confirms improving data and task scalability in multimodal learning is promising. - This paper is well written and easy to understand.

Weaknesses

- Using alternating gradient descent for efficient multimodal multitasking learning is not new, which has been explored in PolyViT. It seems that the proposed method is an extension with more engineering improvement. Also, the success of MoE architecture has been validated on image-text tasks. The contribution of this paper is to incrementally extend this architecture to video and audio modalities. - Compared with other methods, the proposed methods use more training data. Does the improvement mainly come from the scale of the training data? If PolyViT is pretrained on a similar scale of the data, does the proposed method still have advantages? - typo. Line 264, "differ" -> "defer"

Questions

- In Figure 5, the audio modality contributes less to the performance improvement on the image (or video)-text tasks. How about the importance of image (or video) to audio-related tasks?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, the authors have adequately addressed the limitations.

Summary

The paper proposed Integrated Multimodal Perception (IMP) for multi-modal multi-task learning. IMP consists of a shared Mixture-of-Experts (MoE) Encoder as well as modality-specific embedding layers and heads re-projecting representations to modality-specific space. Optimizing towards both supervised classification losses and self-supervised contrastive learning NCE losses, the authors proposed to adopt Alternative Gradient Descent (AGD) during training to accommodate different inputs, outputs and learning objectives without bringing too much memory / computation overhead when incorporating more tasks. Furthermore, a multi-resolution strategy was also proposed to train IMP on large batches of video and audio modalities without losing efficiency or accuracy, where various resolutions, sequence lengths, and batch sizes are used dynamically. Experiments on a good amount of public datasets show that the proposed method can achieve better or competitive performance on various downstream zero-shot classification tasks except audio classification on ESC-50. Ablation studies support some of the design choices.

Strengths

1. The paper introduced a possible way to integrate arbitrary number of modalities into one model. The AGD + unified encoder + MoE design seems an interesting solution without losing efficiency and accuracy when incorporating more modalities **in some situations**. 2. The proposed dynamic resolution strategy is somewhat novel and effective. 3. Experiments on considerably various public datasets demonstrate the effectiveness of the proposed method on zero-shot classification when comparing with previous state-of-the-arts 4. Ablation studies on important model designs were conducted to give a more comprehensive understanding of the methods

Weaknesses

1. One critical problem of IMP is that it seems to suffer from performance drop when incorporating new modalities as per to Fig. 5, possibly due to the shared encoder. Although MoE can alleviate it to some extent in some cases, it is not always working. For example, in Fig. 5, for COCO Text->Image and Image->Text, and K400 zero-shot, IMP can achieve better or similar performance when new modalities of videos are introduced, while IMP-MoE-16Exp obtained worse accuracy. This defect somewhat goes against the major claim of this paper. 2. Another problem is that the model performance on different tasks subjects to change depending on training sample distribution, according to Tab. 1 and Line 261-264. While tuning the distribution of samples of different modalities can improve audio classification, it is not sure whether or not performance on other modality classification will drop, based on Weakness 1. It is also not quite sure how robust the proposed method is against any changes of sample distribution, which may hinder the incorporation of more datasets in the future, going against the major claim again.

Questions

While the paper's major claim/contribution is a new method for multi-modal multi-task learning which can integrate any number of modalities/tasks with arbitrary inputs/outputs/objectives, Weaknesses 1 and 2 are critical and should be addressed before an accept decision can be made, in my opinion. Please address them accordingly.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

N/A

The common problems are what researchers need to solve

I appreciate the authors rebuttal, although it doesn't address any of my concerns. As the authors also agree, what are pointed out in my review are common problems of modern methods and therefore need to addressed properly (I'm not saying "solved") by papers from good conferences like NeurIPS. What the authors are doing in this paper is first downgrading the performance of the baseline model by using a shared encoder and then showing improvements by using existing techniques. Assuming these techniques really work, the authors should apply these techniques to a better baseline model and make some real contribution to the community. This downgrading and improving thing cannot convince me of the merits of the paper. Unless my concerns are well addressed, I will argue for reject.