Summary
The paper studies popular mixup strategies across different datasets and settings and finds that dynamic mixup strategies like SAMix are better across the spectrum than static ones like the original mixup. They also introduce a Python framework named OpenMixUp to promote full reproducibility for their findings and also to allow the community to easily experiment with mixup strategies.
Strengths
1. Extensive pool of mixup strategies, including both static and dynamic ones (18 in total).
2. Going beyond standard classification benchmarks by including experiments on transfer learning (object detection, semantic segmentation).
3. An open discussion of their philosophy for implementing the OpenMixUp project, detailing the technical aspects.
Weaknesses
1. Small datasets, not representative of the scale provided by current open datasets.
2. Architectural ablations are not very comprehensive to establish trends.
3. Many pieces of important information missing from the main text (such as results on non-classification benchmarks).
Questions
1. I don't find strong motivation to do MixUp any longer when the community has already started moving away from self-supervised learning and towards (perhaps more scalable) approaches such as semi-supervised and weakly-supervised learning. I think this is especially true in the emergence of vision-language models across all kinds of scaling regimes. Maybe the authors could make a more compelling argument in the abstract or the introduction, discussing potential applications or benefits of mixup that are still valuable in current research directions.
2. The importance of MixUp isn't that evident in the area of vision-language models. Even when we're in the visual representation learning territory, with the emergence of models like Cambrian [1], the multimodal aspects of this paradigm cannot be neglected. In this paradigm, there hasn't been any benefit of using MixUp.
3. The datasets (such as COYO [2], MetaCLIP [3]) used in the study are not very representative of the scale current open datasets provide. It would have been better to repurpose at least one large-scale dataset (even if trimmed down) to study if the observations transfer well.
4. Most modern backbones and methods (ConvNeXt-v2 [4], HIERA [5], for example) employ MixUp during the fine-tuning stage. So, this could have been studied in this paper, i.e., take a pre-trained backbone and evaluate the mixup strategies.
5. Authors studied different transfer learning scenarios in their setup, including semantic segmentation and object detection. This could be made clearer in the main text of the paper because, from the main text, it's hard to know if the authors go beyond non-classification tasks.
6. L097 - L101 Per-step training latency is another important factor to consider. So, a better trade-off could be to include top-1 accuracy, memory consumption, and time. This gives a more complete picture of the gains/losses. I do see this being reflected in Figure 4. Perhaps the authors could make this more explicit to the readers, letting them know that this trade-off is available already.
7. L101 - L102 Do the static methods perform equally worse on other visual representation learning tasks? VTAB [6] is a representative benchmark that helps to evaluate the effectiveness of a method more completely. I think we need to consider this aspect here because it might so happen that a static method may perform poorly on one task but may perform better on the other. So, comparing the performance of different methods on an established benchmark (like VTAB) provides better signals.
8. MixUp is a training-only technique applied at the data-processing level. Why does OpenMixUp have to have a module for model architectures?
9. From Figure 4, it's not clear what defines the size of the circle markers.
10. When doing the ablations on architectures, I think a better way would have been to first group the architectures with respect to model families and then perform ablations. For example, ResNet [7], ResNeXT [8], could go to native CNN-based architectures. ConvNeXt [9] could go to the modern CNN family with transformer-alike components (like activations) and so on. In my opinion, this could have been followed in the presentation of the results.
References
[1] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs; Tong et al.; 2024.
[2] COYO-700M: Large-scale Image-Text Pair Dataset.
[3] Demystifying CLIP Data; Xu et al.; 2023.
[4] ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders; Woo et al.; 2023.
[5] Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles; Ryali et al.; 2023.
[6] A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark; Zhai et al.; 2019.
[7] Deep Residual Learning for Image Recognition; He et al.; 2015.
[8] Aggregated Residual Transformations for Deep Neural Networks; Xie et al.; 2016.