Embedding Space Interpolation Beyond Mini-Batch, Beyond Pairs and Beyond Examples

Summary

The paper introduces a novel data augmentation method called MultiMix, which aims to go beyond the limitations of existing mixup methods. The authors propose interpolating an arbitrarily large number of examples beyond the mini-batch size in the embedding space, rather than along linear segments between pairs of examples. They also extend the method to sequence data with Dense MultiMix, which densely interpolates features and target labels at each spatial location. The paper demonstrates that their proposed solutions outperform state-of-the-art mixup methods on various benchmarks. *Contribution and novelty* The scientific contribution of the paper lies in the introduction of MultiMix and Dense MultiMix, which address the limitations of existing mixup methods. The authors propose novel approaches for generating an arbitrary number of interpolated examples beyond the mini-batch size, both in the embedding space and for sequence data. The empirical results, as described in the additional explanations, validate the effectiveness of these methods. Overall, this paper seems to improve the research frontier on data augmentation, but empirically and gives some new intuitive understanding of their results as their method leads to a more uniform distribution of embeddings within each class. I do not see any clear weaknesses in the methods that may challenge the main claim of this work, but I have raised some issues in the questions section. Finally, since I have not read some of the papers that are cited in this work or other related papers in the literature, I defer the judgment about novelty of the ideas presented here to other reviewers.

Strengths

- Novelty: The paper introduces a new data augmentation method, MultiMix, that extends the concept of mixup beyond the limitations of existing methods. The idea of interpolating in the embedding space and densely interpolating features and labels in sequence data is original and contributes to the field of data augmentation in machine learning. - Empirical Results: The authors provide empirical evidence to support the effectiveness of their proposed solutions. They demonstrate significant improvements over state-of-the-art mixup methods on multiple benchmarks. The results validate the scientific contribution of the work and highlight its practical relevance. - High quality of presentation of results: the presentation of the results follows a simple and logical flow, with main contributions and their difference to the previsous works clearly stated. The Methods, notations, & results are all clearly stated and easy to understand for the reader. - Intuitions about results: The paper provides some justifications & intuitions about why their approach works. They do so by analyzing the embedding space. The authors show that the classes become more tightly clustered and uniformly spread over the embedding space, explaining the improved behavior of their solutions. This analysis adds depth to the scientific contribution of the work.

Weaknesses

Since I have not read some of the papers that are cited in this work or other related papers in the literature, I defer the judgment about novelty of the ideas presented here to other reviewers. Besides that, I have raised a few issues in the questions section as they are not direct criticism of this work.

Questions

I have the following questions from the authors: - There is a lot of emphasis in the paper about the importance of number of loss terms and its importance for obtaining the results reported in this paper. However, one could achieve similarly high number of high loss terms by only incorporating several steps of pairwise mixup of the embeddings. This will have both random-ness in selection of pairs and value of $\lambda$ and will be equivalent to imposing a sparsity on $\lambda$'s that are sampled in Multimix. Is this point addressed in your evaluations or perhaps some cited works? In other words, what is exactly the benefit of having a dense $\lambda$ if doing mixup in the embedding space? - It seems that the main benefits of the proposed approach is that it imposes new restrictions on the loss such that the solutions have uniform convex hulls per each input class. If authors agree with this statement, can they think of an alternative approach of achieving this? As a hypothetical experiment, can they think of an adding an explicit term in the loss that will encourage this uniform convex hulls, ie, some kind of a uniform convex hull regularisation term?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

yes

Summary

This paper considers a generalized form of Manifold mixup by expanding the concept of a pair-wise interpolation method to the multi-sample-based interpolation on the entire convex hull. The proposed methods called MultiMix and Dense MultiMix show the best performance over prior mixup-based methods for the various classification benchmarks. Also, they are empirically shown to have better robustness than prior methods.

Strengths

$\textbf{Strength 1:}$ The main strength of this paper is the suggestion of the general form of the prior mixup-based methods to cover the entire convex hull which can be formed by a given set of training samples. Because the power of the mixup methods originated from the interpolation of the space, i.e., either input or embedding, in-between different samples, expanding the concept of mixup into the entire convex hull of the given samples are clear and convincing direction. $\textbf{Strength 2:}$ The paper is well-written and easy to understand. To be specific, Fig. 1 is simple yet effective to understand the main strength of the proposed method. Also, the formulations in the paper are quite neat and well-organized. $\textbf{Strength 3:}$ The experiments are done for various kinds of architectures ranging from the ResNet family to ViT, and benchmarks including CIFAR and ImageNet cases.

Weaknesses

$\textbf{Weakness 1:}$ The most crucial weakness of this paper is the novelty of the proposed method. Specifically, I admit that the expansion of the sample interpolation method to the generalized form, i.e., MultiMix, is convincing and the authors show a well-organized formulation of it, but I also feel that the expansion of the concept is quite easy to be anticipated from the prior Mixup methods. For example, in the original Mixup paper, the authors already tried to utilize multiple samples by weighting them with a simplex coefficient in the input space. Although it is reported not to show a further gain, it is hard to find a noticeable difference between the `trial' of the original Mixup paper and the proposed method except for the target space, i.e., input for Mixup vs. embedding space for MultiMix. When focusing on the difference, i.e., mixing on the embedding space, it is natural to ask for the reason why the multi-sample-based mixup becomes effective in the embedding space rather than the input space. However, I cannot find a deep insight into the phenomenon. $\textbf{Weakness 2:}$ A more precise comparison in the Related Work part would be beneficial to highlight the significance of the proposed method. To be specific, in Related Work, a bunch of works are grouped into their attributes and briefly mentioned. The authors drop the detailed explanation of the prior works with a suggested survey paper but I am sure that the authors must provide a compact but clear explanation of the related works. As a simple suggestion, mentioning the names of prior mixup methods along with short descriptions would make it more clear. When a set of important baselines including AlignMix, Co-Mixup, PuzzleMix, and $\zeta$-Mixup are called with names in the Related Work part, readers can easily match the name of the priors appeared in Experiments and the descriptions in Related Work. $\textbf{Weakness 3:}$ The metrics in 4.4 Quantitative analysis, i.e., 'Alignment' and 'Uniformity' do not fully represent the quality of the embedding space. These measurements can evaluate the similarity of embedded features of intra-class samples but do not capture the separation between different classes, i.e., the separability of inter-class distributions. $textbf{Weakness 4:}$ Manifold intrusion is the main challenge of mixup-based methods. However, I cannot find any analysis for the effect of manifold intrusion of MultiMix. The experimental gains empirically show that the effect of manifold intrusion is minimal in practice. However, further discussion is needed on the effect of manifold intrusion of the multi-sample-based mixup approaches. I wonder if the chance of the intrusion might be higher than the pair-wise mixup of prior works.

Questions

$\textbf{Question 1:}$ Would you provide a further explanation for the reason why the multi-sample mixup becomes when the interpolation is done in the deeper layer? Again, I felt that the main significance of this work is the finding of the effectiveness of multi-sample-based mixup on deeper layers. As mentioned in this paper, the authors of Mixup and SuperMix had tried to interpolate multiple samples but they are limited in the input space which leads to minimal performance gain. If the novelty becomes more clear and strong after the rebuttal, I will change the rating of the paper. $\textbf{Question 2: }$ According to Weakness 3, the metrics used in Section 4.4 seem not to capture the separability between different classes. I think that when the 'Alignment' and 'Uniformity' values are divided into the similarities between different classes, they eventually consider the compactness and separability of the embedding. To be specific, when considering class $k$, we can measure the distance between the per-class averaged feature vector of class $k$ and the per-class averaged feature of the nearest interfering class. When we divide `Alignment' with the distance to the nearest different class, then the value will consider both the similarity between intra-class features and the dissimilarity between inter-class features. What happens when we compute the modified 'Alignment' and 'Uniformity'? $\textbf{Question 3: }$ Would you provide a computational cost of the proposed methods and the comparison with other baselines? The results of the training speed guarantee that the proposed method does not suffer from further latency in practice, but I wonder if MultiMix and Dense MultiMix show a similar level of computational cost to the baselines. $\textbf{Question 4: }$ Would you provide a discussion of the thought of manifold intrusion of the proposed methods? I imagine that the multi-sample-based mixup tries to cover the entire convex hull so it probably causes manifold intrusion more frequently.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Weakness 1 and Question 1 are for pointing out the limitation of the proposed work in view of novelty. Also, I provide a suggestion for improving Related Work in Weakness 2. As better metrics for quantitative analysis in section 4.4, I provide the modified measurements in Weakness 3 and Question 2. For emphasizing the computational efficiency of MultiMix and Dense MultiMix, providing the computational costs would be beneficial (as aforementioned in Question 3). For the manifold intrusion issue, I described the limitation and the corresponding question in Weakness 4 and Question 4.

Summary

The paper aims to increase the performance and robustness of deep learning model by increasing sample diversity. The authors propose data augmentation techniques which interpolate samples from a convex hull. The experimental evaluations show that the algorithm outperforms existing techniques. **I have read the rebuttals. Please see the comments below for the discussion.**

Strengths

1. Intuitive idea: the use of convex hull is natural and simple. 2. Applicability: the algorithm can be generally applied in many settings.

Weaknesses

1. Presentation: the presentation of the paper needs significant improvement. For example, the abstract should succinctly summarizes the key ideas of the paper. 2. Novelty: sampling from a convex hull is not novel. The use of Dirichlet distribution has also been done in many other works. Could you please compare your work with the work by Shu et al. ? [1] Shu, Y., Cao, Z., Wang, C., Wang, J., & Long, M. (2021). Open domain generalization with domain-augmented meta-learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9624-9633).

Questions

1. It would be nice to see the analysis on larger/multi-modal models. I am wondering if the gain will be more significant since the sampling diversity will be crucial there. For example, could you conduct experiments on larger Vision Transformer models or language-augmented vision models such as CLIP? 2. In the ablation study, the performance seems to be sensitive to the hyper-parameters selected, especially since the performance difference from the second best algorithm is between 1-2%. Could you expand more on how the parameters are chosen for each experiment?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The authors discuss the limitation of the algorithm in the concluding paragraph.

Summary

The authors introduce MultiMix, an approach aimed at enhancing the Mixup class of techniques, which involve interpolating between input examples and their corresponding targets. This enhancement is achieved through the interpolation of a substantially more training examples, increased number of loss terms for each minibatch, and a more flexible interpolation factors. Further, the authors expand their proposed method with DenseMultiMix, a variation capable of interpolating examples in sequential data. Empirical evidence indicates that the proposed MultiMix and DenseMultiMix not only boost accuracy but also bolster the robustness metric across a diverse set of tasks, including but not limited to classification and object detection.

Strengths

There's a noticeable diminishing return when interpolating additional input examples in Mixup. This presents a limitation to the advancement of Mixup-related methodologies, which focus on interpolating more training examples. However, the approach introduced in this paper appears to overcome this limitation by implementing interpolation in the embedding space, as opposed to the input space, evidenced by Fig 4c. I regard this as a significant contribution of this paper. Furthermore, the adaptation of Mixup to sequential data like text has always been frustrated. There are three primary obstacles: 1. The blending of text tokens doesn't carry the same significance as mixed images. 2. How to interpolate along the temporal dimension or the spatial dimension. 3. Discrepancies in length among examples within a mini-batch. The introduced MultiMixup provides solutions to at least the first two challenges. Interpolating within the embedding space prevents the creation of nonsensical input tokens, while MultiMixup along with attention delivers an effective means of interpolating representations in the spatial dimension.

Weaknesses

There are several weaknesses in this manualscript, as explaint in the following. 1. The motivation for the suggested method is insufficiently compelling within the manuscript. The only statement regarding to the motivation is ```In this work, we argue that a data augmentation process should increase the data seen by the model, or at least by its last few layers, as much as possible. ``` This reasoning seems to be superficial as it fails to distinguish itself from the preceding data augmentation methods. Moreover, it doesn't elaborate on how the proposed approach enhances accuracy and robustness. More explicitly, while it's true that any data augmentation method would increase the volume of data processed by the model, not all such methods that do so effectively improve the model performance. Thus, the motivation fails to pinpoint the core of the research problem. 2. Regarding the empirical evaluation, it would be appropriate for the authors to include comparison with methodologies such as AugMix [1]. Similar to the proposed method, AugMix enhances the volume of data points processed by the model through interpolating examples derived from varying combinations of image pre-processors using the Dirichlet distribution. Moreover, it offers a substantial gain in both accuracy and robustness metrics. Also, the divergence loss could potentially counter the scarcity of dense labels. [1]: Hendrycks, Dan et al. “AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty.” ICLR 2020. 3. The performance improvement over the established baseline is rather modest. In such a scenario, it would be beneficial if the manuscript offered more profound insights to the community to better understand Mixup-oriented techniques. Such insights could include how the proposed method is connected to existing methods like AugMix, as well as an exploration of why the proposed methodology shows greater promise. For instance, while Section 4.4 and Figure 6 in the appendix provide useful information, they lack a comparison between the proposed method and existing methods, as well as a comparison among different variants of MultiMix as done in Section 4.5. Empirical insights of this would offer more compelling evidence that the proposed methodology is a valuable contribution to the community. 4. A less significant shortcoming pertains to the scope of the empirical evaluation. The batch size is held to be 128 in the domain of image classification. Understanding the robustness of the proposed method to batch size selection is crucial, especially as we move towards models with more # parameters where batch size tends to decrease. Specifically, when the batch size is small, MultiMix should offer more advantages over standard Mixup due to its ability to draw from a larger pool of interpolated examples. Another dimension worth exploring is the application of DenseMultiMix in the realm of text data. The manuscript would significantly benefit from showing evidence that the proposed method can address some of the challenges of extending Mixup-style techniques to the domain of text. I understand that interpolating dense text labels is a non-trivial task, preliminary studies on tasks such as text classification could suffice.

Questions

Please see the weakness section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

One limitation as mentioned above is the application of DenseMultiMix in the realm of text data.

Follow-up response to R-Hzfq

Thank you for your feedback. We sincerely appreciate your time and effort in evaluating our work. We are glad to see your rating increase to borderline reject. Since your assessment of our work is still negative, we would like to kindly ask what concerns still remain. This would give us the chance to provide further feedback if needed. We would also like to point out that the **Author Rebuttal by Authors** contains further clarifications on the contributions and motivation of our work.

Check elements of the rebuttal

Dear Authors, Many thanks for your rebuttal. To make some progress in the discussion, could you please confirm the values of the table with CIFAR100-C and ImageNet-C results? Right now, they seem to indicate that `*MultiMix` would worsen the performance (which appears to be in contradiction with the text, if I am not mistaken) The AC of the paper.