Summary
The authors point out that while $\textit{dynamic}$ mixup methods are shown to be effective, they induce too much computational cost. To address this issue, they propose a $\textit{static}$ method called Decoupled Mixup, which utilizes the hard mixed samples. The authors suggest that the Softmax function will suppress the model's confidence on hard mixed samples. Based on this idea, the authors propose a decoupled mixup cross-entropy loss which uses a decoupled version of Softmax that ease the "sum-to-one" constraint of Softmax. This loss is then added in the standard mixup loss as a regularization term.
Empirically, the authors show that decoupled mixup improves the top-1 accuracy performance beyond some standard mixup methods on a variety of benchmark datasets. They also show that this method can be generalized with good performance on semi-supervised learning.
Strengths
1. The idea of utilizing hard mixed samples from the perspective of Softmax function is novel and interesting.
2. The theoretical explanation of the effectiveness of the proposed algorithm is solid.
3. The experiments are conducted thoroughly on plenty of tasks, datasets and Mixup methods.
4. Experiments settings are explained in details, especially the configurations of the hyperparameters of the proposed new method DM, leaving great convenience for future practitioners trying to reproduce the work.
5. The improvement of DM beyond standard Mixup methods is empirically shown to be significant. Also, while there isn't much improvement of DM beyond dynamic Mixup, the saved computational costs are also valuable.
Weaknesses
1. The typesettings of some figures and tables are a bit too dense.
2. A few typo.
Questions
1. Line 72. Should the word "conformation" be "confirmation"?
2. Proposition 1, line 133. What does it mean by saying "$\textit{to regress corresponding}\ \lambda$" in the gradient Equation (2)?
3. Line 144, "... for mixed data point $z_{(a,b)}$". Should it be "$x_{(a,b)}$"?
4. Line 147, "$\textbf{Decoupled Softmax}$". The subtitle seems not in the right place. Probably layout error.
5. Line 144-147. From my understanding the text here is to provide the definition of Softmax function and to introduce $\sigma(\cdot)$ as its denotation. I would simply put it in Section 3.1 before Equation (1), along with the descriptions of all other notations.
6. Line 148-150. The text here is basically telling the same story as line 137-140, that Softmax suppresses the confidence of the model on hard mixed samples, the sum of whose semantic information should be more than $1$. I think they can be combined together, rather than having the idea repeated twice in one paragraph.
7. Equation (4), "$\phi(z_{(a,b)})^{i,j}$". Since it is suggested in the previous text that superscripts denote the index, I think here $j$ can be put as a subscript of $\phi$ as an indication of the function, making the expression clearer.
8. Line 153, "the decoupled Softmax makes all items associated with $\lambda$ becomes $-1$ in gradient". Though the proof of this statement should be straightforward, can you provide it in the Appendix as well, since it's mentioned that "the derivatiopn is given in the A.1" while there is only a proof of Equation 1 in A.1?
9. Line 180, "multi-classification task". Should it be "multi-lable classification task" to be more specific?
10. Line 186, "... the unlabeled data with large $\lambda$ ..." . Does it mean unlabeled data with large combination weight? i.e. the $(1-\lambda)$ as in "$\hat{x}_{(a,b)}=\lambda{x_a}+(1-\lambda)u_b$" actually?
11. Equation under line 192. The expression in RHS (particularly $z_{(a,b)}$) is not clear enough to indicate that only the labeled part is retained in $\mathcal{L}_{DM}$.
12. Line 220 "threshold $t$" and line 222 "$\xi$". The notations are used exchanged.
13. Appendix A.1, first line of the equations under line 526. Should the minus sign "$-$" here be the equal sign "$=$"?
14. The idea of raising confidence for hard mixed samples is interesting, but conventionally from the perspective of calibration, one may wish the confidence to not be too large. Will there be a contradict (or a trade-off) between leveraging hard mixed samples and improving the models' calibration performance?
15. In some tasks (datasets), mixup may not necessarily create hard mixed samples, especially when the type of data doesn't have apparent semantic information, for example points on a 2D plane. Also, when manifold intrusion occurs, the true label of a mixed sample may differs from both the labels of the pair of real samples. In these cases, DM may provide no significant improvement, or even degrade the performance. Do you have any insight or principle to determine the "level of necessity" or "effectiveness" of applying DM in a given task and dataset?
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
Not many obvious limitations. The derivation of some theory statement is not complete.