Summary
This paper considers the problem of mixup generation and classification in a discriminative-contrastive setting where the discriminative part considers mixing samples from different classes while the contrastive part considers instance level mixup. The paper argues that such a setup can be treated as a optimizing for local smoothness versus global discrimination, respectively; however has two shortcomings: i) it is prone to trivial solutions and ii) lower empirical performances in self-supervised learning situations. To this end, the paper suggests scenario-agnostic mix (SAMix) that implements three ideas: a) learns the mixing ratio \lambda, b) avoids trivial solutions via non-linear content modeling (eq. 9), and c) expensive compute for online training via using pre-trained models. Experiments are provided on 12 tasks and demonstrate some promise.
Strengths
The paper is very poorly written and thus it is hard for me to judge the technical strength or its contributions. However, the empirical evidence reported in the paper shows some promise in some situations. For example, while the performance improvements are very minor (less than 0.5%) in Tables 1,2,3,4, when using pre-trained models (Tables 5,6,7,8) they are slightly higher (1-2%).
Weaknesses
The paper needs a thorough revision for clarity and technical correctness, and preciseness in the assumptions it is making. Some suggestions are provided below to consider when revising.
1. It is unclear to me what exactly is meant by "scenario" in the paper? Is it class or instances? Where exactly is the "agnostic" part coming in?
2. The main hypothesis in the paper is that mixup generation can be treated as a sum of local and global terms (Sec. 3.1). But it appears that which one is local and which is global is not quite clear from the exposition. For example, the paper writes "the local term centers around classes, while global term is across classes". Then writes "\ell^CE is a global term" and then writes "at the class level, to emphasize the local term, we introduce binary cross-entropy loss pBCE". Further, in Eq. 4, the paper assume access to the labels y_i and y_j, and so does for the NCE loss in Eq. 5, which is supposed to be label free (as described earlier in the Preliminaries)?
3. In Sec 3.2, Eq. 7 is claimed to be an adaptive mixing factor \lambda, but it is not about updating lambda. Further, the findings in Sec 3.2, which is essentially the main contributions of the paper, the observations are only empirical and the insights are not substantiated in any rigorous manner. The fixes, such as adaptive \lambda, non-linear content modeling, etc. are only treated superficially and it is not clear to me why it makes sense or how is the particular formulation justified. For example, the paper writes "it may be unstable to project high-dimensional features to 1-dim linearly", which the paper claims to be the reason for trivial solutions, and the offering is to use two 1x1 convolution layers with a batch regularization and ReLU.
Other comments:
a. f_\theta(x): x->z. Typically, x and z are replaced by their domains not their variables. Also, the domain of z is not defined. Is it d_c? Do you assume each class has a separate dimension?
b. "closed-loop framework": how is it closed loop?
c. Please double check Eq. 1; it is incorrect. The lambdas should go inside \ell^CE I believe.
d. Why do you use the same \lambda for \ell^NCE and \ell^CE (e.g. in eq.1 and eq. 2), when they look at instance level and class level losses? For that matter, what precisely is meant by parametric loss and non-parametric loss for \ell^CE and \ell^NCE, respectively? When adding these two losses in the same objective (e.g., Eq. 3), how do you ensure it is optimized correctly, as NCE loss I believe do not assume knowledge of the class labels, while CE loss discriminates classes? The term L^{cls} is undefined. What is the precise form for M_\phi, and how is z_i, z_j used?
e. "SSL framework can easily fall into a trivial solution". This is a strong statement and is mostly an empirical observation. It would useful to provide a more technical insight to the reason for this.
Overall, this paper needs a thorough revision for language and technical clarity. As is, the contributions appear as minor hacks to improve performance.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.