Efficient Adversarial Contrastive Learning via Robustness-Aware Coreset Selection

Summary

This paper focuses on the problem of efficient adversarial contrastive learning, the authors propose Robustness-Aware Coreset selection to speed up ACL, and according to the theoretical analysis and experimental results, the proposed framework is effective while does not hurt the performance.

Strengths

1. The topic is interesting and the proposed method is simple yet effective; 2. The experiments are well-designed and solid. Also, several runs are performed on each task, making it more convincing; 3. The writing is easy to follow; 4. The proofs are well-written and sound.

Weaknesses

1. According to the experimental results, it still takes much time to run the proposed algorithm. While the authors discussed it in the limitations, it might a challenge for it to be put into practical use; 2. Figure 2,3,4 are a bit hard to read, might need some re-work;

Questions

Please refer to the previous section

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The authors discussed limitations in the paper appendix.

Summary

This paper introduces a robustness-aware coreset selection (RCS) method without requiring label information to speed up adversarial contrastive learning. RCS selects an informative training subset that minimizes the representational divergence (RD) between adversarial and natural data. Theoretically, the authors prove a greedy search algorithm can solve a proxy problem and provide the lower bound of the solution. Empirically, comprehensive experimental results demonstrate RCS can significantly speed up ACL while maintaining robustness transferability. To my knowledge, it is the first effort to apply ACL on the large-scale datasets ImageNet-1K and obtain effective robust representations.

Strengths

(1) The proposed method is reasonable and novel. The authors propose to select subsets guided by the RD between natural data and its adversarial variant. RCS does not need label annotations while the existing related work requires labels during coreset selection, which supports the originality of the proposed method. This paper is the first to obtain robust representations by ACL pre-training on ImageNet-1K efficiently via RCS. I think this paper has adequately cited the related work. (2) The submission is technically sound. The claims are well supported by theoretical analyses and experimental results. Theoretically, the authors prove solving a proxy problem efficiently via the greedy search can guarantee the optimality of the solution for the original problem. Empirically, the authors apply RCS to speed up ACL and its variant DynACL on various datasets and show that RCS can maintain natural and robust test accuracy on various downstream tasks. Besides, the authors provide extensive results that validate RCS can be applied to accelerate supervised adversarial training including Madry’s and TRADES on CIFAR-10 and ImageNet-1K. (3) Self-supervised robust pre-training can provide improved robustness transferability without requiring label annotations. However, due to the computational prohibition, ACL methods have not been applied to large-scale datasets previously. This paper solves this important issue and enabled ACL to be conducted on large-scale datasets, tackling a very practical meaningful chanllenge. (4) This paper is well-organized and well-written. The reviewer can easily follow most of the content.

Weaknesses

Minor comment - although the authors propose to use the greedy search algorithm to efficiently search the coreset, it still needs to consume extra time for CS during robust pre-training. How to further improve the efficiency of ACL and maintain its effectiveness should be an interesting future direction.

Questions

See weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The limitations and the possible negative societal impacts of this submission have been adequately discussed by the authors in the appendix.

Summary

This paper proposes a robustness-aware coreset selection (RCS) method, which is applied to accelerate adversarial contrast learning (ACL) in the absence of labeling information. Especially, the coreset searched by RCS minimizes the representation difference between the natural data and their adversarial examples, which is achieved by a greedy search method. And experimental results demonstrate that RCS can indeed speed up ACL without significantly compromising the robustness transferability.

Strengths

1. This paper is well-written and easy to follow. 2. The coreset searched by RCS is not only small in number but also beneficial in improving the adversarial robustness of representations. 3. Experimental results demonstrate that RCS can indeed speed up ACL without significantly compromising the robustness transferability.

Weaknesses

1. This paper points out that ACL with RCS trains the model on the previously selected coreset for $\lambda$ epochs, and for every $\lambda$ epochs a new coreset is selected. So, does the value of $\lambda$ affect the effectiveness as well as the efficiency of ACLs? It may be worthwhile for the authors to make further discussion. 2. In the experimental part, it can be found that the coreset searched by RCS without label information achieves better performance compared to the coreset searched by ACS, or even the whole dataset, as shown in Tables 12 and 13. How to explain this interesting phenomenon? 3. The core of this paper is the coreset found by RCS, so the authors should have given a more focused discussion, such as whether it is class-balanced and what the distribution of the subset is.

Questions

Please refer to the weaknesses part.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please refer to the weaknesses part.

Summary

This paper proposes a coreset selection for efficient adversarial self-supervised learning. By selecting a coreset every epoch that can minimize the representation divergence for training, it maintains similar robustness performance despite a learning speed that is more than three times faster.

Strengths

- It is technically sound. The fact that adversarial self-supervised learning (ASL) can be trained more than three times faster using coreset selection is especially remarkable. - The proposed claims are well supported through theoretical analysis and experiments.

Weaknesses

- The biggest weakness of the proposed method is that there isn't a significant difference compared to random selection. It utilizes more computation than random selection, but the gain in performance is small. It would be nice if the difference in computation compared to random selection could be explained. - Moreover, the originality of the proposed method seems more like a simple application of applying the RD loss to previous work [1,2] rather than a new combination. If the author could explain this more clearly, I would be willing to revise my score. - Furthermore, the one-step gradient approximation, warmup, last layer gradients, and adversarial example data approximation also seem to be techniques proposed in previous work, so there seems to be no originality in this regard. - There seems to be a lack of justification that a coreset with a small representational divergence is sufficient to gain robustness. And there's not enough explanation on how the results can achieve comparable robustness. - It would be helpful if there were explanations, perhaps utilizing representation visualization, as to what samples become the coreset selection, and why it helps with robustness. [1] Kilamsetty et al., RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learning [2] Kilamsetty et al., Glister: Generalization based data subset selection for efficient and robust learning. ==After rebuttal I changed my score to 6.

Questions

None

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have well described their limitations.