Summary
This works is situated in the field of **robust generalization**; in particular it studies the problem of how models that were trained on noisy or imbalanced data perform on clean data. They achieve this via **online batch selection**, and in particular they contribute to the development of a **Bayesian framework** for searching for the best datapoints to use in each mini-batch among some canditate points sampled from the training set. In more details, they extend a SotA method, called **RHO-LOSS** [1]. The authors pinpoint some approximation choices that [1] makes in order to make the principled Bayesian approach to data selection practical. As a result, they design an algorithm which, in contrast to [1], they claim 1) to make **less crude approximations about predictive posteriors**, by using variational Laplacian approximations, MC sampling (and other) instead of pure point estimates, and 2) **not to rely on the existence of oracle clean data** during the training of their model. Their experiments demonstrate **improvements in robust generalization and training acceleration for image classification tasks** over other methods in the literature (incl. [1]).
[1] Mindermann, Sören, et al. "Prioritized training on points that are learnable, worth learning, and not yet learnt." International Conference on Machine Learning. PMLR, 2022.
Strengths
The authors contribute to an important direction for robust training methods, in particular developping a bit further the Bayesian data selection framework.
1. The paper is well-written and motivated. The derivations are easy to follow and the deferrals to the appendix are used appropriately. Presentation can further be improved if there was somewhere a paragraph (perhaps in the appendix) with the list of approximation steps taken (for future reference), e.g. MC sample of a variational posterior, based on Laplace approximation of the last layer of a neural network (for which the backbone is taken as a point estimate), and the Laplace approximation further does not use the true Gaussian, but a Gauss-Newton matrix + potentially a KFAC approximation of it.
2. Authors identify correctly potential improvement points of [1], and proceed in mitigating them. In this way, they introduce a novel approximation scheme to the Bayesian framework of data selection, which is effective and computationally-lightweight.
3. Experiments seem to advocate in favor of using their method over [1] or other methods, they are extensive and on various dataset scales in image classification tasks.
Weaknesses
1. Proposed method overly claims that it does not use oracle clean data, however it might still depend on clean data via the unsupervised pre-trained zero-shot classification proxy that they use. In particular, CLIP-R50 might have been trained in a superset of datasets (like ImageNet) which contains CIFAR10/100. In that case, information from oracle clean data has been stored in the pretrained model. While I believe that this is a weaker assumption, there needs to be an explicit statement of this assumption and a quantitative assessment of this potential pitfall. To which extent would a pretrained model on the same noisy/imbalanced dataset as the benchmark task be useful? Nonetheless, the authors acknowledge this potential limitation of an underperforming pretrained model at the last section. The following (lightweight) experiments can be further performed to augment their arguments or awareness of the limitation:
- A zero-shot CLIP baseline is provided, I think they should also provide with a finetuned version using linear probing and training on imbalanced/noisy CIFAR10/100 with uniform sampling. This way it will be more clear that the improvement are more due to their method and not because of an overpowered pretrained model.
- Consider CIFAR10/100 experiments using an unsupervised pretrained model, with linear probing or kNN (so that it is zero-shot if you want that) for the oracle model, via for instance some simple SSL method like MoCov2 [2] on the same noisy/imbalanced training set.
I will increase the assessed score if such experiments are performed and reported.
2. Ablation study reveals sensitivity to some hyperparameters. As a result, model selection and hyperparameter tuning are important for the success of the method. How was model selection performed? Was the validation split iid to the training set or was it a clean/balanced dataset? This needs to be reported clearly, but not addressed in this paper.
3. It would be nice to have a more extensive ablation study on the effects of the approximation choices in terms of final test accuracy, spanning from crude approximation schemes to the one finally used by the authors.
[2] Chen, Xinlei, et al. "Improved baselines with momentum contrastive learning." arXiv preprint arXiv:2003.04297 (2020).
Questions
### Questions
* (related to weakness 2) How was model selection performed?
* How were target accuracies chosen for training acceleration results?
* Lines 151-152: What do we miss by replacing the Hessian matrix in the Laplacian approximation by the Gauss-Newton?
* Lines 131-132 about “the recent trend of exploring the potential of pretrained models” (in robust generalization) needs some citations.
* What does Table 4 in the Appendix refer to? Please complete the appendix to explain.
### Typos
* y-axis on Fig.2,3 and 4 are not in % of test accuracy as indicated.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The authors derive a novel approximation scheme to Bayesian data selection for training robust models (Strength 2), with favorable results over past literature in image classifiacation tasks (Strength 3) and the writing is overall great (Strength 1). There is a concern about the applicability of the method in cases where a pretrained model does not exist for the task at hand, in which case we need to find another proxy or probably pretrain it (Weakness 1), and how much actually the method makes no use of priviledged clean/balanced datasets for training (Weakness 1) and validation (Weakness 2). The limitation about the pretrained model is mentioned briefly, but it would be appreciated if it is expanded and experimented on with the ablation studies suggested in the Weakness section above.