Beyond Efficiency: Molecular Data Pruning for Enhanced Generalization

Summary

The paper presents a new framework MolPeg aimed at improving the efficiency and generalization of training models in molecular tasks using pretrained models. MolPeg introduces a novel DP technique that maintains two models with different update paces during training, leveraging the loss discrepancy between these models to score and select the most informative samples. This approach is applied in a source-free data pruning scenario, which does not require access to the original pretraining data. Extensive experiments across four molecular tasks demonstrate the effectiveness of MolPeg method.

Strengths

1. The paper introduces a novel approach to data pruning by leveraging pretrained models and a loss discrepancy scoring mechanism. 2. The paper is well-organized, with a clear presentation of the problem, the proposed solution, and the experimental results. 3. The experiments are comprehensive, covering multiple datasets and tasks, and provide empirical evidence supporting the efficacy of MolPeg.

Weaknesses

1. The effectiveness of MolPeg is highly dependent on the quality and suitability of the pretrained models utilized. If appropriate pretrained models are unavailable, the benefits of MolPeg may not be fully realized. 2. The proposed framework requires the maintenance and simultaneous updating of two models, which could increase both implementation complexity and computational overhead compared to simpler pruning methods. 3. Most DP methods demonstrate their effectiveness on widely-used datasets such as ImageNet. However, molecular datasets represent a more specialized field. It is crucial to test MolPeg with mainstream validation datasets to further demonstrate its effectiveness.

Questions

Please refer to weakness 3, why not validate the effectiveness of MolPeg with mainstream datasets? Molecular datasets belong to a more specialized field and seem less persuasive compared to datasets like ImageNet.

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

Yes

Summary

By utilizing the pre-trained models, this paper presents a plug-and-play framework (MolPeg) to prune target data without source dataset. By maintaining two models with different updating paces during training, this paper introduces a novel scoring function to measure the informativeness of samples based on the loss discrepancy. The experimental results on 3 datasets shows the enhanced efficiency and superior generalization in transfer learning.

Strengths

- This paper is well-written. Each component of the design space is carefully explained and well-presented. - This method is easy to follow can be adapted to other tasks. - Extensive experiments are conducted to verify the effectiveness of this method.

Weaknesses

(Minor) Missing recent works: 1) static data pruning [1,2], 2) dynamic data pruning [3] 1. Active learning is a strong baseline for data subset selection. NeurIPS workshop, 2022 2. CCS: Coverage-centric Coreset Selection for High Pruning Rates. ICLR, 2023 3. Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt. ICML, 2022

Questions

See weakness.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

See weakness.

Summary

The paper introduces MolPeg, a molecular data pruning framework designed to enhance generalization when applying data pruning to pretrained models for molecular tasks. MolPeg uses two models with different updating rates to develop a new scoring function that assesses the informativeness of data based on loss discrepancies.

Strengths

1. By maintaining dual models that focus on both source and target domains and introducing a novel scoring function that selects both easy and hard samples, MolPeg achieves efficient, lightweight data pruning without the need for retraining. 2. The paper provides the code, ensuring the reproducibility of the method. I will attempt to run the code in the coming weeks and may modify my review comments as necessary.

Weaknesses

1. Although the paper claims to be pioneering in applying data pruning to pretrained models, the motivation may require further exploration. Specifically, how can it ensure that OOD samples crucial for each task are not pruned, which could potentially undermine the very purpose of the pretrained model? 2. The definition of what constitutes a 'hard case' is unclear. Are these cases task-specific? If so, there's a risk that pruning might eliminate hard cases essential for certain tasks, affecting the model's comprehensiveness and utility. 3. The approach might exacerbate the 'molecular cliff' problem, where slight changes in molecular structure lead to significant changes in activity, and such nuances could be lost with aggressive data pruning in pre-training. 4. The experimental section is limited to only three commonly used molecular datasets. Other equally important datasets, such as MUV, were not included in the comparisons.

Questions

None

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

It is suggested that the discussion of limitations be moved to the main text.