Summary
### Summary
This paper presents a new method (EDF) for dataset distillation targeting improved performance in complex scenarios like ImageNet-1K subsets. EDF integrates Grad-CAM activation maps to focus on high-activation, discriminative regions within synthetic data, unlike previous methods that treat all image pixels equally. It also introduces the *Common Pattern Dropout* module to remove low-loss supervision signals, which often contain non-discriminative common patterns. Additionally, the authors contribute a new benchmark, *Complex Dataset Distillation (Comp-DD)*, to help the community evaluate distillation methods in complex scenarios. Experimental results show EDF outperforming state-of-the-art (SOTA) methods in challenging datasets and achieving "lossless" performance on some subsets.
Strengths
- **Novelty**: EDF’s use of Grad-CAM to target high-activation areas in synthetic images is innovative, particularly for complex datasets where discriminative features are limited to small regions. The *Common Pattern Dropout* module’s filtering of low-loss signals is a creative solution in an attempt to minimize non-discriminative features, improving the overall quality of synthetic data.
- **Benchmark Contribution**: The creation of the Comp-DD benchmark provides a valuable tool for future work, establishing a way to test dataset distillation methods on complex scenarios.
- **Strong Results**: EDF achieves SOTA performance across a variety of complex dataset subsets, with substantial gains in accuracy and the ability to achieve near-lossless performance in certain settings.
Weaknesses
- **Inaccuracy of Grad-CAM Definition in Section 2.2**: According to the original Grad-CAM paper, $M^c$ should be a gradient-weighted sum of all feature maps in the last convolutional layer. Therefore, the symbol $l$ in Equation (3) should denote the $l$-th feature map instead of the $l$-th convolutional layer.
**Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D.** (2017). *Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization.* In *Proceedings of the IEEE International Conference on Computer Vision* (pp. 618-626). doi:[10.1109/ICCV.2017.74](https://doi.org/10.1109/ICCV.2017.74)
- **Dependency on Hyperparameters**: EDF’s performance depends heavily on various hyperparameters (dropout ratio $\alpha$, Grad-CAM update frequency $K$, enhancement factor $\beta$), requiring fine-tuning for a particular dataset, which could limit usability in practical applications without extensive experimentation.
- **Generalizability to Non-Complex Scenarios**: EDF shows significant improvements on complex datasets; however, its benefits on simpler datasets (like CIFAR) are less clear. It’s valuable to show EDF’s generalizability across datasets of various complexity.
- **Limited Complexity of Comp-DD**: ImageNet is a curated dataset, with most object-related images in a portrait style. Consequently, the subsets chosen from ImageNet in Comp-DD are likely also portrait-oriented with larger background regions. This setup may limit the true complexity of the benchmark. To further increase complexity, incorporating zoomed-out, cropped objects with relevant labels from uncurated datasets such as COCO, Objects365, or SA-1B could provide a more challenging and diverse set of scenarios, better suited to the intended purpose.
- **Model Bias in Generating Activation Maps**: It is well-known that neural networks can capture biased or non-generalizable features, as discussed in the Grad-CAM paper. If the activation maps focus on such less-generalizable regions, this could undermine the effectiveness of the *Discriminative Area Enhancement (DAE)* module. However, it is unclear how the proposed method addresses this issue of potential bias in the activation maps. Clarification on how EDF mitigates or adapts to these biased features would strengthen the approach.
- **Clarification on Low-Loss Regions and Discriminative Features**: The paper assumes that low-loss regions correspond to common, less-discriminative features. Although the paper demonstrates activation area shifts with different loss levels during distillation (Fig. 2a), additional qualitative results—such as visual analysis with superposed images and highlighted activation regions—would help support this assumption. It is important to note that a smaller activation region does not necessarily indicate less-discriminative features.
Recommendation:
Overall, my recommendation for this paper is borderline reject. However, the score could be improved if the concerns outlined above are properly addressed.