Summary
While synthetic training images generated by diffusion models can be effective for training, diversity and fidelity are challenges. Fidelity can be addressed by image conditioning with few-shot images, but many previous works fine-tune the diffusion model, which can be expensive. The authors propose a frozen alternative to increase the diversity, which conditions the diffusion model not only on few-shot images (done previously) but augmentations (novelty).
They evaluate this across different standard data augmentation techniques, comparing downstream training accuracy and FID to find the most effective augmentation techniques. They also evaluate the effects of CFG scale, compare their method to previous SOTA in long-tailed distributions, and across 4 datasets to previous few-shot SOTA by number of shots.
Strengths
A) Writing is overall quite clear.
B) Experiments are varied across areas (long-tail, few-shot).
C) Method section shows many qualitative examples to supplement the quantitative results in the experimental section.
D) Some of the results show improvement.
Weaknesses
Points are ordered roughly according to my perceived scale, with more important points being listed first.
A) The method itself is quite simplistic from a novelty perspective (simply adding augmentations to the conditioning). I would consider this a strength, if the results were consistent (B) and strong (B, D, E, F) with a clear storyline for effective use-cases (C). However, I do not see this as being the case (see following points for details, as indicated in the corresponding parentheses).
B) The results are mixed. Examples: 1) In table 1, the random image baseline actually has the lowest FID. I do not see this discussed, with lines 323 and 339 pointing out that "the best-performing augmentation-conditioning method has one of the lowest FID scores, supporting our claim...", which is misleading. 2) In table 3, unintuitive bolding hides that your method underperforms in some categories (with LDM t&i, medium is worse and few is tied). 3) In Figure 6, by 16 shots, the novel method is already underperforming on 2 datasets. Once again, the writing does not properly address this.
C) Given the mixed results, there should be an in-depth analysis / explanation to understand when this method is most useful, but this seems to be missing. I want to stress that if done well, this could potentially make up for weakness B.
D) The comparisons with Fill-Up are not clear, as "Ours" does worse but uses less data--it would be better to compare against Fill-Up with the same amount of data as well, otherwise you have not shown that you are beating SOTA.
E) There seems to be some baselines missing in the few-shot section that could strengthen the context. 1) as the ResNet50 is pre-trained, it would be helpful to know the starting accuracy. 2) Figure 6 is missing the random-image baseline included in Table 1.
F) I do not find the CFG scale experiments as adding significant value, although they take up a page in total. They are consistent with previous work, which I don't find unsurprising. While they don't really 'hurt' anything in themselves, they overall weaken the experimental section by taking the space of what could have otherwise been more interesting / surprising / novel results, and in my opinion they water down the impact of the experimental section.
G) In the introduction, it is claimed that methods that fine-tune the diffusion model (Azizi 2023, Trabucco 2023, Shin 2023) are too expensive. However, this is never supported with numbers--it would make the claim stronger to quantify method costs. Especially because these methods have vastly different costs (e.g. in line 134, you claim that Shin 2023 uses textual inversion--this should be less expensive than full fine-tuning, correct? And what about methods that use PEFT?)
H) Table 3 is misleading. The way the sections are split, the entire lower section should be compared. However, Fill-Up is a separate class with more synthetic data, which you are not comparing against. This is not very clear by the visual, and is confusing why the highest numbers are not the ones bolded (as Fill-Up is excluded). Sometimes the bolding seems entirely wrong--e.g. in medium, "ours" is bolded, but LDM (txt and image) is clearly better, and also at comparable synthetic data counts.
Questions
A) In the description of Figure 4, you claim in line 236 that "Augmentation-conditioned generations show more visual diversity in the coloration, pose, and angle of the hamster." Could you please elaborate on what you mean? Because I do not see this as being the case. To me, 1) coloration looks consistent across all images, 2) pose only shows clear differences for CutMix and CutMix-Dropout. All others are primarily face photos staring at the camera and 3) with the same exceptions as 2, angle is primarily straight-on.
B) In table 1, it is shown that the improvements are the most significant on the few classes--this is potentially interesting. Is there some analysis, interpretation, explanation, etc. that could be added on this subject?