We thank the reviewer for their prompt response and willingness to engage in discussion. We believe they bring up two very pertinent points.
## Utility of FLS over FID
> I am not sure if I will use FLS instead of FID unless I want a generative model for data augmentation for discriminative models. Different users have different needs.
We understand the stickiness of FID as a metric and appreciate its utility. In fact, in most scenarios, we show that FLS largely agrees with FID and yields a similar model ranking (Figure 9.). However, we point out multiple failure modes of FID in the paper and show how they are addressed by FLS.
- **Heavy bias for smaller samples**: FID displays heavy bias, even up to 50k samples. FLS on the other hand works better with smaller sample sizes which makes it significantly more amenable to evaluating class-conditional generation (where there are often less than 10k samples per class) and for finding problematic classes/issues with conditional alignment.
- **Insensitivity to overfitting/memorization**: FID as it is currently computed (comparing with train samples) decreases with memorization and even when compared to test samples, a model copying the training set gets a SOTA FID.
- **Sensitivity to imperceptible transformations**: Very slightly altered samples can yield considerably worse FID scores. As shown in Fig.3 of the rebuttal PDF, such transforms cause samples from a SOTA model to be rated as worse by FID than those produced by a 5 year old model. As such, small changes in image processing can have disproportionate impacts on FID.
As well as other issues [1, 2]. We believe it is problematic to continue using a metric with known issues, especially as these issues are becoming more common. As such, we believe that FLS should be favored in most cases and especially when working with smaller datasets. Moreover, note that in our latest experiment (Table 1 of the rebuttal PDF) we observe that FLS with DINOv2 showcases a trend of the superiority of diffusion models in comparison to SOTA GANs (such as StyleGANXL) which was not clear with FID.
## Multi-faceted evaluation
> Different users have different needs. Users do not know which aspect is affecting FLS[...] I think we need more specific measures for different aspects.
We completely agree that specific metrics (such as precision, recall, rarity score, etc.) are very valuable and should be used to evaluate specific aspects of the best-performing models, but such a diagnosis is complementary to benchmarking and ranking using a single holistic metric. In fact, the use of a single metrics by practitioners is undeniable:
- **Generative modeling for image data:** FID is by far the most used metric.
- **Machine translation:** Practitioners have been reporting the F-score [3, 4] which trades off precision and recall, or the BLEU score (a precision oriented metric).
- **Supervised classification:** Practitioners often use standard accuracy to evaluate their models even though some datasets have been saturated. Even for unbalanced dataset practitioners often report F1 score [5].
This can be explained by:
- Comparing two models with respect to several metrics is more challenging since they may be on the Pareto front ($\mathcal{R}^2$ (fidelity, diversity) or $\mathcal{R}^3$ (fidelity, diversity, novelty) are not totally ordered).
- When considering metrics such as precision and recall a model could over-optimize one which corresponds to a pathological behavior while still being on the Pareto front.
#### Flexibility of our Method
We would like to point out the flexibility of the feature likelihood methodology. We believe this form of density estimation allows for measuring various aspects of generative model evaluation not possible with FID:
- **Overfitting evaluation:** The % of overfit Gaussians and the generalization gap allow specifically for evaluating the overfitting behavior quantitatively. If the issue affecting FLS is overfitting, it can be picked up by these methods.
- Ranking of memorized samples provides a way to qualitatively evaluate memorization by the model
- **Fidelity evaluation**: If we estimate the density of the data distribution by using Gaussians centered at the test set, we can then use this to evaluate the likelihood of the generated samples (and thus find high-quality samples, such as in Appendix A.3).
- The likelihood of the whole dataset could be a quantitative measure of fidelity.
[1] Kynkäänniemi, Tuomas, et al. "The Role of ImageNet Classes in Fr\'echet Inception Distance."(2022).
[2] Parmar, Gaurav, Richard Zhang, and Jun-Yan Zhu. "On aliased resizing and surprising subtleties in gan evaluation." 2022.
[3] Van Rijsbergen, Cornelis Joost. "Foundation of evaluation." 1974.
[4] Derczynski, Leon. "Complementarity, F-score, and NLP Evaluation." 2016.
[5]Wang, Xudong, et al. "Long-tailed recognition by routing diverse distribution-aware experts." 2020.