Towards a Scalable Reference-Free Evaluation of Generative Models

Summary

This paper introduces the Fourier-based Kernel Entropy Approximation (FKEA) metric, which efficiently evaluates the diversity of generated samples. The key contributions of this work are twofold: (1) Compared to existing diversity metrics such as VENDI and RKE, the proposed metrics (i.e, FKEA-VENDI and FKEA-RKE) are more computationally efficient, reducing complexity to $O(n)$ for a sample size $n$. (2) The proposed metric is reference-free and can be used to assess the performance of large-scale image, text, and video datasets for generative models. Extensive experimental results demonstrate the effectiveness of the proposed metric.

Strengths

- The paper writing has a clear structure and the theoretical results are technically sound. - The proposed metric exhibits improved theoretical computational efficiency (i.e., Line 13) and can be calculated without sophisticated hardware setups (Line 245). - The method can be broadly applied to assess the performance of various generative models for images, texts, and videos.

Weaknesses

- The proposed evaluation method relies on a number of parameters, e.g., $\sigma$ and $r$. The values of these parameters seem to vary significantly according to different tasks. For example, the authors adopt $2r=4,000$ and $\sigma=7$ for the experiment on the MNIST dataset, and $2r=16,000$ and $\sigma=25$ for the experiment on the ImageNet dataset. How these parameters influence the evaluation results remains unexplored. (See Questions 1 and 2) - The paper could be improved by incorporating a toy example to illustrate the difference between reference-dependent and reference-free metrics. The discussions in Sections 1 and 2 are not concrete enough. (See Question 3) - One of the key contributions of this paper is the theoretical improvements in the computational complexities of the FKEA-based metrics. An evaluation time comparison between FKEA-based metrics (e.g., FKEA-VENDI and FKEA-RKE) and their original metrics (e.g., VENDI and RKE) is missing. (See Question 4) --- **Minor Point** - There is a typo (i.e., $x \to \mathbf{x}$) in Line 157.

Questions

1. The proposed metrics require stochastic approximations on expectations (i.e., $\mathbf{x}$ and $\boldsymbol{\omega}$). Could the authors report the variances of the evaluation results? Additionally, can the authors provide guidelines for choosing a sufficient number of samples? 2. Is there any underlying intuition for selecting $\sigma$ and $r$? Will an inappropriate selection of $\sigma$ and $r$ lead to an inaccurate assessment of the true generative quality? 3. Could the authors provide experimental results to demonstrate that reference-dependent metrics (e.g., Recall and Coverage) may fail to measure sample quality, while reference-free metrics may be successful? 4. Could the authors provide an evaluation time comparison between the calculation of FKEA-based metrics and their original metrics? For example, an evaluation time comparison between FKEA-VENDI and VENDI would be helpful.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have allocated a section (i.e., Line 292) for the discussion of potential limitations.

Summary

The work introduces a new method called Fourier-based Kernel Entropy Approximation (FKEA) to evaluate the diversity of data generated by generative models. Traditional evaluation metrics for generative models often rely on reference datasets, which may not always be available or suitable. Recently, reference-free entropy scores like VENDI and RKE have been proposed but suffer from high computational costs, especially with large-scale models. FKEA addresses this issue by leveraging the random Fourier features framework to reduce computational complexity. It approximates the eigenspectrum of the kernel matrix to estimate entropy scores efficiently. The method utilizes proxy eigenvectors derived from FKEA to identify modes in the diversity assessment of generated samples.

Strengths

1. Paper is well-written and easy to follow. 2. The contribution seems useful to the community. 3. Thorough quantitative evaluation is present on text, image and video datasets. 4. The paper mentions limitations and scope for improvement.

Weaknesses

1. Evaluation seems to be limited to simple datasets. How would the method perform on complex image and video datasets? 2. The reported metrics cover the basics of establishing the advantage of this method. Any other metrics that can be used to establish clear dominance of this method over existing methods? 3. Qualitative results are hard to follow.

Questions

see weaknesses section.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

see weaknesses section.

Summary

This study proposes a computationally efficient metric for evaluating the performance of recent generative models. It highlights the limitations of using reference data, which can restrict the applicability of evaluation methodologies, and instead suggests a method utilizing kernel functions without references. Experiments demonstrate that the proposed metric shows similar trends to existing metrics across various modalities and datasets, and qualitatively reflects the characteristics of the data in the metric computation process.

Strengths

* Compared to existing metrics, the proposed method allows for quick metric computation through relatively simple calculations. * The calculation process reveals that the eigenvectors generated by the metric can semantically distinguish the generated outputs. * It is applicable using various embedding models.

Weaknesses

* Compared to existing metrics, the speed improvement in performance measurement is not experimentally specified. * The numerical alignment with existing metrics is not provided.

Questions

- Please provide the extent of the speed improvement over existing method in actual experiments. - Provide experimental results based on the capabilities of different embedding models (e.g., compare performance measured using BERT for text with that using text-embedding-3-large). - Measure the correlation between the existing metrics and the proposed metric in your experiments.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes