TopP&R: Robust Support Estimation Approach for Evaluating Fidelity and Diversity in Generative Models

Summary

The paper observes that the existing metrics for evaluating fidelity and diversity of generative models can be unreliable when outliers or non-IID perturbations are present. Thus, the authors propose a new metric based on Kernel Density Estimation under topological conditions that is more robust in the earlier scenarios. The main contributions of this paper are the metric that is robust to outliers and theoretical guarantees for robustness.

Strengths

The paper assesses a very important topic of evaluating generative models and proposes a metric that provides robust estimates of sample fidelity and diversity in several toy/synthetic examples. Additionally, bringing concepts from topological data analysis (TDA) to evaluation of generative models might open new future directions in metric research.

Weaknesses

The paper is difficult to follow for readers that are not very familiar with TDA and it could be improved significantly by adding intuition and a visual introduction to the most important concepts TDA and precision and recall. Furthermore, the concept of persistent homology, persistence diagram and confidence band estimation are discussed on a general level in Sec. 2 but the discussion seems disconnected how these are related to the proposed metric – adding concrete and intuitive examples would make this section more approachable. Unfortunately, I believe that in the current state paper is not very accessible to a wider audience. The novelty of the work is challenging to evaluate and potentially limited. In particular, the proposed metric seems to rely on the removal of outliers before evaluating the Top P&R, and the Top P&R part of the metric seems quite similar to previous work. Ideally, I would hope to see real world use cases with state-of-the-art models, where the metric provides more reliable evaluation (now Tab. 1 Top P&R yields very similar results to improved P&R).

Questions

1. The authors claim that robustness is required in practical evaluations of models since these cases are “wild”, i.e., might corrupt/outlier samples or the distributions might be sparse without density. I am curious as to why the proposed metric seems to perform similarly as improved PR with real generative models in Tab. 1? Also, Tab. 1 does not tell which dataset these models were trained and evaluated on. Since the proposed metric is composed of an outlier removal and the Top P&R evaluation, how much of the performance in toy/synthetic datasets can be explained by the outlier removal part? Would adding a similar outlier removal to existing improved PR and DC improve their performance in the toy experiments? 2. I think detecting outliers might prove to be difficult in practice, compared to the toy experiments presented in the paper. Real datasets do not ground-truth “outlier labels” and in some datasets, e.g. in medical imaging, the distribution has a very long tail as there might be more data from patients without a certain disease than with the disease. In this case, modeling the patients with the disease might be more interesting than modeling patients without the disease. However, when evaluating Top P&R the contributions of the patients with the diseases might be suppressed as they are deemed as outliers. 3. How does the dimensionality of the random projection affect the performance of the metric? 4. Is the bandwidth parameter $h$ different for each model in Tab. 1? If so, is it possible that some models obtain artificially better scores because their outlier removal is different than for the other models? Are Top P&R scores with different bandwidth parameters directly comparable?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

1 poor

Contribution

2 fair

Limitations

The authors adequately addressed the limitations of their work.

Summary

The paper presents a novel evaluation metric called Topological Precision and Recall (TopP&R) for generative models. Existing metrics often suffer from unreliable support estimation and yield inconsistent results. In contrast, TopP&R systematically estimates supports by retaining topologically and statistically significant features with confidence. It provides robust evaluation, accurately capturing changes in samples even in the presence of outliers and non-independent and identically distributed perturbations. The metric combines ideas from Topological Data Analysis and statistical inference, offering a reliable approach to support estimation. The paper includes theoretical and experimental evidence to demonstrate the effectiveness and robustness of TopP&R, making it a valuable contribution to the evaluation of generative models.

Strengths

The paper introduces a novel evaluation metric, Topological Precision and Recall (TopP&R), specifically designed for generative models. This metric addresses the limitations of existing evaluation metrics by focusing on robust support estimation, which is crucial for accurately assessing sample quality. TopP&R provides a robust evaluation that is resilient to outliers and non-independent and identically distributed perturbations. The metric systematically estimates supports by retaining topologically and statistically significant features, ensuring reliable evaluation results even in challenging scenarios. The paper demonstrates that TopP&R offers statistical consistency in its results. By incorporating concepts from Topological Data Analysis and statistical inference, the metric provides a systematic approach to distinguishing signal from noise and effectively capturing relevant features in the data. The paper provides both theoretical and experimental evidence to support the effectiveness of TopP&R. The theoretical analysis establishes the robustness and consistency guarantees of the metric, while the experimental results demonstrate its superior performance compared to existing evaluation metrics. TopP&R is a versatile evaluation metric that can be applied to various generative models. Its systematic approach to support estimation makes it suitable for evaluating the performance of models across different domains and datasets. The authors provide code implementation for TopP&R, making it readily accessible for researchers and practitioners. This facilitates the adoption and reproducibility of the metric, contributing to the transparency and advancement of the field. Overall, the paper's strengths lie in its introduction of a novel evaluation metric, its robustness and reliability in assessing sample quality, and its theoretical and experimental validation, demonstrating its effectiveness and broad applicability in the evaluation of generative models.

Weaknesses

The paper mentions Algorithm 1 in line 107 but does not provide the algorithm or any further explanation in the paper. This lack of clarity leaves readers confused and makes it difficult to understand the specific steps and procedures involved. Line 139 mentions the bootstrap bandwidth, but the paper does not clearly explain how this bandwidth is computed. It is important to provide details on the methodology used for computing the bandwidth to ensure transparency and reproducibility. The paper lacks a clear discussion on the dimensionality of the problem that the proposed method can handle. It does not explain how the authors choose the dimensionality of the space for random projections or how they ensure that these projections capture relevant underlying features. Additionally, the paper does not provide theoretical results supporting the choices made in Section 3.4 regarding dimensionality and random projections. In the experimental section, the authors choose a projection dimension of 32 without providing a clear justification or explanation for this specific choice. It is important to provide reasoning or empirical evidence to support the selection of this dimension and explain its impact on the performance of the method. While the paper includes simulations to demonstrate the robustness and effectiveness of the proposed method, it lacks a thorough evaluation on real-world data where changes are not simulated. It would be valuable to see the performance of the method in real data scenarios to assess its practical applicability and validate its effectiveness in capturing genuine changes. Addressing these weaknesses would strengthen the paper by providing clarity, offering detailed explanations for methodological choices, and conducting comprehensive evaluations on real-world data.

Questions

No additional questions in addition to the weaknesses listed above.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Summary

The paper introduces TopP&R, a comprehensive evaluation metric for generative models that enhances the accuracy of sample quality assessment in comparison to existing metrics. TopP&R incorporates topological data analysis and statistical inference, effectively estimating supports through the application of Kernel Density Estimator (KDE) under topological conditions. This metric demonstrates remarkable robustness towards outliers, successfully detects changes in distributions, and provides consistency guarantees and robustness for high-dimensional data with minimal assumptions. Notably, TopP&R is the pioneering metric that prioritizes robust support estimation, ensuring statistical consistency even when confronted with noise. The paper presents compelling theoretical findings and experimental evidence that strongly validate the effectiveness of TopP&R.

Strengths

The paper introduces a novel evaluation metric, Topological Precision and Recall (TopP&R), for generative models. It combines topological data analysis and statistical inference to estimate support robustly and detect distributional changes accurately. The paper provides theoretical and experimental evidence to demonstrate the reliability and consistency of the TopP&R metric. It accurately assesses sample quality, even in the presence of outliers, non-independent and identically distributed perturbations, and other noise conditions. The paper also presents a systematic approach to estimate support using Kernel Density Estimator (KDE) under topological conditions. The paper clearly explains the motivation, limitations of existing metrics, and the design and implementation of the TopP&R metric. It includes visualizations and examples to enhance clarity and uses clear language to facilitate understanding. The paper addresses the need for more reliable evaluation metrics for generative models. The proposed TopP&R metric fills this gap by providing a robust evaluation of sample quality and ensuring statistical consistency. Its findings have implications for improving the evaluation and development of generative models.

Weaknesses

Limited accessibility: The paper may be challenging for readers unfamiliar with persistent homology and topological data analysis to grasp the methodology and results. Providing additional background information and explaining concepts in more detail would improve clarity. Limited experimentation: The experiments are confined to a few datasets and models. Conducting experiments on a broader range of datasets and models would demonstrate the effectiveness of TopP&R in diverse scenarios. Insufficient discussion of limitations: Although the authors mention some limitations, such as the requirement for a large number of samples to estimate the confidence band, a more comprehensive discussion of the limitations would help readers understand the scope and applicability of TopP&R. Recommendations for improvement: Critique existing metrics: Highlight limitations of commonly used metrics, such as Inception Score and Fréchet Inception Distance, due to unreliable support estimates from sample features. Address noise vulnerability: Explain how TopP&R overcomes vulnerability to noise by considering topologically and statistically significant features with confidence. Present experimental validation: Provide comprehensive experimental results comparing TopP&R to existing metrics, demonstrating its superior accuracy and consistency. Discuss real-world applicability: Explore practical implications and applications of TopP&R in various domains, showcasing its relevance beyond theory. Share open-source implementation: Make TopP&R's implementation publicly available, ideally on platforms like GitHub, to promote reproducibility, validation, and further advancements.

Questions

1. How does TopP&R handle the trade-off between precision and recall in evaluating generative models? 2. Can you provide more details on the noise framework and statistical inference used in TopP&R? 3. Have you considered comparing TopP&R to other evaluation metrics based on topological features, such as Persistence Landscape? 4. Can you provide more information on the implementation of TopP&R and its applicability to different types of generative models? 5. How does the choice of threshold for retaining topologically and statistically significant features affect the performance of TopP&R? 6. Have you explored other techniques, like Gaussian mixture models or non-parametric methods, for estimating the density function in TopP&R?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Based on my understanding, the limitations of the TopP&R method mentioned in this paper can be summarized as follows: 1. The method requires a large number of samples to estimate the confidence band accurately, which can be a challenge for datasets with limited samples. 2. The reliance on the choice of kernel function and bandwidth can affect the performance of TopP&R, and the need for a theoretical guarantee on the choice of bandwidth can be difficult to obtain in practice. 3. The method assumes a smooth density function and independence between samples, which may not hold for all types of data, and the reliance on topological features may not capture all aspects of the data distribution.

Summary

The paper discusses robust estimation of precision and recall in high dimensional and potentially noisy data. The approach relies on using kernel density estimation to approximate the underlying distributions and using a bootstrap estimation of confidence interval to ignore support with low probability mass. The authors show the effectiveness of this approach extensively using simulated and real data.

Strengths

The paper is very well written and discusses the motivation and approach clearly. The problem addressed in the paper is very relevant as robust estimation of performance metrics is crucial for better comparing different approaches. The paper presents several interesting simulated and real examples to elaborate the method.

Weaknesses

Although the estimation of confidence interval to remove 'noisy' samples is interesting, it would be great to see a discussion on estimating this uncertainty at individual points separately than working with the worst uncertainty, i.e., the inf norm over the kernel density.

Questions

- Figure 1b: In the persistence plot, what does the shaded area imply? Can more blue points be added from the example on the right showing where in the persistence plot they land? Should the birth in this case always be at zero? - Line 128: while c_X and c_Y are referred to as bootstrap bandwidth, is there a link between them and h_n and h_m, i.e., bandwidth of the kernel density estimate? - Line 261: what does "swapping the position between Xi and Yj" imply? - Line 275: sensitiveness or sensitivity?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Limitation of the method has not been discussed in details. More discussion on the computational complexity will be helpful.

Summary

The paper addresses the need for more reliable evaluation metrics for deep generative models. The existence of noise in real data, such as mislabeled data and adversarial examples, affects the reliability of existing metrics and may lead to false impressions of improvement when developing generative models. To overcome these issues, the paper proposes a new evaluation metric based on the idea of Topological Data Analysis (TDA) and statistical inference. The proposed metric uses persistent homology from TDA and bootstrap bandwidth with confidence level to distinguish between significant patterns (signal) and noise in the data, providing a more robust and reliable measure of generative model performance.

Strengths

Given the recent substantial advancements in generative models, having a metric that is robust and reliable under various real-world noises holds significant importance. The authors present an approach that aims to overcome the limitations of previous metrics, using straightforward yet well-founded methods. Additionally, the paper introduces theoretical properties that show the metric's consistency in the presence of adversarial noises. The paper is well-written and easy to follow.

Weaknesses

1. If I understand correctly, the authors use random linear projection only for the proposed metric, which might be unfair since it is unclear whether the improvements in robustness to outliers or noises stem from the low-dimensional space or the proposed methods. A comparison with previous methods operating in the same low-dimensional space would be more comprehensive. 2. The proposed metric's algorithm relies on multiple iterations for KDE filtration, raising questions about its practicality. It would be beneficial to include a comparison of the computational cost and overall runtime between the proposed metric and previous ones. Gaining insights into its efficiency compared to existing metrics will be valuable for both potential users and researchers. Additionally, it is essential to consider whether the metric can efficiently scale to handle a large number of datasets. However, the paper does not seem to provide details about the number of datasets used in the experiments, which leaves uncertainty about its scalability. 3. In section 5.1.2, the authors assert that precision values smaller than 1 indicate a problem. However, this lacks sufficient justification especially with a small number of samples. Additionally, the behavior of TopP for simultaneous mode dropping in Figure 3 exhibits worse instability compared to other metrics. 4. In the experimental results for mode dropping experiments with CIFAR-10, TopR does not exhibit consistent behavior with the toy experiments in the main paper (Figure 4) and sometimes performs worse than Recall and Coverage. 5. In section 5.2.3, the authors argue that the proposed metric is more sensitive to different levels of diverse noise, similar to FID. However, the results in Figure 6 do not appear to provide compelling evidence for this claim, except for the Gaussian noise case. Notably, in the salt and pepper noise case, TopPR, PR, and DC exhibit equal behavior, and in the black rectangle scenario, TopPR performs worse than PR. To strengthen the authors' claim about the proposed metric's sensitivity to diverse noise levels, additional explanation and experiments are necessary. 6. The sole reliance on FID as the ground truth ranking for models in section 5.2.4 might be insufficient and unconvincing. It is not clear how much relying on FID as GT is reliable. Including more widely used metrics, such as KID or human evaluations, would enhance the credibility of the result. However, even with the FID, the results do not seem to show the clear superiority of the proposed metric. Additionally, the results with HD across different embeddings also raise questions about the effect of the random projection layer, as other metrics do not utilize such a layer. 7. Some statements in the paper are unfounded. For instance, the rationale behind the ideal metric returning zero (line 263) is not clear. Moreover, the assumption in line 278 ("ground truth diversity should linearly decrease") is not correct because the supp(Q) remains the same with simultaneous mode dropping. In that sense, TopR works the worst of all.

Questions

1. This paper employs the median of pairwise k-nearest distances between samples to estimate fixed-width kernels for KDE. I am curious whether this method is as effective as using sample-adaptive varying kernels, especially for datasets with multi-modal or long-tailed distributions. Adaptive kernels that adjust to local data characteristics might offer advantages in capturing complex and diverse patterns. 2. Concerning the experiments presented in Table 1, which dataset was used for these experiments? Can the proposed metric demonstrate effectiveness when applied to large-scale datasets, such as ImageNet? 3. There are some recommended experiments that are missing in the paper but would strengthen the effectiveness of the proposed metric: - Evaluating metrics on Diffusion models such as ADM or EDM, which are the leading and prominent advancements in generative models, would be beneficial to demonstrate the practicality of the proposed metrics. - Experiment for being robust to outliers in real-world dataset as in D&C paper. - Ablation study regarding the hyperparameters alpha and h for real-world datasets. 4. To enhance the readability of the paper, the captions of figures should be more detailed.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Limitations are discussed in the supplement. Potential negative social impact is unlikely for a paper comparing distributions.