Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Data

Summary

This paper contributes a new model evaluation framework called 3S-Testing, which uses (conditional) deep generative models to create synthetic test sets. To provide uncertainty estimations, 3S uses deep generative ensemble method. It is empirically confirmed that the better performance on small subgroups (with real data), and distributional shifted data.

Strengths

The motivation is clear by targeting the subdominant class. Using generative models (and generative ensembles) to evaluate model is an encouraging attempt, especially for uncertainty estimation. The experiment section is clearly organized and easily understood.

Weaknesses

$\bullet$ The major concern is to evaluate a model, more uncertified models are introduced even though they are used for uncertainty estimation. The author(s) gave reasoning from line 155 to 161. The first is reasonable, while the second is not convincing. For example, in figure 2, the samples of green star class are sparse in the test set by nature. Generating samples of green star class becomes (b) is a “bias” introduced by the generative model. $\bullet$ 4.2 is hard to read. While the intention was to keep the discussion for shift as general, more and more notations and assumptions for simplifications are continuously added, sometimes without explanation (see below in questions).

Questions

$\bullet$ How to estimate the biases of the generative models on the subgroup, besides providing uncertainty estimation? One possible case is mentioned in weakness, where the generative model generates new samples based on some its own bias. $\bullet$ Lines 195-198 is confusing to me. After G is trained, how exactly is it used? Steps (2) to (4) are not clear about this. I understand it is talked about later, but it is important to make it clear in the summary paragraph. $\bullet$ Lines 207-208, Where does that assumption come from? $\bullet$ Can the author(s) please briefly summarize the discussion (with minimum information/notation/assumptions needed) for lines 204-218? More specifically, does the final recipe proposed in lines 217 to 218 rely on the assumption “simplest such shift” (line 212)?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

Fully discussed with explanation.

Summary

The authors propose to use synthetic data to evaluate models, especially under distribution shifts or in areas of the input space with low coverage. The authors use CTGAN to empirically validate their idea, and apply it to tabular data. The paper is a resubmission from ICLR 2023 ( https://openreview.net/forum?id=J7CTp-jNyJ ).

Strengths

Originality: While the general idea of using synthetic data for evaluation does not appear new, to the best of my knowledge this is the first thorough evaluation of the idea for tabular data. (I went out of my way to find published studies on this, and the only works I found were on relatively low-quality journals, or very different in scope). Quality: Experiments were done on 6 datasets of relatively small scale. This gives a first idea that the idea might work, but mention two possible extensions under "Weaknesses". Clarity: the paper clarity was fine. Significance: I think this is an important topic that deserves more study, the results of which will be of interest primarily to practitioners, but might also encourage further research on the topic.

Weaknesses

* The paper mostly focuses on 6 small datasets. While the approach itself is probably interesting in cases where data is scarce, it would be interesting to see this applied to harder datasets that have more than just a couple of dozen features, or where there are millions of samples involved. I personally remain unconvinced the method scales to larger data, and would appreciate if the authors could report results on larger datasets. * The authors write "* "an end-user only has access to a single draw of Dtest,f . e.g., we might incorrectly overestimate 265 the true performance of minorities. The use of synthetic data solves this.". I'd like to challenge that statement: I think it's very typical in real life (especially given the relatively small data sets that are the focus in the publication) to at least use cross validation to get multiple test sets, potentially even Leave-1-Out CV. This should give a much better estimate of the loss landscape. I'd appreciate if the authors could add such experiments as baselines. * It would be nice if the authors gave an indication of run times of the various methods they compare too, as this might be useful to practictioners. E.g. They mention in Appendix section C that the GAN was trained with a fairly large hyperparameter search, which I expect is not cheap.

Questions

Figure 4: It is unclear what the x-axis actually represents. It would be nice to get an explanation of what the Different Groups mean. The way it is presented, I don't understand what I am looking at. Line 180: "similar in vain to Deep Ensembles" => similar in vein

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors mention limiations in passing, but have not gone out of their way to find out failure modes of their method.

Summary

This paper proposes to use synthetic test data to improve the estimation of model performance for tabular datasets when insufficient test data is available. Their approach of generating synthetic test data conditioned on subgroups improves performance estimation for underrepresented subgroups and can accurately estimate the model performance under distributional shift towards the underrepresented subgroups.

Strengths

The paper is well-written, and the details are covered by the Appendix. They have done a good job in showcasing the advantage of their synthetic strategy.

Weaknesses

Their proposed method is limited to the tabular data and data synthesis has been looked at in prior studies, but I think this is a good paper with nice experiments.

Questions

Q1. Extreme underrepresentation of the data in some regions, i.e., small subgroups, could affect the training of the generative model as well. At what point does your method fail, i.e., it is simply as good as Dtest strategy? Can you show this for different datasets studied in Figure4? Q2. How do you justify the 3S+ performing worse than 3S for the bank dataset especially for decision tree and random forest in Appendix Figure 10 and Figure 4?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

NA

Summary

In this paper, the authors propose the utilization of synthetic data for evaluating models and introduce an automated suite of synthetic data generators called 3S. 3S offers two key advantages: it enables reliable and detailed evaluation, and it measures model sensitivity to distributional shifts. The paper explores different scenarios involving 3S, providing valuable insights into the application of synthetic data for model evaluation.

Strengths

1. The paper exhibits a clear and easily comprehensible writing style. The authors conduct a comprehensive examination of relevant literature, effectively summarizing its advantages. 2. The issue investigated in this paper, namely the utilization of synthetic data for evaluating models, is both intriguing and significant. 3. The authors effectively discuss various use cases of 3S, offering insights into the practical utilization of synthetic data for model evaluation.

Weaknesses

1. Further discussion on method limitation is needed.

Questions

No questions.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

As mentioned, further discussion on method limitation is needed.

Summary

This paper proposes a model evaluation framework, generating the synthetic test to mitigate the challenges of model evaluation with limited real test sets, such as unbalanced subgroups and distribution shifts.

Strengths

The idea of using synthetic data to improve the testing and evaluation of machine learning models is impressive. This paper analyzes clearly and reasonably the failure of real test data and its corresponding challenge for reliable model evaluation.

Weaknesses

**1. Counterintuition.** Although the empirical benefits of syntenic data are observed in the experimental parts, the inequation of Line 155 is not evaluated theoretically. More precise statements are needed here. **2. Overfitting of deep generative models.** With limited test data in the small subgroup, deep generative models tend to perform overfitting. In this case, the learned manifold could have a huge gap from the real manifold. It could be better to show more visualizations in Figure 2. In detail, if we change the test samples in the green subgroup, what will happen when we compare synthetic manifolds and the real manifold? **3. Failure case.** As a model evaluation framework with synthetic data, which makes a lot of sense in the real world, this paper lacks failure cases to show the limitations of their works.

Questions

Please check the weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please check the weakness.

Thank you for your explanations!

Thank you for your reply. The explanations are much clearer than what was written in the paper. According to the explanation, G is not learning p(X) as stated in line 196, but a conditional probability. It aims to generate more (corrupted maybe) samples depend on the shift, doesn't it? Please consider using these simplified and clearer explanation when editing the draft. I have raised the score. Thank you for your explanation and good luck.