Summary
The authors address the problem of data augmentation for classification tasks by proposing a novel approach that utilizes separate energy-based models (EBMs) to generate synthetic data for each class. These EBMs are derived directly from the logits of a binary classifier whose goal is to distinguish real data from negative data. These logits (their logsumexp over positive and negative classes) are reinterpreted them as an energy function for each class.
In order to avoid training these classifiers on the data itself, the authors take advantage of a pre-trained Prior-Data Fitted Network (PFN).
The authors perform a comprehensive evaluation of their method comparing it to competing methods for data augmentation on a number of small real world datasets. They show that their method is the only one that offers a consistent improvement in classification from the addition of synthetic data.
Strengths
- The paper is well-structured and clearly written, effectively communicating the key ideas and concepts.
- The authors introduce a novel approach by leveraging a Prior-Data Fitted Network (PFN) to obtain energy-based models (EBMs), which, to the best of my knowledge, has not been previously explored.
- The experimental results section is thorough and comprehensive. They compare their approach to a wide range of relevant and competitive methods for synthetic data generation.
- The appendices are well-organized and provide valuable supplementary information. The paper strikes a good balance with the most relevant information presented in main text and additional details provided in the appendix.
Weaknesses
- Interpreting classifier logits as energy functions may not necessarily yield good density models. Given a classifier with logits $f(x)[y]$, one can add an arbitrary function $g(x)$ that is constant over $y$, $f(x)[y] \rightarrow f(x)[y] + g(x)$, without affecting the conditional probabilities $p(y\vert x)$ due to the invariance of the softmax function to the addition of a constant. However, this changes the energy function for the density model to $E(x) \rightarrow E(x) - g(x)$. Consequently, there exists a functional degree of freedom that can arbitrarily alter the energy function without impacting the discriminative loss optimized by a classifier. As a result, a classifier trained solely to distinguish between positive and negative samples is not inherently encouraged to learn a useful energy function for the density over $x$.
My interpretation of the results of [1], from which the authors seem to draw inspiration is that reinterpreting the logits of a classification model as an energy based model by itself accomplishes nothing if the training process of the original classifier is not changed to encourage a good density model to be learned and encoded into the free degree of freedom. [1] achieves this by optimizing the joint likelihood, $E_{x,y} p_\theta(y,x)$, instead of the conditional likelihood, $E_{x,y} p_\theta(y\vert x)$, used in training classifiers. This significantly complicates training by requiring sampling from the EBM during training. Therefore, it is unclear why the authors assume that the pre-trained PFN has learned useful energy functions for density modeling when it is derived from a simple classifier and this is not discussed in the paper.
- Another weakness of the paper is in the experimental setup where only default hyperparameters are used for all competing methods. For many of these methods, tuning hyperparameters can substantially improve their results and the default hyperparameters may not be optimal.
Personally, I would find the presented results more convincing if the paper compared with fewer competing methods but made a better effort of getting good results out of those as it is impossible to know how much the authors tuned the degrees of freedom available during experimentation with their own proposed method.
[1] Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, Kevin Swersky. Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One, 2022 (https://arxiv.org/abs/1912.03263)
Questions
- Can you provide a convincing explanation for why the logits from discriminative classifiers derived from TabPFN should be useful when reinterpreted as a class-conditional energy function in light of the first point above? Without a clear justification, it is difficult to understand the basis for expecting this method to work effectively.
- Since the authors marginalize over $y$ by taking the LogSumExp over the $y=0$ and $y=1$ logits corresponding to negative and empirical class data, respectively, one would expect the energy function to represent a mixture distribution between these two densities. Why aren't the logits for $y=1$ used directly as an energy function instead? In other words, what is the motivation for marginalizing over $y$ when it is simply an indicator of whether the data is real or negative?
- The details of how the negative examples are generated are not clear. The text simply states:
"These surrogate negative samples $\mathcal{X}^{neg}_c$ are constructed to be far from the distribution of real data, ensuring that the classifier can easily distinguish them. This placement prevents class ambiguity and facilitates a robust energy landscape". The appendix does provide some additional details, but it is still ambiguous how this data is generated. Given the potential importance of this process in understanding why the method works, could you provide more clarity on how the negative data is generated and discuss how changing the distribution of negative data affects the results?
- The authors helpfully point out all datasets that were also used as test sets for TabPFN's evaluation since these could potentially be tainted if good practices were not followed on the original TabPFN paper [1].
I am more concerned however with datasets that were part of the meta-validation set that was used to tune the hyperparameter priors of TabPFN. It seems that one of the datasets used for evaluation of TabEBM (stock) is part of this meta-validation set (Tables 7 and 8 in the appendix of [1]).
Even though this is only one of 150 datasets that comprise this validation set and it is not clear in the original paper the extent of this finetuning, I would still consider it good practice to avoid using this dataset for evaluating downstream methods for fear of data leakage. I also would have liked the paper to prioritize evaluating on more datasets that weren't used at all in [1]. I understand if [1] might have somewhat exhausted the pool of small datasets in OpenML but evaluating on larger datasets by first subsampling them would be an option. This would also allow for the use of real data as an ideal baseline to measure TabEBM's improvement against.
- For TabEBM and the other methods that learn class conditional densities (CTGAN, TabDDPM, ARF, GOGGLE, TabPFGen) what distribution of classes (i.e., $p(y)$) was used for the synthetic data? This is not specified in the paper so I am assuming it was an empirical estimate of the p(y) from the training data otherwise this would be somewhat unfair when comparing to methods that learn a joint distribution. Furthermore it would be potentially problematic when evaluating a balanced accuracy in a test set if the p(y) of generated data is not the same across methods (for conditional methods).
- How are statistical fidelity results computed? The metrics reported seem to be univariate ones so how are they being used to compare multivariate distributions? Are only marginal distributions being compared? Is it an average over univariate metrics computed for each marginal separately? How is the KS test used on categorical variables? This section of the experiments is sorely lacking in details and the appendices seem to provide no further information. If it is indeed the case that only marginals are being compared I find this evaluation lackluster since it is easy to achieve similar marginal distributions without learning the relationships between different input variables which is the main difficulty in learning multivariate density models.
[1] Noah Hollmann, Samuel Müller, Katharina Eggensperger, Frank Hutter. TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second, 2022 (https://arxiv.org/abs/2207.01848)
Limitations
The authors aknowledge that due to the reliance of the method on a PFN it is currently constrained for use in small datasets.