Complementary Answers and Information 2/2
> [...]For small timesteps, the loss function seems very closely related to pseudolikelihood estimation [...]
>
Thank you for raising this point. It seems that pseudo likelihood can be derived from energy-discrepancy as follows: Define $q(\mathbf y \vert \mathbf x) = \frac{1}{d} \sum \delta( y_i, \ast)$ which masks exactly one entry of the data vector. Then, energy discrepancy is given for a sampled perturbation $y = x_{\neg I}$ as
$$
U_\theta(\mathbf x) + \log \sum_{x} q(\mathbf x_{\neg i} \vert \mathbf x) \exp(-U_\theta(\mathbf x)) = -\log \frac{\exp(-U_\theta(\mathbf x))}{\sum_{s \in \{1, 2, \dots, S\}}\exp(-U_\theta(\mathbf x_1, \dots, \mathbf x_i = s, \dots, \mathbf x_d))}= -\log p_\theta(\mathbf x_i \vert \mathbf x_{\neg i})
$$
Hence, this specific ED loss function is indeed a MC approximation of pseudo-likelihood. Energy discrepancy is appealing because it is more general and more tunable through the choice of $t$ and $M$. We will discuss this connection in our revised version.
> Since these methods have mixed continuous and discrete data can the authors do a separate benchmark that only models the discrete columns?
>
The performance on continuous data is compared in prior work [9]. In this work, we benchmark ED on various discrete data sets. For a separate benchmark on discrete/continuous variables in tabular data we are missing baselines that perform the same experiment, and an implementation of comparative experiments was not possible in the given time frame.
> I was confused by the statement that method is sensitive to the assumption that the data distribution is positive on the whole space. Why is this more of an issue for ED than other EBM training techniques? [...]
>
Other training techniques for EBMs are adaptive, i.e. the negative samples are created by attempting to self-sample from the model. If the sampler is well-tuned, this produces smoothed estimates of the data distribution (see e.g. Figures 2, 7 to see the smoothing effect compared to ED). Thus, these methods produce biased estimates which can be advantageous when training on image data, where the data support is small compared to the ambient space. ED produces more accurate estimates, but tends to oversaturate quickly when the diffusion produces uninformative negative samples.
> I would also like to see more comparisons of CD with large numbers of MCMC steps v.s. ED with large numbers of samples.
>
- Table 2 reports results for CD with 40 MCMC steps. CD with larger numbers of MCMC steps is rarely used due to the cost of training.
- Similarly, ED with large number of samples would loose the key advantage of ED of being cheaper to compute than CD or MLE with MC approximated normalisation constant.
A major difficulty with increasing the number of MCMC steps in CD is that the sampler likely needs to be retuned. For all these reasons, we chose to compare CD with a typical number of MCMC steps (e.g. 40 steps) and ED with M = 32, i.e. maximising the parallelisation capabilities of our GPU.
Finally, here are the references mentioned in the rebuttal and in the following comments.
[1] Du et al. Compositional Visual Generation with Energy Based Models, NeurIPS 2020
[2] Glaser et al. Maximum Likelihood Learning of Unnormalized Models for Simulation-Based Inference
[3] Grathwohl et al. Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One, ICLR 2020
[4] Grathwohl et al. Oops I took a Gradient: Oops I Took A Gradient: Scalable Sampling for Discrete Distributions, ICML 2021
[5] Xu et al. Modeling Tabular Data using Conditional GAN, NeurIPS 2019
[6] Kotelnikov et al. TabDDPM: Modelling Tabular Data with Diffusion Models, ICML 2023
[7] Campbell et al. A Continuous Time Framework for Discrete Denoising Models, NeurIPS 2022
[8] Lou et al. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution, ICML 2024
[9] Schroeder et al. Energy Discrepancies: A Score independent Loss for Energy-Based Models, NeurIPS 2023
[10] Lyu S. Unifying non-maximum likelihood learning objectives with minimum kl con- traction., NeurIPS 2011
[11] Gao et al. Learning energy-based models by diffusion recovery likelihood. ICLR 2020
[12] Luo et al. Training Energy-Based Models with Diffusion Contrastive Divergences
[13] Lee et al. GUIDING ENERGY-BASED MODELSVIA CONTRASTIVE LATENT VARIABLES, ICLR 2023
[14] Foster et al. A Unified Stochastic Gradient Approach to Designing Bayesian-Optimal Experiments, AISTATS 2020
[15] van den Oord et al. Representation Learning with Contrastive Predictive Coding
[16] Le Cun et al. A Tutorial on Energy-Based Learning