Discrete Distribution Networks

Official Comment by Authors

Thank you for your inspiring feedback and appreciation. **“Theoretical analysis is missing. No mathematical derivation that shows why the distribution of generated images should converge to the real data distribution.”**: We primarily demonstrate the effectiveness of the Gradient Descent with Split-and-Prune algorithm through detailed illustrations and 2D toy experiments. We acknowledge the necessity for further research to understand and formalize the underlying mechanisms. **“Even if this number is big, this suggests that the model essentially is a compressed way of storing all its possible results in a tree based database. This might make a problem to scale up. Since the demonstrated data relatively has few dimensions, this suggests that actually holding the entire dataset might require more or close storage to the model itself. A quick calculation to demonstrate: MNIST has 70k images (train+test) of 28x28, this is 54880000 integers: ~55MB. EDM smallest model has 62M parameters, even if we take these as 1 byte each it is heavier.”**: Admittedly, we are not entirely clear about the background and motivation of this example, but we will attempt to address it from several perspectives: - DDNs perform lossy compression, meaning they do not store every byte of the dataset precisely. Consequently, when reconstructing from the latent space, the generated images will differ from the originals (measurable by L2 loss). Thus, the notion of "actually holding the entire dataset" may not apply directly; instead, the model's parameter count influences the reconstruction accuracy (reflected in reconstruction loss and rFID). - Additionally, latents also contribute information. Although the original ground truth (GT) does not directly input into the network, it influences the DDN through the guided sampling process, transmitting information in latent form. Hence, both network parameters and latents collectively enable the reconstruction of training samples, meaning the dataset's information is carried by both the model parameters and the latents of the entire dataset. - Furthermore, we believe that under the same storage space, the intrinsic information content or complexity of different datasets can vary significantly. Most MNIST pixel values are either 0 or 255, and black pixels near the edges are deterministic and thus carry no information, allowing the model to fit MNIST well. However, replacing this with CIFAR-style images of the same pixel count would significantly increase the modeling difficulty. Therefore, solely relying on the uncompressed storage space of the dataset to reflect the fitting difficulty of a model with a certain parameter size can lead to substantial discrepancies. **“I am adding the weakness of low quantitative results and lack of demonstration of higher scales. To be clear, I think it is legitimate for novel methods but it is still a weakness.”**: As you noted, achieving state-of-the-art (SoTA) results with a novel method like DDN requires more research and effort. We are currently focusing on enhancing the generation quality of our model for Zero-Shot Conditional Generation (ZSCG) tasks. We hope this response addresses your concerns. Thank you for considering our work.

Official Comment by Authors - (1/2)

Thank you for the thoughtful consideration of our paper and the constructive feedback. **1 “model is validated on small scale datasets, it does not compare against more modern approaches”**: As a novel method, achieving state-of-the-art (SoTA) performance with DDN requires further research and effort. We are currently focusing on enhancing the generation quality of our model for Zero-Shot Conditional Generation (ZSCG) tasks. **2 “The method relies on generating multiple samples in parallel and requires multiple layers for further refinement. This seems quite expensive and hard to scale to higher resolutions.”**: - Due to the hierarchical conditional generation approach, similar to autoregressive(AR) models, DDN faces same challenges as AR models in generating high-quality, high-resolution images, which significantly increases computational cost. - We are currently developing a new approach to enhance the efficiency of high-dimensional data representation by expanding the latent space from $K^L$. This involves dividing an image into $N$ patches, independently selecting the optimal patch from $K$ candidates for each patch, and then merging these selected patches into an image. This image serves as the output of the current DDL layer and is input into the next layer, thereby increasing the latent space to $K^{N \cdot L}$. - Another potential solution is to draw inspiration from Latent Diffusion and VQ-VAE, enabling DDN to perform generative modeling in a lower-complexity latent space. **3 “The placement of tables and figures reduces the quality of presentation.”**: We have revised the manuscript to improve its clarity and flow. Specifically, we have moved Figure 5 (2-d toy example) to the Appendix to streamline the main text. Additionally, Table 2 has been relocated to be adjacent to Section 4.2, the Ablation Study, to enhance the logical coherence and accessibility of relevant data. **4 “Can DDN scale to larger and higher resolution datasets, such as FFHQ 256x256, or ImageNet 256x256?”**: - In Appendix Figure 9, we demonstrate conditional generation on FFHQ 256x256, where additional information helps reduce the generative space, allowing DDN to produce clear 256-resolution images. - For unconditional FFHQ 256x256, the current version of DDN can generate recognizable faces but with blurred contours and lacking high-frequency details. - The current version of DDN struggles to generate coherent ImageNet 256x256 images, primarily due to the small model size (74M parameters). Scaling up the model parameters is necessary. For reference, Image GPT (https://openai.com/index/image-gpt/) uses a 1.4B parameters model to generate 96x96 ImageNet images. **5 “What are the parameter counts of the baselines compared?”**: We have supplemented the parameter counts based on the survey paper on generative models (https://arxiv.org/abs/2103.04922): | Method | Type | FID↓ | param(P) | | --- | --- | --- | --- | | DCGAN | GAN | 37.1 | P < 10M | | IGEBM | EBM | 38.0 | P < 10M | | VAE | VAE | 106.7 | P < 10M | | Gated PixelCNN | AR | 65.9 | P > 120M | | GLOW | Flow | 46.0 | 60M < P < 120M | | DDN(ours) | DDN | 52.0 | P=74M |

Official Comment by Authors - (2/2)

**6 “How does a single shot DDN compare against a default DDN allowing for the same number of refinements?”**: - There might be some misunderstanding here. The single-shot paradigm is our default paradigm, leveraging its coarse-to-fine nature to enhance computational efficiency. Only the MNIST experiment used the recurrence iteration paradigm of DDN to demonstrate its effectiveness. To avoid confusion, we have revised the paper to emphasize that single-shot is the default paradigm for most experiments. - In comparing single-shot paradigm with the recurrence iteration paradigm, given the same parameter count and number of refinements, recurrence paradigm requires L times more computation since each parameter is processed L times to obtain the final result. Moreover, the recurrence paradigm cannot effectively utilize the coarse-to-fine nature, resulting in lower computational efficiency. Thus, the single-shot paradigm is a more favorable choice. **7 “Generating multiple samples in parallel for sampling is expensive. What are the memory requirements compared to the baselines in the paper?”**: - Our memory requirements are slightly higher than same architecture of conventional GAN generator, but the difference is negligible. - During training, generating $K$ samples is only to identify the one closest to the ground truth, and the $K-1$ unchosen samples do not retain gradients, so they are immediately discarded after sampling at the current layer, freeing up memory. - In the generation phase, we randomly sample one number from range($K$) as an index and only generate the sample at the chosen index, avoiding the need to generate the other $K-1$ samples, thus not occupying additional memory or computation. - Furthermore, in the single-shot paradigm, which is our default, a Discrete Distribution Layer (DDL) is introduced every two convolution layers. The DDL contains $K$ samples of the same size as the feature. The memory consumption of features stored in convolution layers, including conv and normalization layers, is generally significantly greater than that of $K$ samples. Similarly, the computational cost of convolution layers is significantly higher than that required to compute $K$ samples. Overall, the additional computational and memory overhead introduced by DDLs compared to conventional NN blocks is not significant. We hope this response clarifies our contributions and addresses your concerns. Thank you for considering our work.

Official Comment by Authors - (1/2)

Thank you for the thoughtful consideration of the paper and constructive feedback. **1 “does not explicitly model probability distributions”**: We respectfully disagree with this assessment. As described in Line #97-102, DDN explicitly approximates the target distribution by fitting the training data with a discrete distribution. The network generates multiple samples (K) simultaneously, which collectively represent a discrete distribution. Each generated sample has an equal probability mass of 1/K, and our goal is to make this discrete distribution as close as possible to the target distribution. Figure 17 (Toy examples for two-dimensional data generation) demonstrate that DDN can successfully fit various target distributions by representing them as discrete distributions composed of multiple samples. Additionally, we believe it is unfair to negatively evaluate a generative model solely because it does not explicitly model probability distributions. For instance, Generative Adversarial Networks (GANs) learn distributions implicitly without explicitly estimating probability densities. **2 “Does not model some complex distributions, nor empirically useful due to its worse performance”**: 1.DDN is capable of modeling complex distributions, as evidenced by its application to the FFHQ and CIFAR10 datasets, which are considered complex. Few foundational generative models can effectively model such complexity (five models from https://arxiv.org/abs/2103.04922). DDN, with its simple yet effective concept, demonstrates its capability to model these distributions effectively. 2.As other reviewers have noted: > "I do not think it is fair to hold this novel method to the same standard as more matured, modern approaches." -- Reviewer 1dyp > "Such papers should always be judged under the understanding that for existing methods, a lot of engineering has taken place and should be thought of as the first GAN paper or the first Diffusion models paper." -- Reviewer r4YK We emphasize that DDN is a novel generative model, representing the first work of its kind. It is not fair to compare it to more mature methods. Further research is needed to enhance DDN's performance, particularly in developing better Zero-Shot Conditional Generation models. 3.Moreover, our paper compares DDN with IGN (https://openreview.net/forum?id=XIaS66XkNA), another new generative model proposed at ICLR 2024. On the CelebA-64x64 dataset, DDN outperforms IGN in terms of FID scores and qualitative analysis (Figure 6 Line#412). 4.DDN possesses unique properties not found in other generative models, enabling novel applications. As noted by Reviewer 1dyp: > The ability to condition generation on external signals without gradients is a useful property. **3 “the paper claims that VQ-VAE is unsatisfactory (e.g., in the abstract) but it does not even compared against VQ-VAE.”**: 1. We clarify that VQ-VAE is not mentioned in our abstract or introduction. It is only discussed in the related work section, where we note its 2D latent structure and reliance on an additional prior network for generative tasks. 1. We were unable to find standard FID scores for VQ-VAE on CIFAR10 in several key papers, including VQ-VAE, VQ-VAE-2, and a comprehensive survey on generative models (https://arxiv.org/abs/2103.04922). 1. DDN is a new generative model and we prefer to compare it with other foundational generative models. VQ-VAE, being an extension of VAE requiring an additional generative model (e.g., an autoregressive model) for latent space modeling, is not a fundamental generative algorithm. Thus, we did not attempt to reproduce its results for comparison. 1. Our focus is on introducing a novel generative model with unique properties like a 1D latent space and Zero-Shot Conditional Generation without gradients, rather than focusing solely on performance metrics.

Official Comment by Authors - (2/2)

**4 “Similar to VQ variant of diffusion models”**: While both DDN and VQ-Diffusion model distributions and involve direct input/output of samples, they differ significantly: - Input/Output: VQ-Diffusion processes a single noisy sample at each step, outputting a less noisy sample. DDN, however, takes the selected output sample from the previous layer (**without noise**) and generates **multiple samples** (representing a discrete distribution). - Training: VQ-Diffusion computes gradients independently for each step, whereas DDN computes joint gradients across all layers and uses the "Split-and-Prune" algorithm to balance sampling probabilities. - Data Scope: VQ-Diffusion models only discrete data (VQ-VAE's latent), while DDN can model both discrete and continuous data. - Latent Space: VQ-Diffusion lacks an intuitive latent space and cannot reconstruct samples, unlike DDN, which has a 1D latent space enabling reconstruction. - Zero-Shot Conditional Generation: DDN supports this without gradients, which VQ-Diffusion and other models do not. **5 “fails to connect the proposed method to a lot of existing papers”**: Given the novelty of our method, direct connections to existing methods are limited. However, we highlight the following: - The approach of partitioning the large space into a hierarchical conditional probability model is inspired by autoregressive models. - The "Split-and-Prune" algorithm draws inspiration from evolutionary theory and genetic algorithms. - The technique of learning residuals is inspired by ResNet. - The form of Zero-Shot Conditional Generation(ZSCG) is inspired by SDEdit (https://arxiv.org/abs/2108.01073). We have further emphasized these connections in the revised version of our manuscript. **6 “what is different and novel”**: - DDN introduces a new generative modeling approach that approximates data distributions using hierarchical discrete distributions, achieved through the simultaneous generation of multiple samples. - It supports zero-shot conditional generation across non-pixel domains without gradients, which other models do not. - The latent space is a configurable tree-like structure which differ from autoregressive models. - It employs a novel "Split-and-Prune" algorithm to balance sampling probabilities across multiple samples. Other reviewers also acknowledge the novelty of our approach: > "The novelty is very strong, and this should not be overlooked. This is a whole new method, very different from any of the existing generative models." -- Reviewer r4YK > "The idea of sampling an image and autoregressively feeding it back in through each layer in a forward pass is novel to the best of my knowledge." -- Reviewer 1dyp **7 “why these design choices.”**: - **Generating multiple samples at each layer:** We posit that since network features capture distributional information, generating multiple samples simultaneously can effectively represent distributions (Line #011 in Abstract). - **Hierarchical conditional generating:** Current neural networks cannot generate a vast number of samples simultaneously. We adopt a strategy from autoregressive models to partition this space into a hierarchical conditional probability model (Line #105 in Introduction). - **Split-and-Prune:** We discuss the necessity of this algorithm to balance sampling probabilities across multiple samples in Section 3.1 Optimization with Split-and-Prune (Line #252). We hope this response clarifies our contributions and addresses your concerns. Thank you for considering our work.