Latent Abstractions in Generative Diffusion Models

Rebuttal

We thank the reviewer for the thoughtful feedback and for recognizing the strengths of our work, particularly the connection between nonlinear filtering (NLF) and diffusion models and the complementarity of theoretical and empirical components. Below, we address the reviewer's observations in detail. We understand the reviewer’s concern regarding the presentation of key results and appreciate the suggestion to prioritize Eq. (10) and Figure 1. However, we found during the drafting process of our paper that presenting these elements earlier, without proper introduction, would leave the reader without the necessary context to understand their significance. The sequence of presenting the fundamentals first, followed by their extension to the operational mutual information (MI) equation, is essential to maintain clarity and rigor in our exposition. The connection between theory and experiments indeed hinges on mutual information, as the reviewer interestingly points out. This connection is at the heart of our contribution. While our experiments focus on diffusion models (to reach a broader audience), the theoretical results are intentionally presented in their full generality to emphasize their applicability to a broader class of generative models. The rigorous presentation of the fundamentals ensures that our results are not limited to the specific case of diffusion models but are valid for any model satisfying Assumption 1. **Ultimately, we believe that the merit of our work is that of providing solid theoretical foundations for further research and applications on a broad class of generative models.** *Q1: Could this NLF framework be extended to the deterministic sampling process?* Thank you for the interesting question! It is indeed possible to extend the NLF framework to deterministic sampling processes. However, such an extension requires careful consideration of the differences between deterministic and stochastic dynamics. Specifically: - In deterministic processes, the mutual information between $Y_t$ and $X$ remains constant through time and is equal to its maximum from the outset. This is because the value of the dynamics of the process $Y$ at a given time $t'$ can be obtained from any other value at a different time $t$ (the mapping $Y_t \rightarrow Y_{t'}$ is deterministic and known) - Despite this, our formalism remains relevant, as it helps to analyze when and how such information becomes linearly accessible in the activation space of the architecture under consideration. This is crucial for understanding the role of latent abstractions, even in deterministic settings. This extension highlights the versatility of our framework and its applicability to a wide range of generative models, including those utilizing deterministic sampling schemes.

Rebuttal

We thank the reviewer for the thoughtful feedback and for highlighting important points for discussion. Below, we address them in detail. **Relationship Between Measurements and Latent Abstractions** We agree with the reviewer on the importance of clarifying the relationship between measurements and latent abstractions. This is discussed in lines 323–324, where we explain how to transition from $\mathcal{F}^Y$ (filtration based on measurements) to an enlarged filtration. To summarize, there are two possible cases: - Known Latent World “Ingredients”: If the latent world is known, we have a generative model with an equation in the form of (1), which directly incorporates latent abstractions. - Unknown Latent World “Ingredients”: If the generative process works purely on measurements, latent abstractions can still be introduced mathematically by enlarging the filtration. This step allows us to reason about the hierarchical emergence of abstractions while keeping the theoretical framework general. We will make this connection more explicit in the revised version of the paper to address this point thoroughly, thank you for the suggestion! To further corroborate our experimental findings, we are currently running experiments on the CelebA dataset, where we are observing a similar pattern of hierarchical feature emergence. For example, high-level attributes, such as gender, emerge earlier in the generative process than finer-grained details, such as the presence of eyeglasses, which only appear at later stages. We will place our additional results in the appendix of the camera ready version of the paper. **Relation to Sclocchi et al. (2024)** We acknowledge the connection between our work and the study by Sclocchi et al. (2024), and we appreciate the opportunity to further clarify the differences and similarities. Key distinctions are as follows: - **Scope of Analysis**: The work by Sclocchi et al. focuses specifically on generative diffusion models, while our theoretical framework is broader. As discussed in our response to Reviewer PspT, any model satisfying Assumption 1 can be analyzed using our information-theoretic framework. Diffusion models are a significant instance, but not the sole focus of our analysis. - **Hierarchical Structure**: Sclocchi et al. assume an explicit hierarchical structure for the data (see their Figure 3) and use mean-field theory to study the inference of such variables during the generative process. In contrast, our framework does not assume a pre-defined hierarchy of latent variables. Instead, we show that a hierarchy naturally arises during the generative process, as explained by Eq. (8) and the discussion in lines 224–232. This emergence is a direct consequence of the information-theoretic dynamics of the generative process and does not rely on a specific structural assumption. Note also that our work provides experimental evidence of this phenomenon, see the results obtained in our Figure 3 on the Shapes3D dataset. The attributes that occupy the largest portion of the space (wall hue and floor hue) behave similarly and are the first to appear in the generation process, at $\tau=0.4$. The object hue emerges after these two, at $\tau=0.5$ but before orientation, scale and shape, which appear at $\tau=0.6$. Intuitively, given the nature of the dataset, this hierarchy is reasonable, since (for example) varying the orientation of a shape once the color is fixed, has a lower impact in terms of norms, than changing the color of an object. - **Experimental Comparison**: While the forward-backward experiments in Sclocchi et al. may seem similar to our forking experiments, they are conceptually different. In particular, in the experimental settings described by their Figure 2, it is possible to observe how via a procedure similar to our forking experiments the cosine similarity of the activation functions in a reference ConvNeXt Base vary as a function of the diffusion time $t$. Our forking experiments directly probe when and how latent factors emerge by analyzing coherence across generative pathways and compare such coherence with i) the accuracy of a linear classifier on the hidden activations of the score network and ii) with the mutual information. While the two works are clearly connected, they attack different problems with different techniques. We will revise the manuscript to better highlight these points and further clarify the relationship between the two works.

Rebuttal

We thank the reviewer for the thoughtful feedback and for appreciating the soundness of our theoretical framework. Below, we address the specific questions raised regarding the emergence of the hierarchy of details and the estimation of mutual information. **Emergence of Hierarchy of Details** We agree that understanding the hierarchy of emergent details is a crucial aspect of our work. The submitted paper already provides experimental indications of this phenomenon, such as the results obtained in Figure 3 on the Shapes3D dataset: the attributes that **occupy** the largest portion of the space (wall hue and floor hue) behave similarly and are the first to appear in the generation process, at $\tau=0.4$. The object hue emerges after these two, at $\tau=0.5$ but before orientation, scale and shape, which appear at $\tau=0.6$. Intuitively, given the nature of the dataset, this hierarchy is reasonable, since (for example) varying the orientation of a shape once the color is fixed, has a lower impact in terms of norms, than changing the color of an object (even if the orientation is changed, a reasonable portion of the pixels maintain approximately their value, whereas changing the color modifies all of them). To further corroborate our experimental findings, we are currently running experiments on the CelebA dataset, where we are observing a similar pattern of hierarchical feature emergence. For example, high-level attributes, such as gender, emerge earlier in the generative process than finer-grained details, such as the presence of eyeglasses, which only appear at later stages. We will place our additional results in the appendix of the camera ready version of the paper. **Estimation of Mutual Information** The estimation of mutual information (MI) in our framework is described in the main text and Appendix J. To aid the reviewer’s understanding, we summarize the key aspects of the process below: - Variance-Exploding Scheduler: A variance-exploding (VE) scheduler was selected for these experiments. This corresponds to the limit $\alpha \rightarrow 0$ for $m_t$​ (line 1100 in the appendix). - Denoisers and U-Net Architectures: The two terms in the square norm of Eq. (39) correspond to denoisers for the variable $V$, with or without extra information (the label). These denoisers are implemented using standard U-Net architectures, as detailed in Appendices J.2 and J.4. - Stochastic Process Dynamics: The stochastic process $Y_t$​ evolves according to generation dynamics similar to those described in Eq. (12). Using the unconditional denoisers, it is possible to simulate such dynamics by reparameterizing to obtain the score from the denoisers. - Numerical Integration: The integral in Eq. (39) is computed using the midpoint rule with 999 points for discretization (line 1235 in the appendix). This procedure ensures a robust and accurate estimation of mutual information, which plays a central role in validating our theoretical insights. We will consider including these details directly in the main text to make them more accessible to readers.

Rebuttal

We appreciate the reviewer’s constructive feedback. Below, we provide detailed clarifications and revisions that address the concerns raised. **Not Obviousness and Novelty of Our Contribution** To the best of our knowledge, this is the first work to investigate the emergence of latent abstractions in generative models from an information-theoretic perspective. While empirical studies have highlighted the progressive emergence of high- and low-level abstractions during the generation process, the theoretical underpinnings of this phenomenon have remained largely unexplored. Our framework fills this gap by extending the framework of nonlinear filtering (NLF) and provides a rigorous mathematical description of how such abstractions emerge and influence the generation process. This requires involved mathematical constructs, which we acknowledge can be challenging and require time to be appreciated, but that are essential to justify the gradual emergence of the posterior and its informational dynamics as a stochastic process. Importantly, our approach is general and does not only cater score-based diffusion models, which we use in our work as a simple illustrative example. We agree with the reviewer that score-based diffusion models provide an intuitive lens for understanding generative processes. However, our framework deliberately transcends this scope. Section 4 emphasizes that our findings apply to any generative model satisfying Assumption 1, including neural stochastic differential equations (SDEs) [1], Schrödinger Bridges [2], and stochastic normalizing flows [3]. These models exhibit dynamics where interpreting neural drift as denoising is not straightforward or may not even be feasible. For instance, in Schrödinger Bridges, the connection to a denoising score function is highly implicit. Similarly, in stochastic normalizing flows, the generative pathways often lack a clear denoising interpretation. While the concepts of denoising and score functions are widely accepted in the context of diffusion models, the precise mechanisms through which representations emerge at different stages of the generative process have not been previously formalized. For example: - It is well understood that score functions denoise data. However, it is not clear at which generation stage various details emerge or why simple (linear) classifiers can probe representations within neural network activations. - Our framework elucidates this by linking the emergence of latent factors to information-theoretic measures such as mutual information. **Emergence of Abstractions and Agreement Among Metrics** Our framework provides new insights into the hierarchical emergence of abstractions. Mutual information quantifies the influence of latent factors on observed data and reveals that global abstractions (e.g., semantics) emerge before localized details. This aligns with empirical findings in diffusion models and generalizes to other generative frameworks. Moreover, our work establishes why specific latent representations are encoded in the activations of generative networks. For example: - The sufficient statistics derived in our theoretical framework demonstrate that the posterior (a-posteriori measure) captures all relevant information about latent abstractions. - This explains why latent abstractions emerge gradually, supporting their hierarchical interpretation. A particularly compelling aspect of our empirical results is the exceptional agreement between: - **Mutual Information [our original contribution]**: Theoretical predictions of latent factor emergence based on information-theoretic measures. - **Entropy of Forking [our original contribution]**: Entropy reduction across forked generative pathways, which directly tracks the commitment of the generative process to specific latent abstractions. - **Linear Probe Accuracy**: High probing accuracy on features extracted from the denoising score network at corresponding noise levels. This agreement across three distinct metrics provides a robust empirical validation for our theoretical framework and supports its applicability to real-world generative models. Such consistency underscores the usefulness of our approach for analyzing the interplay between latent abstractions and observable data in generative models. We truly hope that the points above will help the reviewer to appreciate the extent of our theoretical contribution, and that the struggle and time spent to understand our paper (for which we deeply thank the reviewer) will not weigh negatively on the final assessment. [1] Kidger et al, Neural SDEs Made Easy: SDEs are Infinite-Dimensional GANs, ICLR 2021 [2] De Bortoli et al, Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling, Neurips 2021 [3] Hodgkinson et al, Stochastic continuous normalizing flows: training SDEs as ODEs, UAI 2021

Comment

Thank you for your response! I'd appreciate your elaborating further on the significance of this work. You state >To the best of our knowledge, this is the first work to investigate the emergence of latent abstractions in generative models from an information-theoretic perspective. Formally describing the emergence of latent features is perhaps interesting to the information theory community, who may take for granted the importance of seeing if their theory can explain the behavior of diffusion models. But the larger ML community isn't necessarily surprised by the emergence of latent features; one may feel there is nothing to explain as the emergence of latent features is a fundamental behavior of almost all deep learning models (explained using "compression", PAC-Bayes, etc...). What would be nice is if you could advocate for the utility of the information theoretic perspective: does this theory have the potential to help us build better models? Does it offer a satisfying explanation of previously surprising empirical results? Does it suggest new experiments? Does it give us predictive power over the behavior of these models that we didn't have before?