Hamiltonian Score Matching and Generative Flows

Summary

This paper proposes a new matching method based on Hamiltonian mechanics. The proposed method, algorithmically, is essentially a second-order ODE in which the acceleration (i.e., the drift of the velocity channel) is approximated by deep neural networks. The method is well grounded on the theory of classical Hamiltonian mechanics and statistical physics. As a higher-order system, the proposed method naturally absorbs prior diffusion and flow methods as special cases. Experiments were conducted on a synthetic dataset (Gaussian mixtures) and image generation.

Strengths

- Presentation is overall clear and easy to follow. I like how the authors highlight important sentences for improving readability. Figs 1 & 2 are also instructive in showcasing the advantage of their method. - Theoretical contributions are solid. I think many of their results in Sec 4 may as well be of interested for readers from other domains.

Weaknesses

- It's unfortunate that, from an algorithmic standpoint, the resulting algorithm---after all these nice theoretical results---remains somewhat similar to a few recently-proposed second-order methods, e.g., AGM (https://arxiv.org/pdf/2310.07805 ICLR'24 oral) or CLD (ref. [12]). - The use of harmonic oscillator in Sec 7 is not very motivated in my opinion. There could be other alternatives that also fulfill C1 & C2 conditions. Practically speaking, sampling (x,v) in (23) from sin/cos doesn't seem to encourage straighter generative process. - (minor) Typo in L236, I think you meant C2 not C1. Overall, I have a mixed feeling on this paper --- on one side, I do enjoy, and acknowledge, their theoretical contributions, yet on another side, I feel like the resulting algorithm is rather weak.

Questions

- Can the authors comment on the differences compared to AGM? I think AGM also constructs simulation-free (xt, vt) samples then matches the drift of the velocity. - Have the author ever tried to jointly train both $\theta$ and $\phi$, as mentioned in L179? I understand this may most likely be impractical but just curious from an exploratory standpoint. - Could the authors elaborate more on L175 the "optimal" choice of T, and the first hypothesis in L184?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Limitations were mentioned in Sec 10.

Summary

The paper proposes the Hamiltonian Score Matching framework, which is a new general framework of generative models. Inspired by the Hamiltonian dynamics in classical and statistical mechanics, the framework uses the Hamiltonian dynamics to generate data, which is also called Hamiltonian generative flow in the paper. The paper theoretically proves that the force field coincides with the score function if and only if its optimal velocity predictor is zero and proposes the HSM loss function. The paper also shows that the Flow Matching framework and the diffusion model are special cases of the HSM framework. Additionally, a new generative flow called Oscillation Hamiltonian generative flow is constructed. Experiments on several datasets show the experimental performance of the HSM framework.

Strengths

1. Very clear writing. 2. The method is novel and easy to understand. 3. The paper clarifies the connection and difference with previous works.

Weaknesses

1. The experimental result is not so good. As the paper writes, the Oscillations HGFs can surpass most baselines but still lack behind the EDM model. I think more experimental results should be provided to show that the HSM framework is a good framework for generative models.

Questions

What is the motivation for designing Oscillation HGFs? I'm curious about how to design a force field in the HSM framework.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

The authors introduce Hamiltonian velocity predictors (HVPs) as a tool for score matching and generative models. They present two innovations constructed with HVPs: Hamiltonian Score Matching (HSM), a novel generative model that encompasses diffusion models, and flow matching as HGFs with zero force fields. They showcase the extended design space of force fields by introducing Oscillation HGFs, a generative model inspired by harmonic oscillators. They show experiments validating the theoretical insights about HSM.

Strengths

The paper presents an interesting generative model by leveraging Hamiltonian velocity predictors. The authors give a theoretical analysis of the proposed method and its connections with existing methods. The authors also provide an experimental study to support the proposed method.

Weaknesses

a. Why does it produce inferior performance on image generation compared to EDM? The author could present the reasons for this. b. The authors also need to present the computation cost/efficiency comparison of different methods to validate the proposed method.

Questions

See weakness.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors did not provide a clear discussion of the limitations of the proposed method. This should be included in the paper.

Summary

In this work, the authors proposed a new generative modeling approach called Hamiltonian Score Matching, which is motivated by classical Hamiltonian mechanics. This approach estimates score functions by augmenting data via Hamiltonian trajectories, and further motivates Hamiltonian Generative Flows. The authors also discuss the design space of force fields in Hamiltonian Generative Flows and connect it with harmonic oscillators. Experiments are conducted to verify the effectiveness of the proposed approach.

Strengths

1. The connection between Hamiltonian dynamics, force fields, and score matching is interesting and provides a new perspective for the design space of generative models. 2. The paper is well-written and easy to follow. 3. The authors provide a clear proof section to check the correctness of propositions and theorems.

Weaknesses

1. Regarding the methodology: The authors spend efforts to establish Hamiltonian Generative Flows from Parametrized Hamiltonian ODEs, Hamiltonian velocity predictor, and Theorem 1 for Hamiltonian Score Matching. Although the benefits such as bringing additional freedoms for force fields and prediction objectives, the scalability of this approach is restricted as stated by the authors. The necessary trajectory simulation and min-max optimization largely limit the usage of the proposed approach. In Section 7, the authors instead seek to use a pre-defined force field instead of parametrized models, which makes the HGFs degenerate to pre-defined ODEs, further indicating the theory-practice gap of the proposed approach. 2. Regarding the experiments: scales and types of experiments in this work are rather limited. In addition to the simulation experiment, the authors only conducted experiments on CIFAR-10, which even shows that the proposed HGFs cannot bring explicit benefits compared to previous approaches. It would largely improve the quality of this work if the authors could provide more diverse and strong experimental results for the proposed approach. Overall, I think the paper provides an interesting perspective, but the current issues discount the ratings of my evaluation. Currently, I vote for borderline acceptance and I will carefully read the rebuttal and other reviews to decide whether to increase or decrease my scores.

Questions

See the above section.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors discussed the limitations of this work in the Conclusion section.

Summary

The paper explores the application of Hamiltonian formalism for generative modelling. In this framework, the score function is interpreted as a force field F, thus optimizing the parameters of F yields a score-matching objective. Authors build the relation to Flow Matching and Diffusion models as special cases of their framework and suggest another special case, which they validate on an image generation task.

Strengths

- the formulation is novel and implies conservation of energy and volume. - the theoretical foundation and contribution of the work is very solid. - the authors derive diffusion models and flow matching as a special case of HGFs. - the experimental results are solid and indicate the impressive capabilities of the framework.

Weaknesses

- in the general case, the framework requires joint learning of two functions: force field and velocity. The authors themselves compare the optimization to GANs, which raises the question of training stability. - the authors do not compare the runtime and memory consumption of the framework with other SOTA methods, e.g. Flow Matching. - the experimental section is somewhat limited to a toy experiment and a single image-generation task. It would be highly beneficial to evaluate the framework on another task (e.g. physics-related), although it is not critical given the high theoretical contribution of the work. - it is not clear how image generation is influenced by Hamiltonian dynamics. What is x, and what is v?

Questions

- I am curious if it would be simple to enforce equivariance as in Equivariant Hamiltonian Flows by Rezende et al. - as the force field form is a choice now, did authors consider alternative forms other than osciallators? I am curious what is the intuition behind the choice. - authors mentioned conservation properties in the context of physical simulations. Do you think that they will still be beneficial in the general case, e.g. image/video generation? Even in physical systems, if the energy is not conserved, would those properties be a limiting factor somehow?

Rating

8

Confidence

3

Soundness

4

Presentation

3

Contribution

4

Limitations

Perhaps the main limitation for me is the discussion of the framework's theoretical properties, e.g., conservation of energy and volume, in the context of generative modelling. It would be beneficial to see an analysis of how those inductive biases will play out in specific cases (e.g., physical simulations or image generation). Besides, it would be beneficial to provide computational details (runtime / memory) and compare them to SOTA methods.

Author-Reviewer Discussion Phase (ends Aug 13 midnight AOE)

Dear reviewers, Thank you for your efforts so far! The authors have responded to each of your reviews and have also provided some additional results in the PDF attached to the overall response. Could those you take a moment to read the author response and follow up regarding any points that may further inform your opinion on the paper? With thanks, AC

Official Comment by Reviewer g7vb

I thank the authors for their response and additional experiments, which certainly prove the positive contribution of the submission. I will retain my score.