We propose a new score-based approach to generate 3D molecules represented as atomic densities on regular grids. First, we train a denoising neural network that learns to map from a smooth distribution of noisy molecules to the distribution of real molecules. Then, we follow the neural empirical Bayes framework (Saremi and Hyvarinen, 19) and generate molecules in two steps: (i) sample noisy density grids from a smooth distribution via underdamped Langevin Markov chain Monte Carlo, and (ii) recover the"clean"molecule by denoising the noisy grid with a single step. Our method, VoxMol, generates molecules in a fundamentally different way than the current state of the art (ie, diffusion models applied to atom point clouds). It differs in terms of the data representation, the noise model, the network architecture and the generative modeling algorithm. Our experiments show that VoxMol captures the distribution of drug-like molecules better than state of the art, while being faster to generate samples.
Paper
Similar papers
Peer review
Summary
This paper describes a new method of unconditional small molecule generation by parameterizing small molecules as 3d voxel arrays.
Strengths
- Novel characterization of the small molecule generation task and a new way to parameterize the data. This comes with benefits, chief among which is the ability to generate without a known number of atoms. - I appreciate the discussion in section 4.3, where the authors dive into some problems with the metrics that they have chosen. - The ablation studies are illuminating. - The paper is well-written and easy to follow.
Weaknesses
- Unconditional generation of small molecules is a hard task to evaluate. There are no good metrics to optimize for, as the paper indicates. Nevertheless, MiDi (work that the authors cite and compare against) has a more representative set of metrics they compare against. I suggest the authors do the same. - Weaknesses of the one-step denoising procedure are not discussed. These include, for example, the inability to control generation through the use of auxiliary gradient signals, such as classifier-free guidance. - The authors don't mention the implications of having a fixed grid size for their generative model. This means that the molecules generated need to be within a specified volume, restricting the output space. They should contrast this to the fixed atom number of competing models.
Questions
- Line 243 should be as noise *decreases*, correct? - We consistently see a large drop in the validity when the noise level goes from $\sigma = 0.9$ to $\sigma = 1$, why is such a small change in noise so detrimental? What are the characteristics of the generated molecules at these levels of noise? What happens if you train a model with $\sigma = 2$? - What is the relevance of the number of aromatic rings per molecule? Especially given that this is one of the only metrics where the proposed model outperforms, I would like a good rationale for including this metric. - What is the relevance of the validity metric, if that is something that can be automatically detected? It seems in this case that generating unique molecules is more relevant, especially if there are no other metrics to optimize. - How can this model be extended to allow for conditional generation, given that most of the other methods being compared to have this ability?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Limitations
Mainly, the authors should find better metrics to evaluate their work, such as those mentioned in the MiDi paper (at a minimum).
Summary
This paper proposes VoxMol, a novel method for generating 3D molecules in the form of voxel grids. The proposed method adopts neural empirical Bayes as the basic probabilistic framework to develop generative models for 3D voxel grid representations of molecules. Experiments are conducted to show that the proposed method can successfully generate valid 3D molecules.
Strengths
- This work proposes a novel method of generating 3D molecules in the form of 3D voxel grids by neural empirical Bayes framework. The exploration of generating 3D voxel grids representation of molecules is interesting and meaningful to the development of efficient drug discovery applications. - The writing of this paper is well-organized and easy to follow.
Weaknesses
- In this paper, generating 3D molecules in the form of 3D voxel grids by VoxMol is argued to be advantageous compred with 3D molecular graphs because the number of atoms is required to know beforehand, the distributions of continuous atom coordinates and discrete atom features are not treated separately, and simpler and faster training and generation can be achieved. However, in my opinion, these are not strong advantages. (1) Actually, the number of atoms can be sampled from a learned distribution in 3D graph generation model, and VoxMol is also implicitly learn the number of atoms to place in every channel of voxel grids. These two processes are not fundamentally different. (2) It is not a big issue to handle discrete and continuous distributions separately as we can jointly train discrete and continuous generative models (e.g., discrete & contiuous diffusion models). Also, VoxMol need to implicitly learn a "discrete distribution" as for every 3D spatial location in the voxel grid, it needs to decide the atom type, i.e., which channel to place an atom. (3) The simpler and faster training and generation are mainly due to the use of neural empirical Bayes method, which may also be applied to 3D molecular graph generation to achieve fast generation and training. Also, 3D molecular graph generation can be accelerated by some other methods like introducing fragment-based generation. Hence, it is not convincing that the proposed VoxMol method is really advantageous when compared to existing graph-based method. - The use of 3D voxel grids also introduce many disadvantages. 3D voxel grids lose SE(3)-invariance as they are not invariant to rotations or translations. The use of 3D voxel grids hamper the introduction of chemical domain knowledge to guide the generative models to produce chemically valid or drug-like molecules, such as adding constraints based on fragments. The effect of these disadvantages should be evaluated or discussed. - The experimental results in Table 1 do not demonstrate the advantages of the proposed VoxMol method as VoxMol does not outperform strong baseline methods in several metrics.
Questions
In experiments, methods are evaluated by the average number of aromatic rings per generated molecule. Do aromatic rings are detected from 2D molecule graphs inferred from 3D structures? Are the generated aromatic rings are checked in 3D space to decide whether they respect chemical constraints (i.e., all atoms in an aromatic ring should be in the same plane)? Why this metric is used? What can be shown if a method can generate more aromatic rings?
Rating
3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
Yes.
Summary
This paper proposes a novel 3D molecule generation routine dubbed VoxMol. The highlight of it lies in its introduction of a connection between traditional molecular graph representation and 3D voxel representation. In VoxMol, the molecules are first translated into voxel representation. A denoising network is then trained to serve as a score approximation. In the sampling process, a novel walk-jump procedure is introduced to sample noisy p(y) and obtain the denoised x.
Strengths
1. To my knowledge, the method is novel in terms of data representation and the training/sampling process. The paper showcases that instead of the traditional graph representation, molecule generation can also be performed on 3D voxels, shedding light on new research directions of connecting 3D CNNs to equivariant GNNs. The authors also tried to recover the molecular graph representation based on the denoised voxel, which I really appreciate. 2. The evaluations are interesting. Though not always state-of-the-art, the authors have exhibited insightful investigations, e.g., how $\sigma$ influences the tradeoff between validity and uniqueness. 3. The overall presentation is clear and the method is very easy to follow.
Weaknesses
1. I feel a little pity that equivariance is not properly injected into the model. I think it would make a much stronger impact if equivariance could be enforced even on this 3D voxel representation so that physical/chemical priors could be ensured. 2. The method does not significantly beat EDM on these benchmarks. But I understand that as perhaps the first work to challenge the task of 3D molecule generation with 3D CNNs, it is difficult to beat all other GNN counterparts at once.
Questions
As the authors have stated in the paper, efforts have been made for equivariant 3D CNNs but were unsuccessful. I am curious about what detailed experiments have been conducted and how were the results. I believe by involving equivariance the work will make a stronger contribution.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The limitations are not discussed in the main text. I would recommend the authors add these discussions since I think the work is inspiring and will benefit more from future research.
Summary
This paper proposes doing diffusion on voxel space for molecule conformation generation. Because the voxel space is discrete, an efficient sampling method is proposed.
Strengths
This paper proposes doing diffusion on voxel space for molecule conformation generation. Because the voxel space is discrete, an efficient sampling method is proposed.
Weaknesses
## Method: - It is not clear how the SE(3)-equivariant property is guaranteed in the model. This is critical because once the molecules are too large to fit in the voxel grids, we need to rotate it for a better position. - The objective function in Eq 2 is not clear. Concretely, it is not clear how the atom types are trained. ## Experiments: - The authors also referred to MiDi in the paper. I’m wondering why the authors did not add MiDi? Is that because the performance of MiDi is better than VoxMol? - The MiDi paper introduces more domain-specific metrics, and authors should consider add them. - Also a closely-related baseline should be cited at least: https://arxiv.org/abs/2305.05708. ## Minor: - `... capturing long-range dependencies over multiple atoms can become difficult …` It also depends on how the edges are constructured. For point clouds, the edges are often constructured by distance, thus this may not become an issue as long as the spatial distance is close.
Questions
I have some questions on the method section, and would like to confirm with the authors. - I am wondering what’s the advantage of using NEB for estimating p(x)? Based on lines 135-140, it seems to be a standard EBM with Langevine dynamics, puls an extra NEB module. Is this correct? - The authors mentioned that `Chains are initialized with Guassian noise with standard deviation`. I would like to double-check what does standard Guassian mean in the voxel grids?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
N/A
Follow-up Responses
I appreciate authors' hard work in the rebuttal. While some of my concerns have been addressed, several key concerns still remain unsolved to me. - I agree that EDM cannot implicitly learn the number of atoms and implicitly learning it is useful in some conditional generation applications. But graph-based autoregressive generation methods can also achieve it by generating a "STOP" token to indicate the stop of sequential generation while not specifying the number of atoms at start. As authors also agree that jointly learning continuous & discrete distributions are not impossible and simpler training comes from NEB instead of image-based training, I suggest to remove the strong claim that image-based generation is better than graph-based generation in the paper revision, but simply treat the image+NEB as an independent method and discuss its unique advantages. - I agree that the number of aromatic rings is a meaningful metric. However, as authors mentioned, aromatic rings are identified by aromatic bonds, so how the aromatic bonds are detected from the generated 3D molecular structures? Are they detected by by interatomic distances? However, even generating a 6-aromatic-bond cycle graph is not enough for an aromatic ring, as by chemistry knowledge, in a valid aromatic ring, all atoms (6 carbon and 6 hydrogen atoms) should be in the same plane. Do authors check if this constraint is satisfied? - Authors are recommended to discuss the effect of hampering fragment-based generation. Here, fragments are those functional groups or subgraphs frequently appeared in molecules, such as aromatic rings or alkene. By chemical knowledge, the 3D structures of these functional groups have many constraints, e.g., containing some unrotatable bonds that fix some atoms in the same plane. These chemical constraints can be well incorporated into graph-based generation methods, as we can first generate 3D "super-node" graph where each super node is a functional group, then refine the 3D atomic structures in each super node (an example of this method is [1]). However, image-based methods cannot achieve it in my opinion, as images separate atoms in a functional group into different channels in a 3D image volume. This leads to a major concern that molecules generated in the form of 3D images may consistently break chemical constraints and are not practically synthesizable. Due to the above concerns, I tend to keep my decision of rejection. [1] Coarse-to-Fine: a Hierarchical Diffusion Model for Molecule Generation in 3D. ICML 2023.
We thank R3 for the prompt reply. We provide additional discussions in the hope of better answering R3's questions. Below, we clarify the three concerns raised by the reviewer (see below for detailed clarifications): - **Point 1:** we agree with each other. As suggested, we will remove the three initial claims and focus on the model's advantages only. - **Point 2:** we agree the # arom. rings metric is not very informative (we use RDKit to compute it). We follow reviewers suggestions and use MiDi metrics, as it is better for benchmarking. Table 2 on the attached pdf shows that our model clearly beats EDM by a large margin. - **Point 3:** voxel-based models can in principle do fragment-based generation/have chemical constraints (eg, [A, B, C] do this). However, the contributions of this work are (i) proposing a new model for unconditional generation and (ii) beating the SOTA on a large/challenging dataset on unconditional generation. Fragment-based generation is beyond the scope of this paper and can be seen as future applications (same for structure-based or scaffold-based generation). We hope the clarifications above—together with results on GEOM-drugs (better than EDM by a large margin)—can change the reviewer's opinion. If not, please let us know where we can be more clear. Are these three points all the concerns the reviewer have? We would be happy to address any other concern the reviewer might have. --- Detailed comments: - **Comparison with autoregressive (AR) models:** when we mention about learning the #atoms implicitly, we were comparing VoxMol with diffusion models (current SOTA) and not with AR models. AR models have their own advantages/disadvantages (eg, it has not been successfully applied on GEOM-drugs). - **Continuous/discrete spaces:** authors and reviewer agree on this. - **Number aromatic ring:** we computed the number of rings per molecule with RDKit (it is a simple “2D metric”, computed on valid molecules, like many of the other metrics used in this task) and therefore it does not check if atoms are on the sample plane (we observed qualitatively that they do many times). We agree this is not the most meaningful metric and will not report it. Table 1 and 2 on joint pdf show results on MiDi metrics that are more useful for benchmarking models (proposed by reviewers). - **Fragment-based generation**: - Voxel-based models can in principle do fragment-based generation/incorporate chemical knowledge. These constraints can be imposed either (i) during sampling (similar to in-painting, eg, initializing the chain with fragments and keeping them unchanged during sampling) or (ii) apply fragment constraints during training. - _“However, image-based methods cannot achieve it to me, as images separate atoms in a functional group into different channels in a 3D image volume.”_: Previous works have shown evidence that this is possible (eg [A, B, C]). In fact, some chemical priors like 3D pharmacophores are particularly easy to incorporate with voxels. - This paper is about proposing a new model for unconditional generation and beating the SOTA on a large/challenging dataset. Fragment-based generation is beyond the scope of this paper. [A] DeepFrag: a deep convolutional neural network for fragment-based lead optimization, Green et al, 21 [B] Deep generative design with 3D pharmacophoric constraints, Imrie et al, , 21 [C] Incorporating target-specific pharmacophoric information into deep generative models for fragment elaboration, Hadfield et al, 22
Response
I appreciate authors' response. However, my concerns have not been fully addressed. - As authors agree that the original claimed advantages of image-based generation are not persuasive, it seems that there are not outstanding advantages of the proposed method over existing graph-based 3D molecule generation methods. Hence, I think authors should discuss what are the outstanding and unique advantages of the proposed method in the revision of paper, otherwise I am worried that the contribution of this work does not reach the bar of top-tier conferences like NeurIPS. - I do not think fragment-based generation is beyond the scope of this work. This work studies unconditional generation of 3D molecules, and this problem is tightly related to real-world molecule design application. In real-world applications, we have to consider whether the designed 3D molecules satisfy chemical constraints, because a 3D molecule violating these constraints will even not stably exist in nature, let alone be synthesized. This is the central reason why I emphasize fragments, because many chemical constraints are in the level of fragments or functional groups. For graph-based generation, chemical constraints can be intuitively satisfied by considering fragments as super nodes and using coarse-to-fine generation, but in contrast, image-based generation seems not to have an intuitive way to satisfy chemical constraints. But authors have pointed out some papers about this topic, hence, in the revision of paper, I suggest authors to discuss the possibility of introducing chemical constraints or other domain knowledge into the proposed method so as to improve the synthesizability of the generated molecules. - It is nice to see that VoxMol can outperform EDM on GEOM-Drugs by a large margin. I suggest to add an invariant of EDM [1] as an additional baseline in the revision of paper. [1] Diffusion-based Molecule Generation with Informative Prior Bridges. NeurIPS 2022.
We really appreciate the engagement of the reviewer. Below we provide additional discussions in the hope of better answering R3's questions. - **"There are not outstanding advantages of the proposed method over existing graph-based 3D molecule generation methods":** - There are already many papers showing the usefulness of voxel-based representations for molecules (in discriminative, unconditional generation, conditional generation on pockets, scaffolds, fragments, etc). This is not the contribution of this work. - The advantages of our model are the following: - The proposed model is fundamentally different from the current SOTA. It has different data representation, generative modeling algorithm, noise model and the neural network used. - We show for the first time a voxel-based model can be competitive with graph-based on molecule generation on standard benchmarks. - We show it can scale better to larger datasets, achieving better performance on the challenging GEOM-drugs. - The proposed method is so different from SOTA that it "shed light on new research directions of connecting 3D CNNs to equivariant GNNs" (as pointed out by R2). - **"Discuss what are the outstanding and unique advantages of the proposed method":** We clearly state the unique advantages of our method on the abstract (L7-12, and throughout the text): `Our method, VoxMol, generates molecules in a fundamentally different way than current state of the art (i.e., diffusion models applied to atom point clouds). They differ in terms of the data representation, the noise model, the network architecture and the generative modeling algorithm. VoxMol achieves comparable results to state of the art on unconditional 3D molecule generation while being simpler to train and faster to generate molecules.` - **"I do not think fragment-based generation is beyond the scope of this work.":** 1. We agree that conditional molecule generation is the more important task—not only on fragments, but also on pockets, scaffolds, etc. Since this model is the first of its kind, we first study the unsupervised generation case to understand its capacities/limitations. 2. No other work on unconditional generation that we know of--including [1] mentioned by R3– provides information on doing fragment-based generation (as this is beyond what papers want to show). In fact, the only work that mentioning fragments is Vignac et al22: _"In contrast to the proposed method, which operates at the node level, fragment-based methods [13, 24, 32] learn to combine chemically-relevant substructures from a fixed or learned dictionary [48], but are harder to adapt to 3D."_ 3. We will add a paragraph mentioning voxel-based models can in principle do fragment-based generation. Given that (i) there are published works that show that voxel-based models can condition on fragments (and pockets, and scaffolds) and (ii) point 2 above, we argue that providing results on fragment-based generation should not be a requirement for acceptance. - **Comparison with [1]:** - We cited this work in the manuscript but direct comparison is hard. The authors do not provide code (nor generated samples). - Table 1 of [1] shows that their model achieves 82.4/0.0%, on atom and molecule stability, respectively, on GEOM-drugs (they only report those two numbers). In their setup, their performance is 1% better than EDM. However, we can't rely on comparisons, as [1] does not provide details on how training data is preprocessed or how evaluation is done in detail. Were we clear on the unique advantages of our model and why we think fragment-based conditioning should not be a requirement for acceptance? Please, let us know if any other clarification is necessary.
Response
I appreciate authors' response to address my concerns in follow-up responses. Here are my responses. - For the advantages, I am not satisfied with authors' responses. What I would like to know is the unique advantages that exist in VoxMol but does not exist in current graph-based generation methods. But authors' responses emphasize the differences between VoxMol and graph-based methods in methodology design. Differences are not advantages. "First time to show a voxel-based model can be competitive with graph-based on molecule generation on standard benchmarks" is not enough, because I want more insightful analysis about why VoxMol is competitive with (or has the potential to significantly outperform) graph-based methods from the perspective of data representation and model design. - Specifically, I would like to know answers to these questions --- now I want to develop a drug discovery system, why I choose your image-based molecule generation method given that I already have many existing graph-based molecule generation methods as candidates (not only EDM but also some other better options like [1][2][3])? Compared with graph-based molecule representation, can image-based molecule representation in VoxMol improve the quality of generated molecules so that their 3D structures respect chemical constraints better, especially fragment-level constraints? - In authors' responses, it is highlighted that VoxMol achieves better performance than strong graph-based baseline EDM on large benchmark GEOM-Drug. Maybe Authors will claim it as answers to the above questions. However, the "better performance" comes from higher "Validity" metric value, which may not be a reliable metric to me. To my understanding, "Validity" metric bascially converts 3D structure to 2D molecular graphs and check if bond valencies are correct. If this is true, it does not check the quality of 3D structures in a "fragment" view. For instance, the converted 2D molecular graphs may show that you have generated a beneze ring but 3D coordinates of corresponding atoms are not in a plane, then this is not a chemically valid beneze ring but "Validity" metric still counts it as "valid". In this case, good "Validity" metric does not necessarily guarantee that the generated molecules can stably exist in nature and synthesized. - Note that I am not talking about fragment-based generation as something like generating conditioned on some fragments. I mean composing molecules by a set of fragments, such as taking a beneze ring and a carboxyl to form a new molecule. Actually, fragment-based generation have been tried by researchers in molecule generation, e.g., [1][4] ([1] is unconditional generation and [4] is conditional generation from protein pockets, but it is not very different from unconditional generation). Particularly, [1] achieves very good "Validity" metric shown in Table 6, and more importantly, [1] incorporates many prior chemical knowledge into 3D fragment structures so it significantly increases the chances that the generated molecules can stably exist in nature and synthesized. As [1] is published very recently, I do not require to compare with [1] but I expect authors to know that fragment-based generation has already been explored. - To summary, I think the most important thing is to give good answers to the questions in my second point --- find advantages that uniquely exist in image-based molecule representations and discuss incorporating chemical constraints into image-based molecule representations. Given there already exist many literatures and new methods in unconditional molecule generation now, I do not think simplying proposing a novel method and achieving good metric values in benchmarks (particularly "Validity" metric is not so reliable to me) are solid technical contributions. I will not think this paper reaches the acceptance bar unless questions are well answered. [1] Coarse-to-Fine: a Hierarchical Diffusion Model for Molecule Generation in 3D. ICML 2023. [2] Diffusion-based Molecule Generation with Informative Prior Bridges. NeurIPS 2022. [3] Geometric Latent Diffusion Models for 3D Molecule Generation. ICML 2023. [4] Molecule Generation For Target Protein Binding with Structural Motifs. ICLR 2023.
On advantages of vision-based approaches over graph-based ones for molecule generation
We appreciate the continued discussion with the reviewer as we believe it further clarifies the strengths of our paper. We would like to summarize our responses earlier here: - The main advantage of a vision-based approach to molecule generation is **expressivity**. Graph-based methods suffer there due to the message passing formalism, a very well-known problem in the literature [XHLJ19]. In particular, we use the U-Net architecture with attention heads that are currently the workhorse behind state-of-the-art (score-based) generative models on images. With more data and more computation, these models can only become better. - Related to the above comment and highlighting our results, we have already seen much better performance on the larger GEOM-drugs dataset compared to smaller QM9. We suspect this gain will become even more pronounced for (future) larger datasets; this is not a blind suspicion, but rooted in limitations of graph-based message-passing frameworks when it comes to expressivity. "Respecting constraints" has less of a value if one can not capture the modes of a distribution with high fidelity. We agree with the reviewer that current metrics have faults but respecting chemical constraints is not the final answer either. - Another advantage is that what we propose complements the point-cloud/GNN approaches. On this point we agree with **Reviewer j1qM** that >"the paper showcases that instead of the traditional graph representation, molecule generation can also be performed on 3D voxels, shedding light on new research directions of connecting 3D CNNs to equivariant GNNs". - The reviewer's practical question on developing a "drug discovery system" is particularly interesting from this angle. We have shown a clear benefit in having an expressive model that's also efficient with only a *single* hyperparameter, the noise level $\sigma$. The question if we can "respect chemical constraints better" is really a research question. We think about it from a different angle: in our view, respecting chemical constraints is akin to respecting grammar in language modeling. Current state-of-the-art models do not enforce grammar but with enough data and expressive architectures (like transformers) this has become a moot issue. We have made a step in the right direction in demonstrating the power of a score-based computer vision approach for this important practical problem. - Specifically, on the question of *“why I choose your image-based molecule generation method given that I already have many existing graph-based molecule generation methods as candidates (not only EDM but also some other better options like [1][2][3])?”* We believe if the choice is binary the answer will depend on the dataset. We have made a strong case in this paper that our vision-based approach is a viable candidate for large datasets. We have also seen its limitations on small datasets, where having implicit priors are indeed beneficial. This again goes to the heart of this discussion, there are benefits to both having strong priors imposed on the architecture and there are also benefits for having expressive architectures. We believe for larger datasets the latter becomes even more important. - We agree with the reviewer regarding the validity metric, but we have outperformed a strong baseline such as EDM on _eight out of nine_ metrics on GEOM-drugs (see Table 2 on rebuttal pdf)--the dataset that matters for drug discovery going back to the reviewer's earlier question. E.g., 70% of Voxmol generated molecules are stable (meaning all atoms in the mol have right valence), while this degrades to 40% on EDM; VoxMol also captures much better the distribution of atoms (EDM highly underrepresents Carbon), valencies and bond types and angles. Again, we very much appreciate this discussion, and we hope the reviewer has a clearer picture of the advantage of our vision-based approach to molecule generation vs. graph-based ones. We hope the reviewer takes our clarifications and our strong results on GEOM-drugs into account and will support our paper for acceptance. [XHLJ19] K Xu, W Hu, J Leskovec, S Jegelka, *How powerful are graph neural networks?* ICLR 2019.
Response
I appreciate authors' response, but my questions still remains unsolved. - I agree that image-based generative models can take the advantage of expressive models like U-Net. However, it is not appropriate to underestimate the expressivity of graph models. Actually, graph models for 3D molecules or point clouds can be made much more expressive by using high-rotation-order features [1], atomic cluster expansion mechanism [2], or equivariant transformer architecture [3]. Graph-based generative models can become as expressive as image-based generative models by using these networks as backbones. - Even if graph models do not reach the same expressivity as vision models, they are more data efficient as they are internally designed to satisfy SE(3)-equivariance. I agree that if sufficient amounts of data are provided, vision models can also capture SE(3)-equivariance, chemical constraints, or other "grammars", just like large language models. However, while it is relatively easy to collect large amounts of language data, it is very expensive to collect the same amount of 3D molecule structure data, as producing 3D molecule structures need expensive computation like DFT. Considering this practical difficulty, graph models become more advantageous due to its data efficiency. - I do not agree that "Respecting constraints has less of a value if one can not capture the modes of a distribution with high fidelity". Developing unconditional generation methods of 3D molecules must consider how the methods will be applied to real-world applications, and respecting chemical constraints is a requirement as a 3D molecule that cannot stably exist in nature or be synthesized is of little use. Though respecting chemical constraints may limit the flexibility of generation models, I believe they cannot be neglected. [1] Geometric and Physical Quantities Improve E(3) Equivariant Message Passing. ICLR 2022. [2] MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields. NeurIPS 2022. [3] Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs. ICLR 2023.
We thank the reviewer for their follow-up, and we appreciate that we agree on the unique advantages of our model concerning the issue of expressivity. Many years of research have gone into the design of neural architectures in computer vision and our approach readily takes advantage of these developments. We believe the remaining disagreements do not seem to be on the merits of the paper. Our work is the very first score-based generative model for molecule generation that is vision based. The paper is novel with regards to the methodology as acknowledged by the reviewer. In our view, the work is complete in its scope: it is grounded in theory, and we have also shown very strong performance on a large drug-like dataset (GEOM-drugs). We hope the reviewer focuses more on our agreements than disagreements on the issue of SE(3) equivariance. We also believe this somewhat philosophical discussion (we do not mean to diminish its importance) should not be the reason to reject the paper.
Response
Actually, I do not agree that expressivity can be considered as a unique advantage because graph-based generation methods can also take the advantage of many expressive models as I mentioned in my last response. Also, I do not agree that issues of data efficiency and respecting chemical constraints are not the metrits of the paper, because they are unneglectable problems in real-world applications. As my major concerns remains unsolved yet, I will keep my rating.
The score of 3 is reserved for *"a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations."* In this process, the reviewer has not brought a single instance of "technical flaw", "weak evaluation", or " inadequate reproducibility". Please note that expressivity is not something one can *measure*. The reviewer acknowledged earlier that "I agree that image-based generative models can take the advantage of expressive models like U-Net" but the discussion has turned into proving the "unique" advantage of our model. As we mentioned earlier, this discussion (and the reviewer's score as it appears to be the case) is not on the merits of the paper. Regarding the expressive power of message-passing formalism and graph neural networks, we have already cited the influential paper *"How powerful are graph neural networks?"* in our earlier response. Higher order message passing can alleviate this problem to a degree (dictated by how "higher order" one can computationally afford), but they come at a cost and they have been limited to third-order message passing frameworks in practice including in the paper *"MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields"* the reviewer referred to earlier.
Increased score
I thank the authors for a detailed rebuttal. I believe the work is now stronger than when first presented. I have increased my score commensurate with the improved quality to a rating of accept.
We thank the reviewer for the reviews and feedbacks. We particularly appreciate the suggestion of using the new set of metrics. We will incorporate the feedbacks on the updated version of the manuscript.
Follow-ups
Thank you for the replies and correction on the diffusion model. Some minor issues have been addressed. However, my question remains unsolved, and more critical questions come out. Thus, I would like to keep my score. 1. My main concern is on the SE3-equivariance. - Currently, there is no evaluation metric about the SE3-equivariance in the result tables, especially the rotation. Thus, the `outperforming performance` is debatable. - You can think of the main baseline, EDM, as a constrained generation model, where the generation process needs to satisfy the SE3-equivariance, while this work is like an unconstrained generation. - For data augmentation specifically, we can always take rotation as data augmentation. However, as raised in my previous review, what if certain molecules are too large to fit in the voxel grids? How can we guarantee that we can always rotate it to fit in? What about molecules that can never fit it? Will you do truncation? If so, how will this affect the generation? These are the questions critical for data augmentation. - The related works [A,B,C] conduct experiments on images, which I am unfamiliar with. Is there any work providing similar explorations/observations from the chemistry side? - Thus, the claim that `data augmentation is sufficient` is not well supported. If you want to prove that generation with data augmentation is sufficient for SE3-equivariance, you need to design a metric specific for group symmetry and evaluate it for all the methods. 2. Thank you for adding MiDi results, and MiDi is an SE3-equivariant method. There are 18 metrics in total, and MiDi performs better than VoxMol on 15 of them. - As mentioned, MiDi is an SE3-equivariant model, and I'm not sure how authors can do `MiDi+ VoxMol`? Or do you simply mean doing 2D+3D using VoxMol? Then how do you define the covalent bonds in the voxel space? Do they have volumes? If so, then how large? Typically the covalent bonds do not exist in the 3D space, and it's non-trivial to inject them into the voxel space. 3. Yes, as mentioned previously, adding the citation to that paper is sufficient. No need to compare.
We thank R1 for the reply. We provide additional discussions in the hope of better answering R1's questions (detailed clarifications are posted below). 1. **Built-in SE(3)-equivariance (SE3E):** - Voxel representations have been extensively used in molecular modeling (generative and discriminative). These types of models dont have built-in SE3E but are still useful to the community. - The point of the paper is not to have a built-in SE3E model, but to show that we achieve competitive empirical results without it (as also shown by arXiv2305.05708). - Imposing SE3E on top of VoxMol could improve results even further. Future works could include: (i) design 3D CNN architectures with built-in SE3 equivariance, (ii) connect 3D CNNs to SE3E GNNs (as proposed by R2 and in [AA, BB]). 2. **Comparison with MiDi:** - MiDi is "2D+3D models" built on the top of a "3D model" (EDM). It shows that 3D models can be augmented with connectivity graphs to improve performance. In principle, this should work for other generative models as well. - MiDi leverages more information (2D graphs, formal charges) than VoxMol/EDM during training, so quantitative apples-to-apples comparisons are not fair. - The contributions of this work are (i) proposing a novel model for unconditional generation and (ii) beating SOTA on a challenging dataset, assuming same data is used for training. Leveraging 2D information (on top of 3D atoms) is beyond the scope of this paper. We hope we are clear on the clarifications above. Do the reviewer agree with authors that (i) there is value in showing that a novel voxel-based model can be competitive with SOTA despite not having built in SE3-equi., and (ii) comparison between Voxmol and MiDi are not apples-to-apples since the models are trained on different data? Please let us know if any other clarification is necessary? [AA] Point-Voxel CNN for Efficient 3D Deep Learning, Liu et al, NeurIPS19 [BB] 3D Shape Generation and Completion through Point-Voxel Diffusion, Zhou et al, ICCV21 --- **Detailed clarifications:** 1. **Built-in SE(3)-equivariance:** - `outperforming performance`: By "outperforming on GEOM-drugs" we mean that VoxMol has better empirical results than EDM on generative modeling metrics (MiDi metrics, as asked by R1). These numbers are not measuring any "SE3E property", but it clearly shows that the VoxMol better matches the distribution of GEOM-drugs and generates more realistic molecules. - _What if certain molecules are too large to fit in the voxel grids?_ - With a fixed grid size, voxel representations are limited to a fixed volume in space (L127). We can only use molecules that would fit the grid for training, and the model can only generate molecules that fit the grid. - With a 64^3 voxel grid (at resolution .25A), we can fit over 99% of the drug-like molecules from the GEOM-drugs dataset (L207-209), and train w 1 GPU. Rotating molecules with any rotation does not change that. - Every model has limitations. In the case of EDM, the model sampling time scales with n^2, n the number of atoms (since they use fully-connected graphs). Voxels, on the other hand, have constant inference time wrt n (for a fixed grid size) (L257-259). The tradeoff between EDM/VoxMol wrt to voxel grid length and the n of atoms has been discussed in the main text. We will make it clearer on the revised version. - _Related works [A,B,C]_: this is a very new and active research direction and, as far as we know, they haven't been tested on molecular data yet. This work, in addition to, eg, arXiv2305.05708, shows empirically that there might be something interesting to learn about this on molecular data. - `data augmentation is sufficient`: We agree with R1 and we did not mention it is sufficient, nor wanted to prove it. We pointed to a few references [A,B,C] that show that equivariance can be learned in neural networks (eg, vision transformers are sota on images w/o having built-in translation or rotation equivariances–they learn it). We agree it is future work to see how data augmentation helps learn rotation equivariance. 2. **Comparisons w MiDi:** - By `MiDi+VoxMol` we meant methods that could leverage 2D information on the top of VoxMol similar to how MiDi leverages 2D information on top of EDM. Eg, the model could use GNN for modeling connectivity graphs (2D) on the top of VoxMol (3D) and train them together/separately. It is an open question on how this can be done efficiently. - MiDi leverages more information than VoxMol/EDM during training, so quantitative apple-to-apples comparison is not fair. - Leveraging 2D information (on top of 3D atoms) is beyond the scope of this paper. - Finally, the results R1 is comparing (MiDi-adapt) appeared _after_ the submission of the conference and are (still) unpublished.
Thank You for the Detailed Reply
Hi authors, Thank you for the detailed reply, and I am happy to share more insights/concerns. 1. First, let me explain why I gave a score of 4. In terms of the method, I am not convinced that the generative modeling without SE(3)-equivariance would work (too many ways to define **work**, and I am talking about it from the math/physics). But there are solutions to solve this, as will be explained below, so don't worry. Such a technical concern would lead to a score of 3. Then I noticed the performance was quite good, and I turned the score to 4. The reason I wrote this is to tell the authors that there are still merits in this paper. 2. Then, I will go to my main concern. Generative modeling is essentially learning the distribution. This means that SE(3)-equivariant generative modeling is doing the SE(3)-equivariant distribution learning. If there's a metric on the rotation/translation-equivariance, then this metric would be 100% for the SE(3)-equivariant methods, but it is not guaranteed on the voxel method proposed in this work. If the author can give me some numbers to show that modeling on voxel with data augmentations can reach, e.g., `>95% rotation/translation-equivariant socre` (you need to design this metric yourself), then this can convince me. And I believe that this would be a more interesting and fundamental finding for the whole community. 3. The reason why I recommended you to try MiDi is that if your methods can beat MiDi's performance, such kind of `implicit comparison` can be a little much stronger in showing that voxel modeling can work. Yet, I need to point out that such a kind of `implicit comparison` is less persuasive than the one I showed above. Plus now it seems no luck with comparison to MiDi, so you can just skip this experiment. I hope I have explained my concerns more clearly.
We thank the review for the engagement and for clarifying their concerns. Below we provide additional discussions in the hope of better answering R1's questions. - **Comparison to MiDi:** - As mentioned before, `implicit comparison` or any comparison between models trained with different data are not meaningful. - Independently of this fact, R1 is comparing the performance of our model with a paper that appeared on arxiv on _June 5_ while the submission deadline was _May 15_. - Does the reviewer agree that benchmarking with a model that appeared _after_ the submission deadline should not influence the ratings of the manuscript? - If compared with the v1 of that paper (which was on arxiv on May 15), VoxMol outperforms MiDi in most metrics. But again, this does not matter because comparisons are not apples-to-apples. What it tells us is that if we manage to leverage connectivity graphs and formal charges efficiently on top of VoxMol (as MiDi did on top of EDM), we can boost its performance similar to how MiDi boosts EDM. - **" I am not convinced that the generative modeling without SE(3)-equivariance would work (too many ways to define** work **, and I am talking about it from the math/physics)":** - The machine learning community evaluates deep generative models by comparing the quality of the generated samples with a hold-out set of real samples. This is the definition of **work** most used by the community and the one we adopt. Better the empirical distribution of generated samples matches the empirical distribution of the hold-out set, then better the generative model. - Every paper on unconditional molecule generation before this evaluates the quality of their model this way. This includes papers that use voxels, point clouds, surfaces and even text (arXiv2305.05708) as data representation. We do the same. - Therefore, proposing new metrics to evaluate SE3-equivariant property of generative models is not our objective. We focus on benchmarking generative models in the same way other works do. - There are many published papers on voxel-based generative modeling for molecules (unconditional generation, conditional generation on pockets, scaffolds, fragments, etc). Can these works, or arXiv2305.05708 (mentioned by R1), convince R1 that models without built-in SE(3)-equivariance can generate molecules? - **Metric on the rotation/translation-equivariance:** - Generative models for molecules output a set of atoms and their coordinates (eg, `xyz` files), independent of the data representation used for training. - A `rotation/translation-equivariant score` metric makes sense on discriminative tasks, where the _input_ is a molecule (a rotated input molecule should have the same output as non-rotated input). In generative modeling, it is the _output_ that is a molecule while the input is only noise. Since it makes no sense to compute equivariance score on set of atoms, such metric should not be used to measure quality of generative models. Were we clear in (i) how we compare performance of generative models and (ii) the comparisons with MiDi are not apples-to-apples because the training assumptions are different and the model compared appeared after the NeuriPS submission? Please, let us know if any other clarification is necessary.
Quick Reply
Hi there, Thanks for the reply. - For the SE(3)-equivariance, I have explained it is the distribution learning that matters. Not the inference. What you were talking about is the inference (v.s. discriminative). - I agree that `proposing new metrics to evaluate SE3-equivariant property of generative models is not our objective`, but you need to make sure your learned distribution satisfies the underlying physics law. - Works like EDM, and MiDi satisfy this by design (`this metric would be 100% for the SE(3)-equivariant methods`). - But it's not clear on this work. So rigorously, you should prove it. - Existing metrics cannot reveal the physical properties in distribution learning. - `Every paper on unconditional molecule generation before this evaluates the quality of their model this way.` Yes, works like EDM and MiDi are unconditional, but they are constrained distribution/generation learning. These two (conditional and constrained) are different. - For MiDi, I have explained that if your metrics cannot beat MiDi, which is fine. `so you can just skip this experiment` - I'm not sure if arXiv2305.05708 is accepted somewhere. And yes, it has the same issue. I haven't asked you to put too much emphasis on it, just `should be cited at least`.
We thank the reviewer for the reply. Below we provide additional comments. - **SE(3)-equivariance:** - We see your point. What we are trying to point out is that in our paper we are evaluating on the standard way the machine learning community evaluates generative models. - Table 2 shows that, eg, VoxMol has 70.8% molecular stability vs 40.3% of EDM, VoxMol captures much better the distribution of atom types, bond types and the angle between bonds of molecules. These metrics are not perfect, but this huge statistical difference has to be acknowledged. Other SE(3)-equivariant methods before EDM do not even try on GEOM-drugs because its too challenging. - If we understand correctly, the reviewer argues that adding SE(3)-equivariance inductive bias on the NN architecture makes the generated molecules respect the "underlying physical law". This is not true. A model can have SE(3)-equivariance built-in and still generates "non-physical" molecules. - A trivial example would be to, eg, train EDM with a for just one epoch (or with a learning rate 1000x). The model still has built-in SE(3)-equivariance but will likely output noise (therefore not respecting any "underlying physical law"). In the case of published EDM, 60% of the generated molecules are not stable (meaning that at least one atom has wrong valence). Also, it can't capture the angles between bonds. This clearly means the samples do not always satisfy "underlying physical law". - Built-in SE(3)-equivariance is an inductive bias that allows a neural network to learn more efficiently, especially low-data/discriminative regime. By no means it guarantees that physical laws of generated samples are respected. - **Comparison with MiDi:** Do we agree that not comparing with MiDi should not be considered a weakness of this submission? Please, let us know if any other clarification is necessary.
Reply
My questions remain. 1. You need to either provide your method can reach comparatively good SE(3)-equivariance property and can beat the SOTA performance using SE(3)-equivariance. 2. Or another strategy is, for MiDi, which is 2D+3D, your method is better even though you are not using 2D information. But now the second strategy seems to be a dead-end, yet the weakness remains. 3. I am talking about the SE(3)-equivariant distribution learning in math, not sampling or inference. I think the authors remain confused on this point. 4. I don't know why `EDM can't capture the angles between bonds`? EDM is purely 3D, so where do the 2D covalent bonds come from? And why does EDM not satisfy the physical law? Also, are the authors implying that, because EDM doesn't satisfy physical law, this paper can also ignore the physical law? I am happy to follow up when the authors can provide more rigorous and quantitative support for this work's SE(3)-equivariance property.
On "rigorously proving" SE(3) equivariance
We respectfully disagree that we should "rigorously prove" SE3 equivariance. Even a convolutional neural network breaks equivariance due to the presence of non-linearities. We appreciate this discussion, but we have found it very surprising that the reviewer's main reason to give low scores is a philosophical view on SE3 equivariance for a machine learning model. As we pointed out earlier there are very important and impactful models and architectures in machine learning that break equivariance from the start, the most prominent of which are Vision Transformers.
Response
I thank the authors for the detailed feedback. Though there are still spaces to improve regarding this paper, I retain my positive score since the insights are interesting and the approach yields satisfactory novelty.
We thank the reviewer for the reviews and feedbacks. We will update the manuscript to make those points more clear. We particularly appreciate that the reviewer agree that a novel/different method with competitive results _is_ a good contribution for this community.
Decision
Accept (poster)