Systems of interacting objects often evolve under the influence of field effects that govern their dynamics, yet previous works have abstracted away from such effects, and assume that systems evolve in a vacuum. In this work, we focus on discovering these fields, and infer them from the observed dynamics alone, without directly observing them. We theorize the presence of latent force fields, and propose neural fields to learn them. Since the observed dynamics constitute the net effect of local object interactions and global field effects, recently popularized equivariant networks are inapplicable, as they fail to capture global information. To address this, we propose to disentangle local object interactions -- which are $\mathrm{SE}(n)$ equivariant and depend on relative states -- from external global field effects -- which depend on absolute states. We model interactions with equivariant graph networks, and combine them with neural fields in a novel graph network that integrates field forces. Our experiments show that we can accurately discover the underlying fields in charged particles settings, traffic scenes, and gravitational n-body problems, and effectively use them to learn the system and forecast future trajectories.
Paper
Similar papers
Peer review
Summary
In this paper, the authors address the often-overlooked influence of underlying field effects on the evolution of interacting systems, postulating the existence of latent force fields. They suggest utilizing neural fields for learning these dynamics. Equivariant networks, although commonly used, fail to capture the global information, a crucial shortcoming the authors aim to rectify. They propose to separate local object interactions (SE(3) equivariant and dependent on relative states) from the global field effects (dependent on absolute states). Through the use of equivariant graph networks to model these interactions and their integration with neural fields, the authors construct a novel graph network to incorporate field forces. The authors' experimental results demonstrate their ability to successfully identify underlying fields within various scenarios, including charged particle settings, traffic scenes, and gravitational n-body problems. Furthermore, these findings prove the method's effectiveness in system learning and trajectory forecasting.
Strengths
The structure and presentation of the paper are appropriately executed. The authors have used clear and precise language, which aids in the understanding of the complex concepts discussed. This ease of readability can prove beneficial for readers across various levels of familiarity with the subject matter, contributing to the paper's usefulness. The concept of entangled equivariance, as introduced in this work, is a novel approach in system dynamics. This method, which focuses on factorizing out the influence of global fields from local interactions, provides a fresh insight into the understanding of interacting systems. I really enjoyed the range of experiments conducted in the paper, involving diverse systems of interacting entities at different scales. The application of the proposed methodology across different contexts, such as charged particles, traffic scenes, and gravitational n-body problems, illustrates the method's broad applicability.
Weaknesses
I found it interesting that there's no explicit mention of the training objective. This omission can potentially lead to ambiguity in understanding the complete process and methodology involved. For a comprehensive study, it is crucial to explicitly specify each component, including the training objective, to ensure a thorough understanding of the research. The training objective appears to be a reconstruction term (given Fig. 3), which may prove problematic in non-deterministic like trajectory forecasting. As it currently stands, the proposed approach will always predict the same future trajectory given the same input trajectory, a characteristic that is incongruous with the inherent non-deterministic nature of forward predictions in time, especially when the complete dynamical system is not fully known and explicitly defined. In addition, the approach is evaluated primarily in the context of future trajectory prediction, which only accentuates these concerns. I Lastly, there seems to be an omission in the citations pertaining to probabilistic models of neural fields. Specifically, works such as Functa [1] and Diffusion Probabilistic Fields [2] are not referenced. Including these references would enhance the completeness of the literature review and underscore the authors' comprehensive understanding of the existing body of research. [1] Dupont, Emilien, et al. "From data to functa: Your data point is a function and you can treat it like one." arXiv preprint arXiv:2201.12204 (2022). [2] Zhuang, Peiye, et al. "Diffusion probabilistic fields." The Eleventh International Conference on Learning Representations. 2022.
Questions
There are a few open question which are key to the formulation of the training objective of the proposed approach. What are the reasons for choosing a deterministic training objective (eg. L2 reconstruction) for a problem that is inherently non-deterministic. Wouldn't a model that predicts a distribution over future trajectories be conceptually preferred to model that predicts a single point estimate? I am happy to update my score after discussing these questions with authors.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
Limitations and societal impact of the proposed approach is properly discussed
Summary
In this paper, the authors have introduced a novel method, Aether, for discovering the latent field within dynamical systems in order to better model their dynamical evolutions. This is achieved by disentangle the interactions between objects and the effect of the external global field, conditioned on the states for the static field case or an extra latent state vector for the dynamic field case. The experiments on charged particles with external fields, Lorentz force field, traffic scenes, and gravitational field demostrate that Aether can learn to discover the latent external field and leverage the field information to better model the dynamics of the entire system.
Strengths
1. The paper identifies an important problem of latent field discovery which is of good originality. 2. The presentation of the paper is clear and easy to read. The method is well-designed and easy to follow. 3. The experiments are carefully designed which cover a wide range of datasets including physical systems and even traffic scences data.
Weaknesses
1. The experiments only comprise relatively simple scenarios and even methods compared against (see Q1 and Q2). It would be better to showcase the effectiveness and impact of the method on more challenging and complex datasets/benchmarks. 2. Some experiment results seem confusing and might require further explanations (see Q3).
Questions
Q1. Most of the datasets used in the paper are artificial/simulation data. Are there any practical application of the field discovery method on more real-world data (besides the traffic data)? Adding more real-world datasets will largely enhance the credibility of the proposed approach. Q2. Even for the traffic data, it would be of great significance to compare the method with state-of-the-art traffic prediction approaches, while in this paper only weak baselines are considered for this dataset. It is unclear that to what extent the current method would impact these fields of practical applications. Q3. Results in Figure 4 may require additional justifications. The reported MSEs of all methods seem to be around 0.6 at timestep 20 on charged NRI dataset, while in the original paper [1] the numbers were around the scale of 1e-3 when timestep = 20. Are all models properly trained? Moreover, it would be helpful to include ground truth trajectories in the visualizations to see whether the models can offer visually reasonable predictions, as per previous works. [1] Kipf et al. Neural Relational Inference for Interacting Systems. 2018.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors have sufficiently discuss the limitations of their method.
Summary
The main assumption of this paper is that local object interactions generally happen within external global fields. Based on this assumption, the paper then proposes to disentangle the local from the global interactions, respectively, by learning both an equivariant graph network, capturing the local interactions, and a neural network, capturing the global interactions. The latter network modifies the behavior of the graph network. The experimental results of the paper demonstrate that one can accurately discover the underlying fields in charged particles settings, traffic scenes, and gravitational n-body problems, and effectively use them to learn the system and forecast future trajectories.
Strengths
The idea of considering local interactions happening in the presence of global fields is an interesting one, and seems to be new. So is the approach, learning the external fields, and the experimental validation demonstrate an increased accuracy of objects-trajectory prediction.
Weaknesses
The results of this work seem to be confined to physics applications.
Questions
Is it possible to generalize the ideas in this paper to a structuring mechanism relevant to a more general class of ML applications?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
This work only considers fields that do not react to the observable environment.
Summary
This paper proposes to learn the latent fields governing the dynamics of interacting objects in a system without observations of the field. The authors propose unconditional and conditional latent field models to learn the static and dynamic fields respectively. The learned latent field signals are represented in local coordinate representations and treated as additional node features in graph learning to predict future node trajectories. Specifically, the paper adopts a recent VAE framework to make predictions in an autoregressive way.
Strengths
* The general idea of discovering latent neural fields is very interesting and plays an important role in modeling many real-world problems and scientific tasks. * The proposed idea looks simple yet effective in learning the latent fields. It is flexible and potentially can be applied to many graph learning frameworks. * The quantitative and qualitative results showcase the proposed idea and clearly show the performance gains compared with the previous works.
Weaknesses
Unclear notations: * The shape of different variables are often not explained clearly. It’s not easy to infer from the context and therefore not self-contained. * Eq (1): how to understand the shape of angular position $\boldsymbol{Q(w_i^t)}$ and $\boldsymbol{\tilde{R}(w_i^t)}$? The notations do not seem to be self-contained. * Eq (2) - (3) what are the definitions of $\boldsymbol{h_{j,i}^t}$? * Definition of angle in L169 uses the symbol $\boldsymbol{\omega}$, is it the same as in L74? * L94 uses the symbol $\boldsymbol{h}$, is it related to h in Eq (2)? * Latent code $\boldsymbol{z}$ and the latent graph edge type $\boldsymbol{z}$ in the appendix are using the same letter, which is very confusing. L44 and below: In the example of the N-body system, if the E(3) symmetry holds, then SE(3) symmetry must hold since SE(3) is a special case of E(3) without reflection. To the reviewer, this example doesn’t explain the pitched point that SE(3) equivariance doesn’t always hold. About the combination of latent field learning and VAE: * The proposed models use latent edge-based VAE to predict the future node states. It’d be better to contain the VAE formulation in the main paper briefly to be self-contained. * The idea of learning latent fields can be applied in non-VAE frameworks for trajectory predictions as well. It’d be more convincing to show these experimental results. About neural field modeling. * How important is the assumption of static and dynamic fields for learning? Can the model learn different fields in a data-driven manner? * To the reviewer, either conditional or unconditional neural fields modeling methods can be used to learn both static or dynamic fields. It’s not convincing without ablation experiments to justify the model design w.r.t. types of fields. * L155-L162: the naming of unconditional and conditional neural fields look confusing to the reviewer, as both are functions of input states essentially. Although they are different in architecture design, both are neural functions of the input trajectories. * Figure 3 shows the conditional neural field setup for dynamic fields, while most of the experiments are using static fields. To the reviewer, this is a bit misleading and does not showcase the proposed model properly. Please add both static and dynamic fields setup in the figure. About G-LoCS baseline: * How is the artificial velocity defined in the experiments? The details are vague to the reviewer. * This method is essentially an additional baseline with node feature augmentation and is not a novel contribution to the reviewer.
Questions
Please see comments above.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
n/a
Thank you!
Thank you for the efforts you have made in addressing my concerns. I have no futher question and hope the authors will incorporate the new results and explanations into the final paper. I have increased the score.
Thanks for your feedback. As a minor suggestion, I recommend rephrasing the original text concerning `strict E(3) symmetry` to enhance its clarity (for instance, by employing the black hole example you provided in the context of external force fields). > N-body systems from physics, for example, exhibit E(3) symmetries, since gravitational forces only depend on relative positions. Dynamics, however, may be influenced by external force fields, which are either unknown or not subject to transformations. Regarding the unconditional and conditional neural fields, the original paper and the reviewer's rebuttal appear somewhat unclear. The original text states in line 157: > we use an **unconditional neural field**, i.e. a neural field that is **a function only of the input states**, as the field values are common across data sample. The authors explain in the rebuttal with the following statement: > **Unconditional neural fields are not functions of the input states**, on the contrary, they are independent of the input states. These two statements seem to contradict each other. Furthermore, the original paper mentions: > In contrast, for dynamic fields, we use a **conditional neural field**, i.e. a neural field that also depends on a latent vector $z \\in \\mathbb{R}^{D\_z}$ that represents the underlying field. The **latent z will be inferred from the input trajectories** and can be thought of as representing unusual non-equivariant dynamics. What distinguishes the `input states` of the unconditional field from the `input trajectories` of the conditional field? Are they perceived as the same from the network input perspective? Lastly, this question appears to be disregarded in the rebuttal: > How important is the assumption of static and dynamic fields for learning? Can the model learn different fields in a data-driven manner? As the authors acknowledge in the rebuttal, **`a conditional neural field could be used to learn a static field`**. Can we consistently utilize conditional field models to learn both types of data and eliminate the need for assuming the field type beforehand? Regarding empirical performance, how crucial is this assumption across a broader range of experiments? Specifically, the authors mention that for the Lorentz force field (static field) experiment, the MSE of the conditional model is 0.0131, while the original unconditional method had an MSE of 0.0129. This performance difference of **0.0002** seems negligible to the reviewer. Then, what is the rationale behind the unconditional model? Thanks.
Rebuttal by Authors
We would like to thank the reviewer for their detailed comments. Please find our answers below. ``` As a minor suggestion, I recommend rephrasing the original text concerning strict E(3) symmetry to enhance its clarity (for instance, by employing the black hole example you provided in the context of external force fields). ``` Thank you for the suggestion. We will rephrase the original text to enhance its clarity regarding E(3) and SE(3) symmetries. ``` The statements “we use an unconditional neural field, i.e. a neural field that is a function only of the input states, as the field values are common across data sample” and “Unconditional neural fields are not functions of the input states, on the contrary, they are independent of the input states” seem to contradict each other. ``` We apologize for the confusion. The two notions of functions come from different perspectives, namely a practical perspective and a theoretical/mathematical perspective. From a mathematical perspective, the field is a function of x-y coordinates, x-y constituents of a velocity vector, etc. One could compute the field at any valid tuple of positions, velocities etc. In that sense, the field is independent of the input states. In other words, the unconditional neural field will make the same predictions regardless of the inputs. In practice, during training, we only sample the field at the positions/velocities that coincide with the states of the input objects, since we only have supervision there. As such, with a slight abuse of notation, we write $\mathbf{f}_i = f(\mathbf{v}_i)$, where $\mathbf{v}_i$ is the state of node $i$. We acknowledge that this terminology mixup can confuse the reader. We will incorporate parts of this discussion in the camera ready version to increase clarity. Namely, we will mention that “unconditional neural fields are not functions of the input states”, and remove the term “input states” in other occurrences concerning the neural field to avoid confusion. ``` What distinguishes the input states of the unconditional field from the input trajectories of the conditional field? Are they perceived as the same from the network input perspective? ``` The terms “input states” and “input trajectories” describe very similar concepts, yet they are distinct. The input states denote the information about nodes-objects (positions, velocities, orientations) for a single timestep. As we mention in lines 129-130 in the original manuscript, these states - or a subset of the state variables (e.g. positions only) - are used as input to the neural field. The term input trajectories denotes the whole system we have access to, including the temporal information, i.e. the input states for a number of timesteps. In the conditional field case, input trajectories are used to create the latent vector $\mathbf{z}$. The latent vector is used alongside sampled positions/velocities to compute the field. This means that the conditional field depends on the (spatiotemporal) inputs, and would produce different predictions for different inputs. ``` How important is the assumption of static and dynamic fields for learning? Can the model learn different fields in a data-driven manner? ``` We apologize for the omission. Our method discovers dynamic fields conditioned on the observable input system. As such, the working hypothesis is that the input states cover the domain of the field we are trying to discover. If we do not have enough coverage, then we cannot discover the field effectively. Similarly, we cannot model fields that react to the system, unless we have enough spatial coverage for each temporal snapshot of the field. ``` As the authors acknowledge in the rebuttal, a conditional neural field could be used to learn a static field. Can we consistently utilize conditional field models to learn both types of data and eliminate the need for assuming the field type beforehand? Regarding empirical performance, how crucial is this assumption across a broader range of experiments? Specifically, the authors mention that for the Lorentz force field (static field) experiment, the MSE of the conditional model is 0.0131, while the original unconditional method had an MSE of 0.0129. This performance difference of 0.0002 seems negligible to the reviewer. Then, what is the rationale behind the unconditional model? ``` While a conditional neural field be used to learn a static field, this can come at the cost of increased training and inference time, as well as redundant computational resources and model parameters. Indeed, the performance difference of 2e-4 is rather negligible. However, we would argue that when there is expert knowledge that the field at hand is a static field, then the unconditional neural field is the preferred choice. In the absence of such knowledge, e.g. on an exploratory analysis for underlying fields, then the conditional neural field would be preferable.
I appreciate your responses and most of my concerns have been addressed. My curiosity lies in understanding the practical consequences of the described effects, such as longer training and inference times, along with duplicated computational resources and model parameters, stemming from using conditional neural fields for static fields. Is it feasible to **quantitatively** demonstrate the concrete overhead in terms of training and inference time, model size, and computational demands among the experiments detailed in the paper? Providing such comparisons would better validate your claim within the context of practical machine learning applications.
We would like to thank the reviewer for their detailed comments. Please find our answers below. ``` Is it feasible to quantitatively demonstrate the concrete overhead in terms of training and inference time, model size, and computational demands among the experiments detailed in the paper? ``` We would like to thank the reviewer for their suggestion. In the following table, we report the training time per minibatch, the inference time, and the number of parameters for each model in the Lorentz force field setting. While the (unconditional) Aether has 2,515 more parameters than the LoCS backbone, the conditional Aether has 9,985 more parameters on top of the regular Aether, as well as 27% higher inference time. We will include these quantitative results in the camera ready version of the paper. Method | Params | Inference Time | Training time / minibatch ---: | ---: | ---: | ---: | EGNN | 134,020 | 0.0028 | 0.0037 | LoCS | 130,307 | 0.0033 | 0.0038 | Aether | 132,822 | 0.0037 | 0.0049 | Conditional Aether | 142,807 | 0.0047 | 0.0053
Thanks for your feedback. I have no more concerns about the paper and I have raised my score.
I have read the rebuttal and I am satisfied with the answers provided by the authors.
Decision
Accept (poster)