Neural Fields with Hard Constraints of Arbitrary Differential Order

Summary

The authors propose a method to enforce hard constraint points on neural fields. Instead of a single black-box coordinate network predicting the field values, this work uses neural networks to learn basis functions which are then combined in a linear transformation. Given enough basis functions, the weights of this linear transformation can be found by solving a system of linear equations.

Strengths

The paper is well-written and easy to understand. The authors have included their source code which seems reasonably well organized. The method itself is simple and guarantees that constraint points are not violated. It outperforms existing methods on the MERL BRDF dataset. The paper includes a varied selection of experiments to validate the method, and some ablation studies have been performed and reported in the appendix.

Weaknesses

The paper contributes little in terms of theory. While the derivation is simple enough, it would have been nice to see a theoretical argument for why this method converges faster than unconstrained training, at least for special cases. The constraints seem to be applicable only to single points, which is not clear from the abstract. Consequently, initial states and boundary conditions cannot properly be handled using this method. Experiments 4.1 and 4.4 are synthetic and very simple. Only experiments 4.2 and 4.3 measure real-world performance. The paper introduces six basis functions in chapter 3.2 but they are never compared against each other in the experiments. The authors simply employ different basis functions for different experiments. This makes it hard to judge which one to use for any given problem. Comparisons against state-of-the-art is also sparse. Only the BRDF fitting experiment compares to related work. The paper does not include learning curves for the various experiments. Figure 5 is somewhat related to learning curves, but the quantitative evaluation of the training process is severely lacking. Training time is not discussed in the paper either. Learning curves against wall-clock time would be appreciated. Minor: * L11: The claim in “Our approaches are demonstrated in a wide range of real-world applications.” seems a bit of a stretch. It’s more like one to two. * L41: Missing citation for the statement about inequality constraints. * L72-78: The review of previous work is missing stream functions for divergence-free fields (e.g. Deep fluids, 2019) and other conservation properties (e.g. Guaranteed Conservation of Momentum for Learning Particle-based Fluid Dynamics, 2022). Hamiltonian and Lagrangian networks have also been used with conserved properties in mind. * The citations in the main text do not have hyperlinks to the references at the end. * The hybrid kernel basis is poorly explained in 3.2. * Footnote 2 is not explained. It is unclear what kind of LaTeX expressions can be specified. * Chapter 5 is more outlook than summary * A.2 and Figure 5: Which experiment does this belong to?

Questions

In Eq. 11, why do you minimize the log L1 loss, not the log L2 loss?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

While some advantages and disadvantages of various basis functions are mentioned, the authors do not provide a limitations section and only lightly touch on general limitations of the method.

Summary

A broad range of problems can be formulated as linearly constrained problems, e.g., learning material appearance, interpolatory reconstruction, solving linear PDE, etc. In order to solve linearly constrained optimization problems, this paper developed a novel hard constraint method that builds upon neural fields and differentiable linear solver, naming constrained neural fields (CNF). **Contribution**: 1. A novel methodology CNF is proposed, specifically, linear equality constraints are transformed into a linear system, i.e., eq. (3)-(4); 2. Then both weights of neural fields, $\beta_i$, and learnable parameters, $\theta$, of neural fields (see eq. (2)) can be learned from gradient descent of the objective function in eq. (1) given a differentiable linear solver is applied to eq. (3)-(4); 3. Hyper kernel basis is proposed, which is benchmarked with various basis functions (see section 3.2) and demonstrates advantages such as stable conditional number throughout training (Fig. 5); 4. In experiments, 4 examples from very different background are solved by CNF with superior performance, showing the potential of CNF as a general learning framework for linearly constrained problem.

Strengths

**Originality**: CNF is a novel method of solving linearly constrained problem by implementing neural fields based on differentiable linear solver. **Quality**: The paper shows that CNF can solve various problems with high performance. **Clarity**: The methodology is clearly presented. Worthy of mentioning, the analysis of conditioning of the matrix due to different kernel methods (see Appendix A) is quite convincing about why the authors believe Gaussian kernel with hybrid kernel basis (see eq. (9)) is the best choice. **Significance**: This work can be applied to a broad range of linearly constrained problems.

Weaknesses

The work relies on differentiable linear solvers, and therefore the most suitable problem for CNF is a linearly constrained problem. In section 5, the authors discussed that CNF can be applied to nonlinear problems given a differentiable solver. However, such a differentiable solver to nonlinear problems is in general not easy to get. Hence, CNF is currently limited to linear problems.

Questions

N.A.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

N.A.

Summary

The paper presents a method for integrating hard constraints, represented by a linear operator, into neural field basis functions. This is achieved by learning kernel functions as basis functions at specific constraint points. Through experimentation, the paper provides evidence to show the effectiveness of the proposed method in comparison to unconstrained neural fields across diverse practical tasks.

Strengths

Originality: The paper introduces a novel approach by directly applying linear operator constraints to the basis functions constructed with neural field functions and learning a linear representation. Also, the weights are found by applying a solver to a linear system, which is nice and removes some optimization problems when the weights should also be learned. Quality: The results obtained in the paper demonstrate the effectiveness of the proposed method across various tasks. I appreciate the efforts in comparing different common implementations of neural field bases. Clarity: The paper effectively explains the reasoning behind critical implementation choices, such as the selection of basis functions and the choice of kernel. Significance: The proposed approach addresses the significant challenge of the application of explicit hard constraints for neural fields.

Weaknesses

- As a reader, I found it challenging to comprehend the training procedure without delving into the code. Therefore, providing a comprehensive explanation would facilitate understanding. This would make it easier to grasp the methodology. - Furthermore, sharing more details about how regularization is applied would provide valuable insights into the approach. Explaining the specific methods used for regularization and their impact on the model's performance would enhance the clarity and comprehensiveness of the paper.

Questions

- To enhance clarity, it would be better for the authors to dedicate a separate section to explaining the model training process in detail. This section needs to explicitly highlight which parameters or weights are optimized and specify the optimizer/solver employed for this purpose.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors have addressed the limitations of the work in the Summary section.

Summary

The authors look at enforcing hard constraints on neural fields. Here, the problem formulation is to take continuous coordinates as input and predict the solution on these points as output. The neural field is represented as a linear sum of basis functions, and specifically, variants of a neural kernel function. The constraints must be linear operators which are then satisfied via using a linear solver.

Strengths

- Enforcing constraints more precisely on neural fields could improve prediction performance, and trying to do this via harder constraints seems promising. - The method utilizes flexible representations of neural fields (i.e., neural kernel fields; as well as other representations of the basis functions) to enforce the relevant constraints.

Weaknesses

- The evaluation metrics are limited, and it is hard to contextualize these results with respect to other neural field methods. For example 1, the authors compare to other neural representation methods, but what about comparing to “soft constraint” approaches as well? It would be helpful to know how speed vs accuracy compares when enforcing the constraint is a softer way. - There is no discussion of the speed and training time to implement this hard constraint approach. It seems like it would be expensive to do a linear solve on these matrix systems, greatly hindering efficiency. More details about the linear solver would be helpful (as well as how the authors treat the system as fully differentiable). - One major limitation of this approach is that only linear operators are used as constraints (thereby being able to utilize eon 3). The practicality of this method seems especially useful when the operator is non-linear. - It would be helpful to describe the problem formulation for each problem (inputs/outputs, form of constraint, what is being minimized, etc.), as this is not always clear from the paper.

Questions

- Is the constraint enforced at both training and test time? - What linear solvers are being used here? The authors mention “note that when employing general solvers such as SQP and DC3…” but don’t give details about the solvers themselves. - What is the training time and inference time, and how does it compare to other NN fitting approaches? In particular, these linear solvers could be very expensive. - Can you show an example of using the method when the constraint is a non-linear operator? - In example 4.3, is eqn. 12 (Eikonal equation) being solved in a “soft constraint” way? Where are the hard constraints being enforced? - The format of some of the references needs to be corrected. For example, references 7, 16, 25, 39 do not specify a publication venue.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

- Limitations are not discussed (besides that this approach does not work for non-linear operators), but it seems like this approach would also be difficult to scale up for larger systems because of the expensive linear solves on larger systems.

Response to authors

Thank you to the authors for your responses, I appreciate it. I have gone through the responses and the paper again, as well as looked at all the related work. A major limitation of this work is that it is currently only used for linear operators, and that means that it cannot tackle a number of more realistic systems at this point. I know that the authors describe that another solver, like Gauss-Newton, can be used, but it is challenging to converge and the computational graph is expensive. Thus, it seems like this approach is limited because it cannot handle a very large number of important use cases. One of the related constraint works, PDE-CL, seems to be able to work on non-linear differential operators. Edit (2 hours after the above comments): The other point I want to make about training and inference speed is that right now, you are only comparing to other neural network approaches. For a number of these problems, you should also be comparing to classical numerical methods.