Error Estimation and Correction from within Neural Network Differential Equation Solvers

Neural Network Differential Equation (NN DE) solvers have surged in popularity due to a combination of factors: computational advances making their optimization more tractable, their capacity to handle high dimensional problems, easy interpret-ability of their models, etc. However, almost all NN DE solvers suffer from a fundamental limitation: they are trained using loss functions that depend only implicitly on the error associated with the estimate. As such, validation and error analysis of solution estimates requires knowledge of the true solution. Indeed, if the true solution is unknown, we are often reduced to simply hoping that a "low enough" loss implies "small enough" errors, since explicit relationships between the two are not available/well defined. In this work, we describe a general strategy for efficiently constructing error estimates and corrections for Neural Network Differential Equation solvers. Our methods do not require advance knowledge of the true solutions and obtain explicit relationships between loss functions and the error associated with solution estimates. In turn, these explicit relationships directly allow us to estimate and correct for the errors. 1 Basic assumptions and groundwork 1.1 Setup Let D ⊆ Rd denote a closed, bounded, path connected domain of interest for some differential equation. Let ∂D ⊆ D be the portion of the domain over which some constraint conditions exist (often boundary conditions and/or empirical data). Assume that our chosen differential equation (DE), when given unique constraints over ∂D, admits a unique, analytic solution Φ : D → RD. We denote the DE as: F[Φ] = A[Φ] +B[Φ] +C = 0 (1) where A represents the term(s) on which Φ depends linearly, B the represents the term(s) on which Φ depends nonlinearly, and C are the term(s) independent of Φ. This additive decomposition into linear, nonlinear, and independent terms is always possible: for a linear DE, B ≡ 0 ≡ C. Just like Φ, if the operators A,B,C make use of any functions over D, we assume those functions are analytic over D as well. Let us assume we wish to construct a feed forward Neural Network (NN) based approximation N : D → RD. We assume the NN uses analytic activation functions as well: as such, N ∈ C∞. Let W be its width (neurons per hidden layer) and DN be its depth (number of hidden layers). Let w ≡ {b 1 , w 11 , w 12 , ...., b 1 , w 11 , w 12 , ....} ∈ M ⊆ RM be the M weights/biases of the NN.