Simultaneous Online System Identification and Control using Composite Adaptive Lyapunov-Based Deep Neural Networks

We thank the reviewer for the appreciation of our contributions, theoretical results, and presentation. Based on the reviewer's suggestion we removed the technical details in the abstract to make it more concise. Additionally, we have added the following sentence to the abstract: “under the persistence of excitation (PE) condition, the tracking and weight estimation errors are shown to exponentially converge to a neighborhood of the origin, where the rate of convergence and the size of this neighborhood depends on the gains and a factor quantifying PE”. **Regarding composited disturbances** We apologize for the brief statement about the previous work (O'Connell, 2022) in the introduction which may have caused confusion. To make the discussion about (OConnell et al., 2022) more precise, we have moved the discussion about this reference to the Related Work Section in Appendix B.2. For the reviewer's convenience, we are reproducing the discussion here: “The recent result in (O'Connell, Shi, Shi, Azizzadenesheli, Anandkumar, Yue, and Chung, 2022) developed a new learning representation uncertainties involving a composited disturbance given by $f(x,\dot{x},w)=\phi\left(x,\dot{x}\right)a\left(w\right)$, where $\phi\left(\cdot\right)$ denotes a basis function that is learned using a DNN and $a\left(w\right)$ denotes a set of linear parameters accounting for an unknown disturbance time-varying disturbance $w$. Since $f$ is linear in terms of $a$, the composite adaptive approach from (Slotine & Li, 1989) is used to design an adaptation law $\dot{\hat{a}}$ to update the estimates of a given by $\hat{a}$. To obtain a disturbance-invariant representation of $\phi$ using DNNs, a domain adversarially invariant meta-learning (DAIML) algorithm is developed to train the DNN offline. To the best of our knowledge, this is the only existing work using a composite adaptive approach in the context of deep learning-based control. However, since the DNN learning $\phi\left(x,\dot{x}\right)$ has an NIP structure, the aforementioned challenges apply for constructing a Lyapunov-based online adaptation law.” Please note **this is not a criticism of (O'Connell, 2021)**; we acknowledge this method involves a fundamentally different paradigm than ours. We strongly appreciate the DAIML approach to obtain a disturbance-invariant representation in (O'Connell, 2021). Since (O'Connell, 2021) is the only existing work using a composite adaptive approach in the context of deep learning-based control, we believe it deserves to be cited and discussed in the related works section. Furthermore, note that our developed method can be easily extended for composited disturbances. Note that the developed method is agnostic of the DNN architecture used. Therefore, for a composited disturbance $f(x,\dot{x},w)$, the representation $f(x,\dot{x},w)=\phi\left(x,\dot{x}\right)a\left(w\right)$ from (O'Connell, 2022) can be used. The term $\phi$ can be approximated as $\phi\left(x,\dot{x}\right)=\Phi\_\{1}\left(X,\theta\_{1}^*\right)+\varepsilon\_\{1}\left(X\right)$, where $\Phi\_\{1}\left(X,\theta\_{1}^*\right)$ is a DNN with an appropriate ideal parameter $\theta_1^*$, which yields $f(x,\dot{x},w)=\Phi\_1\left(X,\theta\_1^*\right)a\left(w\right)+\varepsilon\_1\left(X\right)a\left(w\right)$. Then $\theta^{\*\top}=[a(w)^\top \theta\_1^{\*\top}]^{\top}$ can be constructed as an augmented parameter to define the parameterization $\Phi\left(X,\theta^*\right)=\Phi\_{1}\left(X,\theta\_{1}^*\right)a\left(w\right)$ and $\varepsilon\left(X\right)=\varepsilon\_1\left(X\right)a\left(w\right)$, which yields $f(x,\dot{x},w)=\Phi\left(X,\theta^*\right)+\varepsilon\left(X\right)$. In this case, $\theta^*$ becomes a time-varying parameter. The analysis in (O'Connell, 2022) treated $a(w)$ as a bounded time-varying parameter with a bounded time-derivative. This argument can then be used to obtain that $\theta^*$ and $\dot{\theta}^*$ are bounded. In the Lyapunov-based analysis the extra term $\tilde{\theta}^{\top}\dot{\theta}^*$ would appear since $\theta^*$ is time-varying. The bound on $\dot{\theta}^*$ can be used to conclude exponential convergence of all the error states to the neighborhood of the origin in a similar manner as in the proof of Theorem 1, except that the size of this neighborhood now also depends on the bound on $\dot{\theta}^*$.

**Revisions in Simulations** Based on the reviewer's suggestion of adding measurement noise to make the simulations more realistic, we performed all of the simulations again with measurement noise, and the results in the simulation section are now updated accordingly. Additionally, although the reviewer is correct that high gains can amplify measurement noise, this is true for all feedback controllers in general; the tradeoff between tracking performance, control effort, and noise sensitivity is well-known. Notably, instead of using numerical derivatives in our update laws, we use a dynamic state-derivative estimator which is shown to be robust to noise in Appendix A.2. As shown in the proof, the ultimate bound on state-derivative estimation error grows linearly with the bound on the noise $\bar{\delta}$. Furthermore, based on the reviewer's suggestion, we have also included the state-derivative observer-based disturbance rejection controller as a baseline for the two link manipulator. The gain selection strategy for ensuring fairness is described in Appendix D. For a fair comparison, the set of gains common to the developed and baseline methods were selected to be exactly the same. We did not perform a comparison with the composite adaptive method in (Slotine and Li, 1989) because the comparison would not be fair in our opinion. The method in (Slotine and Li, 1989) has an unfair advantage because it uses regressor information based on the structural knowledge of the manipulator dynamics. The DNN being a black box model does not have access to this information, so Slotine and Li's method is expected to perform better. This is for similar reasons as why a feedback linearization controller using the exact knowledge of $f(x,\dot{x})$ would be expected to perform better than Slotine and Li's method or our method. **Response to other queries** The proposed framework can be easily combined with offline learning by initializing the DNN weight estimates with pre-trained weights. In fact, initializing with pre-trained weights (e.g., especially using meta-learning approaches such as (O'Connell, 2021)) would likely further improve the controller performance. However, to better showcase the online learning performance with the composite adaptation law, we initialized the DNN with completely random weights in our simulations. Although it might appear the update strategy involves only the current state measurements, the least-squares update law does actually involve a historical data-mining procedure. Specifically, notice the least squares adaptation gain matrix $\Gamma(t)$ evolves according to Eq. (12) in the manuscript, which reads $\frac{d}{dt}\Gamma^{-1}(t)=-\beta(t)\Gamma^{-1}(t)+\Phi^{\prime\top}\left(X,\hat{\theta}\right)\Phi^{\prime}\left(X,\hat{\theta}\right)$. This equation reveals that the adaptation gain matrix $\Gamma(t)$ implicitly incorporates information from all past data points through the dynamics of $\Gamma^{-1}(t)$, which accumulate contributions from the regressor matrix $\Phi^{\prime\top}\left(X,\hat{\theta}\right)\Phi^{\prime}\left(X,\hat{\theta}\right)$ over time. As a result, the update law inherently processes historical data by integrating its influence into the adaptation dynamics, thereby enabling the least-squares framework to adapt based on both current and prior system behaviors. Regarding the comment “It is a kind of traditional adaptive control, instead of modern data-based learning. Maybe a control journal is more applicable to this paper”, we would like to point that our work lies at the intersection of adaptive control and deep learning. Specifically, the use of deep neural networks makes our work different from traditional adaptive control. Furthermore, ICLR explicitly invites contributions involving applications to robotics, autonomy, planning (which is the primary area of this submission). References: O’Connell, M., Shi, G., Shi, X., Azizzadenesheli, K., Anandkumar, A., Yue, Y. and Chung, S.J., 2022. Neural-fly enables rapid learning for agile flight in strong winds. Science Robotics, 7(66), p.eabm6597. Slotine, J.J.E. and Li, W., 1989. Composite adaptive control of robot manipulators. Automatica, 25(4), pp.509-519.

We thank the reviewer for their appreciation of the novelty, theoretical contents, and literature research. While we acknowledge that updating all layers of the DNN in real-time may require high computational resources, many real-world applications, especially those in industrial and robotics settings, have access to more powerful computational platforms, where such methods can be feasibly deployed. Based on the reviewer's suggestion, we performed all of the simulations again with measurement noise, and the results in the simulation section are now updated accordingly. We have also included the control input plots in both simulation examples according to the reviewer's request. Regarding the comment about the nonlinear dynamics in simulations being weak or the question whether this method is effective for highly dynamic systems like quadcopters, please note that the dynamics of the UUV in Section 4.2 (described in further detail in Appendix D.2) are similar to the quadcopter dynamics (see Eq. (1) in (O' Connell et al, 2022) for example). Such similarity is because both systems have an Euler-Lagrange dynamics resulting from the dynamics of rigid body translation and rotation. The UUV dynamics contains strong nonlinearities resulting from centripetal-Coriolis effects and hydrodynamic damping effects. So the developed method is indeed effective for highly dynamic systems. The additional simulation details on control inputs, weights update, and the selected control parameters are provided in Appendix D. References: O’Connell, M., Shi, G., Shi, X., Azizzadenesheli, K., Anandkumar, A., Yue, Y. and Chung, S.J., 2022. Neural-fly enables rapid learning for agile flight in strong winds. Science Robotics, 7(66), p.eabm6597.

We thank the reviewer for their appreciation of the motivation of our paper and the numerical experiments. However, we respectfully disagree with the reviewer on the severity of the assumptions, which we justify as follows. **Regarding fully-actuated dynamics** Many practical nonlinear systems such as robot manipulators and UUVs used in the simulation section, Stewart platforms, hexapod robots meet this assumption. This is a broad variety of applications where the developed method can directly be applied and Assumption 1 holds mathematically and practically. The development is focused on fully-actuated systems to better focus on our unique specific contribution, i.e., the composite adaptive design for DNNs. The reason we do not consider the underactuated case is because, even for underactuated systems with perfect model knowledge, there is no universal nonlinear controller that can be directly applied to every underactuated system. Instead, the the constructive Lyapunov-based control design procedure is unique for each underactuated system. For example, the nonlinear control design for a nonholonomic mobile robot is completely different from the design for quadrotors. However, the steps required to extend the DNN-based control development from a fully-actuated system to an underactuated system mostly involves standard tools (e.g., backstepping) without any challenges specific to composite adaptive DNN development. The tools such as backstepping are well-known since 1980s, so including such an extension for specific cases would add limited theoretical value or novelty and confuse our main contribution. **UPDATE:** Based on the reviewer's suggestion, we have revised the manuscript with a detailed illustration on extending the development for a nonholonomic mobile robot in Appendix F. This extension does not require taking the pseudo-inverse according to Assumption 1 but instead achieves tracking by backstepping the kinematics into dynamics based on the development in (Fierro, Lewis, 1998) work. We hope this addresses the reviewer's concerns about Assumption 1. **Regarding simultaneity of system identification and control** It appears there is possibly a misunderstanding regarding what we mean by simultaneous system identification and control. The term $f(x,\dot{x})$ is unknown and not being used anywhere in the controller. The control input is not designed as $u=g^{+}(r-f(x,\dot{x}))$, but $u=g^{+}(x,\dot{x})(\\ddot{x}\_\{d}(t)-(\alpha_\{1}+k_\{r})r+(\alpha_\{1}^{2}-1)e-\Phi(X,\hat{\theta}))$ according to Eq. (6). The term $\Phi(X,\hat{\theta})$ is a DNN-based adaptive estimate of $f(x,\dot{x})$, updated in real-time. Specifically, the DNN parameter estimate $\hat{\theta}$ updates in real-time based on the adaptive update law $\dot{\hat{\theta}}=\mathrm{proj}(-k_{\hat{\theta}}\Gamma(t)\hat{\theta}+\Gamma(t)\Phi^{\prime\top}(X,\hat{\theta})(r+\alpha_{3}E))$. Therefore, the control and adaptive update laws operate simultaneously, unlike offline training methods which identify the parameter estimates a priori using training datasets. Assumption 1 or fully-actuatedness does not violate this simultaneity. **Regarding Assumption 2** The assumption is not about the existence of $\bar{\theta}$ but about its knowledge. Note that $\bar{\theta}$ is a bound on the unknown constant ideal parameter $\theta^*$, so $\bar{\theta}$ exists because $\theta^*$ is a constant. This is a standard mathematical assumption in NN-based adaptive control literature (e.g., see (Lewis, 1996)). As far as the approximation accuracy $\bar{\varepsilon}$ is concerned, its existence is not an assumption but a fact because due to the universal function approximation property of DNNs. In fact, even without the universal function approximation property, a $\overline{\varepsilon}>0$ satisfying $\sup_{X\in\Omega}\left\Vert f(x,\dot{x})-\Phi(X,\theta^{*})\right\Vert \leq\overline{\varepsilon}$ would exist because the function $f$ and $\Phi$ are continuous and therefore bounded over the compact set $\Omega$. The universal function approximation property further allows $\overline{\varepsilon}$ to be prescribed as arbitrarily small. Based on the reviewer's comment, we revised Section 2.1 to state, “Assumption 2 is reasonable since in practice the user can select $\overline{\theta}$ a priori to restrict the parameter search space. If such a selection does not obey Assumption 2, the selection may no longer allow the user to make $\bar{\varepsilon}$ arbitrarily small as guaranteed by the universal function approximation property. However, a bound $\bar{\varepsilon}$ satisfying $\sup_{X\in\Omega}\left\Vert \varepsilon(X)\right\Vert \leq\overline{\varepsilon}$ still exists due to the continuity of $f$ and $\Phi$ over $\Omega$. Using heuristic approaches, if such $\bar{\varepsilon}$ is found to be larger than the maximum allowable error, then $\bar{\theta}$ can be iteratively increased until it achieves the prescribed $\bar{\varepsilon}$.”

We thank the reviewer for the timely response. We are happy the reviewer found our revisions regarding Assumption 2 satisfactory. As for Assumption 1, it is not clear if the reviewer noticed our more recent update in Appendix F, where we added a brief illustration on extending the procedure to underactuated nonholonomic mobile robots. This example was added based on the reviewer's suggestion to provide more insights on performing the extension to an underactuated case. Since the reviewer agrees with the response "the constructive Lyapunov-based control design procedure is unique for each underactuated system", we hope the unrealisticness of the expectation to consider a general underactuated system is understood. The extension needs to be performed on a case-by-case basis. However, the process of performing such an extension mostly involves standard nonlinear control tools that are not unique to our work. Specifically, note that the composite adaptive DNN (our main contribution) part of the development in Appendix F is mostly the same as in Section 2 of the manuscript. The part of the development specific to mobile robots involved standard nonlinear control insights (i.e.,backsteppping approach well known in literature for mobile robots) from (Fierro, Lewis, 1998) work. The extension to an underactuated case did not involve any theoretical novelty. To better focus on our main contribution, we focus the main development on non-underactuated systems. Additionally, we still think there is a possible misunderstanding regarding simultaneity. The reviewer is requested to clarify on the statement in the original review, > "Assumption 1 implies that performing system identification can achieve arbitrary control performance (arbitrary system dynamics is realized by $u= g^+(r - f(x,\dot{x}))$. In this sense, the problem addressed in this paper is not essentially simultaneous." The briefness of the reviewer's comments is making it difficult for us to interpret what was being meant. Assumption 1 is not precluding the simultaneity of identifying $f(x,\dot{x})$ and implementing the control law. The control and adaptive update laws are simultaneous. The reviewer is requested to clarify if we are on the same page with that. >"Assumption 1 severely limits the class of the plant system to completely eliminate both the challenge (and value) of the simultaneous problem" We would like to bring the reviewer's attention to the challenges involved in the DNN part of the development, which is the focus of our main contribution. There were multiple challenges precluding this development, all of which are explained in great detail in the Introduction and Appendix B. Specifically, developing Lyapunov-based composite adaptive update laws for all DNN layers is a challenging problem especially due to the nested and nonlinear-in-parameter structure. Specifically, the term $\Phi^{\prime}\left(X,\hat{\theta}\right)$ is not the standard regressor in adaptive control, but the DNN's Jacobian with respect to its vectorized weights, an analytical derivation of which is provided in Appendix C. The DNN's Jacobian is derived for an arbitrary number of layers by leveraging its recursive compositional structure. Additionally, the formulation of the prediction error term E has never been done in literature for a DNN because of technical challenges that are specific to a DNN. Slotine's result ([24] in the previous version) used a filtered regression approach to formulate the prediction error. As described in Appendix B.2, there are mathematical challenges in extending Slotine's filtered regression approach to DNNs because the approach utilized the LIP structure of the model. To circumvent those challenges, we used a dynamic state-derivative estimator to construct a new prediction error, and the corresponding estimation errors are also incorporated in the Lyapunov-based analysis to ensure closed-loop stability after combining the developed controller, adaptation laws, and state-derivative estimator. Therefore, even with Assumption 1, the challenges are still significant, which the other reviewers have recognized. Note that this is a very recent topic of research with only a few closely related works, e.g., (Patil et. al., 2022) and (O' Connell et. al., 2021), both highly cited works. Both of these works performed the development assuming fully-actuated dynamics. If Assumption 1 completely eliminated the challenges like the reviewer claims, this topic of research would not have emerged in the first place. Patil, O.S., Le, D.M., Greene, M.L. and Dixon, W.E., 2021. Lyapunov-derived control and adaptive update laws for inner and outer layer weights of a deep neural network. IEEE Control Systems Letters, 6, pp.1855-1860. O’Connell, M., Shi, G., Shi, X., Azizzadenesheli, K., Anandkumar, A., Yue, Y. and Chung, S.J., 2022. Neural-fly enables rapid learning for agile flight in strong winds. Science Robotics, 7(66), p.eabm6597.

We thank the reviewer for their appreciation of the strength, timeliness, and potential impact of our contribution. Based on the reviewer's suggestion to also include more standard controllers used in robotics as baselines, we have updated the simulation section to include nonlinear MPC and nonlinear PD controllers as baselines for both examples, the robot manipulator and the UUV. The developed method was able to achieve improved tracking performance with a similar control effort compared to both of these baselines. Furthermore, while we acknowledge that the comparison with nonlinear MPC provides valuable context, it is important to note that our approach operates under a fundamentally different paradigm. Specifically, using DNN-based online system identification, our method addresses scenarios where prior model knowledge is unavailable or unreliable, highlighting its practical applicability in uncertain environments. We hope this clarifies the distinction and demonstrates the merits of our approach. In future work, we also plan to extend our approach to develop a DNN-based adaptive MPC method, where the DNN-based adaptive system identifier can update the model in real-time for improving future state predictions over the horizon. **Regarding nonconvex dependence** The convexity condition on an underlying loss did not appear in our analysis because we used a Lyapunov-based approach instead of traditional optimization-based approaches involving a loss function. Instead, we require the persistence of excitation (PE) condition and some gain conditions that may be considered analogous to a convexity condition. Specifically, instead of imposing convexity on the DNN, we use a first-order Taylor series approximation $\Phi(X,\theta^{*})-\Phi(X,\hat{\theta})=\Phi^{\prime}(X,\hat{\theta})\tilde{\theta}+\mathcal{O}(\left\Vert \tilde{\theta}\right\Vert ^{2})$ as in Eq. (14) of the paper, where the higher-order term $\mathcal{O}(\left\Vert \tilde{\theta}\right\Vert ^{2})$ is shown to be bounded when $z\in\mathcal{D}$ and because $\left\Vert \tilde{\theta}\right\Vert \leq2\bar{\theta}$ due to the projection operator. The bound on $\mathcal{O}(\left\Vert \tilde{\theta}\right\Vert ^{2})$ carries forward to the gain conditions. Recall that $$\lambda_3=\min \\{ \alpha_{1},k_{r}-\frac{\gamma_{1}}{2},\alpha_{2},k_{f}-\frac{\gamma_{2}+\alpha_{3}\gamma_{3}}{2},\frac{k_{\hat{\theta}}}{2}+\frac{\beta_{1}}{2\varkappa_{0}}-\alpha_{3}\gamma_{3} \\}$$. As mentioned in Remark 3, the gains $\alpha_{1}$,$\alpha_{2}$,$\alpha_{3}$,$k_{r}$, and $k_{f}$ can be selected to be sufficiently high such that $\lambda_{3}=\frac{k_{\hat{\theta}}}{2}+\frac{\beta_{1}}{2\varkappa_{0}}-\alpha_{3}\gamma_{3}$; thus, $ \frac{\lambda_{2}c}{\lambda_{1}\lambda_{3}}=\frac{\lambda_{2}}{\lambda_{1}}\left(\frac{\gamma_{1}+\gamma_{2}+k_{\hat{\theta}}\bar{\theta}^{2}+\alpha_{3}\gamma_{3}\gamma_{1}^{2}}{k_{\hat{\theta}}+\frac{\beta_{1}}{\varkappa_{0}}-2\alpha_{3}\gamma_{3}}\right)$. Since the term $\beta_{1}$ is quantitatively indicative of the PE condition with larger values under more excitation, the bound $\sqrt{\frac{\lambda_{2}c}{\lambda_{1}\lambda_{3}}}$ becomes tighter under more excitation and can be arbitrarily decreased under higher gains and more excitation. Under the absence of PE, a bound is still obtained by selecting an appropriate high $k_{\hat{\theta}}>2\alpha_{3}\gamma_{3}$. **Regarding Input and State Constraints** The input and state constraints can be incorporated using control barrier functions (CBFs) (Ames et. al., 2016). Specifically, the control input from the developed method can be passed as a nominal control input through a CBF-based safety filter involving a quadratic program with CBF-based constraints. However, exploring input and state constraints is beyond the scope of this work as the CBF-based safety constraints are either difficult to obtain or too conservative under the lack of prior model knowledge. We plan to investigate DNN-based adaptive CBFs in future work, where safety constraints can be gradually made less conservative as the DNN learns the system dynamics online. Based on the reviewer's comment, we have added the following sentence in Section 6 to discuss future work: "Furthermore, extensions of the developed online system identification approach in optimization-based control paradigms such as MPC and reinforcement learning can be explored. Moreover, future research efforts can also investigate how to combine the developed method with control barrier functions to satisfy state and input constraints." **References:** Ames, A.D., Xu, X., Grizzle, J.W. and Tabuada, P., 2016. Control barrier function based quadratic programs for safety critical systems. IEEE Transactions on Automatic Control, 62(8), pp.3861-3876.