Robust Model Based Reinforcement Learning Using $\mathcal{L}_1$ Adaptive Control

(Part 1/2) Answers to Weakness 1 and Questions

We thank the reviewer for acknowledging the clarity and effectiveness of our work. Below, we address any potential misunderstandings and subsequently respond to the raised questions. > *W1. Using output of the RL algorithm as reference input for $\mathcal{L}_1$ adaptive control has been explored in Cheng et al., therefore the proposed method does not seem to be novel.* We highlight a crucial distinction between Cheng et al. [1] and our work: 1) we do not presume knowledge of the nominal control-affine dynamics, 2) and we do not assume the availability of a pre-trained (well-performing) policy. In [1], the policy is initially learned in an offline setting (in a model-free sense), where sufficient data—both in quantity and quality—can be collected from a simulator. Subsequently, when implementing this policy in a real environment with uncertainty, $\mathcal{L}_1$ adaptive control (designed based on nominal dynamics) is employed. (For further discussion, please refer to our rebuttal to Reviewer NiYe) In contrast, our work establishes a framework for utilizing adaptive control when neither the nominal model nor a comprehensive simulator is available. In our scenario, we iteratively update the model, which serves as the simulator to train the policy and use this policy to collect data from a noisy environment. This problem setup requires many additional considerations (e.g. treating nonlinear models, designing switching conditions, etc) to implement a robust controller. Providing novel solutions to handle these intricacies, our work significantly diverges from the problem and solution proposed in [1]. In addition, our framework for implementing an $\mathcal{L}_1$ adaptive controller in the training loop has not been presented elsewhere. > *Q1. What is the importance of Theorem 1 or what does it have to do with MBRL? What is the difference with the existing stability result in $\mathcal{L}_1$ -adaptive control theory?* The Theorem 1 serves to quantify the predictive quality of the state predictor in our add-on scheme concerning model approximation errors $\epsilon_l$ and $\epsilon_a$, and parameters governing the adaptation law ($T_s$, $\lambda_{\min}$). It confirms that our state predictor assures a bounded prediction error, which is important in our design where the performance is inherently tied to the predictive quality of the uncertainty estimate. On the other hand, the results in $\mathcal{L}_1$ adaptive control theory encompass the stability of the system in the closed loop. The discrepancy in the objectives of the two analyses is mainly attributed to the availability of the nominal dynamics (i.e. stability, equilibrium, etc) assumed in $\mathcal{L}_1$ adaptive control problems. > *Q2. In equation (9), how is $\nabla \hat{f}_{\theta}(x_t,u_t)$ implemented?* Differentiation of the neural network (NN) model occurs during the backpropagation process used for updating the model. In this procedure, we write the Jacobian of the NN model, denoted as $\nabla_u \hat{f}_{\theta}(x,u)$, as a function and make function calls whenever necessary. An alternative approach is to compute the Jacobian of the NN using Autograd, a technique that numerically evaluates the Jacobian and is commonly referred to as automatic differentiation [3] > *Q3. What does it mean to skip the update in non-differentiable points? Then should the update is over when it the agorihm meets the non-differntiable point?* This is a technical statement to circumvent the nondifferentiability of an NN (e.g. ReLU at zero), at which our affinized dynamics (Eq.(9) in our manuscript) is not defined. The statement clarifies that we apply exceptions to the switching law (10) specifically at those points. In practice, replacing a nondifferentiable activation function with its differentiable counterpart (Swish for ReLU) can easily resolve these nondifferentiability concerns.

(Part 2/2) Comments on Weakness

> *W2. Since the suggested method is an augmentation framework, the performance of the method highly depends on existing MBRL algorithms.* We agree with the reviewer that the suggested framework is constrained by the performance of the current MBRL algorithm, as highlighted in the limitation section of our work. The primary motivation for our proposed framework arises from the fact that introducing noise can have adverse effects on the performance of MBRL algorithms [4], which in turn may limit their applicability to real-world systems. We contend that our proposed methodology holds the potential to narrow the performance gap between algorithms with access to clean datasets and those without. > *W3. Since the true model is assumed to be deterministic, it highly restricts the range of application. Comparison with existing uncertainty dealing methods seems to be insufficient. I would expect several more explanations and experiments comparing with probabilistic models as in Section D.4.* We acknowledge the reviewer's concern that the deterministic dynamics assumption limits the range of application. However, we would like to emphasize that as mentioned in Section D.4 of the Appendix, our approach to using a robust controller is in a similar spirit to that of the probabilistic models but simply differs in architecture to address the same issue. Our approach uses an explicit method to estimate and reject uncertainty based on the deterministic nominal (learned) dynamics, while the probabilistic models use the variance of the prediction to handle uncertainty. Recently, the authors in [2] emphasized that the empirical success of probabilistic dynamics model ensembles is attributed to their Lipschitz-regularizing aspect on the value functions. This observation led to the hypothesis that the ensemble's key functionality is to regularize the Lipschitz constant of the value function, not in the probabilistic formulation itself. The authors have shown that the predictive quality of deterministic models does not show much difference with probabilistic (ensemble) models, leading to the conclusion that deterministic models can offer computational efficiency and practicality for many MBRL scenarios. In this context, our work exploits the practical advantages of using deterministic models, while $\mathcal{L}_1$ adaptive controller accounts for the randomness present in the environment. The above explanation is added to Section D.4 for the clarity of the manuscript, as suggested by the reviewer. [ References ] [1] Cheng, Yikun, Pan Zhao, Fanxin Wang, Daniel J. Block, and Naira Hovakimyan. "Improving the Robustness of Reinforcement Learning Policies With ${\mathcal {L} _ {1}} $ Adaptive Control." IEEE Robotics and Automation Letters 7, no. 3 (2022): 6574-6581. [2] Zheng, Ruijie, Xiyao Wang, Huazhe Xu, and Furong Huang. "Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Function." In The Eleventh International Conference on Learning Representations. 2022. [3] Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. The Journal of Machine Learning Research, 18(153):1–43, 2018 [4] Wang, Tingwu, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. "Benchmarking model-based reinforcement learning." arXiv preprint arXiv:1907.02057, 2019.

(Part 1/2) Answers to Questions

We thank the reviewer for providing insightful comments. We first provide answers to their [Questions], and subsequently address some of the points raised under the [Weakness] section of their review. > *Q1. What is the extension over [1]?* While our work and [1] both integrate RL policy with adaptive control, our work significantly diverges in key aspects such as the underlying assumptions, the scope of the study, and the usability of the method. **[Availability of the nominal model and well-trained policy]** The major difference between Annaswamy et.al [1] and our work is that we do not have access to the nominal dynamics and a well-trained policy. In [1], the policy is assumed to be acquired from an offline setting where sufficient data (both in quality and quantity) can be collected. A well-trained policy is meant to locally stabilize the uncertain system and ensure its uniform boundedness. Then the policy is employed together with an adaptive control (designed based on the nominal dynamics) in the closed loop. In contrast, our work develops a framework to incorporate a robust controller within an MBRL architecture wherein both the model and policy undergo iterative updates. As we state in the paper, one of the main guiding principles of our work is to place no additional burden or requirements on the high-level RL control, whereas, the work in [1] makes explicit requirements from the nominal dynamics and RL policy. **[Affine approximation and Switching conditions]** The reviewer implied in [Weakness] that we can tune a piecewise-affine (PWA) system to approximate the NN model, and then take a similar approach from [1]. However, due to the frequent updates of an NN model in an MBRL algorithm, such an approach becomes infeasible or impractical in an MBRL context. That is, optimizing for both the number of affine submodels and the parameters of each submodel is a well-known NP-hard problem. Even if we heuristically fix the number of submodels, applying a PWA system identification at each iteration will significantly increase the computational burden of the algorithm, and importantly, the accuracy of the affine approximation is not controlled. We concluded this approach is neither scalable nor generalizable, at least in the online MBRL setting that our work is focusing on. For this reason, many works follow the simpler approach of restricting the NN model structure to the control-affine class [2-3]. However, as evident in Figure 2 of our manuscript, this strategy can significantly compromise the performance of the baseline algorithm. To address this issue, we propose to obtain affine dynamics according to a state-dependent switching law, i.e., we do not predefine sub-models, but instead generate affine models along the trajectory. This methodology allows us to explicitly constrain and control the affinization error, a critical factor for providing predictive guarantees for the $\mathcal{L}_1$ adaptive control (Theorem 1 of our manuscript). Moreover, the Jacobian evaluation of a model at the switching point can reuse the stored information from the backpropagation during the model training. Thus, the computational burden of the algorithm is significantly less compromised compared to performing PWA system identification at each iteration. **[Parametric uncertainty]** The authors in [1] only considered linearly parameterizable uncertainty (Assumption 3, [1]) and its extension to bounded (non-state dependent) perturbations (Assumption 5, [1]). While this assumption is made to provide theoretical guarantees, it lacks the accuracy to capture reality. In our case, we place no such restrictive assumptions. > *Q2. Please carefully show differences with the following paper. There also seems to be a big overlap with [4]* While [4] shares some similar ideas with our work, the assumptions and methods to solve an optimal control problem are largely different from our work. This work used the term ‘model-based’ RL to represent the iterative learning process for the value, costate, and policy functions (Eq. (25), [4]) using the globalized dual heuristic programming (GDHP) algorithm when the dynamical model is assumed to be known and the objective function is quadratic. Their contribution is to enhance the GDHP update rule which relies on a noisy state measurement by formulating DNN approximations for each value, costate, and policy function to generate ‘targets’ (Equation (26)-(28), [4]), and their corresponding loss functions (Equation (32)-(34), [4]). Our method does not assume the availability of the nominal model or a quadratic objective function, and we do not model value and costate functions to develop a policy. In addition, although they use the term ‘model-based’ approach, they do not learn the dynamical model, and therefore, our work and theirs share a similarity only in name.

(Part 2/2) Comments on Weakness

Comments regarding the affinization and switching law are discussed in our first comment. Here, we discuss the remaining comments made in the [Weakness] section of the review. > *W1. There are many overblown claims. such as (i). This is not true. Many modern approaches use nonlinear methods. Your example makes no sense.* We agree with the reviewer and we acknowledge that our statement could be misinterpreted as an absence of nonlinear approaches in control theory. While control of nonlinear systems has an extensive history, using RL with adaptive control for unknown nonlinear systems is new. Indeed, the reviewer provided a very new work [1] (published in 2023), and in our answer to Q1 above, we provided ample evidence of simplifying assumptions used in [1] for the applicability of adaptive control. This highlights that while many modern approaches use nonlinear methods, we concern ourselves with one particular type of it. We revised this sentence in the edited version (p.1 in blue) of our manuscript as follows: *Classical control tools often rely on extensively modeled dynamics that are gain scheduled, linear, control affine, and/or true up to parametric uncertainties.* > *W2. The following is over 10 years old and covers many topics in (ii) [5].* We agree with the reviewer that many works including [5] provide robust adaptive control strategies for nonlinear systems. We have edited this sentence to clarify our intent as follows (p.1-2 in our manuscript): *MBRL algorithms frequently update highly nonlinear models (e.g. NNs) to enhance their predictive accuracy. This iterative process and the high nonlinearity of the model present a challenge in the integration of robust and adaptive controllers into MBRL algorithms.* While we revised the sentence for clarity, we point out that in [5], authors assume the knowledge of the nonlinear control-affine time-varying system ($f$ and $g$ in [5]). Furthermore, a Lyapunov function for the nonlinear system is assumed to exist. Exploiting these assumptions, they design a control strategy to compensate for the unmodeled uncertainty $h$. However, we do not place the assumption on the knowledge of a stabilizable system (with Lyapunov certificates) which makes our problem statement significantly different from [5]. > *W3. it is necessary to represent the nominal model in the control-affine form." It is not required to have an affine form, even for NNs [6]* In [6], the authors represent the uncertainties with an NN and update its weights according to Eqn. (44) in [6]. However, the NN used (see Sec. 2.3.1-2.3.2) is represented so that each element of the output can be represented linearly on the network weights. The only two cases where this can happen are if i) the neural network only has a single layer, or ii) one assumes that except the output layer, all other layers’ weights are correct (in some sense). This assumption therefore leads to the same assumption as in [1] of linearly parametrized uncertainties and thus the authors of [6] can apply the standard gradient flow adaptation law. We place no such restrictive assumptions, and we do not assume any knowledge of the system dynamics. Our controller knows only what the high-level MBRL algorithm knows: a purely data-driven NN model of the dynamics. Hence, while the reviewer’s claim that the use of NNs does not require affine forms is technically correct, the example provided in [6] hides many restrictive and simplifying assumptions using which one can easily apply any flavor of adaptive control. Unfortunately, such assumptions are unverifiable for complex systems of the type that MBRL is capable of controlling and the examples we provide in the manuscript. [1] Annaswamy, A. M., Guha, A., Cui, Y., Tang, S., Fisher, P. A., & Gaudio, J. E. (2023). Integration of adaptive control and reinforcement learning for real-time control and learning. IEEE Transactions on Automatic Control. [2] Xiong, Zhihua, and Jie Zhang. "Modelling and optimal control of fed-batch processes using a novel control affine feedforward neural network." Neurocomputing 61 (2004): 317-337. [3] Gil, Paulo, Jorge Henriques, Alberto Cardoso, Paulo Carvalho, and António Dourado. "Affine neural network-based predictive control applied to a distributed solar collector field." IEEE Transactions on control systems technology 22, no. 2 (2013): 585-596. [4] Kim, J. W., Park, B. J., Yoo, H., Oh, T. H., Lee, J. H., & Lee, J. M. (2020). A model-based deep reinforcement learning method applied to finite-horizon optimal control of nonlinear control-affine system. Journal of Process Control, 87, 166-178. [5] Wang, X., & Hovakimyan, N. (2012). L1 adaptive controller for nonlinear time-varying reference systems. Systems & Control Letters, 61(4), 455-463. [6] Padhi, R., Unnikrishnan, N., & Balakrishnan, S. N. (2007). Model-following neuro-adaptive control design for non-square, non-affine nonlinear systems. IET Control Theory & Applications, 1(6), 1650-1661.

Rebuttal reminder

Dear Reviewer NiYe We have carefully addressed your comments in our author response, highlighting key differences from Annaswamy et al. [1] and Kim et al. [4]. We would appreciate your feedback. If there are any further concerns, we are committed to addressing them during the discussion period. If the reviewer’s concerns are addressed sufficiently, we would like to sincerely request a re-evaluation of our manuscript. Best Regards, Authors.