Optimal Exploration for Model-Based RL in Nonlinear Systems

Summary

This paper addresses the problem of exploration when learning to control non linear systems. In particular, they derive a task-driven exploration method which focuses on learning the system parameters that are relevant for the specific task that a controller is trying to achieve. To derive the algorithm, they rely on a quantity which estimates how much the task loss vary, under a certainty-equivalence controller and the current model estimate. This quantity is the key to estimating what parameters in the model are relevant for the task at hand. It can be used by existing exploration algorithms (DynamicOED) as a minimization target. The exploration routine is presented as a second contribution which extends an algorithm originally designed for linear MDPs to non-linear systems.

Strengths

The premises of the article that all parameters of the system might not be useful to know for the task of the controller makes a lot of sense, and the problem of task driven exploration is interesting and relevant. Overall, there is novelty as they extend a lot of existing concepts to a much broader class of systems (non-linear) and the contribution seems significant. However, it would be quite hard to implement the proposed algorithm from reading just the paper itself. The introduction of the motivating example and associated discussion is quite useful to understand the problem space. The related work section helps to position the scope of the paper and is mixing both RL and more traditional control. The experiment section show that the proposed algorithm outperforms non task-driven exploration methods. The set of baselines is quite limited but as the concept is quite specific to this article, I believe they are a sound choice.

Weaknesses

Although the paper is well structured and written, the main weakness is the presentation. Since the paper introduces a lot of mathematical concepts and assumptions, it is useful to help the reader by describing intuitively what they mean. The assumptions presented are justified by citing previous work but some description would be useful to make the paper more self contained. Specifically assumption 3, it would be useful if the authors provide an intuitive justification. Assumption 8 lacks some general explanation. How does the choice of the underlying algorithm affect the performance? and why is the cost different than the goal $\Phi$? A lot of the concepts are derived from reference [2] which essentially solves the problem in the case of linear systems. It makes the contribution more incremental, however the jump from linear to non-linear is still significant. Theorem 1 and 2 introduce bounds on the exploration loss which are claimed to be novel, it would still be useful to compare with the bounds for algorithms that are not task dependent. Is there a trade-off? Since section 5 is presented as a potentially independent contribution, it would have been useful to add more background about the experiment design problem and clearly highlight the difference with previous work. The notation in dynamicOED is hard to follow, the explanation helps only a little. It would be quite hard to implement the algorithm just from reading the paper. It refers a lot to the LC3 algorithm which is not explained (this is certainly not my main expertise). The systems use in the experiments have fairly small scales (6 dimensions for the state). One could wonder whether the approach scale in the number of dimensions. It is mention a linear dependency with the system parameter for the convergence. What about the computational cost of the algorithm? In the experimental section, it is mentioned that MPC sampling based methods are used for exploration. How do we know that they verify assumption 8 which is key for convergence? Maybe the author could have tried linearizing the systems and applied the linear version of the algorithm that is cited.

Questions

- Is there any physical system that would be like the motivating example? - What is D-optimal design? What to the optimization targets presented as example represent at line 270 in section 5? - Why is dynamicOED returning several exploration policies and not just one per step in algorithm 1? - How is the optimal controller $\pi_*$ obtained to compute the excess controller loss?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

They assume a specific form of non-linear system which could limit the applicability of the solution (at least the theoretical results), however a few references are mentioned indicating that many systems fit into this category. The bounds scale polynomially in the number of system parameters, it is not commented whether it is desirable or not. Couldn’t systems have a large number of parameters?

Summary

This paper considers the problem of task-relevant system identification. It starts with the motivation that not all parameters in the dynamics model are relevant to the given task; thus, the exploration should be placed to prioritize the identification of the relevant dynamics parameters, and thus eventually leading to improved efficiency of the model-based policy learning. The authors begin with translating the task loss to the dynamics prediction loss, which is weighted by the cost hessian at ground truth dynamics model. The main result of this paper is to show that this dynamics loss can be approximated tightly by DynamicOED method. The proposed method is validated using some toy problems. Overall, the paper is well-motivated and presented. This paper did a great job in explaining the theorems. Although the interest of parametric dynamics model is restricted to weighted features, the main results of this paper may be of interest to the reinforcement learning community.

Strengths

See my comments in Summary

Weaknesses

- In the motivating example, the explanation of task-irrelevant or relevant parameters are confusing. Shouldn't $a_{3:12}$ be irrelevant to learning the optimal controller, while $a_{1:2}$ critical? - Please further explain "In practice, though they may not formally satisfy 290 the guarantee of Assumption 8, deep RL approaches could be used." - I think Section 5 a bit deviates from the main problem formulation, especially starting with a new system Equ. (5), although I understand the authors are trying to introduce DynamicOED generally for a broad interest of the readers. - The authors could give some perspectives on the promise of the results applied to more general dynamics. - I strongly suggest adding more or complex tasks to demonstrate the efficiency of the proposed methods over non-optimal exploration model-based policy learning.

Questions

See my comments in Weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See my comments in Weaknesses.

Summary

The paper proposes an active exploration algorithm for systems of the form $x_{k+1} = A \phi(x_k, u_k) + w_k$ with known features $\phi$ and unknown matrix $A$, where $w_k \sim \mathcal{N}(0,1)$. The problem is reduced to minimising $\mathrm{tr}(\mathcal{H}(A_\star)\check{\Lambda}_T^{-1})$ where $\mathcal{H}(A_\star)$ is the Hessian at the optimal $A_\star$ (which is iteratively estimated) and $\check{\Lambda}_T^{-1}$ is roughly the expected covariance $\phi\phi^T$ over $T$ episodes of length $H$. Thorough analysis is presented. The algorithm achieves instance-optimal rate. Examples on simulated systems are provided.

Strengths

Very strong theoretical paper, solving optimal exploration for a class of linear-in-features systems. Originality, quality, clarity, are excellent. Significance is strong, but since the features are assumed known, it limits the applicability of the algorithm.

Weaknesses

- The class of systems is quite restricted. If one needs to learn the non-linear features $\phi(x, u)$ first, then one could just as well use the collected data to estimate $A$ afterwards. It would be interesting to see an extension where features are learned as well. - The computational complexity is unclear. Does the algorithm run real-time? Is the numerical implementation stable? - There is no Discussion/Conclusion/Limitations section. - Empirical comparisons are only done with respect to very naive baselines (random and uniform exploration). At least some variance or entropy minimising active exploration could be considered, as e.g., in [1]. But for this paper it is fine, the theoretical contributions are already sufficient, no further experiments are necessary. [1] Schultheis, M., Belousov, B., Abdulsamad, H., & Peters, J. (2020). Receding horizon curiosity. In Conference on robot learning (pp. 1278-1288). PMLR.

Questions

Could one further improve the algorithm for control-affine systems? I.e., $x' = A\phi(x) + Bu$ with unknown $A$ and $B$?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

No. The authors are encouraged to add at least one paragraph of Conclusion, which also discusses the limitations.

Summary

This paper tackles the challenge of controlling unknown nonlinear dynamical systems in reinforcement learning and control theory. The authors propose a novel algorithm, inspired by recent work in linear systems, focusing on the most critical parameters for learning a low-cost controller on the actual system. This method efficiently explores the system to reduce uncertainty in a specific, task-dependent metric. They demonstrate the algorithm's effectiveness on nonlinear robotic systems and provide a theoretical lower bound showing that their approach learns a controller at a near instance optimal rate. This work could have significant implications for reinforcement learning and control theory.

Strengths

The author presents an algorithm which achieves the instance-optimal rate, with controller loss matching the lower bound on the loss of any sufficiently regular control rule.

Weaknesses

The regularity assumptions in 3.1 should be addressed clearly and the author could give a toy example to illustrate when these assumptions are satisfied.

Questions

The formulation of the nonlinear system in (1.1) seems restricted, can you explain more about the formulation?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The simulations are conducted on low dimension systems and the experiment setting should be stated more clearly. Besides, this work faces a high computational burden.

Summary

The paper seeks to quantify formally in the settings of nonlinear dynamical systems (a) which parameters are most relevant to learning a good controller and (b) the best exploration strategy to minimize uncertainty in such parameters. This paper draws theoretical insights where minimizing the controller loss in nonlinear systems translates to estimating the system parameters in a particular task-dependent metric. In light of this characterization, the paper presents an algorithm that achieves the instance-optimal rate for efficient exploration, with controller loss matching a proven lower bound. Several numerical experiments are conducted on nonlinear systems to validate the analysis and the approach.

Strengths

- **S1** The paper is very well written. The problem definitions and contributions are clear. - **S2** For a particular class of nonlinear systems of the form (Eqn. 1.1), the paper characterizes how minimizing the controller loss in nonlinear systems translates to estimating the system parameters in a task-dependent metric, and provides a lower bound on the loss of any control rule. - **S3** The paper proposes an algorithm that achieves the instance-optimal rate, with controller loss matching the proven lower bound. - **S4** The proposed approah demonstrates improvements compared to the baselines in several numerical experiments.

Weaknesses

- **W1** The paper could benefit from having stronger experimental results. The authors could include more simulated environments that fit into the particular class of nonlinear systems, both in simulation and the real world, such as [1]. The authors could also consider comparing with more baseline methods, such as [2] and [3], from the deep model-based RL community. - **W2** Could the authors elaborate on how theoretical insights from the paper relate to problems in practical applications of the particular class of nonlinear systems (1.1)? I believe this would improve the paper's impact on the broader robotics, control, and RL communities. For instance, could applications in safety-critical scenarios be benefited from these insights (e.g., flight maneuvers in [1])? [1] O’Connell et al., "Neural-fly enables rapid learning for agile flight in strong winds." Science Robotics, 7(66):eabm6597, 2022. [2] Kaiser et al., "Model-based reinforcement learning for atari." arXiv preprint arXiv:1903.00374, 2019. [3] Hafner et al., "Dream to Control: Learning Behaviors by Latent Imagination." arXiv preprint arXiv:1912.01603, 2020.

Questions

- The authors mentioned in line 378 that they believe the approach will also scale to deep model-based RL settings. Could the authors explain how their approach can be scaled with deep RL?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

- The authors target a particular class of nonlinear systems characterized in (1.1). This characterization would determine what kind of systems the theoretical insights and the approach can be directly related to. - The authors provide experimental analysis on simulated domains. It would be interesting to see the results in real-world environments.

Summary

In this paper, the authors propose a method for learning an optimal controller that is able to focus the exploration so as to minimize the estimation error of relevant parameters. They show that the proposed method is indeed optimal with tight upper and lower bounds on the gap. Experiments show that the proposed algorithm performs better compared to the baseline reaching lower loss at any given number of learning episodes. **After authors' rebuttal:** Thanks for the added clarification. I agree that the paper's contributions are largely theoretical and there's enough here to merit a publication. The authors mention their intention for addition of proof sketches and intuitive discussion which will improve the readability of the paper further. Given these, I am keeping my rating at 7!

Strengths

The paper is clearly-written, well-motivated and systematically-structured. The assumptions are explicit and the theorems are described to develop intuition. While the idea of certain parameters dictating the system dynamics might make abstract sense, having the motivating example in section 1.1 made the follow-up analysis more grounded for me. Using the second order moment to quantify parameter relevance is neat but that appears to be adopted from previous work. In general, the theory part of the paper is crisp and laid out to facilitate understanding.

Weaknesses

On the other hand, the experiment section is quite thin. However, as a largely theoretical contribution, this is acceptable. I didn’t look through the appendix so I missed out on the proofs, and the paper could have benefitted from some proof sketches without hashing out all the details.

Questions

NA

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

NA

Thank you for the clarifications, please add the explanation about the assumptions.

I acknowledge reading the rebuttal of the authors and thank them for answering all my questions clearly. I recommend that you add the explanation about assumption 3 and assumption 8 to the paper to improve the presentation. The discussion of theorem 1 and 2 and their relations to related work could also be added.

Thank you for the clarifications

Thank you for your answers and clarifications. I recommend that you add the practical insights to the paper to improve its presentation.