Model-Based Control with Sparse Neural Dynamics

Summary

This paper proposes a method for pruning neural networks with ReLu activation functions during training. When employed for learning the dynamics of a control system, this often leads to networks with few activation functions performing similarly as large networks. This allows to apply mixed integer programming techniques to determine optimal control policies. The effectiveness of this approach is demonstrated in a comparison with a sampling based optimizer on several tasks for a robotic manipulator.

Strengths

The idea of formulating planning problems with neural network dynamics as mixed integer program seems novel to me. Moreover, I find the proposed approach of reducing the number of neurons during training very interesting, but I cannot comment on its novelty since I am no expert on this topic. Overall, the paper reads very well and has a good structure. The nice demonstration of the proposed method in a real-world robotic experiment is also a strength of the paper.

Weaknesses

My biggest concern is the missing discussion of computational complexity/computation time. I am not an expert on mixed integer programming, but a quick search suggests that even mixed integer linear programming is already NP hard, so this seems to be a problem in general. I understand that this problem can be mitigated through a sufficiently small number of neurons (‘If only a few ReLUs are left in the model, Equation 10 can be efficiently solved to global optimality’), but I am missing a clear specification what sufficient means in this context. Moreover, I would expect it to be connected to the prediction horizon, i.e., the number of time steps considered in the sum in (10). Therefore, I think a complexity (e.g., in O notation) should be provided to give the reader an impression how severe the computation times grows. This weakness also limits the usefulness of the evaluation in my opinion. Why would you reduce the number of neurons for MPPI in practice? I think the performance in relation to computation time is rather the important metric to look at for this comparison, i.e., a complexity-performance trade-off comparison would be essential. Therefore, the comparison seems a little unfair at the moment. This similarly extends to the robot experiment in Sec. 4.3, where the sampling and the horizon rate of the MPC are not specified. Moreover, it is not clear how fast the robot moves. These are all parameters which crucially influence how challenging the problem is. In general, the method seems tailored to piece-wise affine systems or systems that look almost like that. The examples seem to go into this direction, but I cannot say for sure since I could not find information what dynamics are actually learned in the robotic examples. I think it would be interesting to see how many activation functions are needed for accurately learning highly nonlinear dynamics, e.g., cart-pole swing-up, and how this affects the control performance. Overall, the novelty seems to be mainly the (straightforward) connection of existing ideas, even though I admit that it is a very clever combination. The proposed reduction of neurons is not even targeted to achieve a high control performance, but only to maintain high model accuracy. High control performance is only addressed a posteori by selecting the best model observed in experiments. Doing it as described in lines 206-208 runs the risk of executing a potentially bad controller on a real system. This seems like a dangerous thing to generally do. I think it would be much more interesting to directly optimize the model for achieving the best control performance. Finally, I find the comparison in the numerical evaluation a little weak. When no experiment has to be done like for the open-loop planning performance evaluation, I think more methods than only MPPI should be investigated. I am not an expert, but I know there exists more than one planning/control method for nonlinear systems, e.g., MPC with NN dynamics (Salzmann et al., 2023). Moreover, I do not understand why the learned dynamics model is not used as environment for the classical RL methods. It is apparently accurate enough to allow for a direct transfer to a real-world experiment. So why are the interactions limited to a relatively small number with the real environment, when a large number of interactions with a potentially only slightly more inaccurate model are available? T. Salzmann, E. Kaufmann, J. Arrizabalaga, M. Pavone, D. Scaramuzza and M. Ryll, "Real-Time Neural MPC: Deep Learning Model Predictive Control for Quadrotors and Agile Robotic Platforms," in IEEE Robotics and Automation Letters, vol. 8, no. 4, pp. 2397-2404, April 2023

Questions

What is the point of the regularization in (9) if you bound the total number of ReLu activation functions through $\epsilon$ in (5) anyways? What is a closed-loop planning performance? Or do you mean closed-loop control performance? In (8), is $z_{ij}$ the approximation of $\pi_{ij}$?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Potential negative societal impact is not mentioned in the paper. Some limitations of the proposed approach (e.g., restriction to ReLu activation functions) are mentioned throughout the paper, but there is no dedicated paragraph about limitations. I think there are some limitations (e.g., computational complexity) that should be added.

Summary

This paper proposes a new framework for model-based control. The approach focuses on learning a sparse deep neural network and using a mixed-integer program solver for closed-loop planning. Experimental results are presented on several tasks including object and rope manipulation tasks. The results show that the proposed approach improves performance over strong baselines.

Strengths

- The framework proposed in this paper elegantly combines concepts from deep neural network (DNN) pruning and mixed integer programming (MIP) into a solution for model-based control. - The specific technique for sparsifying a DNN by removing non-linearities rather than simply dropping nodes intuitively matches the goals of eventually using the model with a MIP solver. - The experiments demonstrate that the approach is effective in both simulation and in the real-world on real hardware -- providing strong evidence that this is a generally applicable approach. - The technique is demonstrated with both MLP and GNN based models, showing the versatility of the proposed approach. - The supplemental video presents strong qualitative evidence supporting the efficacy of the approach on real-robots. - The paper is very well written.

Weaknesses

- It is unclear if the performance gains in Figure 5 are significant. On the Object Pushing and Rope Manipulation tasks MIP does not appear to outperform MPPI. And the improvements of MIP on the Object Sorting tasks appear to be within the error bars. - A key ablation is missing. One of the main claims in this work is that the proposed sparsification technique, which focuses on removing nonlinearities rather than neurons, improves performance. Only a partial ablation in support of this claim is provided in the appendix A2. Specifically, A2 shows that the proposed approach leads to lower prediction error. However, to quote L277-279 in the main paper, “what we really care about is the performance when executing optimized plans in the original simulator or the real world. Therefore, it is crucial to evaluate the effectiveness of these models within a closed-loop control framework.” I agree. Thus, it is similarly crucial to perform this ablation in a closed-loop setting.

Questions

I am willing to adjust my rating if these questions can be answered: - Please comment further on the results in Figure 5; there is a trend but why is this a significant advance? - Does the proposed sparsification technique improve closed-loop performance? - What are the limitations of the work?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

4 excellent

Contribution

4 excellent

Limitations

No, the paper does not discuss limitations.

Summary

This paper proposes a framework for model-based planning with forward dynamics represented as sparse neural networks. The paper examines different ways of inducing sparsity in MLP and GNN based forward models, and performs real robot manipulation experiments investigating the tradeoffs with sparsity and performance.

Strengths

- the paper targets an interesting problem of learning efficient world models for control, by reducing optimization costs in learning the forward model. The specific approach of investigating how sparsity can enable this efficient learning of the forward model for real robot manipulation scenarios, is novel in my understanding. - the paper is easy to follow, with detailed descriptions of the different ways of inducing sparsity, and the different architectures. The research questions investigated are well formulated, and adequately addressed in the experiments - the experiments are on interesting real robot manipulation tasks like object sorting, and rope manipulation, so the findings are amenable to be deployed in real world control scenarios.

Weaknesses

- the main weakness of the paper is that there aren't comparisons to prior model-based RL approaches, e.g. PETS, Dreamer, MBPO etc (https://arxiv.org/abs/1805.12114,https://arxiv.org/abs/1912.01603, https://arxiv.org/abs/1906.08253) . If it is difficult to compare them on hardware experiments, there should be comparisons done on simulated robot control tasks. - it is unclear how general are the findings in terms of being applied to different model-based control frameworks. There should be a disucssion (and possibly experiments) about if we can take any prior model-based RL algorithm and replace the forward model with a sparse neural network, and see benefits under certain assumptions - if not, then a discussion of what exactly are the requirements on how the model should be learning and how a policy (or plan) should be learned is needed. - the experiments results in my understanding are on low-dimensional states instead of images. Is there a reason for not evaluating with image observations? Encoding image observations into latent states, and then doing policy learning is a common practice in the community now - so it will be helpful to have a discussion regarding this.

Questions

Please refer to my list of weaknesses above. In summary - comparisons to prior MBRL approaches? - discussion about generality of the findings? - feasibility of experiments with image observations?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations are a bit unclear at the moment (also refer to my list of weaknesses). It will be helpful to provide a detailed discussion of the limitations and generality of the findings.

Summary

This paper focused on the combination of predictive control and model learning. An autogressive dynamic model based on a ReLU neural network is first learned over the observation space. The authors then aimed to sparsify it after introducing the indicator mapping function. To make the optimization feasible, the Gumbel-Softmax trick is applied to replace the greedy operation. Once the sparse dynamic is learned, mixed-integer programming solvers are used to obtain the control policy. Finally, the authors validated the proposed method across a few tasks and showed its promising performance.

Strengths

1. The proposed framework of sparse neural dynamics in predictive control looks interesting. The use of Gumbel-Softmax to reparameterizing the original discrte optimization makes the gradient based methods feasible, and is indeed a reasonable idea. 2. The optimization for the control part is more efficient, compared with the gradient based methods, which constitutes another contribution of this paper.

Weaknesses

Regarding the experimental comparison, it would be more convincing if the authors can test on commonly used reinforcement learning benchmarks. The only comparison regarding control policy is from Figure 3 (b) and it only involves different optimization solvers. The current results would leave the impression that the proposed method may overfit on these tasks only.

Questions

1. What's the complexity of applying MIP? 2. What's the underlying architecture for the ReLu neural network? It would be a bit surprising if a feedforward neural network can capture the features from image inputs.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Yes

response to rebuttal

Dear authors, Thanks for the response and the additional experiments. The comparison to model-based baselines provided is helpful. I am still not convinced by the applicability to image based observations, which is important in order to be able to tackle more realistic tasks with less assumptions on the observed states. However, I am still leaning towards accept for the paper, as my other two concerns have been partially resolved.

Are there any additional questions or concerns?

We hope that our additional experiments following the reviewer’s suggestions, discussion of limitations, and responses addressed the questions and concerns raised. **We would be happy to continue the discussion if the reviewer has additional questions or concerns.**