Thank you for raising the rating and additional comments, we appreciate them! Here are our answers to additional questions.
**Q1. When the model is known, hence no cost is involved in learning the model, how well the new algorithm can outperform model-based RL in terms of both theoretical and empirical advantages.**
A1. According to [1], we believe that your suggested model-based RL with known models falls into the same category as models like AlphaZero [2], which leverage planning methods such as Monte Carlo Tree Search (MCTS). To briefly recap how MCTS works: since the world dynamics model (which are the Go rules for AlphaZero) is fully known, it "simulates" future steps of gameplay using the estimations of its neural network as a prior. That is, it rolls out future gameplay steps based on rough suggestions from the neural networks, builds a search tree based on them, and selects the action that yields the best reward. In this approach, the model is fully known and is thus used as a tool to simulate various scenarios.
However, we’d like to note that this strategy works in a very different environment than ours - it works in the **discrete** action space, while our method is for **continuous** one. It is unclear if we can apply planning methods like MCTS to our case or not, and thus it is difficult to compare it directly to our method. In spite of that, we’d like to point out that even if we could apply MCTS-like planning methods to our physics simulation problems, it could require much more computational cost for training. This is because it has to expand its search tree by taking a large number of timesteps, and take a SINGLE action based on it. For instance, AlphaZero runs 1600 simulation steps to build a search tree for determining a SINGLE action. Considering that the most of the computational cost comes from the expensive simulation costs for physics simulations, it is prohibitive to run such a number of simulation steps for determining action at a single time step, and thus training the network. Therefore, we’d like to underline the difficulty to directly compare model-based RL with a fully known model to our method. At the same time, the planning method like MCTS used for such model-based RL methods could require much more computational cost than ours in the worst case scenario.
Exploring how analytical gradients can be efficiently applied to model-based RL methods would be another very exciting research direction.
**Q2. I still don't understand the difference between completeness and bias based on the authors' explanation. I think a more thorough mathematical definition is required. This is also the case regarding the explanation on the ratio of the two determinants and why this can measure the difference between two policies.**
A2. First, after some reconsideration, we realize that ‘completeness' is not an appropriate term to use in the broader, general context. We will remove the term in the revision and explain our method using bias only. However, it does not change our main claim that our method can leverage possibly biased analytical gradients better than the other baseline methods. Thank you again for pointing it out to improve the exposition of the paper.
About the second issue, we’d like to reemphasize that the ratio of the two determinants does not “measure” the difference between the two policies. Instead, our view is that it is more likely that the two policies are more different from one another when the ratio deviates more from 1, at least locally, because the ratio is 1 when the two policies are the same. In the revision, empirical results will be included to support this claim.
**Q3. I believe the main text of the paper should be self-contained, without relying on any appendices. Hence, the definition of A hat should be in the main text.**
A3. We agree with your point, and will move the definition of A hat to the main paper. Thank you for your suggestions.
[1] Moerland, Thomas M., et al. "Model-based reinforcement learning: A survey."
[2] Silver, David, et al. "Mastering the game of go without human knowledge."