Thank you for your follow-up comments! Please find our responses to your remaining concerns as follows:
**Improvement of Residual-Q over Baseline**: We respectfully disagree with your assessment that the improvements of Residual-Q over the RL fine-tuning baseline are extremely minor. When compared to RL fine-tuning, residual Q-learning consistently achieves better performance when a sufficiently good prior is available. The advantages are particularly apparent in the Parking and Ant environments. Note that the performance advantage is also apparent in the Humanoid environment when the prior policy is trained with RL ---- the greedily customized policy has significantly larger variances in its rewards compared to the residual Q-learning policy. The performance gap between residual-Q and RL fine-tuning depends on the basic and add-on rewards. If the additional add-on reward does not significantly alter the optimal policy, policy divergence is a fairly reasonable IL objective and it is fairly easier to find a good trade-off between the IL and RL objectives, even though policy divergence is a heuristically designed IL objective. For example, it is the case in the Hopper environment ---- while the customized agents jump higher, the gaits are still similar to the prior policy. In contrast, if the add-on reward does significantly alter the optimal policy, regularizing the policy KL divergence does not necessarily maintain the customized policy's performance on the basic task. For example, in the Parking environment, because of the non-holonomic constraint and long task horizon, the agent's optimal behavior varies significantly when the additional non-collision constraint is imposed. Thus, RL fine-tuning fails to find a good customized policy that can reliably complete the basic parking tasks. Conversely, a good prior policy can still be used to encode the basic task objective in the residual Q-learning framework, which explains the dominance of the residual-Q policies over the greedy ones. Therefore, even though the greedy policies perform similarly to residual-Q in some environments, the fact that residual-Q can consistently outperform RL fine-tuning given a good prior validates that residual-Q is a more principled approach to handling the trade-off between IL and downstream RL objectives.
**Novelty**: The novelty of our work lies in both the new problem setting, policy customization, and the proposed solution, residual Q-learning. The residual-Q learning framework provides a theoretical ground to interpret and determine the trade-off between the imitative and add-on objectives, which has never been discussed in the related literature (e.g., RL fine-tuning). While the family of residual Q-learning algorithms is adapted from existing algorithms in a straightforward way, we consider the simplicity in algorithm design an advantage of our proposed residual-Q learning framework. As summarized by Reviewer ePZC, it shows that the proposed framework is flexible and can be easily added to existing maximum-entropy RL algorithms.
**Comparison with IRL**: As mentioned in the author rebuttal thread, since the residual-Q framework does not require inferring the inherent reward from the demo, it allows flexible adoption of any maximum-entropy imitation learning algorithms depending on the environments, including AIRL. Thus, residual-Q should be able to consistently outperform policy customization methods that require explicitly inferring the inherent reward from the demo, since AIRL could perform significantly worse in certain environments. Besides, AIRL also needs to train a policy simultaneously while inferring the reward. Thus, we do not think the demand for a prior policy for online querying makes Residual-Q less favorable than IRL.
**Principled Way**: We understand that determining the trade-off between different rewards can also be difficult in some scenarios. However, we want to emphasize that, compared to RL fine-tuning where the IL objective is added as a heuristic, interpreting the trade-off between IL and RL objectives as a trade-off in rewards as in Residual-Q is a more principled and theoretically sound approach. It has been validated by our experiments comparing Residual-Q versus RL fine-tuning.