Summary
This paper proposes new algorithms for prediction and control in continual reinforcement learning. The new algorithms use permanent and temporary memories. The permanent memory captures a general value function for all the tasks, while the transient memory is used to adapt to the current tasks continually. The paper provides theoretical results for the prediction setting with a tabular learner. Additionally, the paper provides empirical results showing that their method outperforms vanilla TD-based algorithms in various environments using different types of function approximators.
Strengths
The paper directly tackles the continual reinforcement learning setting.
This is a challenging problem setting with few general algorithms and evaluation environments.
The paper provides prediction and control algorithms for the continual RL setting.
Theoretical proofs of convergence are provided for the prediction setting with a tabular learner.
The extensive and sound empirical investigation shows that the proposed method can generally outperform existing methods in various environments and with different various function approximations, including Deep Networks.
Weaknesses
The paper does not have any major weaknesses. I did not check the proofs, so I can not comment on the correctness of the proofs.
However, there are a few minor issues that can be fixed to improve the paper.
1. I'm confused about Equation 5, specifically line 112. Does it mean the "gradient" or the "semi-gradient" of the transient memory?
2. What are the confidence intervals being reported? Line 222 says it is the 90% confidence interval, but is the bootstrapped confidence interval or confidence interval based on the standard error? And if it is based on the standard error, how do you know it is the 90% confidence interval?
3. The continual deep RL algorithms used in the paper use standard deep RL tools like Experience replay and target networks. However, the exact use of these tools is not fully explained in the paper (or the appendix). This creates a lot of questions; for example, are there a total of three networks, target, transient and permanent? I suggest that the authors include a section in the appendix that gives the pseudocode of the PT-DQN algorithm. The lack of details about the PT-DQN algorithm makes it hard to replicate the results given in the paper.
Questions
1. Line 88 says, "... multi-task RL methods ..." What is multi-task RL, and how is it different from continual RL? What limits the use of multi-task RL methods in continual RL? The second question should be answered in the main text.
2. Figure 4d shows the performance of different versions of DQN on Minigrid. Why is the performance of all the methods, particularly DQN(reset), so much worse during the second task? If the DQN is being reset at each task, it should perform similarly on Episodes 500 and 1000. Is this because the first goal state in all the runs is the same?
3. What is PT-DQN-0.5x in Figure 5? 0.5x should be described in the main paper.
4. Lines 733-742 describe the training details for the deep RL experiments. However, many of these choices seem arbitrary, or the paper does not explain the reasoning behind these choices. Why are the conv layers not updated when updating the permanent memory? Why isn't there a fully independent permanent value network? Wouldn't that be closer to the algorithms presented in the paper?
5. Lines 803-805 say that the permanent memory was trained with SGD while the rest of the network was trained with Adam. Why this specific choice? I'm guessing the authors also tried Adam to update the permanent memory. Why didn't Adam work for permanent memory?
6. The plots (like Figure C.13) showing the sensitivity curve will be better with a log scale for the x-axis.
7. Why isn't there any effect of PM LR on the performance in the minigrid environment (Figure C.13c)?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
One limitation of this work is that it only proposes methods for value-based RL. It is unclear how the idea of permanent and transient memory can be extended to policy-gradient based RL. A discussion on this direction of future work will be helpful.