In a standard view of the reinforcement learning problem, an agent's goal is to efficiently identify a policy that maximizes long-term reward. However, this perspective is based on a restricted view of learning as finding a solution, rather than treating learning as endless adaptation. In contrast, continual reinforcement learning refers to the setting in which the best agents never stop learning. Despite the importance of continual reinforcement learning, the community lacks a simple definition of the problem that highlights its commitments and makes its primary concepts precise and clear. To this end, this paper is dedicated to carefully defining the continual reinforcement learning problem. We formalize the notion of agents that "never stop learning" through a new mathematical language for analyzing and cataloging agents. Using this new language, we define a continual learning agent as one that can be understood as carrying out an implicit search process indefinitely, and continual reinforcement learning as the setting in which the best agents are all continual learning agents. We provide two motivating examples, illustrating that traditional views of multi-task reinforcement learning and continual supervised learning are special cases of our definition. Collectively, these definitions and perspectives formalize many intuitive concepts at the heart of learning, and open new research pathways surrounding continual learning agents.
Paper
Similar papers
Peer review
Summary
The paper looks at developing a foundation for continual reinforcement learning (CRL). The authors develop definitions and insights, aiming to formalize the intuitive concepts in continual learning fields. They also provide two examples of CRL to illustrate the difference between traditional RL and CRL, which is viewing learning as endless adaptation instead of finding a solution.
Strengths
1, The concept of CRL as agent designer aiming to identify an optimal agent among available agents is innovative. 2, The use of rigorous math tools and definition in this paper could provide valuable insight into continual learning and potentially inspire future CRL algorithms development.
Weaknesses
1, Though it's not the focus and goal of this paper, the idea of agent basis contains redundancies and may hinder the transformation of it into a memory and computation efficient CRL algorithm. 2, The training process of agent basis is unclear in Figure 2 (b). I guess two examples are presented to show the effectiveness of CRL, but it's not surprising that CRL can beat RL without adaptation in these changing envs. More online learning benchmarks would help to achieve a fair comparison. 3, The paper is not well structured, and only few related works are mentioned. Continual Learning is discussed in many fields, with different names (e.g., online learning, few-shot adaptation, adaptive control), it would be valuable to provide a broader literature review.
Questions
The high-level idea of CRL training is understandable, but I am unsure about the training details in Section 4.2 CRL examples. Including a pseudocode description could clarify the training process and help understand the implementation details.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Overall, the paper requires a lot of additional work to become publishable. Adding figures and pseudocode would enhance readability and understanding. Furthermore, conducting more experiments to demonstrate the effectiveness of the proposed CRL idea would strengthen the paper's contributions.
Summary
In this paper, the authors develop a simple mathematical definition of the continual RL problem. These definitions, insights, and results formalize many intuitive concepts at the heart of continual learning, and may open new research pathways surrounding continual learning agents.
Strengths
The mathematical definition of continual reinforcement learning give us some inspirations when thinking of the continual learning, multi-task RL, continual supervised learning, and so on.
Weaknesses
1.The abstract is too simple that readers cannot get enough information and key ideas from it. 2.This paper is not easy to follow, for the overall mathematical definitions and analysis. I hope the authors can give us more cases or experiments using the definitions proposed in this paper to highlights the benefits of following such definitions.
Questions
1.What information can we get from Figure 2(a) Switching MDP Visual? Do authors just want to emphasis “Each underlying MDP shares the same state space and action space, but varies in transition and reward functions”? 2.How can we prove or evaluate the definitions, the proposed operators, and so on? 3.Can authors add some ablation studies in this paper?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
Please modify the paper as the weaknesses and questions above, and add limitation analysis in the paper.
Summary
The paper proposes a mathematical formulation for the problem of continual reinforcement learning in an infinite horizon setting.
Strengths
1. The authors propose a new mathematical formalism for continual RL. 2. The paper can be of interest to mathematically inclined readers and could potentially lead to theoretical results in continual reinforcement learning.
Weaknesses
1. Abstract: there already are foundations for CRL. You cited a survey paper [16] and multiple other existing works. As unconventional as it is, I wouldn’t be against a short abstract, but I do not believe the current one will do. 2. Missed important opportunities to connect your formalism with existing (continual) RL. E.g. eq 2.1 states an infinite horizon, which is required for example in Theorem 3.1, but the very standard “infinite horizon” vocabulary never appears. I think the paper is not connected enough to existing (continual) RL literature. 3. Generally, I’m not convinced by motivations. The paper says it is important, but I did not really see a convincing motivation for abstracting the problem of continual RL. The theorems provide results that can be conveyed in significantly easier ways with existing formulations: e.g. non commutativity in thm 5.2 signifies that in continual RL, some policy transitions cannot be “unlearnt” in some settings. The formalism incurs a very significant overhead over such high level language – I believe that the authors can improve the motivations to justify why their formalism is worth it. **Typos and suggestions:** 1. L10: we instead of We 2. L41: I would use $\mathcal{Z}$ instead of $\mathcal{X}$ so it flows better in the following sentence. 3. L84: phrasing “instead opt for simply the bounded…” 4. L93: did you mean the sum of rewards to start at index 0 ? or t+1, t+2, etc. 5. L95: I know it got absorbed into v, but I would still prepend this line with “Given <initial conditions> $h_0$, any tuple…” since your work aims to formalise rigorously the problem. 6. “We use as if in the sense of the positive economists” this is terminology that should be defined in the paper. 7. L334: “generates is commutative” -> are 8. L338: I recommend not defining objects used in a main body theorem in the appendix. 9. I believe that the insights, as stated in the conclusion, are trivial; perhaps “insight” isn’t the word that should be used (using the Cambridge dictionary definition of “insight” “ a clear, deep, and sometimes sudden understanding of a complicated problem or situation” – I do not believe the idea for example that agents in CRL either can or cannot find a set of policies/behaviours is insightful).
Questions
No.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
Yes.
Summary
- This paper lays out a foundation for continual reinforcement learning (CLR) from the ground up—establishing definitions for the purpose of building towards a technical definition of CRL itself, proving various properties of CRL, and through employing simple CRL examples, demonstrating some of these properties.
Strengths
- The writing is exceptionally clear and precise. - The framework is well thought-out; the definitions are clear and useful, and the framework as a whole is both technical and intuitive. Importantly the provided definition of CRL seems to get at the root of what researchers have always vaguely meant by CRL. - There is a rich foundation set for future work (e.g. exploring “connections between our formalism of continual learning and some of the phenomena at the heart of recent empirical continual learning studies, such as plasticity loss, in-context learning, and catastrophic forgetting”; line 364).
Weaknesses
- I understand the appeal of a short abstract (i.e. it is to the point, and (albeit somewhat pompously) implies some special finality or importance to the contributions in the work), but the resolution of the abstract, for good reason, usually sits between the resolution of the title and the paper itself. The abstract in this case does not provide much more information than the title, so any reader that wants a quick overview of the paper must read the paper itself, defeating the purpose of the abstract. I suggest the authors to reconsider employing such a short abstract. - I feel that the paper could have developed more connections to previous literature, performed more experiments building off of this framework, etc., i.e. more validation for why we should care about this framework. Better (nonstationary) environments to evaluate agents on? Agent structures? Assessment of how well current methods fit into/exploit elements of this framework? The framework is nice, but it would be useful to see how it can situated into current literature, either by retroactively analyzing current agents/environments w.r.t. this framework, or designing new agents/environments from the principles this framework provides. - Maybe more real-world examples of changing environments? Or where we want the behavior to change in the same environment (but then we just have an oscillating agent…). Something about the environment changing (or being better understood over time) is fundamental to continual RL) - It took me a number of readings to understand agent bases; i.e. the relationship between $\Lambda$ and $\Lambda_B$.
Questions
- Do we really want agents that will adapt their policy forever? Or are the environments we care about simply non-stationary, i.e. because the world changes, and the agent is required to learn more about the environment in order to continue to act optimally? Maybe the policy (in some high-level sense) should remain the same, and it is the world model that should be adapted? E.g. suppose the expressed behavior of the agent policy can be boiled down to “helping humans”, and this policy is fixed, given the agents current understanding of the world as context. Perhaps only this context needs to be updated, i.e. as the world changes, and/or the agent learns more about the existing world. This seems to fit within the proposed framework, if we consider the inferred world model as part of the agent. And while it does seem likely that the discovery of new contexts would require updating the policy, it still appears to me to be beneficial to disentangle these aspects explicitly. What are the authors' thoughts on this?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
4 excellent
Contribution
3 good
Limitations
- The authors have discussed the limitations thoroughly, and I have listed them above as a strength as there are many directions ripe for future work.
Summary
The authors propose a new definition of continual reinforcement learning where an optimal continual learning agent will not converge to a fixed policy. This is formalized through the generate operator, which defines the "searching" between different policies; and the reach operator, which defines whether the agent will stop searching.
Strengths
1. This paper provides a unique definition of the continual learning problem, which is formalized by their mathematical framework. 2. Based on their framework, the authors were able to prove a number of interesting theorems.
Weaknesses
1. It is not immediately obvious how this definition can inspire new learning algorithms. 2. The underlying concept that an optimal learning agent should not converge is not surprising. The author also mentioned that it's an "unsurprising conclusion that it is better to track than converge" for their toy non-stationary environment.
Questions
1. Can you elaborate on how this formal definition of continual learning can address practical problems such as plasticity loss, in-context learning, and catastrophic learning as mentioned in the Discussion section?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors mention the limitations implicitly as future work in the discussion section. Essentially, since this paper is almost purely theoretical, they treat the lack of empirical study as future work or limitation.
Summary
This is an ambitious paper. The paper notes that the problem of "Continual Reinforcement Learning" lacks a rigorous definition and seeks to provide one. It mathematically defines the reinforcement learning problem and then explores the conditions in which an instance of the RL problem is a CRL problem. The legwork is done through the new "generates" and "reaches" operators. The "generates" operator accepts two arguments: 1. a set of "basis" agents, and 2. a particular agent (which may not be in that set, but is exhibits some combined behavior of the basis agents). It returns boolean truth value. The "generates" operator expresses if there exists a learning rule that can search over "basis" agents (with respect to an environment / historical observations) and find a combination of them that is equivalent to the particular agent in the argument. This function is also undecidable in general. Loosely, the "reaches" operator (wrt an environment) accepts two arguments: 1. a particular agent 2. a set of "basis" agents It returns a boolean truth value. It returns True when an agent's behavior settles on an element in the basis set. The authors actually define a more precise "sometimes reaches" and "never reaches" operator. This is also undecidable. A learned agent will always choose a behavior equivalent to some basis agent in the context of the environment and history. Using these formalisms, a CRL agent is one that never reaches an agent basis, and a CRL problem is one where the best agent is a CRL agent. Notably, a problem is a CRL problem depending on the choice of basis. The paper performs mathematical analysis on these operators, relates them to two example instances of CRL, and provides a thoughtful discussion for new questions that can be asked about agent basis sets.
Strengths
The informal definition "An RL problem is an instance of CRL is the best agents never stop learning" is intuitive and clear. I always appreciate explicit definitions for notation. (although defining logical negation, and the quantifiers might not be necessary in a conference paper). I type checked several of the equations and everything seems to work out nicely. The theory gives a good feel for how someone might implement a framework for exploring agents in the language of this theory using a strongly typed language. The paper gives explicit examples and experiments demonstrating intuitive properties of this theory in the setting of Q-learning and finite Markov decision processes. As well as a more complex example in continual supervised learning. The paper proves several basic properties of the new reaches and generates operators. I have not checked the proofs for correctness, but I also don't see any obvious problems. Section 5 made me delete a lot of my questions because it answered them. The idea of exploring choices of the basis is mathematically interesting. The paper raises more questions than it answers.
Weaknesses
The abstract is terse - problematically so. Even if this succeeds at being a theoretically sound definition of continual reinforcement learning, I think it is important to elaborate at least a little bit more. Note this is the main reason I'm rating this a 5 / 10. Given a better abstract I think this is a 7/10, but I think it is critical to give a better overview to the researchers' attention you are competing for. A 9 page paper with a 20 page appendix full of proofs may be better suited for a journal. The paper raises more questions than it answers.
Questions
On line 47 the authors explicitly call out that the set A and O are countable. Does this imply that some other (non-numeric - i.e. non-ℝ) sets may be uncountable in this definition? I think the order of appendix A and appendix B should be switched. Provide the notation first. (The table was very helpful in reviewing). I became confused when I first read Theorem 3.1. Particularly when the selected agent was not an element of the agent basis. In hindsight it makes sense that the new agent is a (linear?) combination behaviors in the agent basis set, but I think the paper could make that more clear. It wasn't intuitive what "switches" meant to me. Or that lambda(h) in Figure 1 was a combination of the basis agents conditioned on history. (At first I thought the learning rule was just choosing an element of the basis set). In Theorem 3.1 are the infinite choices of basis countable? If a problem's status as CRL or not depends on the choice of basis, and there are an infinite choices for the basis, then is this framework all that useful for thinking about problems? Theorem 5.1.2 suggests that it is possible to reduce any CRL problem to an RL problem via a change of basis. Is it possible to reduce any CRL problem to a RL problem by finding a function that changes the basis? Are there problems where such a mapping does not exist? Have you formalized any of the proofs into a proof assistant like Lean? I took a peek at the appendix, but I simply don't have the time to verify these proofs. Being able to point to an automatically checked proof in each theorem would greatly increase the quality of the paper.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
An optimal CRL agent could be difficult to rectify with the alignment problem. In contrast, it might be the case that deciding if a CRL agent will remain aligned with its creators is undecidable. That question is worth exploring.
Summary
In the paper the authors propose a formal framework for reasoning about continual reinforcement learning problems. To this end they introduce mathematical definitions for environments and agents which serve as a basis for defining the general reinforcement learning setting as well as the continual setting. In particular, the authors tie the problem of continual learning to the learning progress an agent makes and two operators describing its limitting learning behaviour. Finally, they provide two examples and derive sevaral characteristics of the proposed operators.
Strengths
The problem of continual RL is most certainly relevant to the community and will arguably become more prominent as real-world agents adapting throughout their deployment become more common-place. To my knowledge the proposed formalism is original in its attempt to characterise this setting formally. Overall I found the framework sound and its description easy to follow. I agree with the authors that such a formalism has the potential to open up new perspectives on and analysis of (long-term) learning behaviour of agents.
Weaknesses
Throughout the paper the terms "agent" and "behaviour" are used almost interchangeably. However, a formal definition of behaviours is missing, making it somewhat unclear whether or not the two terms are actually meant to be equivalent. If so, the statement "we can understand every agent as implicitly searching over a set of behaviors" needs clarification. If not, I would encourage the authors to formally define what a "behaviour" constitutes. A related issue is the separation of agents and their learning rules. Arguably, an agent parameterised by a recurrent network is a continual learner as it changes its "behaviour" (in the sense of the one-step policy) based on the history giving rise to its internal state. However, under the provided definition such an agent would not be considered a continual learner unless it updates its weights. Finally, the paper would benefit from a more in-depth discussion of how previous approaches to continual RL compare to the proposed framework or can be embedded into it. Similarly, the authors mention several avenues for future research such as the examination of catastrophic forgetting - it might be interesting to include at least a characterisation of these problems in the new framework to inspire future work.
Questions
-
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
-
I would like to thank the authors for the extensive replies to every reviewer and acknowledging the importance of rethinking the abstract and litterature/contextualising. > We believe that precise definitions are essential for clarity in science. For example, prior to the discovery of pseudo-randomness, much work in theoretical computer science relied on ad-hoc heuristics for producing random sequences to the detriment of the effectiveness of these approaches. Then, a series of papers carefully defined what it would mean for a sequence to be pseudo-random (for instance, in “The definition of random sequences” by P. Martin-Lof, and “Theory and applications of trapdoor functions” by Yao). On Yao’s view, a pseudo-random sequence is one that a bounded adversary cannot distinguish from truly random. Achieving this clarity allowed for pseudo-random number generators to be developed, and for the community to direct its research at the right concepts, and toward the right objectives. In our view, continual reinforcement learning is at a similar stage in its scientific development to pseudo-randomness in its infancy: we lack precise definitions for what, exactly, it is we are conceptualizing. I agree, as someone who often has to be on the other end of this argument. However, in my opinion, the less accessible a formalism is (and I do believe the one presented here is on the more abstract end of the spectrum), the more burden there is to motivate it. For example, Balduzzi et al.'s *The Mechanics of n-Player Differentiable Games* was a highly impactful paper not only due to its simplicity, but also for proposing in the paper new analyses using their formalism. I understand that the problem here may warrant a more complex formalism, but I still believe proposing something using that formalism is important. > Some people think of non-stationarity, others think of computational constraints, while others think of multi-task learning and transfer. Our goal in developing a single abstract definition is to encompass all of these views as special cases, and the point of formalizing it is to remove any ambiguity in the use of terms like “continual learning”, or “continual learning agent”. That's a unrelated to your submission but 1) I believe *non-stationarity* already captures multi-task learning and transfer, and 2) are those terms really ambiguous ? I have not experienced much disagreement over the definitions of CL that has somehow impeded discussions/led to misunderstandings. > First, to clarify: the second insight does not state an agent in CRL can or cannot find a set of policies. This does not reflect the semantics of Remark 3.2. The remark says that every agent can be classified into exactly one of two families: (1) the agent eventually stops its search over its corresponding basis, or (2) searches forever. I might be missing something re: a third possible outcome. The way I'm reading this is "either A or not A" where A = "the agent eventually stops its search". Again, I assume I'm missing something, so could the authors clarify this point further ? Also, I believe what I stated is the loose equivalent of that, under equating agent with policy, and assuming the search stopping = "finding", but I may very well be wrong. > Second, to justify further why we chose the term “insight”: we provide a formalism of an intuitive fact that holds true of every possible agent in a way that, in hindsight, the fact seems clear and even obvious. Even if the intuition is obvious (in hindsight) While I understand the point of having it stem from the formalism, and you might argue the following is semantics, but it's more foresight than hindsight. The formalism is developed (just like in natural sciences) with the goal of matching current knowledge of CL, but capturing them with a more rigorous formulation. After all, that is the motivation of the paper.
First Response
We thank the reviewer for their thoughtful reply. We have two immediate reactions: > I might be missing something re: a third possible outcome. The way I'm reading this is "either A or not A" where A = "the agent eventually stops its search". We now believe we agree here: the remark is pointing out a claim of the form "either A or not A", exactly as you suggest, and A = "the agent eventually stops its search", as you suggest. Our original comment was indicating that we believed there was misunderstanding on what A was ("finding a set of policies" vs. "agent stopping its search"). We believe that we are all now on the same page. And, regarding the the choice of the term "insight": > While I understand the point of having it stem from the formalism, and you might argue the following is semantics, but it's more foresight than hindsight. The formalism is developed (just like in natural sciences) with the goal of matching current knowledge of CL, but capturing them with a more rigorous formulation. After all, that is the motivation of the paper. Thanks, that resonates. Our main points here are: that: (1) formalizing intuitions about _all_ agents into rigorous statements that reflect common knowledge and intuitions is non-trivial, and (2) the first "insight" (Theorem 3.1) is really intended to be the more "insightful" of the two. We want to emphasize that we are more than happy to change the paper to move away from the term "insight", as we appreciate and understand where the reviewer is coming from with this point.
I thank the authors for their thorough response. I maintain my positive opinion of this paper, especially in light of the authors' (tentative) updates to the draft. I believe this work is ambitious and useful to the community; and that the potential for this work to spur more technically-grounded future research in this area far outweighs the weaknesses pointed out by myself and other reviewers---weaknesses I also believe have been adequately addressed by the authors.
Overall I like this paper, and I think the indended revisions are sufficient for me to raise my rating. > First, we conjecture that any definition of continual learning that involves concepts like “learning” and “convergence” will have to sit on top of some reference object whose choice is arbitrary. I buy this. I'd be interested in seeing a formalization of this conjecture.
I'd first like to thank the authors for the detailed response. I think this is indeed an interesting direction, and I raised my score accordingly. Perhaps a bit outside the scope of this work, but I feel that actually analyzing plasticity using a toy task under the proposed definition, for example, can make the paper a lot more compelling.
To Authors: Thank you for submitting your rebuttal. We appreciate your efforts in addressing the reviewers' comments and providing clarification. To Reviewers: We kindly request you to complete your response to the authors' rebuttal as soon as possible. Time is of the essence, and the deadline is fast approaching. Your timely feedback and expertise are crucial in ensuring a fair and thorough evaluation process. Please prioritize reviewing the authors' response and provide your final feedback accordingly.
Thanks the authors for their thorough response and it really helps my understanding. My concern was that the agent needs to switch between the a set of base agents and this set can be large and needs more training power. Like sec 4.2 CRL example, we have four policies to train and then learn a particular agent by searching over the base agents. Please let me know if I get it right.
Response to Comment
We thank the reviewer for following up, and we are glad to hear our response has helped. > My concern was that the agent needs to switch between the a set of base agents and this set can be large and needs more training power. Like sec 4.2 CRL example, we have four policies to train and then learn a particular agent by searching over the base agents. Please let me know if I get it right. To clarify, when we talk about agents switching between elements of an agent basis, we do so in a general way: we mean that _every agent_, including all of those reinforcement learning algorithms we have designed and implemented to date, can be understood in this way. For instance, consider DQN (Mnih et al. 2015). The Q-network maintained by this algorithm at any point of time can be thought of as the currently active element of the basis. Then, through learning from experience, the algorithm updates the parameters of this Q-network, thereby updating the currently active element of the basis. In this way, all assignments of parameters to the network is the agent basis (the set), and any element of this set is one of the possible elements of the basis (a specific Q-network, and thus, a policy). In the example in section 4.2, there are not four policies to train---both agents (standard Q-learning and Q-learning with a fixed step-size parameter) search through the space of Q functions, roughly. As a result, we do not quite understand the point that that an agent needs more training power, as the above perspective applies to all agents. We do agree that continual RL problems are likely to be _difficult_ learning problems in general: learning to adapt endlessly is likely _harder_ than learning to find a single solution to a problem in many cases. We believe it is important for the community to embrace and confront this difficulty, which is why we think it is useful to carefully define the problem. Does that help? We are happy to help clarify or discuss further.
Thanks for your clarification. Now I get your point: the agent basis is the previous policies. This is interesting and it wouldn't cause extra training power. You have solved my concerns, and I will adjust the paper score accordingly.
I'd like to thank the authors for their thorough responses. I think the proposed changes sufficiently alleviate most of the weaknesses and am updating my score accordingly.
Decision
Accept (poster)