Neural networks are traditionally trained under the assumption that data come from a stationary distribution. However, settings which violate this assumption are becoming more popular; examples include supervised learning under distributional shifts, reinforcement learning, continual learning and non-stationary contextual bandits. In this work we introduce a novel learning approach that automatically models and adapts to non-stationarity, via an Ornstein-Uhlenbeck process with an adaptive drift parameter. The adaptive drift tends to draw the parameters towards the initialisation distribution, so the approach can be understood as a form of soft parameter reset. We show empirically that our approach performs well in non-stationary supervised and off-policy reinforcement learning settings.
Paper
Similar papers
Peer review
Summary
The paper focuses on non-stationary learning of neural networks. The paper proposes a method that automatically adjusts a parameter that influences the stochastic gradient descent (SGD) update to account for non-stationarity.
Strengths
The problem tackled is relevant and the related work is clearly discussed. The method is well-interpreted and relevant for the community. The paper is also very well presented.
Weaknesses
In my view, the paper has some weaknesses, especially regarding the experimental validation: - The experimental validation does not clearly validate the benefits of the method, possibly due to lack of clarity. In Figure 3, it is not clear to me what method this paper proposes. I believe the paper should propose a method and validate it, instead of comparing all its possible variants. Nevertheless, I can see that most (if not all) of such variants are outperforming the chosen baselines. However, only two of the baselines referred in the related work were chosen. The chosen benchmark problems are also not motivated to me. - The reinforcement learning part is not clear to me. Is there an explanation for why, on-policy, the method is outperformed by the simple baseline? Is not on-policy the most non-stationary setting? Why? And why does the proposed method stop earlier than the competitors in the top-right figure? - It is not clear to me what the authors intend to show with Figure 2, and the perfect soft-resets regime. It would be nice to have more explanation here, and possibly move this validation to the end of the experimental section, as appearing before the main validation of the method is confusing. Finally, even though the method is well motivated, it is not theoretically analyzed. We have no theoretical proof that the method will outperform traditional SGD in terms of efficiency, or preventing plasicity loss and catastrophic forgetting, or other dimension. Minor: in page 6, Equation (15), what is the parameter $\tilde{\lambda}$? Was it introduced before?
Questions
I have the following questions: - Could the authors make some clarifications with respect to the experimental validation? Specifically, on the choice of baselines, problems, which method or methods the authors in fact propose to the community. - Can the authors also clarify my concern regarding the reinforcement learning setting and the performance in on-policy settings? - Can the authors clarify what is the intended take-away from the analysis in Figure 2? - Can the authors clarify the Monte-Carlo samples in Equation (7)? Specifically, what is sampled, for instance in the reinforcement learning setting, and if the sampling requires access to a free simulator of the environment.
Rating
5
Confidence
3
Soundness
2
Presentation
3
Contribution
2
Limitations
Yes.
Summary
- The authors study a learning algorithm that can handle the non-stationarity of the data distribution. - They propose a parameter drift model based on the Ornstein-Uhlenbeck process, which models a form of “soft parameter reset” adaptive to the data stream. The drift model has an adaptive parameter $\gamma_t$ which a gradient-based optimizer can learn online. - They illustrate the update rule of the main parameters of a neural network incorporating the learned drift model. The update rule is first proposed under the Bayesian neural network framework and then later adapted to a non-Bayesian neural network. - They numerically corroborate the efficacy of their method on plasticity benchmarks and reinforcement learning experiments.
Strengths
- S1. They propose a novel way to learn online the amount of resetting the parameters. - S2. The method is general enough to be applied broadly to continual learning problems and reinforcement learning problems.
Weaknesses
- W1. Modeling parameter drift is not well-motivated. - The learnable drift model $p(\theta_{t+1} \mid \theta_t, \gamma_t)$ is the main contribution of this work. In my opinion, however, it is unclear why we should care about the drift of “currently learning” neural network parameters. - According to the beginning of Section 3, the local minima of the objective function may change over time. This is of course acceptable. However, I cannot understand what is relevant between the change of the local optima $\theta^{\star}_t$ and the current parameter $\theta_t$. To be more specific, the two sentences in lines 94-96 are not connected well. - W2. The paper is NOT self-contained overall. - The authors claim that they chose the Ornstein-Uhlenbeck (OU) process as a drift model. However, it is unclear how a continuous stochastic differential equation (OU process) can be converted to a discrete Markovian chain defined in Equation (4). The paper does not even introduce the actual form of the OU process. Moreover, it is not very clear why the authors chose the particular model defined in Equation (4) because it is not known (at least in the paper) that the model is a unique option to choose. - There are a lot of equations whose derivation is omitted. Let me list them: Equations (9), (10), (11), (14), (15). In particular, I am suspicious of both the validity and usefulness of Equation (10). - For these reasons, the paper is not so easy to follow. - W3. The soft reset method seems computationally heavy. - I am worried about the computational cost. Although it is good to learn the forgetting parameter $\gamma_t$ online, it increases the computational cost almost twice. It might be more than just twice because learning the $\gamma_t$ requires Monte-Carlo (MC) sampling. - In addition, there seem too many hyperparameters to tune. - W4. Minor comments on typos/mistakes - Do not capitalize the word “neural network”. (e.g., line 35-36) - The term “plasticity loss” may be read as a type of loss function for some people (new to this field). I recommend using the term “loss of plasticity”. - Line 84: “non-stationary” → “non-stationarity” - Line 109: “mportant” → “important” - Line 127: “wrt” → “with respect to”. Do not use abbreviations. - Line 137: “property” → “properties” - Line 154: what is “s” next to a parenthesis? - Line 168: “$\theta = (\theta_i, \ldots, \theta_D)$” → “$\theta = (\theta_1, \ldots, \theta_D)$” - Line 169: “$\mathbb{R}^D$” is a more standard notation than “$\mathcal{R}^D$”. - Line 222: In “$\lambda\_i = \hat{\lambda}\_i \sigma^2_{t,i}$", I think $\hat{\lambda}$ must not have a subscript. - Line 225: “sinc” → “since” - Equation (14): What is “$\tilde{\mathcal{F}}$”? (Partial) gradient of “$\mathcal{F}$”? - Line 232: What does it mean by “we assume that $\mu_0 = \theta_0, \sigma_0^2.$”? - Line 236: “$\theta_{t,i}(\gamma_{t,i}) = \theta_t$” → “$\theta_{t,i}(\gamma_{t,i}) = \theta_{t,i}$” - Line 240: “linearisng” → “linearizing” or “linearising” - Line 292: duplicate “See” - Line 354: “modelling” → “modeling” - Lines 492-493: I guess “*data efficient*” and “*memorization*” are swapped.
Questions
Q1. As far as I know, the Hard Reset method typically resamples the model parameter every time it resets the parameter. With this in mind, what if we slightly modify the drift model as: $p(\theta \mid \theta_t, \gamma_t) = \mathcal{N} (\theta; \gamma_t \theta_t + (1-\gamma_t) \theta’_0 ; (1-\gamma_t^2)\sigma_0^2)$, where we re-sample $\theta’_0 \sim p_0(\theta_0)$ every time $t$?
Rating
4
Confidence
3
Soundness
2
Presentation
1
Contribution
2
Limitations
The paper discusses its limitations in the experiment section. I think it might be better to mention them explicitly in the conclusion section.
The response is not satisfactory yet
I have read the author’s rebuttal, but I still have remaining concerns and questions. Let me leave my comments/questions about the author’s responses one at a time. 1. I don’t think this answer really responds to my question in W1. Here, I am not asking about the motivation of the drift model (although I did it in W2). My question is: why do we care about the conditional distribution of $\theta_{t+1}$, a consequence of a single step of training from $\theta_t$, even though we want to estimate or utilize the dynamics of the moving local optima $\theta^{\ast}\_t \mapsto \theta^{\ast}\_{t+1}$? - After pondering this issue and staring at Figure 1, I suddenly realized that the drift model is actually modeling the dynamics of $\theta^{\ast}\_{t+1}$ given $\theta_t$. So in my understanding, the $\theta_{t+1}$ in “$p(\theta_{t+1} \mid \theta_t, \gamma_t)$” actually means $\theta^{\ast}_{t+1}$, the local optima at time $t+1$, rather than the parameter we will have by updating the parameter from $\theta_t$ using a single step of learning algorithm. (Or, at least, we want to find a local optimum $\theta^{\ast}\_{t+1}$ close to the initialization & $\theta_t$.) Is my understanding correct? If it is, I think this should have been explained in the paper in more detail; also, the term “parameter drift” is a bit misleading in this sense. - By the way, I guess the citation ([2-3]) has a typo, right? It seems that it should include [1]. 2. Thank you for the explanation of the relationship between the OU process, the Gaussian Markov process, and the choice of the drift model. Will the authors add this explanation to their paper? Or do they just ignore it? 3. Thank you for the discussion on the other options for drift models and for considering putting the discussion to the paper. 4. Thank you for derivations of the equations in more detail. However, I think the derivations could be more kind than the current explanation. - Eq 9: Please state to which parameter the linearization is taken (e.g., By linearizing log… around $\theta=\mu_t$ (cf. Line 198)) - Eq. 11 & 12: Is these are just the usual objective functions in BNN training? 5. Thank you for the interesting response based on the drift model and the variance. But what if we change the **variance scheduling**: what if we suitably change the drift model into $p(\theta \mid \theta_t, \gamma_t) = \mathcal{N} (\theta; \gamma_t \theta_t + (1-\gamma_t) \theta’_0(t); 2\gamma_t(1-\gamma_t)\sigma_0^2)$ where $\theta’_0(t) \sim p_0(\theta_0)$? I guess this model does not suffer from the same problem of variance inflation: If $\sigma_t = \sigma_0$, then ${\rm Var}(\theta) = \gamma_t^2 \sigma_t^2 + (1-\gamma_t)^2 \sigma_0^2 + 2\gamma_t (1-\gamma_t) \sigma_0^2 = \\{\gamma_t^2 + (1-\gamma_t)^2 + 2\gamma_t (1-\gamma_t)\\}\sigma_0^2 = \sigma_0^2$...! The motivation behind this is the concern about the dependency on the particular initialization “$\theta_0$” which seems to be fixed throughout the training: “What if the initialization distribution is only the matter?” Please let me know if there are other problems in this drift model, or I would be very happy if you could test this drift model empirically (but I understand if it is impossible due to the time limit)! Also, there is a question that is not answered yet. Let me bring it here: - Line 232: What does it mean by “we assume that $\mu_0 = \theta_0, \sigma_0^2.$”? Although I appreciate the time and effort invested in the rebuttal and am happy with the author's general response, overall, I feel like the author’s response is not really satisfying yet. Even though the authors requested to reconsider my assessment, I cannot do so before I get a satisfying further response. If the further response is not satisfactory as well, sadly and unfortunately, I am **ready** to decrease my score to 3 (but **not yet**) because, in my view, there is still a huge room for improvement in writing and presentation.
Response to a response: Part 1
Dear 4koq, thank you for your feedback and please find our answer to your concerns below. > I don’t think this answer really responds to my question in W1. Here, I am not asking about the motivation of the drift model (although I did it in W2). My question is: why do we care about the conditional distribution of Θt+1 , a consequence of a single step of training from θt , even though we want to estimate or utilize the dynamics of the moving local optima θt∗↦θt+1∗? After pondering this issue and staring at Figure 1, I suddenly realized that the drift model is actually modeling the dynamics of θt+1∗ given θt.. So in my understanding, the θt+1 in “p(θt+1∣θt,γt)” actually means θt+1∗ , the local optima at time t+1, rather than the parameter we will have by updating the parameter from θt using a single step of learning algorithm. (Or, at least, we want to find a local optimum θt+1∗ close to the initialization & θt .) Is my understanding correct? If it is, I think this should have been explained in the paper in more detail; also, the term “parameter drift” is a bit misleading in this sense. We apologize for the confusion, but under the Bayesian perspective depicted in Figure 1 $\theta^*_{t}$ and $\theta^*_{t+1}$ denote **fixed/deterministic** values and therefore are not random variables, while the corresponding random variables which are assigned probability distributions are $\theta_{t}$ and $\theta_{t+1}$. More specifically, we learn the dynamical model distribution $p(\theta_{t+1}|\theta_{t},\gamma_t)$, by learning $\gamma_t$, so that the specific value $\theta^*_{t+1}$ can become more likely or “explainable” under the distribution $p(\theta_{t+1}|\theta_{t},\gamma_t)$. In other words we hope that $p(\theta_{t+1}|\theta_{t},\gamma_t)$ will place a considerable probability mass around the fixed value $\theta^*_{t+1}$. We are going to update the paper to make the above clear and remove the confusion. To further explain here this in Figure 1, consider the stationary case depicted in Figure,1a. There the posterior concentrates around an optimal value (which is fixed value $\theta^*$), in other words $q_t(\theta)$ will gradually converge to a delta mass around the value $\theta^*$. In a non-stationary case, the optimal value can suddenly change over time and e.g. from the value $\theta^*_{t}$ at time $t$ can change to $\theta^*_{t+1}$ . Without a dynamical model (Figure 1,b) the new posterior $q_{t+1}(\theta)$ after observing the new data at time $t+1$ has a small radius/variance (blue dashed circle) and it cannot concentrate fast enough towards the new optimum. The use of the dynamical or drift model (Figure 1,c) introduces an “intermediate step” that constructs an “intermediate prior distribution” $p_t(\theta) = \int q_t(\theta) p(\theta | \theta_t, \gamma_t) d \theta_t$ which can have increased variance and shifted mean (green dashed circle). Then once we fully incorporate the new data point at time $t+1$ the updated posterior $q_{t+1}(\theta) \propto p(y_{t+1}|\theta) p_t(\theta)$ can better concentrate around $\theta_{t+1}^*$. We will update the paper to clarify the above. > citation [2-3] is a typo Thank you, it should be [1-2]. > Q2 Yes we will add this to the paper (we forgot to explicitly write it). > Q3 Thank you. > Eq.11&12 -- usual BNN objective? Yes, except for the temperature defined per-parameter. In BNN, temperature $\lambda$ is either 1, or fixed to a constant for all the parameters. > Line 232: What does it mean by “we assume that $\mu_0=\theta_0,\sigma^2_0.$? It means that the mean of the prior $p_0(\theta)=\mathcal{N}(\theta;\mu_0,\sigma^2)$ is fixed to $\theta_0$, i.e. the initialization and $\sigma^2_0$ is a constant. Normally NNs are initialized from $p_0(\theta)=\mathcal{N}(\theta;0,\sigma^2_0)$, but we make the prior to be initialization-dependent. We will rewrite this sentence to explicitly highlight this. We hope your answer clarifies the questions and concerns you have raised, and we thank you for raising interesting points for discussion!
Part 2
> Eq.9, linearisation As we stated above in the rebuttal, the response to Reviewer 6GY1 (due to space constraints) contains the partial derivation. Linearisation is $\log p(y_{t+1}|x_{t+1},\theta)\sim\log p(y_{t+1}|x_{t+1},\mu_t)-g_{t+1}^T(\theta-\mu_t)$, where $g_{t+1}=\nabla_{\theta} -\log p(y_{t+1}|x_{t+1},\theta=\mu_t)$. Then, $p(y_{t+1}|x_{t+1},\theta) \sim p(y_{t+1}| x_{t+1},\mu_t)\exp^{-g_{t+1}^T\mu_t}\exp^{g_{t+1}^T\theta}$. Since, $q_t^{t+1}(\theta | S_{t}, \Gamma_{t-1}, \gamma_t)$ is Gaussian, we compute the integral eq.6 in a closed form and keep only the terms depending on $\gamma_t$. We provide derivation for a scalar case here and full derivation in the appendix. In eq.6, we have $\log\int p(y_{t+1}|x_{t+1},\theta)\exp^{-\frac{(\theta -\mu_t(\gamma_t))^2}{2\sigma^2(\gamma_t)}}d\theta =\log \frac{1}{\sqrt{2\pi\sigma^2(\gamma_t)}}\int p(y_{t+1}|x_{t+1},\mu_t)\exp^{-g_{t+1}(\theta-\mu_t)} \exp^{-\frac{(\theta-\mu_t(\gamma_t))^2}{2\sigma^2(\gamma_t)}}d\theta$ which becomes $\log p(y_{t+1}|x_{t+1},\mu_t)+\log \int \frac{1}{\sqrt{2\pi\sigma^2(\gamma_t)}}\exp^{-g_{t+1}(\theta-\mu_t)}\exp^{-\frac{(\theta-\mu_t(\gamma_t))^2}{2\sigma^2(\gamma_t)}}d\theta$ Consider only the exponent term in the integral $\frac{-1}{2\sigma^2(\gamma_t)}(2\sigma^2(\gamma_t)g_{t+1}(\theta-\mu_t)+\theta^2-2\theta \mu_t(\gamma_t)+\mu_t(\gamma_t)^2)$ then $\frac{-1}{2\sigma^2(\gamma_t)}\left(\theta^2-2\theta \left[\mu_t(\gamma_t)-\sigma^2(\gamma_t)g_{t+1}\right]+\mu_t(\gamma_t)^2-2\sigma^2(\gamma_t)g_{t+1}\mu_t\right)$ then $\frac{-1}{2\sigma^2(\gamma_t)}\left[ \left(\theta-(\mu_t(\gamma_t)-\sigma^2(\gamma_t)g_{t+1}) \right)^2+2\mu_t(\gamma_t) \sigma^2(\gamma_t)g_{t+1}-\sigma^4(\gamma_t)g_{t+1}^2-2\sigma^2(\gamma_t)g_{t+1}\mu_t)\right]$ then $\frac{-1}{2\sigma^2(\gamma_t)} \left(\theta-(\mu_t(\gamma_t)-\sigma^2(\gamma_t)g_{t+1})\right)^2-\mu_t(\gamma_t)g_{t+1}+0.5\sigma^2(\gamma_t)g^2_{t+1}+g_{t+1}\mu_t$ Now, get back to the integral $\log\int\frac{1}{\sqrt(2\pi\sigma^2(\gamma_t)}\exp^{\frac{-1}{2\sigma^2(\gamma_t)}\left(\theta-(\mu_t(\gamma_t)-\sigma^2(\gamma_t)g_{t+1}) \right)^2-\mu_t(\gamma_t)g_{t+1}+0.5\sigma^2(\gamma_t)g^2_{t+1}+g_{t+1}\mu_t}d\theta$ Which equals to $1-\mu_t(\gamma_t)g_{t+1}+0.5 \sigma^2(\gamma_t)g^2_{t+1}+g_{t+1}\mu_t$ We keep only the terms which depend on $\gamma_t$ and get $G(\gamma_t)=-\mu_t(\gamma_t)g_{t+1}+0.5 \sigma^2(\gamma_t)g^2_{t+1} $ We want to maximize $G(\gamma_t)$ or minimize $F(\gamma_t)=-G(\gamma_t)$ which is exactly the loss we defined in eq.9. We will add the full derivation in the appendix. > Different drift model Assuming that $\theta'\sim\mathcal{N}(\theta;\mu_0;\sigma^2_0)$, the model you wrote will have the mean $\gamma_t\theta_t+(1-\gamma_t)\mu_0$ and the variance $\gamma^2_t \sigma^2_t+(1-\gamma^2_t)\sigma^2_0$, which are the same as for our OU model, see eq.4 and Line 173 after eq.5, therefore they are mathematically equivalent. In order to discuss it in detail, let $\mu_0$ is the prior mean for eq.4 and $\mu'_0$ is the prior mean in your variant, i.e., $\theta'\sim\mathcal{N}(\theta;\mu'_0;\sigma^2_0)$. When we incorporate the Gaussian model drift in the update, we are only concerned with its mean and variance since these affect SGD updates (see eq.16). We have the following three cases: * $\mu_0=\mu_0'=0$, the models have the same mean equal to $0$. They are mathematically equivalent to the model $\theta_{t+1}=\gamma_t\theta_t+\sqrt{1-\gamma^2_t}\theta_0$, where $\theta_0\sim p_0(\theta)=\mathcal{N}(\theta;0; \sigma^2)$ -- the models are the same. The have the mean $\gamma_t\mu_t$ and the variance $\gamma^2_t\sigma^2_t+(1-\gamma^2_t)\sigma^2_0$. * $\mu_0=\mu'_0=\theta_0\neq0$, the models have the same mean which are the one specific initialization of NN parameters. They are mathematically equivalent with the mean $\gamma_t\mu_t+(1-\gamma_t)\theta_0$ and the same variance (as above) * $\mu_0=\theta_0$ and $\mu'_0 \neq \theta_0$. These two models have the same variance, $\gamma_t^2 \sigma^2_t + (1-\gamma^2_t)\sigma^2_0$, but have different means: $\gamma_t\theta_t+(1-\gamma_t)\theta_0$ versus $\gamma_t\theta_t+(1-\gamma_t)\mu'_0$. To implement variant 3, the mean $\mu'_0$ is fixed at the beginning of the training, because otherwise we would be under-counting the variance in the corresponding drift model. It is unclear how to interpret this model nor how it will affect performance. Thus, in order to answer your question about the impact of the particular initialization, we believe variant 1 is the right answer. We conducted an experiment of using Soft Reset variant 1 and 2 on random label MNIST (data efficient), where we swept over hyperparameters. We found that these variants had similar performance on this benchmark. We will add this experiment to the paper in the Appendix. We hope your answer clarifies the questions and concerns you have raised, and we thank you for raising interesting points for discussion!
I thank the authors for a couple of more detailed replies. 1. About original drift model: Thank you for resolving most of my confusion. Now I think I understand the concept of drift model. Still, I am concerned about that, in the author's explanation, the local optimum $\theta^{\ast}\_t$ seems a unique deterministic value/vector. However, in the era of overparameterized deep learning and because of its nonconvex nature, the $\theta^{\ast}\_t$ may not be unique; rather, there might be a set of (non-unique) local optima. Because of this, I thought the drift model is a kind of mechanism of choosing a proper/desired $\theta^{\ast}\_t$ among all local optima. Although I admit that $\theta^{\ast}\_t$ not a random object (but still it can be because of randomness in data sampling) and $\theta_t$ is an estimate of it (thus it is definitely a random object), considering the non-uniqueness of local optima, it seems vague and non-rigorous for me to say "getting closer to $\theta^{\ast}\_t$" or "concentrating aroung $\theta^{\ast}\_t$". 2. About linearization: I am already aware of the derivation explained for reviewer 6GY1. All I wanted is just a kind derivation in your next revised paper. I wanted to take part in improving your paper. 3. About variations of drift model: thank you for clearly answering my question. Personally, I was excited for suggesting a new point of view of your methodology, but it turns out it was somewhat meaningless (which is sad). Anyway, I am happy to understand the significance of the original formulation of the drift model. Thank you again for the detailed response. Nonetheless, in my opinion, it would be better to judge a completely rewritten manuscript with another review process. Therefore, I maintain my score.
Response
We thank the Reviewer 4koq for their response and for sparking interesting discussion! Please see our answer below. > I am concerned about that, in the author's explanation, the local optimum $\theta^{\ast}_t$ seems a unique deterministic value/vector. However, in the era of overparameterized deep learning and because of its nonconvex nature, the $\theta^{\ast}_t$ may not be unique; rather, there might be a set of (non-unique) local optima. Please note that when we discuss local optimas $\theta^*_{t}$ and $\theta^*_{t+1}$, we do not assume their uniqueness. Rather, the only assumption we make is that our algorithm may converge to some local optima $\theta^*_{t}$ (which is a member of a set of local optima), and after non-stationarity, a new set of local optima will appear, where one of the member is $\theta^*_{t+1}$. The reason why we use two specific points $\theta^*_{t}$ and $\theta^*_{t+1}$ in Figure 1, is to simplify the understanding and communicate the intuition behind the non-stationary behaviour and the impact of a drift model. In the paper, in Section 3.1, we mainly refer to a region of new local minima, please see, for example, Lines 110-112: "If the change is such that the region of new good local minima is far away from $\theta^*$, SGD may take considerably higher amount of time to move its parameters towards this region." Hope this clarifies the concerns. > About linearization: I am already aware of the derivation explained for reviewer 6GY1. All I wanted is just a kind derivation in your next revised paper. I wanted to take part in improving your paper. Thanks to the useful discussion with you and the concerns you have raised, we think that including the full derivation of the linearization in the appendix of the paper, will greatly reduce confusion shared by you and other reviewers. We thank you for your suggestions, we will add more context in the paper.
Summary
The paper proposes a method to effectively learn the neural network parameters in non-stationary environments. They propose a modified learning algorithm that adapts to non-stationarity through an Ornstein-Uhlenbeck process with an adaptive drift parameter. Drift parameter is used to track the non-stationarity in the data - specifically, adaptive drift pulls the NN parameters towards initial parameters when the data is not well-modelled by current parameters while on the other hand reducing the degree of regularization (towards initial parameters) when the data is relatively stationary. The paper introduces a Bayesian Soft-Reset algorithm in Bayesian Neural Network (BNN) framework and a General Soft-Reset algorithm in deterministic non-Bayesian framework to effectively learn in non-stationary environments. Empirically, authors demonstrate that their method performs effectively in non-stationary supervised and off-policy reinforcement learning settings.
Strengths
To take into account the non-stationarity in the data - the methodology in the paper modifies the SGD algorithm by modifying the regularization strength and the regularization target of the SGD algorithm - a significant contribution of the work is to do this modification in a principled manner using a drift parameter that is updated online from the stream of data. Overall, the paper is well written. The need and novelty of the methodology are clearly presented, and the illustration of the use of methodology on realistic instances clearly demonstrates how it works and improves on the existing methodologies for learning in non-stationary data regime.
Weaknesses
Some parts can be improved - 1) Section 2 - Simplification of equation (6) to (9) requires some explanation [page -5]. 2) Section 3 - Some intuition behind the objective (11) [page - 6] would be good to understand it better (especially the second term in that objective) 3) Some discussion on how does the methodology perform relative to the degree of non-stationarity in the data? 4) Some minor typos in line 109, 154, 190
Questions
How does the methodology perform relative to the degree of non-stationarity in the data? Specifically, to what extent does this methodology remain effective under varying levels of non-stationarity?
Rating
7
Confidence
3
Soundness
4
Presentation
4
Contribution
3
Limitations
Authors have adequately addressed the limitations of their work.
Summary
This work focuses on the problem of plasticity loss in non-stationary problems. This work proposes a new solution which adaptively drifts the parameters toward the initial distribution. The proposed solution is a form of soft resetting and can be seen as a meta version of L2-init where the degree of drift towards initialization is learned from data.
Strengths
The proposed solution is novel. Current soft resetting methods reset the parameters of the model at a constant rate, irrespective of the degree of non-stationarity. The proposed solution introduces adaptive soft resetting, where the degree of resetting is calculated based on the degree of non-stationarity. This adaptiveness allows the solution to adapt faster and retain more prior knowledge. The experimental results show the effectiveness of the proposed solution.
Weaknesses
Although the proposed solution is novel and effective, the paper has many weaknesses: 1. Hyperparameter sensitivity. Given that the proposed solutions introduce many new hyperparameters and the authors say in line 207 that one of the solutions is sensitive to the choice of hyperparameter, the paper needs to contain hyperparameter sensitivity curves for all new hyperparameters. 2. Computational cost. The proposed solutions seem computationally expensive. It would be good to include the wall of each algorithm beside one of the figures, and Figure 3 might be best. 3. Limited performance improvement. The primary purpose of high replay-ratio is to improve the sample efficiency of algorithms. However, results in Figure 4 show that the proposed solution does not utilize a high replay-ratio. Its sample efficiency at replay ratio 1 is the same as its sample efficiency at replay ratio 128.
Questions
See Weaknesses
Rating
4
Confidence
4
Soundness
3
Presentation
2
Contribution
3
Limitations
Computational cost and hyperparameter sensitivity need to be discussed in the conclusion of the main paper.
Response to authors' rebuttal
I thank the authors for the response. I had some points clarified. However, I am inclined to maintain my score, since the experimental validation is somehow confusing. Regarding the RL results, I can not see clear benefits from the approach. Regarding Figure 3, the authors replied that their method is Soft Reset, but I see it being outperformed by several of its variants. Then why do the authors choose as their proposed method Soft Reset and not one of the variants that outperform it? Thank you.
Response to a response
Dear Reviewer DfC5, thank you for your response. > Regarding Figure 3, the authors replied that their method is Soft Reset, but I see it being outperformed by several of its variants. Then why do the authors choose as their proposed method Soft Reset and not one of the variants that outperform it? It is true that it is outperformed by other variants. Soft Reset is the cheapest method among all the other variants -- it requires only one update on $\gamma_t$ and one update on parameters $\theta_{t+1}$. All the other methods -- Soft Reset more compute, Soft Reset proximal, Bayesian Soft Reset proximal, require significantly more compute. Figure 3 suggests that we can in fact leverage more compute to achieve better empirical performance. This is why we chose to present Soft Reset as the main method because it is relatively cheap and already achieves good performance compared to external baselines. However, when more compute is available, other methods could also be used to further improve performance. Hope this clarifies the confusion.
Decision
Accept (poster)