Summary
This paper proposes two methods - MH-DT and LoRA-DT - to mitigate catastrophic forgetting in continuous offline reinforcement learning using Decision Transformers. MH-DT uses multiple heads and selective replay while LoRA-DT merges weights and adapts with low-rank matrices. Experiments on MuJoCo and Meta-World benchmarks demonstrate performance gains over prior methods.
Weaknesses
1. The experimental evaluation is very limited, with results reported on just a few MuJoCo and Meta-World environments. Important details like the number of random seeds used, variance across seeds, and standard errors are missing. With such sparse experiment details and a lack of reported variance, it is difficult to assess the statistical significance of the results and conclusions. More environments, hyperparameter configurations, and random seeds should be evaluated to substantiate the claims.
2. The comparison to other continual learning techniques is lacking. The paper only considers an offline RL baseline but does not compare against many strong regularization-based continual learning methods like [1,2,3]. Additionally, more offline RL algorithms could be included as baselines beyond just DT variants. The restrictive set of baselines makes it hard to situate the performance of the proposed techniques.
3. Many important implementation details are unclear, like how the datasets are sequentially provided, how rewards are evaluated, the training steps, etc. The hyperparameter values seem arbitrary without proper justification or ablation studies. For instance, the number of training steps likely affects final performance significantly but the impact is not analyzed.
Therefore, the experimental analysis is currently too limited and sparse to demonstrate convincing improvements over existing approaches. A much more thorough evaluation is needed across more environments, seeds, baselines, and design choices to substantiate the conclusions.
[1] Kirkpatrick, James, et al. "Overcoming catastrophic forgetting in neural networks." Proceedings of the national academy of sciences 114.13 (2017): 3521-3526.
[2] Lyle, Clare, Mark Rowland, and Will Dabney. "Understanding and preventing capacity loss in reinforcement learning." arXiv preprint arXiv:2204.09560 (2022).
[3] Xu, Mengdi, et al. "Hyper-decision transformer for efficient online policy adaptation." arXiv preprint arXiv:2304.08487 (2023).
Questions
Some questions remain about the techniques:
1. It is unclear why LoRA and multi-heads differ in their backward transfer capability if they both avoid updating the base model weights. More analysis is needed.
2. The methods of LoRA and multi-heads seem to rely on the base model is well pretrained, such that the pretrained features are rich enough for downstream adaptation, but the paper does not describe any pretraining. This makes the effectiveness unclear. More generally, an ablation study on the base model initialization could provide insights into how reliant the techniques are on pretraining vs. continual adaptation.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.