The goal of continual learning (CL) is to learn a sequence of tasks without\nsuffering from the phenomenon of catastrophic forgetting. Previous work has\nshown that leveraging memory in the form of a replay buffer can reduce\nperformance degradation on prior tasks. We hypothesize that forgetting can be\nfurther reduced when the model is encouraged to remember the \\textit{evidence}\nfor previously made decisions. As a first step towards exploring this\nhypothesis, we propose a simple novel training paradigm, called Remembering for\nthe Right Reasons (RRR), that additionally stores visual model explanations for\neach example in the buffer and ensures the model has "the right reasons" for\nits predictions by encouraging its explanations to remain consistent with those\nused to make decisions at training time. Without this constraint, there is a\ndrift in explanations and increase in forgetting as conventional continual\nlearning algorithms learn new tasks. We demonstrate how RRR can be easily added\nto any memory or regularization-based approach and results in reduced\nforgetting, and more importantly, improved model explanations. We have\nevaluated our approach in the standard and few-shot settings and observed a\nconsistent improvement across various CL approaches using different\narchitectures and techniques to generate model explanations and demonstrated\nour approach showing a promising connection between explainability and\ncontinual learning. Our code is available at\n\\url{https://github.com/SaynaEbrahimi/Remembering-for-the-Right-Reasons}.\n