Reinforcement learning agents often forget details of the past, especially\nafter delays or distractor tasks. Agents with common memory architectures\nstruggle to recall and integrate across multiple timesteps of a past event, or\neven to recall the details of a single timestep that is followed by distractor\ntasks. To address these limitations, we propose a Hierarchical Chunk Attention\nMemory (HCAM), which helps agents to remember the past in detail. HCAM stores\nmemories by dividing the past into chunks, and recalls by first performing\nhigh-level attention over coarse summaries of the chunks, and then performing\ndetailed attention within only the most relevant chunks. An agent with HCAM can\ntherefore "mentally time-travel" -- remember past events in detail without\nattending to all intervening events. We show that agents with HCAM\nsubstantially outperform agents with other memory architectures at tasks\nrequiring long-term recall, retention, or reasoning over memory. These include\nrecalling where an object is hidden in a 3D environment, rapidly learning to\nnavigate efficiently in a new neighborhood, and rapidly learning and retaining\nnew object names. Agents with HCAM can extrapolate to task sequences much\nlonger than they were trained on, and can even generalize zero-shot from a\nmeta-learning setting to maintaining knowledge across episodes. HCAM improves\nagent sample efficiency, generalization, and generality (by solving tasks that\npreviously required specialized architectures). Our work is a step towards\nagents that can learn, interact, and adapt in complex and temporally-extended\nenvironments.\n