We study reinforcement learning in settings where sampling an action from the\npolicy must be done concurrently with the time evolution of the controlled\nsystem, such as when a robot must decide on the next action while still\nperforming the previous action. Much like a person or an animal, the robot must\nthink and move at the same time, deciding on its next action before the\nprevious one has completed. In order to develop an algorithmic framework for\nsuch concurrent control problems, we start with a continuous-time formulation\nof the Bellman equations, and then discretize them in a way that is aware of\nsystem delays. We instantiate this new class of approximate dynamic programming\nmethods via a simple architectural extension to existing value-based deep\nreinforcement learning algorithms. We evaluate our methods on simulated\nbenchmark tasks and a large-scale robotic grasping task where the robot must\n"think while moving".\n