In state of the art model-free off-policy deep reinforcement learning, a\nreplay memory is used to store past experience and derive all network updates.\nEven if both state and action spaces are continuous, the replay memory only\nholds a finite number of transitions. We represent these transitions in a data\ngraph and link its structure to soft divergence. By selecting a subgraph with a\nfavorable structure, we construct a simplified Markov Decision Process for\nwhich exact Q-values can be computed efficiently as more data comes in. The\nsubgraph and its associated Q-values can be represented as a QGraph. We show\nthat the Q-value for each transition in the simplified MDP is a lower bound of\nthe Q-value for the same transition in the original continuous Q-learning\nproblem. By using these lower bounds in temporal difference learning, our\nmethod QG-DDPG is less prone to soft divergence and exhibits increased sample\nefficiency while being more robust to hyperparameters. QGraphs also retain\ninformation from transitions that have already been overwritten in the replay\nmemory, which can decrease the algorithm's sensitivity to the replay memory\ncapacity.\n