In reinforcement learning, temporal difference-based algorithms can be\nsample-inefficient: for instance, with sparse rewards, no learning occurs until\na reward is observed. This can be remedied by learning richer objects, such as\na model of the environment, or successor states. Successor states model the\nexpected future state occupancy from any given state for a given policy and are\nrelated to goal-dependent value functions, which learn how to reach arbitrary\nstates. We formally derive the temporal difference algorithm for successor\nstate and goal-dependent value function learning, either for discrete or for\ncontinuous environments with function approximation. Especially, we provide\nfinite-variance estimators even in continuous environments, where the reward\nfor exactly reaching a goal state becomes infinitely sparse. Successor states\nsatisfy more than just the Bellman equation: a backward Bellman operator and a\nBellman-Newton (BN) operator encode path compositionality in the environment.\nThe BN operator is akin to second-order gradient descent methods and provides\nthe true update of the value function when acquiring more observations, with\nexplicit tabular bounds. In the tabular case and with infinitesimal learning\nrates, mixing the usual and backward Bellman operators provably improves\neigenvalues for asymptotic convergence, and the asymptotic convergence of the\nBN operator is provably better than TD, with a rate independent from the\nenvironment. However, the BN method is more complex and less robust to sampling\nnoise. Finally, a forward-backward (FB) finite-rank parameterization of\nsuccessor states enjoys reduced variance and improved samplability, provides a\ndirect model of the value function, has fully understood fixed points\ncorresponding to long-range dependencies, approximates the BN method, and\nprovides two canonical representations of states as a byproduct.\n