Reinforcement learning can train policies that effectively perform complex\ntasks. However for long-horizon tasks, the performance of these methods\ndegrades with horizon, often necessitating reasoning over and chaining\nlower-level skills. Hierarchical reinforcement learning aims to enable this by\nproviding a bank of low-level skills as action abstractions. Hierarchies can\nfurther improve on this by abstracting the space states as well. We posit that\na suitable state abstraction should depend on the capabilities of the available\nlower-level policies. We propose Value Function Spaces: a simple approach that\nproduces such a representation by using the value functions corresponding to\neach lower-level skill. These value functions capture the affordances of the\nscene, thus forming a representation that compactly abstracts task relevant\ninformation and robustly ignores distractors. Empirical evaluations for\nmaze-solving and robotic manipulation tasks demonstrate that our approach\nimproves long-horizon performance and enables better zero-shot generalization\nthan alternative model-free and model-based methods.\n