Symbolic planning models allow decision-making agents to sequence actions in\narbitrary ways to achieve a variety of goals in dynamic domains. However, they\nare typically handcrafted and tend to require precise formulations that are not\nrobust to human error. Reinforcement learning (RL) approaches do not require\nsuch models, and instead learn domain dynamics by exploring the environment and\ncollecting rewards. However, RL approaches tend to require millions of episodes\nof experience and often learn policies that are not easily transferable to\nother tasks. In this paper, we address one aspect of the open problem of\nintegrating these approaches: how can decision-making agents resolve\ndiscrepancies in their symbolic planning models while attempting to accomplish\ngoals? We propose an integrated framework named SPOTTER that uses RL to augment\nand support ("spot") a planning agent by discovering new operators needed by\nthe agent to accomplish goals that are initially unreachable for the agent.\nSPOTTER outperforms pure-RL approaches while also discovering transferable\nsymbolic knowledge and does not require supervision, successful plan traces or\nany a priori knowledge about the missing planning operator.\n