We propose a method for effective training of deep Reinforcement Learning (RL) agents when the reward is sparse and non-Markovian, but at the same time progress towards the reward requires achieving an unknown sequence of high-level objectives. Our method employs a novel algorithm for synthesis of compact automata to uncover this sequential structure automatically. We synthesise a human-interpretable automaton from trace data generated through exploration of the environment by the deep RL agent. The state space of the environment is then enriched with the synthesised automaton so that generation of an optimal control policy by deep RL is guided by the discovered structure encoded in the automaton. We evaluate performance via a set of experiments including the Atari game Montezuma's Revenge. Compared to existing approaches, we obtain a decrease of two orders of magnitude in the number of iterations required for policy synthesis.