Complex sequential tasks in continuous-control settings often require agents\nto successfully traverse a set of "narrow passages" in their state space.\nSolving such tasks with a sparse reward in a sample-efficient manner poses a\nchallenge to modern reinforcement learning (RL) due to the associated\nlong-horizon nature of the problem and the lack of sufficient positive signal\nduring learning. Various tools have been applied to address this challenge.\nWhen available, large sets of demonstrations can guide agent exploration.\nHindsight relabelling on the other hand does not require additional sources of\ninformation. However, existing strategies explore based on task-agnostic goal\ndistributions, which can render the solution of long-horizon tasks impractical.\nIn this work, we extend hindsight relabelling mechanisms to guide exploration\nalong task-specific distributions implied by a small set of successful\ndemonstrations. We evaluate the approach on four complex, single and dual arm,\nrobotics manipulation tasks against strong suitable baselines. The method\nrequires far fewer demonstrations to solve all tasks and achieves a\nsignificantly higher overall performance as task complexity increases. Finally,\nwe investigate the robustness of the proposed solution with respect to the\nquality of input representations and the number of demonstrations.\n