The language acquisition literature shows that children do not build their\nlexicon by segmenting the spoken input into phonemes and then building up words\nfrom them, but rather adopt a top-down approach and start by segmenting\nword-like units and then break them down into smaller units. This suggests that\nthe ideal way of learning a language is by starting from full semantic units.\nIn this paper, we investigate if this is also the case for a neural model of\nVisually Grounded Speech trained on a speech-image retrieval task. We evaluated\nhow well such a network is able to learn a reliable speech-to-image mapping\nwhen provided with phone, syllable, or word boundary information. We present a\nsimple way to introduce such information into an RNN-based model and\ninvestigate which type of boundary is the most efficient. We also explore at\nwhich level of the network's architecture such information should be introduced\nso as to maximise its performances. Finally, we show that using multiple\nboundary types at once in a hierarchical structure, by which low-level segments\nare used to recompose high-level segments, is beneficial and yields better\nresults than using low-level or high-level segments in isolation.\n