Five years after the first published proofs of concept, direct approaches to\nspeech translation (ST) are now competing with traditional cascade solutions.\nIn light of this steady progress, can we claim that the performance gap between\nthe two is closed? Starting from this question, we present a systematic\ncomparison between state-of-the-art systems representative of the two\nparadigms. Focusing on three language directions\n(English-German/Italian/Spanish), we conduct automatic and manual evaluations,\nexploiting high-quality professional post-edits and annotations. Our\nmulti-faceted analysis on one of the few publicly available ST benchmarks\nattests for the first time that: i) the gap between the two paradigms is now\nclosed, and ii) the subtle differences observed in their behavior are not\nsufficient for humans neither to distinguish them nor to prefer one over the\nother.\n