The Automatic Speaker Verification Spoofing and Countermeasures Challenges\nmotivate research in protecting speech biometric systems against a variety of\ndifferent access attacks. The 2017 edition focused on replay spoofing attacks,\nand involved participants building and training systems on a provided dataset\n(ASVspoof 2017). More than 60 research papers have so far been published with\nthis dataset, but none have sought to answer why countermeasures appear\nsuccessful in detecting spoofing attacks. This article shows how artefacts\ninherent to the dataset may be contributing to the apparent success of\npublished systems. We first inspect the ASVspoof 2017 dataset and summarize\nvarious artefacts present in the dataset. Second, we demonstrate how\ncountermeasure models can exploit these artefacts to appear successful in this\ndataset. Third, for reliable and robust performance estimates on this dataset\nwe propose discarding nonspeech segments and silence before and after the\nspeech utterance during training and inference. We create speech start and\nendpoint annotations in the dataset and demonstrate how using them helps\ncountermeasure models become less vulnerable from being manipulated using\nartefacts found in the dataset. Finally, we provide several new benchmark\nresults for both frame-level and utterance-level models that can serve as new\nbaselines on this dataset.\n