We introduce a FEVER-like dataset COVID-Fact of $4,086$ claims concerning the\nCOVID-19 pandemic. The dataset contains claims, evidence for the claims, and\ncontradictory claims refuted by the evidence. Unlike previous approaches, we\nautomatically detect true claims and their source articles and then generate\ncounter-claims using automatic methods rather than employing human annotators.\nAlong with our constructed resource, we formally present the task of\nidentifying relevant evidence for the claims and verifying whether the evidence\nrefutes or supports a given claim. In addition to scientific claims, our data\ncontains simplified general claims from media sources, making it better suited\nfor detecting general misinformation regarding COVID-19. Our experiments\nindicate that COVID-Fact will provide a challenging testbed for the development\nof new systems and our approach will reduce the costs of building\ndomain-specific datasets for detecting misinformation.\n