In this paper, we compare and evaluate different testing protocols used for\nautomatic COVID-19 diagnosis from X-Ray images in the recent literature. We\nshow that similar results can be obtained using X-Ray images that do not\ncontain most of the lungs. We are able to remove the lungs from the images by\nturning to black the center of the X-Ray scan and training our classifiers only\non the outer part of the images. Hence, we deduce that several testing\nprotocols for the recognition are not fair and that the neural networks are\nlearning patterns in the dataset that are not correlated to the presence of\nCOVID-19. Finally, we show that creating a fair testing protocol is a\nchallenging task, and we provide a method to measure how fair a specific\ntesting protocol is. In the future research we suggest to check the fairness of\na testing protocol using our tools and we encourage researchers to look for\nbetter techniques than the ones that we propose.\n