Fine-tuning of pre-trained transformer models has become the standard\napproach for solving common NLP tasks. Most of the existing approaches rely on\na randomly initialized classifier on top of such networks. We argue that this\nfine-tuning procedure is sub-optimal as the pre-trained model has no prior on\nthe specific classifier labels, while it might have already learned an\nintrinsic textual representation of the task. In this paper, we introduce a new\nscoring method that casts a plausibility ranking task in a full-text format and\nleverages the masked language modeling head tuned during the pre-training\nphase. We study commonsense reasoning tasks where the model must rank a set of\nhypotheses given a premise, focusing on the COPA, Swag, HellaSwag and\nCommonsenseQA datasets. By exploiting our scoring method without fine-tuning,\nwe are able to produce strong baselines (e.g. 80% test accuracy on COPA) that\nare comparable to supervised approaches. Moreover, when fine-tuning directly on\nthe proposed scoring function, we show that our method provides a much more\nstable training phase across random restarts (e.g $\\times 10$ standard\ndeviation reduction on COPA test accuracy) and requires less annotated data\nthan the standard classifier approach to reach equivalent performances.\n