The recent growth in the popularity and success of deep learning models on\nNLP classification tasks has accompanied the need for generating some form of\nnatural language explanation of the predicted labels. Such generated natural\nlanguage (NL) explanations are expected to be faithful, i.e., they should\ncorrelate well with the model's internal decision making. In this work, we\nfocus on the task of natural language inference (NLI) and address the following\nquestion: can we build NLI systems which produce labels with high accuracy,\nwhile also generating faithful explanations of its decisions? We propose\nNatural-language Inference over Label-specific Explanations (NILE), a novel NLI\nmethod which utilizes auto-generated label-specific NL explanations to produce\nlabels along with its faithful explanation. We demonstrate NILE's effectiveness\nover previously reported methods through automated and human evaluation of the\nproduced labels and explanations. Our evaluation of NILE also supports the\nclaim that accurate systems capable of providing testable explanations of their\ndecisions can be designed. We discuss the faithfulness of NILE's explanations\nin terms of sensitivity of the decisions to the corresponding explanations. We\nargue that explicit evaluation of faithfulness, in addition to label and\nexplanation accuracy, is an important step in evaluating model's explanations.\nFurther, we demonstrate that task-specific probes are necessary to establish\nsuch sensitivity.\n