Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition

Natural language inference (NLI) is an increasingly important task for\nnatural language understanding, which requires one to infer whether a sentence\nentails another. However, the ability of NLI models to make pragmatic\ninferences remains understudied. We create an IMPlicature and PRESupposition\ndiagnostic dataset (IMPPRES), consisting of >25k semiautomatically generated\nsentence pairs illustrating well-studied pragmatic inference types. We use\nIMPPRES to evaluate whether BERT, InferSent, and BOW NLI models trained on\nMultiNLI (Williams et al., 2018) learn to make pragmatic inferences. Although\nMultiNLI appears to contain very few pairs illustrating these inference types,\nwe find that BERT learns to draw pragmatic inferences. It reliably treats\nscalar implicatures triggered by "some" as entailments. For some presupposition\ntriggers like "only", BERT reliably recognizes the presupposition as an\nentailment, even when the trigger is embedded under an entailment canceling\noperator like negation. BOW and InferSent show weaker evidence of pragmatic\nreasoning. We conclude that NLI training encourages models to learn some, but\nnot all, pragmatic inferences.\n