MedNLI Is Not Immune: Natural Language Inference Artifacts in the Clinical Domain

Crowdworker-constructed natural language inference (NLI) datasets have been\nfound to contain statistical artifacts associated with the annotation process\nthat allow hypothesis-only classifiers to achieve better-than-random\nperformance (Poliak et al., 2018; Gururanganet et al., 2018; Tsuchiya, 2018).\nWe investigate whether MedNLI, a physician-annotated dataset with premises\nextracted from clinical notes, contains such artifacts (Romanov and Shivade,\n2018). We find that entailed hypotheses contain generic versions of specific\nconcepts in the premise, as well as modifiers related to responsiveness,\nduration, and probability. Neutral hypotheses feature conditions and behaviors\nthat co-occur with, or cause, the condition(s) in the premise. Contradiction\nhypotheses feature explicit negation of the premise and implicit negation via\nassertion of good health. Adversarial filtering demonstrates that performance\ndegrades when evaluated on the difficult subset. We provide partition\ninformation and recommendations for alternative dataset construction strategies\nfor knowledge-intensive domains.\n