Recent advances in neural architectures have revived the problem of\nmorphological rule learning. We evaluate the Transformer as a model of\nmorphological rule learning and compare it with Recurrent Neural Networks (RNN)\non English, German, and Russian. We bring to the fore a hitherto overlooked\nproblem, the morphological gaps, where the expected inflection of a word is\nmissing. For example, 63 Russian verbs lack a first-person-singular present\nform such that one cannot comfortably say "*o\\v{s}\\v{c}u\\v{s}\\v{c}u" ("I\nfeel"). Even English has gaps, such as the past participle of "stride": the\nfunction of morphological inflection can be partial. Both neural architectures\nproduce inflections that ought to be missing. Analyses reveal that Transformers\nrecapitulate the statistical distribution of inflections in the training data,\nsimilar to RNNs. Models' success on English and German is driven by the fact\nthat rules in these languages can be identified with the majority forms, which\nis not universal.\n