The notion of equality (identity) is simple and ubiquitous, making it a key\ncase study for broader questions about the representations supporting abstract\nrelational reasoning. Previous work suggested that neural networks were not\nsuitable models of human relational reasoning because they could not represent\nmathematically identity, the most basic form of equality. We revisit this\nquestion. In our experiments, we assess out-of-sample generalization of\nequality using both arbitrary representations and representations that have\nbeen pretrained on separate tasks to imbue them with structure. We find neural\nnetworks are able to learn (1) basic equality (mathematical identity), (2)\nsequential equality problems (learning ABA-patterned sequences) with only\npositive training instances, and (3) a complex, hierarchical equality problem\nwith only basic equality training instances ("zero-shot'" generalization). In\nthe two latter cases, our models perform tasks proposed in previous work to\ndemarcate human-unique symbolic abilities. These results suggest that essential\naspects of symbolic reasoning can emerge from data-driven, non-symbolic\nlearning processes.\n