Deep pre-trained contextualized encoders like BERT (Delvin et al., 2019)\ndemonstrate remarkable performance on a range of downstream tasks. A recent\nline of research in probing investigates the linguistic knowledge implicitly\nlearned by these models during pre-training. While most work in probing\noperates on the task level, linguistic tasks are rarely uniform and can be\nrepresented in a variety of formalisms. Any linguistics-based probing study\nthereby inevitably commits to the formalism used to annotate the underlying\ndata. Can the choice of formalism affect probing results? To investigate, we\nconduct an in-depth cross-formalism layer probing study in role semantics. We\nfind linguistically meaningful differences in the encoding of semantic role-\nand proto-role information by BERT depending on the formalism and demonstrate\nthat layer probing can detect subtle differences between the implementations of\nthe same linguistic formalism. Our results suggest that linguistic formalism is\nan important dimension in probing studies, along with the commonly used\ncross-task and cross-lingual experimental settings.\n