Neural language models exhibit impressive performance on a variety of tasks,\nbut their internal reasoning may be difficult to understand. Prior art aims to\nuncover meaningful properties within model representations via probes, but it\nis unclear how faithfully such probes portray information that the models\nactually use. To overcome such limitations, we propose a technique, inspired by\ncausal analysis, for generating counterfactual embeddings within models. In\nexperiments testing our technique, we produce evidence that suggests some\nBERT-based models use a tree-distance-like representation of syntax in\ndownstream prediction tasks.\n