Language models trained on large-scale unfiltered datasets curated from the\nopen web acquire systemic biases, prejudices, and harmful views from their\ntraining data. We present a methodology for programmatically identifying and\nremoving harmful text from web-scale datasets. A pretrained language model is\nused to calculate the log-likelihood of researcher-written trigger phrases\nconditioned on a specific document, which is used to identify and filter\ndocuments from the dataset. We demonstrate that models trained on this filtered\ndataset exhibit lower propensity to generate harmful text, with a marginal\ndecrease in performance on standard language modeling benchmarks compared to\nunfiltered baselines. We provide a partial explanation for this performance gap\nby surfacing examples of hate speech and other undesirable content from\nstandard language modeling benchmarks. Finally, we discuss the generalization\nof this method and how trigger phrases which reflect specific values can be\nused by researchers to build language models which are more closely aligned\nwith their values.\n