Classifiers tend to propagate biases present in the data on which they are\ntrained. Hence, it is important to understand how the demographic identities of\nthe annotators of comments affect the fairness of the resulting model. In this\npaper, we focus on the differences in the ways men and women annotate comments\nfor toxicity, investigating how these differences result in models that amplify\nthe opinions of male annotators. We find that the BERT model as-sociates toxic\ncomments containing offensive words with male annotators, causing the model to\npredict 67.7% of toxic comments as having been annotated by men. We show that\nthis disparity between gender predictions can be mitigated by removing\noffensive words and highly toxic comments from the training data. We then apply\nthe learned associations between gender and language to toxic language\nclassifiers, finding that models trained exclusively on female-annotated data\nperform 1.8% better than those trained solely on male-annotated data and that\ntraining models on data after removing all offensive words reduces bias in the\nmodel by 55.5% while increasing the sensitivity by 0.4%.\n