The ability to quantify incivility online, in news and in congressional\ndebates, is of great interest to political scientists. Computational tools for\ndetecting online incivility for English are now fairly accessible and\npotentially could be applied more broadly. We test the Jigsaw Perspective API\nfor its ability to detect the degree of incivility on a corpus that we\ndeveloped, consisting of manual annotations of civility in American news. We\ndemonstrate that toxicity models, as exemplified by Perspective, are inadequate\nfor the analysis of incivility in news. We carry out error analysis that points\nto the need to develop methods to remove spurious correlations between words\noften mentioned in the news, especially identity descriptors and incivility.\nWithout such improvements, applying Perspective or similar models on news is\nlikely to lead to wrong conclusions, that are not aligned with the human\nperception of incivility.\n