Journal Impact Factor and Peer Review Thoroughness and Helpfulness: A Supervised Machine Learning Study

Objectives: To examine the thoroughness and helpfulness of peer review reports submitted to journals with different impact factors. Design: Text analysis using trained machine learning models. Data source: Random sample of 10,000 peer review reports submitted to medical and life sciences journals, drawn from Publons’ database and stratified by journal impact factor. Main outcome measures: Thoroughness of the review, indexed by sentences addressing content categories materials and methods, presentation and reporting, results and discussion, importance and relevance. Helpfulness, gauged by sentences providing suggestions and solutions, examples, praise, or criticism. Association between the prevalence of different content with the journal impact factor, across ten groups defined by journal impact factor deciles. Results: A total of 1,644 journals were represented; 187,240 sentences were analyzed. The median journal impact factor ranged from 1.23 to 8.03 across the ten groups (lowest journal impact factor 0.21, highest 74.70). Sentences on materials and methods were more common in the highest journal impact factor journals than in the lowest impact factor group (difference 7.8 percentage points; 95% CI 4.9 to 10.7%). The trend for presentation and reporting went in the opposite direction, with the journals with the highest impact factors giving less emphasis to such content (difference -8.9%; 95% CI -11.3 to -6.5%). For helpfulness, reviews for higher impact factor journals devoted less attention to suggestions and solutions and provided fewer examples than lower impact factor journals. Few differences were evident for other content categories. The proportion of sentences allocated to different content categories varied widely, even within journal impact factor groups. Conclusions: Peer review in journals with higher journal impact factor tends to be more thorough in discussing the materials and methods used but less helpful in terms of suggesting solutions and providing examples. Differences were modest and variability high, indicating that the journal impact factor is a bad predictor for the quality of peer review of an individual manuscript.