Content moderation is often performed by a collaboration between humans and\nmachine learning models. However, it is not well understood how to design the\ncollaborative process so as to maximize the combined moderator-model system\nperformance. This work presents a rigorous study of this problem, focusing on\nan approach that incorporates model uncertainty into the collaborative process.\nFirst, we introduce principled metrics to describe the performance of the\ncollaborative system under capacity constraints on the human moderator,\nquantifying how efficiently the combined system utilizes human decisions. Using\nthese metrics, we conduct a large benchmark study evaluating the performance of\nstate-of-the-art uncertainty models under different collaborative review\nstrategies. We find that an uncertainty-based strategy consistently outperforms\nthe widely used strategy based on toxicity scores, and moreover that the choice\nof review strategy drastically changes the overall system performance. Our\nresults demonstrate the importance of rigorous metrics for understanding and\ndeveloping effective moderator-model systems for content moderation, as well as\nthe utility of uncertainty estimation in this domain.\n