A Crowdsourcing Approach to Video Quality Assessment

We propose an open-source extension of the ITU-T Rec. P.910 subjective video quality test based on crowdsourcing principles. This extension addresses the speed, usage cost, and barrier to usage issues of P.910. We implement Absolute Category Rating (ACR), ACR with hidden reference (ACRHR), Degradation Category Rating (DCR), and Comparison Category Rating (CCR), and include rater, environment, hardware, and network qualifications, as well as gold and trapping questions to ensure quality. We have validated that the implementation is both accurate and highly reproducible.