Aggregating partial rankings with applications to peer grading in massive online open courses

We investigate the potential of using ordinal peer grading for the evaluation\nof students in massive online open courses (MOOCs). According to such grading\nschemes, each student receives a few assignments (by other students) which she\nhas to rank. Then, a global ranking (possibly translated into numerical scores)\nis produced by combining the individual ones. This is a novel application area\nfor social choice concepts and methods where the important problem to be solved\nis as follows: how should the assignments be distributed so that the collected\nindividual rankings can be easily merged into a global one that is as close as\npossible to the ranking that represents the relative performance of the\nstudents in the assignment? Our main theoretical result suggests that using\nvery simple ways to distribute the assignments so that each student has to rank\nonly $k$ of them, a Borda-like aggregation method can recover a $1-O(1/k)$\nfraction of the true ranking when each student correctly ranks the assignments\nshe receives. Experimental results strengthen our analysis further and also\ndemonstrate that the same method is extremely robust even when students have\nimperfect capabilities as graders. We believe that our results provide strong\nevidence that ordinal peer grading can be a highly effective and scalable\nsolution for evaluation in MOOCs.\n