While automatic performance metrics are crucial for machine learning of\nartificial human-like behaviour, the gold standard for evaluation remains human\njudgement. The subjective evaluation of artificial human-like behaviour in\nembodied conversational agents is however expensive and little is known about\nthe quality of the data it returns. Two approaches to subjective evaluation can\nbe largely distinguished, one relying on ratings, the other on pairwise\ncomparisons. In this study we use co-speech gestures to compare the two against\neach other and answer questions about their appropriateness for evaluation of\nartificial behaviour. We consider their ability to rate quality, but also\naspects pertaining to the effort of use and the time required to collect\nsubjective data. We use crowd sourcing to rate the quality of co-speech\ngestures in avatars, assessing which method picks up more detail in subjective\nassessments. We compared gestures generated by three different machine learning\nmodels with various level of behavioural quality. We found that both approaches\nwere able to rank the videos according to quality and that the ranking\nsignificantly correlated, showing that in terms of quality there is no\npreference of one method over the other. We also found that pairwise\ncomparisons were slightly faster and came with improved inter-rater\nreliability, suggesting that for small-scale studies pairwise comparisons are\nto be favoured over ratings.\n