Do you really code? Designing and Evaluating Screening Questions for\n Online Surveys with Programmers

Recruiting professional programmers in sufficient numbers for research\nstudies can be challenging because they often cannot spare the time, or due to\ntheir geographical distribution and potentially the cost involved. Online\nplatforms such as Clickworker or Qualtrics do provide options to recruit\nparticipants with programming skill; however, misunderstandings and fraud can\nbe an issue. This can result in participants without programming skill taking\npart in studies and surveys. If these participants are not detected, they can\ncause detrimental noise in the survey data. In this paper, we develop screener\nquestions that are easy and quick to answer for people with programming skill\nbut difficult to answer correctly for those without. In order to evaluate our\nquestionnaire for efficacy and efficiency, we recruited several batches of\nparticipants with and without programming skill and tested the questions. In\nour batch 42% of Clickworkers stating that they have programming skill did not\nmeet our criteria and we would recommend filtering these from studies. We also\nevaluated the questions in an adversarial setting. We conclude with a set of\nrecommended questions which researchers can use to recruit participants with\nprogramming skill from online platforms.\n