Designing Scalable, Context-Adapted CS Placement Exams: An IRT-Based Approach

Large introductory CS courses serve students with heterogeneous programming backgrounds; those with informal preparation fall into a gray area that existing placement mechanisms fail to serve. We present a framework for designing scalable, psychometrically validated placement exams anchored to CS0 final performance and using Item Response Theory (IRT) to validate against institutional benchmarks. Applying this framework at our institution with CS0 completers (N = 236) as the benchmark, we found that students placed into CS1 had ability levels that matched the top 15% of CS0 completers, with 80% achieving A or B in CS1, and showed exceptional separation in ability from CS0-placed students (Hedges' g = 2.84). We make three key contributions: (1) A data-driven psychometric framework enabling iterative optimization of exam items and placement criteria against local benchmarks; (2) Demonstration that PrairieLearn's algorithmic randomization provides exam security while enabling rapid scaling from 19-student pilot to 236-student assessment; (3) Post-hoc analysis suggesting ability estimates (?) enable more precise placement than raw scores, with item difficulties remaining stable across cohorts. This reproducible framework enables institutions to develop locally anchored placement exams on auto-grading platforms that support randomized question generation.