Applying Large Language Models to Enhance the Assessment of Parallel Functional Programming Assignments
Courses in computer science (CS) often assess student programming assignments manually, with the intent of providing in-depth feedback to each student regarding correctness, style, efficiency, and other quality attributes. As class sizes increase, however, it is hard to provide detailed feedback consistently, especially when multiple assessors are required to handle a larger number of assignment submissions. Large language models (LLMs), such as ChatGPT, offer a promising alternative to help automate this process in a consistent, scalable, and minimally-biased manner.This paper explores ChatGPT-4’s scalablility and accuracy in assessing programming assignments based on predefined rubrics in the context of a case study we conducted in an upper-level undergraduate and graduate CS course at Vanderbilt University. In this case study, we employed a method that compared assessments generated by ChatGPT-4 against human graders to measure the accuracy, precision, and recall associated with identifying programming mistakes. Our results show that when ChatGPT-4 is used properly (e.g., with appropriate prompt engineering and feature selection) it can improve objectivity and grading efficiency, thereby acting as a complementary tool to human graders for advanced computer science graduate and undergraduate students.CCS CONCEPTS• Software and its engineering → Software maintenance tools; • Applied computing → Computer-assisted instruction.
Paper
Full text
Applying Large Language Models to Enhance the Assessment of Parallel Functional Programming Assignments
Semantic Scholar · Computer Science · 2024
Abstract
Courses in computer science (CS) often assess student programming assignments manually, with the intent of providing in-depth feedback to each student regarding correctness, style, efficiency, and other quality attributes. As class sizes increase, however, it is hard to provide detailed feedback consistently, especially when multiple assessors are required to handle a larger number of assignment submissions. Large language models (LLMs), such as ChatGPT, offer a promising alternative to help automate this process in a consistent, scalable, and minimally-biased manner.This paper explores ChatGPT-4’s scalablility and accuracy in assessing programming assignments based on predefined rubrics in the context of a case study we conducted in an upper-level undergraduate and graduate CS course at Vanderbilt University. In this case study, we employed a method that compared assessments generated by ChatGPT-4 against human graders to measure the accuracy, precision, and recall associated with identifying programming mistakes. Our results show that when ChatGPT-4 is used properly (e.g., with appropriate prompt engineering and feature selection) it can improve objectivity and grading efficiency, thereby acting as a complementary tool to human graders for advanced computer science graduate and undergraduate students.CCS CONCEPTS
- Software and its engineering → Software maintenance tools;
- Applied computing → Computer-assisted instruction.