This study investigates the role of the semantic layer as the contextual foundation of Retrieval-Augmented Automated Assessment, examining how its architecture and semantic extraction mechanism contribute to the reliability, validity, and consistency of automated assessment relative to expert judgment across diverse learning task types. The proposed system is implemented as an integrated architecture combining the Moodle e-learning platform, the n8n orchestration layer for workflow automation, and a dedicated semantic layer responsible for learning context extraction. Moodle manages assignments, rubrics, and student submissions, while automated workflows process the data and activate the Retrieval-Augmented Generation (RAG) mechanism to retrieve relevant learning content and perform automated evaluation. The experimental study was conducted on a purpose-built Moodle environment, designed to ensure full control over assessment processes, event logging, and AI system integration. Results for the full sample yielded a Spearman’s rank correlation coefficient of ρ = 0.874, indicating a very strong positive correlation between expert and automated rankings. Task-level analysis further revealed that system performance is closely tied to the degree of structure and formalizability of the learning tasks: lower agreement was observed for open-ended and interpretive tasks, while structured and logically defined tasks yielded results comparable to expert assessment. These findings provide preliminary evidence that, within the tested course context, the proposed methodology is especially suitable for tasks with clearly formulated criteria and a consistent logical structure, while also identifying directions for future improvement in more subjective and context-dependent assessment scenarios.
Paper
Full text
The Semantic Layer as Contextual Foundation of Retrieval-Augmented Automated Assessment
OpenAlex · Intelligent Tutoring Systems and Adaptive Learning · 2026
Abstract
This study investigates the role of the semantic layer as the contextual foundation of Retrieval-Augmented Automated Assessment, examining how its architecture and semantic extraction mechanism contribute to the reliability, validity, and consistency of automated assessment relative to expert judgment across diverse learning task types. The proposed system is implemented as an integrated architecture combining the Moodle e-learning platform, the n8n orchestration layer for workflow automation, and a dedicated semantic layer responsible for learning context extraction. Moodle manages assignments, rubrics, and student submissions, while automated workflows process the data and activate the Retrieval-Augmented Generation (RAG) mechanism to retrieve relevant learning content and perform automated evaluation. The experimental study was conducted on a purpose-built Moodle environment, designed to ensure full control over assessment processes, event logging, and AI system integration. Results for the full sample yielded a Spearman’s rank correlation coefficient of ρ = 0.874, indicating a very strong positive correlation between expert and automated rankings. Task-level analysis further revealed that system performance is closely tied to the degree of structure and formalizability of the learning tasks: lower agreement was observed for open-ended and interpretive tasks, while structured and logically defined tasks yielded results comparable to expert assessment. These findings provide preliminary evidence that, within the tested course context, the proposed methodology is especially suitable for tasks with clearly formulated criteria and a consistent logical structure, while also identifying directions for future improvement in more subjective and context-dependent assessment scenarios.