What Counts as "Socratic" in LLM-Based Tutors? A Systematic Review of Construct Operationalization and Evaluation Validity

This systematic review characterizes how the "Socratic method" is operationalized in LLM-based conversational tutoring systems and evaluates the validity with which studies demonstrate that their systems implement it. Rather than asking whether Socratic tutors are effective, the review adopts a construct-validity framing: it asks what researchers mean by "Socratic" and whether their evaluations measure that construct. Four research questions are addressed: (RQ1) what distinct operational constructions of "Socratic" appear in these systems (producing a taxonomy of operationalizations); (RQ2) which mechanisms elicit or constrain Socratic behavior (prompting, fine-tuning, agentic orchestration, grounding/RAG) and how faithful they are to the declared construct; (RQ3) how "Socraticness" is evaluated and how valid those measures are with respect to learning versus surface dialogue features; and (RQ4) whether and how these systems are designed and evaluated as a means of mitigating student over-reliance on AI. The review follows PRISMA-P/PRISMA 2020 guidance. A strict eligibility stance requires a learner (human or simulated pedagogical proxy) to be present, excluding learner-free reasoning-prompting uses of "Socratic." Searches were conducted across Scopus, IEEE Xplore, Web of Science, ACL Anthology, and arXiv (2020–2026, English). The ACM Digital Library was searched on 2026-06-28, once institutional access became available; consistent with the prior coverage judgement it overlapped heavily with the other sources (4 unique records of 20). Screening is performed in Rayyan with dual independent reviewers; a first calibration round below the κ ≥ 0.70 threshold triggered guideline refinement and re-calibration (κ = 0.881) before full title/abstract screening. Given expected heterogeneity, synthesis will be narrative and taxonomic rather than meta-analytic.