The convergence of artificial intelligence with structured human oversight has emerged as a defining paradigm for building trustworthy, adaptive, and domain-aware computational systems. This review synthesises the methodological foundations, enabling mechanisms, and applied dimensions of interactive learning pipelines that embed expert judgement into model training, validation, and refinement. It traces the evolution of annotation strategies, active sampling, weak supervision, and feedback-driven optimisation, highlighting how collaborative workflows between algorithms and practitioners mitigate data scarcity, distributional drift, and opaque decision logic. The work examines theoretical underpinnings that link uncertainty quantification, query efficiency, and cognitive load, alongside architectural patterns that operationalise iterative input through dashboards, labelling platforms, and reinforcement signals from preference comparisons. Attention is paid to evaluation protocols that balance accuracy with annotator reliability, fairness, and longitudinal stability in high-stakes settings such as clinical informatics, industrial automation, financial compliance, and public-sector analytics. The review further explores socio-technical dimensions, including governance frameworks, explainability, and the allocation of decisional authority between machines and professionals. Cross-cutting challenges are identified, encompassing labour economics of annotation, verification of subjective tasks, benchmarking heterogeneity, and scalability of collaborative loops in distributed environments. Emerging directions are discussed, spanning foundation-model alignment, multimodal supervision, federated oversight, and domain-adaptive interfaces that reduce friction between subject-matter experts and learning systems. By consolidating evidence from computer science, human factors engineering, data science, and applied professional domains, the article articulates a coherent conceptual map of the field and outlines research priorities for transparent, resilient, and ethically accountable intelligent systems. It offers researchers, engineers, and policymakers a structured reference point for deployments where automated inference and expertise reinforce each other, and sets directions for inquiry, integrating technical innovation with institutional sensibility.
Paper
Full text
Human-in-the-Loop Machine Learning: A State of the Art
Semantic Scholar · 2022
Abstract
The convergence of artificial intelligence with structured human oversight has emerged as a defining paradigm for building trustworthy, adaptive, and domain-aware computational systems. This review synthesises the methodological foundations, enabling mechanisms, and applied dimensions of interactive learning pipelines that embed expert judgement into model training, validation, and refinement. It traces the evolution of annotation strategies, active sampling, weak supervision, and feedback-driven optimisation, highlighting how collaborative workflows between algorithms and practitioners mitigate data scarcity, distributional drift, and opaque decision logic. The work examines theoretical underpinnings that link uncertainty quantification, query efficiency, and cognitive load, alongside architectural patterns that operationalise iterative input through dashboards, labelling platforms, and reinforcement signals from preference comparisons. Attention is paid to evaluation protocols that balance accuracy with annotator reliability, fairness, and longitudinal stability in high-stakes settings such as clinical informatics, industrial automation, financial compliance, and public-sector analytics. The review further explores socio-technical dimensions, including governance frameworks, explainability, and the allocation of decisional authority between machines and professionals. Cross-cutting challenges are identified, encompassing labour economics of annotation, verification of subjective tasks, benchmarking heterogeneity, and scalability of collaborative loops in distributed environments. Emerging directions are discussed, spanning foundation-model alignment, multimodal supervision, federated oversight, and domain-adaptive interfaces that reduce friction between subject-matter experts and learning systems. By consolidating evidence from computer science, human factors engineering, data science, and applied professional domains, the article articulates a coherent conceptual map of the field and outlines research priorities for transparent, resilient, and ethically accountable intelligent systems. It offers researchers, engineers, and policymakers a structured reference point for deployments where automated inference and expertise reinforce each other, and sets directions for inquiry, integrating technical innovation with institutional sensibility.