Solving Jigsaw Puzzles in the Wild: Human-Guided Reconstruction of Cultural Heritage Fragments

Reassembling real-world archaeological artifacts from fragments is challenging due to erosion, missing regions, irregular shapes, and large-scale ambiguity. Traditional jigsaw solvers, typically designed for clean, synthetic data, struggle especially with thousands of fragments, as in the RePAIR benchmark. We propose a human-in-the-loop (HIL) puzzle-solving framework tailored for real-world cultural heritage reconstruction. Our method combines an automatic relaxation-labeling solver with interactive human guidance, enabling users to iteratively lock verified placements, correct errors, and guide assembly toward semantic and geometric coherence. We introduce two complementary strategies, ie., Iterative Anchoring and Continuous Interactive Refinement, that support scalable reconstruction under varying ambiguity and size. Experiments on RePAIR groups show our hybrid approach significantly outperforms both fully automatic and manual methods in accuracy and efficiency, offering a practical solution for human-in-the-loop artifact reassembly.