Illusions of reflection: open-ended task reveals systematic failures in Large Language Models' reflective reasoning

Humans do not just find mistakes after the fact -- we often catch them mid-stream because'reflection'is tied to the goal and its constraints. Today's large language models produce reasoning tokens and'reflective'text, but is it functionally equivalent with human reflective reasoning? Prior work on closed-ended tasks -- with clear, external'correctness'signals -- can make'reflection'look effective while masking limits in self-correction. We therefore test eight frontier models on a simple, real-world task that is open-ended yet rule-constrained, with auditable success criteria: to produce valid scientific test items, then revise after considering their own critique. First-pass performance is poor (often zero valid items out of 4 required; mean $\approx$ 1), and reflection yields only modest gains (also $\approx$ 1). Crucially, the second attempt frequently repeats the same violation of constraint, indicating'corrective gains'arise largely from chance production of a valid item rather than error detection and principled, constraint-sensitive repair. Performance before and after reflection deteriorates as open-endedness increases, and models marketed for'reasoning'show no advantage. Our results suggest that current LLM'reflection'lacks functional evidence of the active, goal-driven monitoring that helps humans respect constraints even on a first pass. Until such mechanisms are instantiated in the model itself, reliable performance requires external structure that enforces constraints. Our code is available at: https://github.com/cruiseresearchgroup/LLM_ReflectionTest