Seeing Nothing, Saying Something: The Lack of Visual Grounding and Confabulation in Gemini Models for Histopathology

Abstract Large vision-language models (VLMs) have demonstrated remarkable performance on computational pathology benchmarks, yet their reliability under adversarial or vacuous inputs remains poorly understood. This paper examines the visual grounding behaviour of two Gemini models Gemini 3.0 Flash Preview ( gemini-flash ) and Gemini 3.1 Pro Preview ( gemini-pro ) on a well known histopathology classification task, and probes for confabulation using a adversarial blank-image set. On the real histopathology dataset both models achieve near-perfect accuracy (98.75% - 100%) across three temperatures (0.0, 0.5, 1.0) and three independent runs. On a controlled adversarial set of blank white images labelled as either benign or malignant, however, a stark divergence emerges. Gemini-flash consistently acknowledges the absence of visual content and assigns zero confidence, while Gemini-pro fabricates detailed, clinically plausible histological descriptions and reports high confidence (mean ≈ 0.95) across the same blank inputs, a behaviour we term confident confabulation . The confabulation rate of gemini-pro reaches 77.8% image-responses at temperature 0.0, dropping to 44.4% at temperature 0.5 and rising to 66.7% at temperature 1.0, while geminiflash records 0% at all temperatures. These findings raise important questions about the safety and trustworthiness of VLMs in clinical decision-support contexts, and underscore the need for comprehensive evaluation beyond standard accuracy metrics.