In lossy image compression, models face the challenge of hallucinating details or generating out-of-distribution samples due to the information bottleneck. Sometimes, hallucinations are necessary to generate in-distribution samples, but the optimal level depends on image content, as humans notice small changes that affect meaning. We propose ConHa, a compression method that dynamically balances hallucination levels based on content. We train a model to predict user preferences on detail and hallucination levels and use this prediction to adjust the perceptual weight in the reconstruction loss. To evaluate our modes performance, we gathered 1,531 comparisons from 40 participants across three bitrates, see Figure 1. Our model outperforms both Hyperprior [1] and HiFiC [2], achieving a middle ground. The performance improvement over HiFiC holds true at all bitrates. When the perceptual loss weight is fixed to the dataset mean, and not predicted for each image, performance drops to near HiFiC levels. This shows the importance of conditioning the perceptual loss weight on the image for optimal performance. ConHa selectively hallucinates details based on content, improving realism and outperforming all baseline models.