Simplicity bias (SB), the tendency of neural networks to learn simpler functions, is a key factor in good generalization. Recent studies have examined SB mainly using simple models or easily controlled synthetic tasks. However, the effect of SB on generalization in real-world settings remains unclear, which hinders the understanding of neural network generalization mechanisms in practice. In this paper, we systematically study how simplicity bias manifests in CLIP models and affects generalization across image classification tasks. First, we theoretically analyze the limitations of existing complexity measures, showing that they often conflate weak model expressivity with SB. To address this, we propose a frequency-aware measure that captures a model's sensitivity along curated 1-D interpolation paths in frequency-restricted input space. This measure reveals the shift in sensitivity toward low-frequency components induced by SB-modulation methods, offering a possible explanation for why SB improves generalization in image classification. Second, we modulate simplicity bias to evaluate generalization in zero-shot classification and in fine-tuning evaluation on Out-of-Distribution (OOD), image corruption, adversarial robustness, and adversarial transferability. These experiments show that the relative ordering of tasks by preferred SB strength is largely model-agnostic: stronger SB is associated with better OOD generalization but weaker adversarial robustness. The resulting insights urge careful consideration of designing models with a proper level of SB that matches the target task. "The code is available at https://github.com/rafa-cxg/SimplicityBiasCLIP."