The Segment Anything Model (SAM) has demonstrated impressive generalization in prompt-driven segmentation. Yet, the potential of semantic text prompts such as class names remains underexplored compared to traditional spatial prompts, including points and boxes. This paper introduces SAM-PTx, a parameter-efficient approach to adapt SAM by leveraging frozen CLIP-derived text embeddings as class-level semantic guidance. Specifically, we propose a lightweight Parallel-Text Adapter that injects text embeddings into SAM’s image encoder, enabling semantically conditioned binary segmentation with point prompts while freezing most of SAM’s parameters. Through supervised experiments and ablations on low-data subsets of COCO and ADE20K, as well as the challenging COD10K benchmark, we show that incorporating fixed text embeddings yields consistent improvements over purely spatial prompt baselines, while improving robustness to prompt variation and train–test prompt mismatch. Importantly, our analysis reveals that the benefits of semantic conditioning become more pronounced as semantic supervision scales (e.g., when the number of class-conditioned image–mask pairs increases), suggesting that SAM-PTx increasingly leverages textual priors beyond spatial disambiguation alone. To the best of our knowledge, this is the first work to use text prompts for segmentation on the COD10K dataset. Overall, our results indicate that integrating semantic conditioning directly into SAM’s architecture provides a practical, scalable pathway for efficient adaptation under limited supervision, with minimal additional parameters.