The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models

Natural-language assistants are designed to provide users with helpful responses while avoiding harmful outputs, largely achieved through alignment to human preferences. Yet there is limited understanding of whether alignment techniques may inadvertently perpetuate or even amplify harmful biases inherited from their pre-aligned base models. This issue is compounded by the choice of bias evaluation benchmarks in popular preference-finetuned models, which predominantly focus on dominant social categories, such as binary gender, thereby limiting insights into biases affecting underrepresented groups. To address this gap, we center transgender, nonbinary, and other gender-diverse (TGNB) identities to investigate how alignment procedures encode and interact with harmful TGNB biases in LLMs. Our key contributions include: 1) a comprehensive survey of bias evaluation modalities across leading preference-finetuned LLMs, highlighting critical gaps in gender-diverse representation, 2) systematic evaluation of gender-diverse biases across 16 publicly-available models across Direct Preference Optimization (DPO) stages, uncovering harms popular bias benchmarks fail to detect, and 3) a flexible framework for measuring harmful gender-diverse biases in implicit reward signals applicable to other social contexts. Our findings suggest that DPO-aligned models are particularly sensitive to supervised finetuning (SFT), and may amplify two forms of real-world gender-diverse harms reflected in their base models: stigmatization and gender non-affirmative language. We conclude with recommendations tailored to DPO and broader alignment practices, advocating for the adoption of community-informed bias evaluation frameworks to more effectively identify and address underrepresented harms in LLMs.