Software often produces biased outputs. In particular, machine learning (ML) based software is known to produce erroneous predictions when processing <italic>discriminatory inputs</italic>. Such unfair program behavior can be caused by societal bias. In the last few years, Amazon, Microsoft and Google have provided software services that produce unfair outputs, mostly due to societal bias (e.g., gender or race). In such events, developers are saddled with the task of conducting <italic>fairness testing</italic>. Fairness testing is challenging; developers are tasked with <italic>generating discriminatory inputs that reveal and explain biases</italic>. We propose a <italic>grammar-based fairness testing approach</italic> (called <sc>Astraea</sc>) which leverages context-free grammars to generate discriminatory inputs that <italic>reveal fairness violations</italic> in software systems. Using probabilistic grammars, <sc>Astraea</sc> also provides fault diagnosis by <italic>isolating the cause</italic> of observed software bias. <sc>Astraea</sc>’s diagnoses facilitate the improvement of ML fairness. <sc>Astraea</sc> was evaluated on 18 software systems that provide three major <italic>natural language processing</italic> (NLP) services. In our evaluation, <sc>Astraea</sc> generated fairness violations at a rate of about 18%. <sc>Astraea</sc> generated over 573K discriminatory test cases and found over 102K fairness violations. Furthermore, <sc>Astraea</sc> improves software fairness by about 76% via model-retraining, on average.