Investigating African-American Vernacular English in Transformer-Based Text Generation

The growth of social media has encouraged the written use of African American\nVernacular English (AAVE), which has traditionally been used only in oral\ncontexts. However, NLP models have historically been developed using dominant\nEnglish varieties, such as Standard American English (SAE), due to text corpora\navailability. We investigate the performance of GPT-2 on AAVE text by creating\na dataset of intent-equivalent parallel AAVE/SAE tweet pairs, thereby isolating\nsyntactic structure and AAVE- or SAE-specific language for each pair. We\nevaluate each sample and its GPT-2 generated text with pretrained sentiment\nclassifiers and find that while AAVE text results in more classifications of\nnegative sentiment than SAE, the use of GPT-2 generally increases occurrences\nof positive sentiment for both. Additionally, we conduct human evaluation of\nAAVE and SAE text generated with GPT-2 to compare contextual rigor and overall\nquality.\n