Recent years have seen a proliferation of attention mechanisms and the rise\nof Transformers in Natural Language Generation (NLG). Previously,\nstate-of-the-art NLG architectures such as RNN and LSTM ran into vanishing\ngradient problems; as sentences grew larger, distance between positions\nremained linear, and sequential computation hindered parallelization since\nsentences were processed word by word. Transformers usher in a new era. In this\npaper, we explore three major Transformer-based models, namely GPT, BERT, and\nXLNet, that carry significant implications for the field. NLG is a burgeoning\narea that is now bolstered with rapid developments in attention mechanisms.\nFrom poetry generation to summarization, text generation derives benefit as\nTransformer-based language models achieve groundbreaking results.\n