What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers

GPT-3 shows remarkable in-context learning ability of large-scale language\nmodels (LMs) trained on hundreds of billion scale data. Here we address some\nremaining issues less reported by the GPT-3 paper, such as a non-English LM,\nthe performances of different sized models, and the effect of recently\nintroduced prompt optimization on in-context learning. To achieve this, we\nintroduce HyperCLOVA, a Korean variant of 82B GPT-3 trained on a Korean-centric\ncorpus of 560B tokens. Enhanced by our Korean-specific tokenization, HyperCLOVA\nwith our training configuration shows state-of-the-art in-context zero-shot and\nfew-shot learning performances on various downstream tasks in Korean. Also, we\nshow the performance benefits of prompt-based learning and demonstrate how it\ncan be integrated into the prompt engineering pipeline. Then we discuss the\npossibility of materializing the No Code AI paradigm by providing AI\nprototyping capabilities to non-experts of ML by introducing HyperCLOVA studio,\nan interactive prompt engineering interface. Lastly, we demonstrate the\npotential of our methods with three successful in-house applications.\n