Given the vast amount of unstructured financial text data available today, there is a high demand for reliable, quality annotations to facilitate robust model development. However, traditional methods can often be expensive and time-inefficient. In this study, we investigate annotations for emotion, sentiment, and cognitive dissonance generated by the large language models (LLMs), GPT-3.5 and GPT-4, for quarterly earnings conference calls and compare them against human annotations obtained via traditional methods. We also investigate different prompt engineering choices on LLM annotation quality, experimenting with 4 styles of prompts centered around varying the amount of contextual information given and how it is presented to the models. Our results show the GPT models are not only more consistent and reliable than human annotators, but also provide annotations in a more cost- and time-efficient manner.
Paper
Full text
Humans vs. ChatGPT: Evaluating Annotation Methods for Financial Corpora
Semantic Scholar · Computer Science · 2023
Abstract
Given the vast amount of unstructured financial text data available today, there is a high demand for reliable, quality annotations to facilitate robust model development. However, traditional methods can often be expensive and time-inefficient. In this study, we investigate annotations for emotion, sentiment, and cognitive dissonance generated by the large language models (LLMs), GPT-3.5 and GPT-4, for quarterly earnings conference calls and compare them against human annotations obtained via traditional methods. We also investigate different prompt engineering choices on LLM annotation quality, experimenting with 4 styles of prompts centered around varying the amount of contextual information given and how it is presented to the models. Our results show the GPT models are not only more consistent and reliable than human annotators, but also provide annotations in a more cost- and time-efficient manner.