SAFE Natural Language Processing

Abstract

The growth of Artificial Intelligence applications based on Natural Language Processing requires to develop risk management models that can balance opportunities with risks, especially in high-stakes scenarios. In this paper, we contribute to the development of risk models presenting two integrated statistical metrics that can measure the "Accuracy" and the "Sustainability" of Artificial Intelligence models based on text processing, in line with the requests of international recommendations and regulations, such as the European Artificial Intelligence Act. The framework is validated through experiments on three distinct binary classification tasks on widely differing data sources. By highlighting vulnerabilities and strengths across diverse NLP pipelines, the proposed metrics provide a practical tool for assessing the reliability of AI applications. These contributions aim to foster safer deployment of AI technologies in finance, by mitigating risks of potential harms to financial stability.