Data Governance on Business/Data Dictionary using Machine Learning and Statistics

Abstract

This paper focuses on business/data dictionary to improve data governance for industries. Companies are expending considerable resources to govern their data. We have proposed applying different natural language method, viz., the statistical method and machine learning method. English terminologies are abbreviated because when they are saved in the database since the database has limited bytes for each column. English terminologies are abbreviated differently by different companies and even within a single company, the same word can be abbreviated differently. Therefore, companies are spending resources to maintain and standardize their data. Further, there are scenarios where two companies combine their data/business dictionary, in which case this integrated dictionary has to be re-standardized. We found a new method of applying these natural language processing approaches to the business/data dictionary for abbreviated English words which will help companies reduce the expenditure on standardization and improve the efficiency of data governance.