Method for Customizable Automated Tagging: Addressing the Problem of Over-tagging and Under-tagging Text Documents

Using author provided tags to predict tags for a new document often results\nin the overgeneration of tags. In the case where the author doesn't provide any\ntags, our documents face the severe under-tagging issue. In this paper, we\npresent a method to generate a universal set of tags that can be applied widely\nto a large document corpus. Using IBM Watson's NLU service, first, we collect\nkeywords/phrases that we call "complex document tags" from 8,854 popular\nreports in the corpus. We apply LDA model over these complex document tags to\ngenerate a set of 765 unique "simple tags". In applying the tags to a corpus of\ndocuments, we run each document through the IBM Watson NLU and apply\nappropriate simple tags. Using only 765 simple tags, our method allows us to\ntag 87,397 out of 88,583 total documents in the corpus with at least one tag.\nAbout 92.1% of the total 87,397 documents are also determined to be\nsufficiently-tagged. In the end, we discuss the performance of our method and\nits limitations.\n