METHOD AND SYSTEM FOR TOKENIZING DOCUMENTS

A method for tokenizing documents. The method includes obtaining a document comprising text to be tokenized, isolating a first string of consecutive characters in the document, searching, in a token tree, for an expression that matches the first string, making a determination that a matching expression exists in the token tree and, based on the determination, storing the matching expression as an extracted token.