MISALIGNED ANNOTATION PROCESSING

Aspects of processing misaligned annotations include receiving a tokenized document and offset annotation file at a processor. The tokenized document includes a source document and corresponding tokens resulting from a low-level segmentation process. Annotations from the annotation file are applied, in conjunction with tokenization rules, to the source document, and a misalignment responsive to the applying is determined. If the misalignment is caused by an offset mismatch, an offset number of characters between the position counts in the annotation file and the source document is calculated, and the position count in the annotation file is adjusted to coincide with the position count in the source document. If the misalignment is not caused by an offset mismatch, a current position count in the source document is reset to a position count of a previous location in which a most recent alignment between the annotation file and the source document was ascertained.