SYSTEMS, COMPUTER-IMPLEMENTED METHODS, AND COMPUTER PROGRAM PRODUCTS FOR DATA SEQUENCE VALIDITY PROCESSING

Embodiments provide for improved data sequence validity processing, for example to determine validity of sentences or other language within a particular language domain. Such improved processing is useful at least for arranging data sequences based on determined validity, and/or making determinations and/or performing actions based on the determined validity. A determined probability (e.g., transformed into the perplexity space) of each token appearing in a data sequence is used in any of a myriad of manners to perform such data sequence validity processing. Example embodiments provide for generating a perplexity value set for each data sequence in a plurality of data sequences, generating a probabilistic ranking set for the plurality of data sequences based on the perplexity value sets and at least one sequence ranking metric, and generating an arrangement of the plurality of data sequences based on the probabilistic ranking set.