ENTROPY-BASED SET BLOCK DECODING UTILIZING AN ADAPTIVE ENTROPY THRESHOLD

Certain aspects of the disclosure provide techniques for language model token prediction. A method generally includes processing an input token sequence to generate, for each respective token position of a plurality of token positions in an output token sequence: a probability distribution for a set of candidate output tokens associated with the respective token position; and an entropy score based on the probability distribution; for each respective token position, determining whether the entropy score satisfies a current adaptive entropy threshold, which is based on the entropy score associated with each respective token position; based on the determination for each respective token position, performing a decoding operation comprising: a set block decoding operation or a single-token decoding operation to generate one or more output tokens that correspond to one or more token positions; and generating a first version of the output token sequence including the one or more output tokens.