MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization

Summary

This paper proposed MAGNET, a gradient-based tokenization method, to address an over-segmentation issue when handling multilingual text data written in different scripts. MAGNET learns to predict segment boundaries between byte tokens in a text sequence. MAGNET has a customizable architecture where byte-level sequence are routed through language-script-specific predictors, which implements a language-dependent tokenization and thus avoids over-segmentation in non-Latin scripts. The authors conducted extensive experiments on nine languages with distinct scripts. Their results show that the proposed approach works well in the downstream task and moreover contributes to the speed-up at inference time. This paper is well organized and clearly motivated. The over-segmentation gets a bigger issue in multilingual model training so the proposed approach will be useful.

Strengths

- Well organized paper. easily to follow each section. enough description is provided. - Extensive results with 9 languages with 3 different scripts.. the authors conducted extensive experiments in the downstream task - Successfully reducing the inference time

Weaknesses

- experiments are carried out in three different scripts. Other interesting languages would be Chinese, Japanese and Korean scripts as another major examples that struggle with the out-of-vocabulary issues. - despite addressing the oversegmentation, performance is still similar to that of byte-level or baselines.

Questions

- I see the proposed approach successfully overcoming segmentation disparities in subword tokenization though what else would be needed to improve the downstream task perfomance? For instance, Figure 3 shows that the byte-level model also achieves good scores across. - Other interesting languages would be Chinese, Japanese and Korean scripts. Have you ever experimented with the languages? If yes, do you observe anything new?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

The authors propose MAGNET (multilingual adaptive gradient-based tokenization) to remove the common problem in non-Latin-script languages getting only tokens (subwords assigned in the vocabulary) representing short character sequences as opposed to English getting high-semantic-content tokens. As opposed to previous gradient based tokenization strategies which globally minimize compression cost across all sequences, they use language specific segmentation predictors to reduce oversegmentation in non-Latin-script languages, to produce a better multilingual byte-level LM. Within a byte-level LM the tokenization acts as an information bottleneck, where the input subword byte sequences have to be predicted again at the output using an "upsampling module", while next word prediction takes place over the tokenized-on-the-fly blocks. They compare MAGNET to DTP (which doesn't have the same conditioning but otherwise has identical structure) and naive BPE over the vocabulary. Figure 2 shows that the avg # tokens per passage across all languages, as opposed to BPE's severe jump for Hindi, Bengali, and Telegu, and DTP's punishment of both Cyrillic and Indic script languages.

Strengths

This tokenization issue comes up a lot in multilingual NLP research. Different segmentation levels between languages can severely hamper LM performance---particularly open source, open weight LM performance---on many cross-lingual tasks including summarization, translation, QA, and reasoning, so fixing this with architectures such as this and scaling them will prove very impactful. This simple proposal to fit the hourglass architecture to script-denominated segementation predictors is a simple but useful refinement. Simple presentation of their method's superior results for simple tokenization, which also generalized to better performance on most multilingual tasks such as XNLI, PAWS-X, and SIB. The savings in token cost for passages translate into better inference time as well.

Weaknesses

Language ID is still an issue here. While they are able to fit these individual segmentation predictors based on implicit language family by script (which can be inferred from position in the UTF-8 table), I think the case could be made that they're basically just kicking the unequal tokenization can down a level. Now words in Telegu might be worse segmented than words in Hindi, etc. Really an inline language identification-conditioned segmentation module would be best here. But that's probably better as a direction for future work anyway.

Questions

N/A

Rating

7

Confidence

3

Soundness

4

Presentation

3

Contribution

3

Limitations

Yes, they mention language-specific issues to expanding their method beyond the alphabetic languages they examined. I would like to see them mention the further potential improvements to using inline language ID prediction (conditioned in training, maybe predicted implicitly in inference?) to control the segmentation module and maybe ensure more equitable performance across the languages.

Summary

In this paper, the authors propose a multilingual adaptive gradient-based tokenization approach to reduce over-segmentation for non-Latin language texts. In particular, they improve the previous Dynamic Token Pooling method by inferring token boundaries with different predictors for different languages. They conduct extensive experiments to demonstrate that not only the over-segmentation issue is reduced, the inference efficiency also increases.

Strengths

* Apply different boundary predictors for different languages/scripts, which is a natural improvement over the previous approach, DTP * The results look reasonable and similar to human tokenization

Weaknesses

* Basic token prediction tasks or text generation tasks are not included in the evaluation * Downstream tasks' performances aren't different * Inference time improvement isn't much compared to DTP

Questions

In Figure 2, are some lines missing or overlapped?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Summary

This paper presents multilingual adaptive gradient-based tokenization (MAGNET), which aims to reduce over-segmentation in non-Latin script languages in multilingual settings. MAGNET processes byte-level sequences and routes them through language-script-specific predictors, each optimized for its respective script, such as Latin, Cyrillic, and Indic. The segmentation is modeled using a sigmoid function, making it differentiable. Experimental results show that MAGNET can maintain downstream task performance while reducing inference latency.

Strengths

1. The main contribution of MAGNET is its ability to maintain performance on downstream tasks while improving inference time by reducing tokenized sequence length. Compared with byte-level tokenizers, inference is more than twice as fast while maintaining or even improving performance. 2. The idea is simple and straightforward, making the paper easy to follow.

Weaknesses

1. The proposed method doesn't conduct experiments on languages with sufficient coverage. The predictors only include Latin, Cyrillic, and Indic, raising concerns about its applicability to languages lacking spaces as word boundaries, such as Chinese and Japanese. 2. The importance of byte-level tokenization and gradient-based tokenizers as research directions is unclear. The experimental results don't demonstrate the significance of byte-level tokenization in terms of downstream performance and inference latency. 3. The training objective remains byte-level without segmentation. This raises questions about whether MAGNET still needs to generate very long byte sequences during inference.

Questions

1. Is there any analysis of the segmentation? For example, can the segmentation find phrase boundaries? 2. Is it possible to incorporate prior domain knowledge (such as word dictionaries or tokenization from other tokenizers) to improve gradient-based tokenization? 3. Are these three predictors sufficient to achieve good performance across all languages? What's the performance for languages that don't have spaces in the sequence, such as Chinese or Japanese?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Please refer to weaknesses.