Language Model Tokenizers Introduce Unfairness Between Languages

Summary

The paper studies the discrepancy in the tokenization length in different languages. It shows the unfairness of utilizing tokenization in different languages due to the cost, latency, and long-term language dependencies. The paper evaluates the unfairness between different tokenization strategies and model architectures. The paper's motivation is clear, and it provides an extensive analysis of multiple languages, showing the significance of the study and why it is essential in research and practical perspectives.

Strengths

**Originality** - The paper introduces an important issue on utilizing LLM in different languages and investigates the aspects of cost, latency, and long-dependency. The problem formulation is important for multilingual researchers and practitioners in deciding what tokenization strategies are cost-effective. **Quality / Significance** - The paper shows a comprehensive analysis of how the different tokenization on different languages was done in different tokenization strategies (byte-based, Unicode-based, and subword-based) on the wide range of pre-trained LLM (on encoder-only, encoder-decoder, and decoder-only) and provides the quantitative evidence on the processing time of each language compared to English. **Clarity** - The paper is well-written, and the examples (in Japanese and Shan) give a high-level idea of why the problem introduced in the paper is important.

Weaknesses

- Besides the aspects mentioned by the paper, it is unclear if the choice of the tokenization strategies would impact the performance of LLM in particular languages. - The suggestion of "building a multilingually fair parallel corpus" is vague. It would be great if the authors could give a more straightforward and practical solution.

Questions

**Questions** - Does the tokenization lengths impact the performance on languages (e.g., Shan)? **Typo** Line 177: XML-R => XLM-R

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Summary

Tokenization is a crucial, yet underappreciated component of language models. This paper presents a welcome investigation into the effects of tokenization choices across languages. The paper introduces the notion of premium for language A relative to B which measures the ratio of the average number of tokens for translations of the same sentence in the two languages. The authors show that the premium relative to English can be over 10 for certain low resource languages. The paper then highlights the negative effects of having a high premium, which include: cost, latency, and the ability to fit less content in the fixed context window of an LM. Based on these findings, the authors make a case for developing multilingually fair tokenizers where the premiums are close to one across languages.

Strengths

1. Tokenization is, in my view, an underappreciated problem and this work provides an insightful analysis on the effects of the tokenizer selection. 2. The discrepancies presented in the paper across languages are surprisingly large. As such, this paper may stimulate new research in the field to address the discrepancies. 3. The paper is very well written and a pleasure to read.

Weaknesses

1. The paper doesn’t discuss how to achieve the goal of training multilingually fair tokenizers, apart from providing an overview of previous approaches to developing multilingual tokenizers. An obvious solution is to reserve more tokens for the languages with high premiums. However, this would mean that the number of tokens for other languages would need to be reduced to keep the vocabulary size fixed. This in turn will likely hurt the performance on high-resource languages such as English and it may even end up hurting the performance for the rest of the languages if there is less cross-lingual transfer from high-resource languages (NB: The paper states that “multilingual models struggle to deliver on the promises of deep transfer learning” but I’m not aware of works suggesting that there’s zero transfer happening across languages.) 2. The paper lacks discussion on the potential negative effects of using a fair tokenizer (such as the one presented above) and discussion on the potential means of developing fair tokenizers. Yet, it argues strongly why the switch to fair tokenizers is necessary. This makes the paper seem a bit like a position paper (which doesn’t reduce the value of the paper in my view but makes me wonder a bit whether NeurIPS is an ideal venue for it). 3. An obvious explanation for the tokenization length differences is the widely varying amounts of available training datasets across languages. However, another possible explanation is inherent differences between languages in terms of characteristics such as morphological richness of a language. Discussion of the different explanations seems to be missing from the paper.

Questions

Could you elaborate on the following sentence “training a model from scratch with this fair tokenizer is necessary, despite potential suboptimal performance in individual languages due to vocabulary limitations”? What are the potential suboptimalities you’re referring to? Are you arguing for using a fair tokenizer regardless of any potential negative side effects?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Yes.

Summary

This paper investigates the disparities between languages caused by tokenization policies used in large language models (LLMs). The authors compare the numbers of tokens needed to represent the translations of the same sentence in different languages and observe that the number of tokens needed in one language can be an order of magnitude larger than that in the target language (mostly English). They argue that these disparities have significant real-world implications such as the increased cost and performance degradations in using LLMs.

Strengths

- I think investigating the disparities of the cost and performance in using LLMs between different languages is an important topic. - The approach taken by the authors (i.e., the use of the FLORES-200 parallel corpus) seems sound. - The observations made in their experiments are informative.

Weaknesses

- There is little technical novelty in the work. - The authors point out the problems caused by the disparities but do not present a concrete solution. They argue that LLMs should be trained from scratch with a multilingually fair subword tokenizer but do not provide any experimental results towards that solution. I would be interested to see how much the performance and efficiency of an LLM in English needs to be sacrificed to achieve the parity.

Questions

- Would it be difficult to conduct some (preliminary) experiments regarding the proposals discussed in Section 6?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Summary

The paper proposes the concept of tokenizer parity as a way to measure the fairness of tokenization across different languages in natural language processing. The authors argue that achieving tokenizer parity is necessary to improve the performance of multilingual models and address potential unfairness in the cost of accessing commercial language services, processing time and latency, and the amount of content that can be provided as context to the models. The paper suggests that training language models from scratch with a multilingually fair subword tokenizer is the only approach that can effectively address tokenization unfairness and achieve tokenizer parity. The authors provide several examples to demonstrate the effectiveness of the proposed method for measuring tokenizer parity and suggest that a variation of subword tokenization is necessary to achieve parity. The paper contributes to the field of natural language processing by proposing a new metric for measuring fairness in tokenization and providing guidance on how to achieve tokenizer parity in multilingual models.

Strengths

In terms of originality, the paper introduces the concept of "tokenizer parity" as a systematic way to assess the fairness of tokenization across different languages. This is a novel idea that has not been explored extensively in the literature. In terms of quality, the paper provides a detailed analysis of the tokenization lengths of different languages using various tokenizers. The authors also propose a method to measure tokenizer parity and demonstrate its effectiveness using several examples. In terms of clarity, the paper is well-written and easy to follow. The authors provide clear definitions of the key concepts and use examples to illustrate their points. In terms of significance, the paper addresses an important issue in natural language processing, namely the fairness of tokenization across different languages. The concept of tokenizer parity has the potential to improve the performance of multilingual models and make natural language processing more equitable across different languages. Therefore, the paper could have a positive impact on the development of multilingual models.

Weaknesses

One major weakness is that the paper does not provide a clear roadmap for how the concept of tokenizer parity could be integrated into existing natural language processing pipelines. While the authors suggest that training language models with a multilingually fair subword tokenizer is the only approach that can effectively address tokenization unfairness, they do not provide guidance on how this could be achieved in practice. More detailed recommendations for how to integrate tokenizer parity into existing natural language processing pipelines would be helpful for researchers and practitioners in the field. Another potential weakness is the lack of discussion on the impact of tokenizer disparity on downstream task accuracies. This aspect is particularly interesting and may attract more attention.

Questions

1. Can you provide more detailed recommendations for how to integrate tokenizer parity into existing natural language processing pipelines? 2. What is the potential impact of tokenizer disparity on downstream task accuracies?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The lack of clear guidance on how to integrate tokenizer parity into existing natural language processing pipelines may limit the impact of the paper's findings. Additionally, the absence of discussion on the impact of tokenizer disparity on downstream task accuracies is a limitation that could be addressed in future research. Without this information, it may be difficult for researchers and practitioners to fully understand the potential benefits of using a multilingually fair subword tokenizer.

Summary

The paper is very interesting in highlighting the disparities in tokenization across different languages, leading to cost, latency, and long-distance modeling inequalities in LLMs. The authors show that this is not limited to one type of tokenizer or a single family of LLMs. They show that different types of tokenizers for LLMs in practice, including subword, multilingual, or byte-level tokenization, all show disparities for languages in FLORES-200. Based on these results, the paper convincingly argues that we need multilingually fair tokenizers for future LLMs, so that some languages are not in disadvantages in terms of the cost to access LLMs, the latency of the service, and the amount of data that can be processed.

Strengths

The paper is well written and the first one (to my knowledge) to study tokenization disparities in LLMs at this scale. The presented results will be very interesting to LLM researchers and practitioners in understanding the disparities in tokenization across different languages in different LLMs. This has major consequences for some languages, putting them at a disadvantage in terms of the cost to access LLMs, the latency of the service, and the amount of data that can be processed.

Weaknesses

Section 6 argues that future tokenizers should be based on subword tokenization and support all Unicode codepoints. However, one of the main weaknesses of the paper is that it does not develop such multilingual fairer tokenizers, and compares them with existing tokenizers in practice. Attempting to do that would have emphasized challenges in developing such tokenizers and how one can overcome them. I found the argument around disparities in long-distance modeling a bit thin. The second paragraph of Section 5.3 discusses this briefly, but additional discussion or experiments are needed to strengthen the argument. For example, it might be helpful to include analysis with multilingual document summarization (e.g., XLSum).

Questions

I think the paper does a very good job of explaining why there are language disparities among different tokenizers. However, I feel these reasons are known already, what are the challenges in overcoming them?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper presents a large-scale study of disparities in tokenization across different languages. It discusses how LLMs are putting some languages to their disadvantage due to this. The paper itself does not propose a new tokenization scheme that needs to address its limitations.