> I find the current justification (relying on semantics) unsupported.
We appreciate the reviewer's insights and understanding of the concerns regarding the use of "morphology" and "phonology" in the context of LLMs where tokens are mapped to indices and vectors. Our justification for stating that byte-level tokens "do not provide any semantic meaning" is based on three primary reasons: semantic integrity, prevention of generating non-existent characters, and improved tokenization efficiency and contextualization.
1. Semantic Integrity:
Byte-level tokens can result in shared tokens for characters with vastly different meanings. For instance, "猟" (hunt) is encoded as <0xE7> <0x8C> <0x9F>, and "猫" (cat) is encoded as <0xE7> <0x8C> <0xAB>, sharing with the identical first two bytes despite their different meanings. This overlap can dilute the semantic integrity of Japanese kanji letters, which are ideograms unlike most of other languages. It is possible to argue that English also suffers from the same problem, e.g., “cat” and “car” are totally different, but the treatment of English tokenization is much better than that of Japanese because the tokenizer of the base model was trained on a corpus with a lot of English text and little Japanese text. In other words, it is unlikely for the base tokenizer to split them into “cat” and “car” into <c> <a> <t> and <c> <a> <r>. While LLMs derive semantics from context, starting with more semantic Japanese tokens is reasonable because Japanese kanji letter are ideograms.
2. Prevention of Generating Non-existent Characters:
Byte-level tokenization can produce byte sequences that do not correspond to any valid characters in the target language. For instance, [it is known](https://okumuralab.org/~okumura/misc/230611.html) in Japan that ChatGPT generated the non-existent Japanese word "視覴," allegedly caused by the impossible combination of byte tokens from "覚(perception)" and "聴(listen)." Such occurrences are less likely if we treat a Japanese kanji as a single token, which respects the boundaries of valid characters and words. This reduces the possibility of generating meaningless or incorrect text and thereby improves the reliability and quality of the model's output.
3. Improved Tokenization Efficiency and Contextualization:
Representing a kanji letter as a single token provides a more consistent and semantically relevant input to the token embeddings, enhancing the model's ability to understand and generate coherent text. For example, "猫"(cat) = <0xE7> <0x8C> <0xAB> shares the first two tokens with 17 other kanji in byte-level tokenization, which potentially increases the semantic ambiguity and degrades its performance. By adding kanji letters to the vocabulary, we can ensure that each token has a clear and distinct meaning, which improves the model's ability to contextualize and generate accurate representations.
> I feel that the statement "impact of vocabulary expansion on performance is minor" is not accurate.
Thank you for your feedback. we would like to clarify our position regarding the statement, "the impact of vocabulary expansion on performance is minor."
Excluding the XL-Sum task, the only dataset where we observed a relative change of more than 10% is JCQA for the Llama-2-JA-7b model. This kind of change was not observed in other model sizes or tasks, as we noted in §5.1.2.
To provide more evidence, we present an additional analysis of the JCQA scores, isolated from the averaged numbers reported in Figure 7. The scores for 7b models’ JCQA with and without vocabulary expansion (VE and $\neg$VE, respectively) are as follows:
|Num. of training tokens (B)|JCQA(VE)|JCQA($\neg$VE)|
|---|---|---|
|0|38.52|38.52|
|20|39.23|45.76|
|40|45.76|39.14|
|60|42.18|49.15|
|80|49.15|52.28|
|100|48.08|54.33|
With 40B tokens used for training, the model with vocabulary expansion (VE) outperforms the one without it ($\neg$VE). This indicates that the JCQA scores are not stable. In contrast, Figure 6 shows that the XL-Sum scores consistently show better performance for the model without vocabulary expansion, exhibiting a different trend.
Furthermore, the average and standard deviation of JCQA scores after training 40B tokens (40,60,80,100 B Tokens) are:
- Vocabulary Expansion (VE): 46.29 ± 3.09
- No Vocabulary Expansion ($\neg$VE): 48.73 ± 6.74
The observed 6.25-point (11.5%) difference of the JCQA scores between the vocabulary expansion (VE) and non-vocabulary expansion ($\neg$VE) of the Llama-2-JA-7b models does not indicate a substantial impact because of the instability and large standard deviation on the dataset.
Therefore, considering these evaluation results and the observed instability, we regard that the impact of vocabulary expansion on overall performance is minor.
In the camera-ready version, we will include these elaborations in the Appendix.