Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?

We appreciate the insightful feedback from the reviewer and have provided detailed responses to each question below. Should there be any follow-up questions, we would gladly engage in further discussion.​​ > Q1. “The number of tokens to train baseline methods is unknown. Is it same with the VocADT?” We are using less number of tokens compared to the baseline for adapter training. We compare the “adapter training” for VocADT with “hypernetwork training” in ZeTT and “external embedding training” for FOCUS. - ZeTT: In Minixhofer et al., 2024, they train English + Code hypernetwork for 200k gradient update steps with a batch size of 128 tokens and a sequence length of 128. For the multilingual decoder-style models, we start from the English + Code checkpoint. Therefore, 6.55B tokens in total. - FOCUS use fastText which is trained on Wikipedia and CommonCrawl. The fastText did not state the exact numbers in their paper for all language. However, we found that Russian only (overlaps with our Mixed and Cyrillic group) has 102B tokens in CommonCrawl and 4% of coverage in training already exceed the numbers of token that we used for adapter training, where the minimum coverage rate they reported is 63.4%. - OFA use ColexNet+ as their external embeddings, which is also unclear to identify the numbers of tokens they used to train the embeddings. | Ours/Baelines | VocADT | ZeTT | |---|---|---| | Token Trained | 3.5B | 6.5B | | Trained Part | Adapter | Multilingual Hypernetwork | > Q2. “It can be found that the language Korean also suffers from severe fragment in Figure 3, while the improvement on Korean is little. How to interpret this phenomenon?” We believe the severity of overfragmentation in Greek far surpasses that in Korean, as Greek exhibits more than twice the severity, and that this distinction makes it hard to apply the same explanation to both Korean and Greek languages. In Figure 3, the token count in Greek (150k, with the bars clipped) is more than twice that of Korean (70k), while Bulgarian and Russian show counts of 55k and 54k, respectively. Therefore, we group Korean’s fragmentation with Russian and Bulgarian as exhibiting similar severity, while Greek stands out as uniquely severe. > W1. “Lack of novelty; incremental work combining the heuristic methods like OFA in the initialization and language adapters like MAD-X.” We would like to highlight that there is a distinctive difference between MAD-X and ours in terms of architecture and as well as between OFA in terms of heuristic methodology, which make it hard to say VocADT is merely an incremental work from those. MAD-X incorporates adapters within the Transformer layers, as illustrated in their paper Figure 1. In contrast, our adapters are positioned at the bottom and top of the original model, creating a clear architectural divergence. Additionally, unlike OFA or FOCUS, which rely on initialization heuristics requiring external embeddings, we deliberately avoid such dependencies. Instead, we adopt a simple heuristic to initialize the adapter which is widely used among vocabulary transfer. > W2. “Missing details and additional experiments to support the findings: only Mistral-7b is used in this work…. The detail settings for baseline methods are unknown in this paper.” Thank you for the suggestions. We will add the detailed settings for baseline methods including the number of tokens used for the adaptation in the future version as well as the result of Llama experiment. > W3. “Writing is not clear: The Figure 3 is confusing and hard to follow: a) The y-axis of 3(a) left part are increase rate(%), thus the negative results are missing like "id-en", "belebele" on Indonesian? b). The mismatched legends for left and right part, which are "VocADT-multi(ours), ZETT-mono"(left) and "mono, multi, mistral"(right). These results will be better to report on Tables." We limited the left side of Figure 3 to the positive range to make it easier to see which languages benefit from vocabulary transfer, but as you mentioned, it may be confusion, so we also include full tables in Appendix B.

We appreciate the insightful feedback from the reviewer and have provided detailed responses to each question below. Should there be any follow-up questions, we would gladly engage in further discussion.​​ > Q1. “Can you further explain the limitations of heuristics-based methods and methods that rely on external embeddings? For example, what does it mean when the introduction states that ‘heuristics-based methods often lack adaptability’?” Instead of relying on methods that heuristically determine weights to initialize the new embeddings using a weighted average of the original embeddings, our approach learns these weights by training adapters. The base principle of most vocabulary transfer methods is to generate new embeddings based on the original ones. For example, methods like FVT and VIPI create the new embeddings by simply averaging the embeddings of subword tokens. Meanwhile, approaches such as WECHSEL, OFA, RAMEN, and FOCUS use external resources to calculate the weights for initializing new embeddings through a weighted average of original embeddings. However, embeddings generated by these heuristic-based methods are not fully integrated with the main body of the original model and typically require an additional training phase involving full-weight updates to adjust to the new vocabulary (i.e. LAPT). In contrast, our method creates new embedding vectors by learning how to combine the existing embedding vectors without requiring external resources. This approach enables the model, with only its embeddings replaced, to perform effectively compared to heuristic-based models. > Q2. “Is your proposed method, VocADT, effective for low-resource languages?” Based on the result, the VocADT method is effective for LR languages. We experimented with two low-resource languages, Swahili and Haitian Creole (Table 1). All the task performance for these two languages increased with VocADT methods, while ht-en translation decreased with ZeTT method (Figure 3).

We appreciate the insightful feedback from the reviewer and have provided detailed responses to each question below. Should there be any follow-up questions, we would gladly engage in further discussion.​​ > Q1. “Can you give an example where the case 3 mentioned in section 3.2 happen?” The third scenario, where a token from the new vocabulary is neither an overlapping token nor subworded by the original tokenizer, does not occur when the original vocabulary supports byte fallback or when the script is “alphabetic,” such as Latin, Cyrillic, or Greek. These cases can easily be decomposed into byte-level components or individual alphabetic characters. However, for scripts like Hangul, which represent combinations of consonants and vowels rather than isolated phonemes, some tokens may fall into this category if byte-level decomposition is not feasible. For instance, the VocADT-Mixed grouped model includes a new token “처럼” (meaning “like” in English), a highly frequent word in Korean—the only non-alphabetic language among the eight we examine. However, the original Mistral does not have a token for “럼,” making it impossible to decompose the new word without relying on byte-level decomposition. Most modern language models now support byte-level decomposition, which makes this third case increasingly uncommon in practice. > W1. “The motivation behind restricting the new embedding to a linear combination of original embeddings has not been explained.” The fundamental principle behind most vocabulary transfer methods is to generate new embeddings based on the original embeddings. Typically, these methods begin by copying embedding vectors for identical tokens from the original to the new embedding space. Subsequently, multiple embedding vectors from the original embeddings are combined to create embeddings for new vocabulary items. For example, methods such as FVT and VIPI compute the new embeddings by simply averaging the embeddings of subword tokens, while approaches like WECHSEL, OFA, RAMEN, and FOCUS utilize external resources to determine the weights to initialize the new embeddings with a weighted average of the original embeddings. While the details of the approach differ across methods, the overarching idea remains consistent. Mundra et al. even establish theoretically that initializing within the convex hull of existing embeddings—such as using a weighted average of source embeddings—is a good initialization. Our motivation stems from the question: rather than deciding how to combine existing embedding vectors heuristically, why not learn this process to create new embedding vectors? Relying on heuristics may lack adaptability that typically requires an additional training phase of full-weight updates to fully adapt to the new vocabulary. Building upon prior works, we propose to learn linear combinations with vocabulary adapters. We will further elaborate on this motivation in a future version.

We appreciate the insightful feedback from the reviewer and have provided detailed responses to each question below. Should there be any follow-up questions, we would gladly engage in further discussion. > Q1. “Adaptation improves performance for fragmented Greek but less so for similarly fragmented Korean. Does this suggest vocabulary adaptation may not be effective for highly fragmented languages, and that other properties of Greek contribute to the improvements?” We argue that vocabulary adaptation is generally effective in severely fragmented languages like Greek, and we believe the severity of overfragmentation in Korean is not similar to that in Greek, as Greek exhibits more than twice the severity, and that this distinction makes it hard to apply the same explanation to both Korean and Greek languages. In Figure 3, the token count in Greek (150k, with the bars clipped) is more than twice that of Korean (70k), while Bulgarian and Russian show counts of 55k and 54k, respectively. Therefore, we group Korean’s fragmentation with Russian and Bulgarian as exhibiting similar severity, while Greek stands out as uniquely severe. > Q2. “Was any ablation conducted comparing initialization methods (e.g., one-hot vs. multi-hot) for the adapter module and their impact on downstream task performance?” The only ablation experiment is with the different alpha values (0 vs 0.1). We will include additional experiments with different initialization methods including different one-hot vs multi-hot and different alpha values. > Q3. “How effective would VocADT be for mixed-script or multilingual groups with shared vocabulary at different scales? Exploring grouping strategies in larger, mixed-language models could provide valuable insights into scalability for multilingual setups.” We will further experiment to explore language scalability. Thank you for the suggestion!