Zero-Shot Tokenizer Transfer

Summary

The paper presents a novel approach towards separating the tokenizer from the language model (LM). All modern LMs are trained with a fixed tokenizer, which prevents them from generalizing well to unseen or rarely seen tokens. Furthermore, the bounding to a specific tokenizer prevents models trained with different tokenizers from being merged and ensembled. To overcome these issues, the authors propose training a hypernetwork capable of mapping embedding parameters between tokenizers. The authors perform extensive experiments to confirm the effectiveness of the proposed approach, even when the tokenizer is transferred in a zero-shot setting.

Strengths

The paper is well-written and easy to follow. The authors describe a zero-shot tokenizer transfer problem and propose a state-of-the-art solution for transferring language models to a new tokenizer without additional training. Unlike popular heuristic-based approaches, the authors propose a novel approach that involves training a hypernetwork. Especially impressive is the attention to the technical details described in the paper, for example, the text sampling strategy or the design of the appropriate loss function. Furthermore, the authors experiment with models of different architectures, different tokenizers, and on a variety of tasks, and verify the effectiveness of the proposed approach.

Weaknesses

In my opinion, the paper is technically sound and contains an appropriate amount of explanation and evaluation. If anything, I would like to see two additional aspects explored: first, more experiments demonstrating how this approach would scale with increasingly large language models (LLMs) in question; and second, experiments with other open-sourced LLMs.

Questions

1. Experiments with TinyLlama show a similar trend to that of the Mistral results. However, I think it's important for the reader to see that the approach works with different models; I would recommend moving Appendix G to the main text. 2. You performed experiments with small language models, up to 7 billion parameters. How do you think your approach would generalize to models with a higher number of parameters, and why? 3. Do you think a similar approach would work for other modalities, such as multimodal models?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

Yes

Summary

The authors propose zero-shot tokenizer transfer, a new task where the goal is adapt language models with unseen tokenizers. They propose to tackle this problem by training a hypernetwork that can directly predict the embeddings for any given tokenizer, and such a hypernetwork is trained by sampling various tokenizations and minimizing a language modeling loss.

Strengths

1. The authors propose an interesting task of zero-shot tokenizer transfer. 2. The authors have non-trivial designs for tokenizer sampling and the hypernetwork architecture.

Weaknesses

1. Training the hypernetwork is time-consuming. However, as the authors mention, this is only a one-time cost. 2. The pretrained hypernetwork is specific to one particular LLM. As a result, this one-time cost needs to be paid for every LLM that wants to benefit from zero-shot tokenizer transfer. Practically, this may not be as efficient as simply retraining the embedding layer for each model.

Questions

How long does it take for a trained hypernetwork to predict the embedding of a typical off-the-shelf tokenizer?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Appendix E and H.

Summary

This paper presents a way to perform "zero-shot" construction of embeddings for a new target tokenizer. It proposes a hypernetwork based approach that learns to take in the embeddings of tokens generated by tokenizing the target token with the original tokenizer and produce the embedding of this token. The authors perform analysis on code and multilingual domains, and their method shows promise with respect to FOCUS, a recent embedding transfer method. Overall, the idea is novel, innovative, and seems promising. However, based on my understanding, I have significant concerns with the evaluation settings and some comparisons. I am willing to reconsider my assessment if my concerns on evaluations are adequately addressed.

Strengths

* The approach is interesting, and the idea of using a hypernetwork to compute new embeddings is innovative. * I can see this approach being useful as the tokenizers/vocabulary grows, since the number of parameters will stay constant. However I have some concerns on the ability to generalize (see weaknesses).

Weaknesses

* I do not think ZETT is truly ‘zero-shot’. While it is important to make a distinction between vocabulary V and tokenization function T, it is likely that a large portion of embeddings are agnostic of T and more reliant on V. For example, I would not expect the embedding of the token ‘car’ to be much different when an LM is trained on two different tokenization functions. * Therefore I think it is important to analyze the overlap of the distribution of the tokens. Appendix F analyzes the overlap with respect to the original vocabulary. But it is also important to study the overlap (both vocabulary and segmentations) between what UnigramLM generates during training versus the ‘target’ tokenizer during evaluation to really contextualize whether this approach is useful. * The above wouldn’t be as big of a problem if the hypernetwork data did not contain the same languages/domains as that it was trained on. But from what I see all the evaluated languages are seen during hypernetwork training. I assume it will be something similar for code too. * I think what would make more sense is to really evaluate on an ‘unseen’ language that uses the same script. Here one can expect a more pronounced distribution shift of both vocabulary and tokenization functions.

Questions

* I cannot seem to find the value that was set for the hyperparameter max token length ‘l’. For a sufficiently large ‘l’ I would assume there would be a significant overlap of the vocabulary produced by UnigramLM during training and the target tokenizer during evaluation, especially when the domain/language is the same. * When the target vocabulary size is increased, does the hypernetwork remain the same? If yes, I could see how some tokens might be zero shot transferred as the vocab size grows. However, without comparing the distributions as mentioned earlier, it’s hard to say what exactly the overlap is. * How does ZETT compare to FOCUS when extra training is done in the more challenging multilingual setup? Would ZETT be a more attractive choice in this case compared to FOCUS which does not require any prior hypernetwork training?

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

I believe the weaknesses outlined above need to be addressed, which are not discussed. There is no separate limitations section in the paper as recommended in the checklist guidelines.

Thank you for your response and addressing my questions.

Looking for ward to see new results on latest LLMs, and multimodal expansion in the future.