SpaceByte: Towards Deleting Tokenization from Large Language Modeling

Summary

The authors propose a byte-level architecture called SpaceByte that involves local blocks (lower dimension, windowed attention) and global blocks (higher dimension, global attention) where the global blocks are between chunks of local blocks and only selectively applied. The global blocks are applied to "spacelike" characters (such as an actual space character), with the intuition being that at such points predicting the next character (like start of word) would be harder. The authors run extensive experiments to show that their proposed architecture outperforms previous byte-level architectures and in fact performs similarly to subword-level transformer baselines.

Strengths

**Originality**: While this work seems to share much in common with past byte-level architecture work, the intuitive idea of some bytes being harder to predict (in a rule predictable way) is very natural and useful (and first presented in this work, as far as I am aware). The architecture itself is also novel. **Quality**: The authors run a lot of experiments to demonstrate the capabilities of their architecture. **Clarity**: The paper was clear, down-to-earth, and easy to understand. **Significance**: This paper shows that byte-level architectures can be competitive with subword tokenization transformers in a flop-matched setting.

Weaknesses

* As a main weakness I'd say that even though this method beats other byte-level approaches and is comparable to subword tokenization, I don't see a compelling reason to use it over standard tokenization. In the Introduction the authors mention several downsides of tokenization, but don't have results that focus on these issues per se. I'm also a bit confused about the "additional modeling complexity (line 20)" point. It seems like tokenization is actually simpler. * It would have been nice to see some ablations on the architecture (for example, the local -> global -> local layer ordering seems a little arbitrary). * It would have been nice to see some modifications of the global rule. For example, only "space" characters like actual space and newline vs the more broad definition of "space" used in the paper. * It might be nice to include more info on what the "spacelike" tokens actually look like (what are the most common "spacelike" tokens and what percentage of all "spacelike" tokens do these make up).

Questions

Please see weaknesses for my questions.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

Yes. The paper does a good job of this.

Summary

The paper introduces SpaceByte, a byte-level Transformer model that incorporates larger transformer blocks at specific byte boundaries to enhance performance in language modeling tasks. While the approach bears similarities to previous work (e.g. MegaByte), it demonstrates improved performance over traditional tokenization models. The study focuses on specific languages, acknowledging limitations in generalizability, particularly in languages like Chinese, which do not use spaces between words.

Strengths

1. The paper presents a novel approach in utilizing byte-level Transformer models with specific block insertions to enhance language modeling performance. 2. The study provides insights into the limitations of tokenization models and the potential benefits of byte-level architectures. 3. The experimental methodology is well-documented, controlling for compute costs and providing detailed training details in the appendices.

Weaknesses

1. The novelty of the SpaceByte approach may be limited by similarities to previous models. 2. The study's focus on specific languages, without detailed experiments on broader language datasets, limits the generalizability of the findings. 3. The paper could benefit from a more extensive discussion on the unique contributions of SpaceByte compared to existing models.

Questions

1. How do the authors plan to address the limitations in generalizability to languages beyond the ones studied in the current work? 2. Are there plans to conduct experiments on a more diverse set of language datasets to further validate the effectiveness of SpaceByte in different linguistic contexts?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have acknowledged the limitations of their work, particularly in terms of language generalizability. To enhance the impact of the study, it would be beneficial for the authors to consider conducting experiments on a broader range of language datasets to strengthen the validity and applicability of SpaceByte across various linguistic domains.

Summary

Byte-level modeling allows transformers to circumvent subword tokenization, thus avoiding the many weaknesses introduced by tokenization. However those models are not very performant compared to subword-based transformers. This paper proposes a novel architecture named *SpaceByte*. The idea is to add an extra wide layer (i.e. "global block") between regular transformer layers (i.e. "local block") if and only if "the byte does not encode a letter, number, or UTF-8 continuation byte". Results show that when normalizing for training FLOPs, SpaceByte achieves the best PPL on multiple datasets. Plus, at different model dimensions and number of layers (?), SpaceByte always achieve the optimal or near-optimal Pareto optimality between PPL and inference FLOPs-per-byte.

Strengths

1. The idea is simple and well-motivated 2. Experiment results show that it is competitive when compared subword-based models

Weaknesses

1. While normalizing for FLOPs, I think there is some potential that the proposal will make the transformer blocks significantly harder to batch (due to the input-dependent dynamic structure of the model), hence the clock-time of the inference might actually be slower with the same FLOPs. It would be good to see some discussions on that. 2. There is no ablations as to whether the global blocks actually need larger dimension. 3. Presentation of the paper could be improved. See my question/suggestion in the next section.

Questions

### Presentation Suggestions 1. It's worth explaining more about how context lengths work differently for global vs. local blocks in Section 2. I'm also not very sure why the larger context length is necessary. 2. I'd suggest switching Section 2 and 3 and talk about related work first. 3. Three suggestions for Figure 3 -- a. make it clear in the caption that you are changing inference FLOPs with different model dimensions & layers (per line 206); b. use dashed/solid lines rather than thin/thick lines; c. mark the pareto frontier for each subfigures. ### Questions 1. Is the FLOP limits you imposed enough to make the models converge adequately on those datasets? It might have been the case that your model simply converges faster, but if trained longer, doesn't work as well as, e.g. sub-word models (which weakens your case). 2. Why set the dimension and context length equal? 3. I'm very confused what's the difference between section 6 and 5. How does the results in section 6 further complements the one in section 5? 4. In Table 2, you only have one context size, but don't global blocks have larger context size? What context size is being reported then?

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

See weakness point 1. I'm relatively confident that this could be a potential limitation that needs to be addressed.

Summary

This paper proposes SpaceByte, a byte-level decoder architecture for language modeling. As opposed to comparable models such as MegaByte, SpaceByte applies global transformer blocks after space-like characters, not after patches of a fixed size. The authors show that this approach leads to a substantially improved performance: compared to several other byte-level decoder architectures (including MegaByte), SpaceByte is the only one that matches or even exceeds the performance of subword-level models trained with the same compute budget.

Strengths

The proposed SpaceByte architecture is novel. The experimental setup is rigorous --- I think the authors did a great job in (a) evaluating a range of different architectures and (b) ensuring a fair comparison by controlling compute costs and using bits-per-byte as the evaluation measure. The results show clear performance improvements for SpaceByte, highlighting its advantages compared to other byte-level decoder architectures. Overall, I liked the paper very much and think that it should be accepted.

Weaknesses

The main weakness that I see is that the authors only evaluate the different architectures using bits-per-byte, not any downstream task. To become a real alternative to subword-level models in practice, it would be important to show that the similar language modeling performance translates to a similar downstream task performance. While the authors cite work indicating that this might be the case (Huang et al., 2024), without actual experiments it is unclear whether this holds for the examined architectures as well.

Questions

Is there any specific reason you did not include evaluations on downstream tasks?

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

Yes, the authors discuss limitations as part of the conclusion.

Post-rebuttal Comment

Thank you for the very detailed rebuttal. It clarified my main confusions during my first read of the paper. Re: Question 1 -- By "adequately" I mean train until convergence, but your budget argument is valid and I will take it. I still have reservations on the practicality of the proposed method (esp. batching), but otherwise I think the paper should be accepted. I'm improving my final evaluation to "weak accept". Since the authors also agree that batching is challenging with this proposal, please make sure you make some space to address this issue/limitation in the final draft of the paper.