MambaByte: Token-free Selective State Space Model

Summary

This paper proposes MambaByte, a byte-level language model without tokenizers. Compared with subword tokenization, byte-level language models are more robust and have fewer inductive biases. However, the resultant sequences are much longer and bring challenges in efficiency, specifically with Transformers. For efficient training, the authors build MambaByte upon Mamba architecture. For efficient decoding, a variant of speculative decoding is proposed. Experiments verified the efficiency and robustness of MambaByte.

Rating

8

Confidence

3

Ethics flag

1

Reasons to accept

1. MambaByte demonstrates promising potential towards tokenizer-free architectures. Looking forward to seeing MambaByte as a larger-scale model and operating for various downstream tasks. 2. Experiments are solid and comprehensive, comparing MambaByte to Transformers, SSMs, and MegaByte. Results show the superiority of MambaByte in both efficiency and robustness against both byte-level models and subword Transformers. 3. Notably, MambaByte shows impressive performance with longer sequences.

Reasons to reject

I do not find any clear reason to reject this paper.

Summary

This paper proposes MambaByte, a byte-level Mamba state space models and observed that Mamba efficiently addresses issues in byte-level language modeling without the need for specialized architectures like global patching. Experiments on language modeling tasks shows MambaByte demonstrates competitive performance both on language modeling performance and efficiency.

Rating

7

Confidence

2

Ethics flag

1

Reasons to accept

- The proposed token-free adaptation of Mamba, MambaByte, demonstrates competitive performance with subword Transformers and maintains the benefits of token-free language models - This paper also proposes an adaptation of speculative decoding for byte-level models (subword model for drafting, byte-level verification), which shows a 2.6× inference speedup - MambaByte shows great performance for length extrapolation

Reasons to reject

I have noticed that different scales of MambaByte are used in various comparisons. Could you provide a comprehensive comparison of MambaByte at different scales? Also, how does scaling affect different models, and will MambaByte demonstrate improved scaling factors?

Summary

Authors propose using state space models to learn a language model on byte sequences utilizing the fact these new family of models have an internal memory that is independent of the context window length. The authors frame the paper as an improvement of byte modeling with other byte models as the baseline. I believe the right baselines are the SOTA LMs regardless of tokenization. In this case, we benefit little by showing one byte model better than the other. We benefit more by understanding how much the gap remaining from the wordpiece/bpe models is.

Rating

5

Confidence

4

Ethics flag

1

Reasons to accept

(1) eliminating the last preprocessing/postprocessing step in language modeling has huge practical implications in simplifying systems and reducing sources of errors. (2) Pushing modeling to support small vocabulary models allow us to extend the lessons learned in language modeling to genomics and image modeling efficiently.

Reasons to reject

(1) The authors compare Mumba (Wordpiece) vs Mumba Byte in terms of equal params (table 3) but do not have a baseline of the wordpeice based models in terms of training compute match. I would like to see figure adjusted with the curve that correspond to Mumba word piece model to figure out if it is more efficient than MumbaByte (training wise). (2) Adaptive decoding seems a general technique that could be also used to speed up any language model model regardless of the tokenization. I would like to see the speedup achieved for MumbaByte vs Mumba both applying adaptive decoding. (3) The way that adaptive decoding still rely heavily on word-piece level models kind of negate the benefits that we are seeking by eliminating the tokenization. (4) Missing references to relevant papers in the field: (a) https://arxiv.org/pdf/1808.04444 (b) https://arxiv.org/abs/1908.10322

Questions to authors

(1) Please, add (Training/inference) isoFLOP analysis of MumbaByte vs Mumba models to prove the computational advantage of your approach.

Summary

The paper presents a token-free selective SSM called MambaByte, which is for improving both modeling performance and computational efficiency. In addition to building upon Mamba architecture, it integrates speculative decoding for subword drafting and byte-level verification. The paper conducts experiments from compute-matched and parameter-matched perspectives to demonstrate the effectiveness and efficiency of the proposed method. The main comparisons are made between MambaByte and Transformers, SSMs, and MegaByte architectures. The results show that MambaByte outperforms the compared byte-level models in several datasets.

Rating

8

Confidence

3

Ethics flag

1

Reasons to accept

1. The motivation is clear and easy to understand. The presentation and writing are of high quality. 2. The idea of using Mamba as the draft model and MambaByte as the verification model is interesting. 3. The experiments are quite extensive to show the superior performance of the proposed model. The parameter-matched and compute-matched evaluations are fair and able to provide more observations into the performance among different architectures.

Reasons to reject

I don't observe very obvious reasons for rejection. This is a solid paper in general.

Questions to authors

1. Mamba architecture has been questioned for how it can be applied to downstream tasks and how it performs on CV/NLP domains. I wonder if the authors have tried the proposed model on downstream tasks to compare the performance with traditional architectures.

Thanks for the response

Thanks to the authors for their response. After reading other reviews, I still hold positive opinions on the submission and maintain the same score. I think it might be influential for future directions.