Summary
A new auto-regressive decoder-only Transformer variant for language modeling is presented which operates well on byte level. Similar to the Vision Transformer, the main idea is to operate on patches instead of single tokens (bytes). Those patches are composed of a fixed number of $P$ tokens. Then, a big global Transformer operates on those patches, i.e. on the spatial dimension $T / P$. After that, a smaller local Transformer operates on the spatial dimension $P$ inside a patch.
Actual values for $P$ are quite small, e.g. 8 for language modeling.
The model is tested on text for classic language modeling, but also on image data and audio data, where PPL on byte-level is measured in all cases. The performance on language modeling compared to other BPE-level models is worse. When comparing to other models on byte-level, it is always better or at least as good.
Strengths
- Novel model which performs well on byte-level
- Many experiments, including ablations.
Weaknesses
- The argumentation on sub-quadratic computation runtime for self-attention is flawed and not true. See details below.
- Proposed model can also be used on BPE-level. How would this perform? Maybe even with this model, BPE would be prefered? This experiment is missing.
Questions
Really sub-quadratic? How? From the figure 1 and 2 it just looks like reduced by constant factor P or thus P^2?
"MegaByte decomposes long sequences into two shorter sequences, and optimal patch sizes reduces the self-attention cost to $O(N^{\frac{4}{3}})$" - I don't understand this. Why two shorter sequences? Where do I see that in the model? What is meant by optimal patch size? $P$ is the patch size? As I understand it, $P$ is a fixed hyper parameter, which you cannot change later on, as it is the dimension for the input.
"By setting $P = T^{\frac{1}{3}}$ - this does not make sense. $T$ is a variable length, depending on the input. $P$ is a fixed hyperparameter, which you can set once and then not change anymore. Or if you want to say, you also never change $T$ in practice, then the whole cost reduces to $O(1)$, but it's not really meaningful to think of $T$ as a fixed size.
Line 136: "the Global model has a sequence of length $\frac{P}{T}$ - I assume it should be $\frac{T}{P}$.
Line 136: "Local model uses $\frac{P}{T}$ sequences" - I assume it should be $\frac{T}{P}$ as well?
"Extensive experiments show that MEGABYTE allows byte-level models to perform competitively with subword models on long context language modeling" - but not on standard language modeling tasks?
$E^{\textrm{global-pad}} \in \mathbb{R}^{P \times D_G}$ should probably be $E^{\textrm{global-pad}} \in \mathbb{R}^{P \cdot D_G}$ instead?
"Section ?? proposes strided inference as a solution" - cross reference broken.
From the NeurIPS template: "The table number and title always appear before the table". However, this is wrong in the paper, where the table caption is incorrectly below the table.
Figure 4, Figure 7: I think that are tables, not figures?
Table 1 caption should describe, what are those numbers for the models? PPL?
In Table 1, Stories dataset, for PerceiverAR, the number looks inconsistent to all other numbers in that it only has 2 digits after the decimal point instead of 3. Is this on purpose?
Image modeling: It is not stated, but the model operates on the raw bytes, and you have 4 bytes per pixel? This should be explained.
The proposed model can be used to also operate on BPE level instead of byte level. While this is against the motivation of the work, I think this is still an interesting experiment. And also an important experiment. Because I would expect that this still improves over the byte-level variant. And this implies that BPE tokens are still overall better, and it is probably better to use BPEs also for this model, just like it is better to use BPEs for the other models.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.