Summary
Experimental study that trains and evaluates a decoder-only Transformer model directly on JPEG byte streams (= JPEG-compressed images). Training is done autoregressively as usual. Evaluation tasks are (i) predict JPEG image quality setting and class of example, (ii) detect/locate/fix single-byte errors, (iii) data generation. The key takeaway is that this works reasonably well.
Strengths
S1. The exploration of LLMs to directly handle compressed data is interesting. Even if it may not turn out to be of notable relevance in practice, it may help to shed more insights into the limitations of LLM.
S2. The evaluation tasks are reasonable first steps and shed some light into the compression/decompression capabilities of plain LLMs.
Weaknesses
W1. Paper positioning not convincing. The paper is motivated by a large argument that directly working on compressed format is beneficial. These arguments, however, are inherently flawed. First, arguments about ubiquity, compactness and generality are invalid because if one simply decompresses before training the LLM, these advantages would still hold. Second, while I do see the worth of studying compressed file formats (see S1), I fail to see practical relevance. On the one hand, real-world machine learning pipelines do consist of many domain-specific techniques; e.g., data augmentation (e.g., crop/scale images), helpful tokenization (e.g., SentencePiece), task-specific training objectives (e.g., BERT training) or models (e.g., CNNs). On the other hand, spending resources to "teach" a model to decompress/compress when we actually know how to do this more efficiently (JPEG encoder/decoder) is a waste of resources.
W2. Training and evaluation setup not convincing. For task (i), the prediction targets of image quality and class are fed into the training process in a somewhat contrived way to deal with problems of decoder-only models for this task. It's not clear why a decoder-only LLM is the right approach in the first place. For task (ii), the authors make "erroneous bytes" are less likely than "correct bytes" arguments. But when doing so, they ignore the entire input after the erroneous token (for localization/correction). This, again, a consequence of using decoder-only models. For task (iii), the automatic check is solely on file validity, but ignores the quality of the generated samples (other than the anecdotal examples of Fig. 3).
W3. Limited insight. This is for two main reasons. First, the papers make broad claims about compressed file formats, but then only considers JPEG, includes JPEG-specific information into the training pipeline (quality setting), and use only one image size. Second, the paper puts too much focus on whether tasks (i)-(iii) work reasonably well with an out-of-the-box LLM training pipeline. What's much more interesting, however, is exploring where such approaches would fail and why.