An Image is Worth 32 Tokens for Reconstruction and Generation

Recent advancements in generative models have highlighted the crucial role of image tokenization in the efficient synthesis of high-resolution images. Tokenization, which transforms images into latent representations, reduces computational demands compared to directly processing pixels and enhances the effectiveness and efficiency of the generation process. Prior methods, such as VQGAN, typically utilize 2D latent grids with fixed downsampling factors. However, these 2D tokenizations face challenges in managing the inherent redundancies present in images, where adjacent regions frequently display similarities. To overcome this issue, we introduce Transformer-based 1-Dimensional Tokenizer (TiTok), an innovative approach that tokenizes images into 1D latent sequences. TiTok provides a more compact latent representation, yielding substantially more efficient and effective representations than conventional techniques. For example, a 256 x 256 x 3 image can be reduced to just 32 discrete tokens, a significant reduction from the 256 or 1024 tokens obtained by prior methods. Despite its compact nature, TiTok achieves competitive performance to state-of-the-art approaches. Specifically, using the same generator framework, TiTok attains 1.97 gFID, outperforming MaskGIT baseline significantly by 4.21 at ImageNet 256 x 256 benchmark. The advantages of TiTok become even more significant when it comes to higher resolution. At ImageNet 512 x 512 benchmark, TiTok not only outperforms state-of-the-art diffusion model DiT-XL/2 (gFID 2.74 vs. 3.04), but also reduces the image tokens by 64x, leading to 410x faster generation process. Our best-performing variant can significantly surpasses DiT-XL/2 (gFID 2.13 vs. 3.04) while still generating high-quality samples 74x faster.

Paper

Similar papers

Peer review

Reviewer HiP78/10 · confidence 4/52024-06-26

Summary

This paper proposes a novel way to tokenize images to benefit image reconstruction and generation. This paper argues that the convention of compressing images into 2D latent spaces in the VQVAE/VQGAN setting limits the VQ model’s ability to fully exploit the redundancies present in images. During encoding, a fixed number of latent tokens are concatenated with the image patches. A vision transformer (ViT) is used to extract the representation from image patches to the latent representation. During decoding, the latent tokens are concatenated with a set of masked tokens. Another ViT is used to reconstruct image patches from the latent representation. However, instead of training encoder and decoder end2end, this paper first trains the model with the discrete codes generated by an off-the-shelf MaskGIT-VQGAN model, and then only fine-tunes the decoder with RGB pixels. During image generation, MaskGIT is used to generate the latent tokens. The model is tested on 256x256 ImageNet for image reconstruction, generation and classification.

Strengths

1. The proposed method is simple yet effective. 2. Scaling up the tokenizer seems able to achieve even more compact image representations. 3. The experiments on image classification is very interesting. As the size of the latent representation decreases, the tokenizer increasingly learns semantically rich representations.

Weaknesses

1. More efforts should be put in writing Two-Stage Training. It is not clear how exactly the "warmup" works. Does it train the whole encoder-decoder but with frozen codebook used in VQGAN? Or it means the model is trained by the quantized value of VQGAN instead of by the RGB pixels, i.e., the input and output of the model are the quantized value of VQGAN? 2. In the ablation test of 2D variant of TiTok-B64, how many tokens are used in this 2D representation? What if we use the same number of tokens for 1D and 2D variants? Will the performance be similar?

Questions

How important is the off-the-shelf codebook? If the MaskGIT-VQGAN codebook is replaced by some other codebook, let's say FSQ, how would the performance look like? Can we use a continuous autoencoder instead of VQGAN as the outside compressor?

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

The limitation and potential negative societal impact have been discussed in the appendix.

Reviewer Mn4Y6/10 · confidence 4/52024-07-08

Summary

This paper introduces TiTok (Transformer-based 1-Dimensional Tokenizer) that can tokenize images as compact 1D sequences instead of 2D latent grids. Accompanied with bidirectional non-autoregressive image generator, TiTok achieves SOTA performance on imagenet 256x256 benchmark with much faster generation process. The key contributions are: 1) A 1D tokenization method that can represent images using significantly fewer tokens compared with traditional 2D approaches without comprising performance (as few as 32 tokens); 2) Scaling up the size of tokenizer model helps to learn more compact and semantic-rich latent representations; 3) Significant speed-up for both training and inference with competitive performance compared with 2D tokenizer.

Strengths

1) The idea of using 1D representation instead of the conventional 2D is novel. It shows as few as 32 tokens are able to reconstruct an image well. The significant reduction of token numbers makes the training and inference much more efficient. Furthermore, scaling model sizes enables more compact representation, which provides insights into scaling laws for image tokenization. 2) The experiments are comprehensive, including different model sizes and token numbers. The ablation of different design choices are well designed. The evaluations on both image reconstruction and generation show superior performance compared with MaskGIT and VQGAN. 3) The paper is well rewritten, and easy to follow. The codes and models are publicly available.

Weaknesses

A major concern is the two-stage training method. From Table 3 (c), the gap with and wo proxy codes is big, 1.7 vs 5.1 for rFID, and 195 vs 120 for IS. However, the proxy codes come from MaskGIT. The authors emphasized that due to the missing of a strong training recipe, the sing-stage method is lagging behind. Thus, it seems the training method is the most important one compared with architecture designs. BTW, recently there is an open-source implementation for MAGVIT2 including training codes: https://github.com/TencentARC/Open-MAGVIT2 It will be great if the authors can adopt their training method and make the single-stage training work.

Questions

Another related work is Seed, which also proposes a 1D tokenizer with semantic meaningful tokens (32 tokens). It's better to cite it: Ge, Yuying, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. "Planting a seed of vision in large language model." arXiv preprint arXiv:2307.08041 (2023).

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

No negative societal impact

Reviewer Pxnc4/10 · confidence 5/52024-07-15

Summary

Background: VQ-GAN with 2D grid of latent tokens and fixed downsampling factors. This paper proposes to use 1D tokens instead of 2D tokens. Key ideas: * Redundancies: adjacent regions are similar. 2D grid explicitly couples the latents and the pixels in the same relative coordinates. * 1D tokens are enough for discriminative tasks (classification, etc.). Maybe they are enough for generative tasks too. Architecture: * Transformer-based encoder (tokenizer) and decoder (de-tokenizer) Training: * Warm-up stage: train the 1D VQ model with the proxy codes (= discrete codes generated by an off-the-shelf MaskGIT-VQGAN model) * Decoder fine-tuning stage: finetune the decoder with frozen encoder and quantizer

Strengths

Technical advantages: * More freedom in designing the architecture of 1D tokenizer * More semantic-rich image embedding // -> definition of semantic-rich? grounding experiment? * Improvement in FID * Faster (70x ~ 410x) generation Thorough analyses: * Scaling experiment -> 32 tokens are sufficient for reconstruction. * Larger tokenizer enables smaller number of latent tokens. * Smaller number of latent tokens have clearer semantics than larger one, regarding linear probing.

Weaknesses

W1. There is no analysis of 1D tokens. What are the advantages of 1D tokens that match the drawbacks of 2D tokens mentioned in the introduction section? What are the effects of masking a subset of 1D tokens compared to 2D tokens? The paper emphasizes the 1D latent sequence but does not analyze whether the observed characteristics (scaling, compact latent, etc.) are due to this 1D nature. A comparison with methods that further reduce the size of 2D tokens is needed. W2. For reconstruction results, it is necessary to compare using MSE, PSNR, and SSIM. FID is not sufficient to evaluate “reconstruction.” Why do we need rFID instead of typical reconstruction metrics such as PSNR, SSIM, and LPIPS? W3. The explanation of decoder fine-tuning is too sparse. Although not a novel contribution of this paper, it plays a significant role in performance improvement as shown in the ablation study, requiring more detailed explanation beyond just the “VQGAN training recipe.” Misc. Sentences could be easier to read. - e.g., L60 ... the ViT decoder (is utilized to reconstruct -> reconstructs) the input images ... - I recommend reading "Style, the basics of clarity and grace".

Questions

Q1. What are the advantages of 1D compared to 2D, and what are the potential disadvantages? Please provide a brief explanation. Q2. Is TiTok compatible with diffusion models? What is the reason for choosing MaskGIT over diffusion models or VAR (Scalable Image Generation via Next-Scale Prediction)? Appendix mentions computational burden. Is MaskGIT cheaper than the others? Q3. Could you discuss the relationship between TiTok and "Vision transformers need registers"? https://arxiv.org/abs/2309.16588

Rating

4

Confidence

5

Soundness

1

Presentation

3

Contribution

2

Limitations

Yes in the appendix sections.

Reviewer YNgD6/10 · confidence 5/52024-07-21

Summary

This paper introduces TiTok, a 1D image tokenization method that can represent images using significantly fewer tokens compared to existing 2D approaches. The key contributions are: 1. A 1D tokenization scheme that breaks the fixed grid constraints of 2D tokenizers, allowing more flexible and compact image representations. 2. A dual-stage training strategy using proxy codes to improve tokenizer performance. 3. Extensive experiments demonstrating state-of-the-art performance on ImageNet generation benchmarks while using 8-64x fewer tokens and achieving 74-410x faster inference compared to leading diffusion models.

Strengths

1. The conclusion is shocking. 32 tokens are enough to construct a picture. Although I know that the information in the picture is very redundant, this conclusion still shocks me. 2. The effect is still good.

Weaknesses

1. Compared with 2D token, the main advantage of 1D image token (in my opinion) is not the efficiency and training data. Although 2D token is cumbersome, it supports the expansion of 2D dimension and has many other advantages. In applications, for higher resolution images and better effects, 2D token is still the mainstream and is difficult to be shaken by 1D token. The impact of 1D token on image generation may not be its main application. The most important value of 1D image token for me lies in its possibility of combining with multimodal technology. Due to its information compression ability and two-dimensional encoding method, 2D token is difficult to combine with language model to form a multimodal language model. And the encoder of the current multimodal language model has no decoder available. 1D token may be able to support multimodal language model and allow language model to directly output images. This is very potential. But this article does not have corresponding thinking, which is a pity. 2. Although the image formed by 32 tokens is shocking, there are still many questions to be answered. Compared with the case of more tokens, can the information encoded by each token be estimated? Can we know what specific information is forgotten in the process of token compression? Can 1D tokens encode 2D relationships? How do 1D tokens understand translation and rotation? Is 1D token similar to the concept of high-level dictionary? These important questions are not studied in depth in this article. 3. There are too few pictures shown in this article to make further judgments.

Questions

I have raised many question in weaknesses.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

yes

Reviewer YNgD2024-08-07

Post-rebuttal review

I have read the author response and the review from other reviewers. I will raise my score to "weak accept". Thanks.

Reviewer HiP72024-08-10

Thanks for the clarification. Since most reviewers have the confusion of the two-stage training, apparently this part is very poorly written. I strongly suggest the authors to revise the manuscript.

Authorsrebuttal2024-08-13

Thanks for Your Review and Support

Thank you so much for the valuable suggestions and for considering our responses! We will revise and refine the confusing parts in the two-stage training as discussed in the rebuttal as suggested. If you need any further information or clarification, please feel free to contact us!

Reviewer Mn4Y2024-08-12

Thanks for providing the detailed rebuttal, especially running the single-stage training experiment by borrowing the training recipe from open-magvit2. I will raise my score.

Reviewer Pxnc2024-08-14

I thank the authors for the rebuttal. Sorry for coming in late. I am lowering my rating to BR because this paper is * We did this and that. It results in this and that. * *We do not know why. It works.* The core argument of this paper is using 1D tokens instead of 2D tokens. However, the paper does not provide the principle behind the advantage of 1D tokens over 2D tokens. 2D tokens are capable of representing anything which can be represented by 1D tokens because 2D tokens have more operational freedom. I value the analysis showing various behavior. But, the advantage of 1D tokens is supported by only the behavior, not the principle or theory. In short, I think using 1D tokens is worth publishing. But the content should be largely revised not to mislead the principle. (I am open to discussion during reviewer-ac discussion period.)

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC