Autoregressive Image Generation without Vector Quantization

Summary

This paper deals with the task of autoregressive image generation using continuous tokenizers. AR image generation has primarily focused on using discrete tokens, and training discrete tokenizers are quite hard. In this paper, the authors train AR models on continuous tokenizers. The idea is to predict continuous valued latent vectors z using a transformer, and modeling the token distribution using a diffusion model conditioned on the latents z. The use of shallow networks for the diffusion helps the models enjoy the fast inference benefit of AR models. Extensive experiments show that diffusion loss can improve the performance of AR models on Imagenet benchmark using various tokenizers.

Strengths

- Use of diffusion loss for AR models is very interesting. The idea is simple, clean and neat. The paper is well written. - Experiments are extensively performed on different tokenizers. - Performance gains are solid on different settings.

Weaknesses

- One weakness with this approach is the slow inference speed of diffusion. The authors suggest that a small MLP is sufficient to model the distribution, so it is fast. But looking at the table 3, increasing the model size improves the performance. So, I wonder if in large-scale text-to-image benchmarks, it might be desirable to use large MLPs. If that is the case, the speed could be slower. - Another weakness is performing experiments only on Imagenet benchmark. As noted by authors, experiments on Imagenet always have noticeable artifacts. It is understandable that some academic labs might not have resources to run experiments on large-scale benchmarks, so I am not going to penalize for this. But, it would have been really nice to see experiments on large-scale text-to-image benchmarks.

Questions

- Can the authors comment on the design choice of using diffusion loss to model P(x|z)? What is the reason for using diffusion? Did you think about other ways of modeling the distribution. One approach which I can think of is to simply represent P(x|z) as a gaussian similar to VAE. Can such simple distributions suffice?

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

N/A

Summary

This paper proposes an autoregressive modeling method without the use of vector quantization tokens. By using a diffusion procedure to model the next-token probability, it is able to apply autoregressive models in a continuous-valued space. To model the probability of one token $x_i$ given the condition $X_{<i}$, the output feature of $X_{<i}$ is treated as a condition in the diffusion process of $x_i$. Afterwards, the authors propose to unify masked modeling and autoregressive modeling through a semi-autoregressive way. Comprehensive experiments show the efficacy of the proposed method.

Strengths

* The idea of integrating the diffusion model and autoregressive model is great. * The experiments are comprehensive and they seem to support the conclusion that the proposed method outperform the previous SOTA.

Weaknesses

Although the experimental results seem great, there exist several non-negligible weaknesses in this paper: * The motivation of this paper seems weird. The authors claim that using vector quantization is not a necessity for autoregressive modeling. However, the authors neither provide sufficient examples supporting "Conventional wisdom holds that autoregressive models for image generation are typically accompanied by vector-quantized tokens." as stated in the abstract, nor demonstrate the unnecessity of vector quantization. Since previous works found it work well, why should we discard it? * The proposed method does not match the motivation. The authors first claim that vector quantization is not necessary in autoregressive modeling. Then the authors propose to fomulate the objective with a diffusion process, which is also complicated and hard to converge. So it seems to me that the authors just replace a complicated method with another complicated method. * The authors claim that they propose the unification of autoregressive and masked generative models by predicting group by group instead of token by token and in raster order instead of in fixed order. However, these ideas have already been proposed in previous works. The pioneer work should be [1], which first proposed predicting tokens group by group. There are also works on masked image modeling exploring combining masked image modeling and autoregressive modeling [2, 3]. Specifically, [3] explored exactly the same thing in Section 3 as the authors do in the paper. So it seems that this unification is not a novel idea. * The proposed method still needs a discrete tokenizer, as shown in the experiment section. Are there any experiment results that do not rely on the discrete tokenizer? [1] Semi-Autoregressive Neural Machine Translation, EMNLP 2018. [2] Self-supervision through Random Segments with Autoregressive Coding (RandSAC), ICLR 2023. [3] Bridging Autoregressive and Masked Modeling for Enhanced Visual Representation Learning, https://openreview.net/forum?id=KUz8QXAgFV

Questions

Besides the above questions, there are one question concerning the implementation of the method: * How do the authors solve the problem of position ambiguity in predicting several tokens at a time in MAR pretraining? To be specific, if the standard ViT structure is used as stated in the paper, when predicting a group of tokens, how does the model know which position is being predicted? This problem has also been stated in [3, 4]. [3] Bridging Autoregressive and Masked Modeling for Enhanced Visual Representation Learning, https://openreview.net/forum?id=KUz8QXAgFV [4] XLNet: Generalized Autoregressive Pretraining for Language Understanding, https://arxiv.org/abs/1906.08237

Rating

3

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors have stated the limitations in Appendix A.

Summary

The paper introduces an autoregressive image generation method without vector quantisation for visual. The authors observe that while discrete-valued spaces facilitate representing categorical distributions, they are not necessary for autoregressive modeling. They propose modeling the *per-token* probability distribution using a diffusion process, allowing the application of autoregressive models in a continuous-valued space. This approach eliminates the need for discrete-valued tokenizers and has been experimentally validated across standard autoregressive models and generalized masked autoregressive variants.

Strengths

This work is articulated clearly, with well-founded theoretical support for its motivation. The proposed method effectively addresses the existing problems and offers significant inspiration for future research: * Clear Motivations: Under the AR, there is no intrinsic connection between the method of next token prediction and quantisation. Using discrete distributions, e.g., categorical or multinomial distributions, for image modeling is also counterintuitive. * Original Contributions: The work introduced the Diffusion loss with various types of AR to model the distribution of continuous tokens, thereby eliminating the need for discretisation. Although some existing literatures have used diffusion [1] or continuous token modeling [2], this work is the first to combine diffusion with AR for generative tasks with solid experiment results * Significant Benefits: This work takes a significant step forward in advancing continuous modeling. I personally believe image quantisation is a simplification and compromise to facilitate AR-based generative models, as modeling discrete distributions is much easier. However, quantisation inevitably leads to information loss. The proposed solution effectively mitigates this potential loss. Additionally, continuous representation is more suitable for modeling objective phenomena governed by physical laws, such as images or videos. [1] Li, Yazhe, Jorg Bornschein, and Ting Chen. "Denoising Autoregressive Representation Learning." arXiv preprint arXiv:2403.05196 (2024). [2] Tschannen, Michael, Cian Eastwood, and Fabian Mentzer. "GIVT: Generative infinite-vocabulary transformers." arXiv preprint arXiv:2312.02116 (2023).

Weaknesses

### Concerns Overall, this work is good, but the theoretical analysis is somewhat insufficient. One concern is that, although quantisation is avoided during generation, the entire process still relies on an encoder/decoder. Can we establish theoretical assumptions to prove that modeling with diffusion loss has lower information loss, or that its upper bound on loss is lower than the version with quantization (although Table 2 has experimentally validated this assumption)? Another concern is after replaced Cross entropy, can we obtain an explicit prior distribution or posterior for the whole AR system? Sec 3.1 discussed about CE in traditional AR system is for Cat distribution, but I feel I cannot find similar discussions for Diffusion in the following work. The next question is can we model the pixel space directly instead an encoder? Is this limited by the computation cost? ### Open Questions An open question: quantisation is not necessary any more for AR, but is the AR still necessary with diffusion loss? Seems image tokens (whether quantised or not) differ significantly from language attributes. The prior distribution of image tokens might be uniform thus current AR might still face the same drawbacks? Another open question is that can we include a decoder in end-to-end training and escape the limitations of the tokenizer? ### Suggestions For readers, some key links appear to be missing in the reasoning chain, e.g.: Why we use Diffusion Loss with noisy input instead of clean image with MSE for continuous modelling? I think one potential explanation is that the objective of MSE is actually minimizing the divergence between Gaussian and token and it differs significantly from the empirical token distribution.

Questions

Please see weaknesses. My Open Questions and Suggestions will not affect my final rating.

Rating

8

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The author has stated limitations and social impact.

Summary

This work challenges the conventional belief that autoregressive (AR) models implemented by a Transformer are best suited for modeling discrete sequences. Instead, it proposes modeling the per-token probability distribution using a diffusion procedure, enabling the application of AR models in a continuous-valued space. By defining a Diffusion Loss function instead of using categorical cross-entropy loss, this approach eliminates the need for discrete-valued tokenizers. Consequently, this framework can directly model unquantized continuous-valued tokens, avoiding information loss and degraded reconstruction performance associated with discretized tokens. Therefore, it is expected to provide better generation quality with the same compression ratio compared to AR models with discrete sequences. Experiments on class-conditional image generation demonstrate that this framework achieves strong results while benefiting from the speed advantage of sequence modeling.

Strengths

#1. Presentation and story-telling. In my personal experience, while trying to improve VQ-GAN in various ways, I always believed that sticking to discrete tokens was necessary to train AR models. This work has a very compelling motivation to challenge this conventional belief, which many researchers and developers in this field presumably hold. I appreciate how the manuscript effectively describes the motivation and explains why quantization is not essential for training AR models. #2. This work proposes a novel and interesting idea. This generalizable framework has full potential and could serve as a fundamental building block in multi-modal LMM, effectively handling multi-modal data naturally, rather than relying on adaptor-based approaches. Previous attempts required continuous signals to be vector-quantized before training an autoregressive model with a Transformer architecture. However, this framework overcomes those limitations, advancing the field in various ways. #3. Simplicity of implementation. The architecture does not contain any specialized components or complex diffusion loss, which increases my confidence in being able to reproduce the results. Furthermore, all important experimental details are thoroughly described in the manuscript.

Weaknesses

The main weakness of this work is the limited scope of experiments, as the manuscript only presents ImageNet 256x256 class-conditional experiments. However, the authors acknowledge this limitation in the Appendix. Additionally, I am confident that the benefits of diffusion loss validated by ImageNet will be generalizable to other datasets and tasks, including text-to-image generation. Therefore, this weakness does not outweigh the strengths of this work.

Questions

#1. In the manuscript, the authors mention that adding 64 CLS tokens at the start of the sequence helps improve the stability and capacity of our encoding when training masked generative models with bidirectional attention. Could you elaborate on why this improves stability? Is it because it sets the minimum sequence length to 64? #2. "Vector-quantized tokenizers are difficult to train and are sensitive to gradient approximation strategies." Can we say this is difficult to train? While dealing with a large codebook size may require special treatment, such as restarting dead codes, training VQ-VAE or VQ-GAN with a moderate codebook size may not be difficult once one is familiar with the framework. This suggests that there is no inherent art to it. #3. Does "16 denotes the tokenizer strides in line 219" refer to the downsampling factor? #4. Why is the gap between CrossEnt and Diff Loss in Table 1 smaller for the AR model compared to the gap for the MAR model? #5. Experiments involving tokenizers with mismatched strides are intriguing. However, I didn't understand why we are considering using these tokenizers, given that KL-8 does not outperform KL-16 despite having the same training and inference costs. #6. Could you elaborate on why AR with random order improves fidelity compared to AR with a raster scan order?

Rating

8

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations of this work are thoroughly discussed in the Appendix. No societal negative impacts have been observed.

Summary

This paper introduces a novel approach to image generation using autoregressive models with continuous-valued tokens, challenging the conventional use of discrete vector-quantized tokens. The authors propose a "Diffusion Loss" function that models per-token probability distributions using a diffusion process, eliminating the need for vector quantization. They demonstrate the effectiveness of this approach across various autoregressive models, including standard and masked variants. The paper also unifies autoregressive and masked generative models under a generalized framework, showing that bidirectional attention can perform autoregression. The authors implement their method using a small denoising MLP and evaluate it on ImageNet, achieving state-of-the-art results for token-based image generation. They demonstrate the flexibility of Diffusion Loss with various tokenizers and explore its properties, such as temperature control and sampling steps. The approach shows favorable speed-accuracy trade-offs compared to existing methods like Diffusion Transformers. Overall, this work opens up new possibilities for autoregressive modeling in continuous-valued domains, potentially impacting various applications beyond image generation.

Strengths

- The paper introduces a novel approach of using continuous-valued tokens in autoregressive models for image generation, challenging the conventional wisdom of discrete vector-quantized tokens. The proposed Diffusion Loss is an innovative way to model per-token probability distributions, bridging the gap between autoregressive and diffusion models. The unification of autoregressive and masked generative models under a generalized framework is a great contribution to the field. - The empirical results are robust and comprehensive, demonstrating consistent improvements across various model variants and tokenizers. - The ablation studies and analyses of different components (e.g., denoising MLP, sampling steps, temperature) are thorough and insightful. - The speed-accuracy trade-off analysis provides a practical perspective on the method's performance. - The paper is well-structured and logically organized, guiding the reader through the concept, implementation, and results. The use of figures (especially Figures 2 and 3) effectively illustrates complex concepts like bidirectional attention for autoregression and the generalized autoregressive framework.

Weaknesses

- While the paper demonstrates good speed-accuracy tradeoffs, I'm concerned about the overall computational complexity, especially during training. The diffusion process adds significant overhead, and training for 400 epochs seems quite intensive. How does the training time and compute requirements compare to other state-of-the-art methods? - While the paper explores different model sizes, a more systematic study of how performance scales with model size (similar to studies in language models) could provide valuable insights into the method's potential for further improvements. Usually diffusion models can be more parameter efficient compared to autoregressive models, I'm curious to see if it's still the case for the autoregressive diffusion modeling which is a hybrid between the two. - For the interpretability of the continuous-valued tokens or the learned representations. An analysis of what these tokens capture compared to discrete tokens could provide valuable insights into why the method works so well.

Questions

- Can authors elaborate more on the rationale behind the idea of using small NLP for denoising conditioning on latent code produced by the transformer? Like comparing to diffusion training without conditioning on latent $z$ ? - Have you experimented with the same model architecture but applied directly to pixels or image patches, rather than using image tokenizers? How significant is the use of more contextualized tokenized codes compared to working with pixels directly? - Is there a theoretical limit to how much continuous-valued tokens can improve performance over discrete ones? Are there any situations where discrete tokens might actually be better? The paper seems to assume continuous tokens are always superior, but is there any proof of this, or are there any counterexamples where this might not be true? - Regarding artifact handling, do you think the issues are more related to the diffusion modeling process, or are they inherent limitations of the image tokenizers themselves?

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

3

Limitations

Yes.