Summary
This paper introduces the Semantic Pyramid AutoEncoder (SPAE) for enabling frozen Large Language Models (LLMs) to perform understanding and generation tasks involving non-linguistic modalities such as images or videos. SPAE converts raw pixels to lexical tokens extracted from the LLM's vocabulary, allowing the LLM to perform multimodal tasks. The proposed method is evaluated using in-context learning experiments with frozen PaLM 2 and GPT-3.5, demonstrating success in image understanding and generation tasks. The main contributions include the first successful method using a frozen language model for image content generation, a new SPAE tokenizer producing interpretable representations, a new progressive prompting method for long cross-modal sequences, and evaluation on visual understanding and generation tasks.
Strengths
1. Compared to LQAE, the method presented in this paper demonstrates superior performance in image reconstruction.
2. By interacting with large language models such as GPT-3.5 and PaLM-2, this paper effectively utilizes the in-context learning capabilities of these models to generate corresponding images, offering an intriguing research direction.
Weaknesses
1. This paper's experiments are limited to 128x128 resolution, significantly lower than the mainstream 256x256 clarity.
2. To achieve comparable performance with VQGAN, a larger latent space (# tokens) is necessary.
3. Is it possible to attain the same performance as VQGAN + frozen codebook + semantic guidance and an expanded latent space (# tokens)? The need for a pyramid structure comes into play here. Whether the pyramid structure is necessary requires experimental support.
4. The VAE proposed in this paper employs perceptual loss and utilizes a VGG network with supervised pre-training on ImageNet during the training process. In contrast, other methods like LQAE do not use perceptual loss. A strategy similar to ViT-VQGAN should be adopted here, using a version without perceptual loss for training on the comprehension task.
5. There is a lack of model variants in the ablation experiments on the understanding tasks.
6. In lines 203-204, it is mentioned that "We train with a batch size of 256 for 450k steps, which takes 1.4k TPUv3-hours." This training cost is larger than that of both VQGAN and RQ-VAE. It would be interesting to investigate the impact of the training cost on the performance of the VAE proposed in this paper.
Questions
Figure 8 in the appendix is intriguing, but the experiments are only conducted on the MNIST dataset. It would be compelling to see the experimental results on datasets like MS-COCO.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.