Thanks for the review, and the strengths you point out.
Regarding novelty: first, we note the link to compressive autoencoders in the paper (Sec 1, 2) and do not claim novelty on the scalar quantizer itself. However, as we note in the general reply, our contribution lies in considering the implied product codebook of FSQ: “We’d like to emphasize that prior to this paper, no one was thinking of using finite scalar quantization (FSQ) as a drop-in for VQ in generative image models or for dense vision tasks. Our key insight is to consider the implicit product codebook created by scalar quantization if you bound each channel, which allows training large generative transformers like on VQ representations (i.e., by bounding each of the $d$ channels to $L$ levels, we get a product codebook of $L^d$ codewords). While it might seem obvious after reading our paper, a priori, it is very surprising that FSQ obtains such competitive results compared to VQ -- despite not using any of its auxiliary losses and tricks.”
Regarding comparison to VQ: we note that in Fig. 3, we see that FSQ scales well to high-bit codebooks. Arguably, this is where the future research lies, as small codebooks are becoming less and less important. We think that people focused on eg 1024 codewords, because VQ did not scale beyond that. Our contribution lies in showing that our simple approach can be highly competitive even though VQ is highly optimized and much more challenging to train.
Regarding “The method obviously lacks compression performance since the quantized vectors' indices are insufficient/irrelevant in this case.”: we note that in this work we do not target compression, which typically operates at much higher bitrates (note that our codebooks of e.g. 10 bits (1024 codewords) correspond to 0.03 bits per pixel without any entropy coding). One can increase the codebook size and eventually obtain a competitive compression method, similar to what was studied in the compression literature (see “Neural Compression” references in Section 2). To improve rate-distortion performance, one probably wants to leverage entropy coding to get an even more compressed bitstream. However, we note that we target generative downstream tasks, and we see in Fig 3d that FSQ can utilize the codebook better than VQ. This is desirable for the tasks we study, and results in better sampling FID (Fig 3b).
Q1) Thanks for this suggestion, this also made us curious. We calculated the representations for 3 images: 2 from class A, and 1 from class B. When calculating the MSE between the two images in class A in FSQ representation space, we obtain 1.5e5. When calculating the MSE between one image from A and the image from B, we obtain 1.4e5. For VQ, the same story emerges, we get 1.67e5 for in-class, and 1.53e5 for between-class. In both cases, see that images from the same class do not lie closer in the representation space. This is in line with what we already report in Section A.3., where we concluded that the representations of both FSQ and VQ mostly store medium level textures. Inspired by your comment, we added a few results to Section A.3:
- Fig 10, which shows the codewords used for two different imagenet classes when encoding 20 examples. As it turns out, the codewords seem to represent textures more than semantic concepts: both models use most codewords to represent very different classes. This explains that we do not see a close relationship when calculating MSE above.
- Fig 11, where we show what happens when adding offsets to the FSQ channels. Again, only low level texture details change.
Overall, both VQ and FSQ seem to store mostly medium level texture detail. Note that both approaches store 1 codeword per 16x16 pixel patch.
Q2) we call the codebook "implicit" because it is defined by the product of the per-channel codebook (eg for L=3, d=2, the per channel codebook is {-1, 0, 1}, and the implied codebook is given by {(-1, -1), (-1, 0), (-1, 1), …}). As you point out, this could be implemented by using VQ with a fixed codebook set to the hyper cube, but in our approach, we never have to construct this codebook: during encoding, the index into the implicit codebook can be calculated with a single dot product (see Sec. A.1), and we thus call it implicit.
Q3) This is likely because in the experiment we ran, the capacity we add is comparatively small, e.g., VQ codebooks for MaskGIT store 1024 * 256 parameters, which is <0.1% of the 227M parameters of the entire model (VAE + transformer). In general, it should help to add a substantial amount of parameters in exchange for more memory and compute requirements.