Thank you for your feedback.
### __Weakness 1__
> Although I am not familiar with object-centric learning, I believe that constructing multi-scale VAE features has already been applied in many generative tasks[1, 2]. Can the author tell me about the difference between MSF and other methods?
Please note that we have updated the related contents into Sect. "A.1 Extended Related Work" of our paper’s new version.
__Short answer__
- VAR [1] vs MSF: VAR auto-regresses from smaller scales of VAE representation to the larger, but with __no information fusion among scales__. In contrast, our MSF realizes fusion among different scales on VAE for the first time.
- SPAE [2] vs MSF: SPAE relies on multi-modal foundation model CLIP, while our MSF does not. SPAE element-wisely averages multiple scales into one, simply mixing different scales together, i.e., __no fusion among scales__. Our MSF augments any scaled representation with all other scales' high-quality representations, i.e., fusion among scales.
__Long answer__
[*__MSF__*] interpolates the input and produces multiple scales of VAE representation, which augment one another. Every scale integrates high-quality representations from the other scales, yielding better visual explanation (Fig. 3). Specifically,
(1) The input image $T$ is spatially interpolated into a pyramid {$T_n$}, from high to low resolutions, and after VAE encoding, we have multiple scales of VAE intermediate representation {$Z_n$};
(2) We project it into two copies, one for scale-invariant quantization using a shared codebook, yielding {$X_n^{si}$}, and the other for scale-variant quantization using different codebooks, yielding {$X_n^{sv}$};
(3) We conduct our unique and novel inter-scale fusion and intra-scale fusion upon {$X_n^{si}$} and {$X_n^{sv}$}, yielding the augmented {$X_n$}, where *__whichever scale is augmented by the other scales with their high-quality representation__*, i.e., fusion among the scales;
(4) The final {$X_n$} is used to guide OCL training.
[*__VAR__*] interpolates the single VAE intermediate representation and produces multiple scales of VAE representations, where there is no augmentation in between. Specifically,
(1) The input image $im$ is VAE encoded once, yielding one single intermediate representation $f$;
(2) The authors spatially interpolate $f$ into multiple scales and residually quantize them with a shared codebook, yielding a sequence of discrete representations $R=[r_1, r_2,... r_K]$, from low to high resolutions;
(3) *__There is no fusion among the scales__*;
(4) Finally the authors conduct auto-regression upon $R$ for image generation, i.e., $p(r_1, r_2, ... r_k) = \Pi p(r_k | r_1, r_2,... r_{k-1})$.
[*__SPAE__*] interpolates one VAE intermediate representation and yields multiple scales of VAE representations, but employs naive element-wise-average to integrate those scales into one. Specifically,
(1) The input $I$ is VAE encoded once, yielding one single intermediate representation $Z$;
(2) The authors conduct spatial interpolation on it and yield {$Z_l$}, and use pretrained multi-modal foundation model CLIP to residually quantize them under some prompt, producing a sequence of lexical tokens {$\hat{Z}_l$}, from low to high resolutions;
(3) The authors streaming average the first $l$ scales into one representation $\hat{Z}\_l$ – *__They have no fusion among the scales; Instead, they just mix all scales together__*.
(4) Finally the mixed representation is VAE decoded to produce the reconstruction or image generation.
__Summary__
Whether with VAE or not, it is not difficult to build multiple scales of representations; Outside of VAE, there are also many techniques to fuse multiple scales. But, __our MSF is the first to both build multiple scales upon VAE and realize fusion among multiple scales__, using unique and innovative designs based on codebook matching.
### __Reference__
[1] Tian et al. Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction. NeurIPS 2024.
[2] Yu et al. SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs. NeurIPS 2023.