Summary
The paper proposes a multi-resolution network for diffusion model which emphasizes the learning features across resolutions. The method further introduces a time-dependent layer norm to boost the model performance with fewer parameters compared to AdaLN in DiT. This proposed network demonstrates a SoTA performance on ImageNet at 1.70 and 2.89 on ImageNet 256 & 512, respectively.
Strengths
- The paper addresses image distortion caused by varying patch sizes by integrating a multi-resolution structure into the network. The choice of patch sizes involves a trade-off between computational complexity and model performance: larger patch sizes reduce computational complexity but degrade performance.
- The method finds a balance between compute complexity and performance by introducing a multi-branch architecture and multi-resolution loss to decompose the learning process from low to high resolution. The idea is a substantial contribution. Through quantitative and qualitative results, the method can alleviate distortion of image generation and achieve SoTA FID scores.
- The time-dependent layer norm is a simplified form of AdaLN with less intensive parameters by removing the cumbersome MLP layer and rearranging class embedding and time embedding for parameter-efficient time conditioning.
Weaknesses
- The discussion between the method and cascaded methods like "Cascaded diffusion models" and "Matryoshka diffusion models" should be included to highlight the advantages of the method.
- In table 1, it needs to be updated with some latest SoTa methods like MDTv2 with the best FID of 1.58 and PaGoDA with the best FID of 1.56 on ImageNet 256x256. Not to compete with them, but to get an overview picture of the latest methods.
- The proposed multi-scale diffusion loss is also introduced in SimpleDiffusion. So, the authors need to mention the difference in the paper.
- Equation 2: have the authors used concat instead of adding the upsampling features directly to the larger-resolution features of the next branch?
- The current method only uses the output features of the previous branch and injects them into the start of the next branch. However, this design lacks interconnections between blocks from the low-resolution and high-resolution branches. Ideally, there should be skip-connections across branches. What is the motivation behind this? If authors have not tested this, it is encouraged to do so.
- In table 2, what is the reason why using AdaLN-Zero with Multi-branch (row 3) causes a bad result?
- The root problem of distortion boils down to the use of patch size which is the main motivation of the work. However, in the design of multiscale network, the authors also patchify the input image to the corresponding resolution for each branch. I wonder why the author do not use different-resolution noisy images as the input to each branch like Matryoshka Diffusion Models. I think this is more effective to completely remove distortion problem. As shown in Figure 5, the method still exhibits a certain level of distortion rates.
- Sampling speed: it is valid to include a comparison of sampling time with baselines.
- Figure 3: What is the red line meaning?
Misc: L37, L130: the figure 7 of DiT paper is incorrectly linked.
Ref:
- Gao, Shanghua, et al. "MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer." arXiv preprint arXiv:2303.14389 (2023).
- Kim, Dongjun, et al. "PaGoDA: Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teacher." arXiv preprint arXiv:2405.14822 (2024).
- Hoogeboom, Emiel, Jonathan Heek, and Tim Salimans. "simple diffusion: End-to-end diffusion for high resolution images." International Conference on Machine Learning. PMLR, 2023.
Limitations
As pointed out in Weaknesses, the method still has a certain distortion rate which is due to the use of strided convolution (a form of patchification) at the start of each branch. I think the author should include this in the revised manuscript.