Thank you for your response! We provide answers below:
Q1. 512 latency
256 (latent size: $32 \times 32$)
| Method | Time | Params |
| --- | --- | --- |
| DiMSUM-L/2 | 2.20s | 460M |
| DiT-L/2 | 3.80s | 458M |
512 (latent size: $64 \times 64$)
| Method | Time | Params |
| --- | --- | --- |
| DiMSUM-L/2 | 2.86s | 461M |
| DiT-L/2 | 4.78s | 459M |
*Note: Previously, we measured our models's latency on two resolutions using the same device and GPU_ID (a single NVIDIA A100). However, for the result of our model upon 512x512, due to resource matter (occupied device), we used a different NVIDIA A100 (different device/GPU_ID), which could cause unfairness. We remesured using the same device/GPU_ID (same specs) for a fairer comparison.
We appreciate your suggestion and have conducted additional benchmarks for 512x512 resolution images, following the same methodology used for 256x256 images in Table 2 of the rebuttal file. Our benchmarking process involves first warming up the model with 10 forward passes, then generating a single image (batch size = 1) 100 times, and finally calculating the average latency from these 100 runs.
Note that the parameter change after changing image size is mainly due to the PatchEmbed layer of the architecture, which both models have.
Q2. MEM and GFLOPs
256 (latent size: $32 \times 32$)
| Method | MEM | GFlops |
| --- | --- | --- |
| DiMSUM-L/2 | 2.42G | 84.49 |
| DiT-L/2 | 2.30G | 80.74 |
512 (latent size: $64 \times 64$)
| Method | MEM | GFlops |
| --- | --- | --- |
| DiMSUM-L/2 | 2.46G | 337.48 |
| DiT-L/2 | 2.34G | 361.14 |
Given our time constraints, we focused our comparison on DiT-L/2, maintaining consistency with our latency comparison. The results, as shown in the table, reveal that DiMSUM-L/2's memory usage is slightly higher than its counterpart. This increase is expected, considering DiMSUM's slightly larger parameter count.
Regarding GFlops, we acknowledge that for 256x256 images, DiMSUM produces about 4% more GFlops than DiT. However, an interesting trend emerges when we examine 512x512 images: DiMSUM's GFlops scaling is actually slower than DiT's. Consequently, at this higher resolution, DiT's GFlops exceed DiMSUM's by approximately 7%. This observation again aligns with the question 1 of reviewer **RLU7.**
This observation aligns with the known quadratic complexity of transformers as sequence length increases. Our hybrid model mitigates this issue; the impact of attention blocks is reduced, while Mamba demonstrates its linear scaling complexity as the token count grows. This architectural choice allows DiMSUM to maintain efficiency at higher resolutions, offsetting the initial GFlops difference at 256 resolution.
Q3-Q4.
Thank you for asking the questions.
In table 2, we use dopri5 to sample image from both models. We hypothesis that our architecture converges to less ODE-curvature solution compared to DiT architecture [1], [2]. This means that we could use less NFE to achieve the better image quality compared to DiT architecture. It is noted that when using dopri5, each sampling process could require different NFEs depending on the initial noise.
Additionally, we acknowledge that our method requires more preprocessing before each MambaBlock, which may introduce latency to the inference speed. Furthermore, as shown in Table 2b, adding more Wavelet level processing costs only 0.01 GFLOPs. We also confirm that all preprocessing components are included in our measurements.
[1] Pooladian, Aram-Alexandre, et al. "Multisample flow matching: Straightening flows with minibatch couplings." *arXiv preprint arXiv:2304.14772* (2023).
[2] Lee, Sangyun, Beomsu Kim, and Jong Chul Ye. "Minimizing trajectory curvature of ode-based generative models." *International Conference on Machine Learning*. PMLR, 2023.
Q5.
To our knowledge, our method achieves state-of-the-art performance on several datasets, including CelebA and LSUN-Church. It's worth noting that while MDT and MaskDIT focused their benchmarks solely on ImageNet, our evaluation is more comprehensive, comparing against both transformer-based models and Mamba-based diffusion models such as DIFFUSSM and ZigMa.
Regarding ImageNet, we acknowledge that our initial claim may cause confusions. We will revise our statement to more accurately reflect our findings: "Our method demonstrates superior results compared to DiT and DIFFUSSM, achieving faster training convergence and delivering high-quality outputs."
It's important to highlight that both MaskDiT and MDT report their results using significantly larger models (-XL/2, approximately 675M parameters), while our results are based on a smaller model (-L/2, 460M parameters). Despite this size difference, our model achieves competitive performance. In fact, DiMSUM-L/2-G attains a slightly better FID score (2.11) than MaskDiT-G (2.28).
We acknowledge your concerns as they are critical for the paper’s clarity and improvement. We will revise them in our manuscript.