General Reply
We thank all reviewers for taking the time to review our manuscript and the encouraging feedback from the reviews. As both reviewer pUdr and n7kc ask about ablation experiments on the autoencoder, we present them here in a joint answer.
**Autoencoder Ablation:**
To assess the effect of (1) larger batch size and (2) using an exponential moving average (EMA) of the weights, we conduct two experiments, where we train the autoencoder (from scratch) on with (a) a total batch size of 8 and (b) a total batch size of 256. Since we only had limited resources and time available for this rebuttal, these models are not trained until convergence. For both experiments, we evaluate the reconstruction performance on a fixed 10k subset of the COCO2014 validation set, comparing EMA and non-EMA weights for both models. We report rFID, PSNR, SSIM, and LPIPS in a table below.
We see, that
1. In all our evaluations the EMA variants give a consistent improvement compared to the non-EMA variant. This effect is amplified after more update steps (compare A1 and A2)
2. To evaluate the effect of the large batch size we first compare the models when they are both trained for the same number of gradient updates (A1(e) and B(e)): In this case, the large batch size provides a clear benefit in terms of rFID: 7.34 vs 8.85, while the other metrics are relatively close. However, when we evaluate the models after they have trained for roughly the same wall time (500k (B) and 1.9M (A2) update steps), the larger number of updates is a more important factor, with A2e outperforming Be in all metrics.
We believe that training until convergence might change this outcome, since we can see from the B/A1 evaluation that the large batch size provide a better update direction for the weights. This will likely help in finding a loss-minimum.
From this experiment, we conclude that EMA-tracking has a significant effect on our performance, but we believe that the choice of the trade-off between batch size and update steps requires more thorough investigation in a future work.
We thank the reviewers for pointing out this missing piece, and we hope to provide more on this for the research community in the future. For the time we have updated the paper with the concern and added this whole experiment to the appendix.
```jsx
| Model Name | Batch Size | EMA | Global Steps | rFID [↓] | PSNR [↑] | SSIM [↑] | LPIPS [↓] |
| ----------- | ------------ | ----- | -------------- | --------- | ---------- | --------- | ----------- |
| B | 256 | ✗ | 500k | 7.52 | 24.30 | 0.71 | 1.27 |
| Be | 256 | ✓ | 500k | 7.35 | 24.33 | 0.72 | 1.24 |
| A1 | 8 | ✗ | 500k | 9.14 | 24.42 | 0.72 | 1.32 |
| A1e | 8 | ✓ | 500k | 8.85 | 24.51 | 0.72 | 1.29 |
| A2 | 8 | ✗ | 1.9M | 6.28 | 25.05 | 0.74 | 1.08 |
| A2e | 8 | ✓ | 1.9M | 5.57 | 25.05 | 0.74 | 1.07 |
```