Response to Reviewer xgEg (Part 1)
Thank you for the time, thorough comments, and nice suggestions. We are pleased that you acknowledged the novel idea, technical soundness, and effective experiments.
> **Q1**: Some unclear explanations and descriptions.
**A1**: Thanks for your detailed review, here are our clarifications.
> **Q1.a**: It is unclear why these two types of methods (GAN or MAE) need to run multiple times.
**A1.a**: (1) For previous methods, in their training stage, they outpaint images from the given central image 128x128 to 192x192, and the training settings are fixed. As a result, there are 2.25 ($\frac{128}{192}^2$) times more contents are generated.
(2) However, If we want to generate 57.6 ($\approx(2.25)^{5}$) times more contents, we have to forward the pretrained generative network 5 times ($1$->$f(x,)$->$2.25$-$f(x,)$->$(2.25)^{2}$->resize->$f(x,)$->$(2.25)^{3}$->resize)->$f(x,)$->$(2.25)^{4}$->resize->$f(x,)$->$(2.25)^{5}$).
(3) The reason for categorizing the existing methods into GAN-based and MAE-based is because GAN-based methods are difficult to converge and MAE-based methods are relatively more efficient due to the dropping of masked patches.
> **Q1.b**: The authors fail to clearly explain why previous methods cannot outpaint images with arbitrary and continuous multiples.
**A1.b**: **Arbitrary and continuous:** For previous methods, they train their model by inputting the central sub-image, and the outpainting multiples (e.g., 1.5x) are pre-fixed in the training stage. Then, in the sampling stage, if we want to outpaint the image 1.7x, they have to retrain their model again. However, our PQDiff is much more flexible. Once PQDiff finishes training, PQDiff can generate images with any multiples.
> **Q1.c**: Why relative positional queries can reduce the number of diffusion steps?
**A1.c**: The relative positional queries do not reduce the number of diffusions **timesteps**, but reduce the number of **iterations** for large multiple settings. As described in A1.a, previous methods have to forward the network 5 times to obtain 57.6x times more content. In our methods, for any outpainting multiples $n$, we only forward the network one times $1$->$f(x)$->$(2.25)^{n}$. Note that $f(x)$ includes all the diffusion steps.
> **Q1.d**: In section 3, the authors say that they set h1 = h2 = h and w1 = w2 = w for notation simplicity. It looks improper.
**A1.d**: Thanks for your nice suggestions. Actually, the regions of the target view are larger than the regions of the anchor view. $h1 = h2 = h$ and $w1 = w2 = w$ are because we use the resize operation to conveniently feed the two views into transformers. We have corrected this expression in our new version.
> **Q1.e**: The mismatch of the training and sampling in Figure 2 and Eq.1.
>
**A1.e**: There are some misunderstandings in the training stage. Actually, in the training stage, we feed the $z_{a}$, noised $z_{b_{t}}$ and relative positional embeddings as input. Please note $z_{b_{t}}$ is relatively equivalent to $x_{t}$ in the original diffusion model, which can be calculated by diffusion forward process (see line 9 in Alg.1 in Appendix B).
Then, **for the sampling stage**, the inputs are the given sub-image (anchor image) $z_{a}$ , a randomly initialized Gaussian noise, and relative positional embeddings.
For better understanding, we have modified the Fig.2. Please check our new version.
> **Q1.f**: In Eq.2, it seems Xb0 should be Zb0 In Eq.2, it seems Xb0 should be Zb0
**A1.f**: Thanks for your detailed review. We have corrected the typo in our new version.
> **Q2**: How does this one-to-many mapping influence the model?
**A2**: We are sorry for the confusion of the figure, and actually, the anchor-to-target is a one-to-one mapping. Since the relative position is one of the **core parts** of our method, we hope our clarification could help your understanding:
Instead of using image-level relative position, we will record the top-left positions for **each patch** of the target view, which means that **the target view is determined**. Take an example:
in the training stage, we take the anchor view as the starting point, suppose the positions of the anchor patches are:
$$\begin{pmatrix}
(0,0)&(0,1)&(0,2)\\\\
(1,0)&(1,1)&(1,2)\\\\
(2,0)&(2,1)&(2,2)\\\\
\end{pmatrix},$$ and the relative positions of the target patches are:
$$\begin{pmatrix}
(-1.6,1.7)&(-1.6,3.4)&(-1.6,5.1)\\\\
(1.4,1.7)&(1.4,3.4)&(1.4,5.1)\\\\
(4.4,1.7)&(4.4,3.4)&(4.4,5.1)\\\\
\end{pmatrix}.$$ From the two positions, we know the top-left **shift bias** of the anchor and target view is $[-1.6-0, 1.7-0]$, the **shift scaling** of the anchor and target view is $[(1.4-(-1.6))/(1-0), (3.4-1.7)/(1-0)]$. Then, the position of the target view is certainly determined.
We also illustrate the details of how to calculate relative positional (both formulation and illustration) in **Appendix C** in our new version.