Response to Reviewer dUar
> We believe that the author should explain in the introduction and method sections whether the method proposed is training or non-training. … I cannot fully understand the details of the method proposed in this paper (e.g., how were the weights in the PC obtained)
Thank you for pointing out aspects of the paper that were not explained clearly in the main text. The PC used in the proposed inpainting algorithm needs to be first trained (unconditionally) on the datasets (e.g., CelebA-HQ, ImageNet, and LSUN-Bedroom) using the MLE objective: $\max \sum_{\mathbf{x} \in \mathcal{D}} \log p_{n} (\mathbf{x})$, where $p_{n}$ is the PC and $\mathcal{D}$ is the dataset.
Following prior art, a natural way to optimize PC parameters w.r.t. the MLE objective is by the Expectation-Maximization algorithm since PCs can be viewed as a latent variable model with a hierarchically nested latent space. The implementation of the EM algorithm is very similar to the backpropagation algorithm since the feedforward computation following Definition 1 can be treated as a computational graph. To update the parameters, we first run the standard backpropagation algorithm to compute the gradient of all parameters (w.r.t. the MLE objective, which is the output of the feedforward PC computation). Suppose the gradient of $\theta_{n,c}$ is $g_{n,c}$, for every sum node $n$, we update all its child parameters by
$$\theta_{n,c} \leftarrow (1 - \alpha) \cdot \theta_{n,c} + \alpha \cdot \frac{g_{n,c} \cdot \theta_{n,c}}{\sum_{m \in \mathtt{in}(n)} g_{n,m} \cdot \theta_{n,m}},$$
where $\alpha$ is the step size. In addition to the EM update, we follow prior arts and use the LVD technique to initialize the parameters. The full details of the learning process of PCs are included in Appendix C of the revised paper. At the end of Section 4.1, we also added a paragraph to explain the PC learning pipeline and points to the Appendix for more details.
We are happy to make further clarifications if there are other unclear parts in the paper.
> LPIPS alone may not be sufficient to assess the quality of generated images.
We conduct a user study based on all three datasets and two masks (“Expand1” and the newly added “Wide” masks adopted from [1,2]). Detailed settings and results are added in the updated paper (Appendix E.1). We copy the result table from the paper:
| Dataset & Mask | Tiramisu | CoPaint | RePaint | DPS |
| CelebAHQ & Expand1 | Reference | 22 | 34 | 14 |
| CelebAHQ & Wide | Reference | 26 | 30 | 22 |
| ImageNet & Expand1 | Reference | 32 | 20 | 24 |
| ImageNet & Wide | Reference | 14 | 6 | 12 |
| LSUN & Expand1 | Reference | 18 | 2 | 8 |
| LSUN & Wide | Reference | 4 | 6 | -6 |
We use the vote difference (%) metric, which is the percentage of votes to Tiramisu subtracted by that of the baseline. A positive vote difference value means images generated by Tiramisu are preferred compared to the baselines, while a negative value suggests that the baseline is better than Tiramisu.
We compare against the three strongest baselines (CoPaint, RePaint, and DPS) as their average LPIPS scores are not significantly worse than Tiramisu. As shown in the table, in all but one task the vote difference score is positive (and often quite large), which indicates the superiority of Tiramisu compared to the baselines.
We also tried the metrics suggested by the reviewer. However, FID is not specific to image pairs and is less informative for image inpainting tasks; when using the official U-IDS metric, we found that in most cases we got a U-IDS close to 0 and the results are not very informative.