Image Inpainting via Iteratively Decoupled Probabilistic Modeling

Response to Reviewer rxeq

Thanks for your comments. We have carefully revised our paper according to your suggestions. 1. **In Eq. 1, does $\mathcal N$ mean the density function of the Gaussian distribution?** Correct, $\mathcal N(\cdot, \cdot)$ is the density function of Gaussian. 2. **In Eq. 1, what's the relationship between the $y$ being integrated and the $\boldsymbol y$ in the integral bounds? Are they the same or different?** They are different. The $\boldsymbol y$ in the integral bounds refers to pixel values of the ground-truth unmasked image. Meanwhile, the $y$ being integrated follows a Gaussian distribution with the estimated mean and variance. We appreciate your highlighting this distinction. For improved clarity, we have represented the integrated variable as $z$ in the revised manuscript. 3. **In Eq. 1, where does $\boldsymbol y^i$ come from during training? Is it the ground-truth unmasked image or the GAN-generated one? If it is the latter, it also depends on $\boldsymbol \theta$?** The $\boldsymbol y^i$ denotes the $i$-th pixel of the ground-truth unmasked image. We have explicitly clarified this in the revised manuscript. 4. **Above Eq. 9, there seems to be a typo: 'filed' -> 'field'.** Thanks for pointing this out. We have fixed this typo.

Response to Reviewer rxeq

Dear Reviewer rxeq, Thank you for your detailed review and the valuable feedback. We have carefully provided clarifications and experiments to your previous comments. We appreciate your thorough evaluation of our work and look forward to hearing from you and addressing any further questions or concerns you may have. Thank you for your continued engagement and support.

Response to Reviewer N7bD

1. **The training strategy is not well explained. Authors should clarify whether and how the iterations are taken into account during the training.** Thank you for your valuable advice. During training, our model undergoes the full pipeline with all iterations (we iterate twice for training efficiency, as stated in Sec.4.2), and the updating scheme between iterations is detailed in Sec.3.1.2. However, for testing, we may iterate for different numbers. As shown in Table 2, using more iterations results in higher FID performance. We have revised our manuscript to provide clearer illustrations of this point. 2. **There are some suspiciously accurate in-painting results in the appendix - particularly in Fig.N7 (Fig.M7 in the original paper corresponds to Fig.N7 in the revised version). Can authors explain how the proposed model can be so accurate? I can understand being realistic, however, the third column shows that the generation is pretty much the same. Similar inpainting results are given in Fig.N8. May there be a problem with training and test data leakage?** There is no problem with training and test data leakage. Following established methodologies like LaMa [a] and MAT[b], we utilize the official training (24183 images) and validation (2993 images) sets of CelebA-HQ. We have carefully checked that individuals present in the validation set do not overlap with the training data. Our model performs well when the input image contains sufficient visible pixel information, enabling high-quality generation while maintaining coherence with the existing content. This success originates from our model's pixel spreading mechanism, which initiates from visible pixels and progressively diffuses valuable information throughout the image. This approach is particularly effective for facial images characterized by strong inherent priors, such as symmetry and structural attributes. In the third example in Figure N.7, where the right half of the face is visible, our model reconstructs a comparatively realistic and consistent outcome using visible features like eyes and beard, albeit with potential discrepancies in details like ears and nose. However, when the entire face is masked, as in the first example in Figure N.7, our generated facial image notably diverges from ground truth. This phenomenon also appears in natural scenes; for instance, in the third example in Figure N.4, our method successfully restores the airplane structure due to the visibility of the frontal part, while other methods fail to inpaint such regions. We will release the training code and models. All results are reproducible. [a] Suvorov, Roman, et al. "Resolution-robust large mask inpainting with fourier convolutions." WACV. 2022. [b] Li, Wenbo, et al. "Mat: Mask-aware transformer for large hole image inpainting." CVPR. 2022.

Response to Reviewer EPEv

1. **What's the meaning of in Eq.1? Is it the ground truth?** The $y^i$ represents the $i$-th pixel of the ground-truth image. We have revised the paper to improve the clarity of this illustration. 2. **More explanation is necessary for Eq.5. What happens when $\boldsymbol m_t=1$ and $\boldsymbol m_0=0$ or vice versa?** In the mask $\boldsymbol m$, we denote known regions with values of 1 and missing regions as 0. In Eq.5, we assign pixel values in the missing regions to 0 while preserving the original pixel values in the initially known area. Subsequently, we progressively fill in the missing regions using sampled results based on newly estimated mean and variance values. It's important to note that the previously inpainted content in the missing regions is updated at each iteration to maintain consistency with the newly inpainted pixels. In cases when $\boldsymbol m_0=0$, the initially visible pixels are absent. We consider this as a generation task. However, in our inpainting mask, such a phenomenon does not occur since we do not set $\boldsymbol m_0=0$. Furthermore, we only set $\boldsymbol m_t=1$ during the final iteration to ensure training convergence. When $\boldsymbol m_0=1$, all pixels are regarded as valid. So we will not update anymore. 3. **Is the uncertainty map computed only in the mask region or the entire image?** The uncertainty map is computed for the entire image. As discussed in the second step (i.e., 'Pick'), the uncertainty score of each initially known pixel is set to 0. Additionally, for the pixels that have been inpainted up to the current time step, their uncertainty scores are computed by using the $\boldsymbol \sigma_t$ map. For the pixels located in regions that are still missing, their values are set as 1.

Response to Reviewer EPEv (Part 2)

4. **Why is $\boldsymbol \sigma_t$ in the range [0,1]?** We do not constrain the range of $\boldsymbol \sigma_t$ to [0,1]. Instead, we normalize $\boldsymbol \sigma_t$ to compute the uncertainty map $\boldsymbol u_t$, which ranges from [0,1]. In the last sampling step, we use the original $\boldsymbol \sigma_t$ values. The $\boldsymbol \sigma_t$ maps in Figure 2 and Figure 3 are normalized only for better visualization. We have clarified this point in the revised paper. 5. **Comparison to Controlnet is necessary.** Thank you for the constructive comments. In addition to ControlNet [a], we also include MI-GAN \[b\] (ICCV 2023) in our comparisons. Although ControlNet is not trained on the Places2 dataset, it undergoes training on a more extensive LAION dataset. Results from our proposed PSM show significant improvements across all metrics, highlighting the effectiveness of our method. We observe that ControlNet might produce undesired results due to its tendency to generate new objects that might not align harmoniously with the existing content. We have added the comparisons to Sec.I of the supplementary material due to the page limit. #### 512x512 Places2 under the small mask ratio setting | | | | | | | |---|---|---|---|---|---| | Method | FID$\downarrow$ | P-IDS$\uparrow$ | U-IDS$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | | **PSM (ours)** | **0.72** | **30.95** | **43.91** | **0.084** | **25.51** | | MIGAN | 1.40 | 18.43 | 39.35 | 0.103 | 24.38 | | ControlNet | 1.86 | 12.63 | 35.71 | 0.117 | 24.68 | | | | | | | | #### 512x512 Places2 under the large mask ratio setting | | | | | | | |---|---|---|---|---|---| | Method | FID$\downarrow$ | P-IDS$\uparrow$ | U-IDS$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | |**PSM (ours)**| **1.68** | **25.33** | **39.30** | **0.161** | **20.89** | |MIGAN| 3.81|13.50|32.42|0.195|20.00| |ControlNet|5.55|6.60|25.65|0.219|19.73| | | | | | | | [a] Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. "Adding conditional control to text-to-image diffusion models." ICCV. 2023. [b] Sargsyan, Andranik, et al. "MI-GAN: A Simple Baseline for Image Inpainting on Mobile Devices." ICCV. 2023. 6. **It seems that sometimes the model is hard to grasp some structure of the image. For example, in the last row of Fig.4, the result of Model A cannot reconstruct the structure of stairs.** We agree that our model sometimes hard to grasp some structures during generation. This is attributed to our pixel spread model, which functions by spreading informative pixels from initially visible regions across the entire image. In the last example depicted in Figure 4, despite some remaining stairs, the spreading process appears to be primarily influenced by the window representation due to its frequent appearance in the input. Consequently, our model effectively reconstructs the windows. However, even in the absence of stairs, our output remains reasonable. Indeed, it is very challenging to consistently generate all subtle structures with high fidelity, as the content of the free-form masked regions is typically complex and ambiguous. This challenging factor also deteriorates diffusion models during generation although they have shown great generation abilities, as shown in Figure 6. We will concentrate on improving the generation quality in our future work.

Response to Reviewer EPEv

Dear Reviewer EPEv, Thank you for your detailed review and the valuable feedback. We have carefully provided clarifications and experiments to your previous comments. We appreciate your thorough evaluation of our work and look forward to hearing from you and addressing any further questions or concerns you may have. Thank you for your continued engagement and support.

Response to Reviewer HZRy

1. **Please explain in detail how the uncertainty is obtained in each iteration.** We provide the details of obtaining the uncertainty in each iteration. Taking the $t$-th iteration as an example, we forward the masked image $\boldsymbol x_{t-1}$, binary mask $\boldsymbol m_{t-1}$ and uncertainty map $\boldsymbol u_{t-1}$ to the network to obtain the estimated mean $\boldsymbol \mu_t$ and variance $\boldsymbol \sigma_t^2$. To obtain a preliminary uncertainty map $\tilde {\boldsymbol u}_t$ in \[0,1\], we normalize the standard deviation map $\boldsymbol \sigma_t$ by subtracting its min value and dividing by absolute max-min value (the smaller $\boldsymbol \sigma_t$, the lower uncertainty). Note that the final sampling (Step 3 in Sec.3.1.2) is performed using the original $\boldsymbol \sigma_t$ without normalization. We then sort the uncertainty scores only for unknown pixels based on $\boldsymbol m_{t-1}$. According to the pre-defined mask schedule, we figure out the pixels that will be newly added in this iteration. Based on the preliminary uncertainty map $\tilde {\boldsymbol u}_t$, by marking pixel locations that are still missing as 1 and the initially known pixel locations (referring to $\boldsymbol m_0$) as 0, while keeping the $\tilde {\boldsymbol u}_t$ values of inpainted pixels unchanged, we obtain the final uncertainty map $\boldsymbol u_t$. 2. **The third paragraph in Section 3.1.1 is inconsistent with Figure 2. The input image $\boldsymbol x_t$ at time t should be $\boldsymbol x_{t-1}$ at time $t-1$, right?** Thanks for pointing this out. We have fixed this typo. It should be $\boldsymbol x_{t-1}$. 3. **Please discuss in detail the effect of the learnable function $\mathcal F$ in Eq.6.** The learnable function $\mathcal F$ introduces the proposed uncertainty-guided attention mechanism. In traditional attention based solely on feature similarity, all pixels have an equal chance to exchange information. However, in image inpainting, where missing pixels are initialized with specified identical values, conventional attention mechanisms are not effective in conveying valuable information from visible regions to missing regions. This phenomenon often yields unsatisfied pixels, resulting in blurry content and undesirable artifacts. By incorporating the uncertainty map as an input, the function $\mathcal F$ discerns between valid and invalid pixels. This enables our network to prioritize attention on valid pixels when filling in the holes. Removal of this design (i.e., 'model F' in Table 1) results in a performance drop compared to our full 'model A'. Additionally, the structures are sharper and clearer in the inpainted results via our uncertainty-guided attention in Figure 4.

Response to Reviewer HZRy (Part 2)

4. **The authors state several times that the proposed method improves efficiency. Please demonstrate in detail, e.g., the comparison of inference time.** We provide efficiency comparisons in Table F.5, where our model takes less time for inference. We also compare inference speed tested on the A100 GPU and inpainting quality in the table below where our model achieves favorable performance. | | | | | | | |---|:---:|:---:|:---:|:---:|:---:| |Method |**Ours**|Stable Diffusion|LDM|ControlNet|MAT| |FID$\downarrow$ | **1.68** | 2.11 | 2.76 | 5.55 | 2.90 | |Speed$\downarrow$ (512×512) |**0.25s**|3.6s|2.7s|3.1s|0.26s| | | | | | | | 5. **What is the mask size in Table 4?** The mask size is 512x512, consistent with the image sizes of Places2 and CelebA-HQ as indicated in Table 4. The small and large mask settings refer to different masking ratios. The mask statistics have been provided in Sec.B of Mat's [e] supplementary file. The small and large masks are established with an average masking ratio of approximately 20% and 50%, respectively. 6. **In the experiments, please add the comparisons with more recent methods, e.g., [a] Chu et al., Rethinking Fast Fourier Convolution in Image Inpainting, ICCV 2023. [b] Sargsyan et al., MI-GAN: A Simple Baseline for Image Inpainting on Mobile Devices, ICCV 2023. [c] Ko et al., Continuously Masked Transformer for Image Inpainting, ICCV 2023.** Thanks for the valuable comments. We have cited all the referred works. In practice, we find out that only MI-GAN [b] provides implementation code and models, so we compare our method to MI-GAN and ControlNet \[d\] (ICCV 2023 Best Paper Award) below. Note that ControlNet is trained on a more extensive LAION dataset rather than the Places2 dataset. The results obtained from our proposed PSM exhibit significant improvements across all metrics. We observe that ControlNet might produce undesired results due to its tendency to generate new objects that might not align harmoniously with the existing content. A detailed comparison is shown in Sec.I due to page limit. #### 512x512 Places2 under the small mask ratio setting | | | | | | | |---|---|---|---|---|---| | Method | FID$\downarrow$ | P-IDS$\uparrow$ | U-IDS$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | | **PSM (ours)** | **0.72** | **30.95** | **43.91** | **0.084** | **25.51** | | MIGAN | 1.40 | 18.43 | 39.35 | 0.103 | 24.38 | | ControlNet | 1.86 | 12.63 | 35.71 | 0.117 | 24.68 | | | | | | | | #### 512x512 Places2 under the large mask ratio setting | | | | | | | |---|---|---|---|---|---| | Method | FID$\downarrow$ | P-IDS$\uparrow$ | U-IDS$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | |**PSM (ours)**| **1.68** | **25.33** | **39.30** | **0.161** | **20.89** | |MIGAN| 3.81|13.50|32.42|0.195|20.00| |ControlNet|5.55|6.60|25.65|0.219|19.73| | | | | | | | [a] Chu, Tianyi, et al. "Rethinking Fast Fourier Convolution in Image Inpainting." ICCV. 2023. [b] Sargsyan, Andranik, et al. "MI-GAN: A Simple Baseline for Image Inpainting on Mobile Devices." ICCV. 2023. [c] Ko, Keunsoo, and Chang-Su Kim. "Continuously Masked Transformer for Image Inpainting." ICCV. 2023. [d] Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. "Adding conditional control to text-to-image diffusion models." ICCV. 2023. [e] Li, Wenbo, et al. "Mat: Mask-aware transformer for large hole image inpainting." CVPR. 2022.