Explanation of why encoding as training objectives fail
Thank you for your response. We found that augmenting the training loss often failed to produce effective counterfactuals, either due to limitations in the model architecture or because of the model's natural training loss.
After many attempts, we discovered that guided diffusion was a straightforward algorithm capable of achieving good performance. Our eventual choice of algorithm was primarily influenced by the available generative models for images (VAEs, GANs, Diffusion), and we elaborate on the challenges associated with each below.
We initially tried VAEs for image repair but found the generated counterfactuals were often blurry, which is a known limitation of VAE [1,2]. Although later work describes methods for high-resolution VAE-based image generation [3,4], these methods often fell short in producing the same level of sharpness and detail compared to GANs and diffusion models—especially within the anomalous regions. So why are VAEs commonly used for reconstruction-based anomaly detection despite the blurry outputs? This is likely because although their ELBO loss tends to favor images that are the "average" of a distribution (i.e., blurry), this often suffices for detecting anomalous regions.
Given the limitations of VAEs, we turned to GANs and diffusion models. However, GAN training was plagued by well-known issues of instability and non-convergence [5,6], thereby leading us to focus on diffusion models.
With diffusion models, we found that incorporating the four properties into the training loss did not enhance counterfactual quality. This was likely because the diffusion model's noise prediction loss often led to static-like reconstructions (very noisy images) that were unsuitable for evaluation against our properties, especially at larger time steps. It was with guided diffusion methods [7] that we achieved high-quality counterfactuals, and these methods now form the foundation of our present methodology.
We thank the reviewer for these questions, and we will update our manuscript to include a discussion of the lessons learned with various model architectures.
**Additional References**
[1] Tomczak, Jakub, and Max Welling. "VAE with a VampPrior." In International conference on artificial intelligence and statistics, pp. 1214-1223. PMLR, 2018.
[2] Dai, Bin, and David Wipf. "Diagnosing and enhancing VAE models." arXiv preprint arXiv:1903.05789 (2019).
[3] Razavi, Ali, Aaron Van den Oord, and Oriol Vinyals. "Generating diverse high-fidelity images with vq-vae-2." Advances in neural information processing systems 32 (2019).
[4] Liu, Zhi-Song, Wan-Chi Siu, and Yui-Lam Chan. "Photo-realistic image super-resolution via variational autoencoders." IEEE Transactions on Circuits and Systems for video Technology 31, no. 4 (2020): 1351-1365.
[5] Saxena, Divya, and Jiannong Cao. "Generative adversarial networks (GANs) challenges, solutions, and future directions." ACM Computing Surveys (CSUR) 54, no. 3 (2021): 1-42.
[6] Lu, Yuzhen, Dong Chen, Ebenezer Olaniyi, and Yanbo Huang. "Generative adversarial networks (GANs) for image augmentation in agriculture: A systematic review." Computers and Electronics in Agriculture 200 (2022): 107208.
[7] Dhariwal, Prafulla, and Alexander Nichol. "Diffusion models beat gans on image synthesis." Advances in neural information processing systems 34 (2021): 8780-8794.