Additional questions and comments to the reviewer based on their review
Dear reviewer,
We want to thank you for your review and comments. We've updated our manuscript based on some of your suggestions, but would also kindly request some clarification of some of your comments.
1. The reviewer mentions that PCA assumes a multivariate Gaussian distribution. To the best of our knowledge, this is not true, but it rather assumes just linear data. See for example https://arxiv.org/abs/1404.1100. Some confusion may arise from the Probabilistic PCA context, where a linear-Gaussian is assumed over the latent variables, not the observed variables. In fact, in a linear-Gaussian framework, in which the marginal and conditional distributions are Gaussian (see also Pattern Recognition and Machine Learning by Bishop, section 12.2). Could the reviewer clarify whether our interpretation is now correct?
2. The reviewer states that our reasoning is flawed due to not using the variance information in the PCA model. Does the reviewer mean that when applying for example probabilistic PCA, a likelihood over the latent variables can be calculated? If so, we would like to stress that PCA is only used as a scaffold in our proof for autoencoders. We furthermore understand that there may be better ways than just using the reconstruction loss, but the reconstruction loss is still applied widely in practice.
3. The reviewer states that "Other autoencoders either output variance directly (such as VAEs) or allow for the estimation of variance and covariance through multiple examples.". Indeed, some methods such as VAEs or PCA can explicitly model the covariance of the data. If I understand the reviewer correctly, they refer to for example the use of Hotelling's T2 in outlier detection in the PCA space, to the explicit constraints put on the the latent space in VAEs. Is that correct?
4. The reviewer then states: "The unreliability of reconstruction error for anomaly detection, as discussed by the authors, arises from disregarding the variance information provided by PCA, not from the PCA process itself." We respectfully disagree here. While the variance information can help in some cases, it certainly does not guarantee outlier detection in the way we state is our objective. For example in "Do Deep Generative Models Know What They Don't Know?" by Nalisnick, it is made clear that VAEs, even while allowing for anomaly detection using the likelihood, still suffer from the inability to detect obvious anomalies. This is further solidified in for example "Autoencoding under normalization constraints by Yoon et al (ICML 2021) "
5. The reviewer further mentions "Anomaly Detection with Robust Deep Autoencoders" by Archambeau et al (2007). I was unable to find the referenced paper, but I believe the reviewer might refer to "Anomaly Detection with Robust Deep Autoencoders by Zhou and Paffenroth (KDD 2017)". However, this paper to my understanding does not use the latent space for anomaly detection, but rather aims to minimize the influence of anomalies in the training of the autoencoder, much like robust PCA, by splitting the data into normal and anomalous data. Interestingly, the results of this paper showcase the failure of normal autoencoders, further solidifying the need for analysis of this phenomenon. Could the reviewer comment on whether the suggested paper is indeed the one they meant to suggest?
7. The reviewer mentions that "Many anomaly detection studies employ autoencoders and model the latent space with Gaussian assumptions without encountering the issues highlighted in this paper. Examples include: ... " However, of these studies, the first and third do not perform any anomaly detection at all. Furthermore, we specifically study the setting where an autoencoder is used in conjunction with just the reconstruction loss, a practice which is common throughout industry and academia, as is outlined in the introduction of our manuscript. Could the reviewer elaborate in more detail on the relevance of these studies on our manuscript?
We've also further clarified the goal of our work, in which we agree with the reviewer, namely: using an autoencoder with the reconstruction loss is insufficient to guarantee reliable anomaly detection. We prove and show how autoencoders can fail, and thereby aim to fill the theoretical gap in our understanding of autoencoder methods when used for anomaly detection.