Nominality Score Conditioned Time Series Anomaly Detection by Point/Sequential Reconstruction

Summary

TSAD techniques are targeted towards detecting either point or contextual anomalies, but often struggle to adequately capture both simultaneously. In this work, the authors propose a novel reconstruction-based AD technique that introduces the notion of a nominality score and a subsequent induced anomaly score. The method achieves a better trade-off between detecting point and contextual anomalies than competing techniques, and outperforms them over a variety of benchmark multivariate datasets.

Strengths

- The problem statement is interesting and relevant, and I appreciate the use of visualisations such as in Figure 1, that convey the difficulty of the task. - I appreciated that the model has several components that could be tweaked, such as the choice of soft or hard gate function, and the ablation study does a good job in highlighting their individual contributions. I would however have liked to see more discussion or synthetic examples that demonstrate whether there are certain properties of the monitored data that can guide the selection (beyond empirical comparison).

Weaknesses

- I found the presentation and clarity to be quite poor overall, as a result of which the contributions are sometimes difficult to follow properly. Section 2.2 introduces a lot of similar notations that are difficult to keep track of – a figure could be very helpful here for conveying the differences between the various time series and deviations. - The authors rely on results reported in earlier papers as performance measures for several competing techniques. While I appreciate that it may be time-consuming to re-implement and re-run all experiments, I do worry about potential inconsistencies in the experimental set-up that may involuntarily skew the comparison. The authors mention how if the results from the original paper are unavailable, *“we search for the highest reported values among other publications.”* I can see this as introducing several inconsistencies, especially since the source of the result isn’t reported alongside the performance figures presented in the table. - I would have liked there to be a dedicated *Related Work* section in the main paper in order to better understand how the work fits alongside other reconstruction-based techniques.

Questions

I would appreciate if the authors could respond to the concerns listed in the *Weaknesses* section, and either defend their approach, or indicate how they can improve on the existing submission. Although I find the problem investigated in the paper to be interesting and well-motivated, I don’t think that the submission is ready for publication in its current state, and believe it would strongly benefit from another revision/round of reviews.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

1 poor

Contribution

2 fair

Limitations

N/A

Summary

This paper proposes an unsupervised time series anomaly detection algorithm called NPSR (Nominality score conditioned TSAD by Point/Sequential Reconstruction), which combines both point-based and sequence-based reconstruction models. Specifically, it proposes a nominality score which is the ratio of a contextual deviation (or in-distribution deviation) to the total deviation (which is assumed to be the sum of the in-distribution deviation and the out-of-distribution deviation). The contextual deviation and the in-distribution deviation is computed by using the sequence-based and point-based reconstruction models. Based on the nominality score and anomaly score (computed using point-based reconstruction model) computed in the neighborhood, the induced anomaly score is further proposed by considering the temporal relationship. Some theoretical results of the proposed algorithm are provided. Experiments on several benchmark time series anomaly detection datasets are performed to demonstrate the performance of the proposed algorithm in comparison with several state-of-the-art algorithms.

Strengths

1. This paper studies an important and interesting problem, i.e., how to detect anomalies in time series data without label data. 2. The proposed algorithm combines both point-based and sequence-based reconstruction models, which achieves quite good performance on several time series anomaly detection benchmark datasets. 3. Some theoretical results are provided for the proposed algorithm. 4. The paper is generally well written and the presentation is clear.

Weaknesses

1. Overall, this paper proposes a heuristic-based unsupervised time series anomaly detection algorithm. The overall pipeline in Algorithm looks Ok to me, but from experiments it seems the most important part is the point-based reconstruction. In other words, the rest of the proposed algorithm may be simplified. The ablation studies in Table 3 also partially confirms it. 2. Flawed experiments. The F1 scores reported in Table 2 for other algorithms, e.g., Anomaly Transformer, are not consistent with results reported in the literature, such as [33]. More justifications are required. 3. The theoretical results look sound, but may not be useful in practice. In other words, can you justify the value of these results in real-world time series anomaly detection applications?

Questions

1. In Table 2, the performance of several algorithms, e.g., Anomaly Transformer, is significantly lower than those reported in the literature. More justification is required. In Figure 4(b), the anomalies on the right of t=14800 are not detected by $M_{pt}+M_{seq}$, but detected by $M_{seq}$, resulting in false negative. Can you provide some explanations? 2. The proposed algorithm is an unsupervised algorithm. Thus, how to adapt the algorithm on different datasets remains not clear to me (besides the threshold used in adjusting the F1).

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Summary

This paper proposed a novel idea to handle the time-series anomaly detection problem by calculating the "nominality score"(concerns on this name are in the question part and I'll keep using this name in the following review) and the induced anomaly score. And the F1 score can be mathematically proved to improve using the proposed scores. The authors then constructed the point-based and sequence-based reconstruction models to estimate the anomaly score and experiments on several datasets demonstrate the superior performances.

Strengths

1. The proposed method is novel and provides a new angle to address the anomaly detection problem. Without directly considering how "abnormal" a point is, this work start with thinking what is "normal" and induce anomality score based on the nominality score. 2. The authors provided both theoretical and experimental supports for the proposed methods which are reasonable. 3. The writing of the paper is clear and easy to follow.

Weaknesses

1. A major concern is on the expectation of improvement by using $\hat A(\cdot ;g_{\theta_2})$. While the Claim 1 and Claim 2 as well as the proofs gives a guarantee that using this method won't result in a worse F1 score, it says nothing about the gap with/without the method. Intuitively, the expected improvement may be related to $\alpha$ in Eq. (2), $d$, the function, the number of data, and other factors. The actual case could be that even if the $\alpha$ is larger than 1 by a margin (say \alpha=1.5, 2, or 5), the expected number of $N(t)\leq \theta$ for abnormal points are still a small portion of the whole dataset, making the proposed method useless in practical use. The analysis of a bound or expectation is missing. 2. The correctness of the method relies on several hypotheses, typically: 1) distribution of abnormal points $\Delta x_{t,a}^p$ has a larger variance than that of normal points $\Delta x_{t, n}^p$. This is used to guarantee that nominality score of normal points are larger than abnormal points; 2) the threshold is well selected so that the nominality score of all the normal points (with $y_t=0$) passes the threshold, while a significant number of abnormal points cannot pass the threshold. While the hypotheses are reasonable theoretically, I wonder whether they still hold when the scores are estimated and calculated based on the outputs from neural networks.

Questions

1. I wonder whether the word "nominality" is a misuse, although it is presented across the whole paper. The antonym to the word "anomaly" is "normality", and "nominality" is the noun of "nominal" which means "existing in name only; far below the real value or cost", and I think it's not related to the topic at all. 2. How is the backbone of the model (i.e., the performer-based autoencoder) chosen? Will that cause an unfair comparison between the proposed method and the baselines due to the neural model's approximation ability? 3. The supplementary file mentions how a best threshold is chosen, but in 3.4 ablation study the threshold is set to 98.5% of the nominality score from the training data. Why and how will the choice of different threshold influence the performance (i.e., whether the method is sensitive to the choice of hyperparameters?)

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

No apparent limitations and negative societal impact observed by the reviewer.

Summary

Basically, this paper aims to consider anomalousness of data points both from a point perspective, which is independent of the temporal relationships in the data, and contextual perspective, which reflects temporal relationships in the data. The paper derives an induced anomaly score that considers both.

Strengths

This paper addresses an important problem of considering both point and contextual anomalies. The paper provides good theoretical background for their work and convincing experimental results, showing particular anomalies that their method finds that are not found by methods that look for either point-based or contextual anomalies.

Weaknesses

The datasets on which testing is performed have rather high anomaly rates. The authors should experiment with some datasets with much lower anomaly rates, perhaps by leaving out some anomalies in the datasets that they use. Post rebuttal comment: Thanks to the authors for the responses. While the anomaly rates that the authors use are worth testing, it is quite common to use lower anomaly rates, and such lower rates would constitute a better test of the performance of the new algorithm. With regard to removing anomalies from datasets, one can remove individual time series that are identified as anomalous.

Questions

At the start of section 2.2, $\mathbf{x}_^0$ is part of the observed dataset but is also defined to be $\mathbf{x}_t^c + \Delta\mathbf{x}_t^p$. This cannot be correct. Did you mean to define $\delta\mathbf{x}_t^p$?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No limitations are described on the work. Societal impact material is not relevant here. However, a description of future work should be provided based on any patterns in the errors that the presented algorithm makes. --- Post rebuttal addition: Thanks to the authors for pointing out the discussions on limitations in the supplementary material. However, such material is critical to understanding the nature of the contribution, and so needs to be in the main paper.

Re: Rebuttal by Authors

I thank the authors for the detailed responses to my questions. Most of my concerns are well addressed. And I'm sorry that I didn't foresee the difficulty to induce a bound or expectation of the improvement by applying your induced anomaly score. The best result I could obtain under your hypothesis (2)~(4) using the hard gate function (11) with $d=1$ and threshold $\theta$ set to cover 100% anomaly points is in a rather complex form. The improvement seems to be relevant to the dimension of data $D$, ratio of std for normal to abnormal points $\alpha$, the size of the dataset, and the proportion of the anomalies. It seems that the increase of $\alpha$, decrease of anomaly proportion, and sadly decrease of the size of dataset (assume still large) will help improve the result. This means the method may be not effective for extreme large dataset containing many anomalies. I'm not sure whether it will be easier under other assumptions, but I admit this is beyond the scope of this paper while I hope to see further analysis in future work. Figure 3 in the provided pdf in the global response gave me the empirical intuition of the possible improvement. I don't have more concerns at current stage. Thus, I'll raise my score to accept.

Acknowledgement

Thank you for the detailed replies to all reviews. I appreciate the additional review of related work presented in your reply. As acknowledged by the authors, there are several refinements and improvements that can be applied to elevate the quality of the paper. As a result, I do not feel inclined to argue for its acceptance, but will raise my score to a borderline accept.