A Comprehensive Review of Anomaly Detection in Web Logs

Abstract

Anomaly detection is a significant problem that has been researched within diverse research areas and application domains, especially in the area of web-based internet services or cybersecurity. Many anomaly detection techniques have been developed for specific application domains, while others are more generic. The Log files of Web-server give insight into the state of web-server and applications running on it and enable the detection of abnormal incidents or behavior. This paper focuses on particularly web-server HTTP logs to the problems of Web-server Log Anomaly Detection (WLAD) due to their own nature and features and aims to provide a brief review of different Data-driven techniques to get to the bottom of recent studies and developments made in the context of WLAD. Moreover, in this paper, the literature related to webserver logs analysis, as well as other closely related to the WLAD topic, are taken into consideration for review. We have classified existing techniques into different categories based on the underlying approach adopted. When applying a particular technique, these assumptions can be used as guidelines to assess the method’s effectiveness in this area. We also provide a basic security anomaly detection approach for each category and compare the existing methods as variants of the basic technique. Further, we identify the cons and pros of the current practices for each category. We also discuss the computational complexity of the methods, which is an essential issue in the domain of Big Data.