Syslog records critical information of network when the system is running, and has been used to help practitioners carry out various network maintenance and operation activities. Because of abundance of syslog, automated log analysis technology is needed to complete the above activities. Usually, log parsing is an important part of log analysis. Previous log parsing methods have good achievements, but they heavily rely on well-designed regular expressions and ignore semantic information of log. To solve these problems, we propose a novel log parsing method SNNLog, which uses Siamese Network to assess the similarity between log messages, and the similarity is used for the subsequent parsing. In addition, after parsing, SNNLog merges log similar events, reducing the misjudgment rate caused by non-numeric token variables at the beginning of messages. We evaluated on five publicly accessible log datasets and the results show that compared with SOTA algorithms such as LogMine, Spell, Drain and MoLFI, SNNLog has an F1-score of 0.999 on five datasets, and parsing accuracy of four datasets is the highest.
Paper
Full text
SNNLog: A Log Parsing Scheme with Siamese Network and Fixed Depth Tree in Networks
Semantic Scholar · Computer Science · 2023
Abstract
Syslog records critical information of network when the system is running, and has been used to help practitioners carry out various network maintenance and operation activities. Because of abundance of syslog, automated log analysis technology is needed to complete the above activities. Usually, log parsing is an important part of log analysis. Previous log parsing methods have good achievements, but they heavily rely on well-designed regular expressions and ignore semantic information of log. To solve these problems, we propose a novel log parsing method SNNLog, which uses Siamese Network to assess the similarity between log messages, and the similarity is used for the subsequent parsing. In addition, after parsing, SNNLog merges log similar events, reducing the misjudgment rate caused by non-numeric token variables at the beginning of messages. We evaluated on five publicly accessible log datasets and the results show that compared with SOTA algorithms such as LogMine, Spell, Drain and MoLFI, SNNLog has an F1-score of 0.999 on five datasets, and parsing accuracy of four datasets is the highest.