Symbolic execution based feature extraction for detection of malware

Abstract

Traditional signature-based Anti-virus (AV) detectors use signatures unique to known malware. But these signatures are manually extracted and cannot detect variants of the same malware or zero-day malware. For automatically detecting zero-day malware and variants of the known malware, Machine Learning (ML) based approaches are proposed by researchers. For an ML-based malware detector, features are extracted using malware analysis techniques like static and dynamic analysis. While dynamic analysis techniques are useful for observing the runtime behavior of the malware (such as API call sequences), it was widely observed that malware can detect the debugging environment and modify its behavior. Static analysis on the other hand is limited by code obfuscation techniques and packers. In addition, analysis of the disassembled code is primarily a manual process due to indirect jumps/calls presented in the disassembled code. To solve these indirect jumps/calls and to automatically extract API call sequences, we propose a symbolic execution based static feature extraction system. This system emulates the binary symbolically and extracts multiple API call sequences as opposed to a single API call sequence obtained in dynamic analysis. These API call sequences are processed and used as input features for the ML classifier. The resulting system using Artificial Neural Networks (ANN) as a classifier, achieved a classification accuracy of 94.62% with the Area Under ROC Curve (AUC) of 0.9827. Combining the API call sequences with features extracted from headers of the Portable Executable (PE) binary, the accuracy improved to 97.42% with 0.9933 AUC.