A Method of Mutating Windows Malwares using Reinforcement Learning with Functionality Preservation

Abstract

Recently, the development in both the quantity and complication of malware has raised a need for powerful malware detection solutions. The outstanding characteristics of machine learning (ML) and deep learning (DL) techniques have been leveraged in the fight against malware. However, they are proven to be vulnerable to adversarial attacks, where intended modifications in malware can flip the detection result and then evade the detector’s eyes. This research area is being focused on and deeply interested in many publications due to its significance in the evaluation of malware detection approaches. In such works, using Generative Adversarial Networks (GANs) or Reinforcement Learning (RL) can help malware authors craft metamorphic malware against antivirus. Unfortunately, the functionality of created malware is not mentioned and verified during the mutation phase, which can result in evasive but useless malware mutants. In this paper, we focus on Windows Portable Executable malware and propose an RL-based malware mutant creation approach to fool black-box static ML/DL-based detectors. Specifically, we introduce a validator to confirm functionality preservation, which is one of our requirements for a successfully created malware. The experiment results prove the effectiveness of our solution in crafting elusive and executable Windows malware mutants.