DETECTION OF MALICIOUS EXECUTABLE FILES USING HIERARCHICAL MODELS

In one embodiment, a device disassembles an executable file into assembly instructions. The device maps each of the assembly instructions to a fixed length instruction vector using one-hot encoding and an instruction vocabulary and forms vector representations of blocks of a control flow graph for corresponding functions of the executable file by embedding and aggregating bags of the instruction vectors. The device generates, based on the vector representations of the blocks of the control flow graph, a call graph model of the functions in the executable file. The device forms a vector representation of the executable file based in part on the call graph model. The device determines, based on the vector representation of the executable file, whether the executable file is malware.