METHOD OF GENERATING IN-KERNEL HOOK POINT CANDIDATES TO DETECT ROOTKITS AND THE SYSTEM THEREOF

A method for determining whether a to-be-tested program contains malicious behavior is disclosed. The method includes steps of providing an emulator having a kernel and a plurality of installed hook points, wherein the kernel has a plurality of in-kernel functions; executing the to-be-tested program in the emulator dynamically to invoke the plurality of installed hook points to obtain a specific in-kernel function set from the plurality of in-kernel functions; and determining whether the to-be-tested program contains instructions for malicious behavior based on an invocation sequence of the specific in-kernel function set.