Exploration of a framework for behavior-based malware detection and classification

Abstract

One of the greatest security threats that we face today is malwares like worms and viruses. But as current defenses against malwares are fast approaching their limits, we propose a new behavioral approach to combat this threat.This thesis attempts to study the feasibility of detecting malwares based on behaviors and forms the basis of a new behavior-based detection system. While the final aim of our research is to study the behaviors of malware, the scope of this thesis is limit to malware detection. The reason for this approach is that we believe all malwares share some common behaviors, and malwares within the same families display more similar behaviors.We will explore a framework that allows the modeling of high-level behaviors from Windows native API system calls. But rather than simply using sequences of API calls to build behavior signatures like many other researches, we built semantically rich behavioral signatures based on context provided the system call and reverse engineering based on descriptions provided by anti-virus companies.In our analysis, we were successfully in identifying some behaviors common to all or most of our malware samples, but not to the set of normal applications used as baseline; thus showing the capability of our system to detect for the presence of known malwares and newer malware variants. We were also able to observe some interesting features of the malwares by studying the behavioral information provided by the framework.