Runtime binary analysis for security

Abstract

Exploitation of buffer overflow vulnerabilities constitutes a significant portion of security attacks in computer systems. Common buffer overflow attacks include return address attacks, format string attacks, vulnerable C function attacks, stack-smashing attacks, heap overflows and global offset table (GOT) modifications. The aim of these attacks is typically to hijack critical information in the process address space so as to redirect the programa??s control flow to any malicious code injected into the process memory. Previous solutions to these problems are based either on hardware or compiler. The former requires special hardware while the latter requires the source code of the application.In this thesis, I will be introducing Transparent RUntime Security Suite (TRUSS) that protects against common buffer overflow attacks. The scheme is implemented using DynamoRIO, a dynamic binary rewriting framework.DynamoRIO is implemented on both Windows and Linux. Hence, this scheme is able to protect applications on both operating systems. TRUSS has been successfully tested on the SPECINT 2000 benchmark programs (on both Windows and Linux), on John Wilandera??s a??Dynamic testbed for twenty buffer overflow attacksa??, on the a??Benchmark Suite for evaluating Architectural Security Systemsa?? as well as on Microsoft Access, PowerPoint, Excel and Word 2002. This thesis will discuss the implementation details of TRUSS. It also provides a performance evaluation, which will show that TRUSS is able to operate with an average overhead factor of up to 0.5 in Linux and 1.5 in Windows. Although implemented in DynamoRIO, the techniques employed in TRUSS are portable to other dynamic binary rewriting frameworks as well.