SECURING APPLICATIONS FROM UNTRUSTED OPERATING SYSTEMS USING ENCLAVES

Abstract

Modern CPUs have started supporting new abstractions which address the threats of an untrusted operating system. However, these architectural solutions offer a trade-off between security, ease of usability, and compatibility with legacy software (both OS and applications). In this thesis, we envision a low-TCB, POSIX-compatible, side-channel resistant, and a formally verified solution which allows users to securely execute their applications on an untrusted operating system. We first build architectural support to execute user-level applications in an isolated execution environment. This architecture design is akin Intel SGX ISA extensions, albeit achieving compatibility and scalability with legacy applications. Second, we build a low-TCB solution to execute legacy applications on Intel SGX platform. Third, we demonstrate the threat of a new class of attacks called page fault side-channel on cryptographic library implementations executing in Intel SGX. We build systematic defenses in hardware as well as software to prevent leakage via this side-channel. Finally, we formally model the class of attacks that the OS can launch against SGX enclaves via the filesystem API; and develop a complete set of formally verified specifications and implementation to disable them.