OPTIMIZING SYSTEM PERFORMANCE BY USING NON-VOLATILE MEMORY

Abstract

The next generation Non-volatile memory (NVM) provides us an opportunity to rethink and redesign the different storage hierarchies. In this thesis, we first explore the way of optimizing the file system journaling by using NVDIMM. We propose NV-Journaling, which presents fine-grained commits along with a locality-aware checkpointing process that significantly reduces checkpoint frequency and gains better space utilization. After the file system, we propose FEDB, a distributed in-memory database system designed to efficiently support online feature extraction. We propose PMem-optimized persistent skiplist to overcome the huge memory consumption, long tail latency, and long recovery time problems. As one of the important use cases of AI databases, we then explore the method of using NVM for the Deep learning recommendation models (DLRM) training process. We propose OpenEmbedding, which applies an efficient training pipeline and a lightweight batch-aware checkpointing scheme that is specially optimized for DLRM batch-based training on PMem.