Lightweight indexing of observational data in log-structured storage

Abstract

Huge amounts of data are being generated by sensing de- vices every day, recording the status of objects and the en- vironment. Such observational data is widely used in scien- tific research. As the capabilities of sensors keep improv- ing, the data produced are drastically expanding in pre- cision and quantity, making it a write-intensive domain. Log-structured storage is capable of providing high write throughput, and hence is a natural choice for managing large-scale observational data. In this paper, we propose an approach to indexing and querying observational data in log-structured storage. Based on key traits of observational data, we design a novel index approach called the CR-index (Continuous Range Index), which provides fast query performance without compromis- ing write throughput. It is a lightweight structure that is fast to construct and often small enough to reside in RAM. Our experimental results show that the CR-index is superior in handling observational data compared to other indexing techniques. While our focus is scientific data, we believe our index will be effective for other applications with similar properties, such as process monitoring in manufacturing. © 2014 VLDB Endowment.