Hidden Markov Models for abnormal event processing in transportation data streams

Abstract

Making sense of big data and big metadata remains a challenge as more and more data are churned out every day. The problem of adding value to unstructured data requires the application of computationally intensive algorithms to discover useful patterns in the data. In terms of data streams from public transport such as buses, we address the problem of performing time-consuming algorithms to model the data while still being able to process abnormal events in real-time. We propose using Hidden Markov Models (HMMs) for identifying conditions for an abnormal event in bus journeys and methods for isolating HMM computations from real-time event processing. Results show that training HMMs with even noisy metadata can generate models that can recognize an abnormal event in a parallel and distributed manner in the cloud. © 2012 IEEE.