Video quality for video analysis

Abstract

Video analysis algorithms are commonly used in a wide range of applications, including video surveillance systems, video conferencing, autonomous vehicles, and social web-based applications. It is typical in such systems to transmit video or images over an IP-network from video sensors or storage facilities to the remote processing servers for subsequent automated analysis. As video analysis algorithms advance to become more complex and robust, they start replacing human observers in these systems. The situation when algorithms are receivers of video data creates an opportunity for more efficient bandwidth utilization in video streaming systems. One way to do so is to reduce the quality of the video that is intended for the algorithms. The question is, however, can algorithms accurately perform on the video with lower quality than a typical video intended for human visual system? And if so, what is the minimum quality that is suitable for algorithms?

Video quality is considered to have spatial, SNR, and temporal components and normally a human observer is the main judge of whether the quality is high or low. Therefore, quality measurements, methods of video encoding and representation, and ultimately the size of the resulted video are determined by the requirements of human visual system. However, we can argue that computer vision is different from human vision and therefore has its own specific requirements to video quality and quality assessment.

Addressing this issue, we first conducted experiments with several commonly used video analysis algorithms to understand their requirements on video quality. We chose freely available and complex algorithms including two face detection algorithms, face recognition, and two object tracking algorithms. We used JPEG compression, nearest neighbor scaling, bicubic scaling, frame dropping, and other algorithms to degrade video quality, calling such degradations \textit{video adaptations}. Experiments demonstrated that video analysis algorithms maintain high level of accuracy until video quality is reduced to a certain minimal threshold. We term such threshold the \textit{critical video quality}. Video with this quality has much lower bitrate compared to the video compressed for human visual system.

Although this result is promising, given a video analysis algorithm, finding its crirtical video quality is not a trivial task. In this thesis, we apply an analytical approach to estimate the critical video quality. We develop a rate-accuracy framework based on the notion of rate-accuracy function, formalizing the tradeoff between algorithm's accuracy and video quality. This framework addresses the dependency between video adaptation used, video data, and accuracy of video analysis algorithms.

The principal part of the framework is to use reasoning about key elements of the video analysis algorithm (how it operates), essential effects of video adaptations on video (how it reduces quality), and if available, the semantic information about video (what is the video's content). We show that, based on such reasoning and a number of heuristic measures, we can also reduce the amount of experiments for finding critical video quality.

To summarize the contribution of the thesis: (i) we demonstrate on the few video analysis algorithms their tolerance to low critical video quality, which can lead to significant bitrate reductions when such an algorithm is the only ``observer'' of the video; (ii) we argue that finding such video quality is a hard task and suggest estimating it using algorithm-tailored metrics; and (iii) we demonstrate benefits in designing algorithms tolerant to reduced video quality and video encoders customized for video analysis.