3D sound localization based on audio and video

Abstract

In this paper, 3D sound localization is presented by fusing audio information with video signals. This localization system consists of two subsystems, an audio setup with 3 microphones in a triangular format and a video source (a camera), all mounted on a pan-tilt unit. The audio subsystem is to generate two independent time differences ΔT 1 and ΔT 2. Based on the set of independent time differences, a 3D curve on which the sound source lies can be derived based on the audio setup alone. On the other hand, the video subsystem gives a plane on which the sound source lies. The intersection between the 3D curve and the plane can be used to determine exact location, i.e., a unique solution, of the sound source. As it is fairly complicated for such an analytical approach, neural network fusion is then proposed to fuse the two systems together conveniently and seamlessly. Simulation results show that this sound localization system can locate the sound source with satisfactory accuracy. © 2003 IEEE.