Adaptive environment map for relighting: Using cameras and projected light

Abstract

Scene relighting has found applications in many areas, from movie special effects to building immersive environments. In this paper, we present a framework to render a scene under any prescribed lighting environments. First, we propose a 3D scanner that captures the scene geometry and reflectance simultaneously, by effectively using an uncalibrated camera, and light and shadows projected from a controlled lighting plane onto the scene. Then, a large camera array, composed of 48 cameras forming a planar array, is built to capture a lighting environment as a 4D incident light field. With the scene properties captured by the 3D scanner, and the lighting environment captured by the camera array, we can then relight the scene using a technique we develop, called the adaptive environment map (AEM). AEM, which fully describes the lighting environment including both spatial and directional variations, can render more realistic relighting results than existing techniques in literature, as shown in extensive experiments.