Active Lighting for Appearance Decomposition Todd Zickler DEAS, Harvard University Appearance Decomposition I = f ( shape, reflectance ) Appearance f -1 ( I ) = ? illumination, Appearance Decomposition Research Overview APPEARANCE CAPTURE COLOR IMAGE FILTERING 3D RECONSTRUCTION PHOTOMETRIC INVARIANTS Appearance Decomposition Getting 3D Shape: Image-based Reconstruction I = f ( shape, reflectance, illumination ) f -1 ( I ) = ? Appearance Decomposition Reflectance: BRDF Bi-directional Reflectance Distribution Function Appearance Decomposition Conventional 3D Reconstruction: Restrictive Assumptions LAMBERTIAN: IDEALLY DIFFUSE Appearance Decomposition Example: Conventional Stereo ASSUMPTION: I l = I r IlIl IrIr Appearance Decomposition Example: Conventional Stereo IlIl IrIr ASSUMPTION: I l = I r Appearance Decomposition Conventional 3D Reconstruction: Restrictive Assumptions Shape from shading [Tsai and Shaw, 1994] Variational Stereo [Faugeras and Keriven, 1998] Space Carving [Kutulakos and Seitz, 1998] Multiple-window stereo [Fusiello et al., 1997] Appearance Decomposition Reflectance: BRDF Appearance Decomposition Reflectance: BRDF Appearance Decomposition Helmholtz Reciprocity [Helmholtz 1925; Minnaert 1941; Nicodemus et al. 1977] Appearance Decomposition Stereo vs. Helmholtz Stereo STEREOHELMHOLTZ STEREO Appearance Decomposition Stereo vs. Helmholtz Stereo STEREOHELMHOLTZ STEREO Appearance Decomposition Stereo vs. Helmholtz Stereo STEREOHELMHOLTZ STEREO Appearance Decomposition Reciprocal Images Specularities fixed to surface IlIl IrIr Relation between I l and I r independent of BRDF Appearance Decomposition Reciprocity Constraint ^ n vlvl ^ vrvr ^ p olol oror = vlvl ^ vrvr ^ p olol oror ^ n Appearance Decomposition Reciprocity Constraint ^ n vlvl ^ vrvr ^ p olol oror = vlvl ^ vrvr ^ p olol oror ^ n Arbitrary reflectance Surface normal Appearance Decomposition Reciprocal Acquisition CAMERA LIGHT SOURCE Appearance Decomposition Recovered Normals [Zickler et al. 2002] Appearance Decomposition Recovered Surface [Zickler et al., ECCV 2002] Appearance Decomposition In Practice 1.Arbitrary Reflectance 2.Off-the-shelf components 3.Direct surface normals 4.Images aligned with recovered shape 5.Self-calibrating (coming) Appearance Decomposition Ongoing Work: Auto-calibration [Zickler et al., CVPR 2003, CVPR 2006,] Appearance Decomposition Research Overview APPEARANCE CAPTURE COLOR IMAGE FILTERING 3D RECONSTRUCTION PHOTOMETRIC INVARIANTS Appearance Decomposition Reflectance Decomposition DIFFUSE =+ SPECULAR [Phong 1975; Shafer, 1985] Appearance Decomposition Reflectance Decomposition [Shafer, 1985] Appearance Decomposition Reflectance Decomposition: Simplifies the Vision Problem =+ LAMBERTIAN: IDEALLY DIFFUSE =+ Appearance Decomposition Reflectance Decomposition: A Difficult Inverse Problem DIFFUSE =+ SPECULAR [Bajscy et al., 1996; Criminisi et al., 2005; Lee and Bajscy, 1992; Lin et al., 2002; Lin and Shum, 2001; Miyazaki et al., 2003; Nayar et al., 1997; Ragheb and Hancock, 2001; Sato and Ikeutchi, 1994; Tan and Ikeutchi, 2005; Wolfe and Boult, 1991,] =+ Appearance Decomposition Known Illuminant: Still Ill-posed G S I RGB B R D? Appearance Decomposition Known Illuminant: Still Ill-posed G S I RGB B R D? Appearance Decomposition Observation: Explicit Decomposition not Required G r2r2 r1r1 S I RGB B R J 1.INVARIANT TO SPECULAR REFLECTIONS 2.BEHAVES LAMBERTIAN Appearance Decomposition Observation: Explicit Decomposition not Required G r2r2 r1r1 S I RGB B R J || J || [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005] Appearance Decomposition Generalization: Mixed Illumination G r2r2 r1r1 S I RGB B R J r1r1 S1S1 B G R S2S2 J SINGLE ILLUMINANTMIXED ILLUMINATION [Zickler, Mallick, Kriegman, Belhumeur, CVPR 2006] Appearance Decomposition Generalization: Mixed Illumination Appearance Decomposition Example: Binocular Stereo [Algorithm: Boykov, Veksler and Zabih, CVPR 1998] Conventional Grayscale (R+G+B)/3 Specular Invariant, ||J|| (blue illuminant) Specular Invariant, ||J|| (blue & yellow illuminants) One image from input stereo pair Recovered depth Appearance Decomposition Example: Optical Flow [Algorithm: Black and Anandan, 1993] Conventional Grayscale (R-+G+B)/3 Specular Invariant, ||J|| (blue illuminant) Specular Invariant, ||J|| (blue & yellow illuminants) Ground truth flow Appearance Decomposition Example: Photometric Stereo [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005] J behaves Lambertian Linear function of surface normal Appearance Decomposition Example: Photometric Stereo [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005] J behaves Lambertian Linear function of surface normal Appearance Decomposition Example: Photometric Stereo [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005] J behaves Lambertian Linear function of surface normal Appearance Decomposition Example: Photometric Stereo [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005] Appearance Decomposition Example: Photometric Stereo [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005] Appearance Decomposition Generalized Hue G r2r2 r1r1 S I RGB B R J Appearance Decomposition Example: Material-based Segmentation [Zickler, Mallick, Kriegman, Belhumeur, CVPR 2006] Input image Conventional GrayscaleSpecular Invariant ||J|| Conventional Hue Generalized Hue Appearance Decomposition Active lighting can provide: 1. Precise shape (surface normals) for a broad class of (non-Lambertian) surfaces 2. Specular and/or shading invariance (e.g., optical flow, tracking, segmentation) 3. Minimal hardware requirements Active Lighting for Image-guided Surgery? Endoscopic imagery: 1. Illuminant(s) is/are controlled and known 2. Non-Lambertian surfaces 3. Lack of texture Appearance Decomposition Acknowledgements Satya Mallick, UCSD Peter Belhumeur, Columbia University David Kriegman, UCSD Sebastian Enrique, Columbia University Ravi Ramamoorthi, Columbia University