multispectral imaging
TRANSCRIPT
Stevens Tech
Multispectral Imaging:From Airborne Sensors
to Mainstream Computer Vision to the Classroom
Elli AngelopoulouStevens Institute of Technology
Stevens Tech
OutlineMultispectral/Hyperspectral Imaging
Traditional Color Imaging in Computer Vision
Multispectral within the Visible RangeData CollectionNew AchievementsChallenges
Multispectral in the Classroom
What does the Future Hold?
Stevens Tech
Multispectral ImagingMultispectral imaging combines the material identification power of spectroscopy with topological information that is available from two-dimensional images. The majority of the work is on remote sensing, where data is collected about the Earth’s surface and atmosphere.Detailed information about:
weather monitoring and forecastinggeographical map consrtuctioncrop countinggeological analysis forest-land managementwater pollution
Stevens Tech
Hyperspectral Sensors1972: NASA’s LANDSAT I with 4 and 7 bands
Late 1970s: NASA’s Scanning Imaging Spectroradiometer (SIS) with 32 bands
1984: NASA’s Airborne Imaging System (AIS 1) with 128 bands
1987: NASA’s (AVIRIS) Airborne Visible/Infrared Imaging Spectrometer with 224 bands http://aviris.jpl.nasa.gov
1995: NRL’s Hyperspectral Digital Imagery Collection Experiment (HYDICE) 210 bandsCommercially available: HyMap, Northrop Grumman, GER, etc.
Stevens Tech
Beyond Remote SensingAgriculture: Crop field quality assessment
Archeology: Dead Sea Scrolls
Manufacturing: Product defect detection
Stevens Tech
Computer VisionComputer vision involves the automatic deduction of the structure and the properties of a possibly dynamic three-dimensional world from either a single or multiple two-dimensional images of the world.
The appearance of an image depends onGeometryOptical properties of the materialsIllumination conditionsSensor
Stevens Tech
Color in Computer VisionCurrent color framework is RGB.
Hal and Greenberg have shown that using only three color bands introduces significant color distortions.
RGB setups fail to capture detailed spectral information.
Complex light interactions that affect the spectrum of the light reaching a sensor are not observed.
Solution: Capture a denser, finer spectral sampling of the visible light. Complex light reflectance interactions affect:
The automated analysis of captured imagesThe creation of photorealistic images
Stevens Tech
Multispectral in Computer Vision
Take a gray-scale image of a scene under different narrow color filtersFor example, each filter is 10nm wide (compared to the traditional 75nm wide RGB filters)
Stevens Tech
Multispectral Vision SensorsGlass filters
MechanicalSlowLimited number of filters (about 10)Affordable
Electronically tunable filtersElectronically controlledFastLarge number of filters (100s)
Newer technology
Stevens Tech
Multispectral Advances in Computer Vision
We can now see differences in colors that can not be seen with a naked eye or a traditional RGB sensor.
A body of work on getting better color descriptors http://www.multispectral.org
Multispectral surface reflectance analysis
We can classify materials or group regions in an image based on the chromophores of the materials and the surface reflectance behavior which can only be observed through multispectral sensors.
Stevens Tech
Reflected LightThe amount of light reflected from each point p=(x,y,z) in a scene depends on:
The incident light, EThe surface reflectance, S
I(p,λ) = E(p,λ)S(p,λ)
At each wavelength, λ, the amount of reflected light varies depending on:
The spectrum of the incident lightThe material propertiesThe scene geometry
Stevens Tech
The Origins of ColorFor diffuse materials:
The intrinsic color of an object is determined by the chromophores of the material.The intensity of the reflected light is a function of geometry.
For specular highlights:The color of a specularity depends on the scene geometry and the index of refraction, a material property that is a function of wavelengthThe intensity of specularly reflected light depends on geometry and the index of refraction, a material property that is a function of wavelength
Stevens Tech
Spectral DerivativeA new technique for extracting color information from dense spectral sampling is by computing the rate of change in reflected intensity with respect to wavelength.After computing the natural logarithm, we get
Lλ(p,λ) = ∂L(p,λ)/∂λ =∂(lne(p,λ)+lnE(p,θi,ϕi)+ lnS(p,λ))/∂λ
More compactlyLλ(p,λ) = (eλ(p,λ)/e(p,λ)) + (Sλ(p,λ)/S(p,λ))
where eλ(p,λ)= ∂e(p,λ)/∂λ and Sλ(p,λ)= ∂S(p,λ)/∂λ
For many illumination spectra one can assume that across the visible spectrum, eλ(p,λ) = 0
Stevens Tech
Diffuse SurfacesThe spectral derivative is purely a function of Surface Reflectance, S
Lλ(p,λ) = Sλ(p,λ)/S(p,λ) According to Lambert’s Law
S(p,λ) = ρ(p,λ) cos(θi(p))where ρ(p,λ) is the albedo of the surface
For Lambertian surfaces, the spectral derivative is the normalized spectral derivative of the surface albedo
Lλ(p,λ) = ρλ(p,λ)/ρ(p,λ) This relationship holds for other diffuse reflectance models like the Oren and Nayar (1995) and the Wolff (1994) models.
Stevens Tech
Specular HighlightsThere is a prevalent assumption that for dielectrics the color of specularities is the color of incident light.Our data (both RGB and multispectral) shows that the color of specularities even in plastics and ceramics is often not the color of incident light.The change in color can be attributed to the Fresnel reflection coefficient and its dependence on the index of refraction, n(λ), which is a function of wavelength.For specular surfaces, the spectral derivative isolates the Fresnel term:
Sλ(p,λ)= Fλ(p,λ)/F(p, λ)where F(p, λ) is the Fresnel reflectance ratio and Fλ(p,λ) = ∂F(p, λ)/∂λ
Stevens Tech
Spectral Gradient Computation
Discrete implementation of Spectral Derivatives:1. For each scene there are n spectral images, Iw,
w=1..n2. Their natural logarithm is computed:
Lw=ln (I)3. Differentiation is approximated via finite-differencing:
Lδw= Lw+1 – Lw
4. The Spectral Gradient is a 7-vector:(Lδ1
, Lδ2,.., Lδn-1
) = (L2 – L1 ,.., Ln – Ln-1)
Stevens Tech
Experimental SetupEach scene was captured using a gray-scale camera with a filter wheel mounted in the front.The filter wheel contained 8 filters, 10nm wide, with central wavelengths 450, 480, 510, 540, 570, 600, 630 and 660nm.
The scene was illuminated by a single tungsten light bulb.For the diffuse objects we took images were under different light source positions and different intensities.
Stevens Tech
Experimental ObjectsMatte Surfaces:
Sheets of foam, each of a distinct colorSheets of construction paper, each of distinct colorSoda cans painted with flat paintSmooth ceramics (porcelain) with localized specularities
Hybrid Surfaces:Plastic peppers of different colorsRough ceramics (earthenware)Smooth ceramics (porcelain) with inter-reflections
Stevens Tech
Different Colors of Paper
Spectral gradients of different colors of paper taken under the same illumination conditions.
Stevens Tech
Objects of the Same Color
Spectral gradients of different white surface patches: white paper, white foam, white mug, white soda can.
Stevens Tech
Similar Colors
Spectral gradients of surface patches of different shades of pink and magenta under the same lighting.
Stevens Tech
Invariance to Illumination
Spectral gradients of a green surface patch as the direction and intensity of the incident light changes.
Stevens Tech
Invariance to Illumination
Spectral gradients of a pink surface patch as the direction and intensity of the incident light changes.
Stevens Tech
Invariance to Geometry
Spectral gradients from different patches of the same curved object (soda can).
Stevens Tech
Fresnel Term of Common Materials
Plastics Ceramics
Different plastics and ceramics have distinct Spectral Gradientsprofiles at specularities, which imply distinct Fresnel terms.
Stevens Tech
Stability of the Fresnel Term
Yellow Pepper Paper Plate
For the same material and within the same specular region, the Spectral Gradient profile remains relatively constant.
Stevens Tech
Stability of the Fresnel Term
Red Pepper Mug
For the same material and within the same specular region, the Spectral Gradient profile remains relatively constant.
Stevens Tech
Different Color Regions
Stevens Tech
Multispectral in the Classroom
The majority of the algorithms in computer vision textbooks are presented for gray-scale images.
Generalizing these algorithms from gray-scale to RGB is very intuitive for the students.
It is a good exercise to ask the students to develop algorithms that expand established computer techniques into both traditional color space and multispectral space.
classroom discussionhomework problemsexam questions
Stevens Tech
A Multispectral FutureNew multispectral sensors.
Multispectral sensors become increasingly more affordable.
We need more publicly available pictures of indoor scenes taken with multispectral sensors.
Publicly available MultiSpectral Scene Analysis software http://www.cs.stevens.edu/~elli/Lab/MultiSSA
Calibration is still a challenge.
“How can I apply your multispectral method to my old-fashioned RGB data?”