an adaptive hyperspectral imager: design, control ......an adaptive hyperspectral imager: design,...
TRANSCRIPT
LAAS-CNRS/ Laboratoire d’analyse et d’architecture des systèmes du CNRS
An adaptive hyperspectral imager: design, control, processing and applications
Simon Lacroix – LAAS/Robotics Antoine Monmayrant – LAAS/Photonics Hervé Carfantan – IRAP/Signal & Image
2
What is color?
> Color: “The property possessed by an object of producing different sensations on the eye as a result of the way it reflects or emits light”
3
What is color?
> “The property possessed by an object of producing different sensations on the eye as a result of the way it reflects or emits light”
> Which eye ?
Humaneye? Insecteye? Man/sshrimpeye?
4
Anenergyspectrum:
What is color?
> “The property possessed by an object of producing different sensations on the eye as a result of the way it reflects or emits light”
> Light: “Electromagnetic radiation within a certain portion of the electromagnetic spectrum”
Thehumanpor/on
5
What is color?
> Color: “The property possessed by an object of producing different sensations on the eye as a result of the way it reflects or emits light”
> Light: “Electromagnetic radiation within a certain portion of the electromagnetic spectrum”
> Light through eyes
3numbers 4numbers 12numbers!
6
Hyperspectral images: “true colors”
> B&W image (aka intensity image): every pixel encodes the integral of energy over a given wavelength interval
> Color image: every pixel encodes the energy captured along three wavelength intervals (“3 image planes”)
> Multi-spectral images: up to a dozen image planes > Hyperspectral images: hundreds of image planes
7
Hyperspectral images: “true colors”
> B&W image (aka intensity image): every pixel encodes the integral of energy over a given wavelength interval
> Color image: every pixel encodes the energy captured along three wavelength intervals (“3 image planes”)
> Multi-spectral images: up to a dozen image planes > Hyperspectral images: hundreds of image planes
➤Hyperspectral cube”
8
What for?
> The reflected spectrum of a surface is characteristic of its physical nature
Various materials Various minerals
9
What for?
> The reflected spectrum of a surface is characteristic of its physical nature
☛ A wide spectrum of applications § Astronomy § Earth observation § Agriculture § Gaz detection § Forensic § Medicine § Art paint analysis § Microscopy § ...
10
What for?
> The reflected spectrum of a surface is characteristic of its physical nature
☛ A wide spectrum of applications § Astronomy § Earth observation § Agriculture § Gaz detection § Forensic § Medicine § Art paint analysis § Microscopy § ...
11
Hyperspectral cameras
> Core component: dispersing element
Gra/ng(diffrac/on) Prism(refrac/on)
> Main difficulty: recovering a 3D data cube with a 2D sensor
12
Hyperspectral cameras
> Classic technology: imaging a slit through a disperser
13
Hyperspectral cameras
> Classic technology: imaging a slit through a disperser > Alternative technologies: snapshot hyperspectral imaging
• Imagingseveralordersofdiffrac/on
• +tomography-likereconstruc/on
14
Hyperspectral cameras
> Classic technology: imaging a slit through a disperser
> Alternative technologies: snapshot hyperspectral imaging
• Codedaperturesnapshotspectralimaging(CASSI)
• Compressedsensingreconstruc/on
15
Hyperspectral cameras
> Classic technology: imaging a slit through a disperser ☛ Main issue: sequential acquisition
16
Hyperspectral cameras
> Classic technology: imaging a slit through a disperser ☛ Main issue: sequential acquisition
> Alternative technologies: snapshot hyperspectral imaging ☛ Several issues
§ High CPU load to produce the HS cube § Performances fixed by design § Tough calibration issues
17
Hyperspectral cameras
> Classic technology: imaging a slit through a disperser ☛ Main issue: sequential acquisition
> Alternative technologies: snapshot hyperspectral imaging ☛ Several issues
§ High CPU load to produce the HS cube § Performances fixed by design § Tough calibration issues
☛ All approaches produce heavy data (~ 1 Gb): § To acquire § To process § To transmit
18
An adaptive hyperspectral imager
> Principle
19
An adaptive hyperspectral imager
> Principle (illustration)
A partial masking of the signal defines a spectral filter
20
An adaptive hyperspectral imager
> Principle (illustration)
A partial masking of the signal defines a spectral filter
21
An adaptive hyperspectral imager
> Principle (illustration)
A partial masking of the signal defines a spectral filter
22
An adaptive hyperspectral imager
> Principle (illustration)
A partial masking of the signal defines a spectral filter
23
An adaptive hyperspectral imager
> Principle (illustration)
A partial masking of the signal defines a spectral filter
24
An adaptive hyperspectral imager
> Principle (illustration)
A partial masking of the signal defines a spectral filter
25
An adaptive hyperspectral imager
> Principle (illustration)
A partial masking of the signal defines a spectral filter
26
An adaptive hyperspectral imager
> Principle (illustration)
A partial masking of the signal defines a spectral filter
27
An adaptive hyperspectral imager
> Principle (illustration)
For the same mask, different spatial rays are differently
spectrally filtered
28
An adaptive hyperspectral imager
> Principle (illustration)
For the same mask, different spatial rays are differently
spectrally filtered
29
An adaptive hyperspectral imager
> Design: dual 4f-line
"Single-shotcompressivespectralimagingwithadual-disperserarchitecture",Ghemetal.Op/csExpress2007.
30
An adaptive hyperspectral imager
> Design: dual 4f-line
Digital Micro-mirror Device
31
An adaptive hyperspectral imager
> Design: dual 4f-line
�! �"#$%&#'()*+,-.&�)�*�/0#1��2#$1�
%34546789:$:;<3<:=>8?<;4:3 +@-8A655<?3 B475<?<C8B43678%$6;<
�"D�!E �"E
�!F �"F
�!D#$E
#$F
#$D
�!E �"E�!F �"F�!D �"D
#!!
#"""
"D"F"E
!D!F!E#! #"
�
�%34546789:$:;<3<:=>8?<;4:3 +@-8A655<?3 B475<?<C8B43678%$6;<
32
An adaptive hyperspectral imager
> Properties § Possibility to acquire an intensity image
33
An adaptive hyperspectral imager
> Properties § Possibility to acquire an intensity image
§ “Colocation”
34
An adaptive hyperspectral imager
> Properties § Possibility to acquire an intensity image
§ “Colocation”
§ Independenceof lines
35
An adaptive hyperspectral imager
> Properties § Possibility to acquire an intensity image
§ “Colocation”
§ Independenceof lines
§ Simplicity of the model
36
An adaptive hyperspectral imager
> Properties § Possibility to acquire an intensity image
§ “Colocation”
§ Independenceof lines
§ Simplicity of the model
§ Programmable acquisition
➤Variousacquisi/onschemes
37
First proof of concept
> Design implementation
38
First proof of concept
39
First proof of concept
> Scanning a slit on the DMD
Imaged object: known light source through a mask
“bi-λ” illumination
!""
#""
$""
%""
&""
'""
()*+,-./0)1
2)345.64
%)"""!"" #"" $"" 7"" %)!""
%)"""
!""
#""
$""
7""
%""
&""
'""
8""
9""
()*+,-./0)12)345.64
:)*+,-./0)1
2)345.64
7!" 7&" 7#" 7'" 7$" 78" 77";4<./.2=>?)*250
@2>.2A,>B
)*46CD)E
D0
!"#
!$#
!%#
&''
&''
40
Overview of the system possibilities
> The control of the DMD allows to: § Configure the system for a given purpose / characteristic § Control the system to optimize its output (active perception paradigm)
> Taxonomy of acquisition schemes, along 2 “dimensions” 1. Type of recovered information (defined by operational goals)
- Full HS cube - Specific information (e.g. a set of given spectra)
2. Way to control the system - Pre-configured DMD patterns - On-line controlled / adapted patterns
> Criteria to consider for application contexts: § Number of acquisitions § Computational load to recover the information § Quantity/quality of recovered information
41
Overview of the system possibilities
> Example 1: scanning slit § Fills the whole cube
42
Overview of the system possibilities
> Example 2: monochromatic imager § Random access to any spectral plane in one acquistion
43
Overview of the system possibilities
> Example 3: spectrum response image § Exhibits the presence of a given spectrum in one acquisition
CMOS
--+------> | V
DMD
--+------> | V
CMOS
--+-+----> | | V V
DMD
--+-+----> | | V V
Corrélation spectrale
λ
Monochromatique
λ
CMOS
--+------> | V
DMD
--+------> | V
CMOS
--+-+----> | | V V
DMD
--+-+----> | | V V
CMOS
--+------> | V
DMD
--+------> | V
CMOS
--+-+----> | | V V
DMD
--+-+----> | | V V
Corrélation spectrale
λ
Monochromatique
λ
CMOS
--+------> | V
DMD
--+------> | V
CMOS
--+-+----> | | V V
DMD
--+-+----> | | V V
Monochroma/cimage
Spectrumcorrela/on
44
Overview of the system possibilities
> Example 4: generalized bayer mosaics § Numerous possible patterns
DMD
CMOS
DMD
CMOS
DMD
CMOS
45
Overview of the system possibilities
> Example 5: near-snapshot partitioning § 2 acquisitions: 1 panchro, 1 controlled
46
Overview of the system possibilities
> Example 6: quadratic regularization § Recovering the full cube from a small set of random DMD acquisitions
(PhD of Ibrahim Ardi, IRAP/LAAS)
Reconstruc/onofa31wavelengthscubefrom5images
47
Latest prototype
> Really engineered > Images the visible 500-700 nm > Integrated acquisition & DMD control
48
Earth observation applications?
> Configurability yields a series of trade-offs in terms of: § Number of acquisitions § Computational load to recover the information § Quantity/quality of recovered information
49
Earth observation applications?
> Configurability yields a series of trade-offs in terms of: § Number of acquisitions § Computational load to recover the information § Quantity/quality of recovered information
> Matrix sensor in a push-broom configuration: § Time delayed integration § Real-time adaptive control of the acquisitions
TDI(acquisi/on1)TDI(acquisi/on2)TDI(acquisi/onn)
…
50
As a conclusion
> Co-design: interdisciplinary cross fertilization
Op/csystems
design
Signal&imageprocessing
Ac/vepercep/onparadigm
Adap*vehyperspectral
imager