an adaptive hyperspectral imager: design, control ......an adaptive hyperspectral imager: design,...

50
LAAS-CNRS / Laboratoire d’analyse et d’architecture des systèmes du CNRS An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics Antoine Monmayrant – LAAS/Photonics Hervé Carfantan – IRAP/Signal & Image

Upload: others

Post on 19-Jun-2020

8 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

LAAS-CNRS/ Laboratoire d’analyse et d’architecture des systèmes du CNRS

An adaptive hyperspectral imager: design, control, processing and applications

Simon Lacroix – LAAS/Robotics Antoine Monmayrant – LAAS/Photonics Hervé Carfantan – IRAP/Signal & Image

Page 2: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

2

What is color?

>  Color: “The property possessed by an object of producing different sensations on the eye as a result of the way it reflects or emits light”

Page 3: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

3

What is color?

>  “The property possessed by an object of producing different sensations on the eye as a result of the way it reflects or emits light”

>  Which eye ?

Humaneye? Insecteye? Man/sshrimpeye?

Page 4: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

4

Anenergyspectrum:

What is color?

>  “The property possessed by an object of producing different sensations on the eye as a result of the way it reflects or emits light”

>  Light: “Electromagnetic radiation within a certain portion of the electromagnetic spectrum”

Thehumanpor/on

Page 5: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

5

What is color?

>  Color: “The property possessed by an object of producing different sensations on the eye as a result of the way it reflects or emits light”

>  Light: “Electromagnetic radiation within a certain portion of the electromagnetic spectrum”

>  Light through eyes

3numbers 4numbers 12numbers!

Page 6: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

6

Hyperspectral images: “true colors”

>  B&W image (aka intensity image): every pixel encodes the integral of energy over a given wavelength interval

>  Color image: every pixel encodes the energy captured along three wavelength intervals (“3 image planes”)

>  Multi-spectral images: up to a dozen image planes >  Hyperspectral images: hundreds of image planes

Page 7: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

7

Hyperspectral images: “true colors”

>  B&W image (aka intensity image): every pixel encodes the integral of energy over a given wavelength interval

>  Color image: every pixel encodes the energy captured along three wavelength intervals (“3 image planes”)

>  Multi-spectral images: up to a dozen image planes >  Hyperspectral images: hundreds of image planes

➤Hyperspectral cube”

Page 8: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

8

What for?

>  The reflected spectrum of a surface is characteristic of its physical nature

Various materials Various minerals

Page 9: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

9

What for?

>  The reflected spectrum of a surface is characteristic of its physical nature

☛ A wide spectrum of applications §  Astronomy §  Earth observation §  Agriculture §  Gaz detection §  Forensic §  Medicine §  Art paint analysis §  Microscopy §  ...

Page 10: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

10

What for?

>  The reflected spectrum of a surface is characteristic of its physical nature

☛ A wide spectrum of applications §  Astronomy §  Earth observation §  Agriculture §  Gaz detection §  Forensic §  Medicine §  Art paint analysis §  Microscopy §  ...

Page 11: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

11

Hyperspectral cameras

>  Core component: dispersing element

Gra/ng(diffrac/on) Prism(refrac/on)

>  Main difficulty: recovering a 3D data cube with a 2D sensor

Page 12: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

12

Hyperspectral cameras

>  Classic technology: imaging a slit through a disperser

Page 13: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

13

Hyperspectral cameras

>  Classic technology: imaging a slit through a disperser >  Alternative technologies: snapshot hyperspectral imaging

•  Imagingseveralordersofdiffrac/on

•  +tomography-likereconstruc/on

Page 14: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

14

Hyperspectral cameras

>  Classic technology: imaging a slit through a disperser

>  Alternative technologies: snapshot hyperspectral imaging

•  Codedaperturesnapshotspectralimaging(CASSI)

•  Compressedsensingreconstruc/on

Page 15: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

15

Hyperspectral cameras

>  Classic technology: imaging a slit through a disperser ☛ Main issue: sequential acquisition

Page 16: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

16

Hyperspectral cameras

>  Classic technology: imaging a slit through a disperser ☛ Main issue: sequential acquisition

>  Alternative technologies: snapshot hyperspectral imaging ☛ Several issues

§  High CPU load to produce the HS cube §  Performances fixed by design §  Tough calibration issues

Page 17: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

17

Hyperspectral cameras

>  Classic technology: imaging a slit through a disperser ☛ Main issue: sequential acquisition

>  Alternative technologies: snapshot hyperspectral imaging ☛ Several issues

§  High CPU load to produce the HS cube §  Performances fixed by design §  Tough calibration issues

☛ All approaches produce heavy data (~ 1 Gb): §  To acquire §  To process §  To transmit

Page 18: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

18

An adaptive hyperspectral imager

>  Principle

Page 19: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

19

An adaptive hyperspectral imager

>  Principle (illustration)

A partial masking of the signal defines a spectral filter

Page 20: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

20

An adaptive hyperspectral imager

>  Principle (illustration)

A partial masking of the signal defines a spectral filter

Page 21: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

21

An adaptive hyperspectral imager

>  Principle (illustration)

A partial masking of the signal defines a spectral filter

Page 22: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

22

An adaptive hyperspectral imager

>  Principle (illustration)

A partial masking of the signal defines a spectral filter

Page 23: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

23

An adaptive hyperspectral imager

>  Principle (illustration)

A partial masking of the signal defines a spectral filter

Page 24: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

24

An adaptive hyperspectral imager

>  Principle (illustration)

A partial masking of the signal defines a spectral filter

Page 25: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

25

An adaptive hyperspectral imager

>  Principle (illustration)

A partial masking of the signal defines a spectral filter

Page 26: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

26

An adaptive hyperspectral imager

>  Principle (illustration)

A partial masking of the signal defines a spectral filter

Page 27: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

27

An adaptive hyperspectral imager

>  Principle (illustration)

For the same mask, different spatial rays are differently

spectrally filtered

Page 28: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

28

An adaptive hyperspectral imager

>  Principle (illustration)

For the same mask, different spatial rays are differently

spectrally filtered

Page 29: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

29

An adaptive hyperspectral imager

>  Design: dual 4f-line

"Single-shotcompressivespectralimagingwithadual-disperserarchitecture",Ghemetal.Op/csExpress2007.

Page 30: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

30

An adaptive hyperspectral imager

>  Design: dual 4f-line

Digital Micro-mirror Device

Page 31: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

31

An adaptive hyperspectral imager

>  Design: dual 4f-line

�! �"#$%&#'()*+,-.&�)�*�/0#1��2#$1�

%34546789:$:;<3<:=>8?<;4:3 +@-8A655<?3 B475<?<C8B43678%$6;<

�"D�!E �"E

�!F �"F

�!D#$E

#$F

#$D

�!E �"E�!F �"F�!D �"D

#!!

#"""

"D"F"E

!D!F!E#! #"

�%34546789:$:;<3<:=>8?<;4:3 +@-8A655<?3 B475<?<C8B43678%$6;<

Page 32: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

32

An adaptive hyperspectral imager

>  Properties §  Possibility to acquire an intensity image

Page 33: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

33

An adaptive hyperspectral imager

>  Properties §  Possibility to acquire an intensity image

§  “Colocation”

Page 34: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

34

An adaptive hyperspectral imager

>  Properties §  Possibility to acquire an intensity image

§  “Colocation”

§  Independenceof lines

Page 35: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

35

An adaptive hyperspectral imager

>  Properties §  Possibility to acquire an intensity image

§  “Colocation”

§  Independenceof lines

§  Simplicity of the model

Page 36: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

36

An adaptive hyperspectral imager

>  Properties §  Possibility to acquire an intensity image

§  “Colocation”

§  Independenceof lines

§  Simplicity of the model

§  Programmable acquisition

➤Variousacquisi/onschemes

Page 37: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

37

First proof of concept

>  Design implementation

Page 38: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

38

First proof of concept

Page 39: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

39

First proof of concept

>  Scanning a slit on the DMD

Imaged object: known light source through a mask

“bi-λ” illumination

!""

#""

$""

%""

&""

'""

()*+,-./0)1

2)345.64

%)"""!"" #"" $"" 7"" %)!""

%)"""

!""

#""

$""

7""

%""

&""

'""

8""

9""

()*+,-./0)12)345.64

:)*+,-./0)1

2)345.64

7!" 7&" 7#" 7'" 7$" 78" 77";4<./.2=>?)*250

@2>.2A,>B

)*46CD)E

D0

!"#

!$#

!%#

&''

&''

Page 40: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

40

Overview of the system possibilities

>  The control of the DMD allows to: §  Configure the system for a given purpose / characteristic §  Control the system to optimize its output (active perception paradigm)

>  Taxonomy of acquisition schemes, along 2 “dimensions” 1.  Type of recovered information (defined by operational goals)

- Full HS cube - Specific information (e.g. a set of given spectra)

2.  Way to control the system - Pre-configured DMD patterns - On-line controlled / adapted patterns

>  Criteria to consider for application contexts: §  Number of acquisitions §  Computational load to recover the information §  Quantity/quality of recovered information

Page 41: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

41

Overview of the system possibilities

>  Example 1: scanning slit §  Fills the whole cube

Page 42: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

42

Overview of the system possibilities

>  Example 2: monochromatic imager §  Random access to any spectral plane in one acquistion

Page 43: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

43

Overview of the system possibilities

>  Example 3: spectrum response image §  Exhibits the presence of a given spectrum in one acquisition

CMOS

--+------> | V

DMD

--+------> | V

CMOS

--+-+----> | | V V

DMD

--+-+----> | | V V

Corrélation spectrale

λ

Monochromatique

λ

CMOS

--+------> | V

DMD

--+------> | V

CMOS

--+-+----> | | V V

DMD

--+-+----> | | V V

CMOS

--+------> | V

DMD

--+------> | V

CMOS

--+-+----> | | V V

DMD

--+-+----> | | V V

Corrélation spectrale

λ

Monochromatique

λ

CMOS

--+------> | V

DMD

--+------> | V

CMOS

--+-+----> | | V V

DMD

--+-+----> | | V V

Monochroma/cimage

Spectrumcorrela/on

Page 44: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

44

Overview of the system possibilities

>  Example 4: generalized bayer mosaics §  Numerous possible patterns

DMD

CMOS

DMD

CMOS

DMD

CMOS

Page 45: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

45

Overview of the system possibilities

>  Example 5: near-snapshot partitioning §  2 acquisitions: 1 panchro, 1 controlled

Page 46: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

46

Overview of the system possibilities

>  Example 6: quadratic regularization §  Recovering the full cube from a small set of random DMD acquisitions

(PhD of Ibrahim Ardi, IRAP/LAAS)

Reconstruc/onofa31wavelengthscubefrom5images

Page 47: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

47

Latest prototype

>  Really engineered >  Images the visible 500-700 nm >  Integrated acquisition & DMD control

Page 48: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

48

Earth observation applications?

>  Configurability yields a series of trade-offs in terms of: §  Number of acquisitions §  Computational load to recover the information §  Quantity/quality of recovered information

Page 49: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

49

Earth observation applications?

>  Configurability yields a series of trade-offs in terms of: §  Number of acquisitions §  Computational load to recover the information §  Quantity/quality of recovered information

>  Matrix sensor in a push-broom configuration: §  Time delayed integration §  Real-time adaptive control of the acquisitions

TDI(acquisi/on1)TDI(acquisi/on2)TDI(acquisi/onn)

Page 50: An adaptive hyperspectral imager: design, control ......An adaptive hyperspectral imager: design, control, processing and applications Simon Lacroix – LAAS/Robotics ... § Forensic

50

As a conclusion

>  Co-design: interdisciplinary cross fertilization

Op/csystems

design

Signal&imageprocessing

Ac/vepercep/onparadigm

Adap*vehyperspectral

imager