SICSURFIS – SPECTRAL IMAGING OF COMPLEX SURFACE

TOMOGRAPHIESPrincipal investigator (PI): Docent Ilkka Pölönen (JyU), Principal Scientist Heikki Saari (VTT), Professor Jarmo Alander (UVa), Professor Annamari

Ranki (HUCH)01.01.2018-31.12.2021

Contents

• Introduction• New MEMS FPI design with metal Ag mirrors• Melanoma classification using convolutional neural networks• Using wave propagation simulations and convolutional neural

networks to retrieve thin coating’s thickness from hyperspectral images

Introduction• SICSURFIS is multidisciplinary project combining expertise from opto-

electronics, embedded computing systems, spectral image processing and dermatology.

Project objectives are:• To create a cost efficient hardware and software solution with miniaturized

spectral sensors and illumination devices for the spectral imaging of surfaces with complex tomography• To develop a novel software tools to analyze and model the detected data

applied to the detection of e.g. skin cancer and caries• To create an optical biopsy - skin and tooth surface topography and

tomography based on measured spectra.

Hyperspectral image

4

Wavelength (nm)

Inte

nsity

Spectrum of single spatial pixel

Collimating optics

Focusing optics

FPI mirrorsSp & Lp filters

456nm 570nm 760nm

Qua

ntum

effi

cienc

y

Hyperspectral imaging concpet based on the RGB image sensor and the Fabry-Perot tunable filter

• Figure shows the measured and simulated transmission curves of an AgMFPI in the wavelength range 450 – 850 nm.

• Several FPI orders are transmitted simultaneously at FPI air gap range 792 – 1200 nm.

• With a RGB image sensor it possible to retrieve spectral signal at three spectral bands for example the range 550 – 950 nm could be used with the measured AgMFPI chip.

Motivation to develop metallig mirror MEMS FPI components

It possible to cover the entire range with just one Ag MFPI

Previous ALD Bragg reflector mirrors -> limited operation range

• Ag based lower and upper mirrors are made by deposition of sandwich film stacks consisting of an ALD Al2O3 under layer, sputtered Ag thin film and an ALD Al2O3 protection layer.

• The Ag thin films within the lower and upper mirrors are also used as electrodes for tuning the air gap distance.

• Ag is patterned to form electrodes and electrical feedthroughs are made by fabricating Aluminum (Al) pads in contact to the Ag thin films.

• The tuning air gap is made by selective etching of a sacrificial material, i.e. PECVD TEOS SiO2, between the lower and upper mirrors through release holes opened in the top mirror.

• The released upper mirror is then a self-supported suspended membrane.

Novel Ag MEMS FPI - process design and integration

Bin Guo & al. “Wide-band large-aperture Ag surface-micro-machined MEMS Fabry-Perot Interferometer (AgMEMSFPIs) for miniaturized hyperspectral imaging”, Proc. SPIE 10545 (2018).

The spectral filter

• The FPI air gap is tunablefrom ca. 1400 nm to 700 nm• Beyond classical pull-in

• Wavelength coverage is ca. 600 – 1100 nm• The FPI operates up to 2.5 um

in theory• Optical aperture 3 mm• Chip size: 5.1 mm x 5.1 mm

19.5.2019 VTT – beyond the obvious 8

Fabry-Perot interferometer

3.2.2019 VTT Microspectrometers

A: Contact pads; B: Actuation electrode; C: Upper movable mirror; D: Lower fixed mirror; E: Optical aperture

Tunable AgMFPI filter hyperspectral imaging concept comparison to alternative solutions

• Smaller module size with optics in comparison to pixel-array spectral imagers

• Tunable FPI can be combined with any camera technology(high pixel density)

• Snapshot 2D images with high spatial resolution• Software-programmable sampling with hundreds of

spectral bands enable more accurate data algorhitms (food applications, health…)

Pixel-array filters (tiled)

Fabry-Perot Mirrors

Air gapOrder sorting

filter

Object of the hyperspectral

imager

Image of the hyperspectral

imager

Front optics for collimation Focusing optics

for imaging

CM

OS

Pa

ssB

an

dF

ilte

r

Tun

ab

leF

PI f

ilte

r

Fabry-Perot Mirrors

Air gap

Incoming light

Transmitted light

400 500 600 700 800 900 10000

0.2

0.4

0.6

0.8

Wavelength/[nm]

Spec

tral t

rans

mis

sion

Pixel-array filters (mosaic)

Key difference: - To enable high spatial resolution

with several WL bands takes upwafer area and is -> not low cost!

Key difference: - Pixel size limits amount of wavelength bands to 4x4

(16).- Plasmonic filters offer more bands, but have poor

extinction coefficient.- Wavelengths cannot be changed later which limits

application potential

Comparison:

Overview of the SICSURFIS Hyperspectral camera 1st prototype

• The Hyperspcetral camera

Photograph of the SICSURFIS Hyperspectralcamera 1st prototype 3D view of the Hyperspectral camera

MFPI chip packaging concept used in firstSICSURFIS Hyperspectral camera

MFPI packaging concept based on MFPI chip glued and wire bonded on the MFPI PCB, one or two external apertures and cover PCB.

Optics design of HSI Camera based on two commercial S-mount and one C-mount objective

• The focal length of the systemis determined by the C-Mount objective

• With 4.15 mm focal length thefull radial FOV is ~24 degrees.

• The F-number of the system is determined by the MFPI PCB external apertures

• The F-number for the 2.0 mm diameter aperture is ~5.0

20/05/2019 14

Technical specifications of the SICSURFIS Hyperspectral 1st prototype

Parameter Typical/targeted value RemarksAgMFPI spectral range 550 – 950 nmSpectral resolution < 25 nm @ FWHM Typical range 12 – 20 nm Spectral step < 1 nmMFPI Clear aperture Ø 2.0 mmMFPI Chip dimensions 5.3 mm x 5.3 mm x 0.675

mmMaximum allowed FPI ray angle < 10ºMinimum NA Ø 0.235CMOS image sensor dynamic range Ø 12 bitsSpectral image dimensions2.2 µm x 2.2 µm pixel size

Ø 640 x 480 pixels The image size is a cropped VGA image at the center of the image sensor

Frame rate for single VGA raw image Ø 25 frames/s

Settling time max from any selected FPI spacing to any other FPI spacing

< 3 milliseconds The settling time of the existing chips is < 3 ms for small wavelength steps.

Frame rate for data cube < 2.0 sDimensions up to C-mount < 81 mm x 60 mm x 58 mm

Weight < 500 g

20/05/2019 15

Spectral transmission measurement results of the AgMFPI chip ID EN14

Spectral transmissions at 121 AC drive voltages (64000 DN corresponds approximately the voltage range 13.5 Vrms) Table contains the drive voltages/DN (column 0), transmitted wavelengths (columns 1, 2, 4 and 4), FWHMs (columns 5, 6, 7 and 8).

Settling time chracterization results for the AgMFPI chip

Contour plots of the AgMFPI chip air gap distribution at 1254 nm (0 voltagedrive voltage) and 1017 nm average gap (~80% of the 0 voltage air gap)

submatrix gdat2_r16 q1 41× 1+, q1 41× 40+, 0, 39, ( )gap1ave 1254.6= gap_std1 3.425=

submatrix gdat2_r16 q1 41× 1+, q1 41× 40+, 0, 39, ( )gap1ave 1017.0= gap_std1 3.401=

Contour plot for at 1254 nm average FPI air gap (0 voltage drive voltage)

Contour plot for at 1017 nm average FPI air gap(80 % of the0 voltage drive voltage air gap)

Overview of SICSURFIS Hyperspectralimaging concept for complex surfaces

• The Led illumination will made from three different angles in thewavelength range 500 – 950 nm (1000 nm). • White Leds are used for the wavelength range 500 – 650 nm and

single wavelength leds 680, 720, 750, 780, 810, 850, 880 and 940 nmfor the range 650 – 950 nm.• The illuminated area at the skin is ~ 30 mm x 40 mm.• The estimated weight of the system is < 900 g• The dimensions are ~70 mm x 65 mm x 200 mm.

SICSURFIS Hyperspectralimaging concept for complex surfaces MEMS-FPI CMOS SENSOR

ARM+FPGA

LENSES

BANDPASS FILTERS

LED ILLUMINATION

COMPLEX SURFACE

Photometric processing• Albedo• Surface normal• Surface elevation modelParameter retrieval• Biophysical and –chemical

parametersClassification and deliniation

PROTOTYPE OF MINIATYRIZED HYPERSPECTRAL IMAGER

Melanoma classification usingconvolutional neural networksPölönen, Annala, Rahkonen, Neittaanmäki

In brief:• 61 lesions imaged with

hyperspectral imagin system• Classification based on

histopathology results after lesion removal• We trained different convolutional

neural networks to classify between • Malignant melanoma• Lentigo maligna (Melanoma in-situ)• Dysplastic nevus• Other benign nevus• Healthy skin

• Leave-one-out cross validation was used• For each lesion own network was trained

based on approx. 1000 samples / class • One sample contained 10x10 pixel area (1

pix = 75 𝜇𝑚) and 50 separate wavebands from 450 nm to 750 nm.

JYU. Since 1863. 2119.5.2019

In brief:• We got pixel-wise classification

for whole lesion.• We had histopathology based

diagnosis for the lesion.

• Whole lesion was classified based on most dangerous pixel in following order:• Malignant melanoma• Lentigo maligna (Melanoma in-

situ)• Dysplastic nevus• Other benign nevus• Healthy skin

JYU. Since 1863. 2219.5.2019

Melanoma detection using convolutional neural networks

JYU. Since 1863. 2319.5.2019

Input

Spectral cube

Separate spectral

bands

Single spectrum

Inte

nsity

Conv3d

(3,3,3,64)+ReLu

Conv3d

(3,3,3,128)+ReLu

MaxPooling3d

(2,2,2)

MaxPooling3d

(1,1,2)

Feature learning

Flat

ten

and

mer

ge

Conv2d

(3,3,64)+ReLu MaxPooling2d

(2,2)

Conv2d

(3,3,128)+ReLu MaxPooling2d

(2,2)

Conv1d

(3,64)+ReLu MaxPooling1d

(2)

Conv1d

(3,128)+ReLu MaxPooling1d

(2)

FC

ReLu

32

FC

ReLu

128

FC

ReLu

64 FC

ReLu

256 Fully connected

ReLu

512

Classification

DropOut

0.5

FC

SoftMax 

6

• Optimization: Adam –algorithm• Cost function: categorical cross-entrophy

JYU. Since 1863. 24

Example: Discrete 2D convolution• 𝐾 is convolution kernel• 𝐹 is image• Discrete convolution:

𝐹 ∗ 𝐾 𝑖, 𝑗 = +,,-𝐹 𝑖 − 𝑢, 𝑗 − 𝑣 𝐾(𝑢, 𝑣)

19.5.2019

JYU. Since 1863. 25

Example: Discrete 2D convolution

• 𝐾 =−1 0 1−2 0 2−1 0 1

19.5.2019

JYU. Since 1863. 26

Example: Discrete 2D convolution

• 𝐾 =−1 −2 −10 0 01 2 1

19.5.2019

Melanoma detection using convolutional neural networks

JYU. Since 1863. 2719.5.2019

Melanoma detection using convolutional neural networks

JYU. Since 1863. 2819.5.2019

Melanoma detection using convolutional neural networks

JYU. Since 1863. 2919.5.2019

Melanoma detection using convolutional neural networks

JYU. Since 1863. 3019.5.2019

Melanoma detection using convolutional neural networks

JYU. Since 1863. 3119.5.2019

Melanoma detection using convolutional neural networks

JYU. Since 1863. 3219.5.2019

Melanoma detection using convolutional neural networks

JYU. Since 1863. 3319.5.2019

34

Some remarks:• Classification seems to work better while we combine feature

types• PPV is currently same or better than clinicians• Limitation• Small data set• Ground truth• Validation

5/19/19

Next steps

• Create stochasticand DEC (1D, 2D and 3D) models for the skin optics• Develop proper inversion function• Develop fast FPGA processing• Start clinical tests, autumn 2019