high-resolution hyperspectral imaging for cultural heritage
DESCRIPTION
High-resolution Hyperspectral Imaging for Cultural Heritage. Rei Kawakami 1 John Wright 2 Yu-Wing Tai 3 Yasuyuki Matsushita 2 Moshe Ben-Ezra 2 Katsushi Ikeuchi 3 1 University of Tokyo, 2 Microsoft Research Asia (MSRA), 3 Korea Advanced Institute of Science and Technology (KAIST) - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/1.jpg)
High-resolution Hyperspectral Imaging for Cultural Heritage
Rei Kawakami1 John Wright2 Yu-Wing Tai3 Yasuyuki Matsushita2 Moshe Ben-Ezra2 Katsushi Ikeuchi3
1University of Tokyo, 2Microsoft Research Asia (MSRA), 3Korea Advanced Institute of Science and Technology (KAIST)
2011 Dunhuang Forum
![Page 2: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/2.jpg)
Giga-pixel Camera
M. Ben-Ezra et al.
Giga-pixel Camera
Large-format lens CCD
![Page 3: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/3.jpg)
Spectrum
200 5000[nm]
100.10.001
700 1000600500400300200 [nm]
Ultraviolet InfraredVisible light
1 meter 3100 meter0.00001
CosmicRays
GammaRays
X-raysUltraViolet
InfraredTV AndRadio Waves
Electric Waves
Electromagnetic Spectrum
Vio
let
Blu
e
Gre
en
Yello
w
Ora
ng
e
Red
0
20
40
6080
100
Solar radiationreachingearth’s surface(Relative Energy)
![Page 4: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/4.jpg)
RGB vs. Spectrum
![Page 5: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/5.jpg)
Applications
Light simulation
Layered surface decomposition
Morimoto et al. CVPR2010: Estimating Optical Properties of Layered Surfaces Using the Spider Model
![Page 6: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/6.jpg)
Why difficult?
![Page 7: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/7.jpg)
Approach
Low-reshyperspectral
High-resRGB
High-resolutionHyperspectral image
Combine
![Page 8: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/8.jpg)
Two-step approach
1. Factorize low-res hyperspectral image into basis functions of spectra and coefficients
2. For each pixel in high-res RGB image, estimate coefficients of the basis functions
![Page 9: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/9.jpg)
Problem formulation
W(Image width)
H(Image height)
S
Goal:
Given:
(Spectral wavelength)
![Page 10: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/10.jpg)
Representation: Basis function
W (Image width)
H (Image height)
S
𝒁
= …
01.00…0
= +x 0 x 1.0 x 0 x 0++
Reflectance vectors
![Page 11: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/11.jpg)
1: Matrix factorization
Sparse
For all pixel (i,j)
Sparse matrix
W (Image width)
H (Image height)
S
= …
00.40…
0.6
𝒀 h𝑠
• At each pixel of , only a few () materials are present
Reflectance matrix
![Page 12: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/12.jpg)
2: Reconstruction
W
H
S Sparse
𝒀 𝑟𝑔𝑏
�̂� (𝑖 , 𝑗 )=argmin‖𝒉‖1
Reconstruction
𝒁 (𝒊 , 𝒋 ,∗ )≈ 𝑨𝒉 (𝒊 , 𝒋 )
𝒁
• At each pixel of , materials should be even much fewer
![Page 13: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/13.jpg)
Simulation experiments
Balloons Beads Sponges Oil painting
Flowers CD Peppers Face
Spectral image database F. Yasuma, T. Mitsunaga, D. Iso and S. K. Nayar.Generalized assorted pixel camera: Postcapture control of resolution,Dynamic range, and spectrum. IEEE Trans. IP, 19(9):2241-2253, 2010
![Page 14: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/14.jpg)
460 nm 550 nm RGB/620 nm 460 nm 550 nm RGB/620 nm
Input images: Balloons and Beads examples
Ground truths
Reconstruction using component substitution method
Reconstruction by the proposed method
![Page 15: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/15.jpg)
430 nm 490 nm 550 nm 610 nm 670 nm
Input images: Sponges examples
Ground truths
Reconstruction by the proposed method
Error images of the proposed method
![Page 16: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/16.jpg)
RGBimage
![Page 17: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/17.jpg)
GroundTruth(430 nm)
![Page 18: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/18.jpg)
Estimated430 nm
![Page 19: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/19.jpg)
Method Balloons Beads Sponges Oil painting
Flowers CD Peppers Face
CSM[2] 13.9 28.5 19.9 12.2 14.4 13.3 13.7 13.1
Global 6.9/4.7 10.5/8.8 15.4/12.3
5.4/3.8 9.8/8.9 10.3/10.0
7.1/5.9 4.7/3.8
Local win 7.0/4.9 10.6/8.9 14.0/10.6
5.7/4.1 7.5/6.3 9.6/9.2 8.8/8.0 10.9/10.5
RGB clust 6.6/4.3 9.7/7.9 13.6/10.0
5.5/4.0 7.8/6.5 9.1/8.6 8.5/7.6 4.7/3.8
Proposed 3.0/3.0 9.2/9.2 3.7/3.7 4.7/4.7 5.4/5.4 8.2/8.2 4.7/4.7 3.3/3.3
RMSE
Balloons Beads Sponges Oil painting
Flowers CD Peppers Face
![Page 20: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/20.jpg)
![Page 21: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/21.jpg)
![Page 22: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/22.jpg)
HS camera
Filter
CMOSLens Aperture
Focus
Translational stage
![Page 23: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/23.jpg)
Real data experiment
Input RGB Input (550nm) Input (620nm)Estimated (550nm) Estimated (620nm)
![Page 24: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/24.jpg)
Summary•Method to reconstruct high-resolution
hyperspectral image from ▫Low-res hyperspectral camera▫High-res RGB camera
•Spatial sparsity of hyperspectral input▫Search for a factorization of the input into
basis functions set of maximally sparse coefficients
![Page 25: High-resolution Hyperspectral Imaging for Cultural Heritage](https://reader036.vdocuments.mx/reader036/viewer/2022062410/56815b4d550346895dc93311/html5/thumbnails/25.jpg)
Acknowledgement
•This work was in part supported by Microsoft CORE 6 project.