single pixel imaging of laboratory and natural light...
TRANSCRIPT
UCSD Photonics
Photonics Systems Integration Lab
Computational Optical Sensing and Imaging 2013
Single Pixel Imaging of Laboratory and Natural Light Scenes
Stephen J. Olivas1
Yaron Rachlin2, Lydia Gu2, Brian Gardiner2, Robin Dawson2, Juha-Pekka Laine2, & Joseph Ford1
1Electrical & Computer Engineering Department University of California, San Diego
2Charles Stark Draper Laboratory, Cambridge, MA
UCSD Photonics
Samples taken all at once Exposure time
Compressive Imaging – Overview
scene lens NxN pixel detector
scene lens NxN mask single detector
Traditional Camera
Masked Sensor
Samples taken one at a time Exposure time Large single detector: Faster or specialty sensor (i.e. Photo multiplier tube, Avalanche Photodiode, infrared) Other benefits: image deblurring, depth information
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 2
UCSD Photonics
Samples taken all at once Exposure time
Compressive Imaging – Overview
scene lens NxN pixel detector
scene lens NxN mask single detector
Traditional Camera
Masked Sensor
Samples taken one at a time Exposure time Large single detector: Faster or specialty sensor (i.e. Photo multiplier tube, Avalanche Photodiode, infrared) Other benefits: image deblurring, depth information
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 3
UCSD Photonics
Samples taken all at once Exposure time
Compressive Imaging – Overview
scene lens NxN pixel detector
scene lens NxN mask single detector
Traditional Camera
Masked Sensor
Samples taken one at a time Exposure time Large single detector: Faster or specialty sensor (i.e. Photo multiplier tube, Avalanche Photodiode, infrared) Other benefits: image deblurring, depth information
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 4
UCSD Photonics Compressive Imaging – Overview
scene lens NxN mask single detector
Compressive Imaging
Samples taken one at a time Less samples needed if the image has a sparse representation
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 5
Jun Ke, Premchandra Shankar, and Mark A. Neifeld, “Distributed imaging using an array of compressive cameras,” Optics Communications, Vol.282, pp.185-197, 2008.
Duarte, M.F.; Davenport, M.A.; Takhar, D.; Laska, J.N.; Ting Sun; Kelly, K.F.; Baraniuk, R.G., "Single-Pixel Imaging via Compressive Sampling," Signal Processing Magazine, IEEE , vol.25, no.2, pp.83-91, March 2008
=
UCSD Photonics Compressive Imaging – Overview
scene lens NxN mask single detector
Compressive Imaging
Samples taken one at a time Less samples needed if the image has a sparse representation
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 6
Jun Ke, Premchandra Shankar, and Mark A. Neifeld, “Distributed imaging using an array of compressive cameras,” Optics Communications, Vol.282, pp.185-197, 2008.
Duarte, M.F.; Davenport, M.A.; Takhar, D.; Laska, J.N.; Ting Sun; Kelly, K.F.; Baraniuk, R.G., "Single-Pixel Imaging via Compressive Sampling," Signal Processing Magazine, IEEE , vol.25, no.2, pp.83-91, March 2008
=
UCSD Photonics Compressive Imaging – Overview
scene lens NxN mask single detector
Compressive Imaging
Samples taken one at a time Less samples needed if the image has a sparse representation
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 7
Jun Ke, Premchandra Shankar, and Mark A. Neifeld, “Distributed imaging using an array of compressive cameras,” Optics Communications, Vol.282, pp.185-197, 2008.
Duarte, M.F.; Davenport, M.A.; Takhar, D.; Laska, J.N.; Ting Sun; Kelly, K.F.; Baraniuk, R.G., "Single-Pixel Imaging via Compressive Sampling," Signal Processing Magazine, IEEE , vol.25, no.2, pp.83-91, March 2008
=
UCSD Photonics Compressive Imaging Hardware – Background
Acquires an image by processing sampled projections Signal can be expressed as a linear combination of basis functions
Measurements Y are taken over the set of basis functions (using an DMD)
If S is sparse then only M measurements are need for perfect reconstruction
Richard G. Baraniuk – Rice University, Houston TX “Compressive Imaging,” IEEE Signal Processing pg 118 July 2007
Advantages of Compressive Sensing Low power
Encoder light / decoder heavy
Signal Acquisition and Compression are Combined
Low pixel count exotic sensors for low light or out of band λ’s
Sample Signal Rate proportional to information (Not BW)
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 8
1st Experimental Demo – Rice Univ.
Objective: Perform a systematic test of a hardware CI system
Texas Instrument Digital Mirror Device (DMD) display
UCSD Photonics
Single Pixel Camera • Transform set displayed on DMD illuminated area resolution 1,048,576 pixels (1MPix) • Image output onto detector
GOALS 1) Compare multiple transform basis sets 2) Compare functionality and performance of optics / electronics / algorithms
Single Pixel Camera System Diagram Optics Design
Backlight illuminated object (variable, incoherent, steady) Lenses (collect most light, non-obstructive, resolution) Digital Mirror Device (DMD)
DMD Specs (timing, speed, resolution, size, memory) Detector w/ DAQ
Signal acquisition (timing, speed, bit depth, sensitivity) System Integration
Custom Mounts (alignment, variable) “Solid Works”
C++ code for DMD & DAQ synchronous operation Transform Generation & Reconstruction code
Linear sum reconstruction
Total Variation reconstruction
L1 reconstruction (NESTA solver)
S. Becker, J. Bobin, and E. J. Candès, NESTA: a fast and accurate first-order method for sparse recovery.
SIAM J. on Imaging Sciences 4 (1), 1-39. 8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 9
UCSD Photonics
Discrete/Continuous Discrete Cosine Transform (DCT) (8-bit) Wavelets, Chirplets
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 10
Binary Hadamard Noiselets
Hadamard Transform
Discrete Cosine Transform
+1
-1
0
Noiselet Transform
Transform basis set made up of 1,048,576 patterns to describe 1MPix image space
𝑺 = 𝒇−Ɣ where 1.8< Ɣ < 𝟐.𝟐
Natural Images contain more low spatial frequencies: Power Law Daniel L. Ruderman, ”Origins of Scaling in Natural Images,” Vision Research, 37 (23), 3385–3398 (1997). R. P. Millane, S. Alzaidi, and W. H. Hsiao, ”Scaling and power spectra of natural images,” in Proceedings of Image and Vision Computing New Zealand, D. G. Bailey, ed. (Massey University, Palmerston North, New Zealand, 2003), 148-153. W. H. Hsiao and R. P. Millane, ”Effects of occlusion, edges, and scaling on the power spectra of natural images,” J. Opt. Soc. Am. A, 22 (9), 1789–1797 (2005). Frenkel, G. and Katzav, E. and Schwartz, M. and Sochen, N., ”Distribution of anomalous exponents of natural images,” Phys. Rev. Lett., 97 (10), 103902 – 103906 (2006).
Compressive Imaging – Transform Basis Sets
UCSD Photonics Compressive Imaging Hardware – DMD grayscale operation
Average each subframe & scale by place value
Average all values in entire frame
Erro
r R
amp
Appr
oxim
atio
n
What is the best sampling technique to measure coefficients during Pulse Width Modulation?
The DMD uses Pulse Width Modulation to represent grayscale basis patterns
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 11
DCT
UCSD Photonics
Single-Pixel Compressive Imaging Built testbed & demonstrated working single sensor system Higher resolution & smaller physical volume than previously published work Timing / illumination artifacts need further investigation
Compressive Imaging – Recent Results
4,096 pixels (Rice 2006)
1.04MPix (UCSD-Draper)
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 12
0.7MPix (Inview Corp.) (2012)
M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly and R. G. Baraniuk “Single-Pixel Imaging via Compressive Sampling," IEEE Signal Processing Magazine, 25 (2), 83-91 (2008). L. McMackin, M. A. Herman, B. Chatterjee and M. Weldon, “A high-resolution SWIR camera via compressed sensing," Proc. SPIE 8353, Infrared Technology and Applications XXXVIII, 835303-835313 (2012).
UCSD Photonics Compressive Imaging – Testbed Optics
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 13
Start with same optical configuration to verify performance
UCSD Photonics
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 14
Compressive Imaging – Test Scenes
Ground Truth Images Taken with Canon 5D Mark II SLR
Binary Image Grayscale Image Outdoor Color Image
UCSD Photonics
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 15
Compressive Imaging of Resolution Targets
Noiselet Transform
0.1% 1% 10% 100%
Hadamard Transform
DCT Transform
0.2212 0.1176 0.0480
0.1797 0.0909 0.0330
0.1912 0.1892 0.1125
UCSD Photonics
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 16
Compressive Imaging of Grayscale Scene 1% 10% 100%
Noiselet Transform
Hadamard Transform
DCT Transform
0.0774 0.0625
0.0647 0.0596
0.2383 0.1450
UCSD Photonics
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 17
Compressive Imaging of Grayscale Scene 1% 10% 100%
Noiselet Transform
Hadamard Transform
DCT Transform
0.0774 0.0625
0.0647 0.0596
0.2383 0.1450
UCSD Photonics
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 18
Compressive Imaging of Natural Light Scene
1% Noiselet Transform
Stac
ked
RG
B Im
ages
In
frar
ed Im
ages
(650
-110
0nm
)
Filter the single pixel camera to form color images & infrared image
UCSD Photonics
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 19
Compressive Imaging of Natural Light Scene
1% Noiselet Transform 1% Hadamard Transform 1% DCT Transform
Stac
ked
RG
B Im
ages
In
frar
ed Im
ages
(650
-110
0nm
)
UCSD Photonics
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 20
Canon downsampled 1% resolution (10,000 pixels)
Canon 1% resolution
Upsampled
Compressive Imager (10,000 samples)
using 1% Hadamard Transform
Comparison to Conventional Focal Plane Imager
UCSD Photonics
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 21
Canon downsampled 1% resolution (10,000 pixels)
Canon 1% resolution
Upsampled
Compressive Imager (10,000 samples)
using 1% Hadamard Transform
Comparison to Conventional Focal Plane Imager
UCSD Photonics
𝑆 = 𝑓−Ɣ where 1.8< Ɣ < 2.2 Power Law: Ensemble of natural images contain more low-spatial frequencies. Generally true?
Noiselet sample spatial frequency randomly Hadamard & DCT transforms can be used to target specific spatial frequencies
𝑙𝑙𝑙 𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 ℎ𝑖𝑖ℎ𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 22
Natural Images and the Power Law
UCSD Photonics
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 23
Canon downsampled 1% resolution (10,000 pixels)
Canon 1% resolution
Upsampled
Compressive Imager (10,000 samples)
using 1% Hadamard transform
Comparison to Conventional Focal Plane Imager
UCSD Photonics
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 24
Compressive Imaging of Point Sources 0.1% 1% 10% 100%
Noiselet Transform
Hadamard Transform
DCT Transform
UCSD Photonics
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 25
Canon downsampled 1% resolution (10,000 pixels)
Canon 1% resolution
Upsampled
Compressive Imager (10,000 samples)
using 1% Hadamard Transform
Hadamard locates bright star’s location, dim star not located
Comparison to Conventional Focal Plane Imager
UCSD Photonics
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 26
Canon downsampled 1% resolution (10,000 pixels)
Canon 1% resolution
Upsampled
Compressive Imager (10,000 samples)
using 1% Noiselet Transform
Noiselet locates star’s location with higher resolution, low SNR on dim star
Comparison to Conventional Focal Plane Imager
UCSD Photonics Conclusion: Single-pixel Compressive Imagers
Acknowledgements
8/8/2013 PHOTONIC SYSTEMS INTEGRATION LABORATORY - UCSD JACOBS SCHOOL OF ENGINEERING 27
S. Olivas, Y. Rachlin, L. Gu, B. Gardiner, R. Dawson, J.P. Laine and J. Ford, “Single Compressive Imaging of Laboratory and Natural Light Scenes," in Computational Optical
Sensing and Imaging (2012), CTu1C.2.
S. Olivas, Y. Rachlin, L. Gu, B. Gardiner, R. Dawson, J.P. Laine and J. Ford, “Characterization of a Compressive Imaging System using Laboratory and Natural Light
Scenes," Appl. Opt. 52 19 4515-4526 (2013).
Built an experimental single pixel camera for performance testing • Solve using Linear sum, Least Squares or Total Variation
Characterize basis functions for compatibility with scene and hardware
Current configuration is not practical as a camera since it is slow
Table: 1% operation 10,486 basis patterns and measurements
Thank Charles Stark Draper Laboratory for funding under the University R&D program Robin Dawson, JP Laine, Christopher Yu, Brian Gardner, Lydia Gu, Yaron Rachlin, and Piotr Indyk for collaboration.
Compressive imaging produces images of comparable quality as traditional cameras
Compressive imaging’s main benefit lies in Feature Specific Imaging where a small number of basis functions are needed to reach a conclusive measurement.