new opportunities: application of lf - mit media...

New opportunities:Application of LF

Se Baek Oh & Ramesh Raskar

Friday, June 19, 2009

Se Baek Oh 3D OpticalSystems Group CVPR 2009 - Light Fields: Present and Future

• LF is a very powerful tool to understand wave-related phenomena

• and potentially design and develop new system and applications

2



Outlinewavefront coding holography rendering

gaussian beam rotating PSF

Online Submission ID: 0344

the screen was very large. As expected, we see (Fig. 9) the typical315

Fraunhofer diffraction pattern.

Figure 9: Diffraction from a square aperture. An animated versionof this experiment with varying the aperture size appears in the sup-plementary material as a video. The distance from the aperture tothe screen is 1 m.

316

Double rectangular apertures: Next we created two rectangu-317

lar apertures and probe them with the AMP. Note that we observe

Figure 10: Diffraction from double square apertures. Both arecentered at y = 0 (horizontal line running through the middle of theimage), the width and height of the aperture is 10 µm and 50 µm,respectively. The separation between the two apertures along thex–direction is 100 µm. The distance from the aperture to the screenis 1 m.

318

strong interference modulation along the x–direction but the side319

lobes of the diffraction pattern along the y–direction.320

In all the above cases, the input light was a plane wave normal to321

the screen axis. In the teaser, we showed how the Mona Lisa image322

would look if it were the source of light, and we were to see it323

through a cloth fabric.324

4.2 Diffraction and Refraction due to Phase Occluders325

We consider rendering diffraction from a lens. The finite aperture326

and the phase variation of the lens creates the Airy disk. For a327

single lens imager shown in Fig. 11, an ideal point spread function328

is an infinitesimally small point if diffraction is neglect. However,329

it is well known that the point spread function is indeed the Airy330

disk due to diffraction by the aperture. Here we explain how the331

augmented LF allows us to model diffraction.332

A lens, focal length f and aperture size A, can be decomposed to333

be a pure phase mask of quadratic phase variation (i.e., quadratic334

change in optical path difference due to the thickness of the lens as335

a function of x) and an amplitude mask of a rectangular aperture336

as shown in Fig. 11(a). Figure 11(b) shows how the augmented LF337

changes throughout the system. The LF of a point source at x = x0338

is !(x ! x0) at the object plane and is sheared along the x–axis339

by the propagation to the lens. By the LF transformer shown in340

Fig. 11(a), the augmented LF transmitted the lens is a tilted pat-341

tern with some negative radiance values; the quadratic phase of the342

lens induces the tilt, and the finite aperture produces radiance vari-343

ations. Then, the ALF is sheared again along the x–axis by the344

second propagation to the image plane. Integrating the augmented345

LF along the "–axis, we obtain the intensity of the point spread346

function, which is the Airy pattern in the flat land.347

Figure 11: Point spread function (Airy pattern) of a single lens im-ager. (a) ALF transformer of the phase and amplitude componentsof a lens with focal length f , (b) ALF shape as the propagationthrough the system. Note that due to the finite size aperture, thePSF is the airy pattern. Negative radiance is shown in blue.

0 0.01 0.02 0.03 0.04 0.05 0.060

0.2

0.4

0.6

0.8

1

x [mm]

! = 650 nm

! = 530 nm

! = 480 nm

Figure 12: The Airy disk introduces blur even in diffracted–limitedoptical imaging systems. (left): original object, (middle): imageblurred by the Airy disk, and (right) half profile of the intensity ofthe RGB channels, where f = 100 mm with f/8, 256"256 pixelsand pixel size is 0.5 µm.

4.3 Performance348

All our experiments have been performed on a Dual-Core AMD349

processor with 2GB RAM endowed with nVidia G80 GPU. As350

seen from the pseudocode mentioned in Section ??, the big-Oh351

complexity of the Augmented Photon Mapping remains the same,352

6

1.1. Double-helix point spread function (DH-PSF)

A DH-PSF system can be implemented by introducing a phase mask in the Fourier plane of an otherwise standard imaging system. The phase mask is designed such that its transmittance function generates a rotating pattern in the focal region of a Fourier transform lens [15-18]. Specifically, the DH-PSF exhibits two lobes that spin around the optical axis as shown in Fig. 1(a). Note that DH-PSF displays a significant change of orientation with defocus over an extended depth. In contrast, the standard PSF presents a slowly changing and expanding symmetrical pattern throughout the same region [Fig. 1(b)].

Fig. 1. Comparison of the (a) DH-PSF and the (b) standard PSF at different axial planes for a system with 0.45 numerical aperture (NA) and 633nm wavelength.

While analytical solutions for helical beams provide valuable insight on wave propagation

[17] and can be used in photon-unlimited applications [16], they do not provide the high-efficiency transfer functions required for photon-limited systems. Hence, we use a design that confines the helical pattern to a specific axial range of interest to attain high efficiency systems [15]. Unlike standard and astigmatic PSFs, the DH-PSF concentrates its energy in its two main lobes throughout this range of operation, and is consequently well suited for photon-limited applications.

2. Cramer-Rao bounds in photon limited systems

The position estimation accuracy of a PSF in the presence of noise can be quantified by computing its Cramer-Rao bound (CRB). The CRB of a PSF represents the lowest possible position estimation variance that can be achieved by an unbiased estimator based on that PSF [14,16,19-21]. CRBs are computed for different noise conditions by appropriately choosing the noise distribution and the noise level relative to the signal level. For the 3D position estimation problem, CRBs for the X, Y, and Z dimensions are obtained from the diagonal elements of the inverse of the 3x3 Fisher information matrix I [14, 19], which is calculated as follows:

!! !!!

"=!

!!

!=

j,i

i,j2

j,i

j,ij,i

]n[]m[

)k|(plnE

]n[

)|k(pln

]m[

)|k(plnE]m,n[I ,

(1)

where = [X, Y, Z], the indices m and n are either 1, 2, or 3, E is the expectation, and pi,j(k|

) is the probability density function for the pixel in ith row and jth column. For a given noise level, the position localization accuracy of a PSF is best when the intensity of the PSF spans the dynamic range of the detector. Because the energy of the rotating PSF is distributed in two of its main lobes, for a given photon budget and NA, the

#102517 - $15.00 USD Received 8 Oct 2008; revised 16 Dec 2008; accepted 17 Dec 2008; published 19 Dec 2008

(C) 2008 OSA 22 December 2008 / Vol. 16, No. 26 / OPTICS EXPRESS 22050

3



Wavefront coding

• ALF of a phase mask(slowly varying ϕ(x))

T (x, !) = "

!! ! #

2$

%&

%x

"

4

conventional wavefront coding

extended DOF (w/ deconvolution)



HolographyRecording

hologram

Reconstruction

object

5



Holography

laser

object wave

Recording

hologram

Reconstruction

object

5



Holography

laser

object wave

reference wave

Recording

hologram

Reconstruction

object

5



Holography

laser

object wave

reference wave

Recording

hologram

Reconstruction

hologram

object

5



Holography

laser

object wave

reference wave

Recording

hologram

Reconstruction

reference wave

hologram

object

5



Holography

laser

object wave

reference wave

Recording

hologram

Reconstruction

reference wave

hologram

observer

virtual image

object

5



Holography

laser

object wave

reference wave

Recording

hologram

Reconstruction

reference wave

hologram

observer

real image

virtual image

object

5



Holography

recording

reconstruction

• For a point object

6



• Using virtual light sources in photon mapping

Rendering Online Submission ID: 0344

the screen was very large. As expected, we see (Fig. 9) the typical315

Fraunhofer diffraction pattern.

Figure 9: Diffraction from a square aperture. An animated versionof this experiment with varying the aperture size appears in the sup-plementary material as a video. The distance from the aperture tothe screen is 1 m.

316

Double rectangular apertures: Next we created two rectangu-317

lar apertures and probe them with the AMP. Note that we observe

Figure 10: Diffraction from double square apertures. Both arecentered at y = 0 (horizontal line running through the middle of theimage), the width and height of the aperture is 10 µm and 50 µm,respectively. The separation between the two apertures along thex–direction is 100 µm. The distance from the aperture to the screenis 1 m.

318

strong interference modulation along the x–direction but the side319

lobes of the diffraction pattern along the y–direction.320

In all the above cases, the input light was a plane wave normal to321

the screen axis. In the teaser, we showed how the Mona Lisa image322

would look if it were the source of light, and we were to see it323

through a cloth fabric.324

4.2 Diffraction and Refraction due to Phase Occluders325

We consider rendering diffraction from a lens. The finite aperture326

and the phase variation of the lens creates the Airy disk. For a327

single lens imager shown in Fig. 11, an ideal point spread function328

is an infinitesimally small point if diffraction is neglect. However,329

it is well known that the point spread function is indeed the Airy330

disk due to diffraction by the aperture. Here we explain how the331

augmented LF allows us to model diffraction.332

A lens, focal length f and aperture size A, can be decomposed to333

be a pure phase mask of quadratic phase variation (i.e., quadratic334

change in optical path difference due to the thickness of the lens as335

a function of x) and an amplitude mask of a rectangular aperture336

as shown in Fig. 11(a). Figure 11(b) shows how the augmented LF337

changes throughout the system. The LF of a point source at x = x0338

is !(x ! x0) at the object plane and is sheared along the x–axis339

by the propagation to the lens. By the LF transformer shown in340

Fig. 11(a), the augmented LF transmitted the lens is a tilted pat-341

tern with some negative radiance values; the quadratic phase of the342

lens induces the tilt, and the finite aperture produces radiance vari-343

ations. Then, the ALF is sheared again along the x–axis by the344

second propagation to the image plane. Integrating the augmented345

LF along the "–axis, we obtain the intensity of the point spread346

function, which is the Airy pattern in the flat land.347

Figure 11: Point spread function (Airy pattern) of a single lens im-ager. (a) ALF transformer of the phase and amplitude componentsof a lens with focal length f , (b) ALF shape as the propagationthrough the system. Note that due to the finite size aperture, thePSF is the airy pattern. Negative radiance is shown in blue.

0 0.01 0.02 0.03 0.04 0.05 0.060

0.2

0.4

0.6

0.8

1

x [mm]

! = 650 nm

! = 530 nm

! = 480 nm

Figure 12: The Airy disk introduces blur even in diffracted–limitedoptical imaging systems. (left): original object, (middle): imageblurred by the Airy disk, and (right) half profile of the intensity ofthe RGB channels, where f = 100 mm with f/8, 256"256 pixelsand pixel size is 0.5 µm.

4.3 Performance348

All our experiments have been performed on a Dual-Core AMD349

processor with 2GB RAM endowed with nVidia G80 GPU. As350

seen from the pseudocode mentioned in Section ??, the big-Oh351

complexity of the Augmented Photon Mapping remains the same,352

6

rectangular aperture

screen

white light

Augmented Photon Mapping for Wavefront Transmission EffectsS. B. Oh et al. (2009)

7



Gaussian Beam

• Beam from a laser

• a solution of paraxial wave equation

8



• ALF (and WDF) of the Gaussian Beam is also Gaussian in x-θ space

Gaussian Beam

x

!

z

x

9



Gaussian Beamx-θ space z-x space

10



Rotating PSF (DH PSF)

1.1. Double-helix point spread function (DH-PSF)

A DH-PSF system can be implemented by introducing a phase mask in the Fourier plane of an otherwise standard imaging system. The phase mask is designed such that its transmittance function generates a rotating pattern in the focal region of a Fourier transform lens [15-18]. Specifically, the DH-PSF exhibits two lobes that spin around the optical axis as shown in Fig. 1(a). Note that DH-PSF displays a significant change of orientation with defocus over an extended depth. In contrast, the standard PSF presents a slowly changing and expanding symmetrical pattern throughout the same region [Fig. 1(b)].

Fig. 1. Comparison of the (a) DH-PSF and the (b) standard PSF at different axial planes for a system with 0.45 numerical aperture (NA) and 633nm wavelength.

While analytical solutions for helical beams provide valuable insight on wave propagation

[17] and can be used in photon-unlimited applications [16], they do not provide the high-efficiency transfer functions required for photon-limited systems. Hence, we use a design that confines the helical pattern to a specific axial range of interest to attain high efficiency systems [15]. Unlike standard and astigmatic PSFs, the DH-PSF concentrates its energy in its two main lobes throughout this range of operation, and is consequently well suited for photon-limited applications.

2. Cramer-Rao bounds in photon limited systems

The position estimation accuracy of a PSF in the presence of noise can be quantified by computing its Cramer-Rao bound (CRB). The CRB of a PSF represents the lowest possible position estimation variance that can be achieved by an unbiased estimator based on that PSF [14,16,19-21]. CRBs are computed for different noise conditions by appropriately choosing the noise distribution and the noise level relative to the signal level. For the 3D position estimation problem, CRBs for the X, Y, and Z dimensions are obtained from the diagonal elements of the inverse of the 3x3 Fisher information matrix I [14, 19], which is calculated as follows:

!! !!!

"=!

!!

!=

j,i

i,j2

j,i

j,ij,i

]n[]m[

)k|(plnE

]n[

)|k(pln

]m[

)|k(plnE]m,n[I ,

(1)

where = [X, Y, Z], the indices m and n are either 1, 2, or 3, E is the expectation, and pi,j(k|

) is the probability density function for the pixel in ith row and jth column. For a given noise level, the position localization accuracy of a PSF is best when the intensity of the PSF spans the dynamic range of the detector. Because the energy of the rotating PSF is distributed in two of its main lobes, for a given photon budget and NA, the

#102517 - $15.00 USD Received 8 Oct 2008; revised 16 Dec 2008; accepted 17 Dec 2008; published 19 Dec 2008

(C) 2008 OSA 22 December 2008 / Vol. 16, No. 26 / OPTICS EXPRESS 22050

1µm 1µm

3D positions

1

2

3

4

5

standard PSF DH PSF

Courtesy of S. R. P. PavaniU. of Colorado@Boulder

Prof. Rafael Piestun’s groupUniv. of Colorado@Boulder

11



Reference• “Wave propagation with rotating intensity distributions,” Y. Y. Schechner, R. Piestun,

and J. Shamir, Phys. Rev. E 54: R50–R53 (1996)• “Wave fields in three dimensions: analysis and synthesis,” R. Piestun, B. Spektor, and

J. Shamir, J. Opt. Soc. Am. A 13:1837-1848 (1996)• “Propagation-invariant wave fields with finite energy,” R. Piestun, Y. Y. Schechner, and

J. Shamir, J. Opt. Soc. Am. A 17:294-303 (2000)

• "Depth from diffracted rotation," A. Greengard, Y. Y. Schechner, and R. Piestun, Opt. Lett., 31(2):181-183, (2006)

• "High-efficiency rotating point spread functions", S. R. P. Pavani and R. Piestun, Opt. Express, 16(5):3484-3489, (2008)

• “Three-Dimensional Single-Molecule Fluorescence Imaging Beyond the Diffraction Limit Using a Double-Helix Point Spread Function,” S. R. P. Pavani, M. A. Thompson, J. S. Biteen, S. J. Lord, N. Liu, R. I. Twieg, R. Piestun, and W. E. Moerner, PNAS, 106: 2995, (2009)

• “Three-dimensional localization with nanometer accuracy using a detector-limited double-helix point spread function system, “ S. R. P. Pavani, A. Greengard, and R. Piestun, APL (2009) In Press

Concept

Implementation

Microscope

12



Gauss-Laguerre modeU(r, t) = u(r) exp [i(kz ! !t)]

unm(r) = G(!, z)Rnm(!)!m(")Zn(z)

G(!, z) =w0

w(z)exp

!!!2

"exp

!i!2z

"exp (!i"(z))

!(!) = exp(im!)

w0

w(z) = w0

!1 + z2

"1/2

!2w0

z0

! =!

w(z) z =z

z0

z0 =!w2

0

"Rnm(!) =

!!2!

"|m|L|m|

(n!|m|)/2(2!2)

Zn(z) = exp {!in!(z)}

!(z) = arctan(z) : Gouy phase

Orthogonal basis in the cylindrical coordinate

(0,0): Gaussian beam

13



Rotating PSF• Rotating beams

• Superposition along a straight line

• Rotation rate related to slope of line

• Both intensity and phase rotate

• Maximum rotation rate in Rayleigh range

-10 -5 0 5 10

10

5

0

m

n

GL modal plane

intensity

Courtesy of S. R. P. Pavani

14



HER-PSF

57.01%

Rotating PSF

1.84% Courtesy of S. R. P. Pavani

Rotating PSF

15



Conceptually...

zx

y

16



Conceptually...

zx

y

17



Conceptually...

z

other orders need to be balanced...

x

y

17



WDF (ALF) of (1,1) order

-10 -5 0 5 10

10

5

0

m

n

GL modal plane

intensity

R. Simon and G. S. Agarwal, "Wigner representation of Laguerre-Gaussian beams", Opt. Lett., 25(18), (2000)

18



intensity in x-y

y

x

19



intensity in x-y

y

x θx

θy

WDF in θx- θy

19



intensity in x-y

y

x θx

θy

WDF in θx- θy

θx

θy

WDF in θx- θy

19



θx

θy

WDF in θx- θy

intensity in x-y

y

x θx

θy

WDF in θx- θy

θx

θy

WDF in θx- θy

19



Future direction

• Reflectance (e.g. BRDR/BTF) model

• Tomography & Inverse problems

• Beam shaping/phase mask design by ray-based optimization

• New processing w/ virtual light source

20



Summary

• big pictures for all light field representations

21



Propertyconstant along rays

non-negativity

coherence wavelength cross term

traditional light field

always constant

always positive

only incoherent zero no

observable light field

nearly constant

always positive

any coherence state

for any wavelength

has cross term

augmented light field

only in the paraxial region

positive and negative

any coherence state

for any wavelength

has cross term

WDF only in the paraxial region

positive and negative

any coherence state

for any wavelength

has crossterm

Rihaczek no; linear drift

complex any coherence state

for any wavelength

reduced cross terms

22



benefits & limitationsability to propagate

representing wave optics

simplicity of computation

adaptability to current pipe line

near field far field

traditional light field x-shear no very

simple high no yes

observable light field

not x-shear yes modest low yes yes

augmented light field

x-shear yes modest high no yes

WDF x-shear yes modest low yes yes

Rihaczek x-shear yesbetter than WDF, not as simple as LF

low no yes

23



Conclusions

• Wave optics phenomena can be understood with geometrical ray based representation

• There are many different phase-space representations

• We hope to inspire researchers in computer vision/graphics as well as in optics graphics to develop new tools and algorithms based on joint exploration of geometric and wave optics concepts

24


new opportunities: application of lf - mit media...

Documents