a) Correspondence and requests for materials should be addressed to R. M. (rmenon@eng.utah.edu)

Lensless Photography with only an image sensor

Ganghun Kim,1 Kyle Isaacson,2 Racheal Palmer,2 Rajesh Menon1,a)

1Department of Electrical and Computer Engineering, University of Utah, Salt Lake City, 84111, USA 2Department of Bioengineering, University of Utah, Salt Lake City, 84111, USA

*Corresponding author: rmenon@eng.utah.edu

Photographyusuallyrequiresopticsinconjunctionwitharecordingdevice(animagesensor).Eliminatingtheopticscouldleadtonew

formfactorsforcameras.Here,wereportasimpledemonstrationofimagingusingabareCMOSsensorthatutilizescomputation.The

techniquereliesonthespacevariantpoint-spreadfunctionsresultingfromtheinteractionofapointsourceinthefieldofviewwiththe

imagesensor.Thesespace-variantpoint-spreadfunctionsarecombinedwithareconstructionalgorithminordertoimagesimpleobjects

displayedonadiscreteLEDarrayaswellasonanLCDscreen.Weextendedtheapproachtovideoimagingatthenativeframerateofthe

sensor.Finally,weperformedexperimentstoanalyzetheparametric impactoftheobjectdistance. Improvingthesensordesignsand

reconstructionalgorithmscanleadtousefulcameraswithoutoptics.

Theopticalsystemsofcamerasinmobiledevicestypicallyconstraintheoverallthicknessofthedevices1-2.Byeliminatingtheoptics,itis

possibletocreateultra-thincameraswithinterestingnewformfactors.Previousworkincomputationalphotographyhaseliminatedtheneedfor

lensesbyutilizingaperturesinfrontoftheimagesensor3-7orviacoherentilluminationofthesample8.Intheformercase,theaperturescreate

shadowpatternsonthesensorthatcouldbecomputationallyrecoveredbysolvingalinearinverseproblem9.Thelattercaserequirescoherent

illumination,which isnot generally applicable to imaging. Inmost instances, codedapertureshave replaced the lenses.Microfabricated coded

apertures have recently shown potential for the thinner systems,4 with thickness on the order ofmillimeters. However, these apertures are

absorbingandhence,exhibitrelativelylowtransmissionefficiencies.Anothermethodutilizesholographicphasemasksintegratedontotheimage

sensorinconjunctionwithcomputationtoenablesimpleimaging.10,11Inthiscase,precisemicrofabricationofthemaskontothesensorisrequired.

Anothercomputationalcamerautilizesamicrolensarraytoformalargenumberofpartialimagesofthescene,whichisthennumericallycombined

toformasingleimagewithcomputationalrefocusing12-14.Here,wereportonacomputationalcamerathatiscomprisedofonlyaconventionalimage

sensorandnootherelements.

Ourmotivationforthiscameraisbasedupontherecognitionthatallcamerasessentiallyrelyonthefactthattheinformationaboutthe

objectenterstheapertureofthelens,thecodedapertureormicro-lensarray,andisrecordedbytheimagesensor.Inthecaseofthecodedaperture

andthemicrolensarray,numericalprocessingisperformedtorepresenttheimageforhumanconsumption.Ifallopticalelementsareeliminated,

theinformationfromtheobjectisstillrecordedbytheimagesensor.Ifappropriatereconstructionalgorithmsweredeveloped,theimagerecorded

bythesensorcanbesubsequentlyrecoveredforhumanconsumption.Itisanalogousthemulti-sensorycompressivesensing6wherespatiallight

modulatorisusedtoformmultiplearbitraryaperturesinfrontofsensorarraytorecordthescene,excepttheapertureinthiscaseisasinglestatic

openaperture.

FIG.1.Experimentsetupandprinciplesofoperation.(a)PhotographofthesetupshowingLEDmatrixinfrontoftheimagesensor(top-rightinset).

(b)SensorimagesarerecordedforeachLEDinthearrayseparately.ThreeexemplarycalibrationimagesareshownatthreelocationsoftheLED

pointsource.(c)Totestimagereconstruction,asimplepatternisdisplayedontheLEDmatrix.Theimagecapturedbythesensorisinputintothe

recoveryalgorithmtoreconstructthescene.

Ourtestobjectwasa32x32RGBLEDmatrixpanel(SparkFun,COM-12584)controlledviaanArduinoboard.TheLEDsareuniformly

spacedwithagapof6.1mm.Thetotalmatrixsizeis19cmX19cm. TheimagesensorwasacommerciallyavailableCMOScolorsensor(DFM

22BUC03-LM,ImagingSource).Thesensorhas640x480pixels,eachofwhichisa6μmsquare.Themaximumsupportedframerateofthecamera

viaaUSB2.0interfaceis76framespersecond.AphotographofoursetupisshowninFig.1(a).Thesensorwasplacedatasetdistance,Dfromthe

objectasillustratedinFig.1(b).TherefreshrateoftheLEDarrayis16Hz.Weusedframeaveragingtoeliminatetheeffectofthisrefreshrateaswell

astoenhancethesignaltonoiseratiooftherawdata.

Asthefirststep,weexperimentallycalibratedtheimagingsystembyrecordingtheimageofeachLEDinthearray,formingwhatwerefer

toasthecalibrationmatrix,A.100frameswereaveragedforeachLED.Exemplaryframes(elementsofmatrix,A)areshowninFig.1(b).Extraneous

lightwasminimizedduring all experiments.Whenanarbitrarypattern isdisplayedon theLEDmatrix, the resulting sensor image is a linear

combinationoftheimagesformedbytheindividualLEDs(elementsofthecalibrationmatrix,A).Thesystemcanbedescribedas

𝑏 = 𝐴𝑥 (1)

wherexistheobject,andbisthesensorimage.CalibrationmatrixAcollectedinthissetuphasconditionnumberof10,912,which

suggeststheposedinverseproblemisill-posed,andthereforemostlylikelyfailifsolvedusingadirectmatrixinversion.Hence,wecanattemptto

solvearegularizedlinearinverseproblemtorecovertheobjectxfromthesensordatab,asformulatedbelow:

𝑥 = argmin 𝐴𝑥 − 𝑏 -- + 𝛼- 𝑥 -

- (2)

whereargminrefers to theL2minimization.The regularizationparameter,αcontrolsnoise suppressionand thesmoothnessof the

solution,x.Regularizationprovidesrobustreconstructioncapabilitythatcanwithstandreasonable levelofnoise,asevidencedbythefollowing

demonstrationsoftheexperimentalcondition.

SinceourLEDmatrixhasgreenLEDs,weusedonlythegreenchannelinourexperiments.Butourtechniqueiseasilyextendedtoall

colors.Notethattheexposuretimewaschosenbetween10msand100mstoensurethatthesensorpixelswerenotsaturated.Foreachimage,100

frameswereaveraged.AnexampleoftherawsensorimageandthereconstructedimageareshowninFig.1(c),wheretheobjectistheletter,“T”.

MoreinterestingobjectsareillustratedinFig.2,wherethefirstcolumnshowsthephotographsoftheLEDmatrixusingaconventionalcamera,the

secondcolumnshowstherawsensorimages,andthethirdandfourthcolumnsshowthereconstructedimagesbeforeandafterthresholding.

Usingthissametechnique,wecanperformvideoimagingaswell,asillustratedbyanexamplevideoofajumpingstick-manincludedas

supplementaryvideo1.Thereconstructionprocess,includingallnecessaryimageprocessing,takeslessthan10msperframeusingaregular

desktopPC(IntelCorei7-4790,32GBmemory).Inourcurrentimplementation,theframeratewaslimitedbyouraveragingof100frames,which

takes1to10seconds.

FIG.2.Exemplaryimagestakenwithoursensor.Theleft-columnshowstheobjectsdisplayedontheLEDmatrix.The2ndcolumnshowstheraw

sensorimages.The3rdcolumnshowsthereconstructedimagesbeforeanyprocessing.Therightcolumnshowsthereconstructedimagesafter

binarythresholding.

As isexpectedfromthegeometryof thesystem, thedistancebetweentheobjectandthesensor,D is thekeyparameterthataffects

imagingcharacteristics.IfDistoosmall,thenthefieldofviewisexpectedtodecreasebecauseoflossoflight(andhence,lossofinformation)fromthe

off-axispointsintheobject.Atthesametime,ifDistoolarge,thenthereislossoflight(andhence,lossofinformation)duetothesensorintercepting

onlyasmallfractionoftheemittedlightfromeachpointontheobject.Asaresult,oneexpectsanoptimalvalueofDforimaging.Wedeterminedthis

empiricallybyanalyzingtheperformanceofthesensoratD=85mm,165mm,242mm,343mmand497mm.ForeachvalueofD,weobtainedthe

matrix,A,calculateditssingularvaluesandalsoreconstructedasampleobject(stickman)assummarizedinFig.3(a).ForeachD,wealsoestimated

thefieldofviewbymonitoringtheextentofonePSFonthesensorplane.However,weusedallthecalibratedPSFsforreconstruction.Wenoticed

thatrestrictingthecontributingpointstowithinthefieldofviewdidnotimpactthefinalimages.Wewereabletosuccessfullyrecoverobjectswith

upto32LEDsinthehorizontaldirectionand30LEDsintheverticaldirection.ThemountingplatformblockedthebottomtworowsoftheLED

matrix,which creates thediscrepancybetween thevertical andhorizontaldirections. Enhancement inLEDdiscriminationwasobservedasa

function of increasing distance and led to improved image reconstruction. For the largest value of D, itwas observed that the quality of the

reconstructiondecreasedslightly,possiblyduetothereductionofSNRintherecordedimage,astheamountofphotoncollectedwithinthesensor

surfacedecreasesquadraticallyasafunctionofdistance.

Sensor Image(640 x 480 pixels)

Object(32 x 32 LEDs)

Reconstructed Image Reconstructed ImageAfter Thresholding

Next,weimagedlinesorientedinthevertical,horizontal,anddiagonaldirectionsatthevarioussensor-matrixdistancestoaccesstheeffect

ofdistanceonthefieldofviewandreconstructionquality.TheresultsaresummarizedinFig.3(b).Reconstructionsofverticallineswerebetterthan

thoseofthehorizontallinesatalldistances.All30LEDs(2LEDswereblockedbyourholder)wereeasilyreconstructedintheverticaldirectionatall

distances.Ontheotherhand,atmostonly26LEDswerereconstructedproperlyinthehorizontaldirection.Weattributethisasymmetrytothe

geometryof thesensor,whichhas640pixels in theverticaldirectioncomparedto480 in thehorizontaldirection.Longerdiagonal lineswere

reconstructedatlargervaluesofDasindicatedbytheinsetimagesinFig.3(b).

Lastly,weperformednumericalanalysisoncalibrationmatrixAbasedonsingularvaluedecomposition.First,wecomparedsingular-

value-decayplotforthecalibrationmatrix,AateachvalueofD.Thisplotisanindicationofhowill-posedtheinverseproblemis15.Notethatthe

singular-value-decayplotiscomputedfromthesamecalibrationdatasetusedtoreconstructtheimages.ThisplotshowninFig.3(c)showsthatatD

=85mm,thesingularvaluesdecaymostquickly,whichisconsistentwithourexperimentalresults.Anotherwaytoestimatethedifficultyofimage

reconstructionisbyanalyzingthecorrelationcoefficientmap(Fig3d).EachelementinthecorrelationmatrixisthePearsoncorrelationcoefficient

betweentwocalibrationimages,indicatedbytheverticalandhorizontalindices.Thehorizontalmatrixiscreatedusingasetof32LEDsalongthe

horizontal line inthecenter.Verticalanddiagonalmatricesarecreatedinthesamemanner.Thecorrelationvaluesoftwosampledcalibration

imagesinanydirectionarealwaysnotablylowerthan1,indicatingthatnotwoimageswithinthecalibrationdatabasecloselyresembleoneanother.

Fromthisdistanceanalysis,wedeterminedthatD=343mmismostappropriateforoursetup.ReconstructionresultspresentedinFigs.2

and4werethereforeacquiredwithcalibrationmatrixcollectedatD=343mm.Inadditiontoprovidinginsightastohowthereconstructionperforms

atvaryingD,theaforementionedanalysisimpliesthepossibilitiesforcomputationalrefocusingandreconstructionof3Dinformation.Giventhe

collectionofthecalibrationdatabaseatmultipleobjectdistances,wecouldapplythesamelineardecompositionprocessusedheretouncoverthe3D

spatialinformationofthescene.Inthisexperiment,however,werestrictedourexperimentstothereconstructiononasingle2Dplane,forsimplicity.

FIG.3.Impactofobjectdistance.(a)Reconstructedimagesobtainedatvariousdistances,D.(c)Lengthofalineobjectimagedatvariousdistances,D

inthehorizontal,verticalanddiagonaldirections.InsetsshowreconstructedimagesofdiagonallinescapturedattheD=85,162,and242mm.(d)

Singular-value-decayplotofthecalibrationmatrix,AcapturedatvariousvaluesofD.(e)Correlationmatrixamongthecalibrationimagesfor

D=343mmalongthehorizontal,verticalanddiagonaldirections.

Finally,wereplacedtheLEDarraywithaliquid-crystaldisplay(LCD)asindicatedinFig.4(a).Usingthecalibrationmatrixobtainedwith

theLEDarray,wewerethenabletoreconstructobjectsdisplayedontheLCDandtheseresultsaresummarizedinFig.4.InFig.4(a),3different

geometries(allofcolorgreen,thesameastheLEDsusedforcalibration)areshown.Fig.4(b)showsastarpatternofvariouscolors,allreconstructed

computationally.Itcanbeseenthatthealgorithmhasdifficultyreconstructingtheredstar.Notethatallthereconstructionsutilizethecalibration

datafromthegreenLEDarray.Inallcases,wenotethatreconstructionsarepossible,butoflowerqualitythaninthecaseoftheLEDarray.Finally,

Fig.4(c)showshowthedistanceD(betweentheLCDandthesensor)affecttheimagingofanobjectcomprisedof2greenletters,“HI.”Itis

noteworthythatthesereconstructionswereallperformedwiththesamematrixA,obtainedatD=343mm.TheresultsinFig.4(c)indicatethatthe

depthoffieldoftheimagingsystemcanberelativelylarge,althoughmoreexperimentsarerequiredforpreciselyelucidatingthiseffect.Nevertheless,

Index100 101 102 103

Sing

ular

val

ue

10-4

10-3

10-2

10-1

100 (c) Singluar value decay plot85mm162mm242mm343mm497mm

Distance(mm)100 200 300 400 500

Max

imum

num

ber o

f LED

s

0

5

10

15

20

25

30

35(b) Line test results

HorizontalVerticalDiagonal

D = 85mm D = 165mm D = 242mm D = 343mm D = 497mm(a) Reconstruction of stickman at varying distances

Horizontal 1

0.5

(d) Correlation coefficient maps of 343mm calibration

Vertical Diagonal

thesepreliminaryresultsindicatethefeasibilityofusingjusttheimagesensorformachinevisionorsimilarapplications,whereanthropomorphic

imagequalityislessimportant.

FordifferentlocationsofLEDtobediscernedbythecomputationalprocess,theremustbedistinctfeaturesintherawimagethatuniquely

differentiateseachcalibrationimagefromanother.Inthisexperiment,weobservedthreelocationsonthesensorwheresmalldustparticlescasta

scatteringpatternontothesensor.Therefore,theintensitypatternonthesensorshiftswiththelocationoftheLED.Asanexample,oneelementof

thecalibrationmatrix(theimageofoneLEDatthetop-leftcorneroftheLEDmatrix)isshowninFig.S1.Inthisframe,wenoticedthreeregionswith

scatteringpatternslikelycreatedbydustparticles(indicatedbythesquareoutlines).Ifwecloselyobserveoneoftheseregions(denotedbythered

square)forvariousframes(elementsofA,correspondingtodifferentLEDpositionsinthematrix),wecanseethatthepatternshiftswiththeposition

oftheLEDinthematrix(bottomrow).Thiswouldbethenbeconsistentwiththewell-knownpinspeckcamera16.However,ifwecloselyexaminethe

sameregionsintherawimageoftheextendedobject(withmultipleLEDs),itcanbeseenthatthesescatteringpatternsdisappearbelowthelevelof

human perception as expected from the incoherent superposition of these patterns frommultiple LEDs (see Figs. S2-S4 and supplementary

information17).Therefore,ourimagingsystemclearlydifferfromconventionalpinspeckcamera.

FIG.4.ReconstructionofobjectsonLCD(iPadMini,Apple)screen.(a)PhotographofsetupandreconstructionresultsatD=343mm.(b)

Reconstructionofastarinfourdifferentcolors.Insetsshoworiginaldisplayedimages.(c)Reconstructionofword‘HI’atvaryingobjectlocations.In

allcasesincluding(c),thesamematrix,AforD=343mmwereused.

Ourhypothesisisthattheimagereconstructionissuccessfulbecauseofthecombinedeffectofthesubtleintensityvariationacrossthe

entiresensorintroducedbythediffractionandscatteringcausedbysubtledefectsandirregularitiesonthecoverglass,aswellasthefew(inourcase,

3)visiblescatteringpatternswithinthefieldofviewofthesensor.Itisimportanttonotethatclearlyscattererscouldbedirectlyengineeredontothe

sensortoenhancesuchreconstructionsinthefuture,whichwouldmakeitsimilartobutmuchmorecompactthanmask-basedlenslessimaging

methods.Suchtechniquescouldobviouslybeextendedtonon-imagingcomputationalproblemsincludinginferenceusingdeeplearningandrelated

algorithms18-19.

Inthispaper,wedescribedproof-of-conceptexperimentsthatdemonstratethat lenslessphotography ispossiblewithonlyan image

sensor.Thisisachievedviathenumericalsolutionofaninverseproblemassumingincoherenceoftheobjectandlinearityoftheimage-formation

process.Thetechniqueisappliedtobothstill-frameandvideoimaging.Bycalibratingthesystematvariousplanes,weshowthatitispossibleto

numericallyrefocusandachievequasi-3Dimagingaswell.Inotherwords,ouronly-sensorcameraisalwaysinfocus(withinlimitsofSNR).The

techniqueisreadilyextendedtocolorimaging.Furtherstudiesarenecessarytounderstandthetrade-offsrelatedtoresolution,fieldofview,depthof

fieldandothermetricsrelatedtophotography.

Funding. We gratefully acknowledge funding from the National Science Foundation (Grant # 10037833) and the Utah Science

Technology and Research Initiative. K.I was supported by a University of Utah Nanotechnology Training Program fellowship.

R.P was supported by University of Utah’s Undergraduate Research Opportunities Program.

Acknowledgments. We thank Peng Wang for his help with the regularization algorithm, and Apratim Majumder for thoughtful

discussions.

REFERENCES 1 J. Bareau and P. P. Clark, in Proceedings of International Optical Design Conference (SPIE 2006), 63421F 2 A. Bruckner and M. Schoberl, in Proceedings of MOEMS and Miniaturized Systems XII (SPIE 2013), 861617 3 T. M. Cannon and E. E. Fenimore, Opt. Eng. 19, 193283 (1980) 4 M. S. Asif, A. Ayremlou, A. Sankaranarayanan, A. Veeraraghavan and R. Baraniuk, arXiv 1509, 00116v2 (2016) 5 X. Yuan, H. Jiang, G. Huang and P. Wilford, arXiv 1508, 03498 (2015). 6 H. Jiang, G. Huang, and P. Wilford, APSIPA Transactions on Signal and Information Processing 3, e15 (2014). 7 G. Huang, H. Jiang, K. Matthews, and P. Wilford, in Proceedings of IEEE International Conference on Image Processing (IEEE, 2013), pp. 2101 8 W. Bishara,S. Mavandadi, F. W. Yu, S. Feng, R. Lau, A. Ozcan, Proc. Natl. Acad. Sci. USA 108, 7296-7301 (2011). 9 B. Adcock, A. C. Hansen, C. Poon, and B. Roman, arXiv 1302, 0561 (2014) 10 P. R. Gill and D. G. Stork, in Proceedings of Computational Optical Sensing and Imaging, Alexandria, VA, 2013 11 D. G. Stork and P. R. Gill, International Journal of Advances in Systems and Measurements, 7(3,4), 201-208 (2014) 12 J. Tanida, O. Kumagai, K. Yamada, S. Miyatake, K. Ishida, T. Morimoto, N. Kondou, D. Miyazaki, and Y. Ichioka, Appl. Opt. 40, 1806-1813 (2001) 13 K. Venkataraman, D. Lelescu, J. Duparre, A. McMahon, G. Molina, P. Chatterjee, R. Mullis and S. Nayar, ACM Trans. on Graphics. 32, 166 (2013) 14 G. Kim and R. Menon, Appl. Phys. Lett. 105, 061114 (2014)

15 A. K. Cline, C. B. Moler, G. W. Stewart, and J. H. Wilkinson, SIAM J. Numer. Anal. 16, 368-375 (1979). 16 A. L. Cohen. Optical Acta. 29(1), 63–67 (1982) 17 Supplementary Information 18 K. Kulkarni and P. Turaga, IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(4), 772-784 (2016) 19 H. Chen, S. Jayasuriya, J. Yang, J. Stephen, S. Sivaramakrishnan, A. Veeraraghavan and A. Molnar, arXiv 1605, 03621 (2016).

1

Supplementary Information for Lensless Photography with only an image sensor

Ganghun Kim,1 Kyle Isaacson,2 Rajesh Menon1,a)

1Department of Electrical and Computer Engineering, University of Utah, Salt Lake City, 84111, USA 2Department of Bioengineering, University of Utah, Salt Lake City, 84111, USA

*Corresponding author: rmenon@eng.utah.edu

1. Dust and particle shadow Following four figures are enlarged copy of CMOS images presented in the main text. We marked areas where shadow appeared during the calibration procedure. Shadow completely disappears when an arbitrary object was drawn, while reconstruction was successful. Therefore, we concluded the shadow plays minor role in encoding spatial information. Images are shown in landscape orientation to maximally fit the image.

2

Figure S1. Detailed image of one LED located at the top-left corner shows 3 defects (possibly dust particles) on the sensor. These are indicated by the square outlines. The region marked with the red square is shown in the bottom for different LEDs, clearly indicating a shift of the “defect” scattering pattern.

3

Figure S2. Enlarged CMOS image of the stickman object. Photograph of the LED is shown in the inset.

Object 1. Stickman

4

Figure S3. Enlarged CMOS image of the UTAH object. Photograph of the LED is shown in the inset.

Object 2. Utah

5

Figure S4. Enlarged CMOS image of the space invader object. Photograph of the LED is shown in the inset.

Object 3. Space invader

6

Figure S5. Reconstructions are performed after removing sections of images where particle shadows are casted both from the sensor data. Removed regions are marked by red boxes drawn on one of the calibration image. Results show successful reconstruction of the object without particle shadows, although with increased noise level.

Areas where particle shadows cast

Reconstructions with full CMOS image

Reconstructions without particles shadows

One LED ON atthe center (16,16)

7

2. Imaging with CCD sensor Same imaging technique can be easily extended to other sensor type. We replaced our CMOS sensor with a CCD sensor (Clara Interline CCD, Andor) and successfully reconstructed objects displayed on LED panel.

Figure S6. Lensless imaging performed using CCD sensor array. Objects similar to ones used in CMOS imaging were reconstructed. CCD sensor we used offers higher dynamic range and lower electrical noise, which we believe lead to better reconstruction quality.

3. Supplementary Video 1 shows the animation of a jumping stick-man. Left: original scene. Center: Raw sensor video. Right: Reconstructed video.

CCD Image(1040 x 1040 pixels)

Object on LED Panel (32 x 32 pixels) Reconstructed Image