Logarithmic CMOS Logarithmic CMOS image sensorsimage sensors

Dr. Dileepan JosephDr. Dileepan Joseph

Dept. of Engineering ScienceDept. of Engineering Science

University of Oxford, UKUniversity of Oxford, UK

OutlineOutline

MotivationMotivation

BackgroundBackground

MethodMethod

ConclusionsConclusions

Future workFuture work

Motivation: socialMotivation: social

Society has invested over many millennia in Society has invested over many millennia in developing technology to record observed developing technology to record observed scenes on an independent mediumscenes on an independent mediumArtistic license aside, the aim has been to render Artistic license aside, the aim has been to render images with a maximum of perceptual accuracy images with a maximum of perceptual accuracy using a minimum of effortusing a minimum of effortThe digital camera is a culmination of the above The digital camera is a culmination of the above but its development is far from complete…but its development is far from complete…Although digital cameras have in many ways Although digital cameras have in many ways surpassed film cameras, human vision routinely surpassed film cameras, human vision routinely outperforms the best camerasoutperforms the best cameras

Motivation: economicMotivation: economic

A digital camera consists of many components A digital camera consists of many components (optics, housing, battery, memory etc.), of which (optics, housing, battery, memory etc.), of which the image sensor is considered principalthe image sensor is considered principal

With market revenues of $1.7 billion in 2003, With market revenues of $1.7 billion in 2003, there is widespread research and development there is widespread research and development in a variety of image sensor designsin a variety of image sensor designs

Modern designs may be either charge coupled Modern designs may be either charge coupled device (CCD) sensors or complementary metal-device (CCD) sensors or complementary metal-oxide-semiconductor (CMOS) sensorsoxide-semiconductor (CMOS) sensors

Motivation: technologicalMotivation: technological

CriterionCriterion Human eyeHuman eye Film photoFilm photo CCD sensorCCD sensor CMOS sensorCMOS sensor

Pixel pitchPixel pitch 22––3 3 μμmm 1010––20 20 μμmm 55––10 10 μμmm 55––10 10 μμmm

Image pitchImage pitch 3 cm3 cm Film sizeFilm size 1 mm1 mm––11 cm11 cm 1 mm1 mm––2 cm2 cm

Dynamic rangeDynamic range 22––5 decades5 decades 11––4 decades4 decades 4 decades4 decades 33––5 decades5 decades

Max. frame rateMax. frame rate ≈≈ 15 Hz15 Hz 1 shot only1 shot only 10 kHz10 kHz >> 10 kHz>> 10 kHz

Pre-processingPre-processing ExtensiveExtensive NoneNone NoneNone PossiblePossible

Unit priceUnit price InvaluableInvaluable € € 0.10.1 € € 100100 € € 1010

Background: CCD image sensorBackground: CCD image sensor

Marches photo generated Marches photo generated charge systematically charge systematically from an array of pixels to from an array of pixels to an output amplifieran output amplifier

Established technologyEstablished technology

High resolution, high High resolution, high sensitivity, low noisesensitivity, low noise

Fabrication process is Fabrication process is optimised for imagingoptimised for imaging

Market share of 93% in Market share of 93% in 1999 (49% in 2004?)1999 (49% in 2004?)

Background: CMOS image sensorBackground: CMOS image sensor

Works like memory array Works like memory array with photosensitive pixels with photosensitive pixels instead of memory cellsinstead of memory cells

Signal processing may be Signal processing may be included on the same dieincluded on the same die

High yield and good video High yield and good video performanceperformance

May be fabricated by the May be fabricated by the makers of microchipsmakers of microchips

Market share of 7% in Market share of 7% in 1999 (51% in 2004?)1999 (51% in 2004?)

Background: linear pixelsBackground: linear pixels

Linear pixels (CCD or Linear pixels (CCD or CMOS) integrate CMOS) integrate photons over discrete photons over discrete periods of timeperiods of timeThey produce a They produce a voltage directly voltage directly proportional to the proportional to the light intensitylight intensityThe response may The response may saturate white or saturate white or black easilyblack easily

Background: logarithmic pixelsBackground: logarithmic pixels

Logarithmic pixels Logarithmic pixels (CMOS only) can (CMOS only) can measure photon flux measure photon flux continuouslycontinuouslyThey produce a They produce a voltage proportional voltage proportional to the logarithm of to the logarithm of light intensitylight intensityThe response is The response is similar to that of similar to that of human visionhuman vision

Background: image qualityBackground: image quality

Images are noisy with Images are noisy with logarithmic sensorslogarithmic sensors

Colour is worse than Colour is worse than with linear sensorswith linear sensors

Quality improves with Quality improves with digital processingdigital processing

No comprehensive No comprehensive treatment of either treatment of either problem or solutionproblem or solution

Method: theoryMethod: theory

Model logarithmic CMOS image sensors Model logarithmic CMOS image sensors using optical & integrated circuit theoryusing optical & integrated circuit theory

Use the model to hypothesize the cause Use the model to hypothesize the cause and solution of image quality problemsand solution of image quality problems

Calibrate the model and test hypotheses Calibrate the model and test hypotheses using constrained regression theoryusing constrained regression theory

Optimise digital image processing using Optimise digital image processing using multilinear (or array) algebramultilinear (or array) algebra

Method: simulationMethod: simulation

Simulation of integrated circuits is more Simulation of integrated circuits is more accurate than a theoretical analysisaccurate than a theoretical analysisCost of simulation in time and money is Cost of simulation in time and money is small compared to that of experimentsmall compared to that of experimentIntegrated circuits may be studied under Integrated circuits may be studied under controlled and well-defined conditionscontrolled and well-defined conditionsInternal states and variables may be Internal states and variables may be observed without specialised equipment, observed without specialised equipment, circuit disruption and/or foresightcircuit disruption and/or foresight

Method: experimentMethod: experiment

Experiments were performed using a Fuga Experiments were performed using a Fuga 15RGB camera from C-Cam Technologies15RGB camera from C-Cam Technologies

The camera was operated from a portable The camera was operated from a portable PC via a custom Windows applicationPC via a custom Windows application

The image sensor had 512 The image sensor had 512 × 512 pixels × 512 pixels and a full frame rate of about 8 Hzand a full frame rate of about 8 Hz

Until recently, it was the most successful Until recently, it was the most successful commercial logarithmic image sensorcommercial logarithmic image sensor

Conclusions: fixed pattern noiseConclusions: fixed pattern noise

yy = = aa + + bb ln ( ln (cc + + xx) + ) + εε for illuminance for illuminance xx and and response response yy of a pixel of a pixel

Variation of offset Variation of offset aa, , gain gain bb, bias , bias cc or a or a combination thereof combination thereof causes FPNcauses FPN

Calibration possible Calibration possible within limits of the within limits of the stochastic error stochastic error εε

Conclusions: fixed pattern noiseConclusions: fixed pattern noise

Left to right: FPN Left to right: FPN correction for single, correction for single, double and triple double and triple variation modelsvariation modelsTop to bottom: two Top to bottom: two decade attenuation of decade attenuation of illuminance in halfilluminance in half decadedecade steps stepsInter-scene plus intra-Inter-scene plus intra-scene dynamic range scene dynamic range of 3.5 decadesof 3.5 decades

Conclusions: transient responseConclusions: transient response

The transient response of The transient response of a pixel is fast enough for a pixel is fast enough for most applicationsmost applications

Greater demands are Greater demands are placed on the row and placed on the row and column readoutscolumn readouts

Premature digitization Premature digitization results in a predictable results in a predictable non-uniformity or FPNnon-uniformity or FPN

Affects only a few rows Affects only a few rows due to slow scanningdue to slow scanning

Conclusions: transient responseConclusions: transient response

Premature digitization is Premature digitization is more serious for column more serious for column readout due to speedreadout due to speed

For example, columns For example, columns need scanning at 100 need scanning at 100 MHz for HDTV videoMHz for HDTV video

Column-to-column gain Column-to-column gain variation is caused by variation is caused by transient responsetransient response

Resolve with careful Resolve with careful timing and designtiming and design

Conclusions: temperature Conclusions: temperature dependencedependence

Unlike with humans, Unlike with humans, digital cameras do not digital cameras do not regulate temperatureregulate temperature

Hence, responses to Hence, responses to illuminance depend illuminance depend on temperatureon temperature

When temperature When temperature dependence varies dependence varies from pixel to pixel, from pixel to pixel, FPN occursFPN occurs

Conclusions: temperature Conclusions: temperature dependencedependence

The dark response of The dark response of a pixel depends only a pixel depends only on temperatureon temperatureIt may be used to It may be used to correct FPN due to correct FPN due to temperature in the temperature in the light responselight responseExperiments support Experiments support this conclusion but this conclusion but simulation results are simulation results are shown for clarityshown for clarity

Conclusions: colour renditionConclusions: colour rendition

Combine the theories of Combine the theories of colour linear sensors and colour linear sensors and b/w logarithmic sensorsb/w logarithmic sensorsCalibrate FPN, using Calibrate FPN, using images of uniform stimuli, images of uniform stimuli, by a by a relativerelative analysis analysisCalibrate colour, using Calibrate colour, using images of a colour chart, images of a colour chart, by an by an absoluteabsolute analysis analysisFuga 15RGB competes Fuga 15RGB competes with conventional digital with conventional digital cameras (which have a cameras (which have a perceptual error of 15)perceptual error of 15)

Conclusions: colour renditionConclusions: colour rendition

Image of a colour chart, Image of a colour chart, in 11 lux of illuminance, in 11 lux of illuminance, was rendered using was rendered using calibrated modelscalibrated models

Single, double and triple Single, double and triple variation results and ideal variation results and ideal colours are showncolours are shown

As with vision, rendition As with vision, rendition improves in brighter improves in brighter lighting and worsens in lighting and worsens in dimmer lightingdimmer lighting

Future workFuture work

Digital cameras aim to render images with Digital cameras aim to render images with a maximum of perceptual accuracy using a maximum of perceptual accuracy using a minimum of efforta minimum of effortBy modelling and calibrating logarithmic By modelling and calibrating logarithmic CMOS image sensors, problems with CMOS image sensors, problems with image quality may be solvedimage quality may be solvedPast work has focused on maximising Past work has focused on maximising perceptual accuracy but future work will perceptual accuracy but future work will focus on minimising effortfocus on minimising effort

Future workFuture work

Shrinking feature Shrinking feature sizes may be used to sizes may be used to improve imagingimprove imaging

There are challenges There are challenges with deep submicron with deep submicron CMOS processes that CMOS processes that need overcomingneed overcoming

What about industrial What about industrial and biomedical uses and biomedical uses of the technology?of the technology?

AcknowledgementsAcknowledgements

This work was funded thanks to the engineering This work was funded thanks to the engineering research councils of both Canada and the UKresearch councils of both Canada and the UK

Thanks also to colleagues at the Microelectronic Thanks also to colleagues at the Microelectronic Circuits and Analogue Devices research groupCircuits and Analogue Devices research group