Download - Research 2003
-
7/31/2019 Research 2003
1/24
Logarithmic CMOSLogarithmic CMOS
image sensorsimage sensorsDr. Dileepan JosephDr. Dileepan Joseph
Dept. of Engineering ScienceDept. of Engineering ScienceUniversity of Oxford, UKUniversity of Oxford, UK
-
7/31/2019 Research 2003
2/24
OutlineOutline
MotivationMotivation
BackgroundBackground
MethodMethod
ConclusionsConclusions
Future workFuture work
-
7/31/2019 Research 2003
3/24
Motivation: socialMotivation: social
Society has invested over many millennia inSociety has invested over many millennia indeveloping technology to record observeddeveloping technology to record observedscenes on an independent mediumscenes on an independent medium
Artistic license aside, the aim has been to renderArtistic license aside, the aim has been to render
images with a maximum of perceptual accuracyimages with a maximum of perceptual accuracyusing a minimum of effortusing a minimum of effort
The digital camera is a culmination of the aboveThe digital camera is a culmination of the abovebut its development is far from completebut its development is far from complete
Although digital cameras have in many waysAlthough digital cameras have in many wayssurpassed film cameras, human vision routinelysurpassed film cameras, human vision routinelyoutperforms the best camerasoutperforms the best cameras
-
7/31/2019 Research 2003
4/24
Motivation: economicMotivation: economic
A digital camera consists of many componentsA digital camera consists of many components
(optics, housing, battery, memory etc.), of which(optics, housing, battery, memory etc.), of whichthe image sensor is considered principalthe image sensor is considered principal
With market revenues of $1.7 billion in 2003,With market revenues of $1.7 billion in 2003,there is widespread research and developmentthere is widespread research and development
in a variety of image sensor designsin a variety of image sensor designs
Modern designs may be either charge coupledModern designs may be either charge coupled
device (CCD) sensors or complementary metal-device (CCD) sensors or complementary metal-
oxide-semiconductor (CMOS) sensorsoxide-semiconductor (CMOS) sensors
-
7/31/2019 Research 2003
5/24
Motivation: technologicalMotivation: technological
Criterion Human eye Film photo CCD sensor CMOS sensor
Pixel pitch 23 m 1020 m 510 m 510 m
Image pitch 3 cm Film size 1 mm11 cm 1 mm2 cm
Dynamic range 25 decades 14 decades 4 decades 35 decades
Max. frame rate 15 Hz 1 shot only 10 kHz >> 10 kHz
Pre-processing Extensive None None Possible
Unit price Invaluable 0.1 100 10
-
7/31/2019 Research 2003
6/24
Background: CCD image sensorBackground: CCD image sensor
Marches photo generatedMarches photo generatedcharge systematicallycharge systematicallyfrom an array of pixels tofrom an array of pixels to
an output amplifieran output amplifier
Established technologyEstablished technology
High resolution, highHigh resolution, highsensitivity, low noisesensitivity, low noise
Fabrication process isFabrication process isoptimised for imagingoptimised for imaging
Market share of 93% inMarket share of 93% in1999 (49% in 2004?)1999 (49% in 2004?)
-
7/31/2019 Research 2003
7/24
Background: CMOS image sensorBackground: CMOS image sensor
Works like memory arrayWorks like memory arraywith photosensitive pixelswith photosensitive pixelsinstead of memory cellsinstead of memory cells
Signal processing may beSignal processing may be
included on the same dieincluded on the same die
High yield and good videoHigh yield and good videoperformanceperformance
May be fabricated by theMay be fabricated by themakers of microchipsmakers of microchips
Market share of 7% inMarket share of 7% in1999 (51% in 2004?)1999 (51% in 2004?)
-
7/31/2019 Research 2003
8/24
Background: linear pixelsBackground: linear pixels
Linear pixels (CCD orLinear pixels (CCD orCMOS) integrateCMOS) integratephotons over discretephotons over discreteperiods of timeperiods of time
They produce aThey produce avoltage directlyvoltage directlyproportional to theproportional to thelight intensitylight intensity
The response mayThe response maysaturate white orsaturate white orblack easilyblack easily
-
7/31/2019 Research 2003
9/24
Background: logarithmic pixelsBackground: logarithmic pixels
Logarithmic pixelsLogarithmic pixels(CMOS only) can(CMOS only) canmeasure photon fluxmeasure photon fluxcontinuouslycontinuously
They produce aThey produce avoltage proportional tovoltage proportional tothe logarithm of lightthe logarithm of lightintensityintensity
The response isThe response issimilar to that ofsimilar to that ofhuman visionhuman vision
-
7/31/2019 Research 2003
10/24
Background: image qualityBackground: image quality
Images are noisy withImages are noisy with
logarithmic sensorslogarithmic sensors
Colour is worse thanColour is worse than
with linear sensorswith linear sensorsQuality improves withQuality improves with
digital processingdigital processing
No comprehensiveNo comprehensivetreatment of eithertreatment of eitherproblem or solutionproblem or solution
-
7/31/2019 Research 2003
11/24
Method: theoryMethod: theory
Model logarithmic CMOS image sensorsModel logarithmic CMOS image sensors
using optical & integrated circuit theoryusing optical & integrated circuit theory
Use the model to hypothesize the causeUse the model to hypothesize the cause
and solution of image quality problemsand solution of image quality problems
Calibrate the model and test hypothesesCalibrate the model and test hypotheses
using constrained regression theoryusing constrained regression theory
Optimise digital image processing usingOptimise digital image processing using
multilinear (or array) algebramultilinear (or array) algebra
-
7/31/2019 Research 2003
12/24
Method: simulationMethod: simulation
Simulation of integrated circuits is moreSimulation of integrated circuits is moreaccurate than a theoretical analysisaccurate than a theoretical analysis
Cost of simulation in time and money isCost of simulation in time and money is
small compared to that of experimentsmall compared to that of experimentIntegrated circuits may be studied underIntegrated circuits may be studied undercontrolled and well-defined conditionscontrolled and well-defined conditions
Internal states and variables may beInternal states and variables may beobserved without specialised equipment,observed without specialised equipment,circuit disruption and/or foresightcircuit disruption and/or foresight
-
7/31/2019 Research 2003
13/24
Method: experimentMethod: experiment
Experiments were performed using a FugaExperiments were performed using a Fuga
15RGB camera from C-Cam Technologies15RGB camera from C-Cam Technologies
The camera was operated from a portableThe camera was operated from a portable
PC via a custom Windows applicationPC via a custom Windows application
The image sensor had 512The image sensor had 512 512 pixels 512 pixels
and a full frame rate of about 8 Hzand a full frame rate of about 8 Hz
Until recently, it was the most successfulUntil recently, it was the most successful
commercial logarithmic image sensorcommercial logarithmic image sensor
-
7/31/2019 Research 2003
14/24
Conclusions: fixed pattern noiseConclusions: fixed pattern noise
yy== aa ++ bb ln (ln (cc++ xx) +) + for illuminancefor illuminance xxandandresponseresponse yyof a pixelof a pixel
Variation of offsetVariation of offset aa,,gaingain bb, bias, bias ccor aor a
combination thereofcombination thereofcauses FPNcauses FPN
Calibration possibleCalibration possible
within limits of thewithin limits of the
stochastic errorstochastic error
-
7/31/2019 Research 2003
15/24
Conclusions: fixed pattern noiseConclusions: fixed pattern noise
Left to right: FPNLeft to right: FPNcorrection for single,correction for single,double and tripledouble and triplevariation modelsvariation models
Top to bottom: twoTop to bottom: twodecade attenuation ofdecade attenuation ofilluminance in halfilluminance in halfdecadedecade stepssteps
Inter-scene plus intra-Inter-scene plus intra-scene dynamic rangescene dynamic rangeof 3.5 decadesof 3.5 decades
-
7/31/2019 Research 2003
16/24
Conclusions: transient responseConclusions: transient response
The transient response ofThe transient response ofa pixel is fast enough fora pixel is fast enough formost applicationsmost applications
Greater demands areGreater demands are
placed on the row andplaced on the row andcolumn readoutscolumn readouts
Premature digitizationPremature digitization
results in a predictableresults in a predictable
non-uniformity or FPNnon-uniformity or FPN
Affects only a few rowsAffects only a few rows
due to slow scanningdue to slow scanning
-
7/31/2019 Research 2003
17/24
Conclusions: transient responseConclusions: transient response
Premature digitization isPremature digitization ismore serious for columnmore serious for columnreadout due to speedreadout due to speed
For example, columnsFor example, columns
need scanning at 100need scanning at 100MHz for HDTV videoMHz for HDTV video
Column-to-column gainColumn-to-column gain
variation is caused byvariation is caused by
transient responsetransient response
Resolve with carefulResolve with careful
timing and designtiming and design
-
7/31/2019 Research 2003
18/24
Conclusions: temperatureConclusions: temperature
dependencedependence
Unlike with humans,Unlike with humans,
digital cameras do notdigital cameras do notregulate temperatureregulate temperature
Hence, responses toHence, responses toilluminance dependilluminance depend
on temperatureon temperature
When temperatureWhen temperature
dependence variesdependence varies
from pixel to pixel,from pixel to pixel,
FPN occursFPN occurs
-
7/31/2019 Research 2003
19/24
Conclusions: temperatureConclusions: temperature
dependencedependenceThe dark response ofThe dark response ofa pixel depends onlya pixel depends onlyon temperatureon temperature
It may be used toIt may be used tocorrect FPN due tocorrect FPN due totemperature in thetemperature in thelight responselight response
Experiments supportExperiments supportthis conclusion butthis conclusion butsimulation results aresimulation results areshown for clarityshown for clarity
-
7/31/2019 Research 2003
20/24
Conclusions: colour renditionConclusions: colour rendition
Combine the theories ofCombine the theories ofcolour linear sensors andcolour linear sensors andb/w logarithmic sensorsb/w logarithmic sensors
Calibrate FPN, usingCalibrate FPN, usingimages of uniform stimuli,images of uniform stimuli,by aby a relativerelative analysisanalysisCalibrate colour, usingCalibrate colour, usingimages of a colour chart,images of a colour chart,by anby an absoluteabsolute analysisanalysis
Fuga 15RGB competesFuga 15RGB competeswith conventional digitalwith conventional digitalcameras (which have acameras (which have aperceptual error of 15)perceptual error of 15)
-
7/31/2019 Research 2003
21/24
Conclusions: colour renditionConclusions: colour rendition
Image of a colour chart, inImage of a colour chart, in11 lux of illuminance, was11 lux of illuminance, wasrendered using calibratedrendered using calibrated
modelsmodels
Single, double and tripleSingle, double and triplevariation results and idealvariation results and ideal
colours are showncolours are shown
As with vision, renditionAs with vision, rendition
improves in brighterimproves in brighterlighting and worsens inlighting and worsens in
dimmer lightingdimmer lighting
-
7/31/2019 Research 2003
22/24
Future workFuture work
Digital cameras aim to render images withDigital cameras aim to render images witha maximum of perceptual accuracy usinga maximum of perceptual accuracy usinga minimum of efforta minimum of effort
By modelling and calibrating logarithmicBy modelling and calibrating logarithmicCMOS image sensors, problems withCMOS image sensors, problems withimage quality may be solvedimage quality may be solved
Past work has focused on maximisingPast work has focused on maximisingperceptual accuracy but future work willperceptual accuracy but future work willfocus on minimising effortfocus on minimising effort
-
7/31/2019 Research 2003
23/24
Future workFuture work
Shrinking featureShrinking feature
sizes may be used tosizes may be used toimprove imagingimprove imaging
There are challengesThere are challengeswith deep submicronwith deep submicron
CMOS processes thatCMOS processes thatneed overcomingneed overcoming
What about industrialWhat about industrial
and biomedical usesand biomedical uses
of the technology?of the technology?
-
7/31/2019 Research 2003
24/24
AcknowledgementsAcknowledgements
This work was funded thanks to the engineeringThis work was funded thanks to the engineering
research councils of both Canada and the UKresearch councils of both Canada and the UK
Thanks also to colleagues at the MicroelectronicThanks also to colleagues at the Microelectronic
Circuits and Analogue Devices research groupCircuits and Analogue Devices research group