Sensory Information Processing

Introduction to imaging sensors

Camera

CCD driver / Signal processing

DigitalCapture

CCD

lens

What is principal parameter of the lens?What is aberration?

What is the image degradation by lenses?How to select the lenses?

How the image is recorded electrically?What is the artifact of CCD?

How to select the CCD?

Camera

How to drive the CCD?How the color value is recovered?

How the Synchronize and Sampling is performed?

CCD driver / Signal processing

DigitalCapture

CCD

lens

Measuring the light• History of light measurement

– Exposure meter for cameras(Human can not percept absolute amount of the light)

Leica MC meter (1954)Contaflex(1935) : world’s first camera with exposure meter

Before that..

• Rely on inspiration and experience..

Senoguide

Device to measurethe amount of light

• Transducing energy of light to electricity– Photo diode– Solar cell (similar to photo diode)

• Change of resistance according to the light– CdS (sulfureted cadmium) cell– Photo tube

• Light displacement, speed, etc..– So weak, very difficult

CdS cell

• Change of resistance of sulfureted cadmium (compound semiconductor)– Free electrons are emerged by the light, and

conducts electric current well– Wide range of sensitivity– Slow reaction– cheap

Inadequate for image sensor

CdS cell(2)

• Winding CdS area– Wider width, shorter path

Burn out

・ very good linearity

electrode

Rel

ativ

e se

nsiti

vity

Wavelength(nm)

Ele

ctric

cur

rent

(mA

)

Maximum power consumption 100mW

Voltage (V)

phototube(1)• Photoelectric effect

– electron is kicked off from metal by photon

metal

photon

e-

• Characteristics of photoelectric effect– No electron for the light with longer wavelength

than threshold which depends on materials– Number of electron is proportional to the

intensity of light

phototube(2)• Capturing the electron out from the metal

– Material: Ag, Cs, Na, Te, Ga, As

– Very high speedof response : up to 10ps

•Electron is attracted by electric potential by high voltage

Phototube(3)• Very high sensitivity

(photomultiplier)

•Dynodes (SbCs, AgHg etc.) emit lots of electron by hitting the accelerated electron ．•http://en.wikipedia.org/wiki/Photomultiplier

photomultiplier(4)

• Super-kamiokaNDE (observing facility of neutrino)http://en.wikipedia.org/wiki/Kamiokande

• Broken photomultiplier of kamiokaNDE

photodiode(1)• Major light sensing device

– Photoelectron emerged between p and n semiconductor

– No vaccum!

history of exposure meter

• Solar cell type : require no battery– １９５０

• CdS type : high sensitivity (but slow)– １９６０

• Photodiodes– １９７０

Leica MR meter

Leica MC meter

Current products

2-D image sensor

• How to capture the 2-D distibution of light?

Array of sensors?Connecting all sensors each?

2-D image sensors

• To reduce the wiring cost, scanning mechanism is necessary

Electric signal

Camera tube

• Magnetic field is used to deflection (scanning)

Camera and TV tubes

• Magnetic field for scanning

Image orthicon

CCD (charge coupled device)

• Electrostatic effect is used to transfer the electron– Not limited to the imaging

device– CCD is the term of the way to

transfer the electron– http://en.wikipedia.org/wiki/

Charge-coupled_device

Aside : delay line (1)• How to delay temporal signal?

– “dead time component”

Aside : delay line (1)

Aside : delay line (1)• Delay line is a memory

– Recording information for a short time– No sampling, no quantization

Delay line (2)

• SAW (surface acoustic wave) filter– http://en.wikipedia.org/wiki/Surface_acoustic_wave

Delay line (3) BBD

• BBD = Bucket Brigade delay

Predecessor of CCD MN3004

Delay line (4) current standard

• Long time delay for sound, video is based on digital technology

AD converter CPU DA converter

RAM

CCD types

• Full frame• Interline transfer• Frame transfer• Frame interline transfer

• http://laser.physics.sunysb.edu/~jose/CCDInfo.html

• CCD university– http://www.ccd.com/ccdu.html

Full frame

• While transfer, CCD should be covered

• (mechanical shutter is necessary)

• High aperture ratio• Easy to manufacture• Commonly used in

digital SLR

Interline transfer

• Most popular• Mechanical

shutter is not necessary

• Very short exposure is possible

• Used in most video cameras

Frame transfer• Image is

transferred to the storage array quickly

Image degradation in CCD

• smear– Highlight spreading to the transfer direction

(vertical)• blooming

– Highlight spreading to the peripherals• Thermal noise

– Electron is emerged by the thermal agitation of free electron

smear

Source of smear

• Imperfect mask of transfer CCD areas, spilling out of electron to the transfer area

mask mask mask mask

mask

Anti-blooming

• Discarding too much electrons by highlight

X-Y address image sensor

CMOS image sensor

photodiodes are connected by wire and transistor

CMOS transistor is used, therefore, commonly called as CMOS sensor

JFET type

CMOS image sensor

• Low power consumption– CCD needs high voltage to attract the electron

• Other circuit can be embedded on a chip(usual LSI process is used)– AD converter, image processor can be

embedded• Noise was more than CCD

– Now it is well improved• Flexibility to access to each pixel

Rolling shutter

• For most CMOS sensors, exposure timing is varied pixel by pixel– Called “rolling shutter”– If all pixels are exposed simultaneously, it is

called “global shutter”

Shutter types and image skewLens shutter（ interchanging the lens is not easy)

Focal plane shutter（ suit for interchanging lens)SLR (single lens reflex) etc.

Skew of moving object

• Shape of the object is skewed by the varied timing of exposure

Ancient example of skew

• Lartigue “A.C.F grand prix”World’s most famous example of skew

Color image• Sequencial shots of R, G, B

– filter wheel• Split R, G, B to three image sensors

optically– ３ tubes, 3-CCD

• Each pixel is sensitive either R, G or B– Primal color, complementary color– Foveon

Filter wheel

３ tubes

lens

prism

Camera tube

３ -CCDcameraEach surface in the prism is “interference filter” which consist of transparent materials with different refractive index, and interference of light reflect or transmit the energy of the light without loss.

• No loss of light energy, high sensitivity• Color reproduction is good

Single CCD

• Each pixel of CCD is colored– Primal color

○natural color reproduction– Complementary color

○high light efficiency

R GG B

Primal color（ Bayer pattern)

Cy Ye Cy Ye Cy Ye Cy Ye

Mg G Mg G Mg G Mg G

Complementary color

R GG B

R GG B

R GG B

R GG B

R GG B

R GG B

R GG B

R GG B

R GG B

R GG B

R GG B

MgGCy Ye

Mg GCy Ye

Mg GCy Ye

Mg GCy Ye

Cy Ye Cy Ye Cy Ye Cy YeMgG Mg G Mg G Mg G

Color reproduction (2)• Simple method (interpolation)

• See Fredo Durand and Bill Freeman’s slide– http://groups.csail.mit.edu/graphics/classes/

CompPhoto07/index2.html– 03_colorInterpCP.ppt

GG

GG

GG

GG

GG

GG

GG

GG

GG

GG

GG

GG

R R R R

R R R R

R R R R

Low resolution