Vision, Video
and Virtual Reality SensorsSensors
Lecture 4Sensors
CSC 59866CDFall 2004
Zhigang Zhu, NAC 8/203A
http://www-cs.engr.ccny.cuny.edu/~zhu/Capstone2004/Capstone_Sequence2004.html
Vision, Video
and Virtual Reality AcknowledgementsAcknowledgements
The slides in this lecture were adopted and modified from lectures by
Professor Allen HansonUniversity of Massachusetts at Amherst
Vision, Video
and Virtual Reality SensorsSensors
Static monocular reflectance data (monochromic or color) Films Video cameras (with tapes) Digital cameras (with memory)
Motion sequences (camcorders) Stereo (2 cameras) Range data (Range finder) Non-visual sensory data
infrared (IR) ultraviolet (UV) microwaves
Many more
Vision, Video
and Virtual Reality The Electromagnetic SpectrumThe Electromagnetic Spectrum
Visible Spectrum
700 nm 400 nm
C = f
f
Vision, Video
and Virtual Reality The Human EyeThe Human Eye
Vision, Video
and Virtual Reality The EyeThe Eye
The Retina: rods (low-level light, night vision) cones (color-vision) synapses optic nerve fibers
Sensing and low-level processing layer 125 millions rods and cones feed into 1 million nerve fibers Cell arrangement that respond to horizontal and vertical lines
Retina
RodsCones
Vision, Video
and Virtual Reality Film, Video, Digital CamerasFilm, Video, Digital Cameras
Black and White (Reflectance data only) Color (Reflectance data in three bands - red, green, blue)
Vision, Video
and Virtual Reality Color ImagesColor Images
Blue Green Red
‘Dimensions’ of an Image
Spatial (x,y)Depth (no. of components)Number of bits/channelTemporal (t)
Pixel
Spatial Resolution
Spectra Resolution
Radiometric Resolution
Temporal Resolution
Vision, Video
and Virtual Reality Across the EM SpectrumAcross the EM Spectrum
Crab Nebula
Vision, Video
and Virtual Reality Across the EM SpectrumAcross the EM Spectrum
Cargo inspection using Gamma Rays
Mobile Vehicle and Cargo Inspection System (VACIS®)
Gamma rays are typically waves of frequencies greater than 1019 Hz
Gamma rays can penetrate nearly all materials and are therefore difficult to detect
Courtesy:Science Applications International Corporation (SAIC),
Vision, Video
and Virtual Reality Across the EM SpectrumAcross the EM Spectrum
Cargo inspection using Gamma Rays
Mobile Vehicle and Cargo Inspection System (VACIS®)
Gamma rays are typically waves of frequencies greater than 1019 Hz
Gamma rays can penetrate nearly all materials and are therefore difficult to detect
Courtesy:Science Applications International Corporation (SAIC),
Vision, Video
and Virtual Reality Across the EM SpectrumAcross the EM Spectrum
Cargo inspection using Gamma Rays
Mobile Vehicle and Cargo Inspection System (VACIS®)
Gamma rays are typically waves of frequencies greater than 1019 Hz
Gamma rays can penetrate nearly all materials and are therefore difficult to detect
Courtesy:Science Applications International Corporation (SAIC),
Vision, Video
and Virtual Reality Across the EM SpectrumAcross the EM Spectrum
Medical X-Rays
Vision, Video
and Virtual Reality Across the EM SpectrumAcross the EM Spectrum
Chandra X-Ray Satellite
Vision, Video
and Virtual Reality Across the EM SpectrumAcross the EM Spectrum
From X-Ray images to 3D Models: CT Scans
Vision, Video
and Virtual Reality Across the EM SpectrumAcross the EM Spectrum
Flower Patterns in Ultraviolet
Dandelion - UV
Po
ten
till
a
Vision, Video
and Virtual Reality Across the EM SpectrumAcross the EM Spectrum
Messier 101 in Ultraviolet
Vision, Video
and Virtual Reality Across the EM SpectrumAcross the EM Spectrum
Traditional images
Vision, Video
and Virtual Reality Across the EM SpectrumAcross the EM Spectrum
Non-traditional Use of Visible Light: Range
Vision, Video
and Virtual Reality Across the EM SpectrumAcross the EM Spectrum
Scanning Laser Rangefinder
Vision, Video
and Virtual Reality Across the EM SpectrumAcross the EM Spectrum
IR: Near, Medium, Far (~heat)
Vision, Video
and Virtual Reality Across the EM SpectrumAcross the EM Spectrum
IR: Near, Medium, Far (~heat)
Vision, Video
and Virtual Reality Across the EM SpectrumAcross the EM Spectrum
IR: Finding chlorophyll -the green coloring matter of plants that functions in photosynthesis
Vision, Video
and Virtual Reality Across the EM SpectrumAcross the EM Spectrum
(Un)Common uses of Microwaves
CD MovieExploding Water
Movie
Vision, Video
and Virtual Reality Across the EM SpectrumAcross the EM Spectrum
Microwave Imaging: Synthetic Aperture Radar (SAR)
San Fernando Valley Tibet: Lhasa River
Thailand: Phang Hoei RangeAthens, Greece
Red: L-band (24cm)
Green: C-band (6 cm)
Blue:C/L
Vision, Video
and Virtual Reality Across the EM SpectrumAcross the EM Spectrum
Radar in Depth: Interferometric Synthetic Aperture Radar - IFSAR(elevation)
Vision, Video
and Virtual Reality Across the EM SpectrumAcross the EM Spectrum
Low Altitude IFSAR
IFSAR elevation, automatic, in minutes
Elevation from aerial stereo, manually, several days
Vision, Video
and Virtual Reality Across the EM SpectrumAcross the EM Spectrum
Radio Waves (images of cosmos from radio telescopes)
Vision, Video
and Virtual Reality Stereo GeometryStereo Geometry
Single Camera (no stereo)
Vision, Video
and Virtual Reality Stereo GeometryStereo Geometry
P(X,Y,Z)
f = focal length
Optical Center
pr(x,y)
Film plane
pl(x,y)
Optical Center
f = focal length
Film plane
LEFT CAMERA RIGHT CAMERA
B = Baseline
Vision, Video
and Virtual Reality Stereo GeometryStereo Geometry
LEFT IMAGE RIGHT IMAGE
Disparity = xr - xl
P
Pr(xr,yr)Pl(xl,yl)
≈ depth
Vision, Video
and Virtual Reality Stereo ImagesStereo Images
A Short Digression
StereoscopesStereoscopes
Vision, Video
and Virtual Reality Stereo ImagesStereo Images
Darjeeling Suspension
Bridge
Vision, Video
and Virtual Reality Picture of you?Picture of you?
Vision, Video
and Virtual Reality StereoStereo
Stereograms
Vision, Video
and Virtual Reality Stereo X-RayStereo X-Ray
Vision, Video
and Virtual Reality Range SensorsRange Sensors
Light Striping
David B. Cox, Robyn Owens and Peter HartmannDepartment of BiochemistryUniversity of Western Australia http://mammary.nih.gov/reviews/lactation/Hartmann001/
Vision, Video
and Virtual Reality MosaicsMosaics
A mosaic is created from several images
Vision, Video
and Virtual Reality MosaicsMosaics
Stabilized Video
Vision, Video
and Virtual Reality MosaicsMosaics
Depth from a Video Sequence (single camera)
P(X,Y,Z)
Height H from Laser Profiler
GPS
Vision, Video
and Virtual Reality MosaicsMosaics
Brazilian forest…..made at UMass CVL
Vision, Video
and Virtual Reality Why is Vision Difficult?Why is Vision Difficult?
Natural Variation in Object Classes: Color, texture, size, shape, parts, and relations
Variations in the Imaging Process Lighting (highlights, shadows, brightness, contrast) Projective distortion, point of view, occlusion Noise, sensor and optical characteristics
Massive Amounts of Data 1 minute of 1024x768 color video = 4.2 gigabytes
(Uncompressed)
Vision, Video
and Virtual Reality The Need for KnowledgeThe Need for Knowledge
Knowledge
Function
Context
Shape
SpecificObjects
GenericObjects
Structure Size
Shape
Motion
Purpose
Variation
Vision, Video
and Virtual Reality The Figure Revealed The Figure Revealed
Light Source
Vision, Video
and Virtual Reality The Effect of ContextThe Effect of Context
Vision, Video
and Virtual Reality The Effect of Context - 2The Effect of Context - 2
Vision, Video
and Virtual Reality Context, cont.Context, cont.
….a collection of objects:
Vision, Video
and Virtual Reality ContextContext
The objects as hats:
Vision, Video
and Virtual Reality
And as something else…..
‘To interpret something is to give it meaning in context.’
ContextContext
Vision, Video
and Virtual Reality Vision System ComponentsVision System Components
…..at the low (image) level, we need Ways of generating initial descriptions of the image data Method for extracting features of these descriptions Ways of representing these descriptions and features Usually, cannot initially make use of general world
knowledge
IMAGE(numbers)
DESCRIPTION(symbols)
Vision, Video
and Virtual Reality
….at the intermediate level, we need Symbolic representations of the initial descriptions Ways of generating more abstract descriptions from the initial ones (grouping) Ways of accessing relevant portions of the knowledge base Ways of controlling the processing
Intermediate level processes should be capable of being used top-down (knowledge-directed) or bottom-up (data-directed)
IMAGEIINTERMEDIATEDESCRIPTIONS
KNOWLEDGE
Vision System ComponentsVision System Components
Vision, Video
and Virtual Reality Vision System ComponentsVision System Components
….at the high (interpretation) level, we need Ways of representing world knowledge
Objects Object parts Expected scenarios (relations) Specializations
Mechanisms for Interferencing Beliefs Partial matches
Control Information Representations of
Partial interpretations Competing interpretations Relationship to the image descriptions
Vision, Video
and Virtual Reality NextNext
Anyone who isn't confused really doesn't understand the situation.
--Edward R. Murrow
Next:Image Formation
Reading: Ch 1, Ch 2- Section 2.1, 2.2, 2.3, 2.5
Questions: 2.1. 2.2, 2.3, 2.5
Exercises: 2.1, 2.3, 2.4