Computer vision for non-visual spectral regimes and non-

traditional applications

Rama ChellappaUMD

Opening remarks – non-visual sensors

• Sensors in visual regimes are only a small % of the possible sensors

• Infrared, hyperspectral, LADAR, SAR, FOPEN SAR, mmWave, polarimetric

• Provide thermal signatures, material properties, 3D, all weather/all day coverage, looking behind trees, walls, …

• Sensor physics is important.• Low signal to clutter ratio.• Characterization of image statistics driven by

sensor physics

Problems addressed using hyperspectral images

• Sensor design • Material classification/segmentation• Anomaly detection

– More than two decades of work• ARO MURI (2002-2007) on the science of land-based target

signatures.• DARPA DDB program, NGA• Glenn Healey: pioneer in HSI and computer vision• Larry Wolf, Equinox Corporation

• Unmixing of pixels (Remote sensing) • Compressive sensing possibilities (SPC)• Numerous books, papers, conferences…

Estimating object reflectance

4

• Radio transfer function

– irradiance to the camera

– reflectance at location (x,y)

– main light source (e.g. sun light)

– ambient light source coming from all directions, assuming constant intensity for all directions

– fraction of unblocked sky from (x,y) view

geometry factor

Nguyen and Chellappa,CVPR 2010 Workshop onBeyond visible spectrum

5

Tracking radiance

Reflectance tracking

• Robust against illumination, abrupt motion• Capable of recovering after losing track

6

Detection of land mines

• Statistical models of clutter (non-Gaussian)

• Optimal detection methods • Sup-pixel detection methods• Detection of disturbed earth

– Detection of mass graves in the Balkans

Broadwater and Chellappa, PAMI 2007, SP, 2010

Radar images

• Synthetic aperture radar (SAR), foliage-penetrating SAR (FOPEN SAR), …• Speckle noise (random fluctuations in the return signal from an object that is no

bigger than a single image-processing element)• Shape from shading – radar clinometry (USGS, Frankot, Chellappa, AI

Journal, 1990)• 3D from interferometric SAR (Zebker and Goldstein), stereo SAR• Object detection, indexing recognition (DARPA programs MSTAR, SAIP,

DDB)– MSTAR program (feature extraction, indexing, prediction, recognition) – Typical

object recognition framework– Features can correspond to scatterers (supported by physics)

• FOPEN SAR– Low signal to clutter ratio (tree trunks produce stronger returns than objects)– Symmetric alpha-stable noise

• Global hawk, TESAR sensors

R.T. Frankot and R. Chellappa, Estimation of Surface Topography from SAR Imagery Using Shape from Shading Techniques, in Physics-Based Vision: Shape Recovery, (eds.), L.B. Wolff, S.A. Shafer and G.E. Healey, Jones and Bartlett Publishers, Boston, MA, pp. 62-101, 1992.

LADAR images• Took off in the mid eighties• From machine vision to outdoor conditions• Feature (step, crease edges, surfaces,) extraction,

matching and recognition (Hypothesis prediction/verification)

– Fundamental forms I and II (Besl and Jain)

• Pulsed-Doppler LADAR (2km) for ATR• Better resolution at longer ranges• Kinect

R. Chellappa, S. Der and E.J.M. Rignot, Statistical Characterization of FLIR, LADARand SAR Imagery, in Statistics and Images, K.V Mardia, (ed.), Carfax publishers, Oxfordshire, U.K., pp. 273-312, 1994.

Opening remarks – non-traditional applications

• Road following, lane tracking and other automotive applications– Dickmanns, Pomereleu, DOT, FHWA

• Computer vision for the blind– Navigation, face/expression recognition

• Analysis of Schlieren images – Detection of oblique structures such as shock waves

and shear layers

• Understanding bee dances• Industrial inspection

Vision for Schlieren data reduction• Schlieren Imaging

– Aerodynamic visualization technique in wind tunnels -> long established

– Capture density gradients in supersonic flow

– Shock waves, shear layers and turbulent structures

– High speed imaging -> data deluge• Images complete the picture

– Offer what surface measurements may not

– Are we taking advantage of the data collected?

• Can vision extend analysis capabilities?

– Additional insight to flow unsteadiness– Desire for automation– Removal of human subjectivity

• Recast as a segmentation/feature extraction problem

Extraction of oblique structures

• Oblique flow structures ubiquitous• Physically meaningful

segmentation– Bilateral filter -> isoperimetric cut

• Shock wave and shear-layer inclination

– Canny & Hough transform• Classification

– Length & quantifiable bounds– Location enforced from labeling

• Scale implementation for robustness?

– Hough transform binning– Convergence from two Hough grids

• Success of automation 92-94%• Outer shock motion history• Recommended for Publication

in AIAA Journal• Vision: viable analysis tool

Waggle dance

Orientation of waggle axis Direction of Food source.(with respect to sun).Intensity of waggle dance Sweetness of food source.Frequency of waggle Distance of food source.Parameters of interest in the waggle dance

Waggle Axis : Average orientation of Thorax during Waggle.Duration of Waggle : Number of frames of waggle in each segment of the dance.

Anatomy/behavior modeling - prior

• Three major body parts; each body part modeled as an ellipse.

• Anatomical modeling ensures– Physical limits of body parts are

consistent; accounts for structural limitations and correlation among orientation of body parts

– Insects move in the direction of their head.

Veeraraghavan, Chellappa and Srinivasn, IEEE TPAMI, March 2008

Insects display very specific behaviors - priorsModeling behavior explicitly improve

Tracking performanceBehavioral understanding

Position tracking and behavioral analysis in a unified framework.

Result

Detect Frames of Waggle Dance by looking at

Rate of change of Abdomen OrientationAverage absolute motion of center of abdomen in the direction perpendicular to the axis of the bee.

Parameters of Interest in the Waggle Dance

Waggle Axis : Average orientation of Thorax during Waggle.Duration of Waggle : Number of frames of waggle in each segment of the dance.

Looking for a screw amid screws (MERL)

My advice to the young ones

• Look for collaborations outside EE, CS• Helps with multidisciplinary credentials• Will help with winning MURIs: best source of

support.• There are top Transactions and journals that

accept these papers!• Computer vision presents immense

opportunities outside traditional areas.