next generation intelligent camera technologies - composec 2014 keynote

Next Generation Intelligent Camera technologies

Jacob JoseProduct ManagerTexas Instruments

Agenda• TI’s investment in Video and Vision markets• Current technologies for intelligent camera markets• What’s coming next?

TI Confidential - NDA Restrictions3

TI has over 30 years of Video & Imaging innovation

TI leads in intelligent video technologies

Military Public Innovators Early adopters Early majority

DM642 Surveillance Processor with DSP

for analytics

2001

Lower Power

Higher Performance

Decreasing Form Factor• Lower system cost

• Affordable software options• Solve specific market needs• Smaller form factors• Reduced power

Thermal Efficiency

Support Evolving Standard

• Cost of ownership reduction• Standardization begins• Productivity maximized• Easy to implement• Standardization begins

TI’s DMVAx-based products in market NOW

Residential Do It YourselfTI DMVAx based

products in market NOW

Critical Facilities

Gaming Retail

Banking

DM6467 Surveillance Processor with 1GHz

C64x DSP for analytics

2005

VLIB, a library of 50+ optimized

kernels available for C64x, C64x+ DSPs

2007

TI announces first video SoC (DMVA1)

with Vision Co-Processor

2009

TI adds HD compression & Tamper

Detect, Motion Detection, People

Counting on DMVA2

2010

First customers in production using DMVA1 & DMVA2

2011

4

TI opens Vision Co-Processor for

programming using the Vision Co-

Processor Library

2013

Various Analytics technology

development

1990 -

http://www.logitech.com/en-us/webcam-communications/video-security-systems/master-systems/devices/7251

http://www.logitech.com/en-us/webcam-communications/video-security-systems/master-systems/devices/7251

5

TI’s DMVA3:A high performance programmable vision coprocessor

• 720MHz ARM® Cortex™-A8 for control heuristics

• Up to 2 megapixel real-time H.264/SVC-T

• Capable of compressing videos at 20 megapixels or higher resolution while running video analytics algorithms

• Customers can port their own algorithms onto the DMVA3

Loitering Detection

Left Object Detection

Object Classification

Animal

TI’s DM8127:Leading solution for Traffic Cameras

• Up to 750 MHz of C674x floating point DSP

• 1GHz ARM Cortex-A8 for control heuristics

• 5 Megapixel Fish-eye correction + perspective correction (bird’s eye view

• TI’s stereo module library to sense depth and increase accuracy by 5x-10x

• Reference design available

6

Left Image Right Image

Leopard Imaging’s LNC-DUALCAM-3Dbased on TMS320DM8127

Depth Map

DMVA3 and DM8127 enables customizable analytics at the edge for mainstream cameras

7

TI’s DaVinci video processors enable megapixel smart analytics

IP

People counting Trip zone Streaming metadata Tamper detection Motion detection

8

Intelligent Motion Detection (IMD)Configuration Example

Intelligent Motion Detection

Add ROI Delete ROI Clear All

Save Cancel Advanced

Enable ROI Label

Region of Interest Settings

IMD Global Settings

Settings Info

Enable IMDDisplay Object Bounding BoxDisplay Enabled Zones

Sensitivity Moderate

Max. Object Size

Min. Object Size 100

8000

Opening page of IMD Control PanelGUI Behavior

For illustrative purposes, the yellow arrow represents an end-

user’s mouse.

This is the opening page for IMD’s GUI, which is called the IMD

Control Panel. It contains a live preview of the camera’s field of

view. ‘IMD Global Settings’, which establish parameters that apply for all regions of interest (ROIs), are

set in the sub-window below preview. ROIs are managed by

the sub-window to the right of the preview window. The lower-right sub-window displays feedback

messages to the user when mouseover events occur or to

acknowledge user input.

When this page is viewed for the first time, we assume that no ROI

has been defined by the user. Note that default settings are

installed and buttons to delete or clear ROIs have been disabled.

Whenever the ‘Defaults’ button is clicked, all settings are restored to

those shown here.

Intelligent motion detection is disabled. Click ‘Enable IMD’ to turn this feature on.

Defaults




Enable ROI Label


IMD Global Settings

Settings Info



Max. Object Size


8000

Enabling IMD for the 1st timeGUI Behavior

When a mouseover event occurs over the ‘Enable IMD’ checkbox,

its present status is acknowledged in the Settings Info sub-window.

If the checkbox is clear and the user left-clicks it, report ‘Intelligent

motion detection is enabled.’ in Settings Info. Likewise, if the

checkbox has been checked and the user left-clicks it, report

‘Intelligent motion detection is disabled.’

If the mouse returns to the checkbox (after loosing focus), report ‘Click this button to turn

intelligent motion detection off.’ in Settings Info.

Click this button to turn intelligent motion detection on.

Defaults




Enable ROI Label


IMD Global Settings

Settings Info



Max. Object Size

Min. Object Size 3

67

Changing default settingsGUI Behavior

The ‘Sensitivity’ dropdown list can be set to the following states:• Low• Low-to-Moderate• Moderate• Moderate-to-High• High

When a mouseover event occurs over the ‘Sensitivity’ dropdown list, instructive information is shared in the Settings Info sub-window: ‘Try

Low sensitivity to improve performance in outdoor or

challenging environments. High-sensitivity works best for fast

moving objects in stable lighting environments.’

The Min. Object Size slider changes the radius of a green circle rendered on the screen. Likewise, Max. Object Size will affect the radius of a red circle.

These circles are drawn anytime a mouseover event occurs on the slider or adjacent input window.

The user is free to move the circles to any location inside of the

preview window.

Move the slider or type in a value to set the minimum object size. Objects that are smaller than this value will be ignored by Intelligent motion detection.

Defaults

Min Obj Size

Max Obj Size




Enable ROI Label


IMD Global Settings

Settings Info



Max. Object Size


8000

Enabling with no user-defined ROIsGUI Behavior

If the user clicks ‘Save’ when no user-defined regions of interest have been selected, a DMVA

Settings Manager dialog pops up and displays a message. This

dialog has the focus of attention over any other control owned by the browser window. The dialog

has redundant controls (same as those presented within the IMD

Control Panel).

If the user clicks ‘Save’, a new ROI is automatically defined by a 4-point polygon that shares the same coordinates as the entire frame. This ROI is enabled and labeled as ‘Field_of_View’. The dialog is closed and the page is

redirected to DMVA home.

If the user clicks ‘Cancel’, the dialog is closed and the focus

returns to the IMD Control Panel.Intelligent motion detection is enabled.

Defaults

Since no regions of interest have been explicitly defined, an alert event will be

signaled when any object motion is detected within the entire field of view.

Press Save to accept or Cancel to return to the IMD Control Panel.

DMVA Settings Manager

Save Cancel




Enable ROI Label

ROI_0


IMD Global Settings

Settings Info



Max. Object Size


8000

Adding a region of interest (1)GUI Behavior

If a mouseover event occurs over the ‘Add ROI’ button, an

informative message is presented in ‘Settings Info’.

If the user clicks this button, the top row in the scrolling list is

highlighted in a brighter color. In this row, an enabled checkbox appears in the 1st column and a default ROI label, e.g. ROI_0,

appears in the 2nd column.

Click on this button to define a polygon to represent a new region of interest..

Defaults




Enable ROI Label

ROI_0


IMD Global Settings

Settings Info



Max. Object Size


8000


After the ‘Add ROI’ button is clicked, the mouse pointer

changes from arrow to crosshairs when the mouse moves inside preview window. An instructive

message is presented in ‘Settings Info’ that explains how to define

the ROI.

Clicks outside the preview window force a warning message inside Settings Info, e.g. Invalid point selected! Please pick a point inside the preview window.

If 16 points have been defined while setting the ROI, the next single click will complete the

polygon based on those points.

Define the region of interest. Each left mouse button click within the live preview window defines a point in the polygon. Up to 16 points can be used. Double-clicking the left mouse button will automatically complete the polygon.

Defaults




Enable ROI Label

ROI_1

ROI_0


IMD Global Settings

Settings Info



Max. Object Size


8000


After the ROI has been established, the area inside the

lines is shaded and the label displayed. Buttons ‘Delete ROI’ and ‘Clear All’ become enabled and the top row of the ROI list

loses the focus (no highlighting).

To add more ROIs, the user must click ‘Add ROI’ for each region. Once activated, the user must

click the left mouse button inside the preview over a location not

already defined by an existing ROI to create a new ROI. The

elements in the ROI list will grow accordingly.

Define the region of interest. Each left mouse button click within the live preview window defines a point in the polygon. Up to 16 points can be used. Double-clicking the left mouse button will automatically complete the polygon.

Defaults

ROI_0




Enable ROI Label

DESK

ROI_0


IMD Global Settings

Settings Info



Max. Object Size


8000

Editing a region of interestGUI Behavior

The user can change the label name at any point by click on the

label in the ROI list. Any name under 64 characters is exceptable

as long as the name is unique, case-insensitive, begins with a letter, etc. (same restrictions as filenames). When editing, the

cursor is displayed and the entire row is highlighted.

If the another ROI was being defined when the user tries to edit a label name, store the settings to

a valid state, e.g. close the polygon as if a double-click were

registered.

Re-label the name of the region of interest. Limit names to 64 characters and avoid leading numbers and the following characters: ` , ~ ! @ # $ % ^ & * ( ) / ? ‘ { } \ |

Defaults

ROI_0

DESK




Enable ROI Label

DOOR

DESK

MAIN_FLOOR


IMD Global Settings

Settings Info



Max. Object Size


8000

Deleting a region of interestGUI Behavior

If a mouseover event happens over the ‘Delete ROI’ button, Settings Info reports ‘Click to

delete a highlighted ROI.’

Pressing this button without first selected a ROI label name to do nothing; Settings Info will report

‘You must select a ROI label name first to delete.’

Double-clicking a label name gives it the focus of attention, e.g. highlights it. If ‘Delete ROI’ is

clicked after the row is highlighted, the corresponding overlay is also brightened. Clicking this button a

second time will delete the element in the list as well as the

overlay region in the preview window. If the focus of attention is

diverted to the keyboard or another location on the page not

over the button, the delete operation is canceled.

No overlap tolerated. Max. ROI = 16

Click again to delete a highlighted region of interest.

Defaults

ROI_0

DESK

DOOR

18

DMVA Event Monitor

Excerpts from DMVA Graphical User Interface (GUI) Specification

DMVA Event Monitor

Timestamp EventType Event Details

Tu Oct 7 0913:29:39.1 IMD Zone = LOBBY

Duration = 00:1:39.2





Tu Oct 7 0913:45:19.0 COUNT Num = 2

Direction = LATERAL

Tu Oct 7 0913:54:35.8

TRIPZONE Direction = Zone 12

Tu Oct 7 0913:55:19.3 IMD Zone = DESK

Duration = 00:19:49.2

Event List

DMVA Message Window

TimestampMetadata Overlay

DMVA Live Event MonitorGUI Behavior

The DMVA Event Monitor page is the main DMVA page, e.g. homepage. It

features a video window, a scrolling list of event summaries, function

indicators/selectors, controls for display and event recording, and a message

window.

The video window shows both live (default) and archived video, which can

be marked up by graphical overlays. The ‘Display’ subwindow allows the user

to select the desired overlay. A row of indicators/selectors sits below.

Indicator “lamp” for an active analytic function glows “green”; Tamper, which runs at all times, can also glow “red”

during active tamper events. Lamps also used to 1 of the 4 analytic functions

that can run simultaneously with tamper. After function is selected,

user can click ‘Configure’ to bring up the dedicated configuration page.

During each event, the compressed audio and/or video, metadata, and

detailed event log file can be saved.

Tamper alerts are displayed in the DMVA Message window. Overall DMVA health and status information is shown

here as well.

Tamper alert: Camera movement detected Tues Oct 7 2009 13:56:09.2!

LOBBYDESK

Tamper

Analytics

DisplayObject Bounding BoxEnabled Zones

Event RecordingVideo StreamAudio StreamMetadata StreamEvent Log

CancelConfigure Advanced

IMD

CountingTripZone

SMOS

DMVA Event Monitor








Tu Oct 7 0913:45:19.0 COUNT Num = 2

Direction = LATERAL

Tu Oct 7 0913:54:35.8



Duration = 00:19:49.2

Event List

DMVA Message Window


DMVA Live Event Monitor

Cllick ‘Advanced’ to fine-tune camera tamper detection parameters.

LOBBYDESK

Tamper

Analytics

DisplayObject Bounding BoxEnabled Zones

Event RecordingVideo StreamAudio StreamMetadata StreamEvent Log


IMD

CountingTripZone

SMOS

GUI BehaviorIf the Advanced button is pushed, a dialog box appears. For now, it will show camera tamper options.

Press Save to accept or Cancel to return to

the DMVA Event Monitor.

Save Cancel

Advanced DMVA Settings

Detect lights on-to-off eventsDetect lights off-to-on events

Camera Tamper Detection

Sensitivity Moderate Reset Sec. 30

DMVA Event Monitor








Tu Oct 7 0913:45:19.0 COUNT Num = 2

Direction = LATERAL

Tu Oct 7 0913:54:35.8



Duration = 00:19:49.2

Event List

DMVA Live Event MonitorGUI BehaviorDMVA generates an output

response for every frame of video. For each event, an summary

record is created that establishes the duration of the event and

associates the timestamp, type, and relevant zone information

(context dependent based on the enabled analytic function).

When a user selects an event from the list and ‘Video Stream’

has also been selected (at time of event as well as while user is

interacting) under ‘Event Recording’, the compressed video

of the event is retrieved and replayed in the window. If

‘Metadata Stream’ has also been select, bounding box information is available for display (depends on whether ‘Metadata Overlay’ has

been checked or not).Video and metadata recorded for this event.

LOBBYDESK

Tamper

AnalyticsIMD

CountingTripZone

SMOS



Object Bounding BoxEnabled Zones

Video StreamAudio StreamMetadata StreamEvent Log

Display Event Recording

DMVA Message Window

Image Signal Processing (ISP) video processing smart plug-ins

22

Video stabilization: Camera shakes due to weather, traffic → degrades video quality

TI’s advanced video stabilization reduces jitter, improves visual quality

Face detection tracking, recognition:For identification, access control, privacy mask

TI’s face detection HW tracks multiple faces (>30) with detection in <20 milliseconds

TI’s Video Noise Filters preserve detail, enhance visual quality, enable low light imaging

Image noise: Degrades low light image quality, details are lost

Problem Solution

JohnSmith

Problem Solution

Image Signal Processing (ISP) video processing smart plug-ins

TI Confidential - NDA Restrictions

23

Lens Distortion, Fish Eye CorrectionFish eye is lenses can get to 360 degree view

TI’s Lens Distortion Correction technology enables to use cheaper fish eye lenses

Auto White Balance, Auto ExposureFor exposure correction and compensate for type of illumination

TI’s H3A hardware provides for Auto Exposure, Auto White Balance and Auto Focus

Security application: Provide clear picture in high dynamic range scenes

TI’s WDR brings details in the shadow & compensatefor low cost sensors

WHAT’S NEXT?

24

What problem we are trying to solve?

• Two eyes help humans to perceive depth. Still, humans are prone to illusions.

Images from: http://www.qianqin.de/2007/10/15/clockwise-or-counterclockwise/

• Most security cameras have only one “eye”, so they cannot easily distinguish between– shadows and a person– two people standing behind each other

• Depth perception can reduce false detections by 5-10x

http://www.qianqin.de/2007/10/15/clockwise-or-counterclockwise/

50 Years of Intelligent Technology Development at TI

• TI started very early– In 1964, TI started developing a guidance system for a laser-guided artillery shell [1], By

1969, the first guidance kits were delivered [2]

• TI’s analytics algorithms are internally developed, robust video and non-video based intelligence technologies enhanced over a long period of time.

• Today, TI is a leading technology supplier to the automotive market where self-driving cars require far more accuracy than for IPCameras.

26

[2] Note: In 1997 TI exited the defense business, selling it’s defense unit to Raytheon

[1] http://www.ti.com/corp/docs/company/history/timeline/defense/1960/docs/65-ti_demos_laser_guidance.htm

Front Camera

Surround View

Smart Rear Camera

Long Range Radar Others

FusionNight VisionMirror Replacement

http://www.ti.com/corp/docs/company/history/timeline/defense/1960/docs/65-ti_demos_laser_guidance.htm

Time of Flight technology• Time of Flight technology measures

depth based on how soon the modulated pulses reflect back to source

• Works great at night when stereo is ineffective

• More on TOF technology: http://www.ti.com/ww/en/analog/3dtof/index.shtml

http://www.ti.com/ww/en/analog/3dtof/index.shtml

http://www.ti.com/ww/en/analog/3dtof/index.shtml

Digital Light Processing (DLP) technology• DLP technology relies on structured

lighting to illuminate a grid on the scene and observe the distortions in the grid to perceive depth

• DLP offers higher resolution than TOF technology

• More on DLP technology: http://www.ti.com/solution/3d_machine_vision

Left Hand Electric Drill Metal Screw

Color maps of measured objects

http://www.ti.com/solution/3d_machine_vision



Comparison of 3D Imaging Technologies Time of Flight (ToF) Stereoscopic Vision Fixed Structured Light Programmable Structured

Light (DLP)

Operational Principle IR pulse, measure light transit time

Two 2D sensors emulate human eyes

Single pattern visible or IR illumination, detects distortion

Multiple pattern visible or IR illumination, detects distortion

Point Cloud Generation Direct out of chipset High SW Processing Medium SW Processing SW Processing scales with # of patterns

Latency Low Medium Medium Medium

Active Illumination Yes No Yes Yes – Customizable Spectrum

Low Light Performance Good Weak Good Good

Bright Light Performance Medium GoodMedium / WeakDepends on illumination power

Medium / WeakDepends on illumination power

Power Consumption Medium/HighScales w/ distance Low Medium Medium

Scales with distance

RangeShort to long rangeDepends on laser power & modulation

Mid rangeDepends on spacing between cameras

Very short to mid rangeDepends on illumination power

Very short to mid rangeDepends on illumination power

Resolution QQVGA, QVGA ->Roadmap to VGA Camera Dependent Camera Dependent WVGA to 1080p ->

Roadmap to WQXGA

Depth Accuracymm to cmDepends on resolution of sensor

mm to cmDifficulty with smooth surface mm to cm µm to cm

Scanning Speed FastLimited by sensor speed

MediumLimited by software complexity

FastLimited by camera speed

Fast / MediumLimited by camera speed

Applications Location X X XIdentification X X X XMeasurement & Inspection X X X XBiometrics XAugmented Reality X X X

next generation intelligent camera technologies - composec 2014 keynote

Documents