ibm research © 2007 ibm corporation tuning sss analytics rick kjeldsen [email protected]

IBM Research

© 2007 IBM Corporation

Tuning SSS Analytics

Rick [email protected]

IBM Research


Analytic Tuning

Metadata Tuning

– Ensuring Events show real objects / tracks as much as possible

– Events Represent what the system is seeing

– Without good quality event data most alerts will not work well.

Alert Tuning

– Adjusting Alerts to trigger on the goal activity as often as possible.• i.e. Not generating too many False Positives, nor missing False Negatives

– Ensuring alerts satisfy the Use Case

IBM Research


Metadata Tuning

Verify good quality Event data under all expected conditions

– Visual Inspection

– Statistics

– Sample searches

IBM Research


Visual Inspection of Events

Accuracy of object bounds

– Valid object

– Single object

– Complete object

– Distorted by shadows

– Will never be pefect

Accuracy of Track

– Complete or broken

Bunching of objects

Quality of color information

Examine at different times of day

IBM Research


Examples

IBM Research


Sample Event Searches

Do typical searches return expected results ?– Search for “wild” events

• Monitor scene for interesting object– e.g. red shirt

• Search for that object

– Create events and search for them

Histograms look as expected ?– Peaks and quiet times– Common problem: Under reports events during high

activity • No current fix

Situations to watch carefully– Night– Rain (esp at night)– High Activity periods– Times important to customer

IBM Research


Diagnose problems

Obvious causes of errors

– Obstructing objects

– Moving or windblown “background”

Patterns in Errors

– Time

– Lighting

– Object characteristics

– Location in image

Examine Thumbnail for clues

Think about how the system works !

IBM Research


Remediation Actions– RoU

• Regular motion• Distant activity

– e.g. cross road in distance• Hide distant views of objects you see nearer

– Small objects can be combined, then not get split apart• Reflections

– Change camera view (pan / zoom / change position)• Increase / decrease object size

– Larger objects track better and generally produce better metadata (e.g. color)– Whole object must be visible for long enough to track

• Avoid occlusions• Decrease number of objects in view• Improve viewing angle

– Looking down from high is generally better than looking long from low

– Shield camera from rain, light, etc.

– Live with deficiencies• Document issue• Discuss with customer

– Operational work-around– Modify use case

– Reject view

IBM Research


Alert Tuning

IBM Research


FP / FN Trade-off

IBM Research


SSS Release Certification

Ensures each release meets performance expectations

IBM internal standard data set

The results are examined by

– Research Staff, Product Staff and Management

Any discrepancies are addressed thru software testing and bug fixing.

In-general it is expected that accuracy improves from one release to the next

Labor intensive

– 1 hr of video can take up to 50 man / hours to certify

IBM Research


Release Certification Process

1. Define Use Cases and Requirements1. Alerts, Search, Operating Conditions (Day/Night/etc)

2. Data Collection1. Simulated events

2. Training vs Test Sequences

3. Environmental Condition Sampling

3. Ground Truth Marking

4. Select operating point using training set

5. Running the Video Analytic

6. Collating Video Analytics Results

7. Comparing Results to Ground Truth

8. Calculate Performance Metrics

9. Return to Step 4 – to test at different operating point.

IBM Research


Certification Performance Criterion

fntp

tpr

fptp

tpp

pr

rpF

)1(

1

Where is the recall bias. The lower the recall bias, the more emphasis on false positives.

tp – true positive – correct detection by the system

fp – false positive – Alert that triggers when it should not

fn – false negative – Alerts missed by the system

r – recall - Of the Alerts that should have been detected,

how many were ?

p – precision - Of the Alerts which were triggered,

how many were TP ?

F1 – balance between precision and recall

IBM Research


Alert Tuning Process

Camera ConfigurationBest guess parametersbased on similar views

Data CollectionStaged or Natural TPBackground Activity

Field EvaluationFPR

Recall ProcessingRun each TP with every PC

Data PrepGround Truthing

Video PrepParameter Config (PC) Prep

FP ProcessingRun each PC

against baseline video

PCs meeting Recall target

PCs meeting Recall and FPR target

Select and Deploy PC

Manual analysis

Field EvaluationRecall

Done

Done

Field Evaluation In-house (lab) evaluation Field verification

Off-line

On-line

Field EvaluationFPR

IBM Research


On-line Alert Tuning Process

Install / modify Alert

Collect data

Determine performance

Tune– View

– Parameters

– RoI, RoU

– Alerts

– Use Case

Repeat as needed

IBM Research


Collect Alert Examples

Need enough examples to see patterns

– 10 - 100 depending on alert

– Use “Natural” Alerts if frequent enough

– Generate alerts where needed

Alerts should cover most expected conditions

– night, rain, rush hour, weekends, etc

IBM Research


Evaluate Alerts

Classify Alerts as FP/TP

– Customer classification is ideal• Will be supported in future releases if UI

– IBM classification• Look back for Alerts over representative period

– Cover most expected situations• Classify each as TP or FP

– Use thumbnail and video as necessary– Record results manually

• Some cases are subjective– Be consistent– Understandable Miss

> System does right thing, but does not meet use case

Approximate # FNs if possible

IBM Research


Determining Alert Performance in the Field

Ground truth is very difficult to obtain on live data– False Negative statistics are impractical to obtain during deployment

– Therefore Recall statistics are not available

Metrics for deployment tuning– Total Alerts (= TP+FP) / Hr

• Acceptable value depends on customer workload• If few FPs, talk to customer• If many FPs, work to reduce

– TP / FP ratio (~= precision) • >1 for good user acceptance

Because system has gone through certification, we can be confident that if we tune for these metrics, recall will also be acceptable

If customer requires detailed Recall statistics, we can provide as an extra service.

IBM Research


Problem Determination

Examine individual errors for basic problems– Poor view of target

• Lighting– Reflections, Shadows, Headlights, Changing ?

• Size• Occlusion

– High activity– Target easy to confuse with other activity– Shaking camera– Confusing background

Look for patterns over errors– Happen in clusters ?

– Common circumstances ?

IBM Research


Problem Determination cont.

Look at Event data

– Often give you hint as to problem

– Object not tracked

– Track broken

– Bounding box large / small

– Etc.

Basic condition of Alert broken ?

– Not in RoI long enough

IBM Research


Remediation Actions

– RoI

• Size / shape

– Alert Parameters

• Especially size

– Camera angle / view

• Improve object separation, reduce occlusion, etc.

– RoU

– Schedule

– Alert Debounce

– Change Alert used

IBM Research


Tracking and Non-Tracking Alerts

Tracking Alerts

– Tripwire

– Directional Motion

– Region

– Object Alerts

– Motion Detection (tracking version – v3.6 only)

Non-Tracking Alerts

– Abandoned Object / Object Removal

– Motion Detection (default version)

IBM Research


Tracking Alert Issues

Poor quality Event data

Tracking Alert near edge of image

Large bounding box of nearby objects intercept RoI (TW, MD)

– Move / Resize RoI slightly

– Use alert with more restrictive conditions• Directional Motion, Region

– Adjust object size parameter Objects merged in distance and never split

– RoU objects till they get closer (larger)

IBM Research


Issues for Tracking and Non-Tracking Alerts

Object not segmented properly

– Clumped with other objects

• Change camera angle to improve separation

– Only part of object detected

• Object looks similar to background

Alert parameters exclude object

– Size / Color / etc.

IBM Research


Abandoned Object Algorithm

Foreground “blob” remains still long enough to heal into background

Examined healed region to see if it is foreground or background (arrived or removed)

– Texture & Edges

If Arrived (foreground), monitor location to ensure object does not leave

If object disappears for too long (“occlusion time”), quit monitoring

At end of alert timeout, if object still in view trigger alert

Only supported by City Surveillance (xxx) or Alert Detection profiles

IBM Research


Evaluating Parking Alerts

Determine if Alert is FP or true parking violation

– First evaluate Visualization• False alerts

– Empty Box– Box around non-car object

• Valid alerts– Box fits car (or distinct part) well– Box covers part of car

– Look for mitigating circumstances• e.g. unloading truck

– If still unsure, examine video• Click on Alert to start video• FFWD/RWD to observe car since arrival

IBM Research


Abandoned Object Failure Modes

“Heal Type”: object is detected as removed rather than abandoned– Texture in background– Indistinct object boundary– Common when slow moving object moves through RoI

• Causing FPs

Appearance changes during wait time– Open car door– Lighting change– Partial occlusion

Occluded for too long

IBM Research


Common Abd Obj False Positives

FPs can occur when SSS is fooled into thinking an object has arrived by:

– Shadows appearing / disappearing

– Puddles / snow piles

– Water drops on dome

– Slow moving vehicles

– Opening door or other change to car’s appearance

– Other optical illusions

Patterns

– FPs are more common:• at night• in the rain• when there is lots of activity in the ROI• When there is lots of occlusion

IBM Research


Abandoned Object Tuning

Refine Min/Max size

Camera View

– Minimize occlusions, background clutter

RoI

Sensitivity Profiles

– Designed to address a range of causes of poor performance

– Change profile in increase / decrease “sensitivity”• Start with Indoor Tracking (S0)• Move to (S-n) profiles to decrease sensitivity

– Fewer FPs, but more FNs

• Move to (S+n) profiles to increase sensitivity– More TPs, but more FPs

IBM Research


If all else fails…

If none of the previous has improved the problem consider…

– Try another approach to solving use case

– Modify use case• Work with customer

– Give up on this camera / use case.

If you MUST get this use case working and have tried all your alternatives then

– Use IBMSSE tool to examine visualizations• May provide hints as to what is happening

– Contact Global Team• Include video and configuration file and detailed description• We may be able to provide a custom profile to help with the issue

IBM Research


Internal (Profile) Parameter Adjustment

Tuning Internal Parameters can cause several problems

– Maintenance / Support issues

• Confusion about exactly what is running

• Potential loss of changes on system update

• Future tuning

– Improve performance in one situation, decrease performance in another

– Unexpected results

• Even analytic developers often can not predict affect of changes accurately.

IBM Research


Tuning Internal Parameters

Edit Engine Configuration– Program Files/IBM/SSE\SSESystem.xml

– Restart Engine

– * NOT RECOMMENDED*

Create custom profile– Modify existing file (“ProfileName.xml”)

• Save modified version for future reference– Register Profile

– Configure View Analytics to load profile

– Reconfigure Alerts/RoU

IBM Research


Analytic Profile sample

IBM Research


Tuning Guide

IBM Smart Vision Suite 3.6.5 Camera Placement and Analytic Tuning Guide.doc

IBM Research


Tuning Tool

C:\Program Files\IBM\SSE\Release\IBMSSE.exe

Shows internal workings of analytic plug-ins

Can connect to existing SSE or create a dedicated one

Can connect to existing engine or create new engine

IBM Research


Per-camera Acceptance Testing is Dangerous !

Assuming an alert has a true detection rate of 75%:

– If 4 bags dropped, 26% chance <3 detected

– If 8 bags dropped, 32% chance < 5 detected




– If 100 bags dropped, 0.94% chance < 65 detected

– If 1000 bags dropped, 99.97% chance between 700 and 800 bags will be detected.

– Note: these numbers roughly correspond to a 65% success rate on the drops.

Message: Many good performing cameras WILL fail any reasonable demo / acceptance test.

IBM Research


Acceptance Testing Criteria

Requiring each camera to pass performance thresholds is unrealistic

– Requires detailed tuning of each camera (does not scale)

– Requires large amounts of data from each camera to get accurate performance numbers

• Tuning• Testing

Alternative:

– Tune a representative set of cameras

– Propagate tuned profiles to similar views

– Capture small amount of data for each camera

– Report global performance numbers

– Address important or problem cameras individually

IBM Research


Analytic Maintenance

Cameras require periodic attention

– Performance degrades• Cameras move• Lighting conditions change• Activity level changes

– Requirements change

– Expectations change as users gain experience with system and understanding of analytics

ibm research © 2007 ibm corporation tuning sss analytics rick kjeldsen [email protected]

Documents

ibm research

ibm corporation examples

view slide

research staff

customer slide

needed slide

use case slide

rou alerts