behavior analytics in retail - accuracy
DESCRIPTION
Definition and Best Practices in Auditing Accuracy in People Counting (Footfall Traffic), Queue Management, and In-Store Analytics SolutionsTRANSCRIPT
Silicon Waves LLC. All Rights Reserved.
Behavior Analytics in Retail - Accuracy
Definition and Best Practices in Auditing Accuracy in People Counting (Footfall Traffic), Queue
Management, and In-Store Behavior Measurement Solutions
Silicon Waves LLC. All Rights Reserved.
Why should we care about Accuracy?
Silicon Waves LLC. All Rights Reserved.
Accuracy is
the single most important factor in Behavior Analytics
Technologies, such as video, thermal or wireless
Silicon Waves LLC. All Rights Reserved.
Return On Investmentfrom improving sales
conversion and optimization of labor
depends on the accuracy of measuring the
behaviors (activities) of the staff and customers
Silicon Waves LLC. All Rights Reserved.
Increase Conversion by 1%Sales up 3%+
Silicon Waves LLC. All Rights Reserved.
Why 80% Accuracy is Not Enough?
• This system under-counts randomly between 5% to 20%, therefore the Sales Conversion is higher than the actual conversion
• The system’s data distorts the sales opportunities
• Accuracy errors can accumulate….
System’s Conversion
Rate
ActualConversion
Rate
Silicon Waves LLC. All Rights Reserved.
High Accuracy, Better Store
Performance, More Sales!
Silicon Waves LLC. All Rights Reserved.
The Accuracy Rateis the percentage of system
counts to actual behavior, i.e. 95%
Silicon Waves LLC. All Rights Reserved.
Accuracy for Line Counts
refers to the number of people crossing a line, per
period of timeFor example
95% (or 105%) accuracyper 30 minutes
for the raw data of Arrivals
Silicon Waves LLC. All Rights Reserved.
Accuracy for Queues refers to the behavior of people waiting (standing and moving)
inside a specific area, per period of timeSuch as
95% accuracy for calculating the average number of people
waiting for service, in the last 10 minutes
Silicon Waves LLC. All Rights Reserved.
The Accuracy Rateis defined by
Data ConsistencyData Granularity
and Behavior Anomalies
Silicon Waves LLC. All Rights Reserved.
Data consistency refers to the consistency of the
error, over-count or under-count but not both, and avoiding
inaccuracy spikes and troughs
Silicon Waves LLC. All Rights Reserved.
Data Consistency - Pass
• Compare Arrivals Data with 2 video clips.
• Each video clip runs 15 minutes, for a total of 30 minutes auditing time, and actual count of 100 people entering a store.
• If the system counted 100 people and the auditor viewed the video clips and manually counted 100 people entering the store during the 30 minute period, then the accuracy rate is 100%.
Silicon Waves LLC. All Rights Reserved.
Data Consistency - Pass
• If in each of the 15 minute videos, the system’s data is 50, and the auditor saw 50 people, then the Accuracy Rate is 100%, for each of the 15 minute periods.
• 100% correlation for the 30 minute period
• 100% correlation for each 15 minute period
Audit Result – Pass!
Silicon Waves LLC. All Rights Reserved.
Data Consistency - Fail
• If in the first 15 minutes, the system counted 75 arrivals, but the auditor saw only 25 people enter the store, then the accuracy is 0%.
• If in the 2nd clip, the system output is 25 arrivals, but the auditor counted 75 people in the video clip, then again the accuracy is 0%
Silicon Waves LLC. All Rights Reserved.
Data Consistency – Fail For 30 minute period
total counts are the same for system (75+25=100)
and actual (25+75=100) but
inconsistency of the 15 minute counts fails the audit
Silicon Waves LLC. All Rights Reserved.
The System canOver-Count or Under-
Count not Both!
Silicon Waves LLC. All Rights Reserved.
Data Granularity refers to the time period
of raw data such as the number of people
entering the storeevery 15 minutes
Silicon Waves LLC. All Rights Reserved.
Data Granularity in Arrivals
• Arrivals: An audit of the number of people entering a Bricks-and-Mortar store includes 12 segments of 15 minute video clips
• 3 Hours Accuracy Rate compares the system’s 354 people to the actual arrivals of 352, for accuracy of 99%
99% Accura
cy
Silicon Waves LLC. All Rights Reserved.
Data Granularity in Arrivals
• Over-Count: At the 4pm segment, the system’s data of Arrivals was higher than the Actual Arrivals by 22 people (55 versus 34)
• The 15 Minute Segment of 4pm to 4:15pm has the over-count variance of 64%
64% Over
-Coun
t
Silicon Waves LLC. All Rights Reserved.
Data Granularity in Arrivals
• Under-Count: At 5pm, the Arrivals of the system was lower than the Actual Arrivals (11 versus 28)
• The 15 minute segment of 5pm to 5:15pm has the under-count variance of 60%
60% Under
-Count
Silicon Waves LLC. All Rights Reserved.
Data Granularity in ArrivalsFor 3 hour audit, the Accuracy Rate is 99%
In fact, the audit is a failure, since the system over-counts and under-counts, in the required 15
minute segments
Under-Count
Over-Count
Silicon Waves LLC. All Rights Reserved.
Best Practices in Granularity
are30 Minutes for Sales Conversion,
15 Minutes for Labor Optimization,
5 Minutes for Queue Management,
2 Minutes for Predictive Scheduling
Silicon Waves LLC. All Rights Reserved.
Counting Employees
• Traffic counters capture the behaviors of all people, including employees.
• For many retailers, the behavior of employees is consistent enough to be part of the demand trend.
• For example, if the employees enter the stores before shifts and exit when they are done for the day.
• For example, if the staff uses a Back Door.• For example, if the staff uses the Exit Door.
Silicon Waves LLC. All Rights Reserved.
Why Not Factor?When we factor, we define the
behavior of employeesas constant
and consistent for every period of time and for every traffic level
Silicon Waves LLC. All Rights Reserved.
Why Not Factor?
• In Factoring, we estimate the percentage of staff movements and deduct it from the actual count.
• For example, 10% factoring will take out 10 out of 100 arrivals in the peak hour, and 2 out of 20 people at the closing hour.
Silicon Waves LLC. All Rights Reserved.
Factoring omits a constant percent of traffic out of total demand,
in all time periods, and therefore is
not recommended!
Silicon Waves LLC. All Rights Reserved.
Many retailers, can ignore the bias in
Counting Employees and focus on the trends
Silicon Waves LLC. All Rights Reserved.
Behavior Anomalies - Guards & Greeters
• In luxury stores, it is typical to post greeters, who also serve as guards, at the main entrance.
• In addition to security duties, the guard also opens and closes the door for customers, especially in high-street stores located in cold climates.
• Since the movements of the guard may not be a complete entry or exiting behaviors, most premium counting technologies can identify and discard the guard’s counts.
Silicon Waves LLC. All Rights Reserved.
Behavior Anomalies – Racks & Displays
• Common challenge to accurate counts occurs when a display or a rack is positioned close to the entrance and customers hover around the counting zone.
• For this behavior, the counts can be adjusted by separating between passing and standing behaviors.
• By identifying the length and direction of motion, a technology can pin point a customer who walks by the entrance area and a person who actually enters or exits.
Silicon Waves LLC. All Rights Reserved.
Behavior Anomaliespose challenges to the accuracy of the count, and the ability to
deal with such behaviors is a key differentiator among vendors
Silicon Waves LLC. All Rights Reserved.
Best Practices in Auditing Compare
System Counts to Feeds For example
Arrivals Raw Data to Video Clip
Silicon Waves LLC. All Rights Reserved.
Best Practices in Auditing
No Manual Counting
No Clickers
Seriously!
Silicon Waves LLC. All Rights Reserved.
Best Practices in Auditing Compare
Different Time Periods, Traffic Levels,
and Changes in the Store’s Environment
Silicon Waves LLC. All Rights Reserved.
Best Practices in Auditing
Define Requirements for
Line (Door) Counting Queue Management
and Service Time
Silicon Waves LLC. All Rights Reserved.
Auditing Bi-Directionality
Audit how many people cross the
line per period of time for each direction - Entry and Exit
Silicon Waves LLC. All Rights Reserved.
Auditing Time Segment
Audit for the minimal time period required for the solution
such as 15 minute segments
for Labor Optimization
Silicon Waves LLC. All Rights Reserved.
Auditing Traffic LevelsFor statistical purposes,
an acceptable level of traffic todetermine the accuracy rate of the audit should cover traffic
levels ofno less than 100 people
Silicon Waves LLC. All Rights Reserved.
Auditing Queuesrequire special parameters
includingWaiting or Waiting Time
Time ThresholdDropping Error
Silicon Waves LLC. All Rights Reserved.
Auditing QueuesKey Performance Metrics are
Per Period of TimeWaiting – Average Number of
people waiting in line for service
Waiting Time – Average Waiting Time in the queue
Silicon Waves LLC. All Rights Reserved.
Auditing #Waiting
• Data Table: We use a system that tracks individuals, each with their own random Identification Number (ID) and Waiting Time.
• 5 Minute Segment: Summary for the 5 minute segment is 19 people each, and the error is 0%.
• System = 19 people (ID 1016 = 0)
• Actual = 19 people (ID 1011 = 0)
Silicon Waves LLC. All Rights Reserved.
Auditing Time Threshold
• Waiting Behavior: ID 1016 is discounted by the system because the real person stood in the queue for only 4 seconds
• Time Threshold: This event is not considered as an error because Waiting in Queue has a 5 seconds Time Threshold
• 4 seconds is Passing Behavior and therefore is discarded from the count
Silicon Waves LLC. All Rights Reserved.
Auditing Dropping Error
• Dropping Error: The system captured images of ID 1010 and ID 1011 incorrectly. The video clip showed an image of 1 person, but the system counted the single individual as two separate people
• Typically, this error occurs when a person stands in a spot for too long without moving
Silicon Waves LLC. All Rights Reserved.
Auditing #Waiting
• Calculate the Accuracy Rate: Discounting the Threshold event of ID 1016, we have 18 correct counts out of the actual count of 19 people
• 18/19= 95% Accuracy Rate.
95% is a superior Accuracy Rate for a 5 minute audit of the Queue Length (Waiting
metric)
Silicon Waves LLC. All Rights Reserved.
Auditing Waiting Time
• Waiting Time: For simplicity, we audit the average of the 20 recorded events
• System’s Average Waiting Time is 1:62 minutes
• Actual Average Waiting Time (manually measured from the video clip) is 1:78 minutes
• 5 minute audit period has an Accuracy Rate of 91%
Silicon Waves LLC. All Rights Reserved.
Auditing Waiting TimeCorrectly
Capturing Waiting Time may be the most challenging
component in counting technologies
Proceed with caution!
Silicon Waves LLC. All Rights Reserved.
Auditing Queues85% is a Good Accuracy Rate for Waiting
Time
Silicon Waves LLC. All Rights Reserved.
Waiting Time ModelsSuch as
Serve 90% of Customers in less than 3 minutes
Are cutting-edge techniques for Workforce Optimization
Silicon Waves LLC. All Rights Reserved.
Auditing Service Time
• Service Time refers to how long people stand inside a defined virtual area (i.e. Zone)
• Shopping Unit: The business function classifies the occupants of the zone as a single buying transaction
• Occupied / Not Occupied: This means the number of people inside the zone can be either zero (none) or 1 (occupied), and as long as the zone is occupied, the system continues to count the Service Time
Silicon Waves LLC. All Rights Reserved.
Auditing Wireless TechnologyWi-Fi and Bluetooth Technologies
depend on active features in the Customer’s Smartphone
thereforeAccuracy relates only
to customers captured in the sample
Silicon Waves LLC. All Rights Reserved.
The Accuracy Statement – Technology Vendor
• Solutions: The company performed absolute accuracy tests for all device configurations, including counting, queuing, detection, service and tracking.
• Granularity: Tests include granularity of audit per individual track and per 15 minute averages
• Behaviors: individual and group behavior; adults and children height differentiation; and extreme and fluctuating ambient and light environments
• Accuracy Rate: consistent accuracy of 98%.
Silicon Waves LLC. All Rights Reserved.
The Accuracy Statement – Value Added Reseller
• Local Environment: The accuracy of the sensor in use depends on the quality of the installation and on the anomalies of behavior.
• Installation Standards: For maximum accuracy, the company uses common auditing parameters and certified installers.
• Accuracy Rate: For typical installations, minimum consistent accuracy level is 95%.
Silicon Waves LLC. All Rights Reserved.
Managing for Accuracy
• While Accuracy is the single most important factor in identifying a solution, system, and vendor, there are other aspects to consider in the decision.
• Each thread of the project comes from a different company—Technology Vendor, Installation Company, Value Added Reseller, and the Retailer itself.
• Follow Best Practices for conducting transparent audits, and promote communication across all levels.
Silicon Waves LLC. All Rights Reserved.
Best Practices in Managing for Accuracy
The Retailer’s Request for Proposal (RFP)
should specify auditing requirements, and ask for the
vendor’s guidelines in defining the Accuracy Rate
Silicon Waves LLC. All Rights Reserved.
Our mission is to nurture, train, and educate a community of behavior analytics professionals
Ronny Max is the author of “Behavior Analytics in Retail” (October 2013), and the founder of Silicon Waves, a consultancy specializing in Behavior Analytics, including People Counting, Queue Management and Schedule to Demand.
Website: BehaviorAnalyticsRetail.com