human-behavior sensing and visualization for service...
TRANSCRIPT
Human-Behavior Sensing andVisualization for Service Quality Control
Takeshi Kurata, Masakatsu Kourogi, Takashi OkumaTomoya Ishikawa, Ryoko Ueoka, Ryuhei Tenmoku, Koji Makita
Center for Service Research, AIST, Japan
CSCW 2012 Workshop: Exploring collaboration in challenging environments: From the car to the factory and beyond
Evidence-based Service (EBS) Through Real-World Virtualization
• Real-Virtual correspondences of products with POS (Point-Of-Sales) systems– Facilitate modeling and designing the flow of the
products by not strongly relying on tacit knowledge.– Brought about drastic changes in retail, chain
restaurant, logistics, etc.– Realized EBS to some extend for Service Quality
Control (QC).
On the analogy…• One of the next key issues for service innovation How to Make better correspondence betweencustomers/employees/environmentsand the computerized ones.
14
Virtualizing Human Behaviors and Environments in Service Fields
• Current Progress in virtualizing human activities– ID-POS (POS in cooperation with Customer
Relationship Management (CRM))– Driving recorders in logistics and transportation
services– Participatory sensing such as Foursquare
• Not yet realized to sufficiently grasp– Where customers are in a shopping mall– What employees are doing in a large facility
• Needs for virtualizing environments– In a laboratory it should be known or controllable.
2
MRIS in Service Engineering• If human behavior and environments are clarified…
– More comprehensive and synthetic visualization (Mieruka: 見える化) for Evidence-based Service (EBS ) by the combination of
• “Results” such as POS accounting data• “Behavior” that brought the results• “Environmental stimuli and Constraints” that influenced the
behavior
• In this talk,– Human behavior measurement of employees and the
visualization by using “Mixed-Reality Information Sharing” (MRIS) technologies
– Applications in actual service fields• Japanese restaurant, Nursing home, Japanese-style hotel
3
HBIplus for Service Quality Control4
* HBI: Human-Behavior Indicator
Human Behavior Human Behavior Indicators and
Responses
Labor quantity, Trajectory, Service operation log
Resulting Indicators
Environment Stimuli and Constraints
Diverse and Normalized Indicators
Accounting history, Nurse call log, Energy consumption indicators, Plant operation log, Subjective evaluation results
Alleviation of issues and biases due to Intangibility and Heterogeneity
Layout, Appearance, Perception, Skill, Economy, Brand image
MRIS Technologies for Multi-Stakeholders5
PDR(Pedestrian Dead-Reckoning)
Estimates velocity vector, relative altitude, and actionsby measurements from waist-mounted sensor module.Wearing sensor module on waist Easy to wear and maintain Easy to measure data for action
recognition Relatively easily to apply for handheld
setting compared to she-mounted PDR based on Zero Velocity Updates (ZUPTs)
Recognition of walking locomotion Low-cost sensors
M. Kourogi and T. Kurata, “Personal Positioning Based on Waling Locomotion Analysis with Self-Contained Sensors and a Wearable Camera”, ISMAR2003, pp. 103-112, 2003.
15
Sensor module・Accelerometers・Gyro-sensors・Magnetometers・Barometer
Indoor Pedestrian Tracking Using SDF
Pedestrian Dead-Reckoning (PDR)
IDreader
IDRSSI
Acceleration /angular velocity
Building Structure/Layout
Magneticvector
Magnetometer
Output of position/orientation
Pair-wise-sensorbased positioning
Atmospheric pressure
Barometer
TrajectorySensor/Data Fusion (SDF)
(Particle filter)
Accelerometers/ gyro-sensors
Walking velocity
Position / Orientation
Trajectory matching/Velocity estimation
Absolute position
3D environment model
Velocity vector /Relative altitude /Action type
Sensor module
ActiveRFID tagID
Surveillance camera/
RGB-D sensor
ID-LEDID
Video/Depth
T. Ishikawa, M. Kourogi and T. Kurata: "Economic and Synergistic Pedestrian Tracking System with Service Cooperation for Indoor Environments", Int’l Journal of Organizational and Collective Intelligence, Vol.2, No.1 pp.1-20 (2011)
Application 1: QCC in Japanese Cuisine Chain Restaurant
7
Staying-time rate at each premise area per personSales at each premise area per employee
Visualization tool combining human-behavioral and accounting history
Employee taking order while cleaning up the
guest room
Icons showing the number of customers at each table
POS data log
QCC in manufacturing industryPurpose: Productivity improvement
Conventional QCC in service industryPurpose: Productivity improvement
Subjective QCC in service industryPurpose: Improvement of CS/ES
Computer-supported QCC (CSQCC)Purpose: Productivity improvement
w/ reasonable ways to gather objective data in plants
In 1980s, applying QCC for service industry
w/o reasonable ways to gather objective data in service fields
In 1990s, Service industry lost interest in QCC
In 2000
In 2010
CSQCC in the futureProductivity improvementImprovement of CS/ES
QCC in the Service Industry in Japan8
w/ reasonable ways to gather subjective data continuouslyw/ reasonable ways to
gather objective data in service fields
During Discussion in CSQCC9
Trajectory of a waitress (hostess) in lunch time: 12:00-14:00
R. Ueoka, T. Shinmura, R. Tenmoku, T. Okuma, T Kurata: “Introduction of Computer Supported Quality Control Circle in Japanese Cuisine Restaurant”, In Proc. Int’l Conf. Human Side of Service Engineering (HSSE2012) jointly with AHFE2012, July, 2012.
Fact: Going in and out of the kitchen/office to no small extent.Possible result: Difficulty in concentrating on guest service.Cause: Cell phone everywhere, but reservation book only in the office room.Possible improvement: e-reservation book
Front of the house
Kitchen
Office room
Application2: Job Analysis in Nursing Home
• How long it took to respond to the nurse call (NC)• Which care worker responded to the NC• Where and how far the care worker and other workers were when the NC was switched on• How each care worker actually took action until the NC was turned off
10
Indicator for on-state of NC by the resident of room #201
Care worker responding to
nurse call (NC)
Timeline showing on/off-state of NC with room #
NC data log
40
60
80
100
40
60
80
100
40
60
80
100
• Nurse R: Role as a leader. Mainly desk work and sometimes vital check of residents.• Nurse S: Taking care of each resident while relatively flexibly circulating.
Care worker E, I, K Care worker D, H, MCare worker A, G
• Flexibly changing the role?• Or low skill?
• High skill?• Or assigned at specific floor?
• Mainly desk work?
# of steps
# of utterance(VAD)
# of floor change
Time spent inresidents’ rooms
Nurse R
Nurse S
# of steps
# of utterance
# of floor change
Time spent inresidents’ rooms
# of steps
# of utterance
# of floor change
Time spent inresidents’ rooms
Voice Activity Detection (VAD) FrequencyLow High
RestroomBath/Dressing roomResidents’ rooms Corridor Nurse Station Stairs/EV Dining room
Application2: Job Analysis in Nursing Home
Validation of the hypotheses on what is related to high skills:e.g. ‘Workers who are skillful at comprehensive awareness is to talk to residents frequently everywhere, but each conversation is basically short.’
Interview with FPV
Passage of Time
+ Over 50% cost reduction on labor cost and preparation time compared with existing time studies+ Consideration of customer privacy by not using cameras+ FPV with less motion sickness+ Effective in episodic memory retrieval for retrospective interviews based on definitive rationality
Worker’s trajectory
3D model built from a set of photos
First-person view (FPV)
CCE (Cognitive Chrono-Ethnography) Lite
Results, Behavior, Stimuli, and Constraints(Transforming Service Fields into Laboratories)
Human Behavior
Results
Environment Stimuli and Constraints
InfluenceInfluence
Decision-making process• Definitive rationality• Satisficing principle
Long-term Trend
Accounting, Energy consumption, Plant operation, Others’ condition (CS/ES)
Layout, Appearance, Weather, Economy, Brand image
InfluenceInfluence
13
Virtualization in indoor environments
AR/DR Presentation
AR/DR Presentation
Skill, Perception
Influence