arpan pal csi2012
TRANSCRIPT
1
Personal Context Discovery using Unobtrusive Sensing
1st December 2012CSI 2012
Arpan PalPrincipal Scientist and Research HeadCyber physical SystemsInnovation Lab, KolkataTata Consultancy Services (TCS)
2
Human-in-Loop Cyber-physical Systems
Humans
Computing Infrastruct
ure
ICT Systems
3
Human-in-Loop Cyber-physical Systems
Humans
Physical Objects
and Infrastruct
ure
Computing Infrastruct
ure
Cyber-physical Systems
4
Human-in-Loop Cyber-physical Systems
Humans
Physical Objects
and Infrastruct
ure
Computing Infrastruct
ure
Human-in-loop Cyber-physical Systems
Perso
nal
Conte
xt
Disco
very
5
Personal Context Discovery
Context - patterns of individual, group and societal behaviours.
Broadly classified into three categories –Personal Physical Network Discovery
Who is interacting with whom? What is the level of interaction? Who all are part of similar-interest networks?
Individual Context Discovery Who is doing what? Who is thinking what?
Community Context Discovery Can we discover how a community / group behaves as a whole?
6
Example Use Cases
Customer Behavior Study in Retail Stores Customer movement pattern Customer interaction pattern with shelves / merchandize
Crowdedness measure in public places Efficient scheduling of public transport
Wellness Activity and Work-out Quantification Pulse and other physiological parameter measurement
Organizational Behavior Team Efficiency / Best Practice Study Workspace Ergonomics - Stress Analysis People Profiling – Cognitive Load Analysis
Ref. - Alex Pentland et. al., MIT media Lab
7
What do we need to Sense
Location and Proximity
Activity
Identity
Cognitive Load
Physiological Parameters
Provide Personal Context
discovery as a Service
8
How to Sense
Requirement• Needs to be Ubiquitous and Unobtrusive• There should not be any new hardware / device to carry
for an individual
Approach• Use smartphone-based sensors (GPS, accelerometer,
compass, microphone, camera)• Use 3D surveillance cameras (like Kinect)• Use wearable EEGs with mobile phone as gateway• Augment with social network data and email data
analytics• Multimodal fusion of all the above
Issue• Privacy can be an issue – needs to be handled on an use
case-by-use case basis• Privacy vs. Utility
9
Mobile Phone Based Sensing
Proximity / presence– Using Bluetooth – Using Wi-Fi
Location– Using ultrasound beacon– Using GPS (outdoors)– Using Accelerometer /
compass / gyroscope
Activity– Using Accelerometer
Interaction Level– Using Microphone Audio
Identity– From Network ID
Physiological Sensors– Pulse rate using camera - PPG
On-board sensorsAccelerometer, GPS, Compass, GyroscopeCamera, Microphone
NetworkBluetooth, WiFi, 2G/GPRS, 3G
Network2G/GPRS, Bluetooth
On-board sensorsMicrophone, Camera
10
Kinect Based Sensing
Human Identification– Skeleton Model Based– External Stimulus based
refinement
Network Discovery– Network discovery through
proximity– Level of Interaction through
Audio
• 2D Camera with IR depth sensor• Excitation by IR light pattern• Directional Mic.
Human Interaction – Activity Detection on 3D Point
Cloud– Physical object Identification– Interaction with ObjectsExample Activities• People Discussion• Give/Put/Take an object• Enter/leave a room• Handshaking
11
EEG Based Sensing
Cognitive Load Detection– Wearable EEG headset – Biggest challenge is collecting
annotated data and removing movement artifacts
Signal Acquisition
Pre-processing (Common Spatial
Pattern filter)
Feature Extraction
Classification / Score generation
Measure of the cognitive load
Feature-1
Featu
re-
2
High cognitive load
Low cognitive load
12
Soft sensing from Web
Unstructured Data• Social network posts such as
tweets, facebook• Blog posts• Email bodies
Structured Data• Social network profiles and
network information• Email headers• Tweet Attributes
Personal
Network
13
Summary
Multimodal Fusion
Location Identity Activity
Sensors• GPS, accelerometer, compass, microphone, camera on SmartPhone • 2D / 3D surveillance cameras (like Kinect)• Wearable / Mobile-phone hosted Physiological Sensors – Pulse, ECG, EEG• Soft sensors from web and social network
Cognitive Load & Phys. Sensing
Context DiscoveryPhysical, Individual & Community
Applications
