1

Personal and Community Context Discovery

15 April 2023

Arpan PalPrincipal Scientist and Research HeadInnovation Lab, Cyber physical SystemsTata Consultancy Services (TCS)

2

Tata Consultancy Services Ltd. (TCS)

Pioneer & Leader in Indian IT

TCS was established in 1968

One of the top ranked global software service provider

Largest Software service provider in Asia

250,000+ associates

USD 10B + annual revenue

Global presence

First Software R&D Center in India

- 2 -

3

The Heart of Innovation – TCS Innovation Labs

Bangalore, India1

TCS Innovation Labs - Bangalore

Chennai, India2

TCS Innovation Labs - ChennaiTCS Innovation Labs - RetailTCS Innovation Labs - Travel & HospitalityTCS Innovation Labs - InsuranceTCS Innovation Labs - Web 2.0TCS Innovation Labs - Telecom

Cincinnati, USA3

TCS Innovation Labs - Cincinnati

Delhi, India4TCS Innovation Labs - Delhi

Hyderabad, India5

TCS Innovation Labs - HyderabadTCS Innovation Labs - CMC

Kolkata, India6

TCS Innovation Labs - Kolkata

Mumbai, India7

TCS Innovation Labs - MumbaiTCS Innovation Labs - Performance Engineering

Peterborough, UK8

TCS Innovation Labs - Peterborough

Pune, India9

TCS Innovation Labs - TRDDC - Process EngineeringTCS Innovation Labs - TRDDC - Software EngineeringTCS Innovation Labs - TRDDC - Systems ResearchTCS Innovation Labs - Engineering & Industrial Services

1 2

3

4

597

6

8

2000+

Associates in Research, Development and Asset Creation

19 Innovation Labs

4

Integrated Platform for Intelligent Infrastructure

People Feedback & Emotions

Social Media

Integrated Services

Sensors & IoTPlatform

Traditional Monitoring & Control Systems Citizen Data

Smart Integration Platform

Transportation Healthcare Electricity

WaterPublic Safety Tourism

Smart Integrated Services

Sense

Analyze

Extract

Respond

Intelligence

Smart Domain Services

Community

etc.

Sense: People Activity, Appliances, Vehicles , Road, Home/Bldg, Utility Infrastructure

Detect gas leakage/water contamination : mobilize rescue team, suggest optimum route

Divert Road Traffic in case of Water Pipeline Burst

Correlate Electricity/Water /Gas consumption patterns

Intelligent Integration Platform

Integrated Intelligent Services

OutlineWhat we mean by ContextExample Use CasesProposed SystemConclusion

6

Personal and Community Context Discovery

Context - patterns of individual, group and societal behaviours.

Broadly classified into three categories – Personal Physical Network Discovery

Who is interacting with whom? What is the level of interaction? Who all are part of similar-interest networks?

Individual Context Discovery Who is doing what?

Community Context Discovery Can we discover how a community / group behaves as a whole?

7

Example Use Case - Campus

Source: Zhang et. al., “Extracting Social and Community Intelligence from Digital Footprints: An Emerging Research Area”, UIC 2010, LNCS 6406, pp. 4–18, 2010. © Springer-Verlag Berlin Heidelberg 2010

8

Other Example Use Cases

Organizational Behavior Analysis Team Efficiency Study Best Practice Study Workspace Ergonomics Study

Customer Behavior Study in Retail Stores Customer movement pattern Customer interaction pattern with shelves / merchandize

Crowdedness measure in public places Efficient scheduling of public transport

Crowd Behavior analysis Evacuation planning during disaster

Ref. - Alex Pentland et. al., MIT media Lab

9

What do we need to Sense

Location

Proximity

Activity

IdentityProvide Context discovery as a

Service

10

How to Sense

Needs to be Ubiquitous and Unobtrusive There should not be any new hardware / device to

carry for an individual

Proposalo Use smartphone-based sensors (GPS, accelerometer,

compass, microphone)o Use 3D surveillance cameras (like Kinect)o Augment with social network data and email data

analyticso Multimodal fusion of all the above

Privacy can be an issue – needs to be handled on an use case-by-use case basis

o Privacy vs. Utility

11

Proposed Architecture

Platform Service

c

Sensors (location

, proximit

y, activity)

CameraWeb-

based soft sensing

Personal Physical Network

Discovery

Individual Context Discovery

Microphone

Community Context Discovery

Behavioral Analytics

Applications

Context Discovery Service

Mobile Phones, Kinect, Email, Social Network

cMultimodal Fusion

12

Mobile Phone Based Sensing

Proximity / presence– Using Bluetooth for finding nearby mobiles– Using Wi-Fi to discover other mobiles nearby

Location– Using ultrasound beacon– Using GPS (outdoors)– Using Accelerometer / compass

Activity– Using Accelerometer

Interaction Level– Using Microphone Audio

Identity– From Network ID

On-board sensors

Accelerometer, GPS, Compass

Camera, Microphone

NetworkBluetooth, WiFi, 2G/GPRS, 3G

Network2G/GPRS, Bluetooth

On-board sensors

Microphone, Camera

13

Sensor Penetration and power consumption in Mobile Phones

0 20 40 60 80 100

Bluetooth

USB

Edge

GPRS

Wifi

3G

Camera

GPS

Accelarometer

Digital Compass

Consolidated Market Penetration

Source: Nericell: Rich Monitoring of of Road and Traffic Conditions using Mobile Smartphones, Prashant Mohan et. al., Microsoft Research, SenSys 2008, North Carolina, USA

14

Kinect Based Sensing

Human Identification– Skeleton Model Based– External Stimulus based refinement

Network Discovery– Network discovery through proximity– Level of Interaction through Audio

• 2D Camera with IR depth sensor• Excitation by IR light pattern• Directional Mic.

15

Kinect Based Sensing (contd …)

Working on a public Kinect dataset• People Discussion• Give/Put/Take an object• Enter/leave a room• Leave baggage unattended• Handshaking• Typing on a keyboard• Telephone conversation (Mobile, landline)

Image and the corresponding 3D cloud point

Human Interaction – Activity Detection on 3D Point

Cloud– Physical object Identification– Interaction with Objects

Human activities recognition and localization competition (HARL), ICPR 2012

16

Soft sensing from Web

Unstructured Data• Social network posts

such as tweets, facebook

• Blog posts• Email bodies

Structured Data• Social network profiles

and network information

• Email headers• Tweet Attributes

Personal Network

17

IoT Platform from TCS

Internet

End Users Administrators

Device Integration & Management Services

Analytics Services

Application Services

Storage

Messaging & Event Distribution Services

Ap

plic

ati

on

Serv

ices

Presentation Services

Application Support ServicesM

iddle

ware

Edge Gateway

Sensors

Internet

Back-end on Cloud

RIPSAC – Real-time Integrated Platform for Services & AnalytiCs

TraditionalInternet

Service Delivery Platform & App Development Platform

Security/Privacy Framework

Lightweight M2M Protocols

Analytics-as-a-Service

Social Network Integration

SDKs and APIs for App developer

18

Summary

Introduced the concept of personal and community context discovery as a service with help of example use cases

Proposed an unobtrusive and ubiquitous way to gather the context through mobile phone based sensors, 3D camera and web data

Each method has its own limitation both from application and technology perspective Need for a multimodal fusion for improved

accuracy

A generic IoT platform to implement and deploy the services for application developers

Thank You

arpan.pal@tcs.com