1 Copyright © 2014 Tata Consultancy Services Limited

Internet-of-Things and Cyber-physical Systems- Exploratory Research in Signal Processing, Communication and Computing

Dr. Arpan PalPrincipal Scientist and Research HeadInnovation Lab, KolkataTCS

Apr 15, 2023

2

Cyber-physical Systems – Internet-of-Everything

Humans

Physical Objects

and Infrastruct

ure

Computing Infrastruct

ure

Peo

ple

Con

text

Dis

cove

ry

PhysicalContext Discovery

INTERNET OF EVERYTHING

Physical Context

Discovery

What is happening, where and when

People Context

DiscoveryWho is doing what, where and when,

who is thinking what

Internet of

Digital

Internet of

Things

Internet of

Humans

ABI Research. May 7, 2014

"In the next century, planet earth will don an electronic skin. It will use the Internet as a scaffold to support and transmit its sensations.“ - Neil Gross 1999

3

It’s a Connected Universe

NEVER FORGET YOUR PILLS MONITOR YOUR ACTIVITYMONITOR THE AGED

source: http://postscapes.com/internet-of-things-examples/

HEAT YOUR HOME EFFICIENTLY

MAKE SURE THE OVEN IS OFF

TRACK DOWN THOSE LOST KEYS

KEEP THE CITY CLEAN RECEIVE POLLUTION WARNINGSUSE ELECTRICITY MORE EFFICIENTLY

ConnectedIndividual

ConnectedHome

ConnectedCity

4

Research Programs Outline

Mobile Phone Sensing

Camera Sensing Other SensorsBio-Sensing

Signal and Image ProcessingProtocols and Networking

Parallel and Distributed ComputingData Analytics (Computational and Semantic) and Modeling

EDGE

CLOUD

Personal Context Discovery (Location, Activity, Psychology)• Mobiles• Cameras • Bio-sensing

• Manage Scale •Reduce Network Load•Increase Compute Capacity•Reduce Storage Requirement

• Handle Interoperability

• Easy-to-use Analytics

Physiological Sensing using Mobile Phones

Mobile phone and Robot based sensing

Human Activity Detection and

Behavior Modeling

IoT Platform Solutions

Affordable Wellness

&Healthcare

Mobile Interactive Remote Sensing

PROGRAMS

5

Click to edit Master title styleProgram: IoT Platform Solutions

6

Integrated Platform for Intelligent Enterprise

People Feedback & Emotions

Social Media

Integrated Services

Sensors & IoTPlatform Legacy Monitoring & Control Systems Enterprise Data

Smart Integration Platform

Transportation Human Resources Energy

OperationsSafety Asset Tracking

Smart Integrated Services

Sense

Analyze

Extract

Respond

Intelligence

Smart Domain Services

Supply Chain

Security and Surveillance

Sense: People Context, Appliances, Building, Plant, Utility Infrastructure

Sync Transportation with Remote Operations Link Asset Tracking and Safety with Surveillance

Employee Wellness and Energy Preservation as Community Initiatives

Intelligent Integration Platform

Integrated Intelligent Services

7

Requirements and Challenges for IoT – Need for a Platform

Applications need support for

Visibility Capture & store data from sensors

InsightsPatterns, relationships and models

Control Optimize and actuate

TCUP – TCS Connected Universe PlatformA horizontal platform for addressing the IoT Software and Services market

Model-driven DevelopmentModel the Domain Knowledge

Model the Infrastructure – Network, Storage, Compute

Model the Analytics – map to Domain Requirements

Model the Architecture – Device and Cloud

TCUP Platform

Model the Sensor – Semantics, Phenomenon

8

TCUP Design and Architectural Highlights

18 patents filed, Standard Body Contribution - IETF and Singapore ITSC

• Fog Computing – Utilize unused compute power of edge devices

Distributed Computing on Edge Devices

• To reduce network congestion• Adequate Security and Reliability

Adaptive, Lightweight yet Secure Communication Protocols

• For economical scaling of sensor data store

Efficient Compression

• Statistical and Information-theoretic measure to find out potential privacy-breaching content

Sensitivity Measurement and Privacy Preservation

• Semantic annotation of sensors• Sensor Search Engine

Semantic Enabled Sensor Explorer

• Algopedia – Algorithm Repository, Search and Recommendation• Semantic Sensor Web

Model-driven Development

ManageScale,

Reduce Cost

Handle Privacy

Manage Diversity and Inter-

opEase of

Development

9

Horizontal operators(semantic integration) operates on data from heterogeneous sources to created integrated data streams.

Semantic Sensor Web - From Data to Wisdom

temperature

humidity

odor

image

high temperature

gaseous odor

light

concentrated light

high temperature indicates fire

gaseous odor indicates gas discharge

Fire from Gas Leak, evacuate

immediately, send fire fighting team

equipped with gas leakage

data

information

knowledge

wisdom

Vertical operators(semantic abstraction) operates on artifacts at each level and transcends them to the next level

F PCS(Data, KB*) → Information

F PCS(Knowledge, KB) → Wisdom

F PCS(Information, KB) → Knowledge

KB: Knowledge base

Adopted from: Physical-Cyber-Social Computing: An early 21st Century Approach, Amit Sheth et. al.

10

Research Outcomes – Some of the Results

Publications in ACM Sensys, Ubicomp, Infocomm, Middleware

7 times less network load

10 times less latency

Lightweight Security – 5

times less overhead

Improved Privacy via Sensitivity Detection

Improved Compute through Edge Devices

11

Program: Human Behavior Modeling and Data Collection

12

Human Data Collection and Behaviour Modelling - Program Overview

Research Goals - Given a context, predict behaviour. Given behaviour, find out context.Focus Domains – Organization Behavior, Consumer BehaviorCurrent Available Models – Statistical, Need of the Day – Models based on physical data

Meetings – Group or One-on-one

Digital Communication

Individual @Work or Leisure

Individual – Day in the Life Of

• Strength and Polarity of Relationships

• Meeting Flow• Emotive State• Outcome

• Formal / Informal, Business / Social

• Strength and Polarity of Relationships

• Tenor of Communication

• Nature of Communication

• Location and Time spent

• Mood and Physical State

• Engagement Level

• Cognitive Load

• Location and Time spent

• Activity• Behavioral

Routine• Social

Interaction• Moment of

decision makingMicrophone

Email / Knome

Mobile Phone, Desktop Mounted Cameras, EEG/GSR

Mobile Phone, Kinect, EEG/GSR,Smart MeterSurveillance Camera

13

Human Identification and Activity Detection using Kinect

Human Identification– Skeleton Model Based / Depth

based– 20 joints of skeleton data

• 2D Camera with IR depth sensor

• Excitation by IR light pattern

Human Identification • Gait cycle detection• Feature extraction from

skeleton joints• Training• Recognition

Papers in IEEE Fuzz, CEC, IEEE SMC, UbiComp and ECCV

Activity • Sitting• Standing• Walking

Human activity recognition using RGB-DAccuracy is above 90%

14

Activity Detection using Mobile Phone Inertial Sensors

Activity Detection– Uses Accelerometer Data– Gyroscope and Magnetometer for orientation

correction– Step Count, Stride Length Estimation– Walking, Brisk Walking, Running, Sitting, Falling

Classification

Peak Detection and Step Validation using IPA;Calculating Step cycle lengths for all valid steps in the window

Classification of window activity using step frequencies derived from step cycle lengths

Noise Cancellation and pre-processingCalorie Count from Step Count and Type of Activity

Papers in UbiComp

~90% Accuracy~80% Accuracy

15

Geo-fencing– Using Magentometer

Proximity Detection– Using Bluetooth RSSI

Inertial Navigation– Step Count + Stride Length (personalized

model)– Gyroscope and Magnetometer-corrected

Inertial Navigation Wi-Fi based Zoning and Triangulation

– Based on RSSI of known location of 3 or more access points

– Attenuation modeling of the building– Unsupervised Learning through physical

modeling Fusion, Tracking and Correction

– Kalman Filter based Tracking– Particle Filter based Correction

Mobile Phone based Indoor Localization – Inertial and Wi-Fi

• Colleague Finder in Large Offices

• Shopper Localization in Retail Stores

• Emergency Evacuation in Large Buildings

Papers in Mobiquitous, UbiCompSelected for Indoor Localization

Competition in IPSN 2014

16

Cognitive Load on Human Brain

Cognitive Load 23+45=? 1846890129 + 2374609823=?

Use Cases Personalized education User interface design

EEG GSR

Papers in IEEE SMC, IEEE Fuzz, ACM BIBE

17

Emotion and Engagement level using Camera

Why Camera− Unobtrusive Sensing @Work

Purpose− Identifying the mood a person at work− How a person is engaged at work

Scope− Facial emotion analysis on unconstrained environment using

camera on desktop/laptop − Mood and human engagement at work place using desktop camera

Partial occlusion for facial expression recognition Engagement analysis Micro emotions / expressions

http://www.ecse.rpi.edu/~cvrl/tongy/aurecognition.html

Irfan Essa (1994), “Analysis, interpretation and synthesis of facial expressions“, PhD Thesis, MIT, Cambridge, MA, USA. (Advisor: Alex (Sandy) Pentland)

18

Program: Affordable Healthcare and Wellness

19

Affordable Healthcare using Mobile Phones

Sensing various physiological parameters using smartphone sensors with minimal attachments

Low cost solution for initial screening in preventive healthcare / wellness

Solutions need to be stable, repeatable and robust

New disease diagnostics and treatment protocols

– Requires long observation over large set of patients

Geriatric care, monitoring chronic patients Go into Wearable in future Heart condition

– Heart rate, Heart rate variability, Blood Pressure, ECG

– Foetal Heart rate Lung condition

– Spirometry, Respiratory Rate Pupil condition

– Pupillary dilation response Pulse Diagnostics

Purpose

Scope

Robust Solutions using camera, microphone and accelerometer

20

PoC Approach and Expected Novelty

HR, HRV, RRPPG extractionRealtimeVideo

Audio from Mic

Accelerometer

Cardiovascular Model

SpO2

BP, ECG

Spirometry

Breathing Rate

Image of eyePupillary Reflex

21

Photo-plethysmography (PPG) using Mobile Phone Camera

Subject1 Subject2 Subject3

Actual Detected Actual Detected Actual Detected

68 66 66 63 85 84

2.9% 4.5% 1.1%

Papers at Mobihoc, IEEE BIBE, SenSys, ICASSP

Data set Pd Ps PP-diff < 15

Standard dataset (14 features) 92.9% 74.7% 77.9%

TCS dataset - add height, weight, age

99.3% 82.7% 85.5%

22

Program: Mobile Interactive Remote Sensing

23

Sensing the Physical World

Mobile phone based crowd sensing

Robot assisted sensing

www.popularmechanics.com  

www.engadget.com  

www.allthingssd.com  

apollo2.cs.illinois.edu  

Camera based sensing

24

Intelligent Transportation – Vehicle Model Driven Sensor Data Analysis

KNOWN PARAMETERS EFFECTS TARGET INDUSTRYVehicle Type & Driving Behavior Road Condition Monitoring City MunicipalityRoad Condition & Driving Behavior Car Prognosis Automotive

Road Condition & Vehicle Type Driving Behavior Analysis Insurance

Acceleration a(t) = f (H(t), v(t), R(t), D(t))

H(t)

Papers in ICST, Percom

25

Phone Microphone based Sound Scaping

Solution OverviewEvent driven with participatory sensing aided audio surveillance system • Classification of Traffic Noise (Honk Detection) and Crowd Noise

Papers in CODIS, ISSNIP, ISDA

26

Phone Camera based 3D Reconstruction from 2D images

Input Images

Dense Reconstruction without using mobile inertial sensors

- 20 images, compute time (4 core, 1GPU) ~ 20 min

- 120 images, compute time (16 core, 1GPU) ~ 30 min

- Bandwidth saving ~ 8 times, if done on mobile

Sparse Reconstruction using Mobile Inertial Sensors for Camera Position Estimation

• 20 images, compute time (4 core, 1GPU) ~ 3 min (without using inertial sensors)

• 20 images – compute time (4 core, 1GPU) ~10 sec. (with inertial sensors)

• Bandwidth saving ~ 200 times, if done on mobile

• Sparse good enough for many applications• Mobile Sensing and ACCV (submitted)• Dense Reconstruction with mobile

inertial sensors under progress with more number of images target < 1min

Dense Reconstruction -120 images

Dense Reconstruction - 20 images

• Low cost solution for 3D reconstruction from multiple 2D images captured from mobile device.

• Motion information from the inbuilt inertial sensors – for camera position estimation

• Applications in Agro-advisory service, Remote Diagnostics, Remote Healthcare

27

Multi-sensor Fusion for Robot-assisted Sensing

Application in remote sensing in hazard-prone areas• Robot carries 2D camera and heat / chemical sensors on a rotating arm• 3D reconstruction from the 2D vision• Estimation of Heat / gas leak / sound Source (direction and range) through passive

directional signal processing• Fusion of heat / gas / sound map on reconstructed 3D vision map

www.ese.wustl.edu 

Ongoing Work Possible reuse from 2D-3D reconstruction and sound

classification

Cloud point from 3D vision

Possible gas / heat

source (ROI)

Source direction

and intensity

28

Innovation @TCS

29

TCS - Pioneering IT Innovation in India

1970s

Offshore Model Alliances with Major IT

Players

University Alliances, Systems Engineering

1970s -1980s

New Labs, Bio SuiteDhruvam

Silicon Valley Ecosystem

Mastercraft, Revine,Quartz Program,

Tools Foundry, Rice Husk Ash

2000 - 2005

2007-2011

IPR Focus and Policy Talent Management

TCS COIN TM

Software Tools -Casepac Migration

Re-engineering

Industry & Services Practices Set Up

Foundation of TRDDC-India’s first Industrial R&D

Center in IT, S/W Engg

CTO/R&D structureInnovation Framework

More Domain Labs

2006

1981 1987

1991

1998

2012

4E ModelScaled

Invention, Co-Innovation

The journey continues…

30

Innovation@TCS - Innovation Labs

Bangalore, India1

TCS Innovation Labs - Bangalore

Chennai, India2

TCS Innovation Labs - ChennaiTCS Innovation Labs - RetailTCS Innovation Labs - Travel & HospitalityTCS Innovation Labs - InsuranceTCS Innovation Labs - Web 2.0TCS Innovation Labs - Telecom

Cincinnati, USA3

TCS Innovation Labs - Cincinnati

Delhi, India4

TCS Innovation Labs - Delhi

Hyderabad, India5

TCS Innovation Labs - HyderabadTCS Innovation Labs - CMC

Kolkata, India6

TCS Innovation Labs - Kolkata

Mumbai, India7

TCS Innovation Labs - MumbaiTCS Innovation Labs - Performance Engineering

Peterborough, UK8

TCS Innovation Labs - Peterborough

Pune, India9

TCS Innovation Labs - TRDDC - Process EngineeringTCS Innovation Labs - TRDDC - Software EngineeringTCS Innovation Labs - TRDDC - Systems ResearchTCS Innovation Labs - Engineering & Industrial Services

1 2

3

4

597

6

8

• Associates in R&D2000+

• Innovation Labs19• Papers Published in last two years1000+

• Patents filed in last two years350+

• Patents granted till date100+• TCS RSP funded Research Scholars150+• Internships in Labs in last two years200+

Thank You

IT ServicesBusiness SolutionsConsulting