Instrumentation &

Measurement Society

www.ieee-ims.org

Unobtrusive Smart Sensing and Pervasive

Computing for HealthCareCardio-Respiratory and Physical Therapy Assessment

IEEE SM Octavian PostolacheIEEE IMS Distinguished Lecturer

IEEE IMS TC-13 Chair, IEEE I&M Portugal Chapter Chair

Instituto Universitario de Lisboa, Portugal

Instituto de Telecomunicações, Lisboa, Portugal

IEEE IMS DL Seminar, IEEE IMS Romania Chapter Universitatea

Politehnica Bucharest, Romania, October 22, 2018

10/22/2018 7:51 AM 2

IEEE IMS

DL Program

10/22/2018 7:51 AM 3

IEEE IMS

DL Program

– Kristen Donnell

• I&M AdCom (2016-2019);

– Distinguished Lecturer Program Chair

- http://ieee-ims.org/education/distinguished-

lecturers-program

Now: Active 9 IEEE IMS DL

How to apply?Every year competition organized by

10/22/2018 7:51 AM 4

Outline

- Population ageing phenomena: Facts and Motivation

- Assistive Environments for Healthcare

- Daily used object with sensing and computation

capabilities for health status monitoring

- Smart sensing solutions for training and objective

evaluation of physical therapy

- Serious Games Physical Rehabilitation

- Thermography for Objective Evaluation

- Conclusions

10/22/2018 7:51 AM

Instrumentation & Measurement Society

www.ieee-ims.org 5

Ageing:

Facts

– Elderly people are the fastest

growing segment of the

population

– According with UN the

population over 60 years old will

increase worldwide from11%

now to 22% in 2050;

– The healthcare system is under

pressure

6

Physical Therapy

Stroke Facts

The number stroke patients

recovering in rehabilitation clinics

increase.

15 million people suffer stroke

worldwide each year (WHO).

5 million having to work for their

motor and cognitive recovery

Prevalence of stroke

elderly population (over 60 years

old).

Physical Therapy

Reality and Challenges

9

8

Physical Therapy Today evaluation

Subjective evaluation

based on:

o scales (e.g. pain

scales),

o manually data record

Objective evaluation

based on:

o basic mechanical

tools,

o no data record,

o no patient feedback

10/22/2018 7:51 AM

Instrumentation & Measurement Society

www.ieee-ims.org 9

Challenge

Develop new assistive

environments for healthcare

services:

- High QoS

- Efficiency

- Objective evaluation of

cardiac and motor

condition

Increase the usability and

the acceptance by Elderly

of new healthcare devices

Assistive Environments for Healthcare

Smart Sensing

smart sensors combination of sensor, signal conditioning,

embedded algorithms and communication interface

E. Song. K. Lee, et al, ICEMI 2011,

R. Morelo , O. Postolache , Guest Editor, Advanced, IEEE Sensors Journal, 2015

Assistive Environments for HealthcareUbiquitous Computing

Ubiquitous (pervasive)

computing embedding

microprocessors in everyday used

objects so they can communicate

information(Mark Weiser- - father of ubiquitous computing, PARC

Xerox company)

Mark Weiser, “The Computer for the 21st Century”, Scientific American. 1991

… Beyond ..

pervasive sensing & computing?

O.Postolache ,”Pervasive Sensing and M-Health”, Springer 2012

Sensing and Computing

Implemented

HealthC- Ecosystem

Assistive Environment for Healthcare

HealthC-Ecosystem

10/22/2018 7:51 AM 13

Smart WheelchairBCG non-intrusive cardiac exam

Cardio-respiratory monitor

based on EMFIT and MEMS

accelerometer

BCGyesterdayBCGtoday

Isaac Starr – 50’ cardiac

monitoring fashion

10/22/2018 7:51 AM

Instrumentation & Measurement Society

www.ieee-ims.org 14

Smart Wheelchair

BCG Sensing and Signals

• BCG captures pressure oscillations due to heart

activity

• Cardiac output through HR, HRV is assessedBCG wheelchair seat

BCG wheelchair backrest

10/22/2018 7:51 AM

Instrumentation & Measurement Society

www.ieee-ims.org 15

Smart Wheelchair

HR through BCG

IEEE EMBC 2015, Osaka, Japan

10/22/2018 7:51 AM

Instrumentation & Measurement Society

www.ieee-ims.org 16

Smart Wheelchair

BCG processing

IEEE EMBC 2013, Osaka, Japan

Cardiac output through

CWT Scallogram of

BCG energy

HR extraction based on

peak detection of a

selected scale and

threshold=0).

10/22/2018 7:51 AM

Instrumentation & Measurement Society

www.ieee-ims.org 17

Smart Wheelchair

ccECG

Capacitive Coupled Electrocardiogram (ccECG)

2 copper electrodes (33.75 cm2 ) at 20 cm distance , embedded within the wheelchair

backseat cover., 1 cooper plane of 550 cm2

IIEEE 2MTC 2013, Mineapolis, USA

Smart Wheelchair

ccECG processing

• Artifact removal and denoising

• DWT and SWT (Doubechies mw)

• Empirical Mode Decomposition (with PCA-IMD optimization)

Pinheiro, E.C.; Postolache, et al, Measurement, Vol. 45, No. 2, pp. 175 - 181, February, 201218

10/22/2018 7:51 AM

Instrumentation & Measurement Society

www.ieee-ims.org 19

Smart Wheelchair

rBCG

Ballistocardiography-seismography monitoring system

based on 24GHz FMCW Doppler radar

IEEE EMBC2011, Boston, USA

VCO

Vtune

coupler

M1

M2

Q-IF2

I-IF1

Transmit

antenna

Receive

antenna

10/22/2018 7:51 AM

Instrumentation & Measurement Society

www.ieee-ims.org 20

Smart WheelchairRadarSignal - CardioResp

IEEE EMBC2011, Boston, USA

FMCW Doppler radar output signals - balistocardiography (BCG)

signals include information about cardiac and respiration activity,

10/22/2018 7:51 AM

Instrumentation & Measurement Society

www.ieee-ims.org 21

Smart Wheelchair Respiration signals obtained from

EMFIT BCG and Radar BCG signals

MeMeA 2011, Bari, Italy

3rd order

DWT

decompo

sition

db4

mother

wavelet

respS=

approx.

10/22/2018 7:51 AM

Instrumentation & Measurement Society

www.ieee-ims.org 22

Smart Wheelchair PPG

Reflective photo - plethysmogram embedded in the

wheelchair arms Electrocardiogram (rPPG)

Robust measuring on arm/hand/wrist HR and SpO2 are obtained

I2MTC 2013

Physical Therapy

Today and Tomorrow Clinics

– Exclusively Usage of Mechanical Devices by the

physiotherapists in Regular Clinics

- No Feedback for user or physiotherapists

- No Data Records or/end Internet Connectivity

IoT

Smart Physiotherapy

Smart Walker Net

Instrumentation & Measurement Society

www.ieee-ims.org 25

O.Postolache et. al., IEEE, ICST 2011, NZ, O. Postolache et. Al., IEEE MeMeA 2015, Turin, Italy

O. Postolache IEEE EHB 2015, Iasi, Romania

standardW 2wheelsW 4wheelsW

Smart PhysiotherapySmart walkers of Walking Aids Net

10/22/2018 7:51 AM

Instrumentation & Measurement Society

www.ieee-ims.org 26O. Postolache et Al. IMEKO

TC-4, Conf. 2015

Smart Walker

standardW prototype

y

x

W43

F1F2

F3 F4

W12

L

d

D

COFx

COFy

COF: Center of Forces

» Risk index

(un)balance:

𝐼1 % = 100 ×𝑑

𝐷× 𝛼

: weighting factor

» motor

incoordination index

𝐼2(%) = 100 ×𝐵

𝑁,

where B is the

number of ‘incorrect

’ steps in N last

steps.

e-textile electrodes

Smart Walker

standardW prototype main components

28

0

1

2

3

4

5

0

10

......

T40

T20

T410

T210

step

Smart Walkerstate machine for step classification

0: Waiting for the walker to

lift off;

1: Walker is flying;

2: Waiting for the injured

foot to move forward;

3: Injured foot is moving

forward;

4: Waiting for the healthy

foot to move forward;

5: Healthy foot is moving

forward.

classified “correct” step if the user completes all states from 0 to 5

The incoordination index I2 is calculated.

29

Smart Walkerindexes and state machine evolution

classified “correct” step if the user completes all states from 0 to 5.

(the squared boxes numbered 0 to 5 indicate gait states)

30

Smart Walker

indexes during tests

10/22/2018 7:51 AM

Instrumentation & Measurement Society

www.ieee-ims.org 31

Force

sensor

Single

Doppler

Radar

3D

accelerometer

O. Postolache et Al.

ICST 2011, NZ

Smart Walker

2wheelsW prototype

VCO

Vtune

coupler

M1

M2

Q-IF

I-IF

Transmit

antenna

Receive

antenna

4

-4

-3

-2

-1

0

1

2

3

Time (s)

100 1 2 3 4 5 6 7 8 9

Instrumentation & Measurement Society

www.ieee-ims.org 32

IoT Healthcare Gait Pattern Radar based Remote Sensing

VCO

Vtune

coupler

M1

M2

Q-IF

I-IF

Transmit

antenna

Receive

antenna

FMCW IVS-162

IVS-948

Walking Aids Net Nodes

2wheelsW mobile software

19

34

Wearable Gait RehabilitationSmart Insole

3

-0,5

0

0,5

1

1,5

2

2,5

Time(s)

4020 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39

ATEE 2015, Bucharest, Romania

voltage signals normal gait from

metatarsal calcaneus area

ArduinoGLOVES

Microcontroller

Unity Exergames

Xamarin

iOS APP

Windows Universal APP

Android APP

AzurePortal

App Service Backend

App Database

WIoTPhyR- IT-IUL

IoT: Healthcare - WBAN-VR – Internet

connected

10/22/2018 7:51 AM 36

IoT for Smart Physiotherapy

Kinect Computation Architecture

Kinect Serious Games on the client side are GRANTED

IR speckle pattern Depth estimation

IoT for Smart PhysiotherapyKinect sensor

20 body joints SDK

NUI based Remote Sensing for physical

rehabilitation

Kinect Serious Games

Therasoup,

AppleHarvesting

JustPhysioKidding

Serious Games concept refers to

the use of computer games

without the main purpose of

providing pure entertainment,

Serious games based therapy

(TheraGames) is currently

gaining a lot of interest by the

healthcare community.

Unity 3D and C#

technologies were used to

develop the game

IoT for Smart PhysiotherapyTherasoup 2.0

web based: game configurator

game score and pain assessment

IoT for Smart Physiotherapy

Therasoup v2.0 web configuration and

results

IoT through Internet connectivity of the Kinect Client (miniPC)

O. Postolache et. al:MeMeA 2016

IoT for Smart PhysiotherapyMultimodal Sensing &Therasoup 2.0

42

Amplitudes

A-P antero posterior

M-L medio-lateral

0,0

-70,0

-60,0

-50,0

-40,0

-30,0

-20,0

-10,0

xCOP

20,0-30,0 -20,0 -10,0 0,0 10,0

-15,0

-50,0

-45,0

-40,0

-35,0

-30,0

-25,0

-20,0

xCOP

20,0-10,0 0,0 10,0

Left arm training Right arm training

IoT for Smart PhysiotherapyMultimodal Sensing &Therasoup 2.0

Computation architecture

Based on API Implemented VR game scenario

IoT for Smart PhysiotherapyApple Harvesting

IoT for Smart PhysiotherapyApple Harvesting

Kinect Serious Game for RehabilitationTailoring “Apple Harvesting”

Kinect Serious Game for RehabilitationTailoring “Apple Harvesting”

47

Kinect Serious Games

For Rehabilitation

Apple Harvesting APP- Xamarin

48

Cerebral Palsy Rehabilitation

JustPhysioKidding

Gaming Scenarios

IoT for Smart PhysiotherapyJust Physio Kidding

49

JustPhysioKidding

Results GUI

Plan and Motion

Reconstruction

IoT for Smart PhysiotherapyJustPhysioKidding – data analysis interface

50

IoT for Smart PhysiotherapyLeap Motion Serious Game Platform

51

- two cameras track infrared light

produced by three infrared

LEDs.

3D stereoscopic image

providesGesture and

position tracking with sub-

millimeter accuracy

IoT for Smart PhysiotherapyLeap Motion Serious Game Platform

• Hand and finger position,

velocities are provided

during the serious game

Leap Motion SDK

IoT for Smart Physiotherapy

Game: Collect Cube

• The user hands are selected according with: male, female, children

(robo) to increase the player motivation .

• The physiotherapist can chose the training plan for particular hand

• THEME: OBJECTS• CUBES

• BOXES

Animals

Sports

Fruits

Game: Collect Cube

• Extended Adaptation of the Game to the Cognitive Rehabilitation

Needs are considered

Game: Collect Cube

• Motor and cognitive rehabilitation together

Collect Cubes Serious Game

Hand Motion Assessment Results

30

-25

-20

-10

0

10

20

Time

220 2 4 6 8 10 12 14 16 18 20

30

0

5

10

15

20

25

Time (s)

200 2 4 6 8 10 12 14 16 18

30

0

5

10

15

20

25

Time (s)

200 2 4 6 8 10 12 14 16 18

30

5

10

15

20

25

Time (s)

200 2 4 6 8 10 12 14 16 18

The Z palm positions graph of left and right hand for two participants

Collect Cubes Serious Game

Hand Motion Assessment Results

30

-25

-20

-15

-10

-5

0

5

10

15

20

25

z

30-25 -20 -10 0 10 20

RH Z1

RH Z2

RH Z3

RH Z4

35

-5

0

5

10

15

20

25

30

z

35-5 0 5 10 15 20 25 30

LH Z3

LH Z4

LH Z1

LH Z2

Poincaré plot using data for Z positions of right and left

hand from four participants

user and training plan

records

Collect Cubes Serious Game

Game Configuration and Analysis by APP

Score and joint position

by date

Collect Cubes Serious Game

Game Configuration and Analysis by APP

Infrared Thermography for Smart Physiotherapy

Unobtrusive objective evaluation

Infrared Thermography: Every object whose surface

temperature is above absolute zero (-273 °C) radiates energy at a

wavelength (short wave 3-5um and long wave 7-9um) corresponding

to its surface temperature.

To obtain Thermographic Image highly sensitive (0.05ºC) infrared

camera (e.g. FLIR E60) capture thermal radiated energy with good

accuracy (error 2% of reading).

FLIR Tools+ and Thermonitor™ software applied to extract

specific temperatures (min, max, avg).

Thermographic images provides patient

cutaneous temperature in non-intrusive

mode the intensity of muscular

activity is revealed.

Temperature increase caused by

increased blood flow during the physical

therapy sessions.

Flir Tools+ is used to extract the

temperature values in different

cutaneous regions

Infrared Thermography for Physical Therapy

Unobtrusive sensing of muscular activity

How about real training versus virtual training using “Cubes”?

4 volunteers, 4 tests (0.5min, 1 min, 1.5 min, 2min

Thermographic images

VR versus Real Scenario Serious game

Training Effectiveness Tests & Results

Infrared Thermography for Physical Therapy Unobtrusive sensing of muscular activity

Average temperature evolution of

hand skin temperature before and

after 2 min training

Score Evolution for different

training durations (IDi

volunteer ID)

64

Conclusions

• Development of assistive environments based on non-

intrusive smart sensors and pervasive computing designed

for vital signs and physical therapy interventions supports:

• Preventive medicine;

• Personalized medicine;

• Participative medicine;

• Smart environments useful solutions for in-home and

remote rehabilitation services.

• NUI Serious Games and Thermography new challenge

regarding usability and acceptance

Thank you!