PIECEWISE LINEAR DYNAMICAL MODEL FOR HUMAN ACTIONS CLUSTERING

FROM INERTIAL BODY SENSORS WITH CONSIDERATIONS OF HUMAN FACTORS

Jiaqi Gong1, Philip Asare1,2, John Lach1, Yanjun Qi2 1Charles L. Brown Department of Electrical and Computer Engineering

2Department of Computer Science University of Virginia

{jgong, pka6qz, jlach, yq2h}@virginia.edu

ICST BodyNets, September 29th, 2014

Motivation 2

¨  Dividing motion data into discrete segments is useful ¤  Training

¤  Diagnosis

¨  Segmentation can be at different granularities ¤  Walking vs. not-walking

¤  Where in the gait cycle

1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

Walking Jumping Running Punching

Left  Wrist

Left  Ankle

Right  Wrist

Right  Ankle

Problems 3

¨  Typical BSN Processing Pipeline

S1

S2

Sk

… … … … …

Raw  Data𝐘 𝒕 = (𝒚𝟏, 𝒚𝟐 …𝒚𝒌)

…

𝒚𝟏(𝒕) ∈ 𝑹𝒑

…

𝒚𝟐(𝒕) ∈ 𝑹𝒑

…

𝒚𝒌(𝒕) ∈ 𝑹𝒑

t Samples

Preprocessing𝒀′(𝒕) = (𝒚′𝟏, 𝒚′𝟐 …𝒚′𝒌)

t Samples

…

𝒚′𝟏(𝒕) ∈ 𝑹𝒑

…

𝒚′𝟐(𝒕) ∈ 𝑹𝒑

…

𝒚′𝒌(𝒕) ∈ 𝑹𝒑

SegmentationW= {𝒘𝟏,𝒘𝟐 …𝒘𝒎}

…

…

…𝒘𝟏 𝒘𝒎…

m  Segments

Feature  Extraction𝑿𝒊∈[𝟏,𝒎] = {𝒙𝒊𝟏, 𝒙𝒊𝟐, … 𝒙𝒊

𝒇}

𝒙𝟏𝟏

𝒙𝟏𝒇

𝒙𝟐𝟏

𝒙𝟐𝒇

𝒙𝒎𝟏

𝒙𝒎𝒇

Classification  𝑷𝒊(𝒔|𝑿𝒊, 𝜽)

𝒑𝟏𝟏(𝒔𝟏|𝑿𝟏)

𝒑𝟏𝟏(𝒔𝟐|𝑿𝟏)

𝒑𝟏𝟏(𝒔𝒄|𝑿𝟏)

𝒔𝟏

𝒔𝒎

Class

m  Segments C  Classes

Assumes repeatability of signals

Credit: A. Bulling, U. Blanke, and B. Schiele, “A Tutorial on Human Activity Recognition Using Body-Worn Inertial Sensors”, ACM Computing Surveys (CSUR), vol. 46, no. 3, pp. 33, 2014.

Problems 4

¨  ‘Human Factors’

100 200 300 400 500 600 700 800 900 1000-150

-100

-50

0

50

100

o *

100 200 300 400 500 600 700 800 900 1000-300

-200

-100

0

100

200

300

400

o *

100 200 300 400 500 600 700 800 900 1000-1000

-500

0

500

1000

100 200 300 400 500 600 700 800 900 1000-500

0

500

1000

Gyroscope  data  of  inertial  BSNs  mounted  insecurely

Gyroscope  data  of  inertial  BSNs  mounted  incorrectly

Gyroscope  data  of  inertial  BSNs  mounted  correctly

Heel  Strike  Event

Chien et al. (2013)*: Model-based estimation

*C. C. Chien, J.Y. Xu, H-I Chang, X. Wu and G. J. Pottie, “Model Construction for Human Motion Classification using Inertial Sensors”, IEEE Workshop on Information Theory and Applications, San Diego, Feb 2013

Physical placement affects signal

Addressing Problems 5

¨  Our Approach

S1

S2

Sk

… … … … …

Raw  Data𝐘 𝒕 = (𝒚𝟏, 𝒚𝟐 …𝒚𝒌)

…

𝒚𝟏(𝒕) ∈ 𝑹𝒑

…

𝒚𝟐(𝒕) ∈ 𝑹𝒑

…

𝒚𝒌(𝒕) ∈ 𝑹𝒑

t Samples

Preprocessing𝒀′(𝒕) = (𝒚′𝟏, 𝒚′𝟐 …𝒚′𝒌)

t Samples

…

𝒚′𝟏(𝒕) ∈ 𝑹𝒑

…

𝒚′𝟐(𝒕) ∈ 𝑹𝒑

…

𝒚′𝒌(𝒕) ∈ 𝑹𝒑

SegmentationW= {𝒘𝟏,𝒘𝟐 …𝒘𝒎}

…

…

…𝒘𝟏 𝒘𝒎…

m  Segments

Feature  Extraction𝑿𝒊∈[𝟏,𝒎] = {𝒙𝒊𝟏, 𝒙𝒊𝟐, … 𝒙𝒊

𝒇}

𝒙𝟏𝟏

𝒙𝟏𝒇

𝒙𝟐𝟏

𝒙𝟐𝒇

𝒙𝒎𝟏

𝒙𝒎𝒇

Classification  𝑷𝒊(𝒔|𝑿𝒊, 𝜽)

𝒑𝟏𝟏(𝒔𝟏|𝑿𝟏)

𝒑𝟏𝟏(𝒔𝟐|𝑿𝟏)

𝒑𝟏𝟏(𝒔𝒄|𝑿𝟏)

𝒔𝟏

𝒔𝒎

Class

m  Segments C  Classes

Body  Motion  M

odeling

X(t) X(t+1)X(t-­‐1)

S(t) S(t+1)S(t-­‐1)

Y(t) Y(t+1)Y(t-­‐1)

Linear  Dynamical  Transition

Nonlinear  Observation

HumanBody

InertialSensorsMotion

Stimulus𝑿𝒊

Sensor  Data𝐘 𝒕

Our Approach 6

¨  Basic Approach: Piecewise Linear Dynamical Modeling

S1

S2

Sk

… … … … …

Raw  Data𝐘 𝒕 = (𝒚𝟏, 𝒚𝟐 …𝒚𝒌)

…

𝒚𝟏(𝒕) ∈ 𝑹𝒑

…

𝒚𝟐(𝒕) ∈ 𝑹𝒑

…

𝒚𝒌(𝒕) ∈ 𝑹𝒑

t Samples

Preprocessing𝒀′(𝒕) = (𝒚′𝟏, 𝒚′𝟐 …𝒚′𝒌)

t Samples

…

𝒚′𝟏(𝒕) ∈ 𝑹𝒑

…

𝒚′𝟐(𝒕) ∈ 𝑹𝒑

…

𝒚′𝒌(𝒕) ∈ 𝑹𝒑

SegmentationW= {𝒘𝟏,𝒘𝟐 …𝒘𝒎}

…

…

…𝒘𝟏 𝒘𝒎…

m  Segments

Feature  Extraction𝑿𝒊∈[𝟏,𝒎] = {𝒙𝒊𝟏, 𝒙𝒊𝟐, … 𝒙𝒊

𝒇}

𝒙𝟏𝟏

𝒙𝟏𝒇

𝒙𝟐𝟏

𝒙𝟐𝒇

𝒙𝒎𝟏

𝒙𝒎𝒇

Classification  𝑷𝒊(𝒔|𝑿𝒊, 𝜽)

𝒑𝟏𝟏(𝒔𝟏|𝑿𝟏)

𝒑𝟏𝟏(𝒔𝟐|𝑿𝟏)

𝒑𝟏𝟏(𝒔𝒄|𝑿𝟏)

𝒔𝟏

𝒔𝒎

Class

m  Segments C  Classes

S(t+1)=AS(t)+Bx(t) y(t)=CS(t)+P

sensor data (observations)

human action (motion stimulus/input)

noise

dynamics/ properties

System generally non-linear, but approximately piecewise linear (over short time segments)

Our Approach 7

¨  Identifying Motion Stimulus

Left  Wave Heel  StrikeToe  offRight  Wave

(a) (b)

Left  Wave Heel  StrikeToe  offRight  Wave

(a) (b)

100 200 300 400 500 600 700 800 9000

10

20

30

40

50

100 200 300 400 500 600 700 800 9000

1

2

3

4x 104

-150-100

-500

50100

-200-150

-100-50

050

100150

200250-400

-300

-200

-100

0

100

200

300

-1.5-1

-0.50

0.51

-1.5

-1

-0.5

0

0.5

1-3

-2

-1

0

1

2

3

4

x

Point1Point2

y

z

Transformation  of  3D  curvature

Scalar  time-­‐series  curvature(a) (b)

(c)

Cur↓gyro =𝛻↓3 (𝛻↓1 (𝛻↓1 (y(t,  g(1)),  y(t,  g(2)),y(t,  g(3)))))

Our Approach 8

¨  Identifying Motion Stimulus

Left  Wave Heel  StrikeToe  offRight  Wave

(a) (b)

100 200 300 400 500 600 700 800 9000

10

20

30

40

50

100 200 300 400 500 600 700 800 9000

1

2

3

4x 104

100 200 300 400 500 600 700 800 900 10000

2000

4000

6000

8000

10000

12000

100 200 300 400 500 600 700 800 900 10000

2000

4000

6000

8000

10000

12000

Curvature  of  gyroscope  data  (insecure  mounting)

Curvature  of  gyroscope  data  (mounting  error)

Curvature  of  gyroscope  data  (correct  mounting)

Left  Wave Heel  StrikeToe  offRight  Wave

(a) (b)

Overall Algorithm 9

1000 2000 3000 4000 5000 6000 7000 8000 9000 100000

0.5

1

1.5

2x 105

1000 2000 3000 4000 5000 6000 7000 8000 9000 100000

5

10

15x 104

1000 2000 3000 4000 5000 6000 7000 8000 9000 100000

1

2

3x 106

1000 2000 3000 4000 5000 6000 7000 8000 9000 100000

2

4

6x 105

0.20.4

0.60.8

11.2

0.20.4

0.60.8

11.2

0

0.2

0.4

0.6

0.8

1

1000 2000 3000 4000 5000 6000 7000 8000 9000 10000-2000

-1000

0

1000

2000

1000 2000 3000 4000 5000 6000 7000 8000 9000 10000-2000

-1000

0

1000

2000

1000 2000 3000 4000 5000 6000 7000 8000 9000 10000-2000

-1000

0

1000

2000

1000 2000 3000 4000 5000 6000 7000 8000 9000 10000-2000

-1000

0

1000

2000

S1

S2

S3

S4

Raw  Data      𝒀 𝒕 = (𝒚𝟏(𝒕), 𝒚𝟐(𝒕)…𝒚𝟒(𝒕)) Curvature  Calculation   𝜵𝟏 𝜵𝟏 𝐲 𝐭, 𝐠 𝟏 , 𝐲 𝐭, 𝐠 𝟐 , 𝐲 𝐭, 𝐠 𝟑

EM  Algorithm

1) Initialize a random sparse input 𝜲   satisfying theconstraints: 𝐱 𝒕 ≤ 𝟏        ∀𝐭

2) Repeat:(a) Given 𝜲:

Identify a LDS: 𝐀, 𝑩, 𝐂, 𝑺𝒐;(b) Given 𝐀, 𝑩, 𝐂, 𝑺𝒐:

𝒎𝒊𝒏𝒊𝒎𝒊𝒛𝒆𝑿  (𝐂𝐨𝐬𝐭  𝐅𝐮𝐧𝐜𝐭𝐢𝐨𝐧)𝐱 𝒕 ≤ 𝟏        ∀𝐭

3) Calculate histogram of X into 11 bins after recursion.

20 40 60 80 100-500

0

500

20 40 60 80 100-500

0

500

20 40 60 80 100-2000

0

2000

20 40 60 80 100-2000

0

2000

Data  Piece𝒀(𝒕)𝒊  𝟎 < 𝐭 < 𝒕𝓵

PLDM  Features𝑿𝒊∈[𝟏,𝒎] = {𝒙𝒊𝟏, 𝒙𝒊𝟐, … 𝒙𝒊

𝒇}    

ClusteringPreprocessing

Hierarchical  Temporal  Selection  of  Data  Pieces  based  on  detected  moments  of  motion  stimulus  

Motion  Stimulus  Detection(Peaks  of  Calculated  Curvature)

What does this buy us? 10

¨  Coarse-grain segmentation

Start 1 minute 2 minute 3 minute 4 minute

Start 1 minute 2 minute 3 minute 4 minute

Start 1 minute 2 minute 3 minute 4 minute

Start 1 minute 2 minute 3 minute 4 minute

Start 1 minute 2 minute 3 minute 4 minute

Start 1 minute 2 minute 3 minute 4 minute

Start 1 minute 2 minute 3 minute 4 minute

Ground  Truth

CorrectMounting

ErrorMounting

InsecureMounting

CorrectMounting

ErrorMounting

InsecureMounting

PLDM

ACAWalk

StraightRun Jump Side  Walk Punching Body  Rotation

What does this buy us? 11

¨  Fine-grained segmentation ¤ Heel-strike and toe off detection

0

50

100

150

200

250

300

350

400

450

500

1 2

Method  in  [5] PLDM

No.

of

pairs

of

even

ts

[5] S. Chen, C. L. Cunningham, J. Lach, and B. C. Bennett, “Extracting Spatio-Temporal Information from Inertial Body Sensor Networks for Gait Speed Estimation”, IEEE International Conference of Body Sensor Networks (BSN), pp. 71-76, 2011.

Expected range

Humans walk at about two steps per second (one per leg) so we expect about 360 event pairs per leg with some variance

Recap 12

¨  Segmentation is important for BSNs ¨  Human factors can be a problem ¨  Linear dynamical systems modeling can help for

¤ Fine grained ¤ Coarse grained

Future Work 13

¨  Reduce Computation Complexity ¨  Optimize clustering process ¨  Other Applications

¤ Surgery education data analysis ¤ Head impact identification (in sports) ¤ …