![Page 1: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/1.jpg)
Behavior analysis based on coordinates of body tags
Mitja Luštrek,Boštjan Kaluža, Erik Dovgan,
Bogdan Pogorelc, Matjaž Gams
Jožef Stefan Institute,Department of Intelligent Systems
Špica International, d. o. o.
Slovenia
![Page 2: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/2.jpg)
Introduction
• Problem:– Number of elderly increasing– Too few young people to care for them
![Page 3: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/3.jpg)
Introduction
• Problem:– Number of elderly increasing– Too few young people to care for them
• Solution:– Ambient assisted living technology– Confidence project:
detect falls, monitor well-being
![Page 4: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/4.jpg)
Work presented
• Input: coordinates of tags attached to body
![Page 5: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/5.jpg)
Work presented
• Input: coordinates of tags attached to body
• Task 1: activity recognition (falls)• Task 2: recognition of abnormal walking
![Page 6: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/6.jpg)
Work presented
• Input: coordinates of tags attached to body
• Task 1: activity recognition (falls)• Task 2: recognition of abnormal walking
Machine
learning
![Page 7: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/7.jpg)
Work presented
• Input: coordinates of tags attached to body
• Task 1: activity recognition (falls)• Task 2: recognition of abnormal walking
• Analyze how recogntion is affected by:– Number of tags– Quality of tag coordinate measurements
Machine
learning
![Page 8: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/8.jpg)
Presentation overview
• Sensing hardware• Task 1: activity recognition (falls)• Task 2: recognition of abnormal walking
![Page 9: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/9.jpg)
Sensing hardware
Infraredmotioncapture
Volunteer wearing 12 tagsperforming an activity
![Page 10: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/10.jpg)
Sensing hardware
(x, y, z)for all tags
at 10 Hz
Infraredmotioncapture
Volunteer wearing 12 tagsperforming an activity
![Page 11: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/11.jpg)
Sensing hardware
(x, y, z)for all tags
at 10 Hz
Infraredmotioncapture
Volunteer wearing 12 tagsperforming an activity
Add noise to simulate realistic
hardware
![Page 12: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/12.jpg)
Task 1: activity recognition
![Page 13: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/13.jpg)
FeaturesReference coordinate system
• z coordinates• absolute velocities, z velocities• absolute distances between tags,
z distances
![Page 14: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/14.jpg)
FeaturesReference coordinate system
Body coordinate system
• z coordinates• absolute velocities, z velocities• absolute distances between tags,
z distances
• x, y, z coordinates• absolute velocities,
x, y, z velocities
![Page 15: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/15.jpg)
Features
• z coordinates• absolute velocities, z velocities• absolute distances between tags,
z distances
• x, y, z coordinates• absolute velocities,
x, y, z velocities
Reference coordinate system
Body coordinate system
Angles
![Page 16: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/16.jpg)
Feature vectorst
Snapshot
Snapshot
![Page 17: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/17.jpg)
Feature vectorst-1 tt-2t-9 ...
Activity
Feature vector
![Page 18: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/18.jpg)
Feature vectorst-1 tt-2t-9 ...
t+1
t+2
t+3
...
![Page 19: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/19.jpg)
Feature vectorst-1 tt-2t-9 ...
t+1
t+2
t+3
...
Machine learning with SVM
![Page 20: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/20.jpg)
Experimental setup
• 6 activities:– Walking– Sitting down– Sitting
– Lying down– Lying– Falling
![Page 21: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/21.jpg)
Experimental setup
• 6 activities:– Walking– Sitting down– Sitting
– Lying down– Lying– Falling
• Number of tags: 1 to 12• Noise level: none to Ubisense × 2
![Page 22: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/22.jpg)
Recognition accuracy
![Page 23: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/23.jpg)
Fall detection
• Simple rule:if 3 × recognized fallingfollowed by 1 × recognized lyingthen fall
![Page 24: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/24.jpg)
Fall detection
• Simple rule:if 3 × recognized fallingfollowed by 1 × recognized lyingthen fall
• Fall detection accuracy:– Mostly independent of noise– 93–95 %
![Page 25: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/25.jpg)
Summary of activity recognition
• SVM to train a classifier for activity recognition
• Accuracy: 91 % with Ubisense noise and 4–8 tags
![Page 26: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/26.jpg)
Summary of activity recognition
• SVM to train a classifier for activity recognition
• Accuracy: 91 % with Ubisense noise and 4–8 tags
• Simple rule for fall detection• Accuracy: 93–95 %
![Page 27: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/27.jpg)
Task 2: recognition of abnormal walking
![Page 28: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/28.jpg)
Feature vectors
• 1 feature vector = 1 left + 1 right step
![Page 29: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/29.jpg)
Feature vectors
• 1 feature vector = 1 left + 1 right step
• Features from medical literature on gait analysis
![Page 30: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/30.jpg)
Features
Double support time
![Page 31: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/31.jpg)
Features
Swing time
![Page 32: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/32.jpg)
Features
Support time
![Page 33: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/33.jpg)
Features
Distance of the footfrom the ground
![Page 34: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/34.jpg)
Features
Ankle angle
Knee angle
Hip angle
![Page 35: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/35.jpg)
Features
Ankle angle
Knee angle
Hip angleAnd others...
![Page 36: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/36.jpg)
Machine learning = outlier detectionLocal outlier factor algorithm
![Page 37: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/37.jpg)
Machine learning = outlier detectionLocal outlier factor algorithm
Normal walking
Abormalwalking
![Page 38: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/38.jpg)
Experimental setup
• Normal walking• Abnormal walking:– Limping– Parkinson’s disease– Hemiplegia
![Page 39: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/39.jpg)
Experimental setup
• Normal walking• Abnormal walking:– Limping– Parkinson’s disease– Hemiplegia
• Number of tags: 2, 4, 6, 8• Noise level: none to Ubisense × 2
![Page 40: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/40.jpg)
Recognition accuracy
![Page 41: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/41.jpg)
Summary of recognition of abnormal walking
• Medically relevant features• Outlier detection to recognize
abnormal walking• Accuracy:
92 % with Ubisense noise and 6 tags
![Page 42: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/42.jpg)
Conclusion
• Tag localization + machine learningsuitable for ambient assised living
• Results comparable to competitive approaches (inertial sensors)
![Page 43: Behavior analysis based on coordinates of body tags Mitja Luštrek, Boštjan Kaluža, Erik Dovgan, Bogdan Pogorelc, Matjaž Gams Jožef Stefan Institute, Department](https://reader030.vdocuments.mx/reader030/viewer/2022032804/56649e445503460f94b383b6/html5/thumbnails/43.jpg)
Conclusion
• Tag localization + machine learningsuitable for ambient assised living
• Results comparable to competitive approaches (inertial sensors)
• Future work:– Test with realistic hardware– Analyze other activities (besides walking)