October 4, 2016

Santa Clara Convention Center

Mission City Ballroom

Contextual Awareness Solutions

MEMS Sensor Solutions Software Team

2:50 Introduction Sankalp Dayal

3:30

Applications

Algorithms

Solutions

Integration

Agenda

Time

Speaker

Presentation

2

Sense

(Sensors)

Recognize Context

(Solutions)

Take Action

(Your Magic)

New Things to

augment life

Existing Things

augmented

3

Applications

• Diverse applications of context-aware systems

• Smart Me

• Smart Home

• Smart Office

• Industrial

• Healthcare

• Fitness

• Gaming

• Virtual/Augmented Reality

• Human Machine Interaction

• Security and Monitoring

• ……

4

Some Applications of context-Aware systems

• HealthCare:

• Monitoring posture and patterns to prevent conditions before they actually happen.

• Actively support elderly or disabled people in performing everyday activities

• System to detect potentially dangerous situations in a person’s life

• Industrial Applications

• Activity-aware applications can support workers in their tasks, help to avoid mistakes and increase

workplace safety

• Multimodal sensing using on-body and environmental sensors to determine context and user’s

current activity

• Entertainment and Games

• Provide feedback to athletes, artists in performing arts, etc, to augment their performance

• Entertainment based on context aware applications for mood recognition and then using playlist

recommendation

5

Building Algorithms to Recognize Context

• Real value is in recognition of high level contexts.

• Shopping in a mall, having lunch with friends, busy with office work

• Classification of low-level contexts form the basis to model and infer more

complex contexts.

• e.g. Shopping in mall requires location, human activity, human object interaction

• Low Level Contexts

• Human Activity

• Human Environment

• Human Location

• Human Interaction (with human or objects)

6

Generic Context Awareness Architecture 7

Understanding Human Activity

• Gestures/ Movements/ Motifs

• Brief and distinct body movements,

• e.g. taking a step, bending the arm

• (Low-Level) Activities

• Sequence of movements/ a distinct posture,

• e.g. walking, sitting, cleaning windows

• High-Level Activities/ Scenes/ Routines

• Collection of activities,

• e.g. office work, lunch, shopping at mall

8

Seconds

Minutes

Hours

Accelerometer

Barometer

Gyroscope

Windowing

and

Feature extraction

Windowing

and

Feature extraction

Compute

P(Activity/Features) in

Motion Activity Posteriorgram

Compute

P(Activity/Features) in

Motion Activity Posteriorgram

Compute

P(Activity/Features) in

Motion Activity Posteriorgram

Windowing

and

Feature extraction

MicrophoneWindowing

and

Feature extraction

Compute

P(Activity/Features) in

Motion Activity Posteriorgram

Activity

Inference

Human Motion Activity Detection Architecture

9

Understanding Human Environment 10

MicrophoneWindowing and

Feature extraction

Compute Prob.

(Environment/Features) in

Spatial

Environment Posteriorgram

Understanding High Level Context (1/2)

Motion activity vector (MAV)

= [stationary; walking; jogging; escalator; elevator; bicycling; driving;

none of these]’

Voice activity vector (VAV)

= [silence; face-to-face talking; phone conversation; none of these]’

Spatial environment vector (SEV)

= [street; nature: garden/park; beach; stadium; office;

mall/supermarket/restaurant/cafeteria; home; in moving vehicle;

lecture/conference room; none of these]’

11

Understanding High Level Context (2/2)

Time sequence of Motion Activity

Posteriorgram

(MAP)

Time sequence of Voice Activity

Posteriorgram

(VAP)

Time sequence of Spatial

Environment Posteriorgram

(SEP)

Embedded Application

producing Soft Decision of

High-Level Context

12

Building Good Solutions

• Accurate

• Large user testing, diverse scenarios

• Low Power

• Smart Sensor Usage, Hardware Accelerators, On Sensor Computation

• Low Footprint

• Minimalist models, optimized software

• Flexible

• Multi Platform, Tunable, Personalization

13

ST Sensor Solutions enabled by MEMS

Sensor fusion for device orientation (Any Device)

• Gives rotation, orientation, quaternions, linear acceleration, calibration

User Activity recognition (Mobile/Wrist)

• Travel mode, Pedestrian mode, workout, sports

Gesture recognition (Mobile/Wrist)

• Glance, pick-up, look-at, shake, tap, swipe, rotate, symbols, user defined

Carry position (device placement on body) determination (Mobile)

• Shirt pocket, holster, trouser pocket, backpack, handbag, near the head, ..

Pose Detection (Wrist)

• Detect Standing, Sitting, Lying Down

14

ST Sensor Solutions enabled by MEMS

Stairs Counting/Vertical Context (Any Device)

• Elevator, Escalator, Stairs Up/Down, Number of stairs, Floor Change

Bicep Curl Detection and Counting (Wrist)

• Detects bicep curls and counts reps

Situational awareness using Audio + Motion sensors

• Environment detection, number of speakers, key word spotting

Pedestrian Dead-Reckoning (PDR)

• Indoor location, Augmenting GPS based location

15

Sensor FusionosxMotionFX

• Magnetometer calibration routine

• Gyroscope bias compensation

• Dynamic distortion (hand jitter), measured by the accelerometer

• Power saving

osxMotionFX – GUI

16

Activity Recognition 17

osxMotionAR provides real-time information on user activity:

• Stationary

• Walking

• fast walking

The algorithm manages only the data acquired from the accelerometer at a

low sampling frequency of 16 Hz, contributing to reduction of power

consumption of the hosting platform.

The application recognizes the performed activities

and stores them in the on-board memory for offline analysis.

• Jogging

• biking

• driving

osxMotionAR

Activity Recognition Test Results 18

Stationary Walking Fast Walking Jogging Biking DrivingDetection

Probability

16279 1 0 0 98 1431 Stationary 91.41%

3 49030 51 9 483 25 Walking 98.85%

0 116 3143 6 10 3 Fast Walking 95.88%

0 14 11 2781 8 2 Jogging 98.76%

63 132 4 0 5292 633 Biking 86.41%

1113 6 1 0 436 7912 Driving 83.57%

Classified As

Actual

Activity

Test Data

• 682 data sets (71 unique individuals) 48 hours of activity data

• Activities included (stationary, walking, fast walking, jogging, vehicle, bicycle) for different carry positions (body placement)

• Pedestrian: Trouser pocket, in-hand, shirt pocket, in back pocket, near-the-head, ..

• Vehicle: in cup-holder, in-shirt pocket, in- trouser pocket, ..

• Bicycle: in-shirt pocket, in-trouser pocket

Gesture Recognition

osxMotionGR provides real-time information on Gestures:

• Glance

• Pick-up

• Wake-up

The algorithm manages only the data acquired from the

accelerometer. Designed for always-on operation. Machine learning algorithms

are used for the feature

The application recognizes the performed activities

and stores them in the on-board memory for offline analysis.

19

osxMotionGR

Carry Position Recognition

osxMotionCP provides real-time information on Carry Position:

• On-Desk

• In-hand (like texting)

• Near Head (talking on Phone)

The algorithm manages only the data acquired from the accelerometer.

Uses Machine Learning based models to achieve higher accuracy

The application recognizes the performed activities

and stores them in the on-board memory for offline analysis.

• Shirt Pocket

• Trouser Pocket

• Arm Swing

20

osxMotionCP

Pose Estimation

Estimates pose of human body

• Standing

• Sitting

• Lying Down

The algorithm manages only the data acquired from the Pressure

Sensors at 10 Hz and Accelerometer Data at 16 Hz

Precise pressure sensor allows to differentiate between standing and

sitting at desk

Uses Machine Learning models to achieve higher accuracy

Performance: maximum latency < 2 sec

21

Pose Estimation Library

Scenarios for Data CollectionHands in trouser pocket - standing

Head resting on hand - sitting

Hand resting on side of couch while watching TV

Eating with Fork and Knife

Driving

Cooking

Hands resting on thigh/pillow while watching TV

Holding phone in hand Reading or Texting - standing

Holding phone in hand Reading or Texting -sitting

Holding a cup of coffee/can of soda - standing

Holding a cup of coffee/can of soda - sitting

Talking on phone – standing

Talking on phone - sitting

Washing face

Talking with gestures - standing

Talking with gestures - sitting

Typing

Mouse Handling

Doing dishes

Brushing teeth

Shaving

Hands in jacket

Writing

Sitting vs Standing Confusion Matrix

Activity Duration Sitting Standing Probability of Detection

Sitting 7386 97.49 2.51 97.49

Standing 3843 11.04 88.96 88.96

Total Duration 11229

22

Vertical Context Detection 23

Detects:

• Elevator

• Escalator

• Stairs Up/Down (uses step detection function)

• On-floor

The algorithm manages only the data acquired from the Pressure

Sensors at 10 Hz.

Corrects drift to achieve higher accuracy Able to prevent drift < 50 cm.

Performance: maximum latency < 2 sec

Vertical Context Library

Bicep Curl Detection

• Detects Bicep Curls

• Counts number of repetitions

The algorithm manages only the data acquired from the

Pressure Sensor at 25 Hz and Accelerometer at 25 Hz

Corrects drift to achieve higher accuracy

Precise pressure sensor allows to differentiate

between random motion and actual bicep curl motion

Uses Machine Learning models to achieve higher accuracy

Acc X

Acc Y

Acc Zθ angle

Vertical

direction

24

Bicep Curl Detection Library

Start Building Your Own SolutionIn less than 5 minutes

25

Steps to Build Your Own Solution 26

Choose

Preferred

Hardware

Download

Firmware

Get Library

License

Install App

Activity, Carry, Gesture,

Pedometer, Vertical Context,

Pose, …

Software Developer’s

Platform

$10 $86

Free

Free

Free

• Gerber files

• BOM

• Schematics

• Source Code

• Application Note

• User Guide

• Getting Started

• API

• Library

• Free License

• Data Logging & Viewing

• Complete Project

• Source Code

STM32 Open Development EnvironmentSTM32 Nucleo

27

One STM32 MCU flavor with 64 pins

Arduino™ extension connectors

Easy access for add-ons

Morpho extension headers

direct access to all MCU I/Os

2 push buttons, 2 color LEDs

Flexible board power supply

Through USB or external source

Available for STM32F401, STM32L152, STM32F030 and STM32F103. Other STM32s coming soon

www.st.com/stm32

Integrated ST-Link/V2-1

Mass storage device flash programming

Features

• Complete solution with STM32 Nucleo board includes:

• LSM6DS0: MEMS 3D accelerometer + 3D gyroscope

• LIS3MDL: MEMS 3D magnetometer

• LPS25HB: MEMS pressure sensor, (absolute digital output barometer)

• HTS221: capacitive digital relative humidity and temperature

• DIL 24-pin socket available for additional MEMS adapters and other

sensors (UV index)

Product Specification Gerber Files

Bill of Material

Schematics

SW Packages download

HW description of the X-Nucleo

Getting Started HW User Manual of the X-Nucleo

Quick User Guide User Manual for SW packages

Motion & Environmental SensorsSTM32 Nucleo Expansion Board

28

X-NUCLEO-IKS01A1

SensorTile Core SystemSensing, processing and BLE connectivity

Sensors

Ultra Low Power

Connectivity

Low-Power MCU

MP34DT04

LPS22HB

LSM6DSM

LSM303AGR

STM32L4

BlueNRG-MS

• SD Card Slot for Extra Memory

• Wearable Form Factor

• Easy Programmability

13.5 mm

13

.5 m

m

Miniaturized Tile that can be

soldered or plugged on a host board

Motion MEMS

Environmental sensors

MEMS microphone

Low-power brain

Sensor fusion

Bluetooth Smart

29

Free SW Licensing Program 30

Unleashing the power of embedded software

with algorithms and complete demonstrators

for the Internet of Things.

Bring your ideas to now!

31

32

OpenSoftwareThe Middleware Libraries

33

Open Software – Middleware Libraries

• Open.MEMS: sensor processing libraries

• Open.RF: protocols and profiles for the wireless connectivity

• Open.Audio: libraries and processing algorithms for audio capturing systems based on digital MEMS microphones

Includes several free libraries:

• Open.MEMS

• Open.RF

• Open.Audio

Open.Framework

(Implementation Examples):

• BlueMicrosystem

• BlueVoiceLink

Mobile and PC Apps 34

.Thank You!