mobile sensing talk - kleomenis katevas · magnetometer no drift problem (depend on magnetic north)...

Introduction to Mobile Sensing TechnologyKleomenis Katevas

[email protected]://minoskt.github.io

Cognitive Science Research Grouphttp://cogsci.eecs.qmul.ac.uk

mailto:[email protected]?subject=

https://minoskt.github.io


Image by CRCA / CNRS / University of Toulouse


• What is Mobile Sensing?

• Sensor data, but what can we actually sense?

• Sensor Fusion.

• Mobile Sensing on Android and iOS.

• Continuous Sensing Platforms.

• Mobile Sensing in Research.

In this talk

Camera(s)

Proximity Sensor

Ambient Light Sensor

Microphone(s)

GPS

Accelerometer

Gyroscope

Bluetooth Magnetometer

WiFi

NFC

Barometer

Water Sensor

Motion Coprocessor


Satellite Navigation Systems• It calculates the location (latitude and longitude), altitude, and speed

of the device based on the distance from at least four satellites (trilateration).

• Average accuracy is ~3m.

• Not suitable for indoor positioning.

• GPS is the most popular one, but others are available or coming in the future (GLONASS, COMPASS, Galileo, IRNSS).


• Wireless networking technology.

• Can been used as an indoor positioning system (IPS) by triangulating the Received Signal Strength Indication (RSSI) of other Wi-Fi hot-spots.

• Some mobile devices combine GPS, cell tower triangulation and WiFi-based location to improve accuracy.

• Read “Survey of Wireless Indoor Positioning Techniques and Systems” by Hui Liu et al.


• Wireless networking technology for exchanging data over short distances. (~10 meters for Class 2 radios)

• Can also be used for indoor positioning.

• Very useful for spatial-temporal data gathering (participant mobility and social interactions).

• Bluetooth LE (BLE), marketed as Bluetooth Smart, has very low battery consumption.


iBeacon™• Apple’s implementation of Bluetooth

Smart, used as an indoor positioning system.

• Classifies the location of the device as Immediate, Near or Far.

• All devices can be iBeacon transmitters, receivers (or both), but the technology is also open to third-party hardware.

• Also available in Android using third-party libraries.


iBeacon™ Data Format

Company ID (Apple: 0x004C)

Type(always 0x02)

UUID (16 byte hex)

Major(uint 0-36535)

Minor(uint 0-36535)

Measured Power(rssi in 1m distance)

Estimote Indoor Location


Eddystone™• Google’s answer to iBeacon.

• Fully open-source specification (Apache v2.0) with source code available for Android and iOS.

• Three frames:

‣ Eddystone-UID (similar to iBeacon™)

‣ Eddystone-URL (broadcasts a URL)

‣ Eddystone-TLM (broadcasts telemetry information)


Six degrees of freedom (3D Translation & 3D Rotation)

Some terminology


Magnetometer

• Magnetic field sensor.

• Tells you the actual orientation of the device relative to the magnetic north (not the true north).

• Three axis magnetometer:x, y, z (East, North, Up)


Accelerometer• A sensor that measures the force of

acceleration of the device.

• 3 axis accelerometer:x, y, z (Surge, Heave, Sway)

• Can also calculate Pitch and Roll (but not Yaw!) using trigonometric calculations.

• Also used to understand the device orientation, by measuring the acceleration caused by the gravity.


Gyroscope

• 3 axis gyroscope:x, y, z (Pitch, Roll, Yaw)

• Tells you how much your device is being rotated over time.

• Less computationally expensive Pitch and Roll.

• Also provides Yaw.


Demo time…


Comparing the three sensors+ -

Magnetometer No drift problem(depend on magnetic north)

Noisy. Hard to calibrate.Not accurate for rotation detection.

Tilt compensation.

Accelerometer No drift problem(depend on earths gravity)

Noisy, especially in high frequencies.Mixes linear acceleration and gravity.

GyroscopeSmooth reading even in high

frequencies.Great dynamic response.

Large errors due to the drift problem.


What is sensor fusion?

• “The use of multiple sensors so that they compensate each other’s weaknesses.”

• Users (developers) do not want to know about sensors. What they really want is accurate and responsive 3D translation, 3D rotation and orientation.


Accelerometer and Gyro Fusion

(Sachs, 2010)


Accelerometer and Gyro Fusion

(Sachs, 2010)Integration


Sensor Fusion (Sachs, 2010)

Accelerometer

Gyroscope

Gravity

Orientation 3D Rotation



Accelerometer

Gyroscope

Gravity


Integration


Magnetometer and Gyro Fusion

(Sachs, 2010)



Accelerometer

Gyroscope

Gravity


Magnetometer Direction


Gravity and Linear Acceleration

• Linear Acceleration = Accelerometer Data - Gravity

• In most cases, it is very hard to separate Gravity from Linear Acceleration as gravity changes really fast.

• High pass filters do not work very well.



Accelerometer

Gyroscope

Gravity



Linear Acceleration 3D Translation



Accelerometer

Gyroscope

Gravity



Linear Acceleration 3D Translation

Double Integration


Motion Coprocessors• Apple’s M7 / M8 / M9 chip.

• Continues measures motion data from Magnetometer, Accelerometer, Gyroscope and Barometer sensors.

• Automatically performs activity detection: stationary, walking, running, automotive, unknown. M8 also provides distance and elevation using the Barometer sensor. M9 can also track Pace and Cadence.

• Apps can query data of the past 7 days when loaded.

• Does not provide the raw data.


Sensing Motion in iOS and Android

• Raw and Calibrated data of:

- Accelerometer

- Gyroscope

- Magnetometer

• Fused data of:

- User Acceleration

- Gravity

- Attitude

- Rotation rate

- Magnetic field


Mobile Sensing on Android• Sampling Rate in motion sensors is not consistent.

• Timestamp is not consistent.

• Lack of a motion coprocessor, only Pedometers.(this will change very soon…)

• Android fragmentation problem.

• Bluetooth Smart (BLE) is only fully supported in Lollipop.


Sampling rate on Android

Samsung Galaxy S2 Google Nexus 4

FASTEST 100 Hz 200 Hz

GAME 50 Hz 50 Hz

UI 15 Hz 30 Hz

NORMAL 5 Hz 15 Hz


Mobile Sensing on iOS

• Running a process in the background is (almost) not possible.

• Access to some sensors is limited (e.g. WiFi, NFC).

• Restrictions in the App Store.


SensingKit Server

Web

Ser

vice

s

Com

mun

icat

ion

Mod

ule

Plug-inSystemModel

Manager

SensorModules Data

Data DataExtraction

Internet

SensingKit: A Multi-Platform Mobile Sensing Framework

• Works in Android and iOS mobile systems.

• Captures Motion, Location, Proximity, Environmental data.

• Power efficient using Bluetooth Smart (4.0).

• Easily extensible using a modular design.

• Automated time sync and data processing on the server.

• Available in open-source under the GNU LGPL v3.0.

Platform Characteristics

For more info, check www.sensingkit.orgCognitive Science Research Grouphttp://cogsci.eecs.qmul.ac.uk

http://www.sensingkit.org

SensingKit: Battery Consumption

Battery consumption of SensingKit running on an iPhone 5S.


00 03 06 09 12 15 18 21 24 27 30 33 36 39 42 45 48 51 54Time (hours)

0

10

20

30

40

50

60

70

80

90

100

Pow

er

(%)

iPhone 5S

IdleAccelerometer (100Hz)Gyroscope (100Hz)Magnetometer (100Hz)Device Motion (100Hz)Location (Best Accuracy)iBeacon Broadcast (1Hz)iBeacon Scan (1Hz)iBeacon Scan + Broadcast (1Hz)Microphone (44100.0 Hz)


Detecting Group Formations


141

142

143

144

145

146

147

148

149

150

time (sec)

P1

P2

P3

P4

P5

P6

P1

P2

P3

P4

P5

P6

P1

P2

P3

P4

P5

P6

P1

P2

P3

P4

P5

P6

P1 P2 P3 P4 P5 P6

P1 5.72 21.78 29.07 39.95 30.60

P2 5.72 15.45 29.06 19.67 25.41

P3 21.78 15.45 4.60 3.94 1.22

P4 29.07 29.06 4.60 10.49 2.43

P5 39.95 19.67 3.94 10.49 3.04

P6 30.60 25.41 1.22 2.43 3.04

P1 P2 P3 P4 P5 P6

P1 0 0 0 0 0

P2 0 0 0 0 0

P3 0 0 1 1 1

P4 0 0 1 0 1

P5 0 0 1 0 1

P6 0 0 1 1 1

(a)Weighed Adjacency Matrices

(b)Binary Adjacency Matrix

(c)Graph

}


Results

Confusion Matrix

Thank you!


David Sachs, “Sensor Fusion on Android Devices: A Revolution in Motion Processing”, Google Tech Talk, February 2010. Retrieved from https://www.youtube.com/watch?v=C7JQ7Rpwn2k.

Kleomenis Katevas, Hamed Haddadi, Laurissa Tokarchuk, Richard G. Clegg, "Detecting Group Formations using iBeacon Technology", 4th International Workshop on Human Activity Sensing Corpus and Application (HASCA2016) in conjunction with UbiComp 2016, September 2016, Heidelberg, Germany.

Kleomenis Katevas, Hamed Haddadi and Laurissa Tokarchuk, “SensingKit: Evaluating the Sensor Power Consumption in iOS devices”, 12th International Conference on Intelligent Environments (IE'16), September 2016, London, UK.

Kleomenis Katevas, Hamed Haddadi and Laurissa Tokarchuk, “Poster : SensingKit -- A Multi-Platform Mobile Sensing Framework for Large-Scale Experiments”, Extended abstract, ACM MobiCom 2014, Maui, Hawaii, September 2014.

https://www.youtube.com/watch?v=C7JQ7Rpwn2k

mobile sensing talk - kleomenis katevas · magnetometer no drift problem (depend on magnetic north)...

Documents