zhiphone: a mobile phone that learns context and user preferences jing michelle liu raphael hoffmann...

Zhiphone: A Mobile Phone that Learns Context and User Preferences

Jing Michelle Liu Raphael Hoffmann

CSE567 class project, AUT/05

CSE567 project 212/8/2005


Objective

Adapt the alarm type of a mobile phone to its context.


Objective

Adapt the alarm type of a mobile phone to its context and the preferences of its user.


This is what you want

Track context

Receive call

User reaction

Learn user preferences


Outline

Related Work Architectural Overview Feature Extraction Learning Context Learning User Preferences Future Work


Related Work

SenSay: A Context-aware mobile phone Four states: uninterruptible, idle, active and

normal Five functional modules: the sensor box,

sensor module, decision module, action module and phone module

Sensors include light, motion and microphone Query sensor data and the electronic calendar

of the user Use thresholds to classify into different states

There is no learning from user preferences


Related Work

SenSay: A Context-aware mobile phone Four states: uninterruptible, idle, active and

normal Five functional modules: the sensor box, sensor

module, decision module, action module and phone module

Sensors include light, motion and microphone Query sensor data and the electronic calendar of

the user Use thresholds to classify into different states

There is no learning from user preferences

CSE567 project 1012/8/2005

Feature Extraction

Architectural Overview

0

111

Supervised Learning

Learning context

Reinforcement Learning

Learning user preferences

ringing tone vibration

offline

online

voice mail

raw data

features

soft sensors

alarm type

close to body conversation extr. noise outdoor physical act.

FFT1 variance1 variance2 derivative3

CSE567 project 1112/8/2005

Feature Extraction

potentially useful features running mean running variance derivative exponential smoothing frequency components (FFT)

short (10 sec) and long (1 min) running windows010111011010010011010100011110101010111010010111010100011010101

CSE567 project 1212/8/2005

Feature Extraction (cont’d)

system of pluggable components(using observer/composition design patterns)

rawsensor

windowfilter

windowfilter

FFTfilter

meanfilter

var.filter

windowfilter

meanfilter

joinfilter

CSE567 project 1312/8/2005

Feature Extraction - Experiments

measured execution times on the iPAQ

Feature Extractor Time

Mean (window 20) 0-1 ms

Variance (window 5500) 15-18 ms

Derivative 0-1 ms

Exponential Smoothing 0-1 ms

Fast Fourier Transform (window 4096)

1032-1083 ms

Window (window 5500, w/ overlap)

102 ms

CSE567 project 1412/8/2005

Feature Extraction


0

111




online

voice mail

raw data

features

alarm type

Supervised Learning

Learning context offline

soft sensorsclose to body conversation extr. noise outdoor physical act.


CSE567 project 1512/8/2005

Learning Context

identified five relevant soft sensors Close to Body Conversation Extreme Noise Outdoor Physical Activity

Using annotated user traces, we build a Support Vector Machine classifier for each soft sensor offline

CSE567 project 1612/8/2005

Learning Context (cont’d)

Support Vector Machines

++++

++ +

++

+ +

+

++

-

-

--

--

-

Fea

ture

2

Feature1

? ??

CSE567 project 1712/8/2005


Support Vector Machines

++++

++ +

++

+ +

+

++

-

-

--

--

-

Fea

ture

2

Feature1

Maximize distance between closest

point and boundary(Optimization)

CSE567 project 1812/8/2005


Support Vector Machines w/ Gaussian Kernel

++

++

++ +

++

+ +

+

+

+

-

-

-

-

- -

-

Fea

ture

2

Feature1

-

CSE567 project 1912/8/2005

Learning Context – Experiments

recorded and annotated traces at Allen Center

Annotation Trace1 Trace2 Total

Close To Body 48 min 31 min 79 min

Conversation 24 min 5 min 29 min

Extreme Noise 4 min 9 min 13 min

Outdoor 6 min 10 min 16 min

Physical Activity

5 min 5 min 10 min

Total recorded time

62 min 55 min 117 min

CSE567 project 2012/8/2005

Learning Context – Experiments (cont’d) 2-fold Cross-Validation results (in %)

Annotation Precision

Recall Accuracy

Close To Body 88.058 87.647 82.799

Conversation 77.879 32.247 81.678

Extreme Noise 81.569 88.547 96.511

Outdoor 68.478 61.204 91.874

Physical Activity

94.309 79.293 97.876

Overall 82.059 69.788 90.148

CSE567 project 2112/8/2005

Learning Context – Experiments (cont’d) measured execution times on iPAQ and

desktopAnnotation Classificatio

nTraining

Close To Body 127 ms 4 sec (±2)

Conversation memory error

212 sec (±134)

Extreme Noise 74 ms 3 sec (±2)

Outdoor 132 ms 4 sec (±2)

Physical Activity

12 ms 3 sec (±2)

Overall >86 ms 45 sec (±28)

offlineonline

CSE567 project 2212/8/2005

CSE567 project 2312/8/2005

Feature Extraction


0

111

Supervised Learning

Learning context offline

raw data

features

soft sensors




online

voice mail alarm type

close to body conversation extr. noise outdoor physical act.


CSE567 project 2412/8/2005

Learning User Preferences

Scenario 1 Soft sensors detect CloseToBody,

Conversation Call comes in and phone is ringing User hangs up phone without taking call

Scenario 2 Soft sensors detect CloseToBody,

Physical Activity Call comes in and phone vibrates User does not notice call

bignegativereward

smallnegativereward

CSE567 project 2512/8/2005

Learning User Preferences

Reinforcement Learning (Q-Learning)

Neural Network

soft sensor states+ current alarm type

new alarm type

CSE567 project 2612/8/2005

What we can do in the future

use an open-source VoIP library, e.g. JVOIPLIB, to enable real cell phone capability

apply more advanced reinforcement learning on user preferences

evaluate the effectiveness of learning user preferences

CSE567 project 2712/8/2005

End

zhiphone: a mobile phone that learns context and user preferences jing michelle liu raphael hoffmann...

Documents

sensor module

sensor box

action module

decision module

microphonequery sensor

cse567 projectthis

phone modulesensors

annotated traces