an overview of context-aware computing jason i. hong group for user interface research university of...

An Overview of Context-Aware Computing

Jason I. Hong

Group for User Interface ResearchUniversity of California at Berkeley

Apr 25 2001 2

Motivation

• Modern computers are divorced from reality Unaware of who, where, and what around them Leads to mismatch

• Computers have extremely limited input Aware of explicit input only Can take a lot of effort to do simple things

• Context-Aware Computing Making computers more aware of the physical

and social worlds we live in Breaking computers out of the box

Apr 25 2001 3

Why Context-Aware Computing?

Context TypesExisting Examples Human Concern

Room ActivitySmoke Alarm Safety

Room ActivityAuto Lights On / Off Convenience

Object IdentityBarcode Scanners Efficiency

Personal Identity & Time

File Systems Finding Info

TimeCalendar Reminders Memory

Apr 25 2001 4

Technology Trends

• Sensors GPS, Active Badges, Active Bats Smart Dust Cameras and microphones

• Recognition algorithms MSR Radar location from 802.11 Smart Floor footstep force

• Wireless technologies Bluetooth, 802.11, cell phones

Apr 25 2001 5

Existing Examples

Why Context-Aware Computing?

Context Types Potential Examples Human Concern

Activity Convenience

Activity Finding Info

Identity Memory

Identity & Time Safety

Time Efficiency

Identity

Time

Location

Proximity

Activity

History

…

Smoke Alarm

Auto Lights On / Off

Barcode Scanners

File Systems

Calendar Reminders

Health Alert

Auto Cell Phone Off In Meetings

Service FleetDispatching

Tag Photos

Proximal Reminders

Apr 25 2001 6

Outline

Background and MotivationSample Context-Aware AppsBuilding Context-Aware AppsOpen Research ChallengesSummary

Apr 25 2001 7

Sample Context-Aware Apps

1 2

3 4

Apr 25 2001 8

Sample Context-Aware AppsActive Badges

Active BadgeOlivetti / AT&T

Hopper, Harter, et al

• Badges emit infrared signals Gives rough location + ID

• Teleport Redirect screen output from

"home" computer to nearby computer

• Phone forwarding Automatically forward phone

calls to nearest phone

Apr 25 2001 9

Sample Context-Aware Apps ParcTabs

ParcTabsXerox PARC

Want, Schilit, et al

• Active badge + wireless Rough location + ID

• Proximate selection Interfaces for nearby objects

• Auto-diaries People, places, and time

• Triggers Alerts on preset events

• Reconfiguration Bind device to room

Apr 25 2001 10


ParcTabsXerox PARC







Apr 25 2001 11

ParcTabsXerox PARC








Apr 25 2001 12







ParcTabsXerox PARC


Apr 25 2001 13

Apr 25 2001 14

Apr 25 2001 15

Apr 25 2001 16







ParcTabsXerox PARC


Apr 25 2001 17

"Like living in a rule-based expert system!"

Apr 25 2001 18







ParcTabsXerox PARC


Apr 25 2001 19

Sample Context-Aware Apps Tour Guides

CyberguideGeorgia TechAbowd et al

• GPS or infrared tracking Fairly precise location

• Display location on screen• Predefined points of interest

Automatically pop up if nearby

• Travel journal Keep log of places seen and

photographs taken

Apr 25 2001 20



Apr 25 2001 21



Apr 25 2001 22

Sample Context-Aware Apps Enhanced PDAs

Microsoft ResearchHinckley et al

Apr 25 2001 23

Sample Context-Aware Apps Enhanced PDAs

• Voice memo recording Hold like phone near mouth to

start recording

• Portrait / Landscape mode Just physically rotate screen

• Tilt scrolling Tilt instead of scrollbars

• Power management Turn on if being held and tilted

Microsoft ResearchHinckley et al

Apr 25 2001 24

Some Issues in Context-Aware Computing

• Sensor ambiguity Sensors not 100% reliable Precision / Accuracy / Granularity

• Sensor Fusion Merging different sensor inputs together

• Self-contained vs. distributed systems PDA doesn't need location sensors if it can ask nearby

sensors to approximate

• Requires lots of knowledge and effort to build Sensors, recognition algorithms, devices, application Few kinds of context beyond location + ID used There haven't been many rigorous evaluations of utility

Not PreciseNot Accurate

PreciseNot Accurate

Rough Granularity

Fine Granularity

Apr 25 2001 25

Some Issues in Context-Aware Computing

• Sensor ambiguity Sensors not 100% reliable Precision / Accuracy / Granularity

• Sensor Fusion Merging different sensor inputs together

• Self-contained vs. distributed systems PDA doesn't need location sensors if it can ask nearby

sensors to approximate

• Requires lots of knowledge and effort to build Sensors, recognition algorithms, devices, application Few kinds of context beyond location + ID used There haven't been many rigorous evaluations of utility

Apr 25 2001 26

Outline


Apr 25 2001 27

Building Context-Aware Apps

• Describe support at app-level ParcTab System Context Toolkit MUSE Cooltown Mobisaic Stick-E Notes Context Fabric

• Try to make it easier to build a certain class of context-aware apps

Apr 25 2001 28

A Rough Taxonomy of Context-Aware Apps

• Triggers• Metadata Tagging• Reconfiguration and Streamlining• Input specification• Presentation

Apr 25 2001 29


• Triggers On X do Y "Notify doctor and nearby ambulances if

serious health problem detected" "Remind me to talk to Chris about user studies

next time I see him"

• Metadata Tagging "Where was this picture taken?" "Find all notes taken while Mae was talking" Memory prosthesis

Apr 25 2001 30


• Reconfiguration and Streamlining Telephone forwarding and Teleport Turn off cell phone in theaters Automatically adjust brightness / volume Automatic file pre-caching Select modes in multimodal interaction Multimedia / Bandwidth adaptation

Apr 25 2001 31


• Input specification Send mail only to people in building now Print to nearest printer "Find gas stations nearest me"

• Presentation Current location Idle? Currently in? Contextual info about objects Proximate selection

Apr 25 2001 32

ParcTab System(Schilit 1995)

• Three key abstractions Device Agents, User Agents, Active Maps

Apr 25 2001 33

ParcTab System(Schilit 1995)

• Queries Check context state in Agents or Active Maps {RoomLID:35-2-*} {{Printer*}{modellw}}

• Events Standing queries Notified on changes in context state

Apr 25 2001 34

Context Toolkit(Dey, Salber, Abowd 2001)

• Toolkit for distributed context-aware apps Framework for acquiring and handling context Standard components

• Three key abstractions Widgets, Interpreters, and Aggregators

Apr 25 2001 35


• Widgets abstract out sensors

GPSActiveBadge

App App

LocationWidget

Cell PhoneLocation

ActiveBadge

Apr 25 2001 36


• Interpreters transform context data

App

LocationWidget

Location to Room

Interpreter

Location to Street

Interpreter

Apr 25 2001 37


• Aggregators group related widgets together

App

PersonAggregator

ActivityWidget

AffectWidget

App

LocationWidget

Location to Room

Interpreter

Apr 25 2001 38


In / Out Board

Apr 25 2001 39


Dummbo

Apr 25 2001 40

MUSE(Castro and Muntz 2000)

• Sensor management issues Sensor discovery (currently via Sun's Jini) Specify performance goals and resource

constraints "Find a printer but minimize power consumption"

• Probabilistically modeling sensor data Can measure service cost Sensor fusion (via Bayesian nets and HMMs)

Apr 25 2001 41

CoolTown(Hewlett-Packard 2000)

• Since web is universal, link context awareness to web

• Literally everything has a URL People, places, things Infrared beacons, bar codes, etc

• Literally everything has a web page Current status, contact info, services offered, etc

Apr 25 2001 42


• SF Exploratorium• Get extra info• Create bookmarks for

later use

Apr 25 2001 43


• Buses equipped with GPS and net

• In bus Show location Show nearby

points of interest

• Waiting for bus Show location Show wait time

Apr 25 2001 44

Mobisaic(Voelker and Bershad 1994)

• Context-awareness + web page authoring• Dynamic URLs

Variables processed by modified web browser

http://www.offices/$(Location).html

http://www.offices/525.html

• Active Documents Extensions to HTML

<!- subscribe to $(Location), reload ->

Apr 25 2001 45

Stick-E Notes(Pascoe 1997)

• Authoring tool for using context

<note><required><at> (1,4) .. (3,5)<facing> 150 .. 210<during> December

<body>The large floodlit building at the bottom of the hill is the cathedral.

Apr 25 2001 46

Context Fabric(Hong and Landay 2001)

• Distributed sensors, services, and devices Won't be designed a priori for interoperability

• Shift burden of context-awareness from individual devices onto infrastructure Provide basic services usable by any device Provide a high-level way of specifying context

needs Dynamically coordinate sensors and services

Apr 25 2001 47

Context Fabric(Hong and Landay 2001)

• Push burden of context-awareness from individual devices onto network Share computational power, data, sensors Support incremental evolution Support richer set of devices

• Provide core set of context services Basic services that exist everywhere Uniform level of abstraction hiding nasty details

Apr 25 2001 48

Automatic Path Creation

ZIP Weather

GPS ZIP

Cell GPS

Cell ZIP

ZIP TheatersGPS ZIP

ZIP WeatherGPS ZIP

• A way of assembling small services into larger, more interesting ones

• Operators

• Paths

Apr 25 2001 49


Sensor Input ZIP Theaters

GPS

GPS ZIP

What are the nearby movie theaters?


Cell

GPS ZIPCell GPS


Cell

Context OutputCell ZIPMovie

Theaters

• Suppose we had a way to specify context needs• Suppose we had a way to specify service semantics• Automatically assemble paths based on resources• Don't have to know specific sensors and services

Apr 25 2001 50

Context Fabric Architecture

Sensor Management Service

ServicesContextEvent

Service

ContextQuery

Service


AppApplication

Layer

ContextLayer

Logical SensorLayer

Physical SensorLayer

ContextSpecification

Language

Apr 25 2001 51

Outline


Apr 25 2001 52

Open Research ChallengesSystems Issues

• Programming model Programming the physical world Unreliable sensors, recognition algorithms,

plus standard distributed computing issues• Interoperability

Sensors, services, and devices Useless if everyone has proprietary / custom

systems Need standard data formats, protocols, and

frameworks Varying capabilities of sensors, services, and

devices

Apr 25 2001 53

Open Research ChallengesSystems Issues

• Need clearer definitions of context What is and isn't context? Temperature? The monitor I'm looking at?

Personal history? Have to avoid the AI tarpit (ie does it matter?)

Apr 25 2001 54

Open Research ChallengesPeople Issues

• Avoiding embarrassing situations Active Badges + bathrooms Inconvenient phone forwarding

• Avoiding dangerous situations Need to take into consideration cost of mistake Smoke alarms when cooking Lights that turn off when you're still there Woman locked in "smart toilet stall"

• Will adding more context really help here?

Apr 25 2001 55

Open Research ChallengesPeople Issues

• Making it predictable and understandable Setting preferences "I want my cell phone to ring except in theaters

and when I'm in a meeting unless…" Why the heck did it do that?

• Privacy What does the computer know about me? What do others know about me? What do I gain? What do I lose?

Apr 25 2001 56

Take Home Ideas

• Strides in sensors, recognition, and wireless are enabling new context apps

• Still need to lower barriers to entry• Lots of systems issues, even more people

issues• Lots of potential here for new kinds of

interaction and applications

Apr 25 2001 57

The End

Group for User Interface ResearchUniversity of California at Berkeley

Context Fabric Linkhttp://guir.berkeley.edu/cfabric

Apr 25 2001 58

Backup Slides

Apr 25 2001 59

Sensor Management Service

Services

Proposed Solution

ContextEvent

Service

ContextQuery

Service


AppApplication

Layer

ContextLayer

Logical SensorLayer

Physical SensorLayer

ContextSpecification

Language

Agents

Device Agents

User Agents

Apr 25 2001 60

Context Specification Language

• Need to express context about and relationships between People, Places, Things

• Predicates Identity (Who is…? What is…? Is with…?) Location (Near? Nearest? Distance? Path?) Activity (Is busy? Is in meeting? Current task?) Time (In past? In present? In future? On date?)

• Some of this vocabulary done by Schilit Implicitly encoded in his APIs One goal is to extend his work in spec language Another is to make it extensible for future

context types

Apr 25 2001 61

Context Specification Language

• Common parameters Max number of results wanted Return name Return data type (e.g. String, List, Table) Minimal probability of correctness desired Relevant sensor input requestor has

• Event parameters Event rate (e.g. at most 1 event per second) Event callback (e.g. RPC, socket port) Max number of events desired Granularity of change (e.g. 1 meter)

Apr 25 2001 62

What is Context?

• Situated Action (Suchman) Responsiveness, improvisation to situation Actions are moment-by-moment, highly fluid on

situation

• Activity Theory (Nardi) Subject, Object[ive], Operations, Artifacts, Env Activity defines context but it changes as list

above changes Context depends on sensing and inferring, but

difficult to know people or goals

Apr 25 2001 63

What is Context?

• Provides fair criticisms on existing context-aware apps Response: Focus on the more routine and

predictable actions Response: Focus on simple first-order approx Examples: Smoke alarms, Calendar reminders

Response: Is occasionally wrong, but won't get better unless we do more research here

Response: Infrastructure enables more research in this area

Apr 25 2001 64

What is Context?

• Merging of virtual with physical and social Mostly implicit input, often secondary to

task at hand

• People / Places / Things / Virtual / Time People: Location, Identity, Task, Affect Places: Activity, People, Temperature, Audio Things: Location, Identity Virtual: Services, Bandwidth Time: All of these Past, Present, and Future

Apr 25 2001 65

What is Context?

• But it's a rathole to try to define it… Workflows? (Greenberg 2001) Intent and Desire? Natural Language Processing? What isn't context?

• Instead, operationalize it Distributed, multi-device, sensor-based Fairly well-defined and computable concepts

(e.g. location and identity, but not intent or workflow)

Apr 25 2001 66

Why Context-Awareness?Skeptics toward the ubiquitous computing movement quickly point out that those involved in promoting Mark Weiser's initial vision are mainly interested in pushing technology…

As a reaction against this fair criticism, many researchers spend countless hours trying to dream up the "killer application" that will cause everyone to adopt ubiquitous computing technology.

I think this is the wrong goal to have in mind as a researcher for two reasons. First, we are not schooled in the practices to judge a market…

Second, my interpretation of Weiser's vision is not that there would be a single, most compelling use of ubicomp technology, but rather that the totality of the experience would cause a significant paradigm shift in the way we as humans perceive our relationship to interactive computing.

In short, Weiser argued for what I would like to call a "killer existence," in which many disparate devices would provide an array of services in ways that seamlessly complement our lifestyles.

-- Gregory Abowd, 2001

Apr 25 2001 67

Why Infrastructure Approach?

• Sharing of data Collaborative filtering, large data sets

• Sharing of processing power Heavy-duty processing on dedicated machines

• Sharing of sensors Sensors in the environment

• In-place evolution of services New algorithms, new features

• Supports heterogeneity Sensors, Devices, OS, Programming Langs

Apr 25 2001 68

Why Infrastructure Approach?

• Building and evaluating now can Provide more design space for interaction Let us rapidly prototype and evaluate these apps Find problems now Develop the right mechanisms Evolve the right policies

Apr 25 2001 69

Some Sensors

• Temperature• Pressure• Humidity• Soil makeup• Acceleration• Noise• Material stress

• Biometric• Motion• Distance• Touch• Location• Orientation• Light

Apr 25 2001 70

Context-Aware Mechanisms

• Events Something interesting has just happened e.g. "Larry has entered room 306"

• Queries Introspect current context state (discrete) e.g. "Where is Larry now?"

• Filters Modify events and queries based on

preferences and privacy e.g. "Only show others if in office or not"

Apr 25 2001 71

Are Queries and Events Enough?

Triggers

Metadata Tagging

Reconfiguration

Input Specification

Presentation

Query

Query for relevantcontext state

Query current context state before and while running

Query current context stateto fill in the blanks

Query current statefor more information

Event

On Event do Action

On Event do Reconfigure

On Event do Update Presentation

an overview of context-aware computing jason i. hong group for user interface research university of...

Documents

overview of context

berkeley slide

box slide

aware computing jason

nearest phone slide

room parctabs xerox

cell phones slide

time triggers alerts