pervasive computing: vision and challenges m. satyanarayanan ieee personal communications, 2001

Pervasive Computing: Vision and Challenges

M. SatyanarayananIEEE Personal Communications, 2001

Evolution & Related Fields

• Pervasive computing represents a major evolutionary step in a line of work dating back to the mid-1970s.

• Two distinct earlier steps in this evolution:– Distributed systems– Mobile computing


Figure 2. Taxonomy of computer systems research problems in pervasive computing.


• Distributed systems– Arose at the intersection of personal computers and

local area networks.

– The research that followed from the mid-1970s through the early 1990s created a conceptual framework and algorithmic base (for various types of networks)

– Spans many areas that are foundational to pervasive computing


• Mobile computing– Led by the appearance of full-function laptop computers

and wireless LANs in the early 1990s

– How to build a distributed system with mobile clients??

– Many basic principles of distributed system design continued to apply

– Four key constraints of mobility forced the development of specialized techniques:

• Unpredictable variation in network quality• Lowered trust and robustness of mobile elements• Limitations on local resources imposed by weight and size

constraints• Concern for battery power consumption

Other related fields

• Sensor networks• Human-computer interaction• Artificial intelligence


• Sensor Networks– A sensor network consist of a

large number of tiny autonomous computing devices, each equipped with sensors, a wireless radio, a processor, and a power source.

– Sensor networks are envisioned to be deployed unobtrusively in the physical environment to monitor a wide range of environmental phenomena

• e.g., environmental pollutions, seismic activity, wildlife


• Human Computer Interaction (HCI)– HCI is the study of interaction between people

(users) and computers.

– Goal of HCI: to improve the interaction between users and computers by making computers more user-friendly and receptive to the user's needs.

– Long term goal of HCI: to design systems that minimize the barrier between the human's cognitive model of what they want to accomplish and the computer's understanding of the user's task.


• Artificial Intelligence– AI can be defined as intelligence exhibited by an

artificial (non-natural, manufactured) entity.

– AI is studied in overlapping fields of computer science, psychology and engineering, dealing with intelligent behavior, learning and adaptation in machines, generally assumed to be computers.

– Research in AI is concerned with producing machines to automate tasks requiring intelligent behavior.

Key research thrusts

• Pervasive computing incorporates four additional research thrusts:– Effective use of smart spaces– Invisibility– Localized scalability– Masking uneven conditioning

Effective use of smart spaces

• By embedding computing infrastructure in building infrastructure, a smart space brings together two worlds that have been disjoint until now.

• The fusion of these worlds enables sensing and control of one world by the other.– Automatic adjustment of heating, cooling, and lighting

levels in a room based on an occupant’s electronic profile.

Invisibility

• The ideal expressed by Weiser is complete disappearance of pervasive computing technology from a user’s consciousness (minimal user distraction).

• If a pervasive computing environment continuously meets user expectations and rarely presents him with surprises, it allows him to interact almost at a subconscious level.

Localized scalability

• As smart spaces grow in sophistication, the intensity of interactions between a user’s personal computing space and his/her surroundings increases.

• This has severe bandwidth, energy, and distraction implications for a wireless mobile user.

• The presence of multiple users will further complicate this problem.

• Good system design has to achieve scalability by severely reducing interactions between distant entities.

Masking uneven conditioning

• Huge differences in the “smartness” of different environments — what is available in a well-equipped conference room, office, or classroom may be more sophisticated than in other locations.

• This large dynamic range of “smartness” can be jarring to a user, detracting from the goal of making pervasive computing technology invisible.

• One way to reduce the amount of variation seen by a user is to have his/her personal computing space compensate for “dumb” environments.

Detailed design and implementation issues

• User intent• Cyber foraging• Adaptation strategy• High-level energy management• Client thickness• Context awareness• Balancing proactivity and transparency• Impact on layering• Privacy and trust

User intent

• For proactivity to be effective, it is crucial that a pervasive computing system track user intent. – Otherwise, it will be almost impossible to determine which

system actions will help rather than hinder the user.

• For example, suppose a user is viewing video over a network connection whose bandwidth suddenly drops. Should the system:– Reduce the fidelity of the video?– Pause briefly to find another higher-bandwidth connection?– Advise the user that the task can no longer be

accomplished?

• The correct choice will depend on what the user is trying to accomplish.

Cyber foraging (aka “living off the land”)

• Dynamically augment the computing resources of a wireless mobile computer by exploiting wired hardware infrastructure.

• As computing becomes cheaper and more plentiful, it makes economic sense to “waste” computing resources to improve user experience.

• In the forseeable future, public spaces such as airport lounges and coffee shops will be equipped with compute servers or data staging servers for the benefit of customers, much as table lamps are today. – Today, many shopping centers and cafeterias offer their

customers free wireless internet access.

Adaptation strategy

• Adaptation is necessary when there is a significant mismatch between the supply and demand of a resource (e.g. wireless network bandwidth, energy, computing cycles or memory).

• There are three alternative strategies for adaptation in pervasive computing:– A client can guide applications in changing their behavior so that

they use less of a scarce resource. This change usually reduces the user-perceived quality, or fidelity, of an application.

– A client can ask the environment to guarantee a certain level of a resource (reservation-based QoS systems). From the viewpoint of the client, this effectively increases the supply of a scarce resource to meet the client’s demand.

– A client can suggest a corrective action to the user. If the user acts on this suggestion, it is likely (but not certain) that resource supply will become adequate to meet demand.

High-level energy management

• Sophisticated capabilities such as proactivity and self-tuning increase the energy demand of software on a mobile computer in one’s personal computing space.

• Making such computers lighter and more compact places severe restrictions on battery capacity, so the higher levels of the system must be involved in memory management.

• One example is energy-aware memory management, where the operating system dynamically controls the amount of physical memory that has to be refreshed.

• Another example is energy-aware adaptation, where individual applications switch to modes of operation with lower fidelity and energy demand under operating system control.

Client thickness (or client’s hardware capabilities)

• For a given application, the minimum acceptable thickness of a client is determined by the worst-case environmental conditions under which the application must run satisfactorily.

• A very thin client suffices if one can always count on high-bandwidth low-latency wireless communication to nearby computing infrastructure, and batteries can be recharged or replaced easily.

• If there exists even a single location visited by a user where these assumptions do not hold, the client will have to be thick enough to compensate at that location.

• This is especially true for interactive applications where crisp response is important.

Context awareness

• A pervasive computing system must be cognizant of its user’s state and surroundings, and must modify its behavior based on this information.

• A user’s context can be quite rich, consisting of various attributes – Ex) physical location, physiological state (e.g., body temperature

and heart rate), emotional state (e.g., angry, distraught, or calm), personal history, daily behavioral patterns, and so on.

• If a human assistant were given such context, he or she would make decisions in a proactive fashion, anticipating user needs.

• In making these decisions, the assistant would typically not disturb the user at inopportune moments except in an emergency.

• A pervasive computing system should emulate such a human assistant.

Balancing proactivity and transparency

• Unless carefully designed, a proactive system can annoy a user and thus defeat the goal of invisibility.

• A mobile user’s need and tolerance for proactivity are likely to be closely related to his/her level of expertise on a task and familiarity with his/her environment.

• A system that can infer these factors by observing user behavior and context is better positioned to strike the right balance.

• For transparency, a user patience model can be implemented to predict whether the user will respond positively to a fetch request. – So the user interaction is suppressed and the fetch is handled

transparently.

Impact on layering

• Proactivity and adaptation based on corrective actions seem to imply exposure of much more information across layers than is typical in systems today.

• Layering cleanly separates abstraction from implementation and is thus consistent with sound software engineering.

• Layering is also conducive to standardization since it encourages the creation of modular software components.

Privacy and trust

• As a user becomes more dependent on a pervasive computing system, it becomes more knowledgeable about that user’s movements, behavior patterns and habits.

• Exploiting this information is critical to successful proactivity and self-tuning (invisibility), but also may cause serious loss of privacy.

• User must trust the infrastructure to a considerable extent and the infrastructure needs to be confident of the user’s identity and authorization level before responding to his/her requests.

• It is a difficult challenge to establish this mutual trust in a manner that is minimally intrusive and thus preserves invisibility.

Charting Past, Present, and Future Research in Ubiquitous

Computing

Gregory Abowd and Elizabeth MynattACM TOCHI 2000

• Natural interfaces• Context-aware computing• Automated capture and access• Everyday computing• System evaluation challenges

Computing with natural interfaces

• Ubicomp inspires “off-the-desktop” applications– Needs “off-the-desktop” means of interaction– Speech, gestures, writing

• More accessible• Easier to use???


• Need for first-class natural data types – To ease the development of more apps w/

natural interfaces• Should be easy keyboard and mouse

– Support for “fundamental” operations• E.g., text and speech recognitions

– Support for “extra” operations• Tivoli Live Board: special input like structured

gestures such as dragging and freeform selection

• Classroom 2000: annotating pen strokes with timestamps


• Error prone interaction– Permit new and numerous mistakes– People do not have perfect recognition

• As low as 54%; cursive handwriting 88%; printed handwriting 96.8%

– Recognition accuracy == user satisfaction?? • Not really: complexity of error recovery dialogues and

value-added benefit of any given efforts• Entering a command vs. writing journal entries

• Several research areas– Error reduction (about 5-10%)– Error detection– Reusable toolkit for error handling

Context aware computing

• Current Systems– Generally using position and identification of

objects– Still do not provide a complete context– Definition of context is limited

• Research areas– Context toolkits

• Toolkit for sensing environment• Explicit use of sensed information is up to program

– What is context?– How is context represented?

What is context?

• Who– Currently generally tailored to one user– How important are others in determining our behavior

• How could this be captured?

• What– Attempt to figure out what is currently happening – Sense environment, use calendar software etc.

• Where– Location based information, e.g., GPS– Most explored context information

• When– Easily obtained information -- Computer is good at remembering time

• Although determining when one event stops and another begins is not easy

• Why– Even harder than the “what” question, biometric sensors might help

(e.g., body temperature, heart rate, etc)

Toward context aware computing

• Context representation– Requires universal context schemes or toolkits with

standard context representations• Context sensing and fusion

– How to make context-aware computing “ubiquitous”?

– In practice, there are few truly ubiquitous, single-source context services

• E.g., GPS does not work indoors; different indoor localization schemes have different characteristics (e.g., cost, range)

– Like sensor fusion, context fusion handles seamless handling of sensing responsibility between boundaries of different context services

– Combining multiple context sources can increase the accuracy of context information

Automated capture and access

• Recording information and data as it occurs– Computers are inherently good at recording, people are not– People freed up to summarize and understand– Most work in academic/ classroom settings

• Time stamping lectures, digital whiteboards

• Challenges in “capture and access”– Sometime we don’t know we want to capture something until

after its already happened• How could the computer know that?• If it captures everything then we need a system of sorting and

filtering (access)– Gmail Problem

– Access is a problem because capturing of raw data can be burdensome for sifting through; systems need to recognize important events facilitate access

Everyday computing

• Continuous interactions (i.e., no clear beginning or end)– Both fundamental activities like communication and long-term

endeavors do not have predefined starts and ends; information from past can be recycled

– Very different traditional HCI design which assumes “closure” with clear goals like spell checking, dialogue, etc.

• Interruption is expected:– People are constantly interrupted– Computer systems must recognize interruption and change

state• Also computers must appropriately inform users

• Multiple activities operate concurrently: – People multitask and rapidly switch task based on external

unpredictable environment– Systems need to adapt to this opportunistic behavior and

change accordingly

Toward everyday computing

• Develop continuously present interface– No current model of continuously present interfaces,

even people are not continuously present– Create an interface that doesn’t get annoying (e.g.,

wearable devices)

• Determine what information should require my attention and what should be display peripherally

• Connect events in the physical and virtual worlds (e.g., face to face vs. email, document, webs)

• Modify/fuse existing HCI schemes to efficiently support everyday computing (but evaluation is challenging and laborious)

System evaluation challenges

• Hard to evaluate Ubicomp Systems– Little publish on ubicomp evaluation– Systems often required to be fully connected leading to

systems that are hard to build– Lack of development toolkits make system creation

difficult– Systems often need to be integrated into peoples lives

which using big clunky prototypes does not lead itself well too

– Task/Goal centric approaches don’t work in ubicomp

Example Projects

• Pervasive computing projects have emerged at major universities and in industry:– Project Aura (Carnegie Mellon University)– Oxygen (Massachusetts Institute of Technology)– Portalano (University of Washington)– Endeavour (University of California at Berkeley)– Place Lab (Intel Research Laboratory at Seattle)

Example Projects : Project Aura (1)

• Aura (Carnegie Mellon University)– Distraction-free (Invisible) Ubiquitous Computing.


• Moore’s Law Reigns Supreme– Processor density– Processor speed– Memory capacity– Disk capacity– Memory cost– ...

• Glaring Exception– Human Attention

Adam & Eve 2000 AD

Human Attention


• Aura Thesis:– The most precious resource in computing is human

attention.

• Aura Goals:– Reduce user distraction.– Trade-off plentiful resources of Moore’s law for human

attention.– Achieve this scalably for mobile users in a failure-prone,

variable-resource environment.


• The Airport Scenario

– Jane wants to send e-mail from the airport before her flight leaves.

• She has several large enclosures• She is using a wireless interface

– She has many options.• Simply send the e-mail

– Is there enough bandwidth?• Compress the data first

– Will that help enough?• Pay extra to get reserved

bandwidth– Are reservations available?

• Send the “diff” relative to older file– Are the old versions around?

• Walk to a gate with more bandwidth– Where is there enough bandwidth?

– How do we choose automatically?


• The Mobile Task Scenario

– Aura saves Scott’s task.– Scott enters office and gets strong

authentication and secure access.– Aura restores Scott’s task on

desktop machine and uses a large display.

– Scott controls application by voice.– Bradley enters room.– Bradley gets weak authentication,

Scott’s access changes to insecure.– Aura denies voice access to

sensitive email application.– Scott has multi-modal control of

PowerPoint application.– Aura logs Scott out when he leaves

the room.

Example Projects : Oxygen

• Oxygen (MIT)– Pervasive human-centered computing.– Goal of Oxygen is bringing abundant computation and

communication, as pervasive and free as air, naturally into people's lives.

Example Projects : Oxygen (2)

• To support highly dynamic and varied human activities, the Oxygen system must be

– pervasive— it must be everywhere, with every portal reaching into the same information base;

– embedded— it must live in our world, sensing and affecting it; – nomadic— it must allow users and computations to move around

freely, according to their needs; – adaptable— it must provide flexibility and spontaneity, in response to

changes in user requirements and operating conditions; – powerful, yet efficient— it must free itself from constraints imposed by

bounded hardware resources, addressing instead system constraints imposed by user demands and available power or communication bandwidth;

– intentional— it must enable people to name services and software objects by intent, for example, "the nearest printer," as opposed to by address;

– eternal— it must never shut down or reboot; components may come and go in response to demand, errors, and upgrades, but Oxygen as a whole must be available all the time.

Related Projects: Portalano

• Portolano (University of Washington)– An expedition into invisible computing.

– Expedition goals:• Connecting the physical world to the world-wide information fabric

– Instrument the environment: sensors, locators, actuators – Universal plug-and-play at all levels: devices to services– Optimize for power: computation partitioning, comm. opt.– Intermittent communication: new networking strategies

• Get computers out of the way– Don’t interfere with user’s tasks– Diverse task-specific devices with optimized form-factors– Wide range of input/output modalities

• Robust, trustworthy services– High-productivity software development– Self-organizing, active middleware, maintenance, monitoring– Higher-level, meaningful services

Related Projects: Portalano (2)

• Scenario– Alice begins the day with a cup of coffee and her

personalized newspaper.– When her carpool arrives, she switches to reading the

news on her handheld display, where she notices an advertisement for a new 3-D digital camera.

– It looks like something that would interest her shutterbug-friend Bob, so Alice asks her address book to place the call.


• Scenario (2)– Bob's home entertainment system softens the volume of

his custom music file as his phone rings.– Alice begins telling Bob about the camera, and forwards

him a copy of the advertisement which pops up on his home display.

– Bob is sold on the product, and after hanging up with her, he asks his electronic shopping agent to check his favorite photography stores for the lowest price and make the purchase.


• Scenario (3)– When the camera arrives, Bob snaps some photos of his

neighbor's collection of antique Portuguese navigation instruments.

– After reviewing the photo album generated automatically by a web-based service, Bob directs a copy of his favorite image to the art display in his foyer.

– He also sends a pointer to the photo album to Alice and instructs his scheduling agent to set up a lunch date so that he can thank her for the suggestion.

Example Projects : Endeavour

• The Endeavour Expedition (UC Berkeley)– Charting the Fluid Information Utility

• Endeavour Goal:– Enhancing human understanding through the use of

information technology.

• Principles of Pervasive Computing• Evolution & Related Fields• Example Projects• Other Scenarios• References

Other Scenarios

• Buy drinks by Friday (1)– Take out the last can of soda

– Swipe the can’s UPC label, which adds soda to your shopping list

– Make a note that you need soda for the guests you are having over this weekend

Other Scenarios

• Buy drinks by Friday (2)– Approach a local supermarket

– AutoPC informs you that you are near a supermarket

– Opportunistic reminder: “If it is convenient, stop by to buy drinks.”

Other Scenarios

• Buy drinks by Friday (3)

- Friday rolls around and you have not bought drinks

- Deadline-based reminder sent to your pager

Other Scenarios

• Screen Fridge– Provides:

• Email• Video messages• Web surfing• Food management• TV• Radio• Virtual keyboard• Digital cook book• Surveillance camera

Other Scenarios

• The Active Badge– This harbinger of inch-scale computers contains a small

microprocessor and an infrared transmitter.

– The badge broadcasts the identity of its wearer and so can trigger automatic doors, automatic telephone forwarding and computer displays customized to each person reading them.

– The active badge and other networked tiny computers are called tabs.

Other Scenarios

• The Active Badge

Other Scenarios

• Edible computers: The pill-cam– Miniature camera– Diagnostic device– It is swallowed

• Try this with an ENIAC computer!

Other Scenarios

• Artificial Retina– Direct interface

with nervous system

– Whole new computational paradigm (who’s the computer?)

Other Scenarios

• Smart Dust– Nano computers that

couple:• Sensors• Computing• Communication

– Grids of motes (“nano computers”)

pervasive computing: vision and challenges m. satyanarayanan ieee personal communications, 2001

Documents

pervasive computing

wildlife slide

computing infrastructure

uneven conditioning

function laptop computers

additional research

local area networks

study of interaction