Prof. Joe Paradiso Responsive Environments Group, MIT Media Lab

http://www.media.mit.edu/resenv Seoul

August-09

Living with Ubiquitous Sensing and Dynamic Responsive Media

2

JAP 4/08

A Historical Perspective

3

JAP 4/08

…to experimental high-energy physics

PhD with Ulrich Becker and Sam Ting (MIT LNS) 1981

4

JAP 4/08

It’s great to do a thesis in Switzerland

CERN is a city of physicists Geneva is between Alps and Jura Any European culture is a short train ride

5

JAP 4/08

PreHistory - 1960

Me - 1976 1974

1976

At Ars Electronica Festival – Sept. 2-9, 2004

8

JAP 4/08

World’s largest home-designed-built synth (1975-1985)

Newton living room - 1986

Musical Interfaces

Sensor Networks as Extension of the Nervous System

Cast our awareness across space, time, scale, modality…

11

JAP 4/08

Bootstrapping a Ubiquitous Sensor Infrastructure

•  Sensor networks are the foot soldiers at the front lines of ubiquitous computing

•  At this point, few if any customers will buy an ensemble of “UbiComp” sensors

•  They will aggregate from established devices –  Home security, appliances, utility devices, entertainment…

Just as the web sprouted from a networked ensemble of personal computers, true “ubicomp” will arise from an

armada of networked devices installed for other purposes.

12

JAP 4/08

Power Strips are Everywhere

•  Needed in Homes, offices, especially the Media Lab! •  Sensors are becoming commodity items

–  Cost of adding sensors to a design is becoming incremental •  Power strips are ideal base platforms for hosting a

sensor network –  Ready access to power –  Power line can be a network port –  Can monitor the status of devices that are plugged in

13

JAP 4/08 PlugPoint – Power Strips as the backbone of a UbiComp Sensor Infrastructure J. Lifton, M. Feldmeier, Y. Ono (Ricoh)

Power Line provides energy & comm

Monitor current profiles, Switch individual sockets Hosts basic sensors (mic,

light, motion) Expansion Port for others Hub for wireless sensor

network

Collaboration with Ricoh Research

Army of Plugs

35 ON MEDIA LAB THIRD FLOOR

MIT Media Lab

Joe Paradiso / ResEnv

Distributed Acoustic Conversation Shielding

Yasuhiro Ono

Ricoh ML affiliate

Paper in ACM SANET 2007

17

JAP 4/08

Cross Reality

IEEE Pervasive Computing July 09 MIT ML, MIT CMS, Stanford, ETH, University of South Australia, Georgia Tech, Sun, etc.

Shadow Lab - Binding real sensor data to virtual worlds

Simple sensor apparitions to explore basic ideas -  Energy use ➔ Amount of smoke

-  Sound Levels ➔ Ripples -  PIR Motion ➔ Waving Fronds -  Temperature ➔ Frond Color

-  Light Levels ➔ Frond Height

Third floor of ML built in Second Life

ResEnv Lab rendered in detail - other areas currently

derived from map

Sensor data piped in and interpreted as real-time

graphic phenomena

Lifton 07

Portal Details – Camera System Deployed to cover entire

building (~50 nodes) 3MP Camera Motorized Panning and Focus Dedicated Video DSP/ARM - (TI DaVinci chip) Real-time Linux OS LCD display (Touchscreen) Contains Spinner Gateway/

Sensor board (detailed on next slide)

45 distributed across Media Lab since October 08

Mat Laibowitz

Device Details – Spinner Gateway Works with or without camera

board Communicates with wearable/

mobile devices in mesh network Serves as reference beacon for

location system Ethernet or slave to linux board Audio system with DSP

AVR32 Environmental Sensing

Motion Humidity and Temperature Light Infrared Communication and

Detection/Proximity Talks to badge systems

45 all over the Media Lab

Power Switch

Invitation to Interaction

Live Portal-Portal Video Feed

Portals and Cross Reality

Real World

Virtual World

Will the market demand privacy?

People are bringing these sensors into their homes and lives on the backs of dedicated appliances and devices

If they don’t feel in control of them, they won’t buy them!

How will they manage their dynamic privacy in a world full of potentially invasive sensors?

- You can’t turn them all off because there are too many, and they are often attached to things that need to stay on or can’t be turned off…

world

group

user Accessible for user’s social network

Accessible for public

Accessible for data owner & close colleagues only

Video, Images, Clean Audio, fine localization

Scrubbed images, garbled audio, coarse localization

Image & audio features only, building-level localization

No ID, building-level features

User-Configurable Dynamic Privacy Settings

Privacy can depend on physical location, time, and inquiry identity

Nan-Wei Gong

NoNo Badge for Dynamic Privacy

Dynamic Privacy…

NoNo Badge for Dynamic Privacy

Dynamic Privacy…

Nan-Wei Gong

MIT Media Lab

Joe Paradiso / ResEnv

Privacy requests (button pushes) per user vs time of day

31

JAP 4/08

32

JAP 4/08

Public Misinformation…

“Human” Energy harvesting will do little for sustainability It will only be valuable in extending/eliminating batteries in

portable devices, wearable sensors, etc.

33

JAP 4/08

Sensor networks for energy conservation

•  Leveraging dense sensor networks for optimal energy management –  40% of US energy is spent in buildings – Pervasive sensor/actuator network can reduce this

•  Optimize heating, AC, lighting for Person not room •  Anticipating behavior & build usage models over time

Wireless Monitor

Wearable Integrated vibration, T & H, Light @ µW Mark Feldmeier

Ceiling PIR 802.15.4 bridge

Ceiling HVAC Damper Control

Measure temperature, humidity, airflow, PIR motion Continuous Control of Damper

Open/Close Windows

37

JAP 4/08

Learning Comfort

38

JAP 4/08

Very Early Data – Learned Setbacks

39

JAP 4/08

SportSemble 3x HiG Accl, 3x LowG Accl, 3x HiRateGyro, 3x magnetometer

40

JAP 4/08 Network In Action

To be presented at ISWC 09

Return to Spring Training, 3/09

New device has onboard removable flash for continual data storage (RF synch), Bi-range full IMU

Soon measuring entire team and streaming data to MIT/MGH

Red Sox Spring Training - March 09

Video by Alex Reben

44

JAP 4/08

Exchange Business Card, Bookmark Demos

45

JAP 4/08

Bookmark data posted on website

46

JAP 4/08

Bookmark data posted on website

47

JAP 4/08

Badge Accelerometer & Audio Data

48

JAP 4/08

Interest Meter & Group Dynamics

Collaboration with Human Dynamics Group

Badge-Badge Badge-Demo

Uninterested

Interested

Uninterested

Interested

Classification Value = f (voice, motion, face-face time…)

80% Accuracy

49

JAP 4/08

Affiliated Wearers

Affiliated people tend to spend more time face-face and move together!

Accuracy of inferred affiliation: 93%

Face-Face Time (IR) Correlated Motion (accels)

50

JAP 4/08

Affiliated Wearers from Energy Only

Timekeeping for Talks

T – 3 mins T – 2 mins T – 1.5 mins T – 1 mins T – 30 sec

Out of time!! •  24 talks in the morning (research updates) •  5 Minute time limit on each! •  Audience badges flashed time queues •  We didn’t run over (first time ever…)!!

SPINNER •  Sensate Pervasive Imaging Network for Narrative

Extraction from Reality

•  Unites wearable human sensing with video capture •  Use of wearable sensing allows access to subject/data

channels far beyond what can be achieved with standard image pixel processing

Mat Laibowitz - PhD finished fall 09

Query multimodal sensor networks in human terms (through narrative/stories)

Overall System Diagram

Behavior Models

Sensor /Action Models

Captured Events

(video clips w / sensor data )

Assembling of Narrative(video )

Meaningful Narrative Discourse

(video)

StoryModel

Design /Input

UserInterface

Sensor NetworkInternal Mappings

Sensor /Imaging Physical Network

Sensor Data

TrainingDataFE

EDB

ACK

FEED

BAC

K

Camera System

The Age of Ubiquitous Media

•  Is here, whether you like or not

•  Can record your entire life, too much video to ever watch

•  Need a mechanism to create/extract meaningful content

Device Details – Spinner Social Sensor •  Wearable on collar or as pin/

badge •  Audio system with DSP for

analytics and CD quality recording

•  Mini-SD Slot for copious memory •  Compass for orientation •  3-axis Accelerometer & 2-Axis

Gyro •  IR communication and line of

sight detection/proximity •  802.15.4 Radio with RF Location

engine •  Captures social signal and group

dynamics •  New Badge has OLED Display

System talks to Portals •  Cameras will record whenever Spinner badges

are nearby •  May also use wearable cameras (e.g., N95) for

1’st person perspective

First ever video edited automatically using sensor data and filtered

from a >40 node distributed camera network

Bright and calm - Active - Dark and calm

Story Arc…

59

JAP 4/08

Conclusions •  Sensing, computation, and communication become tightly integrated and

commonly embedded •  Low power and energy scavenging enable active nodes to be embedded and

“forgotten” •  The Ubicomp infrastructure permiates our cities, our dwellings, our objects, our

clothing, and eventually our bodies –  Pervasive-Wearable-Implantable

•  The 5 human senses locked into our body are augmented by interfaces into ubiquitous sensor network data –  Marshall McLuhan for real –  Interface devices now - implantables some day –  Omniscience…

•  This infrastructure mediates everything –  Collaboration, business, social interaction, resource use… –  Context engines filter, represent, and manifest information

•  Google for reality •  Brave New World

–  Privacy, security, promises vs. perils…