smart u-things & internet of things (iot) · u-things physical things with aeb two fundamental...

Smart u-Things &

Internet of Things (IoT)

e-Thing & e-Activity in Cyber World

e-Things: digital things in Cyber World on cyber space

Computers/Networks/Internet

Cyber Space

Web, WbS, SmW, Grid, P2P, EaaS, Cloud

e-Thing

e-Activity

Cyber World Physical World

Physical Thing

Usual Activity

u-Things Physical Things with AEB Two Fundamental Technology Trends

Continuing miniaturization of devices (Moore’s law, new material, nanotech., …)

Available interconnections by ubiquitous/pervasive wired and wireless networks

u-Things: Physical things with some kind of Attachment, Embedment, Blending

(AEB) of computers, sensors, tags, networks, and/or other devices

0.15x0.15 mm, Hitachi/06

u-Things in Physical/Real World

Physical World

u-Thing

Usual Activity

Sensor/M/NEMS, Comps & Per. Nets

Physical Space

UC, ID, Context, Emb. Sys., etc.

u-Things: on the physical space in physical-digital form

e-Thing

e-Activity

Cyber World

Computers & Networks/Internet

Cyber Space

WbS, SmW, Grid, P2P, EaaS, Cloud

Characteristics in 3 Computing Waves/Eras

m-to-1MC

1-to-1PC

1-to-mUC

Main Role/Obj

Proc. Content

Calculations Information u-Services

Data Media

u-Things + Cloud Mainframes Personal ComputersComp Element

Contexts

Size/Form/User

Weiser’s3 Relations3 Waves

Basic Behavior Passive ActiveInteractive

Large/Stationary/Few Small/Portable/Many Invisible/Everywhere/All

Theory/Tech Comp Sci/Eng Info Sci/Eng (? Science: Cybermatics)Smartness/Intelligence

Central Goal Fast/Precise/Reliable Rich/Convenient/Secure Aware/Comfort/Safe

Essential Questions

Can machines think ? AI (Artificial Intelligence)

- By A. Turing in “Computing Machinery and Intelligence” in 1950

Can things think ? Intelligent real/virtual entity

- Consortium of Things That Think (TTT) in MIT Media Lab since 1995

- “When things start to think” by Neil Gershenfeld of MIT in 1999

Can smart u-things be everywhere ? UI (u-Intelligence)

- Ubiquitous Intelligence (UI) or Pervasive Intelligence ( ) from 2003, Ma-Yang, etc

- Residing in everyday objects, environments, systems, ourselves, plant, animal, …

- Real Things u-Things Smart/Intelligent u-Things Smart World

p

IBM’s Smart Planet, New Intelligence 2008

Hyperworld and Smart World Hyperworld (1995)

- “Hyperworld Modeling”, Keynote in VIS, Melbourne, Feb. 1996 (Kunii, Ma, Huang)- “Its basic characteristic is direct mapping between virtual and real worlds via

active devices including sensors, actuators, micro-machines, robots, etc.” - 1996- “A Study on a Hyperworld System of One-to-Many Interaction”, ICAI’97 (Weiser’s 1-to-m relationship)

- “Towards Direct Mapping between Information Worlds and Real Worlds”, LNCS1306, 1997

Smart World (2003)- Based on ubiquitous computers, sensors, actuators, networks, etc.- Pervaded with smart, life-like ubiquitous things/u-things- Characterized by u-services & u-intelligence in the real world

Smart e-Thing

e-Service

Cyber World Smart World

Smart u-Thing

u-Service

Hyperworld

Cyber-Physical-Social Spaces

Web-Intelligence

WIUbi-Intelligence

UI

Ubiquitous IntelligencePervasive smart real things able to sense, think, act, …

Information Explosion Intelligence Explosion!

- The Intelligence Revolution, Interview by Wise Media, ID People Magazine, Apr./05- Ubiquitous Intelligence Summit 2005, A Think Tank, Oulu, Finland, June 15-17, 2005- Journal of Ubiquitous Computing and Intelligence (Yang & Ma), American Sci. Pub./05- IEEE Task Force on Intelligent Ubiquitous Computing (Duman, Yang, Ma), since 2006 - Journal of Autonomic and Trusted Computing (Yang and Ma), ASP, USA, ASP, 2006

Smart Object (Smart u-Object, Smartifact)- A physical object with AEB devices and some smartness/intelligence- Handheld, card, label, sensor, artifact, appliance, goods, furniture, textile, robot, ……

Smart Environment (Smart u-Environment)- A physical spatial environment/site integrating smart u-objects &/ devices- Smart u-Services via these objects/devices and their commun./cooperation

Smart System (Smart u-System)- May be a physical system/network for management, monitoring, emergence, …- May be a platform or middleware for smart u-objects/environments/services

Ubicomp/Percomp/AmI/CPS/IoT Computing of Smart u-Thing

Smart u-Things

Plant talks, IEEE Spectrum, May/09

Smart Grid, IEEE Spectrum, July/09

Basic features of smart u-things- Sense & effect for perception and interaction- Networking & naming for interconnection and communication- Computation & storage for information processing and memory

Smart u-Things – MIT Lover’s Cups

Explore the idea of sharing feelings of drinking between two people in different places

Two cups are connected with each other, and glow when your lover is drinking.

Smart u-Things – Smart Slippers & Nike

Smart u-Things – GrowCaps, etc.

Smart u-Things – Smart Home

Videos About Smart Object and System

Smart Trashbox – 勝手に入るゴミ箱

Smart Systems – Toyota’s Smart Mobility Society

Sony SmartWatch2 , Samsung Galaxy Gear ,

Galaxy Gear Concept 2013

Smart Shoe – Nike smart trainers

Smart Pill Bottle , Smart Pill Box , Smart Pill

Smart Toy – Smartphone + Toy

Smart Space/Environment

Smart Space/Environment Computerized active one

- Physical environments but digitally enhanced/integrated

- Awareness of contexts: users, ambience, resources, etc. - Context-Aware responses or services with certain intelligence- Where are users? Supervisors & computing elements!!

Sensors

Actuators

Space

Environment

Object

Artifact

Plant

Body

(User)

Ubi/Pervasive

Information

- Acquisition

- Archive

- Analysis

- Awareness

- Application

- …

Multimedia

Information

Convenience

Comfort Service

(Context)

Loop

Smart Space/Environment

Status/ attend

Status/ absent

RFID

Reader

Room １ Room ２

Room ３

Motion‘Organ’

Memory‘Organ’

Percep.‘Organ’

Brain

BathToilet

StudySleep

Entrance Kichen

Living

Dinning

Action‘Organ’

Think‘Organ’

Control‘Organ’

Smart Physical Objects Management

Things Mapping - toy- game- key- …

Thing Reminder- umbrella- book- pencil- …

Users’ IndoorLocation Sharing

ユーザの屋内位置情報の共有

ROOM

サーバー　　　

ROOM

アクセスポイント１

データベース

位置推定行動推定

　アクセスポイント２ログデータ

ログデータ

電波強度加速度データ

EV２

EV１

User Location-Aware Music Player

Music data sent to speaker(s) near a user’s location

Sound-Aware Smart Space

Status/ attend

Status/ absent

RFID

Reader

Room １ Room ２

Room ３

Sound-aware Reminder- Ambient sound capture- Ambient analysis- Advice judgment- Advice presentation- …

Sound Disturbance

- home

- classroom- theater- …

Robots forMobile Sensing & Object Finding

R-Reader

R-Controller (PDA)

R-Machine

Ultrasonic Sensor

Motor

Camera

R-Reader

R-Controller (PDA)

R-Machine

Ultrasonic Sensor Motor

Lifelog – Personal Experience/DB

LabLog – Info Sensing & Management

Sun SPOTS

Wireless networks

& Sensors

Java program

SSG

SSGOn Robot

(Time)

24℃

(Raw) SSG (Door)

(Time)

(Raw)

25℃

SSG (Robot)

HomeLog – Kid’s Activity Summarization

ID Tag and Sensor for User2007/11/10

12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11

1 Sensor A(study)

Sensed Activity Period

4

3 Sensor F(study)Sensor G(study)

2 Sensor E(study)

5 Sensor H(study)

Sensed Activity Change Time

Sensor for User12 1 2 3 4 5 6 7 8 9 10 11 12 1 432 5 6 7 8 9 10 11

2007/12/11

部屋入口で認識した子どものRFIDタグ

机の椅子のタッチセンサー

教科書のRFIDタグ

机付近で認識した子どもののRFIDタグ

漫画本のRFIDタグ

ID Tag and Sensor for User2007/11/10

12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11

1 Hosei CIS

Kid’s Name12 1 2 3 4 5 6 7 8 9 10 11 12 1 432 5 6 7 8 9 10 11

2007/12/11

Sleep Game Study Sleep

Videos About Smart Things and Services

Green Wheel - Smart Mobility & Ubiquitous Computing- 10 scenarios on Smart Ubiquitous Services

Display Glass - Smart Ubiquitous Displays- Displays in wall, surface, mirror, veneer, etc.

Smart Home - Home Energy Management System (HEMS)(community) - Smart energy supply, use, control, etc.

Japan Smart City Pilot Projects

http://jscp.nepc.or.jp/

Japan Smart Community Alliance

Established on April 6, 2010 to promote Smart Community through government-private sector cooperation

As of Nov. 2012, 397companies and organizations (including universities)

Cooperation across companies and organizations

https://www.smart-japan.org/default.aspx

Smart City Project

On Sep. 17th, 2009, Smart City Planning Inc. was established as Joint Venture Company of leading companies for the management of Smart City Project.

As of Sep. 2012, 22leading companies joined the project.

http://www.smartcity-planning.co.jp/

Nikkei Smart City Consortium

http://expo.nikkeibp.co.jp/scw/2011/exhibitors/english/http://expo.nikkeibp.co.jp/scw/2012/exhibitors/english/

http://bizgate.nikkei.co.jp/smartcity/

Numbers of Smart City Projects

http://itpro.nikkeibp.co.jp/article/Keyword/20120906/421044/ As of September 12, 2012

China

JapanIndia

Africa

Far-East

NorthAmerica

SouthAmerica

PacificOcean

Europe

Toyota Smart Center in 20xx

Smart u-Things – Real World Challenges

Smart u-Things are emerging- Many scenarios: Weiser’s Sal, AmI’s Maria/Dimitrios/Carmen, Aura’s Jane/Fred, …- Lots of research and many various prototypes Research Booming comes- But rare practical ones Application Booming is to come real world complexity

Ideal Smart u-Things expected and related challenges Able to act adaptively and automatically according to

1. Surrounding Situations Challenge 1: Situation Approximation

2. Users’ Needs Challenge 2: Knowing Users’ Needs

3. Things’ Relations Challenge 3: Complex Things’ Relations

4. Common Knowledge Challenge 4: Knowledge Management/Growth

5. Own Goal, Role, etc. Challenge 5: Self Awareness

6. Error and Exception Challenge 6: Looped Self Adjustment

7. Safety & Satisfaction Challenge 7: UbiSafe

Above challenges from real world intrinsic characteristics- RW == physical + social + natural + … uncertain, unpredictable, changing, …- RW computing: complicated/abstruse phil., social, ethical & other implications- Understanding real world (RW) diversity and complexity Extremely Hard !!- Novel cyber dimensions newly added in physical-digital combined u-things

Challenge 1 – Situation Approximation

Context- Information characterizing the situation of an entity (Dey) or 5Ws (Abowd)

- Whole contexts are a collection of various values from sensors or other sources

C(t) = {Ci(Bi, L, th), i∈I(t), th∈[t-h, t]}

Situation (Activity)- Relatively compact, more semantic, directly used for judgment/decision

S(t) =φ[C(t)] = {Sj(Wj, t), j∈J(t)}Designing a Context-aware System to Detect Dangerous Situations in School Routes for Kids Outdoor Safety Care, LNCS, Proc. of EUC'05.

C(t), S(t) approximations to a real environment- Are the contexts C(t) sufficient and precise enough to characterize a real environment?

- How correctly be S(t) determined using available but incomplete & uncertain contexts?

- What are consequences of situation judgment errors to context-aware/situated u-things?

Nyquist Sampling Theorem & Shannon Entropy/R(D) Theory Theory for context/situation?

Keep incompletion, uncertainty and misjudgment as basic design assumptions !

Challenge 2 – Knowing Users’ Needs

Interactive Mechanism- A request-response dialog process betw. users (activator) & computer (passive)- 3 user limits: too small/many/complex computers to be visible/interact-able/manageable

Proactive and Autonomic Mechanisms- Take actions proactively by anticipating users’ needs (Tennenhouse, Intel, 2000)

- Manage themselves under human supervisions/needs (Paul Horn, IBM, 2001)"Proactive Control of Group Revision Assistance Management Using P2P Technology" (Takata & Ma, ISCIT04)

Journal of Autonomic and Trusted Computing (JoATC, EiC, Laurence T. Yang and Jianhua Ma), ASP, USA

1st Int’l Wksp on Trusted and Autonomic Ubiquitous & Embedded Systems (TAUES-05, Japan, Dec. 2005)

2nd International Workshop on Trusted and Autonomic Computing Systems (TACS-06, Austria, Apr. 2006)

3rd Int’l Conf. on Autonomic & Trusted Computing (ATC06, China) ATC07 (HK) ATC08 (NW) ATC09 (Australia)

Users’ Needs- Multi aspects and subtle affection by many factors

N(t) = {Nu,v[Pu,v(t), S(t), S(t-Δ)], u∈U(t), v∈V(t)}

Hardness of knowing users’ true needs- How much can be known correctly and promptly for users’ true needs in changing situations?

- “Know you and know your face, but don’t know your mind” - Chinese saw

Challenge 3 – Complex Things’ Relations

Complex Relations among u-Things- “Everything will be connected to everything else” - by R. Lucky, 1999

- “what are the consequences of any given action (including simply walking into a room)”

- by M. Weiser, 1999

Cyber dimensions, except spatial/temporal/other conventional dimensions Complex dynamic relations among users, u-things, and other things

R(t) = {Rq(Tq, Uq, t), Uq∈U(t), q∈Q(t)}

- How to define, find, describe and use complex relations necessary for u-things’ systems?Smart Hyperspace: a set of interconnected smart spaces with situational-spatial-temporal relations

(“Modeling Interface with a Multimedia Hyperworld”, HIS’96, “Smart Hyperspaces and Project Ubikids”, USW05)

users

non u-thingsu-things

Challenge 4 – Knowledge Mgmt/Growth

Why common knowledge needed?- Knowledge is the base of analysis, reasoning, anticipation and judgment (Ref AI)

- Smart u-things are in the physical environments and serve people’s daily life and work- Smart u-things need some common knowledge about physical world, human society, etc.

K(t) = {Km(Fm, th), m∈M(t), th∈[t0, t]}

Issues are- What knowledge is necessary for smart u-things? - What knowledge should be initially set? - What knowledge can be added later on?- What knowledge may be self-learned during uses?- How knowledge is used for rich and varied real situations?

Challenge - how to abstract, learn, use complex knowledge about human & world

How about DAI, swarm, softcomp, agent, semantics, etc.?

Challenge 5 – Self-Aware u-Things

Why smart u-things should be self-aware- Physical, not virtual, things with attached/embedded/blended (AEB) computers/etc.- AEB devices are function parts or components of the real physical things- AEB is for enhancing original functions of real things following common rules- Smart u-things should be aware their roles and function without against the rules

A(t) = {Az(Gz, t), z∈Z(t)}

Context-awareness vs self-awareness- Context-awareness: knowing others- Self-awareness: knowing oneself

“Knowing others is wisdom, knowing yourself is enlightenment” - Tao Tzu

- What this exactly means? How to equip self-awareness to u-things?

Extreme Challenging in this research“we seem so ill equipped to understand ourselves”, The Society of Mind, Minsky, 1986

Workshop on Self-Aware Computer Systems - Chaudhri & McCarthy, SRI/DARPA, 2004

Challenge 6 – Looped Self-Adjustment

Decisions cannot be correct always in 100% Imprecise decision & exception are common

Loop mechanism is indispensable !

Challenge - adaptively correct errors & make adjustments How is this related to automatic control theory?

D(t) =ψ[S(t), N(t), R(t), K(t), A(t), D(t-Δ), E(t)]

ψS(t)

N(t)

R(t)

D(t)

K(t) A(t)

actions

D(t-Δ)

E(t)χ

Error(t)

Exception(t)

Challenge 7 – Ubisafe (Ubiquitous Safety)

Unobtrusive AEB devices in real objects and spaces Physical unsafe factors

Diverse Users with different ages/backgrounds/demandsHuman unsafe factors

Desired States of ubicomp environment

Calm Technology by Weiser and Brown

UbiSafe Ubiquitous Safety & Satisfaction

Basic Question

How to build a computing environment in which all people can fully enjoy the u-services under complex situations without worrying or thinking the safety problem any more?

Ubiquitous Comp Ubiquitous Services Ubiquitous Dangers UbiSafe

Ubicomp: Physical-Cyber Loop & Consequence

Cyber EnvironmentsComputers, Components

Software, Service, …

UserObject

Physical Input

Physical Output

Trust Safety

measure, model, monitor, management, analysis, evolution, …

Consequence of misbehavior or

unexpected outputfrom trustor/trustee?

Sensors

Actuators

Automatic PhyCybPhy

Loop

User Protection- Tech. + Social

Safety Guarantee- Sec. + Rel. + …

LossUnsafeDanger

B-2 Spirit Crash

On 23 February 2008, a B-2 crashed back onto a runway shortly after takeoff from Andersen Air Force Base in Guam. The aircraft was completely destroyed, a total loss estimated at US$1.4 billion.

The findings of the subsequent investigation stated that the B-2 crashed after "heavy, lashing rains" caused water to enter skin-flush air-data sensors, which feed angle of attack and yaw data to the computerized flight-control system. The water distorted preflight readings in three of the plane's 24 sensors, causing flight-control system to send an erroneous correction to the B-2 on takeoff.Because of the faulty readings, the flight computers determined inaccurate airspeed readings and incorrectly indicated a downward angle for the aircraft, which contributed to an early rotation and an un-commanded 30-degree pitch up and left yaw, resulting in the stall.

A Boy Killed by an Auto-door

2004/3/27、Asahi Newspaper

2004/4/20、

日本経済新聞2004.3.26 : a boy died due to a rotation door

SensingRanges

From Prof. Z. Cheng

Elevator Accidents

朝日新聞2006年6月8日

朝日新聞・産経新聞2006/6/17

2006.06.03, a high school student dead when he took a bicycle into a

elevator, which started to move without completely closing the door.

If pushing the open button just after the door

has been closed, the elevator may start to go

up with the door openedFrom Prof. Z. Cheng

Novel Features and Unsafe Factors

Unobtrusive AEB in real objects and environments Physical characteristic oriented unsafe factors

- Limited computation, open/changing/worse working conditions

- Used consciously or unconsciously

Diverse users with different backgrounds/demandsAll people from baby to elder, normal to disabled, ……

Human characteristic oriented unsafe factors

- No comp knowledge, no expected usage, no awareness of

dangers, no ability of handling, ……

- Safe/unsafe is not absolute, relative, up to individual/situation

Life-like systems, smart u-things from small to large scales Passive Interactive Active Life-like

life-like system characteristic oriented unsafe factors

- Imprecise sensing data, insufficient context, complex relation, ……

Ubiquitous Comp/Dev./Net Ubiquitous Risks/Dangers

The Internet … a Network of Networks that consists of millions of

private, public, academic, business, and government networks, of

local to global scope. - From Wikipedia

The Internet Internet of Computers (IoC)Leonard Kleinrock Lawrence Roberts Steve CrockerJon Postel Vinton Cerf Robert Kahn

What are the Internet?

Originated from the ARPANET around 1970

Available from 1980, got popular from 1990.

Key components

- Hardware: Routers connecting networks

- Software: TCP/IP protocol suite, IPv4 IPv6

- Addressing: 2**32 (IPv4) 2**128 (IPv6)

- Naming: DNS symbolic names

A Global Net

Connecting All

Computers

physical net

router

The World Wide Web, abbreviated as WWW or the Web, is a system

of interlinked documents accessed via the Internet. - From Wikipedia

The Web was originated from Tim Berners-Lee around 1990.

The Web, like Email, is one of the services that runs on the Internet.

Key components

- Uniform Resource Locator (URL) & Uniform Resource Identifier (URI)

- HyperText Markup Language (HTML)

- Hypertext Transfer Protocol (HTTP)

- Web server and web browser (client)

What are the Web?

Tim Berners-Lee Ted Nelson Hypertext Mark Andreesen

Netscape

1st Web Server

1st Web

Graphical

Browser

The Web Internet of Documents (IoD)

How about Social Media/Web2.0?

SM/Web2.0 Internet of People (IoP)

How about Cloud Computing?

Software

Cloud Internet of Resources (IoR)

Thing - An object, an entity, an idea, or a quality perceived, known, or

thought to have its own existence, … (dictionary)

Object – A tangible/visible thing; a person or thing seen as a focus or

target for feelings, thought, etc.; a purpose/objective; ... (dictionary)

Everyday Things/Objects – used in human daily lives

Inner Things – mind, directly insensible things, ...

Physical things, digital things, real/virtual things, …

What are Things?

Various Things! Many IoX!

IoCIoD

IoPIoR

What Kinds of Things in IoT?

u-Things in Physical/Real World

Physical World

u-Thing

Usual Activity

Sensor/M/NEMS, Comps & Per. Nets

Physical Space

UC, ID, Context, Emb. Sys., Cloud, …

u-Things: on the physical space in physical-digital form

e-Thing

e-Activity

Cyber World

Computers & Networks/Internet

Cyber Space

WbS, SmW, Grid, P2P, XaaS, Cloud

IoT u-Things in Physical World

IoT Not Anything Special Objects

The term “Internet of things” was first coined by the former Auto-ID

Center, founded in 1999, based at the time at MIT. (Kevin Ashton

and David L. Brock) RFID-based EPC Only term!

(Sean Dodson, 2003) "IoT" can be expressed as the building of a

global infrastructure for RFID tags.

@ You could think of it as a wireless layer on top of the internet

where millions of things from razor blades to euro banknotes to car

tyres are constantly being tracked and accounted for.

@ A network … is for computers to identify "any object anywhere in

the world instantly".

@ Put a tag - a microchip with an antenna - on a can of Coke or a

car axle, and suddenly a computer can 'see' it.

The Origination of IoT

Formal Introduction of IoT“We are heading into a new era of ubiquity, where the users of the Internet will be counted in billions, and where humans may become the minority as generators and receivers of traffic. Changes brought about by the Internet will be dwarfed by those prompted by the networking of everyday objects” – UN report, 2005

The Internet of things, also known as the Internet of objects, refers to

the networked interconnection of everyday objects. It is described as a

self-configuring wireless network of sensors whose purpose would be

to interconnect all things.

From Wikipedia

(IoT is) a network of interconnected computers to a network of

interconnected objects, from books to cars, from electrical appliances

to food, and thus create an ‘Internet of things’. These objects will

sometimes have their own Internet Protocol addresses, be embedded

in complex systems and use sensors to obtain information from their

environment and/or use actuators to interact with it.

From Internet of Things — An action plan for Europe (2009)

“Internet of Things” to refer to the general idea of things, especially everyday objects, that are readable, recognizable, locatable, addressable, and/or controllable via the Internet—whether via RFID, wireless LAN, wide-area network, or other means.

From SRI Consulting Business Intelligence

Other Definitions of IoT

Examples of Things in IoT

Many physical things are connected, can talk 全てのモノが繋がる世界 対話（通信）できる

Ubiquitous NetworksObject-To-Object (O2O), Thing-To-Thing (T2T)

Ubiquitous Connection of All Things

Internet of Everything

Companies/Societies

Individuals

SEA-net

Internet

/WWW

Cyber space

Physical Space

Social Space

Cyber networks

Physical networks

Social networks

Big Data

Big Data

Big Data

IoP

IoD

IoT

IoC

IoT Platform and Applications

From CERP-IOT

Videos About IoT

IOT will change everything from Cisco (30 videos)

Internet of Things by Dr. John Barrett at TEDxCIT

Harnessing the Internet of Things by Steve Lucas

Freescale's IoT Applications

An Introduction of Internet of Things from IBM