bridging the gap between humans and the physical world
DESCRIPTION
Bridging the Gap between Humans and the Physical World. A Step Toward Reducing Energy Consumption and Increasing Comfort. Dr. Fred Jiang / 姜小凡 In collaboration with Mike Liang, Jeff Hsu, Caiquan Liu, Jie Liu, and Feng Zhao Mobile and Sensing Systems Group Microsoft Research Asia - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/1.jpg)
Bridging the Gap between Humans and the Physical World
A Step Toward Reducing Energy Consumption and Increasing Comfort
Dr. Fred Jiang / 姜小凡In collaboration with Mike Liang, Jeff Hsu, Caiquan Liu, Jie Liu, and Feng Zhao
Mobile and Sensing Systems GroupMicrosoft Research Asia
National Taiwan University / 2011-4-1
![Page 2: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/2.jpg)
MotivationWe interact with our environment in very limited ways
Simple time series data collectionActuation is limited because of lack of uniform interfacePhysical objects cannot proactively reach out to us
Result: waste in energy and suboptimal comfort
Example 1: environment (physical objects) aware of who is acting on it, collect usage / energy data in real-time (personal energy footprint)Example 2: alert me if something is not normalExample 3: objects interacting with objects
![Page 3: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/3.jpg)
Related Work
Ambient intelligenceRoomwareSmart objectsLocalizationX10Beaver monitoringUbicomp literature
![Page 4: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/4.jpg)
We Propose
Bi-direction interaction and information flow between humans and physical objectsFor humans to have real-time finer-grained knowledge and control over physical objectsEnvironment to be aware of humans, and make intelligent decisions, proactively and reactivelyBridging the gap between humans and their environmentOur vision of IoT (it’s not about simply putting a networking interface on everything)
![Page 5: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/5.jpg)
Three Primary Components
Precise indoor localization primitiveReliable IPv6 based networkingUniform interface for abstracting services of physical objects
![Page 6: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/6.jpg)
Virtual Overlay
Entrance to 4th floor
name: fridgestates: on : true temp : 5 door : opencommands: turn_on() turn_off()events: notifyDorOpn notifyPwrUsgchildren: []
![Page 7: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/7.jpg)
Indoor Localization
Previous workWhat we needChallenges
![Page 8: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/8.jpg)
Localization Options
Wifi radioBT radioInertial sensorOptical (Kinect)RFIDMagnetic (PKE)
![Page 9: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/9.jpg)
Magneto-Induction
Slide courtesy of Andrew Markham (http://www.comlab.ox.ac.uk/people/andrew.markham/)
![Page 10: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/10.jpg)
Magneto-Induction
Slide courtesy of Andrew Markham (http://www.comlab.ox.ac.uk/people/andrew.markham/)
![Page 11: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/11.jpg)
Magneto-Induction
Slide courtesy of Andrew Markham (http://www.comlab.ox.ac.uk/people/andrew.markham/)
![Page 12: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/12.jpg)
Challenges for Indoor
4m x 1m antenna -> 5m^3 trackingWe don’t have that much space!We use a 8cm x 1.5cm transmitter antennaAnd 3D coil IC (2cm x 1cm) as receiving antenna
Power supply issuesOutdoor vs indoorInterference issuesPhone magnetometer (digital compass)
![Page 13: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/13.jpg)
Microsoft LivePulse
![Page 14: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/14.jpg)
Mains Power
AD/DC/DC Converstion90-240VAC -> 12VDC -> 5VDC -> 3.3VDC
Mechanical Relay
Hall-effectSensor
MI MOSFET Driver
125kHz MI Transmitting Antenna
CC430 SoCEnergy Meter
IC
900MHzPCB
Antenna
![Page 15: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/15.jpg)
Microsoft LivePulse• Precise indoor
localization
Range: ~ 3m omnidirectional
~10cm dropoff zone
Consistent over timeRobust against interference and multipathBlock by metal
• Wireless energy monitoring
![Page 16: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/16.jpg)
Microsoft LiveLink
3D MI Receiving Antenna
Wakeup
![Page 17: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/17.jpg)
Networking
RequirementsReliableGlobally addressableScalableLow power
![Page 18: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/18.jpg)
Radio Consideration
• 802.15.4– 2.4GHz– 863 MHz to 928MHz– 100-150m range– US
• 802.15.4c (c for china)– 779-787 MHz– 250m range
• Core module choices– 60USD Epic core– 20USD SuRF core
• Energy
![Page 19: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/19.jpg)
Networking Overview
One-hopIPv6 communication over 802.15.4c
JSON / RESTful services over Internet
Repository of virtual objects
![Page 20: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/20.jpg)
Virtual Object Abstraction Layerapartment
bedroom office bathroom hallway
desk desktop room portable lamp computer lamp heater
coffee laptop piano pinballmachine computer keyboard machine
LCD computer monitor tower
• Hierarchical• Event based
![Page 21: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/21.jpg)
Object Representation
StatesE.g. On/off
CommandsE.g. turn on / read power / set temp / count people
EventsE.g. door opened / light-off / over-threshold
ChildrenPtr to children objects
![Page 22: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/22.jpg)
name: officestates: lightOn? : True doorOpen? : False roomOccupied? : True roomTemp? : 30C roomEnergy? : 200W commands: toggleLight numOccRoom lastTimeDoorOpened
events:
notifyDoorOpen notifyNewOccupant notifyHourlyEnergyUsechildren [ipv6]: coffeeMachine deskLamp laptopComputer desktopComputer pianoKeyboard roomLamp pinballMachine portableHeater
1. DISCOVERY_REQ
2. DISCOVERY_REPLY
to http://<ipv6OfOffice>
3. EVENT_SUBSCRIPTION [notifyNewOccupant]
4. NOTIFICATION
“let me know if someone enters room”
“someone just entered your room”
![Page 23: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/23.jpg)
Open Source Standard
JSON is the object encodingSOAP / WSDL / Zigbee SE Profile – Nooo
REST/HTTP is the interchange schemeHTTP_GET / HTTP_GET / HTTP_RESPONSE
IPv6Interoperate with existing IP devices without understanding additional protocols
![Page 24: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/24.jpg)
Applications
Personal energy footprintSocial networkingHealth care
![Page 25: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/25.jpg)
Related EffortsSmart meters (via utilities)
Enables time-of-day billingGoogle PowerMeter
Visualization of whole-house usageDefining a API for energy data
Microsoft HohmComplimentary work (help learn signatures)House-level / NILM / can benefit from DB of appliance signatures
In the right direction, but not enoughOnly at the whole-house levelLack of actuation / control
Source: http://www.microsoft-hohm.com
Source: X. Jiang Dissertation
![Page 26: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/26.jpg)
Individualized Energy Feedback and Control
• Real-time energy apportionment– Individualized energy
accounting / billing– Energy map / trail of occupants
• Alerts and abnormally detection– Via cellphone– Remote alerts
• Control and actuation– Automated control of devices– Scheduling
![Page 27: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/27.jpg)
An Energy-Centric Ecosystem – Social Network of Energy
Social network based on real-time individual energy usagesWindows 7 GadgetWindows Mobile 7 AppFacebook appReward systems
Building managers have fine-grain control and view
![Page 28: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/28.jpg)
mobile application
![Page 29: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/29.jpg)
web application - synergy
![Page 30: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/30.jpg)
Social Network of Energy
LocalizationLocation Status updates, kind of like Foursquare Checkins.• “Mike is now in the coffee room.”
ApportionmentNow that we have localization primitive, we can specifically attribute the use of public energy to individual users. Very useful.• “Attribute the last 3 minutes of coffee energy usage to Jeff.”Add this to personal monitoring of private appliances, and we now have a complete carbon footprint.
![Page 31: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/31.jpg)
web application dashboard - map
![Page 32: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/32.jpg)
Social Network of Energy
Map of User’s owned areas…and where energy usage is taking place the mostSocial Interaction
Compare Energy Usage of Specific Appliances, or aggregateGames• User VS. User [One-Month Challenge, percent
improvement?]• Team VS. Team
![Page 33: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/33.jpg)
web application dashboard - energy
![Page 34: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/34.jpg)
Social Network of Energy
Add Appliances [Home and Work]Energy Usage Trends
Day to Day Comparisons in a tableEnergy Usage Graphs [Compare different devices]
Email Summary of Energy Usage to User [each day, or week, or month]
![Page 35: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/35.jpg)
web application – user profile
![Page 36: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/36.jpg)
Social Networking “Wine Party” App
Frequently, it is very difficult to meet people you click with at social gatherings.Ex. You are at a company party. You are bored. You pull out your mobile phone. A map of the room you are in shows up, and there are markers on the map. Click on a marker, and you can see that person’s interests, details, hobbies, etc.See twitter feed, facebook profile, etc.When someone who match your interest pass by, your phone vibrates to alert you.
![Page 37: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/37.jpg)
Health Care
Room is aware of status of elderly occupants 1. phone insert trigger / event2. object representing the room 3. room send a signal to alert
![Page 38: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/38.jpg)
Conclusion• Bi-direction interaction
between humans and physical objects
• For humans to have real-time fine-grained knowledge and control over physical objects
• Environment to be aware of humans, and make intelligent decisions, proactively and reactively
• Precise indoor localization primitive
• Reliable IPv6 based networking
• Uniform virtual representation of physical objects
![Page 39: Bridging the Gap between Humans and the Physical World](https://reader036.vdocuments.mx/reader036/viewer/2022062400/5681694c550346895de0e7d4/html5/thumbnails/39.jpg)
Thank you
Questions?
Feel free to contact me at [email protected] http://research.microsoft.com/people/fxjiang