(slides) ubiremote: framework for remotely controlling networked appliances through interaction with...
DESCRIPTION
Kiyokawa, K., Yamamoto, S., Shibata, N., Yasumoto, K., Ito, M.: UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space, Proc. of ACM Multimedia Systems 2010 (MMSys2010), pp.271-280, DOI:10.1145/1730836.1730870 (Feb. 2010). http://ito-lab.naist.jp/mediawiki/images/6/60/100223mmsys.pdf In this paper, we propose a framework named “UbiREMOTE”for controlling information appliances connected to a home network with a unified and intuitive user interface from a remote place. The UbiREMOTE framework provides users with a way to control appliances in a home through a virtual space drawn on a mobile terminal screen which reflects the latest conditions of the real appliances and the rooms in the home. With UbiREMOTE, a user controls appliances by (1) moving to the front of an appliance, (2) choosing the appliance to control and (3) pushing buttons on the virtual remote controller which imitates the real remote controller for the appliance or the real console. In this paper, we propose a method to improve the drawing speed of 3D virtual space on mobile terminals and a method for automatically reflecting condition changes of the real space in the virtual space. We implemented the methods and evaluated the performance. The results showed that the proposed methods can be practically used on small mobile terminals.TRANSCRIPT
![Page 1: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/1.jpg)
Kohta Kiyokawa, Shinya Yamamoto †, Naoki Shibata ‡,
Keiichi Yasumoto and Minoru Ito
Nara Institute of Science and Technology, † Tokyo University of Science Yamaguchi, ‡ Shiga University
February 23, MMSys2010
![Page 2: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/2.jpg)
• Propose framework “UbiREMOTE” for remotely controlling appliances with 3D virtual space I/F– Intuitive operation for anyone – Intuitive monitoring of target spaces where appliances exist– Smooth operation on mobile terminals
• Implement and evaluate the framework– Runs smoothly on mobile terminals– Has better usability than text-based
remote controller
Overview
2
![Page 3: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/3.jpg)
Outline
3
1. Background and Goals
2. UbiREMOTE Framework
3. Proposed Techniques
4. Evaluation
5. Conclusions and Future Works
![Page 4: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/4.jpg)
Background Several frameworks for controlling appliances
through network are available UPnP, DLNA, OSGi, ECHONET, etc
These frameworks enable remote control of appliances Examples
warm/cool the living room before arriving home check if he/she switched off an appliance (e.g.,
heater) and turn it off remotely
4
![Page 5: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/5.jpg)
5
• Several existing services– E.g., “JIG multi remocon” by jig.jp– Operate multiple appliances
through a cell phone– User I/F is based on figures/icons
and texts
• Drawbacks– Usability rapidly reduces when operating many similar
kind of appliances at different places (rooms)• Difficult to find target appliance from many
figures/texts on screen• UI is different from dedicated remote controllers
Existing services and drawbacks
© jig.jp co.ltd http://remocon.jig.jp/
![Page 6: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/6.jpg)
Our goals
• Remote controller service that allows users to remotely control networked appliances in their home by– Intuitive operation from a remote place– Intuitive monitoring of target spaces
• Current states of appliances (on/off, …etc)• Conditions (temperature, humidity, …etc) of rooms
– Smooth operation by mobile terminals
6
![Page 7: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/7.jpg)
Outline
7
1. Background and Goals
2. UbiREMOTE Framework
3. Proposed Techniques
4. Evaluation
5. Conclusions and Future Works
![Page 8: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/8.jpg)
Smooth operation on mobile terminals
Intuitive monitoring of target spaces
8
Challenges and ApproachesIntuitive operation
![Page 9: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/9.jpg)
Reproduce building with many networked appliances (target space) in 3D virtual space Imitate the real remote controller for each appliance by
using control window Show states of
appliances by visual effects
9
Challenges and ApproachesIntuitive operation
![Page 10: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/10.jpg)
There are too many 3D objects in buildingPoor 3D capability on mobile deviceWe introduce lightweight 3D rendering
mechanism
Smooth operation on mobile terminals
Reproduce building with many networked appliances (target space) in 3D virtual space
10
Challenges and ApproachesIntuitive operation
![Page 11: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/11.jpg)
Gather data by sensors deployed in each roomDetection/reflection of appliance positions
Lightweight 3D rendering mechanism
Smooth operation on mobile terminals
Intuitive monitoring of target spaces
Reproduce building with many networked appliances (target space) in 3D virtual space
11
Challenges and ApproachesIntuitive operation
![Page 12: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/12.jpg)
12
Intuitive UI based on 3D virtual space
Appliance (lamp) in real space is turned on
Monitor real space conditionsMove viewpoint
25.0℃
Select target appliance (lamp), and turn it on
![Page 13: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/13.jpg)
Target environment
13
Home server
Remote controller terminal
Sensors
Control
•Reflect information in virtual space
•Allow intuitive operations for appliances
Appliances
User
Wireless communication
3D Virtual Space (UI)
Operation
Synchronization
Target Real Space
Communication
![Page 14: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/14.jpg)
Outline
14
1. Background and Goals
2. UbiREMOTE Framework
3. Proposed Techniques– Lightweight (LW) 3D Rendering– Reflecting Real Space Info into Virtual Space
4. Evaluation
5. Conclusions and Future Works
![Page 15: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/15.jpg)
Why we need LW 3D rendering?• Motivation
– Render 3D objects on mobile terminals– However, there are so many 3D objects in home, and
mobile device’s 3D rendering capability is not sufficient
• Requirements– Smooth viewpoint movement– Quick response when operating an appliance– Sufficient quality of rendered image
15
![Page 16: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/16.jpg)
16
Straightforward approach• Thin Client Method
– A powerful server renders 3D image, captures a still image from a specified viewpoint, and sends the image to a client
• Drawback– Large operation delay (response time) is produced
4. Drawing image2. Renderingand capturingstill image
3. Sending captured image
1. Operation
Server Client
Some time lag produced
![Page 17: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/17.jpg)
17
Tradeoff for 3D rendering
short
Client renders all objects
Client renders all objects
Thin C
lient Method
Thin C
lient Method
Response time
3D renderingheavy light
long
• Our approach– Propose compromise between two methods– Utilize a fact that users are likely to operate
appliances near viewpoint
![Page 18: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/18.jpg)
18
• Surround viewpoint with a cube, where 6 still images are captured for virtual space and put on its faces
• Client renders only objects inside the cube
Proposed LW 3D rendering technique
These objects are drawn as background images (rendered by server)
These objects inside the cube are rendered by client
Viewpoint
VP
Cube
![Page 19: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/19.jpg)
Updating cube when viewpoint moves
Set a sphere of radius R with the same center as cube
While viewpoint (VP) stays within the sphere
No update of cube When VP goes out of sphere
Prefetch BG images for
new cube, and replace
current cube by new oneR
BG image
![Page 20: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/20.jpg)
20
Message sequence for cube update
viewpoint movement
Viewpointmovement
ServerClient
New center pos for cube
BG images for new cube
User going out of sphere
Rendering BG images
Updating cube
Updating cube
User going out of sphere RendereingBG images
![Page 21: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/21.jpg)
Outline
21
1. Background and Goals
2. UbiREMOTE Framework
3. Proposed Techniques– Lightweight (LW) 3D Rendering– Reflecting Real Space Info into Virtual Space
4. Evaluation
5. Conclusions and Future Works
![Page 22: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/22.jpg)
22
• Difference between virtual and real spaces causes the following situations
• A user remotely operated a heater in living room (on the screen), but it actually had been moved to children room
• Approach to avoid above situations– Show some real space information in virtual
space• Temperature, humidity, etc• Position of appliances / furniture
Why need to reflect real space info?
![Page 23: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/23.jpg)
23
Reflecting state change in real space into virtual space
Home server
Wireless communication
Communication
Virtual space
living roomTemp:25CHum:60 %
Sensors at each room detect temperature change, movement of appliance/ furniture, etc
Sensors at each room detect temperature change, movement of appliance/ furniture, etc
Deploy sensors at each room, and update information when there is a big change in sensor values
![Page 24: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/24.jpg)
Detection/reflection of appliance positions
• Attach to each appliance a sensor node equipped with– Acceleration sensors for detecting movement– ZigBee communication capability for estimation of
new position
• Use existing ZigBee based localization techniques to estimate new position of appliance
24
![Page 25: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/25.jpg)
Outline
25
1. Background and Goals
2. UbiREMOTE Framework
3. Proposed Techniques
4. Evaluation
5. Conclusions and Future Works
![Page 26: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/26.jpg)
26
Experimental evaluation
• Implemented UbiREMOTE prototype on tablet PC • Evaluated performance
– Smoothness : Frame rates and response time– Usability : Questionnaire for User satisfaction in remote operation
TVElectric
Fan
Ceiling Light
Stand Light
CD Player
Floor plan and position of viewpoint
Controller of viewpoint
Controller foran appliance
![Page 27: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/27.jpg)
27
Exp1 : Rendering speed
will achieve practical rendering speed on mobile terminal
Environment : Tablet PC
# objects (# objects inside cube)
# polygonsFrame rate (fps)
Render all objects
20 200000 1
Proposed 20 (1) 108 276
Proposed 20 (2) 2100 262
Proposed 20 (3) 4100 250
OS: Windows XP tablet PC editionCPU: Pentium M 1.2GHzMemory: 512MBGraphics Chip: Mobile Intel ®915GM
![Page 28: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/28.jpg)
Exp2 : Response time (system configuration)
28
Mobile Terminal( Tablet PC )
Home Server
UPnPTCP/IP
3G cellular
network /WiFi
Electric Fan
Light
Heater
infrared Remote Controller
![Page 29: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/29.jpg)
Exp2 : Response time (Result)
Reasonable response time for practical use 29
UbiREMOTE
appliances (a)WLAN (b)CellularElectric
fanON 0.94 3.22
OFF 0.92 3.14Light ON 0.73 2.98
OFF 0.61 3.13Heater ON 0.93 3.18
OFF 0.84 3.30average 0.828 3.141
![Page 30: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/30.jpg)
30
Exp3:Usability evaluation by questionnaire
• 6 testees (graduate students)– operated 4 appliances by Remote controller that has
text-based interface and UbiREMOTE, – Return questionnaires.
ComparedText-based
remote controller
![Page 31: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/31.jpg)
• It is easy to monitor appliances/conditions with UbiREMOTE31
サンプル数:24
サンプル数:24
■ Very Good■ Good■ Not good■ Bad
Q1: Is operation comfortable?
Q2:Can monitor conditions of appliances?
Text-based
Exp3 : Result of usability evaluation by questionnaire(1)
6 testees×4 appliances= 24samples
![Page 32: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/32.jpg)
32
• Q3 : Did you feel something wrong in UbiREMOTE?
Exp3 : Result of usability evaluation by questionnaire(2)
Proposed LW 3D rendering does not spoiluser feeling and comfort in operation
![Page 33: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/33.jpg)
3333
Video Demonstration
![Page 34: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/34.jpg)
34
Conclusion and Future work• Conclusion
– UbiREMOTE framework• Intuitive and unified GUI for remotely controlling appliances
– Techniques to realize UbiREMOTE• LW 3D rendering based on cubic virtual space• Automatic reflection of real space information
– Performance evaluation• Achieved great improvement of 3D rendering performance• Usability was much better than Text-based interface
• Future work– Implementation on iPhone and Android– Usability evaluation by more detailed scenario
![Page 35: (Slides) UbiREMOTE: Framework for Remotely Controlling Networked Appliances through Interaction with 3D Virtual Space](https://reader033.vdocuments.mx/reader033/viewer/2022060115/55763ce1d8b42ac31b8b45f1/html5/thumbnails/35.jpg)
35