tools for ubiquitous hci eduardo velloso. traditional hci
TRANSCRIPT
Tools for Ubiquitous HCIEduardo Velloso
Traditional HCI
Ubicomp HCI
Embedded
Ubicomp HCI
Tangible
Ubicomp HCI
Multitouch
Ubicomp HCI
Whole body
Overview
Embedded Interaction Tutorial 1: Digital Camera with .NET Gadgeteer
Tangible and Multitouch Interaction Tutorial 2: Homemade Reactable
Whole-Body Interaction Tutorial 3: Motion Capture with Kinect
Putting it all together
Embedded Interaction
“Embedded means enclosed; these chips and software are not considered computers. They are unseen parts of everyday things.”
Malcolm McCullough, Digital Ground, 2004
Components
Input Buttons, knobs, pressure, temperature, light,
accelerometers…
Output LEDs, displays, DC motors, servos, sound, vibration…
Microcontrollers PIC, Arduino, Gadgeteer, Phidgets…
Toolkits Arduino (starter kit £37)
arduino.cc
“Arduino is an open-source electronics prototyping platform based on flexible, easy-to-use hardware and software. It's intended for artists, designers, hobbyists, and anyone interested in creating interactive objects or environments.”
Toolkits Lilypad (starter kit £85) arduino.cc/en/Main/ArduinoBoardLilyPad
“The LilyPad Arduino is a microcontroller board designed for wearables and e-textiles. It can be sewn to fabric and similarly mounted power supplies, sensors and actuators with conductive thread. ”
Toolkits Phidgets (starter kit $230) www.phidgets.com
“Phidgets are a set of "plug and play" building blocks for low cost USB sensing and control from your PC.”
Toolkits .NET Gadgeteer FEZ Spider
(starter kit £177)
www.netmf.com/gadgeteer/
.NET Gadgeteer
Platform conceived to make it easier to design and build custom electronic devices and systems for a range of ubiquitous and mobile computing scenarios.
Consists of 3 components: Solder-less modular electronic hardware Object-oriented managed software libraries 3D design and construction tools
+ +
Philosophy
Flexibility Software + Hardware + Form factor
Accessibility Extensibility Low threshold for new users
Versatility High ceiling for advanced users Scales up to sophisticated device and complex systems
Extensibility Open source
Connections with sockets
Software Designer
Software
Automatic generation of code
Programming
Object-oriented C# The modules are the objects!
Each modules has Properties Methods Events
Arduino x Gadgeteer
Arduino
Setup() Initialisation of pins
and variables
Loop() Continuous read and
write from pins
Gadgeteer
Constructor Initialise variables Set event listeners
Listeners Set the behaviour of
the system when the event happens
Tutorial: Digital Camera
Resources
Papers Hodges, Steve, et al. "Exploring Physical Prototyping
Techniques for Functional Devices using .NET Gadgeteer." (2013).
Villar, Nicolas, et al. ". NET gadgeteer: a platform for custom devices."Pervasive Computing (2012): 216-233.
Official website www.netmf.com/gadgeteer/
Tangible Interaction
Prioritises as principles of design: Tangibility and materiality Physical embodiment of data Bodily interaction Embeddedness in real spaces and data
Blurs the boundaries between input and output
Reactable
Tangible modular synthesizer Controlled by manipulating tangible acrylic pucks Multi-touch interaction Collaborative performances
reacTIVision
Underlying sensor component for the Reactable
Open-source, cross-platform computer vision framework for the fast and robust tracking of fiducial markers attached onto physical objects, as well as multi-touch finger tracking.
reacTIVision
reacTIVision Standalone
application Sends TUIO messages
via UDP port 3333
TUIO protocol Designed for encoding
the state of tangible objects and multi-touch events
TUIO client Your application!
reacTIVision vision engine
TUIO Client
Events
TUIO Objects Represent the fiducial markers Add, remove and update X, Y, orientation, motion and rotation speed and
acceleration
TUIO Cursors Represent fingers Add, remove and update X, Y, speed and acceleration
Tutorial: Reactable
Material: Box Glass Tracing paper Webcam Computer
Tutorial: Reactable
Download the reacTIVision server reactivision.sourceforge.net/
Print the symbols
Build your controllers
Program your client with the functionality you want by listening to TUIO messages
Tutorial: Reactable
Example TUIO client
Whole Body Interaction
The integrated capture and processing of human signals from physical, physiological, cognitive and emotional sources to generate feedback to those sources for interaction in a digital environment.
Motion Tracking
Optical: Infrastructure-based
Inertial: Wearable
OpticalIR projection
Reflective markers
IR cameras
Triangulation
Optical
Commercial solutions: Vicon (£70,000 - £90,000)
www.vicon.com Optitrack (£6,500 - £20,000)
www.naturalpoint.com/optitrack/ Kinect (£90 - £200)
www.microsoft.com/en-us/kinectforwindows/
InertialAccelerometer
Gyroscope
Magnetometer
Inertial
Commercial solutions Xsens (£30,000-£60,000)
www.xsens.com Animazoo (£30,000)
www.animazoo.com Wiimote (£20)
www.nintendo.com/wii
Motion Tracking
Optical
Easier setup on the user
More complex setup on the infrastructure
Not subject to magnetic interference
Cameras need a clear field of view to the markers
Inertial
Harder setup on the user
Not much infrastructure needed
Subject to magnetic interference
Occlusion is not a problem
Microsoft Kinect
RGB camera
Depth camera
Microphone
Motor
Microsoft Kinect
Developing for Kinect
Kinect for Windows Runtime
Kinect for Windows SDK
Kinect for Windows Developer Toolkit
Kinect Studio
Allows you to record and playback the data from the Kinect to test applications.
Putting it all together
What if we’ve got a whole bunch of sensors and we would like to combine what they are sensing to generate more sophisticated output?
Problems
Different timestamps
Different sample rates
Modularity
Context Toolkit
“Facilitate the building of context-aware applications using the Java programming language”.
Georgia Tech
Dey, A. K., Abowd, G. D., and Salber, D. 2001. A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. Hum.-Comput. Interact. 16, 2 (Dec. 2001), 97-166.
Java Context Awareness Framework
“Java-based set of API’s for creating context-aware applications”
Aarhus University
Jakob E. Bardram. The Java Context Awareness Framework (JCAF) - A Service Infrastructure and Programming Framework for Context-Aware Applications. In Hans Gellersen, Roy Want, and Albrecht Schmidt, editors, Proceedings of the 3rd International Conference on Pervasive Computing (Pervasive 2005), Lecture Notes in Computer Science, Munich, Germany, May 2005. Springer Verlag.
Context Recognition Network Toolbox
“The Context Recognition Network (CRN) Toolbox allows to quickly build distributed, multi-modal context recognition systems by simply plugging together reusable, parameterizable components.”
ETH Zurich
Bannach, D., Amft, O., Lukowicz, P.: Rapid Prototyping of Activity Recognition Applications. In: IEEE Pervasive Computing. Vol 7:2, 2008, 22-31. ISSN: 1536-1268.
Cosm “Cosm is a secure, scalable platform that connects
devices and products with applications to provide real-time control and data storage”
Originally called Pachube
What’s else?
Eye interfaces
Gestural interfaces
Wearable computing
Sensory interfaces
Affective interfaces
