best practices in natural interaction design (gestures, posture, speech, gaze)
TRANSCRIPT
NUI vs. GUI
3
• NUI hardware is still in a quite early stage
• Precision and recognition rate are much lower
– A gesture may not be recognized immediately/definitely and a gaze click might not exactly be at the place the user looks at
• Interfaces can be more intuitive, but on the other hand also not as clear as traditional user interfaces
– Which voice commands or gestures are available?
• Direct manipulation of objects displayed to the user is possible
www.nuia.com
Best practices - General
• Do not try to recreate traditional interfaces
– Design for as little learning as possible and use familiar interaction principles
• Think of usage scenarios, e.g. in open space office a speech interface may not be optimal due to distraction of other users. Similarly, do not require the users to perform the same hand gesture a hundred times since they will tire of it quickly.
• Actively plan for false positive and false negatives: Just like human interaction misunderstandings are common. Today users expect devices to respond error free to any input, so you need strategies
– Direct feedback ideally with option to continue or undo
– “Double check” by combining sensors, based on history and competing detection algorithms
– Minimize requirements regarding precision
• Test, evaluate with several people and environments and tune the user experience
4www.nuia.com
Best practices – Gesture control
5
• Use gestures for simple commands and direct manipulation
• Use natural and intuitive gestures, ideally inspired by RL or at least iconic
• Stay consistent: The meaning of gestures should not change wildly with context
• Performing the gestures should be possible with both hands
• Keep in mind cross-cultural audiences and user variation, since a gesture can mean very different things in different places and users can have very different habits
• Do not occlude the relevant content with the users hands
• Give feedback about the camera FOV. The hands should not be "just lost" because the user accidentally left the camera's FOV
• Make sure recognition is robust across different platforms with different camera angle and user distance (Notebook, AIO, Add-on)
www.nuia.com
Best practices – Voice control
6
• Use voice for more complex commands and when not within range of other sensors
• Voice commands should be based around natural language usage
• Use synonyms for the commands to be more flexible and tolerant in interpreting the input
• Avoid using a single word as command (ideally two or three words)
• Either avoid dangerous commands or implement confirmation/undo due to possible false negatives
• Be aware of pauses and interjections during speech input
• Consider environmental noise and social appropriateness
• Always show the status of the speech engine (listening, not listening, processing sound)
• If you implement voice feedback make it short and intelligent
www.nuia.com
So why not just use these new sensors?
7
• Sensors cannot just be connected to existing devices and applications due to missing hooks
• Uncertainty which vendor will “win the race”
• Multiple sensors will conflict with each other without “rules”
• Very few interaction standards have been established
www.nuia.com
NUIA provides a standardized integration layer
8
• NUIA connects all next generation sensors with any environment
– Eye Trackers, 3D Cameras, Microphones, IoT Devices, Wearables, Mobiles and AR/VR Devices
• NUIA incorporates the user context in real-time, enabling the devices to interact intelligently and automatically with us
• Hundreds of use cases with most used applications and web services already integrated
• NUIA SDK enables rapid prototyping of new use cases to save time from idea to implementation
• Install and use - No source code changes in controlled applications necessary
www.nuia.com
100+ Application Extensions
NUIA Software & SDK
Your Environment(Workstation, HUD, IoT Device,
Control Panel, 3D Cave, …)
9
NUIA platform as base for all products and projects
Your Application
?
Next Generation Sensors(Eye Tracking, 3D Cameras, Voice Recognition, Mood Detection,
User Authentication, VR/AR Headsets, Neuro Headsets, …)
www.nuia.com
10
Humanizing Interaction with Devices
„… [NUIA’s] vision for reshaping how the masses interact with computers could become a reality sooner rather than later.“ - TechCrunch
www.nuia.com