Download - An introduction to NUI

An Introduction to NUI

NUI programming seminar

Jun Iio, Ph.D.Chuo University

An introduction to NUI Copyright © Jun Iio 2

Self-introduction● Prof. Jun Iio, Ph.D.

Dept. of Socio-Informatics, Faculty ofLetters, Chuo University

● 1994-2013, Working for Mitsubishi ResearchInstitute

● 2013-2014, Assoc. Prof. of Chuo University● 2014-Current, Prof. of Chuo University● Also, 2009-, Guest Assoc. Prof. of Tokyo University

of Agriculture and Technology● My interests are

– Software Engineering, Image Processing,Open-Source Software, User Interface, etc.


What is NUI?● NUI stands for Natural User Interface

● Improvement of the User Interface?– CUI -> GUI -> NUI？

● Let's look back on the history of UI...


The first step of data input methodsfor computers

source: Wikimedia


Emerging personal computers

source: Wikimedia (CC BY-SA 2.0 fr)

Commodore PET 2001 computer. On display at the Musée Bolo, EPFL, Lausanne. The making of this document was supported by Wikimedia CH.


CUI（Character User Interface）


GUI（Graphical User Interface）


WIMP

Ｗ … WindowsＩ … IconsＭ … MenusＰ … Pointers


Physical peripherals● Keyboard● Pointing devices

– Mouse– Track-point– Track-ball– Touch pad– Touch panel


Novel operations● Tap

– Double tap– Tap and hold

● Flick, drag● Pinch

– Pinch in– Pinch out

● Shake, bump(!)– (Grab) source: http://bump.softonic.jp/iphone


Elemental technologies● Contact-type

– Touch interface● Multi touch, touch display, pressure-sensitive input● (Image input, acceleration sensor)

● Contactless-type– Space devices

● Image input, three dimensional measurement– Gesture interface

● Human body modeling, inverse kinematics● Acceleration sensor (motion, direction, inclination)● Muscle potential measurement


An impact of Windows 8

source: http://fesoku.net/archives/6029501.html


Multi-touch● http://kivy.org/


TUIO● Tangible User Interface Objects


CCV（Community Core Vision）● Poorman's multi-touch device? Video


3D mouse


Kinect


OpenNI


Leap MotionVideo


MyoVideo


RingVideo


Gesture interface● Maximum benefit

– “Direct manipulation”– It enables to control the target directly

source: ”Make It So”


Note for the NUI designs● Natural User Interface development is still

in progress● Considerable features:

1. How to realize input methods which can notmake users tired?

2. “Mode” problem of gesture input3. Haptic information (feedbacks of user's

operation)● Learning from SciFi movies

– What are applications using NUI efficiently?


References“Make It So” (Japanese Translation)


“Minority Report”● A “prototype meter” for the gesture

interface● Lessons learned:

– A grate demo hide many flaws– A gesture interface should understand intent



Enthusiasts for keyboards

source: http://www.pfu.fujitsu.com/hhkeyboard/hhkbpro/black.htmlsource: http://www.kmecs.com/products/detail_267.html


Haptic devices● An input was completed? or not?● Why does this keyboard make user feel

unpleasant?

source: http://www.itmedia.co.jp/pcuser/articles/1204/06/news033.html


Haptic devices● It enables the

feedbacks of user'sinput by touchinformation such asvibration, resistantforce, and/ormotion.

● Cf. “Myo uses apattern of vibrationas the notice foruser's input”

source: http://lab.sekaimon.com/index/index/c/58058/ym/201201/p/2/ajax/1


Grammar for the gesture input

source: Leap Motion SDK API document


Grammar for the gesture input

source: https://support.getmyo.com/hc/en-us/articles/201135025-Controlling-media-with-your-Myo-armband


Hollywood's pidgin● Wave to activate● Push to move● Turn to rotate● Swipe to dismiss● Point or touch to select● Extend the hand to shoot● Pinch and spread to scale


Wave to activate



Push to move



Turn to rotate



Swipe to dismiss



Point or touch to select



Extend the hand to shoot



Pinch and spread to scale



Be careful if you propose a novelgesture out of the seven patterns

● Another patter canlead usersmisunderstandingor confusing

● Considering globalrules and local rule– Gestures in

Japanese culture– Gestures in another

country's culture

source: Fuji television


Dependency of local cultures

source: http://matome.naver.jp/odai/2133536386030621501


An application of Kinect● Y. Nakano, T.

Tohyama, and S.Kanemune,“Comparison ofJapanese InputMethods for Kinect” Inproc. of IPSJ-SSS2013, 173-175.– https://www.faceboo

k.com/media/set/?set=a.183589891823257.1073741846.172555706260009


Applications using NUI● It should be considered what scene

requires NUI fundamentally?

● In the case of promotion videos for Myo:– Controlling drone, sphero, and UMV...

“It would be enough to use a controller!”

● It is suitable for the situations where auser cannot use his/her own hands– Controlling recipe video when cooking– tele-surgery


Future of interfaces● NUI does not replace {CUI, GUI}

– Programming using NUI? -> a distant idea

Practical

Abstract

Professionaluse

Generalpurpose

CUI

GUI

NUI


Conclusions● Progress of interacting user and computer:

CUI -> GUI -> NUI– NUI study is the blue ocean and in progress

● Elemental technologies for NUI– multi-touch, 3D measurement, acceleration

sensors, muscle potential measurement● How to solve some problems in gesture UI

– Tiredness, mode-problem, haptic interface● How to make a grammar for gesture UI

– Not only hardware but also cultural constraintsshould be considered!

Download - An introduction to NUI

Top Related