An Introduction to NUI

NUI programming seminar

Jun Iio, Ph.D.Chuo University

An introduction to NUI Copyright Jun Iio 2

Self-introduction Prof. Jun Iio, Ph.D.

Dept. of Socio-Informatics, Faculty ofLetters, Chuo University

1994-2013, Working for Mitsubishi ResearchInstitute

2013-2014, Assoc. Prof. of Chuo University 2014-Current, Prof. of Chuo University Also, 2009-, Guest Assoc. Prof. of Tokyo University

of Agriculture and Technology My interests are

Software Engineering, Image Processing,Open-Source Software, User Interface, etc.

An introduction to NUI Copyright Jun Iio 3

What is NUI? NUI stands for Natural User Interface

Improvement of the User Interface? CUI -> GUI -> NUI

Let's look back on the history of UI...

An introduction to NUI Copyright Jun Iio 4

The first step of data input methodsfor computers

source: Wikimedia

An introduction to NUI Copyright Jun Iio 5

Emerging personal computers

source: Wikimedia (CC BY-SA 2.0 fr)

Commodore PET 2001 computer. On display at the Muse Bolo, EPFL, Lausanne. The making of this document was supported by Wikimedia CH.

An introduction to NUI Copyright Jun Iio 6

CUICharacter User Interface

An introduction to NUI Copyright Jun Iio 7

GUIGraphical User Interface

An introduction to NUI Copyright Jun Iio 8

WIMP

Windows Icons Menus Pointers

An introduction to NUI Copyright Jun Iio 9

Physical peripherals Keyboard Pointing devices

Mouse Track-point Track-ball Touch pad Touch panel

An introduction to NUI Copyright Jun Iio 10

Novel operations Tap

Double tap Tap and hold

Flick, drag Pinch

Pinch in Pinch out

Shake, bump(!) (Grab) source: http://bump.softonic.jp/iphone

An introduction to NUI Copyright Jun Iio 11

Elemental technologies Contact-type

Touch interface Multi touch, touch display, pressure-sensitive input (Image input, acceleration sensor)

Contactless-type Space devices

Image input, three dimensional measurement Gesture interface

Human body modeling, inverse kinematics Acceleration sensor (motion, direction, inclination) Muscle potential measurement

An introduction to NUI Copyright Jun Iio 12

An impact of Windows 8

source: http://fesoku.net/archives/6029501.html

An introduction to NUI Copyright Jun Iio 13

Multi-touch http://kivy.org/

An introduction to NUI Copyright Jun Iio 14

TUIO Tangible User Interface Objects

An introduction to NUI Copyright Jun Iio 15

CCVCommunity Core Vision Poorman's multi-touch device? Video

An introduction to NUI Copyright Jun Iio 16

3D mouse

An introduction to NUI Copyright Jun Iio 17

Kinect

An introduction to NUI Copyright Jun Iio 18

OpenNI

An introduction to NUI Copyright Jun Iio 19

Leap MotionVideo

An introduction to NUI Copyright Jun Iio 20

MyoVideo

An introduction to NUI Copyright Jun Iio 21

RingVideo

An introduction to NUI Copyright Jun Iio 22

Gesture interface Maximum benefit

Direct manipulation It enables to control the target directly

source: Make It So

An introduction to NUI Copyright Jun Iio 23

Note for the NUI designs Natural User Interface development is still

in progress Considerable features:

1. How to realize input methods which can notmake users tired?

2. Mode problem of gesture input3. Haptic information (feedbacks of user's

operation) Learning from SciFi movies

What are applications using NUI efficiently?

An introduction to NUI Copyright Jun Iio 24

ReferencesMake It So (Japanese Translation)

An introduction to NUI Copyright Jun Iio 25

Minority Report A prototype meter for the gesture

interface Lessons learned:

A grate demo hide many flaws A gesture interface should understand intent

source: Make It So

An introduction to NUI Copyright Jun Iio 26

Enthusiasts for keyboards

source: http://www.pfu.fujitsu.com/hhkeyboard/hhkbpro/black.htmlsource: http://www.kmecs.com/products/detail_267.html

An introduction to NUI Copyright Jun Iio 27

Haptic devices An input was completed? or not? Why does this keyboard make user feel

unpleasant?

source: http://www.itmedia.co.jp/pcuser/articles/1204/06/news033.html

An introduction to NUI Copyright Jun Iio 28

Haptic devices It enables the

feedbacks of user'sinput by touchinformation such asvibration, resistantforce, and/ormotion.

Cf. Myo uses apattern of vibrationas the notice foruser's input

source: http://lab.sekaimon.com/index/index/c/58058/ym/201201/p/2/ajax/1

An introduction to NUI Copyright Jun Iio 29

Grammar for the gesture input

source: Leap Motion SDK API document

An introduction to NUI Copyright Jun Iio 30

Grammar for the gesture input

source: https://support.getmyo.com/hc/en-us/articles/201135025-Controlling-media-with-your-Myo-armband

An introduction to NUI Copyright Jun Iio 31

Hollywood's pidgin Wave to activate Push to move Turn to rotate Swipe to dismiss Point or touch to select Extend the hand to shoot Pinch and spread to scale

An introduction to NUI Copyright Jun Iio 32

Wave to activate

source: Make It So

An introduction to NUI Copyright Jun Iio 33

Push to move

source: Make It So

An introduction to NUI Copyright Jun Iio 34

Turn to rotate

source: Make It So

An introduction to NUI Copyright Jun Iio 35

Swipe to dismiss

source: Make It So

An introduction to NUI Copyright Jun Iio 36

Point or touch to select

source: Make It So

An introduction to NUI Copyright Jun Iio 37

Extend the hand to shoot

source: Make It So

An introduction to NUI Copyright Jun Iio 38

Pinch and spread to scale

source: Make It So

An introduction to NUI Copyright Jun Iio 39

Be careful if you propose a novelgesture out of the seven patterns

Another patter canlead usersmisunderstandingor confusing

Considering globalrules and local rule Gestures in

Japanese culture Gestures in another

country's culture

source: Fuji television

An introduction to NUI Copyright Jun Iio 40

Dependency of local cultures

source: http://matome.naver.jp/odai/2133536386030621501

An introduction to NUI Copyright Jun Iio 41

Dependency of local cultures

source: http://matome.naver.jp/odai/2133536386030621501

An introduction to NUI Copyright Jun Iio 42

An application of Kinect Y. Nakano, T.

Tohyama, and S.Kanemune,Comparison ofJapanese InputMethods for Kinect Inproc. of IPSJ-SSS2013, 173-175. https://www.faceboo

k.com/media/set/?set=a.183589891823257.1073741846.172555706260009

An introduction to NUI Copyright Jun Iio 43

Applications using NUI It should be considered what scene

requires NUI fundamentally?

In the case of promotion videos for Myo: Controlling drone, sphero, and UMV...

It would be enough to use a controller!

It is suitable for the situations where auser cannot use his/her own hands Controlling recipe video when cooking tele-surgery

An introduction to NUI Copyright Jun Iio 44

Future of interfaces NUI does not replace {CUI, GUI}

Programming using NUI? -> a distant idea

Practical

Abstract

Professionaluse

Generalpurpose

CUI

GUI

NUI

An introduction to NUI Copyright Jun Iio 45

Conclusions Progress of interacting user and computer:

CUI -> GUI -> NUI NUI study is the blue ocean and in progress

Elemental technologies for NUI multi-touch, 3D measurement, acceleration

sensors, muscle potential measurement How to solve some problems in gesture UI

Tiredness, mode-problem, haptic interface How to make a grammar for gesture UI

Not only hardware but also cultural constraintsshould be considered!

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45