vision-based finger detection and its applications

Vision-Based Finger Detectionand Its Applications

基於電腦視覺之手指偵測及其應用

Yi-Fan Chuang

Advisor: Prof. Yi-Ping Hung Prof. Ming-Sui Lee

Outline

IntroductionRelated WorkFingertip Detection and TrackingApplications

i-m-Top Magic Crystal Ball (MaC Ball)

Conclusion & Future Work

2

Outline




3

Introduction

Interactive systemBare-handed operations are more intuitive to

manipulate digital objects directlyInteractions

• Users: hover/touch gestures• Objects: rotation, scaling, selection, special effects….

Use computer vision techniques• Fingertip detection• Fingertip tracking

4

Outline




5

Related Work

Enhance DeskFingertip finding

• Template matching

Palm finding• Morphological erosion

H. KOIKE and Y. KOBAYASHI, “Integrating paper and digital information on enhanceddesk: a method for realtime finger tracking on an augmented desk system,“ ACM Transation Computer-Human Interaction, vol. 8, no. 4, pp. 307-322, 2001.

Related Work

FTIR (Frustrated Total Internal Reflection)

J. Y. Han, “Low-cost multi-touch sensing through frustrated total internal reflection," in Proceedings of the 18th annual ACM symposium on User interface software and technology (UIST '05). New York, NY, USA: ACM Press, 2005, pp. 115-118.

Related Work

Visual TouchpadFingertip detection

• Curvature of contour• Finger Orientation

Touch detection• Two warp images

S. Malik and J. Laszlo, “Visual touchpad: a two-handed gestural input device," in Proceedings of the 6th international conference on Multimodal interfaces (ICMI '04). New York, NY, USA: ACM Press, 2004, pp. 289-296.

Related Work

PlayAnyWhereTouch and Hover

• Use appearance of shadows

A. D. Wilson, “Playanywhere: a compact interactive tabletop projection-vision system," in Proceedings of the 18th annual ACM symposium on User interface software and technology (UIST '05). New York, NY, USA: ACM Press, 2005, pp. 83-92.

Related Work

Comparison

10

Enhance Desk

(CHI 01’)

FTIR

(UIST 05’)

Visual Touchpad (ICMI 04’)

PlayAnyWhere

(UIST 05’)

MethodTemplate matching

Connected component

analysis

Curvature of contour

Shadow

Detection Touch TouchTouch/Hover(2 cameras)

Touch/Hover(1 camera)

DisadvantageOne user

Only touchOnly touch One user

One finger for one palm

Our Work: Touch/Hover (1 camera) Multi-user with multi-finger

Outline




11

Overview

System setupAn IR camera with IR illuminators to observe

hands

J. Rekimoto and N. Matsushita, “Perceptual surfaces: Towards a human and object sensitive interactive display," Workshop on Perceptural User Interfaces (PUI'97), 1997.

Direct observation• Clear shapes

Indirect observation• Unclear shapes

System setupDirect observation

• Clear shapes


OverviewClear shapes (MaC Ball)

GestureCaptured

image

System setupDirect observation

• Clear shapes


Overview

14

Unclear shapes (i-m-Top)

Gesture Captured image

Overview

Fingertip detection

Fingertip tracking

Detection results Kalman filtering

Backgroundsubtraction

Morphological opening

Difference of previous two

Principalcomponent

analysis

Fingertipdetection

15

Fingertip Detection

Background subtraction

ObservationContact area leaves strong reflection

(Enhanced image)



Difference of previous two PCA

Fingertipdetection

16

Fingertip Detection

Extract finger partUse a morphological opening operation

• The size of structuring element is larger than that of a normal finger and smaller than a palm.

Original


Finger

(Enhanced image)




Fingertipdetection

17

Fingertip Detection

Difference and binarization

Principal component analysisFinger direction

Possible fingertips’ positions

(Enhanced image)Backgroundsubtraction



Fingertipdetection




Fingertipdetection

18

Fingertip Detection

Template matching (pattern matching)TemplateRemove false matchings

• Distance check• Diagonal check




Fingertipdetection

Finger patch

Detection Results on i-m-TopFingertipdetection

Difference of previous two

& PCA

Morphologicalopening


(Enhanced image) Separate finger touching and palm hovering

Detection Results on MaC Ball

21

Fingertip Tracking

Kalman filterSmooth the path

• Predict the new state and its uncertainty• Correct the tracker with its new measurement• Assume white noise and uniform velocity

Original

After Kalman filter

t

t

t

t

t

t

t

t

t

t

t

t

t

t

t

t

t

t

t

t

t

t

t

t

zvy

zy

zvx

zx

vy

y

vx

x

vy

y

vx

x

tEquationMeasuremen

wvy

wy

wvx

wx

vy

y

vx

x

vy

y

vx

x

ionStateEquat

0000

0100

0000

0001

:

1000

1100

0010

0011

:

1

1

1

1

1

1

1

1

22

Performance Evaluation

Real-time system is possible

23

Outline

IntroductionRelated WorkFingertips Detection and TrackingApplications



24

Hardware Configuration

i-m-TopInteractive Multi-resolution Tabletop

Cooperate with Yi-Wei Chia25


i-m-TopInteractive Multi-resolution Tabletop

Cooperate with Yi-Wei Chia26

Software Implementation

Fingertip detection and trackingPalm detection and trackingAssociation between fingertips and palmsPen detectionPriority of pens and palmsEvents definitions

27

Palm Detection and Tracking

Palm detectionThe average

position is picked as palm position

Palm trackingKalman filtering

Palmdetection


(Palm part)


(Enhanced Image)

Association between Fingertips and Palms

Allocate fingertips to palmsUse the angle and the distance between

fingers and palms

,*

,),(cos1,minarg

m

diri

dirposi

pos

j

pp

pfAnglepfm

(Enhanced Image)29

Pen Detection and Priority

Pen DetectionUse a higher threshold

Priority of pens and palmsHigher priority for the pen tip

• If there is a palm around the pen tip, this palm and the fingertips associated to the palm will be ignored.

30

Events Definitions

HoverPalm position

TouchPalm positionFinger position and its correspondent palmPen position

31

Application I: Browsing and Editing

OperationsDrag photo/documentZoom in and zoom outStrokeMove the foveal area

32

Application II:Video Retrieval System

OperationsQueryBrowseFeedback

33Video

Top button

Video plane Video wall

Scroll bar

E:\VRST_Paper\Final\MacBall_FinalVersion.mov

Outline




34

Magic Crystal Ball (MaC Ball)

An interactive 3D displaySlide fingers (like a wizard) on the ball surface

to control the content

35


IR Camera IR Illuminator

Pressure Sensors

User

Transparent Glass Ball

Reflection Mirror

Fresnel Lens

LCD

Display Module Detection Module


Fingertip detection and trackingDetect fingertips’ positionsPointing gesture

Motion detectionOptical flow algorithmWaving gesture

Contact DetectionPressure Sensor (Flexi Force)

• Detect whether users touch MaC Ball


Contact detection – Pressure SensorsObservation

p1 p2p3p1 p2 p1 p2

38

Events Definitions

Gesture switch

Pointing Gesture

FingertipsDetection

MotionDetection

MotionDetection

FingertipsDetection

Waving Gesture

Pressuresensors

Touch / Hover 39

Application:Virtual Exhibition

OperationsRotateSelectMagnifierChange relics

40Video

E:\VRST_Paper\Final\MacBall_FinalVersion.mov

Outline




41


Multi-finger detection techniquesSimple and real-timeCost-effective and flexible

Applications i-m-Top: interactive interfaceMaC Ball: interactive 3D display

Future workUse FTIR to enhance the accuracy of fingertip

detection More gestures for interactive systems

42

vision-based finger detection and its applications

Technology