VISION for ASSISTIVE TECHNOLOGIES

Edwige Pissaloux, ISIR/UPMC & CNRS (UMR 7222), Paris

Andrea Carbone, ISIR/UPMC & CNRS (UMR 7222), Paris

Christophe Veigl, FHTW, Vienna

Christophe Weiss, FHTW, Vienna

Institut des Systèmes Intelligents et de Robotique (ISIR)

U. Roma La Sapienza (IR) EC7T (Eye Com)Pertech (IR)RIT (IR)

Outline

1. Interaction and human capabilities : definition & new technologies.

2. Vision based technologies for interaction with pointing.3. AsTeRICS project contribution to vision technologydesign.4. Results of first user evaluations of AsTeRICS gaze tracker. 5. Conclusions.

UPMC

1. Interaction and humancapabilities :definitions & new technologies.

Definitions• Interaction = reciprocal actions/influences

- key concept of modern societies- based on human attention- involves different actors

(real person, virtual characters, objects, ubiquitous environments, etc.)

• Interaction allows- accessibility to all computer based ICT solutions- new skills acquisition & development- new communication modes elaboration

(ART= Attention Responsive Technology; multimodal paradigms, etc.).

New technologies for interaction

• New assistive technologies consider a mix of humancapabilities not integrated in classic interactive tools:- brain physiological signals (BCI)- brain plasticity- visual perception (visual attention)- pure technological « intelligent tools »

(tactile, haptic, robots, orthotics, computer vision).

This presentation objective : • visual perception & computer vision• pointing operation

2. Vision basedAT for interaction withpointing

• Pointing = the designation of an object by mediation of the arm, hand and sight.

• Two steps natural pointing operation usual execution : (1) pointed object localization in space and identification via sight

(point of regard (PoR) “touches” the object);

(2) arm/hand lifting for /physical or virtual (distal) “touch” of the object.

• In the case of upper limb impairments the second step should be executed by other means (other body parts).

New AT for pointingof an object on a PC

• Implementation outline

� use of image and vision processing

� targeted body part(s) detection and tracking

� mouse different operation simulation.

Finger as a pointer• Distal touch typing system (University of Tokyo)(1)

principle : fingertip spatial movements interpreted as different gestures

are linked to a usage of a standard mouse

(1) Niikura,T , Hirobe, Y. , Cassinelli, A. , Watanabe, Y. , Komuro,T. , Ishikawa, M. “In-air typing interface for mobile devices with vibration feedback”,

ACM International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 2010).

Fingers are detected with a single high-frame-rate optical camera attached to a portable device of a

small (PDA-size) screen.

Mouse operations:

° mouse spatial displacements (by tracking the fingers)

° mouse click (by dwell time).

Limb as a pointer

• the concept of a pictorial structure (not fully connect graph) to represent the investigated part of the human body.

(1) Felzenszwalb, P.F., Huttenlocher, D.P. “Pictorial structure for object recognition”, Int. J. of Computer Vision, 61(1), pp. 55-79, 2005

(2) Siddiqui, M., Medioni, G., Robust Real-Time Upper Body Libm Detection and Tracking, VSSN’06, October 27, 2006, Santa Barbara

• The USC system (a monocular video stream)

- detects and tracks in the acquired images the human upper limbs (a tree structure)

- explores combinatorially using a body-global model the space of solutions over affine transforms and scales.

- optimizes the limb tracking via the gradient-descent approach

University of Southern California (USC)(2)MIT & Cornell (1)

Shoulder & elbowas a pointer

• EU 6th FWP IST IP AMI, University of Twente (1)- approach oriented at the estimation and recognition of poses

which generalizes detection of different body parts

(different limb sections such as shoulder, elbow)

Original image detected shoulders (and face) (2)

(1) Broekhuijsen, J., Poppe;, R., Peol, M. “Estimating 2D Upper Body Poses from Monocular Images”, Int. J. of Computer Vision, 54(1-3),

pp.181-207, 2006.

(2) Sidenbladh, H., Black, M. “Learning the statistics of people in images and video”, Int. J. of Computer Vision, 54(1-3):181-207, 2003.

• No published data on AT applications

Face and headas pointer

(1) Wahdi, M. , Koggalage, R. “Control of Mouse Movements using Human Facial Features”, 3rd Int. Conf. on Information and Automation

for Sustainability (ICAISP’2007), Melbourne, Australia, pp.13-18, 2007

(2) Gorodnichy, D., Roth, G., Nouse “use your nose as a mouse” perceptual vision technology for handfree games and interfaces,

Image and Vision Computing, 22(12), Oct. 2004, pp. 931-942.

• Different facial features : nose, global face, eyebrows, and their combination

• Processing- targeted feature is detected in acquired images during the calibration- targeted feature is tracked during the interaction.

Mouse functionalities emulation:- the mouse spatial displacements can be deduced from nose/head/eyebrows movements, - the mouse click (or object selection) is implemented through the (left or right)

eye double blinks or through dwell time.

(1) (2)

Eye& gaze as a pointer• Object selection via gaze is a fundamental interaction modality,

as the gaze position anticipates and finally allows, an action execution on the gazed object.

• Two configurations for eye- gaze trackers

Wearable (head mounted, Rochester Inst.) Remote (Tobii)

• The main characteristics :- remote systems : no devices/sensors have to be mounted on the subject’s body,

but restriction of the movement in the interaction space ; - Wearable systems : high accuracy, gaze-estimation in natural viewing context,

unlimited interaction space

3. AsTeRICSproject contribution to vision technology design (http://www.asterics.eu)

Objectives- the design of an adaptable and scalable system supporting

also unconventional peripherals (BCI, vision, etc.) for people with severely reduced motor capabilities interactions

- the evaluation of system with primary and secondary users.

AsTeRICS construction set (ACS) supports a wide range of user input and actuators (switches, grippers etc.)

that can be integrated with the vision-based system :

� tailoring the interaction scheme to the specific end-user needs and capabilities

� giving the possibility to explore completely new sensor combinations for human-computer interaction.

AsTeRICSwearable gaze trackerMain characteristics - adjustable to the end-user anatomy (head size, distance eye/camera, etc.) - adaptable to specific needs (small-amplitude head movements compensation,

easy to wear, precision of detection compatible with targeted skills for interaction, etc.)

Hardware - hot-mirrors or telescopic arms - multiple combinations of sensors coping with different capabilities and interaction

needs- batteries for long autonomy.

Foreseen combinations : - a three camera system using only visible lighting

for full gaze tracking in 3D space ;- a ‘minimal’ set with only one IR eye-camera and

a custom PCB integrating an IR tracker, a gyroscope and a Sip/Puff sensor.

AsTeRICS wearable gaze tracker (1)

Iris detection Visible & IR

Particle filter & Radial transform

Martinez, F., Carbone, A., Pissaloux, E., Radial Symmetry guided Particle Filter for Robust Iris Tracking, Proc. CAIP 2011, Spain

AsTeRICS gaze tracker

targeted applications � Wearable gaze tracker

• Vision « only » gaze tracker :

environmental control applications

• IR gaze tracker :

eye-movements interaction with a computer monitor

coupled with a range of actuators driven by the sip/puff sensor.

� Remote head/gesture tracker (webcam, Kinect)

• simple interaction purposes via head pose on low performance

computing platforms

• a nose- and chin-tracking for mouse simulation (mouse displacements

and an object selection/clicking) (Haar transform; KLT)

• Complex facial expression and gesture tracking (Active shape models)

ASM detection & tracking (left) ; landmark detection (right)

4. Results of first user evaluations.

AsTeRICS system prototype 1 – remote (web-camera based) gaze tracker- June-August 2011- different sensors and sensor-combinations - 50 users in Austria, Poland and Spain- spasms and involuntary head movements

Tests - Interaction with a screen

Results- spasms and involuntary head movements represented a big problem,

preventing precise pointing or computer mouse control- BUT highest level of acceptance.

Future developments- tremor reduction algorithms and - evaluation of the head mounted eye tracker

5. Conclusion

Vast impact of the vision technology a on the quality of life,

Vision allows new modes- to access ICT and internet-based services

(games, e-library, e-shopping, e-health, e-rehabilitation, e-learning, etc.)- to access smart environments which constitute the infrastructure for gaze interaction

with environmental control systems(lighting control, heating/ventilation or usage of home entertainment devices);

- for new skills acquisition (such as navigation in virtual worlds) (with training).

Future (second phase) of the AsTeRICS project- the head-mounted gaze estimation system will be finished, - both, the remote and the head-mounted solution,

will be evaluated in qualitative and quantitative user tests.

Thanks to- EU FP7 ICT program- France Soudage

Francis Martinez, ISIR/UPMCDarius Mazeika, ISIR/UPMC & Kaunas University (Lithuania)

Isabelle Liu, Master Student, UPMCJaza Gul Mohammed, Master Student, UPMC

Jacky Chen, E&M, MITFaith Keza, CS, MIT

Thankto youfor your participation