22/01/2013

1

Haptics Technologies:Bringing Touch to Multimedia

C2: Haptics Applications

Outline Haptic Evolution: from Psychophysics to Multimedia

Haptics for Medical Applications Surgical Simulations

Stroke-based Rehabilitation

Support of the Visually Impaired

Tele-robotics and Tele-operation Tele-Surgery

Military Applications

Media Haptic Broadcasting

E-Commerce

Video Games

Arts and Design

Mobile Haptics

Haptics and Virtual reality

Education and Learning

Haptics for Security

Closing Remarks

Haptic Evolution: from Psychophysics to Multimedia

Experimental psychology Beginning of 20th century

Understanding human touch perception and manipulation

Robotics era Seventies and eighties of last century

Create devices with a dexterity inspired by human abilities

Tele-operation era Late eighties

extension of sensing and manipulation to a remote location

Haptics Early nineties

Tactile and kinaesthetic interfaces and computer haptics

Multimedia Haptics

22/01/2013

2

Haptic Evolution: from Psychophysics to Multimedia

The application spectrum is quite vast, and its trend of expansion is anticipated to increase

Haptics for Medical Applications

Already a strong presence in the area of medical training Professionals guide students through a simulated

surgical process Example: Trainee feels the surgeon performing a precision cut on

a simulated human organ

Statistically, surgeons have higher success rates the more often they perform a procedure

Allows surgeons to practice a procedure with less fatigue and stress

If geographically separated, realistic haptic rendering is near impossible!

Realistic haptic models of human tissue and organs

Surgical Simulations

May be used to train surgeons at remote locations In the future, tele-haptic applications will be used to perform surgical

procedures on patients at remote locations. e.g., Arctic regions, the space station

Examples: Needle-based procedures use needles, catheters, guide-wires, and small

bore instruments for teaching straightforward procedures

Minimally invasive surgery (called laparoscopic surgery)

Robot-assisted surgery (da Vinci Surgical System)

Simulation environments

Suturing video 1

Suturing video 2

22/01/2013

3

Surgical Simulations

Virtual eye

Surgical tool

Phantom Haptic Device

Eye Surgery Training Simulation

X. Shen et al., “Haptic-Enabled Telementoring Surgery Simulation", IEEE Multimedia, 2008

Stroke-based Rehabilitation

Involves applying forces to the injured/disabled organ (such as the finger, arm, ankle) to help it recover its strength and range of motion

helps to measure performance and to tailor performance-based exercises for each patient

Enables home located rehab and remote monitoring

Stroke-based Rehabilitation Game-like tasks to address certain parameters of hand movement:

Based on Jebsen Test for Hand Function (JTHF)

Offers continuous evaluation of the patient’s improvement.

Example exercise: Arranging eight cubes according to a color pattern.

22/01/2013

4

Stroke-based Rehabilitation Based on Augmented Reality (AR) technology

Post-Stroke Interactive and Entertaining Rehabilitation with ReActive objects (SIERRA)

Reactive Objects

react with vibration to inform the subject of being interacting with virtual objects

Components: A real cup

A vibrotactile actuators (pager motors)

An actuator controller

a bluetooth module

a 9V battery

Features: Inexpensive

light

portable

The actuators are arranged to cover a whole hand when grabbing.

Stroke-based Rehabilitation

Rehab test 1

Rehab test 2

Rehab test 3

Rehab test 4

Rehab test 5

Rehab test 6 (foot Stroke)

22/01/2013

5

Support of the Visually Impaired

Haptics enhances the perception of blind or visually impaired people by converting visual or sound information into a haptic modality

Example applications: European Union GRAB project

the exploration of chart data,

a city map explorer, and

a simple adventure game

TACTICS system interpret complex scientific data

Chromo-Haptic Sensor-Tactor (CHST) To feel colors

VIbroTActiLe (VITAL) system Tactile feedback as a navigation tool for the visually impaired

Inner fabric to protect the skin

Support of the Visually Impaired

How it works? Touch by hand, cell phone, mouse

For hand Hand tracking using depth image

For touch-enabled cell phone

For mouseVirtual touch

pad

22/01/2013

6

Tele-robotics and Tele-operation

Collaborative Design Designers at various locations

collaborate on the same project using a HVE

Haptics offers the participants additional information regarding the project

Objects may be moved, grasped, assembled, disassembled

Requires position sampling and collision detection for N users

Geographically separated users –network performance is key

Tele-robotics and Tele-operation

To remotely handle dangerous or radioactive substances

Applications: underwater exploration,

assembly and manufacturing,

micro-assembly

Tele-Surgery

Training surgeon on a simulated human organ

Tele-surgery

Train surgeons at remote locations

Perform surgical procedures on patients at remote locations

Brain surgery

Reachin Display[Reachin Technologies]

22/01/2013

7

Military Applications

Battlefield conditions, such as the presence of artillery fire or smoke, might mask sound and vision modalities

Haptics become an efficient communication channel

Media

Started in 1959, when tactile stimulation was used to enhance the movie “The Tingler” by attaching vibrating devices to the theatre seats

Provide higher immersion in multimedia contents

two content scenarios: Authored content

Viewers are able to interact with and influence the presentation of pre-recorded content

Real-time content The acquisition and display of haptic content occurs in real time

Haptic Broadcasting

Broadcasting multimedia streams with haptic channel Examples:

Home shopping environment Viewers can touch the 3D model through a haptic device

A fiducial marker position and orientation are calculated

Haptic Video Annotate video stream with haptic stimuli

Users feel what they see and hear

Haptic YouTube video Annotated with tactile feedback using XML notation

Used a haptic enabled browser and an arm band device

22/01/2013

8

E-Commerce

Haptic virtual environments gives new meaning to the phrase “Try before you buy”

Consumers could experience aspects of the product not available using current E-Commerce technology Example: “How heavy is this new PDA?” “How soft is this sweater?” –

haptic interfaces can answer these questions

Browser plug-in or Java application offers haptic and graphical display of a 3d model of the object on a vendor website

Requires that accurate models of new products can be acquired quickly and cheaply

Video Games

Gaming experience comprises four aspects: physical, mental, social, and emotional.

Force feedback technology enhances the physical aspects of the gaming experience

Greater immersive experience to gamers

Haptic information is recorded and played-back in synchronization with audio-visual data

BUT: Multiplayer games require high-bandwidth, low-latency networks

Logitech MOMO racing wheel[Logitech]

Logitech Rumblepad[Logitech]

Logitech Force 3D joystick[Logitech]

Gaming

S. Andrews et al., “HaptiCast: A physically-based 3D game with haptic feedback”, in Proc. of Future Play 2006.

When applied to video games, it gives players a higher sense of immersion/realism.

22/01/2013

9

Arts and Design

Arts & Culture Due to a “hands-off” policy,

museums do not permit visitors to handle sculptures and artifacts

Haptic equipped kiosks could be used to permit visitors to feel and manipulate these objects virtually

Requires realistic models of museum artifacts to be acquired so that users may truly explore them Example: tracing the

contours of Michelangelo’s “David”

Digital 3D model of ‘David’[The Digital Michelangelo Project]

Mobile Haptics

Mobile devices utilize gesture to allow easier and more intuitive user interaction

Haptics would enhance such intuitivism

Tactile haptic feedback is becoming common in smart phones

Example: Samsung haptic mobile phone

Haptic SCH-W420

Mobile Haptics

The KissMe system A neck piece to display kisses

Mobile device connected to neck piece via Bluetooth

Users can send kiss messages to each other

22/01/2013

10

Mobile Haptics

The HugMe system 2.5D video camera

A haptic Jacket (mobile device)

A haptic device

Local person Remote person

Depth cameraForce feedback device

Haptics and Virtual reality

Integrating haptic interactions with existing virtual environments: Linden Lab's multiuser online virtual world

Second Life

Education and Learning

Learners learn better a sensorimotor skill task using haptic modality

Example: Haptic Multimedia learning tool: Teach handwriting of alphabets (Japanese, Arabic, etc)

Three modes of operation: Full guidance

Partial guidance

No guidance (testing)

InteraSense (too)

22/01/2013

11

Haptics for Security

Haptic Biometrics More “advanced” handwritten signature

Environment: Virtual Check Application.

Visual stimuli: Virtual pen and a virtual check.

Haptic stimuli: Force and frictional feedback

Haptics for Security

Graphical password with haptics Utilizes the physical attributes captured during human-computer

interaction (pressure and velocity)

Overcomes shoulder surfing attacks

Continuous verification and authentication

Closing Remarks

The incorporation of haptics into virtual environments is still in its infancy!

A wide range of human activities, such as communication, education, entertainment, commerce, and science, would forever change if we learned how to capture, manipulate, and create haptic sensory stimuli that are nearly indistinguishable from reality

Business models and frameworks are needed to make haptic devices practical, inexpensive, and widely accessible (as easy as mouse)

22/01/2013

12

متشکرم

谢谢！ありがとう！