transparency improvement of haptic-based networked systems

DEPT. OF INFO. & COMM., GISTNetworked Media Lab.

Networked Media LaboratoryDept. of Information & Communications

Gwangju Institute of Science & Technology (GIST)[email protected]

http://nm.gist.ac.kr/~shlee

1

Transparency Improvement of Haptic-based Networked Systems

Seokhee Lee


2

Contents Introduction

Transparency Analysis for Haptic-based Net-worked Systems

Delay Compensation Scheme for Haptic-based NVEs

Transmission and Error Control Scheme for Haptic-based NVEs

Haptic Synchronization Scheme for Force-re-flecting Teleoperation

Conclusions and Future Work


3

Introduction

Overview of HNS Motivation and Contributions


Overview of HNS HNS (haptic-based networked system)

System which provides haptic feeling for a user who interacts with remote environments

Haptic-based NVE (networked virtual environment) + force-reflecting teleoperation

Local haptic system + remote haptic systems

4


5

Requirements of HNS Main goal of HNS

Executing a task in a remote environments (real or virtual) With stability and transparency

Stability The primary requisite for safe system Instability

Uncontrollable oscillations and chaotic behavior Factors causing instabilities

Quantization error, time delay, packet losses, asynchronous switching between continuous- and discrete-time subsystems ……

Transparency Transparency ≈ haptic realism Mathematically more difficult to analyze since the ultimate goal is

to make the user experience a “good feeling” Optimal transparency

The user cannot distinguish between direct and tele-interaction with a remote environment.

Focused requirement


6

Network Problems Instability

The delayed and lost haptic data destabilize HNS. The instability may cause serious damages to the user.

Transparency deterioration The human haptic sense is more sensitive than the visual

and auditory senses. Because of the sensitivity, the transparency is deteriorated

severely with network variations of delay and loss. High transmission rate

A high update rate (approximately 1 kHz) of haptic rendering leads to high transmission rate of 1000 packets/s.

The available network bandwidth for multiple haptic data may not be sufficient over current Internet.

Focused network problems


7

Existing Haptic Data Net-working Schemes Network layer solutions

Improve the performance of the network itself for HNS. QoS provision for haptic data

Class-based weight fair queue (Marshall08), DiffServ & IntServ (Hirche05)

QoS routing for haptic data QoS-guaranteed overlay routing (Cen05)

High speed network for haptic data Optical networks (LaMarche07)

Application and transport layer solutions Improve the system performance on the assumption that

the current network performance is not sufficient. Delay/jitter compensation Transmission control Error control Focused networking

schemes


Motivation

Problems of the existing networking schemes (application and transport layer) There was little consideration to enhance the human hap-

tic perception. Network adaptation schemes for conventional multimedia

were applied to the haptic interaction without careful consideration of the difference between the haptic event and the other data.

Remaining challenge Minimizing the transparency (haptic realism) degradation

By determining the optimum adaptation parameters for transparent haptic interactions

Transmission rate, error control level, buffering time, spring co-efficient for delay compensation,……

8


Contributions Distinguishing point of the proposed net-

working schemes Consideration of human perception characteristics (trans-

parency) for the optimization Transparency analysis

Deterioration of haptic interaction quality caused by network delay is quantified as distortions of mass, damping, and spring coefficients of a virtual object.

In order to formulate the importance of a haptic event with respect to a packet loss, loss effect of each haptic event is quantified for haptic-based NVEs with a prediction algorithm.

For force-reflecting teleoperations stabilized by a control al-gorithm, the force feedback distortions caused by network delay and packet loss are quantified when robot keeps in contact with a wall.

9


10

Contributions Networking schemes based on transparency

analysis Delay compensation scheme

Allowable delay prediction Predicts the maximum allowable delay based on the quantified

values of mass and damping distortions. Spring coefficient modification

Modifies spring coefficient according to delay based on the quanti-fied values of spring coefficients’ distortions.

Verification Experimental results

The proposed scheme improves the haptic interaction quality more ef -ficiently compared with existing delay compensation schemes while avoiding unnecessary trial-and-errors over network delay.

Representative paper Elsevier Computer Communications, 2009


11

Contributions Transmission and error control scheme

Haptic event prioritization Based on the quantified loss effect, all haptic events are classified into sev-

eral groups with different network QoS requirements. Priority-based haptic event filtering

Based on the prioritization, select more appropriate data to be transmitted for realistic haptic interaction over bandwidth-limited lossy network.

Verification Simulation and experiment results

The proposed scheme provides lower packet rate than the existing schemes for a transparent haptic interaction over a bandwidth-limited network.

The proposed scheme guarantees less processing delay and better haptic interac-tion quality compared with previous error control schemes over a lossy network.

Representative paper Springer Multimedia Systems, 2009

Patent Haptic event transport method for haptic-based collaborative virtual environment

and system therefor


12

Contributions Haptic synchronization (buffering) scheme

Allowable delay and loss time prediction Predicts maximum allowable delay and loss time based on quanti-

fied force feedback distortions and predefined transparency re-quirements.

Network-adaptive buffering time control Improves transparency by controlling the playout time of the

transmitted haptic event with the transparency-related parameters from transparency analysis.

Verification Remote calligraphy system

The proposed scheme guarantees less force feedback error compared with the existing haptic synchronization schemes over time-varying network delay.

Representative paper ACM NetGames, 2009

Patent application The method for synchronizing haptic data and the haptic system


13

Transparency Analysis for Haptic-based Networked System

Transparency Deterioration Caused by Network Variations

Transparency Analysis for Hap-tic-based NVEs

Transparency Analysis for EBA-based Teleoperations


Transparency Deterioration Caused by Network Variations Transparency deterio-

ration caused by delay If there is network delay while a

user manipulates a virtual object, the virtual object does not move immediately.

During that time period, penetra-tion depth increases.

Eventually, the force feedback in spring-damper model also in-creases.

14

<Spring-damper model>force_feedback = spring_coefficient penetration_depth+damping_coefficient velocity


15

Transparency Deterioration Caused by Network Variations Transparency deterio-

ration caused by packet loss Haptic-based NVE without a

data prediction scheme The force feedback also in-

creases in the same manner of the delay case.

Sudden movement Haptic-based NVE with a

data prediction scheme If the haptic event can be pre-

dicted based on past patterns of events, the quality degradation will be small.

Otherwise, the quality degrada-tion is more severe.

With data prediction scheme<Original movement of a virtual ob-

ject>

<Unintended movement caused by loss>


16

Transparency Deterioration Caused by Network Variations Transparency deteriora-

tion caused by delay jit-ter More severe deterioration of hap-

tic interaction quality Out-of-order arrivals as well as de-

layed data transmission and empty sampling instances

(Lee06)


Transparency Analysis for HNS Quantification of transparency deterio-

ration according to network delay and packet loss In order to determine the optimum adaptation pa-

rameters of haptic data networking schemes Focused HNS

Haptic-based NVEs With CS (client-server) architectures

Force-reflecting teleoperation With EBA (energy bounding algorithm)

17


18

Transparency Analysis for Haptic-based NVEs Position-position interaction

HIP (haptic interaction pointer) position (client -> server) Virtual object position (server -> client)

Consistency server Updates movements of all virtual objects.

Distributed force calculation Each client calculates force feedback by using spring-damper model.

1

( ) ( ) ( )i

o o ein

mx t bx t f t

( ) ( - )outei hi ix t x t T

( ) -{ ( ( ) - ( )) ( ( ) - ( ))}ei ei o ei ei o eif t k x t x t b x t x t

( ) ( ( ) - ( )) ( ( ) - ( ))d dhi hi o hi hi oi hif t k x t x t b x t x t

( ) = ( )d inoi o ix t x t T


19

Transparency Analysis for Haptic-based NVEs Transparency

Equality between the human (Zh) and the environment (Ze) impedances

Simplification The number of clients is not

related to transparency. A virtual object is repre-

sented by a mass and damp-ing.

Impedance

2

3 2

( ) ( )( ) 2 2( )( ) ( ) ( )

2 2 2

h hh h h e

hh

ehe

k Rm k Rbs k m s k b k k RF sZ s RkRm bRX s s m s b s k

2

( )( )( )

e e ee

e e

F s k ms k bZ sX s ms bs k

( )e eG s k

( )h hG s k2

1( )oG sms bs


20

Transparency Analysis for Haptic-based NVEs Magnitude responses

of impedances with respect to network delay Confirm that if delay in-

creases, force feedback also increases

It is called transparency (force feedback) distor-tion Mass and damping dis-

tortion (mh, bh) Spring coefficient distor-

tion (khm, kh

b)

kh=0.5, ke=0.5, m=0.25, b=0.0025

R=RTT (round trip time)


21

Transparency Analysis for Haptic-based NVEs Mass and damping distortion

If delay increases, a user perceives a virtual object as having larger mass and damping coefficients than the actual values.

Quantification of mass (mh) and damping (bh) distortion Low frequency approximation

Human operator usually generates low frequency input (Hirche05) [10-6~102]

Approximated human impedance

Transparency condition

Distorted mass

Distorted damping

2

( ) ( )2( )

( ) ( )2 2

hh h h e

apph

ee

k Rb k m s k b k k RZ s

RkbR m s b s k

2hRbm m

h eb b k R

( ) ( )apph eZ s Z s


22

Transparency Analysis for Haptic-based NVEs Distortion of spring coefficient

If delay exists, user is provided with unrealistically large force feedback that seems to be obtained with the larger spring coefficient than the actual value.

Quantification of spring coefficient distortion Mass-based distortion (kh

m)

Damping-based distortion (khb)

( )2 2

m m hh h h h

k RbRbk m k m k km

( )b b h eh h e h h

k k Rk b k b k R k k

b

If 2 then

If 2 then

If 2 then

m be h h

me h

be h

b mk k k

b mk k

b mk k

If virtual object dynamics are affected by mass more than by damping, spring distortion follows mass-based distortion; otherwise it follows damping-based distortion.


Transparency Analysis for Haptic-based NVEs Force feedback distortion

according to packet losses Without a prediction scheme

Quantified in the same manner as the force distortions according to network delay

By using loss time (Tloss) With a prediction scheme (fdis(n))

Quantified by using the difference between the actual and the pre-dicted (xpre(n)) positions

Loss effect (LE(n)) of the nth haptic event

In order to formulate the impor-tance of a haptic event with re-spect to a packet loss

23


Transparency Analysis for EBA-based Teleoperation

EBA (energy bounding algorithm) Stability algorithm of a haptic simulation system (Kim04)

EBA restricts the energy generated in the ZOH (zero order hold) within a consumable energy limit in the haptic device.

Can be applied to teleoperation to ensure robust stability regardless of the amount of time delays and packet losses (Seo08).

24


Transparency Analysis for EBA-based Teleoperation

Master EBA

25

Slave EBA

,max ,max

,min

( ) ( 1) ( ) ( )where

( ) ( 1)( ) ( ) 0

( )( ) ( ) ( 1),

( ) ( ) ( ),with the following bounding laws:

( ) ( ) ( ) ( )

( ) (

sEBA sEBA s s

s sEBAs s

s

s s s

s sd s

s s s s

s s

F n F n n e n

F n F nn for e n

e ne n e n e n

e n x n x n

if n n then n n

if n n

,min

,max 1 ,max

,min ,min

,01 1

2

0

2

2,max 2 2

2,min 2 2

) ( ) ( )

where( ) min( , ( )),

( ) ( ),

( ),

( 1)

( 1) ( 1)( ) ,( ) ( )

( 1) ( 1( )

( )

s s

s s s

s s

SD SDs n

sk

s ss s s

s s

s ss s s

s

then n n

n c n

n n

P nc

e k

F n F nn c ce n e n

F n F nn c c

x n

2

2

),

( )

where is a positive constant.s

s

x n

c

,max ,max

,min ,min

,max 1 ,max

,min ,min

,01 1

2

0

,max

with the following bounding laws:( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

where( ) min( , ( )),

( ) ( ),

( ),

( 1)

(

m m m m

m m m m

m m m

m m

MD MDm n

mk

m

if n n then n n

if n n then n n

n c n

n n

P nc

X k

n

2

22 2

2

2,min 2 2

2

( 1) ( 1)) ,

( ) ( )

( 1) ( 1)( ) ,

( ) ( )

where is a positive constant.

m mm m

m m

m mm m m

m m

m

F n F nc c

X n X n

F n F nn c c

X n X n

c

( ) ( 1) ( ) ( )where

( ) ( 1)( ) ( ) 0( )

( ) ( ) ( 1)

mEBA mEBA m m

md mEBAm m

m

m m m

F n F n n X n

F n F nn for X nX n

X n X n X n

Control law Control law

Bounding law Bounding law


Transparency Analysis for EBA-based Teleoperation Definition of Transparency

Similarity between the force feedback for a slave robot (FsEBA) and that for a user in master side (FmEBA)

Assumptions Robot keeps in contact with a wall and a user feels the force feed-

back by moving haptic device facing the wall with constant veloc-ity (vm).

Force feedback increase according to the user's input From the control law in master EBA

26

/

1

/

,max ,max1

/

1 ,max1

( ),

( )

min( , ( ))

inter

inter

inter

T

in m mn

T

in m mn

T

m m mn

F v n

F v n

v c n

,

: update time period ( ): interaction time ( )

: increase of ( ): maximum value of ( )

inter

in mEBA

in max in

secT secF F NF F N


Transparency Analysis for EBA-based Teleoperation Approximation of force

feedback increase Assumption: c1m≥2c2m

γm,max(n) in bounding laws con-verges into c2m when Fm(n-1) increases monotonically.

Force feedback decrease caused by delay Network delay reduces Tinter as

much as the delayed time. If the network delay in-

creases, the force feedback decreases in proportional to c2m∙vm.

27

,max 2in m m interF c v T

, 2de delay m m delayF c v T

, : decrease according to delay

: network delayde delay mEBA

delay

F F

T


Transparency Analysis for EBA-based Teleoperation Force feedback de-

crease (Fde,loss) caused by loss Packet losses reduces Tinter as

much as the loss period. Transparent loss time (Ttr,loss):

time period when the force feedback continuously in-creases even though there exist the packet losses

28

,,

2

loss mEBA losstr loss

m m

F FT

c v

, 2 ,( )de loss m m loss tr lossF c v T T

, mEBA

: the latest received force feedback before the packet losses

: the latest updated F

before the packet losses: loss time (time period when

the pack

loss

mEBA loss

loss

F

F

Tet losses happen)

Loss

F mEB

A

Time

Floss

FmEBA,loss

Ttr,loss

Tloss

Beginning of loss

End of loss


29

Delay Compensation Schemefor Haptic-based NVEs

Related Work Delay Compensation Scheme Experimental Results


30

Related Work Spring coefficient modification scheme (Fuji-

moto04) Dynamically changes the reaction force applied to a user by

adjusting a spring coefficient of a spring-damper model. The spring coefficient (Kh) is modified according to the current

end-to-end delay (ΔT).

Problems An accurate spring coefficient for the realistic haptic interac-

tion can only be found by a process of trial and error. A significant challenge remains in developing an optimum

method of determining the spring coefficient.

( , )

_ _

where denotes initial spring coefficient

inith h

inith

inith

K min K b T

b K initial buffering time

K


31

Transparency Analysis-based Approach Transparency Analysis

Transparency deterioration caused by delay is quantified as distortions of mass, damping, and spring coefficients of virtual objects.

Delay compensation scheme based on transparency analysis Predicts the maximum allowable delay from the

quantified values of mass and damping distor-tions.

Modifies the spring coefficient according to net-work delay based on the quantified value of spring coefficient distortion.


32

Delay Compensation Scheme

Haptic interface

HIP position

ith ClientHaptic & graphic

rendering

Allowable delay

prediction

Force Feedback

New spring coefficient (kh

new)

Assumption- Time- varying network delay

Network monitoring

Spring coefficient

modification

Calculation of force feedback

Maximum allowable delay

(Rmaxm, Rmax

b)

Current network

delay (R)

Network transport

Haptic event

transport

ServerHaptic & graphic

rendering

Update of virtual object

Network transport

Haptic event

transport

Virtual object

position

Delay com-pensation scheme


33

Delay Compensation Scheme Allowable delay prediction

Quantified mass and damping distortions are proportional to network delay.

Predefined transparency requirements Maximum allowable mass (mallowable) and damping (ballow-

able) Allowable mass and damping gradients (cm, cb)

Mass-based:

Damping-based:

allowablem

m mc

m

allowableb

b bc

b

2m mmax

c mRb

b bmax

e

c bR

k

( , )m bmax maxR R

: original mass: original damping: original spring coefficient

of environmente

mbk


Delay Compensation Scheme Spring coefficient modi-

fication Estimation of the spring coeffi-

cient increment perceived by user By using the spring coefficient dis-

tortion in transparency analysis

Modified new spring coefficient By subtracting the spring coeffi-

cient increment from original spring coefficient

34

mincreased h h

bh h

k k k

k k

newh h increasedk k k

, : spring coefficient distortions: original spring coefficient

m bh h

h

k kk


35

Experimental Results Verification of the

transparency anal-ysis and delay compensation scheme Pushing motion of a

virtual object With constant velocity

Haptic devices PHANToM Omni

Network emulation NIST Net


36

Experimental ResultsVerification of trans-parency analysis Force feedback comparison Force feedback 1 Original coefficients (m, b, kh) Delay 300 ms

Force feedback 2 Original spring coefficient Distorted mass and damping coeffi-

cients when delay=300 ms (m=mh, b=bh)

No delay Force feedback 3 Original mass and damping coefficients Distorted spring coefficient when de-

lay=300 ms (kh=khm)

No delay

All force feedbacks are similar -> quantifications of force feedback distortions are ac-ceptable.


37

Experimental ResultsVerification of the proposed scheme Force feedback comparison

Force feedback 1 No delay (transparent force feedback)

Force feedback 2 No delay compensation scheme Delay of 300 ms

Force feedback 3 and 4 Existing schemes with different

initial settings (Fujimoto04) Delay of 300 ms

Force feedback 5 Proposed scheme Delay 300 ms

Force feedback with proposed scheme is most similar to original force feedback -> proposed delay compensation scheme is useful for transparency improvement.

Transparent force feed-back


38

Transmission and Error Control Scheme for Haptic-based NVEs

Related Work Haptic Prioritization Priority-based Filtering Simulation and Experiment

Results


39

Haptic-based NVE over Bandwidth-limited Lossy Network

Transmission control Transmission rate adaptation Transmission rate reduction (traffic reduction)

Haptic interface

HIP position

ith ClientHaptic

rendering (1kHz)

Force Feedback Predicted data

Assumptions- Network unreliability- Limited bandwidth

Data prediction

Force calculation for force feedback

Network transport

ServerHaptic rendering

(1kHz)

Force calculation for virtual objects

& Position update of

virtual object

Network transport

Haptic event transport

Data prediction

Predicted data

Time

Display device

Graphic rendering (30Hz)

Virtual object & HIP position

Visual feedback

Graphic rendering (30Hz)

Virtual object & HIP position

Network monitoring

Network monitoring

Transmission control

Error control

Haptic event transport

Transmission control

Error control

Virtual object position

Focused transmission control


40

Related Work Transmission rate reduction

Statistical approach Focuses on the statistical properties of haptic signal Applies conventional data compression schemes to haptic data

Haptic compression scheme based on Huffman coding (Hikichi01) Haptic compression scheme based on DPCM (differential pulse code

modulation) (McLaughlin02) Problem

Even very good compression on the payload itself is useless if a big share of the necessary network bitrate is wasted by packet headers.

Perception-based approach Packet rate reduction rather than payload compression by using

the limitations of human haptic perception

40 bytes 12 bytes


Related WorkPerception-based approach

Deadband-based filtering (Hirche05; Zadeh08) Events are only sent if the change ex-

ceeds a given threshold value. Prediction-based filtering (Kanbara04; Clarke08) Sender transmits the event only when the

difference between predicted and actual events is larger than a threshold value.

Problem They make the haptic applications very

sensitive to packet losses. Should be used together with a error con-

trol scheme over lossy network.

41


Related Work Error control scheme ARQ (automatic repeat request) Teleoperation layer (LaMarche07)

TCP for critical haptic data Smoothed SCTP (Dodeller2004)

Selective ARQ for last update message FEC (forward error correction) STRON (Cen05)

To compensate undesirable jitter caused by ARQ

Error-correcting code such as Reed-Solomon code

Problem The existing researches use conven-

tional error control schemes for video, audio, and events

Large processing delay for haptic inter-actions

Transparency deterioration

42

STRON (supermedia transport for teleopera-tions over overlay networks)

e.g. robot arm controlcontrol: robot control, feedback1: force, feedback2: video, feedback3: ori-entation

Teleoperation layer


43

Transparency Analysis-based Approach Transparency analysis

The packet loss effect of each haptic event is formulated based on the difference between the actual and predicted positions.

Transmission and error control scheme based on transparency analysis Haptic event prioritization

All haptic events are classified into several groups with different network QoS requirements based on the formulated values.

Priority-based haptic event filtering Based on the prioritization, selects more appropriate data to be

transmitted over bandwidth-limited lossy network. Guarantees less processing delay than existing error control

schemes. By reconstructing lost high-priority event with received low-priority

events without any error-correcting code and retransmission


Haptic Prioritization Comparison between loss ef -

fect (LE) and a threshold value (THHLI)

HLI (haptic event loss index) HLI=0 (HLI0)

Predictable haptic event (LE(n) < THHLI)

HLI=2 (HLI2) Unpredictable haptic event (LE(n) ≥ THHLI)

Critical haptic event HLI=1 (HLI1)

Redundant haptic event Predictable but necessary events

for the lost unpredictable events To compensate the loss effect of

HLI2 event

44

THHLI Derived empirically Set to 0.4 mm

A user cannot perceive the deteriora-tion of interaction quality within the range of 0.01 to 0.4 mm (Hikichi01)

Dat

aSample index

THHLI

Actual sampled data

HLI2 event

Transmitted data with priority-based filtering

Predicted data

HLI1 event

HLI0 event

LE


45

Haptic Prioritization Decision of HLI1 event

Because haptic events include the position every millisecond, two sequential events have similar values.

In order to compensate the loss effect of HLI2 event, several haptic events following a HLI2 event can be HLI1 event.

Maximum number (NHLI1) of HLI1 events is calculated based on Pi Pi: the probability that a HLI2 event and i HLI1

events are all lost.Gilbert loss model α

A sufficiently small value (e.g., 10-4),

Indicating that a packet loss rate of less than α is considered negligible.


Priority-based Filtering Transmission rate reduction

HLI0 events are always filtered. Filtering HLI1 events

When current transmission rate (r) > al-lowable bandwidth (R)

Filtering type Sequential: filtering events sequentially Alternate: filtering events alternately Probability increments of packet losses when

j HLI1 events are filtered (Pi,j

sequential and Pi,jalternate)

Filtering number (j) Probability (Pi,j) that a HLI2 event and i HLI1

events are lost when j HLI1 events are filtered Predefined allowable packet loss probability

(Pallowable)

46

2 1 1

,

1 2

,

2 2

(1 ) (1 )( 2)

(1 )( 2)

( (1 ) ) ( ) ( 2)

j i jalternate singlei j i

j j i jsequential singlei j i

single ii

p q q qP jP u jp q

p q qP jP u jp q

P p q q p q i


47

Priority-based Filtering Haptic event reconstruction (xrec(n))

To reduce loss of fidelity caused by lost HLI2 event Reconstruct lost nth HLI2 event with HLI1 events

Required buffering time (processing time) NHLI1 (ms)

All haptic events are generated every time when haptic rendering module is updated (i.e., every millisecond).

Worst-case scenario for reconstruction of nth haptic event Only n+NHLI1

th haptic event is received successfully

,,

( ) ( )( ) ( ) o orec o n n l

n m n l

x n m x n lx n x n l TT

Tn,n-l : The elapsed time between nth and n-lth eventsAssumptions

n-lth event was received successfullynth HLI2 event is lost but n+mth (1<m<k) HLI1

event is transmitted successfully


48

Simulation Results MATLAB/SIMULINK

Simulation Verification of priority-based

filtering scheme Virtual object contact motion

A haptic device is fixed at a zero point.

A virtual object moves back and forth from -0.1 to 0.

Penetration depth 0~10 cm Transparent force feedback

Force feedback when packet loss rate = 0 %


Simulation Results Verification of trans-

parency with low transmission rate Comparison between prior-

ity-based and deadband-based filterings

Loss rate 20% and allowable bandwidth 70 Kbps Priority-based filtering

Satisfies the transmission rate and transparency re-quirements (error < 0.1N).

Deadband-based filtering Threshold = 2.5 mm

Large force feedback error Threshold = 0.7 mm

Large transmission rate

49


Experimental Results Verification of trans-

parency over lossy network Transparency comparison (peak force feedback error) Prediction-based with no er-

ror control: 0.11N Prediction-based with FEC:

0.3N Prediction-based with ARQ

(μ=25ms, ν=4): 0.1N Prediction-based with ARQ

(μ =4ms, ν=1): 0.104N Priority-based: 0.014N(μ =retransmission timeout, ν=maximum retransmission)

50

Transparent force feed-back


51

Haptic Synchronization Scheme for Force-reflecting Teleoperation

Related Work Haptic Synchronization Simulation and Experiment

Results


EBA with Delay Jitter No instability prob-

lem EBA guarantees stable

teleoperation over net-work delay and packet losses

Limitation It cannot overcome

transparency deteriora-tion according to time-varying network situa-tion.

52


53

Related Work Moving-average adap-

tive buffering (MAB) for remote surgery simula-tion (Wongwirat06) Adaptive buffering control us-

ing moving-average smooth-ing technique Buffering time is adjusted to

twice the moving average delay. Problem: the authors men-

tioned the importance of an optimum buffer size for the transparency but it remained further study.

Remote surgery simulation


54

Related Work

Adaptive buffering control with interpolation scheme (Ber-estesky04) Stability of wave-variable-based teleoperation is guaranteed by compressing

and expanding buffered data. Problem: although this scheme improved performance of position tracking and

stability, it did not focus on transparency of force feedback. VTR (virtual time rendering) (Ishibashi04)

Dynamically adapting the buffering time to improve the interactivity of haptic events in haptic-based NVEs.

Problem: little attention has been given to the transparency in the force-reflect-ing teleoperation.


55

Transparency Analysis-based Approach Transparency analysis

Quantifies the force feedback distortions caused by network delay and packet loss when robot keeps in contact with a wall.

Haptic synchronization scheme based transparency analysis Improves transparency over time-varying delay.

By controlling the playout time of the transmitted haptic event with the transparency-related parameters

By synchronizing the local haptic event with the trans-mitted event according to the transparency analysis


Haptic Synchronization Scheme Maximum allowable

delay (Tal,delay) and loss time (Tal,loss) Fmax : maximum force feed-

back without delay and loss

Predefined transparency requirements Maximum allowable force

feedback decrease by packet loss (Fal,loss)

Maximum allowable force feedback (Fal)

56

,, ,

2

al lossal loss tr loss

m m

FT T

c v

max ,,

2

al loss alal delay

m m

F F FT

c v

, 2 ,( )de loss m m loss tr lossF c v T T , 2de delay m m delayF c v T

Transparency analy-sis


Haptic Synchronization Scheme Transparency im-

provement Output time control of

delayed force Controls playout time of

transmitted event. With transparency-re-

lated parameters Output time control of

local position Synchronizes local event

with transmitted event. To minimize the de-

crease of interaction time caused by delay

57


Haptic Synchronization SchemeOutput time control of delayed force Ideal target output time xn

f

The time at which the event should be output in the case where network jit-ter is always smaller than an estimated maximum network delay jitter Jmax We cannot always output

each event at its xnfn be-

cause there can exist net-work jitters larger than Jmax.

Target output time tnf

By adding total slide time to the ideal target output time

58

1 1 max 1 1 ,1

1 ,

1 1

( ), if , otherwise

( ) ( 2)

f f fal delayf

fal delay

f f f fn n

D A J D T Tx

T T

x x T T n

1 1

1

*

( 2)

( 1)

f f

f fn n n

f fn n n

t x

t x S n

t t S n

: output time of force, : arrival time

: generation time of force

fn n

fn

D A

T

0 1

*

: slide time: total slide time

0, ( 1)

: modified target output time of force

n

n

n n n

fn

SS

S S S S n

t


Haptic Synchronization SchemeOutput time Dn

f

By comparing the arrival time An and the target output time tn

f Virtual time expansion

Delays the target output time Loss rate decrease, total delay increase To minimize the transparency degradation

caused by delay Only when packet loss time is larger than

allowable packet loss time Virtual time contraction

Advances the target output time Total delay decrease, loss rate increase To minimize the transparency degradation

caused by loss Only when the haptic interactions do not

happen

59

,if

loss al loss

fn n n

T T

S A t

Virtual time expan-sion

Virtual time contrac-tion

,

1

if [( or 0) and

( )]

( , , )else 0

m mEBA

f fn n al delay

fn n n n

n

X F

t T T

S min r S t A

S

*

, ,

ff n n nn f

n

A if A tD

t otherwise


Haptic Synchronization Scheme Output time control of local position

In order to minimize the decrease of the interaction time Tin-

ter caused by delay, the playout time of the local position Xm is synchronized with the transmitted haptic event Fds.

Ideal target output time xnP

Output time DnP

If the virtual-time expansion or contraction is executed for transmitted event, the target output time and output time of Xm are also changed.

60

1 1 1 1

1 1

( )

( ) ( 2)

p p f f

p p p pn n

x T t T

x x T T n

1 1

( 2)

p p

p pn n n

p pn n

t x

t x S n

D t

: generation time of positionpnT

: target output time of positionpnt


Simulation Results MATLAB/SIMULINK sim-

ulation Verification of the trans-

parency analysis Wall contact motion

Slave robot keeps in con-tact with the wall.

Haptic device movement vm=0.05 m/s

61

1 2

0.5 , 20000PD controller: _ 200 /EBA: 1000 , 200

s e

s s

M Kg K N mproportional gain N m

c N m c N m

1 2EBA: 1000 , 200m mc N m c N m


Simulation Results Delay effect

Transparency analysis Fde,delay=10∙Tdelay

Delay increase of 100ms -> force decrease of 1N

Simulation results With delay 0~600ms

Loss effect Transparency analysis Ttr,loss=0.002 s Fde,loss=10 ∙(Tloss-0.002)

Packet losses for 50 ms -> force decrease of 0.48N

Simulation results With loss time 2~300ms

62

Simulated results closely follow predicted values -> the transparency analysis is valid


Experimental Results

63

Verification of the proposed scheme No network delay

Standard shape and force (FsEBA) Realistic force feedback (FmEBA)

Transparency comparison With existing schemes (MAB, VTR, skipping) Over time-varying network delay

Pareto-normal distribution average=300 ms, standard deviation=50 ms

Master

Haptic data

transport

Haptic synchronization

FmdMaster EBA

Xm

FmEBA

Slave

Haptic synchronization

Haptic data

transport

Xsd

Fs

Xm

Slave EBA

Xs

FsEBA

PD control

+-

es

Fs

Fds

Xdm

HIP position

Virtual brush position

HIP

Remote calligraphy sys-tem


Experimental Results The user writes the character as feeling the similar

force feedback to the realistic force feedback Force feedback error is less than 0.2 N

The force errors between the force feedbacks for the robot and the standard force 0.6 N (MAB), 0.4 N (VTR), 0.3 N (skipping), and 0.1 N (proposed scheme)

64

0

0.5

1

1.5

2

2.5

0 10 20 30 40 50 60 70 80 90

Forc

e (N

)

Time (s)

FmEBA without delay FmEBA with MABFmEBA with VTR FmEBA with skippingFmEBA with the proposed scheme

0

0.5

1

1.5

2

2.5

0 10 20 30 40 50 60 70 80 90

Forc

e (N

)

Time (s)

FsEBA without delay FsEBA with MABFsEBA with VTR FsEBA with skippingFsEBA with the proposed scheme


Experimental Results User can write the character most similarly to

the standard shape by using the proposed scheme.

With the other schemes, although the user thinks that he or she is writing the character well, unintended tick lines are drawn.

65

with MAB with VTR with skippingwith the proposed scheme


66

Conclusions and Future Work


67

Conclusions Haptic data networking schemes based on

transparency analysis for HNS Delay compensation, transmission and error control and

haptic synchronization schemes Transparency analysis

The force feedback distortions according to network variations are quantified.

The optimization of the adaptation parameters of network-ing schemes for transparency Transmission rate, error control level, buffering time, and

spring coefficient for delay compensation Performance evaluations

Verification of transparency improvement More realistic haptic interactions over time-varying network Comparison with the existing schemes tailored for haptic data


68

Future Work Accuracy and generality improvement of the trans-

parency analysis More simulations and experiments with various haptic interaction

scenarios Other approaches such as series approximations

Other transparency analysis and networking schemes For multi-user HNS

Transparency analysis Transparency degradation according to the haptic inconsistency among

users Group synchronization scheme

For haptic interactions supported by audio and video data Transparency analysis

Transparency deterioration with respect to synchronization error between haptic and the other data

Inter-media synchronization scheme


69

Future Work Combination issues between application

(transport) and network layer solutions For HNS requiring the strict network QoS such as remote

surgery Haptic-specific network compensation and transmission

schemes of application and transport layers QoS-guaranteed haptic data communication services of net-

work layer Application to practical HNS

Virtual museum (Kwon02; Ishibashi04), tangible tele-meet-ing (Kwon05), remote drawing systems (Ishibashi07), net-worked penalty shootout game (Ishibashi08)……

The proposed networking schemes can improve the overall system performance of the practical applications.


70

Questions & Comments

END

transparency improvement of haptic-based networked systems

Documents

haptic feeling

gistnetworked media

multiple haptic data

hapticbased nves transmission

overview of hnshns haptic

khz of haptic rendering

networked media laboratorydept

network problemsinstabilitythe