transparency improvement of haptic-based networked systems
Post on 24-Feb-2016
34 Views
Preview:
DESCRIPTION
TRANSCRIPT
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Networked Media LaboratoryDept. of Information & Communications
Gwangju Institute of Science & Technology (GIST)shlee@nm.gist.ac.kr
http://nm.gist.ac.kr/~shlee
1
Transparency Improvement of Haptic-based Networked Systems
Seokhee Lee
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
2
Contents Introduction
Transparency Analysis for Haptic-based Net-worked Systems
Delay Compensation Scheme for Haptic-based NVEs
Transmission and Error Control Scheme for Haptic-based NVEs
Haptic Synchronization Scheme for Force-re-flecting Teleoperation
Conclusions and Future Work
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
3
Introduction
Overview of HNS Motivation and Contributions
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Overview of HNS HNS (haptic-based networked system)
System which provides haptic feeling for a user who interacts with remote environments
Haptic-based NVE (networked virtual environment) + force-reflecting teleoperation
Local haptic system + remote haptic systems
4
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
5
Requirements of HNS Main goal of HNS
Executing a task in a remote environments (real or virtual) With stability and transparency
Stability The primary requisite for safe system Instability
Uncontrollable oscillations and chaotic behavior Factors causing instabilities
Quantization error, time delay, packet losses, asynchronous switching between continuous- and discrete-time subsystems ……
Transparency Transparency ≈ haptic realism Mathematically more difficult to analyze since the ultimate goal is
to make the user experience a “good feeling” Optimal transparency
The user cannot distinguish between direct and tele-interaction with a remote environment.
Focused requirement
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
6
Network Problems Instability
The delayed and lost haptic data destabilize HNS. The instability may cause serious damages to the user.
Transparency deterioration The human haptic sense is more sensitive than the visual
and auditory senses. Because of the sensitivity, the transparency is deteriorated
severely with network variations of delay and loss. High transmission rate
A high update rate (approximately 1 kHz) of haptic rendering leads to high transmission rate of 1000 packets/s.
The available network bandwidth for multiple haptic data may not be sufficient over current Internet.
Focused network problems
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
7
Existing Haptic Data Net-working Schemes Network layer solutions
Improve the performance of the network itself for HNS. QoS provision for haptic data
Class-based weight fair queue (Marshall08), DiffServ & IntServ (Hirche05)
QoS routing for haptic data QoS-guaranteed overlay routing (Cen05)
High speed network for haptic data Optical networks (LaMarche07)
Application and transport layer solutions Improve the system performance on the assumption that
the current network performance is not sufficient. Delay/jitter compensation Transmission control Error control Focused networking
schemes
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Motivation
Problems of the existing networking schemes (application and transport layer) There was little consideration to enhance the human hap-
tic perception. Network adaptation schemes for conventional multimedia
were applied to the haptic interaction without careful consideration of the difference between the haptic event and the other data.
Remaining challenge Minimizing the transparency (haptic realism) degradation
By determining the optimum adaptation parameters for transparent haptic interactions
Transmission rate, error control level, buffering time, spring co-efficient for delay compensation,……
8
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Contributions Distinguishing point of the proposed net-
working schemes Consideration of human perception characteristics (trans-
parency) for the optimization Transparency analysis
Deterioration of haptic interaction quality caused by network delay is quantified as distortions of mass, damping, and spring coefficients of a virtual object.
In order to formulate the importance of a haptic event with respect to a packet loss, loss effect of each haptic event is quantified for haptic-based NVEs with a prediction algorithm.
For force-reflecting teleoperations stabilized by a control al-gorithm, the force feedback distortions caused by network delay and packet loss are quantified when robot keeps in contact with a wall.
9
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
10
Contributions Networking schemes based on transparency
analysis Delay compensation scheme
Allowable delay prediction Predicts the maximum allowable delay based on the quantified
values of mass and damping distortions. Spring coefficient modification
Modifies spring coefficient according to delay based on the quanti-fied values of spring coefficients’ distortions.
Verification Experimental results
The proposed scheme improves the haptic interaction quality more ef -ficiently compared with existing delay compensation schemes while avoiding unnecessary trial-and-errors over network delay.
Representative paper Elsevier Computer Communications, 2009
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
11
Contributions Transmission and error control scheme
Haptic event prioritization Based on the quantified loss effect, all haptic events are classified into sev-
eral groups with different network QoS requirements. Priority-based haptic event filtering
Based on the prioritization, select more appropriate data to be transmitted for realistic haptic interaction over bandwidth-limited lossy network.
Verification Simulation and experiment results
The proposed scheme provides lower packet rate than the existing schemes for a transparent haptic interaction over a bandwidth-limited network.
The proposed scheme guarantees less processing delay and better haptic interac-tion quality compared with previous error control schemes over a lossy network.
Representative paper Springer Multimedia Systems, 2009
Patent Haptic event transport method for haptic-based collaborative virtual environment
and system therefor
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
12
Contributions Haptic synchronization (buffering) scheme
Allowable delay and loss time prediction Predicts maximum allowable delay and loss time based on quanti-
fied force feedback distortions and predefined transparency re-quirements.
Network-adaptive buffering time control Improves transparency by controlling the playout time of the
transmitted haptic event with the transparency-related parameters from transparency analysis.
Verification Remote calligraphy system
The proposed scheme guarantees less force feedback error compared with the existing haptic synchronization schemes over time-varying network delay.
Representative paper ACM NetGames, 2009
Patent application The method for synchronizing haptic data and the haptic system
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
13
Transparency Analysis for Haptic-based Networked System
Transparency Deterioration Caused by Network Variations
Transparency Analysis for Hap-tic-based NVEs
Transparency Analysis for EBA-based Teleoperations
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Transparency Deterioration Caused by Network Variations Transparency deterio-
ration caused by delay If there is network delay while a
user manipulates a virtual object, the virtual object does not move immediately.
During that time period, penetra-tion depth increases.
Eventually, the force feedback in spring-damper model also in-creases.
14
<Spring-damper model>force_feedback = spring_coefficient penetration_depth+damping_coefficient velocity
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
15
Transparency Deterioration Caused by Network Variations Transparency deterio-
ration caused by packet loss Haptic-based NVE without a
data prediction scheme The force feedback also in-
creases in the same manner of the delay case.
Sudden movement Haptic-based NVE with a
data prediction scheme If the haptic event can be pre-
dicted based on past patterns of events, the quality degradation will be small.
Otherwise, the quality degrada-tion is more severe.
With data prediction scheme<Original movement of a virtual ob-
ject>
<Unintended movement caused by loss>
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
16
Transparency Deterioration Caused by Network Variations Transparency deteriora-
tion caused by delay jit-ter More severe deterioration of hap-
tic interaction quality Out-of-order arrivals as well as de-
layed data transmission and empty sampling instances
(Lee06)
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Transparency Analysis for HNS Quantification of transparency deterio-
ration according to network delay and packet loss In order to determine the optimum adaptation pa-
rameters of haptic data networking schemes Focused HNS
Haptic-based NVEs With CS (client-server) architectures
Force-reflecting teleoperation With EBA (energy bounding algorithm)
17
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
18
Transparency Analysis for Haptic-based NVEs Position-position interaction
HIP (haptic interaction pointer) position (client -> server) Virtual object position (server -> client)
Consistency server Updates movements of all virtual objects.
Distributed force calculation Each client calculates force feedback by using spring-damper model.
1
( ) ( ) ( )i
o o ein
mx t bx t f t
( ) ( - )outei hi ix t x t T
( ) -{ ( ( ) - ( )) ( ( ) - ( ))}ei ei o ei ei o eif t k x t x t b x t x t
( ) ( ( ) - ( )) ( ( ) - ( ))d dhi hi o hi hi oi hif t k x t x t b x t x t
( ) = ( )d inoi o ix t x t T
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
19
Transparency Analysis for Haptic-based NVEs Transparency
Equality between the human (Zh) and the environment (Ze) impedances
Simplification The number of clients is not
related to transparency. A virtual object is repre-
sented by a mass and damp-ing.
Impedance
2
3 2
( ) ( )( ) 2 2( )( ) ( ) ( )
2 2 2
h hh h h e
hh
ehe
k Rm k Rbs k m s k b k k RF sZ s RkRm bRX s s m s b s k
2
( )( )( )
e e ee
e e
F s k ms k bZ sX s ms bs k
( )e eG s k
( )h hG s k2
1( )oG sms bs
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
20
Transparency Analysis for Haptic-based NVEs Magnitude responses
of impedances with respect to network delay Confirm that if delay in-
creases, force feedback also increases
It is called transparency (force feedback) distor-tion Mass and damping dis-
tortion (mh, bh) Spring coefficient distor-
tion (khm, kh
b)
kh=0.5, ke=0.5, m=0.25, b=0.0025
R=RTT (round trip time)
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
21
Transparency Analysis for Haptic-based NVEs Mass and damping distortion
If delay increases, a user perceives a virtual object as having larger mass and damping coefficients than the actual values.
Quantification of mass (mh) and damping (bh) distortion Low frequency approximation
Human operator usually generates low frequency input (Hirche05) [10-6~102]
Approximated human impedance
Transparency condition
Distorted mass
Distorted damping
2
( ) ( )2( )
( ) ( )2 2
hh h h e
apph
ee
k Rb k m s k b k k RZ s
RkbR m s b s k
2hRbm m
h eb b k R
( ) ( )apph eZ s Z s
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
22
Transparency Analysis for Haptic-based NVEs Distortion of spring coefficient
If delay exists, user is provided with unrealistically large force feedback that seems to be obtained with the larger spring coefficient than the actual value.
Quantification of spring coefficient distortion Mass-based distortion (kh
m)
Damping-based distortion (khb)
( )2 2
m m hh h h h
k RbRbk m k m k km
( )b b h eh h e h h
k k Rk b k b k R k k
b
If 2 then
If 2 then
If 2 then
m be h h
me h
be h
b mk k k
b mk k
b mk k
If virtual object dynamics are affected by mass more than by damping, spring distortion follows mass-based distortion; otherwise it follows damping-based distortion.
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Transparency Analysis for Haptic-based NVEs Force feedback distortion
according to packet losses Without a prediction scheme
Quantified in the same manner as the force distortions according to network delay
By using loss time (Tloss) With a prediction scheme (fdis(n))
Quantified by using the difference between the actual and the pre-dicted (xpre(n)) positions
Loss effect (LE(n)) of the nth haptic event
In order to formulate the impor-tance of a haptic event with re-spect to a packet loss
23
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Transparency Analysis for EBA-based Teleoperation
EBA (energy bounding algorithm) Stability algorithm of a haptic simulation system (Kim04)
EBA restricts the energy generated in the ZOH (zero order hold) within a consumable energy limit in the haptic device.
Can be applied to teleoperation to ensure robust stability regardless of the amount of time delays and packet losses (Seo08).
24
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Transparency Analysis for EBA-based Teleoperation
Master EBA
25
Slave EBA
,max ,max
,min
( ) ( 1) ( ) ( )where
( ) ( 1)( ) ( ) 0
( )( ) ( ) ( 1),
( ) ( ) ( ),with the following bounding laws:
( ) ( ) ( ) ( )
( ) (
sEBA sEBA s s
s sEBAs s
s
s s s
s sd s
s s s s
s s
F n F n n e n
F n F nn for e n
e ne n e n e n
e n x n x n
if n n then n n
if n n
,min
,max 1 ,max
,min ,min
,01 1
2
0
2
2,max 2 2
2,min 2 2
) ( ) ( )
where( ) min( , ( )),
( ) ( ),
( ),
( 1)
( 1) ( 1)( ) ,( ) ( )
( 1) ( 1( )
( )
s s
s s s
s s
SD SDs n
sk
s ss s s
s s
s ss s s
s
then n n
n c n
n n
P nc
e k
F n F nn c ce n e n
F n F nn c c
x n
2
2
),
( )
where is a positive constant.s
s
x n
c
,max ,max
,min ,min
,max 1 ,max
,min ,min
,01 1
2
0
,max
with the following bounding laws:( ) ( ) ( ) ( )
( ) ( ) ( ) ( )
where( ) min( , ( )),
( ) ( ),
( ),
( 1)
(
m m m m
m m m m
m m m
m m
MD MDm n
mk
m
if n n then n n
if n n then n n
n c n
n n
P nc
X k
n
2
22 2
2
2,min 2 2
2
( 1) ( 1)) ,
( ) ( )
( 1) ( 1)( ) ,
( ) ( )
where is a positive constant.
m mm m
m m
m mm m m
m m
m
F n F nc c
X n X n
F n F nn c c
X n X n
c
( ) ( 1) ( ) ( )where
( ) ( 1)( ) ( ) 0( )
( ) ( ) ( 1)
mEBA mEBA m m
md mEBAm m
m
m m m
F n F n n X n
F n F nn for X nX n
X n X n X n
Control law Control law
Bounding law Bounding law
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Transparency Analysis for EBA-based Teleoperation Definition of Transparency
Similarity between the force feedback for a slave robot (FsEBA) and that for a user in master side (FmEBA)
Assumptions Robot keeps in contact with a wall and a user feels the force feed-
back by moving haptic device facing the wall with constant veloc-ity (vm).
Force feedback increase according to the user's input From the control law in master EBA
26
/
1
/
,max ,max1
/
1 ,max1
( ),
( )
min( , ( ))
inter
inter
inter
T
in m mn
T
in m mn
T
m m mn
F v n
F v n
v c n
,
: update time period ( ): interaction time ( )
: increase of ( ): maximum value of ( )
inter
in mEBA
in max in
secT secF F NF F N
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Transparency Analysis for EBA-based Teleoperation Approximation of force
feedback increase Assumption: c1m≥2c2m
γm,max(n) in bounding laws con-verges into c2m when Fm(n-1) increases monotonically.
Force feedback decrease caused by delay Network delay reduces Tinter as
much as the delayed time. If the network delay in-
creases, the force feedback decreases in proportional to c2m∙vm.
27
,max 2in m m interF c v T
, 2de delay m m delayF c v T
, : decrease according to delay
: network delayde delay mEBA
delay
F F
T
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Transparency Analysis for EBA-based Teleoperation Force feedback de-
crease (Fde,loss) caused by loss Packet losses reduces Tinter as
much as the loss period. Transparent loss time (Ttr,loss):
time period when the force feedback continuously in-creases even though there exist the packet losses
28
,,
2
loss mEBA losstr loss
m m
F FT
c v
, 2 ,( )de loss m m loss tr lossF c v T T
, mEBA
: the latest received force feedback before the packet losses
: the latest updated F
before the packet losses: loss time (time period when
the pack
loss
mEBA loss
loss
F
F
Tet losses happen)
Loss
F mEB
A
Time
Floss
FmEBA,loss
Ttr,loss
Tloss
Beginning of loss
End of loss
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
29
Delay Compensation Schemefor Haptic-based NVEs
Related Work Delay Compensation Scheme Experimental Results
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
30
Related Work Spring coefficient modification scheme (Fuji-
moto04) Dynamically changes the reaction force applied to a user by
adjusting a spring coefficient of a spring-damper model. The spring coefficient (Kh) is modified according to the current
end-to-end delay (ΔT).
Problems An accurate spring coefficient for the realistic haptic interac-
tion can only be found by a process of trial and error. A significant challenge remains in developing an optimum
method of determining the spring coefficient.
( , )
_ _
where denotes initial spring coefficient
inith h
inith
inith
K min K b T
b K initial buffering time
K
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
31
Transparency Analysis-based Approach Transparency Analysis
Transparency deterioration caused by delay is quantified as distortions of mass, damping, and spring coefficients of virtual objects.
Delay compensation scheme based on transparency analysis Predicts the maximum allowable delay from the
quantified values of mass and damping distor-tions.
Modifies the spring coefficient according to net-work delay based on the quantified value of spring coefficient distortion.
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
32
Delay Compensation Scheme
Haptic interface
HIP position
ith ClientHaptic & graphic
rendering
Allowable delay
prediction
Force Feedback
New spring coefficient (kh
new)
Assumption- Time- varying network delay
Network monitoring
Spring coefficient
modification
Calculation of force feedback
Maximum allowable delay
(Rmaxm, Rmax
b)
Current network
delay (R)
Network transport
Haptic event
transport
ServerHaptic & graphic
rendering
Update of virtual object
Network transport
Haptic event
transport
Virtual object
position
Delay com-pensation scheme
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
33
Delay Compensation Scheme Allowable delay prediction
Quantified mass and damping distortions are proportional to network delay.
Predefined transparency requirements Maximum allowable mass (mallowable) and damping (ballow-
able) Allowable mass and damping gradients (cm, cb)
Mass-based:
Damping-based:
allowablem
m mc
m
allowableb
b bc
b
2m mmax
c mRb
b bmax
e
c bR
k
( , )m bmax maxR R
: original mass: original damping: original spring coefficient
of environmente
mbk
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Delay Compensation Scheme Spring coefficient modi-
fication Estimation of the spring coeffi-
cient increment perceived by user By using the spring coefficient dis-
tortion in transparency analysis
Modified new spring coefficient By subtracting the spring coeffi-
cient increment from original spring coefficient
34
mincreased h h
bh h
k k k
k k
newh h increasedk k k
, : spring coefficient distortions: original spring coefficient
m bh h
h
k kk
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
35
Experimental Results Verification of the
transparency anal-ysis and delay compensation scheme Pushing motion of a
virtual object With constant velocity
Haptic devices PHANToM Omni
Network emulation NIST Net
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
36
Experimental ResultsVerification of trans-parency analysis Force feedback comparison Force feedback 1 Original coefficients (m, b, kh) Delay 300 ms
Force feedback 2 Original spring coefficient Distorted mass and damping coeffi-
cients when delay=300 ms (m=mh, b=bh)
No delay Force feedback 3 Original mass and damping coefficients Distorted spring coefficient when de-
lay=300 ms (kh=khm)
No delay
All force feedbacks are similar -> quantifications of force feedback distortions are ac-ceptable.
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
37
Experimental ResultsVerification of the proposed scheme Force feedback comparison
Force feedback 1 No delay (transparent force feedback)
Force feedback 2 No delay compensation scheme Delay of 300 ms
Force feedback 3 and 4 Existing schemes with different
initial settings (Fujimoto04) Delay of 300 ms
Force feedback 5 Proposed scheme Delay 300 ms
Force feedback with proposed scheme is most similar to original force feedback -> proposed delay compensation scheme is useful for transparency improvement.
Transparent force feed-back
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
38
Transmission and Error Control Scheme for Haptic-based NVEs
Related Work Haptic Prioritization Priority-based Filtering Simulation and Experiment
Results
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
39
Haptic-based NVE over Bandwidth-limited Lossy Network
Transmission control Transmission rate adaptation Transmission rate reduction (traffic reduction)
Haptic interface
HIP position
ith ClientHaptic
rendering (1kHz)
Force Feedback Predicted data
Assumptions- Network unreliability- Limited bandwidth
Data prediction
Force calculation for force feedback
Network transport
ServerHaptic rendering
(1kHz)
Force calculation for virtual objects
& Position update of
virtual object
Network transport
Haptic event transport
Data prediction
Predicted data
Time
Display device
Graphic rendering (30Hz)
Virtual object & HIP position
Visual feedback
Graphic rendering (30Hz)
Virtual object & HIP position
Network monitoring
Network monitoring
Transmission control
Error control
Haptic event transport
Transmission control
Error control
Virtual object position
Focused transmission control
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
40
Related Work Transmission rate reduction
Statistical approach Focuses on the statistical properties of haptic signal Applies conventional data compression schemes to haptic data
Haptic compression scheme based on Huffman coding (Hikichi01) Haptic compression scheme based on DPCM (differential pulse code
modulation) (McLaughlin02) Problem
Even very good compression on the payload itself is useless if a big share of the necessary network bitrate is wasted by packet headers.
Perception-based approach Packet rate reduction rather than payload compression by using
the limitations of human haptic perception
40 bytes 12 bytes
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Related WorkPerception-based approach
Deadband-based filtering (Hirche05; Zadeh08) Events are only sent if the change ex-
ceeds a given threshold value. Prediction-based filtering (Kanbara04; Clarke08) Sender transmits the event only when the
difference between predicted and actual events is larger than a threshold value.
Problem They make the haptic applications very
sensitive to packet losses. Should be used together with a error con-
trol scheme over lossy network.
41
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Related Work Error control scheme ARQ (automatic repeat request) Teleoperation layer (LaMarche07)
TCP for critical haptic data Smoothed SCTP (Dodeller2004)
Selective ARQ for last update message FEC (forward error correction) STRON (Cen05)
To compensate undesirable jitter caused by ARQ
Error-correcting code such as Reed-Solomon code
Problem The existing researches use conven-
tional error control schemes for video, audio, and events
Large processing delay for haptic inter-actions
Transparency deterioration
42
STRON (supermedia transport for teleopera-tions over overlay networks)
e.g. robot arm controlcontrol: robot control, feedback1: force, feedback2: video, feedback3: ori-entation
Teleoperation layer
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
43
Transparency Analysis-based Approach Transparency analysis
The packet loss effect of each haptic event is formulated based on the difference between the actual and predicted positions.
Transmission and error control scheme based on transparency analysis Haptic event prioritization
All haptic events are classified into several groups with different network QoS requirements based on the formulated values.
Priority-based haptic event filtering Based on the prioritization, selects more appropriate data to be
transmitted over bandwidth-limited lossy network. Guarantees less processing delay than existing error control
schemes. By reconstructing lost high-priority event with received low-priority
events without any error-correcting code and retransmission
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Haptic Prioritization Comparison between loss ef -
fect (LE) and a threshold value (THHLI)
HLI (haptic event loss index) HLI=0 (HLI0)
Predictable haptic event (LE(n) < THHLI)
HLI=2 (HLI2) Unpredictable haptic event (LE(n) ≥ THHLI)
Critical haptic event HLI=1 (HLI1)
Redundant haptic event Predictable but necessary events
for the lost unpredictable events To compensate the loss effect of
HLI2 event
44
THHLI Derived empirically Set to 0.4 mm
A user cannot perceive the deteriora-tion of interaction quality within the range of 0.01 to 0.4 mm (Hikichi01)
Dat
aSample index
THHLI
Actual sampled data
HLI2 event
Transmitted data with priority-based filtering
Predicted data
HLI1 event
HLI0 event
LE
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
45
Haptic Prioritization Decision of HLI1 event
Because haptic events include the position every millisecond, two sequential events have similar values.
In order to compensate the loss effect of HLI2 event, several haptic events following a HLI2 event can be HLI1 event.
Maximum number (NHLI1) of HLI1 events is calculated based on Pi Pi: the probability that a HLI2 event and i HLI1
events are all lost.Gilbert loss model α
A sufficiently small value (e.g., 10-4),
Indicating that a packet loss rate of less than α is considered negligible.
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Priority-based Filtering Transmission rate reduction
HLI0 events are always filtered. Filtering HLI1 events
When current transmission rate (r) > al-lowable bandwidth (R)
Filtering type Sequential: filtering events sequentially Alternate: filtering events alternately Probability increments of packet losses when
j HLI1 events are filtered (Pi,j
sequential and Pi,jalternate)
Filtering number (j) Probability (Pi,j) that a HLI2 event and i HLI1
events are lost when j HLI1 events are filtered Predefined allowable packet loss probability
(Pallowable)
46
2 1 1
,
1 2
,
2 2
(1 ) (1 )( 2)
(1 )( 2)
( (1 ) ) ( ) ( 2)
j i jalternate singlei j i
j j i jsequential singlei j i
single ii
p q q qP jP u jp q
p q qP jP u jp q
P p q q p q i
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
47
Priority-based Filtering Haptic event reconstruction (xrec(n))
To reduce loss of fidelity caused by lost HLI2 event Reconstruct lost nth HLI2 event with HLI1 events
Required buffering time (processing time) NHLI1 (ms)
All haptic events are generated every time when haptic rendering module is updated (i.e., every millisecond).
Worst-case scenario for reconstruction of nth haptic event Only n+NHLI1
th haptic event is received successfully
,,
( ) ( )( ) ( ) o orec o n n l
n m n l
x n m x n lx n x n l TT
Tn,n-l : The elapsed time between nth and n-lth eventsAssumptions
n-lth event was received successfullynth HLI2 event is lost but n+mth (1<m<k) HLI1
event is transmitted successfully
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
48
Simulation Results MATLAB/SIMULINK
Simulation Verification of priority-based
filtering scheme Virtual object contact motion
A haptic device is fixed at a zero point.
A virtual object moves back and forth from -0.1 to 0.
Penetration depth 0~10 cm Transparent force feedback
Force feedback when packet loss rate = 0 %
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Simulation Results Verification of trans-
parency with low transmission rate Comparison between prior-
ity-based and deadband-based filterings
Loss rate 20% and allowable bandwidth 70 Kbps Priority-based filtering
Satisfies the transmission rate and transparency re-quirements (error < 0.1N).
Deadband-based filtering Threshold = 2.5 mm
Large force feedback error Threshold = 0.7 mm
Large transmission rate
49
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Experimental Results Verification of trans-
parency over lossy network Transparency comparison (peak force feedback error) Prediction-based with no er-
ror control: 0.11N Prediction-based with FEC:
0.3N Prediction-based with ARQ
(μ=25ms, ν=4): 0.1N Prediction-based with ARQ
(μ =4ms, ν=1): 0.104N Priority-based: 0.014N(μ =retransmission timeout, ν=maximum retransmission)
50
Transparent force feed-back
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
51
Haptic Synchronization Scheme for Force-reflecting Teleoperation
Related Work Haptic Synchronization Simulation and Experiment
Results
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
EBA with Delay Jitter No instability prob-
lem EBA guarantees stable
teleoperation over net-work delay and packet losses
Limitation It cannot overcome
transparency deteriora-tion according to time-varying network situa-tion.
52
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
53
Related Work Moving-average adap-
tive buffering (MAB) for remote surgery simula-tion (Wongwirat06) Adaptive buffering control us-
ing moving-average smooth-ing technique Buffering time is adjusted to
twice the moving average delay. Problem: the authors men-
tioned the importance of an optimum buffer size for the transparency but it remained further study.
Remote surgery simulation
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
54
Related Work
Adaptive buffering control with interpolation scheme (Ber-estesky04) Stability of wave-variable-based teleoperation is guaranteed by compressing
and expanding buffered data. Problem: although this scheme improved performance of position tracking and
stability, it did not focus on transparency of force feedback. VTR (virtual time rendering) (Ishibashi04)
Dynamically adapting the buffering time to improve the interactivity of haptic events in haptic-based NVEs.
Problem: little attention has been given to the transparency in the force-reflect-ing teleoperation.
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
55
Transparency Analysis-based Approach Transparency analysis
Quantifies the force feedback distortions caused by network delay and packet loss when robot keeps in contact with a wall.
Haptic synchronization scheme based transparency analysis Improves transparency over time-varying delay.
By controlling the playout time of the transmitted haptic event with the transparency-related parameters
By synchronizing the local haptic event with the trans-mitted event according to the transparency analysis
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Haptic Synchronization Scheme Maximum allowable
delay (Tal,delay) and loss time (Tal,loss) Fmax : maximum force feed-
back without delay and loss
Predefined transparency requirements Maximum allowable force
feedback decrease by packet loss (Fal,loss)
Maximum allowable force feedback (Fal)
56
,, ,
2
al lossal loss tr loss
m m
FT T
c v
max ,,
2
al loss alal delay
m m
F F FT
c v
, 2 ,( )de loss m m loss tr lossF c v T T , 2de delay m m delayF c v T
Transparency analy-sis
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Haptic Synchronization Scheme Transparency im-
provement Output time control of
delayed force Controls playout time of
transmitted event. With transparency-re-
lated parameters Output time control of
local position Synchronizes local event
with transmitted event. To minimize the de-
crease of interaction time caused by delay
57
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Haptic Synchronization SchemeOutput time control of delayed force Ideal target output time xn
f
The time at which the event should be output in the case where network jit-ter is always smaller than an estimated maximum network delay jitter Jmax We cannot always output
each event at its xnfn be-
cause there can exist net-work jitters larger than Jmax.
Target output time tnf
By adding total slide time to the ideal target output time
58
1 1 max 1 1 ,1
1 ,
1 1
( ), if , otherwise
( ) ( 2)
f f fal delayf
fal delay
f f f fn n
D A J D T Tx
T T
x x T T n
1 1
1
*
( 2)
( 1)
f f
f fn n n
f fn n n
t x
t x S n
t t S n
: output time of force, : arrival time
: generation time of force
fn n
fn
D A
T
0 1
*
: slide time: total slide time
0, ( 1)
: modified target output time of force
n
n
n n n
fn
SS
S S S S n
t
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Haptic Synchronization SchemeOutput time Dn
f
By comparing the arrival time An and the target output time tn
f Virtual time expansion
Delays the target output time Loss rate decrease, total delay increase To minimize the transparency degradation
caused by delay Only when packet loss time is larger than
allowable packet loss time Virtual time contraction
Advances the target output time Total delay decrease, loss rate increase To minimize the transparency degradation
caused by loss Only when the haptic interactions do not
happen
59
,if
loss al loss
fn n n
T T
S A t
Virtual time expan-sion
Virtual time contrac-tion
,
1
if [( or 0) and
( )]
( , , )else 0
m mEBA
f fn n al delay
fn n n n
n
X F
t T T
S min r S t A
S
*
, ,
ff n n nn f
n
A if A tD
t otherwise
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Haptic Synchronization Scheme Output time control of local position
In order to minimize the decrease of the interaction time Tin-
ter caused by delay, the playout time of the local position Xm is synchronized with the transmitted haptic event Fds.
Ideal target output time xnP
Output time DnP
If the virtual-time expansion or contraction is executed for transmitted event, the target output time and output time of Xm are also changed.
60
1 1 1 1
1 1
( )
( ) ( 2)
p p f f
p p p pn n
x T t T
x x T T n
1 1
( 2)
p p
p pn n n
p pn n
t x
t x S n
D t
: generation time of positionpnT
: target output time of positionpnt
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Simulation Results MATLAB/SIMULINK sim-
ulation Verification of the trans-
parency analysis Wall contact motion
Slave robot keeps in con-tact with the wall.
Haptic device movement vm=0.05 m/s
61
1 2
0.5 , 20000PD controller: _ 200 /EBA: 1000 , 200
s e
s s
M Kg K N mproportional gain N m
c N m c N m
1 2EBA: 1000 , 200m mc N m c N m
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Simulation Results Delay effect
Transparency analysis Fde,delay=10∙Tdelay
Delay increase of 100ms -> force decrease of 1N
Simulation results With delay 0~600ms
Loss effect Transparency analysis Ttr,loss=0.002 s Fde,loss=10 ∙(Tloss-0.002)
Packet losses for 50 ms -> force decrease of 0.48N
Simulation results With loss time 2~300ms
62
Simulated results closely follow predicted values -> the transparency analysis is valid
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Experimental Results
63
Verification of the proposed scheme No network delay
Standard shape and force (FsEBA) Realistic force feedback (FmEBA)
Transparency comparison With existing schemes (MAB, VTR, skipping) Over time-varying network delay
Pareto-normal distribution average=300 ms, standard deviation=50 ms
Master
Haptic data
transport
Haptic synchronization
FmdMaster EBA
Xm
FmEBA
Slave
Haptic synchronization
Haptic data
transport
Xsd
Fs
Xm
Slave EBA
Xs
FsEBA
PD control
+-
es
Fs
Fds
Xdm
HIP position
Virtual brush position
HIP
Remote calligraphy sys-tem
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Experimental Results The user writes the character as feeling the similar
force feedback to the realistic force feedback Force feedback error is less than 0.2 N
The force errors between the force feedbacks for the robot and the standard force 0.6 N (MAB), 0.4 N (VTR), 0.3 N (skipping), and 0.1 N (proposed scheme)
64
0
0.5
1
1.5
2
2.5
0 10 20 30 40 50 60 70 80 90
Forc
e (N
)
Time (s)
FmEBA without delay FmEBA with MABFmEBA with VTR FmEBA with skippingFmEBA with the proposed scheme
0
0.5
1
1.5
2
2.5
0 10 20 30 40 50 60 70 80 90
Forc
e (N
)
Time (s)
FsEBA without delay FsEBA with MABFsEBA with VTR FsEBA with skippingFsEBA with the proposed scheme
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
Experimental Results User can write the character most similarly to
the standard shape by using the proposed scheme.
With the other schemes, although the user thinks that he or she is writing the character well, unintended tick lines are drawn.
65
with MAB with VTR with skippingwith the proposed scheme
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
66
Conclusions and Future Work
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
67
Conclusions Haptic data networking schemes based on
transparency analysis for HNS Delay compensation, transmission and error control and
haptic synchronization schemes Transparency analysis
The force feedback distortions according to network variations are quantified.
The optimization of the adaptation parameters of network-ing schemes for transparency Transmission rate, error control level, buffering time, and
spring coefficient for delay compensation Performance evaluations
Verification of transparency improvement More realistic haptic interactions over time-varying network Comparison with the existing schemes tailored for haptic data
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
68
Future Work Accuracy and generality improvement of the trans-
parency analysis More simulations and experiments with various haptic interaction
scenarios Other approaches such as series approximations
Other transparency analysis and networking schemes For multi-user HNS
Transparency analysis Transparency degradation according to the haptic inconsistency among
users Group synchronization scheme
For haptic interactions supported by audio and video data Transparency analysis
Transparency deterioration with respect to synchronization error between haptic and the other data
Inter-media synchronization scheme
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
69
Future Work Combination issues between application
(transport) and network layer solutions For HNS requiring the strict network QoS such as remote
surgery Haptic-specific network compensation and transmission
schemes of application and transport layers QoS-guaranteed haptic data communication services of net-
work layer Application to practical HNS
Virtual museum (Kwon02; Ishibashi04), tangible tele-meet-ing (Kwon05), remote drawing systems (Ishibashi07), net-worked penalty shootout game (Ishibashi08)……
The proposed networking schemes can improve the overall system performance of the practical applications.
DEPT. OF INFO. & COMM., GISTNetworked Media Lab.
70
Questions & Comments
END
top related