![Page 1: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/1.jpg)
A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems
T. K. Ho and Jack Y. B. LeeDepartment of Information Engineering
The Chinese University of Hong Kong
Presented by T.K. HoP2P 2003, 1-3 Sep, Linköping, Sweden
![Page 2: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/2.jpg)
Outline
� Background� Previous Works� Transpositional Redundant Data
Update (TRDU)� Multiple Redundant Nodes� Performance Evaluation� Conclusion and Future Works
![Page 3: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/3.jpg)
Background� Server-less Video Streaming Systems[1]
� Pool of user hosts, or call nodes� Each node acts as both sender and receiver� Data redundancy to achieve reliability
Data Nodes
Redundant Nodes
ReceiverNetwork
......
[1] Jack Y. B. Lee and W. T. Leung, “Study of a Server-less Architecture for Video-on-Demand Applications,” Proc. IEEE International Conference on Multimedia and Expo., August 2002.
![Page 4: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/4.jpg)
Background
� Data Placement Policy
v0 v1 v2 v3 c0,0 c0,1
d2d1d0 d3 r0 r1
v4
v8
v12
c1,0
c2,0
c3,0
c1,1
c2,1
c3,1
v5 v6 v7
v9 v10 v11
v13 v14 v15
v16 c4,0 c4,1v17 v18 v19
Same Parity Group
Data Nodes Redundant Nodes
Q-bytes Data Blocks
Q-bytes Redundant Data Blocks
v20 c5,0 c5,1v21 v22 v23
![Page 5: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/5.jpg)
Background� New nodes join the system� Video data blocks are reorganized to utilize their
storage and streaming capacity� Video data blocks in the parity group change� Redundant data blocks need updating� Transmitting video data blocks to redundant nodes
v0 v1 v2 v3 c’0,0 c’0,1
d2d1d0 d3 r0 r1
v6
v12
v18
c’1,0
c’2,0
c’3,0
c’1,1
c’2,1
c’3,1
v7 v8 v9
v13 v14 v15
v19 v20 v21
v4
d4
v10
v16
v22
Data Blocksin Parity GroupCHANGE!
Redundant Data Blocks CHANGE!
v5
v11
v17
v23
d5
![Page 6: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/6.jpg)
Previous Works
� Redundant Data Regeneration� Data blocks are fully distributed
• Transmit all data blocks to redundant node• Compute the new redundant
data in redundant node• Overhead is too high
RedundantNode
Data Nodes ...
![Page 7: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/7.jpg)
� Redundant Data Regeneration� All data blocks stored in central server
• Regenerate new redundant data in server• Transmit them to redundant node• Overhead is low• Costly for server, not desirable/ feasible
Redundant Node
Central Server
Previous Works
![Page 8: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/8.jpg)
Previous Works� Sequential Redundant Data Update (SRDU)
� Old redundant blocks contain valuable information� Possible to reuse for linear code, e.g. RSE code� Vandermonde matrix
• Number of nodes, N• Number of redundant nodes, h• Element at row i, column j = ji-1
block from data node 0
Redundant block in redundant node 0
,0 ,0
,1 ,1
1 1 1, 1 , 1
1 1 1 11 2 3
1 2 3 ( )
i i
i i
h h hi h i N h
c dc dN h
c dN h− − −− − −
− = −
…
…
� �� � � �
…
![Page 9: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/9.jpg)
Previous Works
v0 v1 v2 v3 c’0,0 c’0,1v4
d2d1d0 d3 r0 r1d4
v0 v1 v2 v3 c0,0 c0,1
d2d1d0 d3 r0 r1
0
0,0 1
0,1 2
3
1 1 1 11 2 3 4
vc vc v
v
=
0
1
0,0 2
0,1 3
4
5
' 1 1 1 1 1 1' 1 2 3 4 5 6
vv
c vc v
vv
=
0,1 0 1 2 31* 2* 3* 4*c v v v v= + + +
Only need to send v4, v50,1 0,1 4 5' 5* 6*c c v v= + +
� Updating parity group 0� Before addition of nodes
� After addition of two nodes
v5
d5
![Page 10: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/10.jpg)
Previous Works
� Updating parity group 1� Before addition of nodes
8
2,0 9
2,1 10
11
1 1 1 11 2 3 4
vc vc v
v
=
8
9
1,0 10
1,1 11
6
7
' 1 1 1 1 1 1' 1 2 3 4 5 6
vv
c vc v
vv
=
2,1 8 9 10 111* 2* 3* 4*c v v v v= + + +
1,1 2,1 6 7' 5* 6*c c v v= + +
v8 v9 v10 v11 c2,0 c2,1
d2d1d0 d3 r0 r1
� After addition of two nodes
v6 v7 v8 v9 c’1,0 c’1,1v10
d2d1d0 d3 r0 r1d4
v11
d5
Only need to send v6, v7
![Page 11: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/11.jpg)
Previous Works� Problem in SRDU
� Reusing of old redundant block is limited� c’0,1 can only be in term of one of c0,1 and c1,1
� Both c0,1 and c1,1 are constructed by multiplying 1to 4 with the data blocks
7
0,1 0,14
' ( 1)* jj
c c j v=
= + +∑3
0,1 1,10
' ( 5)* jj
c c j v=
= + +∑
v0 v1 v2 v3 c’0,0 c’0,1
d2d1d0 d3 r0 r1
v8
v4
v12 c’1,0
c’2,0
c’1,1
c’2,1
v9 v10 v11
v5 v6 v7
v13 v14 v15
v16 v17 v18 v19
d4 d5 d6 d7
v20 v21 v22 v23
![Page 12: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/12.jpg)
Transpositional Redundant Data Update� Compute redundant block using
transposed coefficients � Vandermonde matrix
� (Nmax-h) columns� Nmax is max system size, irrespective of current size N
max
,0,0
,1,1 max
1 1 1, 1, 1 max
1 1 1 11 2 3
1 2 3 ( )
ii
ii
h h hi N hi h
dcdc N h
dc N h− − −− −−
− = −
…
…
�� � � � �
…
![Page 13: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/13.jpg)
Transpositional Redundant Data Update� For parity group i, the starting position is
shifted by (i(N-h) mod (Nmax-h))� All other elements are zeroed� Example
� Let (Nmax-h) be 10� Parity group 0
v0 v1 v2 v3 c0,0 c0,1
d2d1d0 d3 r0 r1
v4
v8
c1,0
c2,0
c1,1
c2,1
v5 v6 v7
v9 v10 v11
0
1
20,0
30,1
1 1 1 1 1 11 2 3 4 5 10
0
0
vvv
cv
c
=
�
�
�
0,1 0 1 2 31* 2* 3* 4*c v v v v= + + +
![Page 14: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/14.jpg)
Transpositional Redundant Data Update� Parity group 1 and 2
� Repeat coefficient in c2,1
� Max size of 10 prevents v0 and v10 in residing in the same parity group
v0 v1 v2 v3 c0,0 c0,1
d2d1d0 d3 r0 r1
v4
v8
c1,0
c2,0
c1,1
c2,1
v5 v6 v7
v9 v10 v11
4
7
0000
00
v
v
�
1,1 4 5 6 75* 6* 7* 8*c v v v v= + + +
2,1 8 9 10 119* 10* 1* 2*c v v v v= + + +
10
11
8
9
0
0
vv
vv
�
![Page 15: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/15.jpg)
Transpositional Redundant Data Update
� Two nodes are added� Updating parity group 0
v0 v1 v2 v3 c’0,0 c’0,1
d2d1d0 d3 r0 r1
v6 c’1,0 c’1,1v7 v8 v9
v4
d4
v10
v5
v11
d5
0
1
2
30,0
40,1
5
' 1 1 1 1 1 1' 1 2 3 4 5 10
0
0
vvvv
cv
cv
=
�
�
�
0,1 0 1 2 3 4 5
0,1 4 5
' 1* 2* 3* 4* 5* 6* 5* 6*c v v v v v v
c v v= + + + + += + + Need to send v4, v5
![Page 16: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/16.jpg)
Transpositional Redundant Data Update
v0 v1 v2 v3 c’0,0 c’0,1
d2d1d0 d3 r0 r1
v6 c’1,0 c’1,1v7 v8 v9
v4
d4
v10
v5
v11
d5
10
11
1,0
1,16
7
8
9
0
' 1 1 1 1 1 10
' 1 2 3 4 5 10
vv
cc
vvvv
=
��
�
1,1 6 7 8 9 10 11
1,1 4 5 2,1
' (7* 8* ) (9* 10* 1* 2* ) ( 5* 6* )c v v v v v v
c v v c= + + + + += − − +
No need to send
Cached in previous update
� This equation is not allowed in SRDU, as v6 shares the same coefficient with v10
� Updating parity group 1
![Page 17: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/17.jpg)
Transpositional Redundant Data Update
1,1 8 9 10 11 6 7
2,1 6 7
' (1* 2* 3* 4* ) (5* 6* ) 5* 6*c v v v v v v
c v v= + + + + += + +
8
9
1,0 10
1,1 11
6
7
' 1 1 1 1 1 1' 1 2 3 4 5 6
vv
c vc v
vv
=
1,1 4 5 6 71* 2* 3* 4*c v v v v= + + +
2,1 8 9 10 111* 2* 3* 4*c v v v v= + + +
� In SRDU� Before addition of nodes
� After addition of two nodes
![Page 18: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/18.jpg)
Transpositional Redundant Data Update� Zero overhead point
� Number of additional nodes equals integral multiples of original system size
v0 v1 v2 v3 c’0,0 c’0,1
d2d1d0 d3 r0 r1
v4
d4
v5
d5
� Tradeoffs� Specify the max system size Nmax
� Larger Vandermonde matrix increases the computational complexity in decoding
v6
d6
v7
d7
0,1 0,1 1,1'c c c= +
![Page 19: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/19.jpg)
Multiple Redundant Nodes� Total number of data blocks, B� Redundant Data Regeneration
� Fully distributed data blocks• Total overhead = B + (h-1)(B/(N-h))
All data blocks are sent, overhead = B
All redundant data are constructed here overhead = B / (N-h)
......
...
![Page 20: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/20.jpg)
Multiple Redundant Nodes
� Data blocks are stored in central server• Total overhead = h * B/(N-h)
overhead = B / (N-h)
All redundant data are constructed in server
......
![Page 21: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/21.jpg)
Multiple Redundant Nodes� SRDU/TRDU
� Total overhead = blocks under SRDU/TRDU + (h-1)(B/(N-h))
Blocks under TRDU or SRDU are transmitted
Compute partial result of each new redundant blocks
overhead = B / (N-h)
Combine the old redundant data and received partial result
1,1 1,1'c c P= +
4 55* 6*P v v= +
......
...
![Page 22: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/22.jpg)
Multiple Redundant Nodes
� Overhead is dominated by updating the first redundant node
![Page 23: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/23.jpg)
Performance Evaluation
� Simulation Details� One redundant node� One video title� 40,000 data blocks
� Metric� Redundancy Update Overhead
• Number of video data blocks needed to transmit for generating the new redundant data
![Page 24: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/24.jpg)
Performance Evaluation� Continuous system growth
• Grow from 1 node to 400 nodes• One node added each time
100
1000
10000
100000
0 100 200 300 400
System Size (nodes)
Red
unda
ncy
Upd
ate
Ove
rhea
d (b
lock
s)
Regeneration by Fully Distributed Data
SRDU
TRDU
Regeneration by Archive Server
![Page 25: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/25.jpg)
Performance Evaluation� Batched redundancy update
• Defer update until a fixed number of nodes, W• Initial system size = 80, W ranges from 1 to 200
10
100
1000
10000
100000
0 50 100 150 200
Batch Size W (nodes)
Per-n
ode
Red
unda
ncy
Upd
ate
Ove
rhea
d (b
lock
s)
TRDU, Starting at 80
SRDU, Starting at 80
Zero Overhead Points
![Page 26: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/26.jpg)
Conclusion
� Reduce the redundancy update overhead significantly by� TRDU� Batched update
� Overhead in multiple redundant nodes is investigated� Additional overhead is insignificant
![Page 27: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/27.jpg)
Future Works
� Node Removal� Addition of redundant nodes� Scheduling problem� Optimal batch size
![Page 28: Server-less Video Streaming Systems Update Algorithm for ... · A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems T. K. Ho and Jack](https://reader035.vdocuments.mx/reader035/viewer/2022062920/5f028cef7e708231d404d26d/html5/thumbnails/28.jpg)
Thank you
� Q&A Section