antfarm: efficient content distribution with …...antfarm: efficient content distribution with...
TRANSCRIPT
![Page 1: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/1.jpg)
Antfarm: Efficient Content Distribution with Managed Swarms
Ryan S. Peterson and Emin Gün Sirer
Department of Computer Science, Cornell UniversityUnited Networks, LLC
April 22, 2009
![Page 2: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/2.jpg)
Problem Domain
What is the most efficient way to disseminate a large set of files to
a large set of clients?
![Page 3: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/3.jpg)
Client-Server
server
clients
![Page 4: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/4.jpg)
Client-Server
clients
InefficientHigh cost of ownership
![Page 5: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/5.jpg)
Peer-to-Peerpeer
block transfer
![Page 6: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/6.jpg)
Peer-to-Peer
Limited informationNo control or performance guarantees
![Page 7: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/7.jpg)
Peer-to-Peer
swarm
![Page 8: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/8.jpg)
Antfarm Goals
• High performance
• Low cost of deployment
• Performance guarantees
• Administrator control over swarm performance
• Accounting
• Enables different resource contribution policies
![Page 9: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/9.jpg)
Antfarm Approach
• Key insight: view content distribution as an optimization problem
• Hybrid architecture
• P2P swarming with a logically centralized coordinator
• Clean slate protocol
![Page 10: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/10.jpg)
Antfarm System Modelcoordinatorseeder
altruist
![Page 11: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/11.jpg)
Antfarm System Modelcoordinatorseeder
Coordinator optimally allocatestotal seeder bandwidth B
![Page 12: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/12.jpg)
Antfarm
OverviewThe
SystemEvaluation
![Page 13: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/13.jpg)
Antfarm
OverviewThe
SystemEvaluation
![Page 14: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/14.jpg)
Strawman Coordinator
• One could schedule every data transfer in the system
• All packets for all time
• Unscalable, impractical!
• Antfarm coordinator makes critical decisions based on observed dynamics
![Page 15: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/15.jpg)
Antfarm Coordinator
• Models swarm dynamics
• Measures and extracts key parameters
• Formulates optimization problem
• Calculates optimal bandwidth allocation
• Enacts allocation decisions
• Maximizes aggregate bandwidth
• Minimizes average download time
![Page 16: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/16.jpg)
Antfarm Formalization
Maximize system-wide aggregate bandwidth subject to a bandwidth constraint
![Page 17: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/17.jpg)
Swar
m a
ggre
gate
ban
dw
idth
Seeder bandwidth
Response Curves
slope
= 1
slope = 0
![Page 18: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/18.jpg)
Response CurvesSw
arm
agg
rega
te b
andw
idth
(K
B/s
) 1500
1000
500
0
Seeder bandwidth (KB/s)0 25 75 10050
![Page 19: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/19.jpg)
Swarm Dynamics
Swarms exhibit different dynamics based on size, peer resources,
network conditions. . .
![Page 20: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/20.jpg)
Swarm Dynamics
![Page 21: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/21.jpg)
Antfarm OptimizationSw
arm
agg
rega
te b
andw
idth
Seeder bandwidth
A
B
C
![Page 22: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/22.jpg)
Antfarm OptimizationSw
arm
agg
rega
te b
andw
idth
Seeder bandwidth
!A !C!B
!A + !B + !C = B
A
B
C
![Page 23: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/23.jpg)
Performance Control
• Can provide swarm performance guarantees
• Guarantee minimum level of service
• Prioritize swarms
![Page 24: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/24.jpg)
Swar
m a
ggre
gate
ban
dw
idth
Seeder bandwidth
Antfarm Allocation
!A" !C"!B"
!A"+ !B"+ !C"= B
A
B
C
![Page 25: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/25.jpg)
Adapting to Change
• Swarm dynamics change
• Churn
• Network conditions
• Antfarm updates response curves
• Coordinator explores around point of operation
![Page 26: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/26.jpg)
peer A
purse ledger
Wire Protocol
• Coordinator mints small, unforgeable tokens
• Peers trade each other tokens for blocks
• Peers return spent tokens to the coordinator as proof of contribution
coordinator
peer B
purse ledger
![Page 27: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/27.jpg)
Antfarm
OverviewThe
SystemEvaluation
![Page 28: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/28.jpg)
Antfarm
OverviewThe
SystemEvaluation
![Page 29: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/29.jpg)
Antfarm Performance
Zipf, 60 KB/s seeder Zipf, 200 KB/s seeder0
1000
2000
3000
4000
Agg
rega
te b
andw
idth
(K
B/s
)
Client-server BitTorrent Antfarm
![Page 30: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/30.jpg)
Swarm Starvation
0
10
20
30
BitTorrent Antfarm
Avg
ban
dw
idth
per
pee
r (K
B/s
)
self-sufficient swarm singleton swarm
BitTorrent starves the singleton swarm
![Page 31: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/31.jpg)
BitTorrent: Starves New Swarmtotal seeder bandwidth avg bandwidth per peer
0
5
10
15
20
25
Ban
dw
idth
(K
B/s
)
Swarms, ordered largest to smallest
newself-sufficient singleton
![Page 32: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/32.jpg)
total seeder bandwidth avg bandwidth per peer
0
5
10
15
20
25
Ban
dw
idth
(K
B/s
)
Antfarm: Seeds New Swarm
Swarms, ordered largest to smallest
newself-sufficient singleton
![Page 33: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/33.jpg)
Scalability
Number of peers
0 20K 40K 60K 80K
1-m
achin
e co
ord
inat
or
4-m
achin
e co
ord
inat
or
2-m
achin
e co
ord
inat
or
8-machine coordinator
5 GB/s
0
1 GB/s
2 GB/s
3 GB/s
4 GB/s
Agg
rega
te b
andw
idth
![Page 34: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/34.jpg)
Scalability
Number of peers
0 20K 40K 60K 80K
1-m
achin
e co
ord
inat
or
4-m
achin
e co
ord
inat
or
2-m
achin
e co
ord
inat
or
8-machine coordinator
5 GB/s
0
1 GB/s
2 GB/s
3 GB/s
4 GB/s
Agg
rega
te b
andw
idth
Single PC can compute allocations for 10,000 swarms with 1,000,000 peers
in 6 seconds
![Page 35: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/35.jpg)
Antfarm Implications
• No fine-tuning
• Subsumes hacks devised for BitTorrent
• Share ratio
• Manual pruning
![Page 36: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/36.jpg)
Related Work• Content Distribution Networks
• Akamai, CoBlitz, CoDeeN, ECHOS, Coral, Slurpie, YouTube, Hulu, GridCast, Tribler, Joost, Huang et al. 2008, ...
• P2P Swarming
• BitTorrent, BitTyrant, PropShare, BitTornado, BASS, Annapureddy et al. 2007, Guo et al. 2005, ...
• Incentives and microcurrencies
• Dandelion, BAR Gossip, Samsara, Karma, SHARP, PPay, Kash et al. 2007, ...
![Page 37: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/37.jpg)
Conclusions
• Model swarm dynamics and allocate bandwidth optimally
• Novel hybrid architecture
• PlanetLab deployment shows that Antfarm outperforms client-server and P2P
![Page 38: Antfarm: Efficient Content Distribution with …...Antfarm: Efficient Content Distribution with Managed Swarms Ryan S. Peterson and Emin Gün Sirer Department of Computer Science,](https://reader034.vdocuments.mx/reader034/viewer/2022052423/5f097e987e708231d427183f/html5/thumbnails/38.jpg)
Questions?