von: a scalable peer-to-peer network for virtual environments

1

VON:A Scalable Peer-to-Peer Network

for Virtual Environments

Shun-Yun Hu (胡舜元 ) ([email protected])

CSIE, National Central University, Taiwan

2007/06/15

2

Outline

Introduction Voronoi-based Overlay Network (VON) Evaluation Conclusion

3

What isNetworked Virtual Environment?

Virtual Reality + Internet

3D worlds with people, agents, objects, terrain

Military simulations (’80) Massively Multiplayer Online Games (mid-‘90)

Trends: larger scale, more realistic simulation

Massively Multiplayer Online Games

MMOGs are growing quickly 8 million registered users for World of Warcraft Over 100,000 concurrent players Billion-dollar business

Can we scale to millions at the same time?

8

The Scalability Problem

Many nodes on a 2D plane ( > 1,000) Message exchange with those within Area of Interest (AOI) How does each node receive the relevant messages?

Area of Interest

9

A simple solution (point-to-point)

But…too many irrelevant messagesN * (N-1) connections ≈ O(N2) Not scalable!

Source: [Funkhouser95]

10

A better solution (client-server)

Message filtering at server to reduce trafficN connections = O(N) server is bottleneck


11

Current solution (server-cluster)

Still limited by servers. Expensive to deploy & maintain.


12

Scalability Analysis

Scalability constrains Computing resource (CPU) Network resource (Bandwidth)

Non-scalable system vs. Scalable system

x: number of entitiesy: resource consumption at the limiting system component

Resource limit

13

What Next?

Strategies Increase resource More servers Decrease consumption Message filtering

Architectures Scale Point-to-point (LAN) tens 101

Client-server hundreds 102

Server-cluster thousands 103

? millions 106 …

Peer-to-Peer

14

What is Peer-to-Peer (P2P)?

[Stoica et al. 2003] Distributed systems without any centralized control

or hierarchical organization Runs software with equivalent functionality

Examples File-sharing: Napster, Gnutella, Kazza, eDonkey Distributed Search: Chord, CAN, Tapestry, Pastry VoIP: Skype

15

Peer-to-Peer Overlay

A P2P overlay network source: [Keller & Simon 2003]

16

Promise & Challenge of P2P

Promises Growing resource, decentralized

Scalable Commodity hardware Affordable

Challenges Topology maintenance dynamic join/leave Efficient content retrieval no global knowledge

17

Outline


18

Design Goals

Observation: for virtual environment applications, the contents we want

are messages from AOI neighbors Content discovery is a neighbor discovery problem

Solve the Neighbor Discovery Problem in a fully-distributed, message-efficient manner.

Specific goals: Scalable Limit per-node message traffic Responsive Direct connection with AOI neighbors

19

Voronoi Diagram

2D Plane partitioned into regions by sites, each region contains all the points closest to its site

Can be used to find k-nearest neighbor easily

Neighbors

Site

Region

20

Design Concepts

Identify enclosing and boundary neighbors Each node constructs a Voronoi of its neighbors Enclosing neighbors are minimally maintained Mutual collaboration in neighbor discovery

Circle Area of Interest (AOI)

White self

Yellow enclosing neighbor (E.N.)

L. Blue boundary neighbor (B.N.)

Pink E.N. & B.N.

Green AOI neighbor

L. Green unknown neighbor

Use Voronoi to solve the neighbor discovery problem

21

Procedure (JOIN)

1) Joining node sends coordinates to any existing node

Join request is forwarded to acceptor

2) Acceptor sends back its own neighbor list

joining node connects with other nodes on the list

Acceptor’s region Joining node

22

Procedure (MOVE)

1) Positions sent to all neighbors, mark messages to B.N.

B.N. checks for overlaps between mover’s AOI and its E.N.

2) Connect to new nodes upon notification by B.N.

Disconnect any non-overlapped neighbor

Boundary neighbors

New neighbors

Non-overlapped neighbors

23

Procedure (LEAVE)

1) Simply disconnect

2) Others then update their Voronoi

new B.N. is discovered via existing B.N.

Leaving node (also a B.N.) New boundary neighbor

24

Dynamic AOI

Crowding within AOI can overload a particular node

It’s better if AOI-radius can be adjusted in real time

25

Adjustment Conditions

AOI-radius decrease Number of connections > connection limit

AOI-radius increase Maximum connections not exceeded Current AOI-radius < preferred AOI-radius

Mutual awareness rule Do not disconnect a neighbor who sees me

26

Demonstration

Simulation demo Random movements (100 nodes, 1200x700 world) Local vs. global view Dynamic AOI adjustment

27

Outline


28

Simulation Method C++ implementation of VON (open source VAST library)

World size: 1200 x 1200 (AOI: 100) Trials from 200 – 2000 nodes Connection limit: 20 3000 time-steps

(~ 300 simulated seconds, assuming 10 updates/seconds)

Behavior model Random movement: random destination Constant velocity: 5 units/step Movement duration: random (until destination is reached)

29

Scalability: Avg. Transmission / sec

0

5

10

15

20

25

30

0 400 800 1200 1600 2000Number of Nodes

Siz

e (kb /

basic

dAOI

basic (fixed density after 1000 nodes)dAOI (fixed density after 1000 nodes)

30

Scalability: Max. Transmission / sec

0

10

20

30

40

50

60

70

0 400 800 1200 1600 2000Number of Nodes

Siz

e (kb /

basic

dAOI

basic (fixed density after 1000 nodes)dAOI (fixed density after 1000 nodes)

31

Scalability: Avg. Neighbor Size

0

10

20

30

40

50

60

70

80

90

0 400 800 1200 1600 2000Number of Nodes

Neig

hbor Siz

e

connected neighbors (basic)

AOI neighbors (basic)

connected neighbors (dAOI)

AOI neighbors (dAOI)

32

Reliability: Effects of Packet Loss

0

10

20

30

40

50

60

70

80

90

100

0% 20% 40% 60% 80% 100%Loss Rate

Units

Topology Consistency (%)

Recovery Steps

33

Analysis of Design

Scalability Bounded resource consumption dynamic AOI

Consistency (Topology) Topology is fully connected enclosing neighbors

Reliability Self-organizing distributed neighbor discovery

Responsiveness Lowest latency direct connection, no relay

34

Outline

Introduction Voronoi-based Overlay Network (VON) Simulation results Conclusion

35

Summary

NVE scalability is achievable with P2P architecture and is a neighbor discovery problem

A promising solution: Voronoi-based P2P Overlay Leverage knowledge of each peer to maintain topology

Properties Scalable: fully-distributed, dynamic AOI Efficient: low irrelevant messages, zero relay Simple: simple protocol and procedure

36

Potential Applications

Online gamesPosition updates in current MMOGs, Voice-chats

MilitaryEnable large-scale, affordable military training simulation

3D WebProvide multi-user interactivity to static 3D world

Scientific simulationsDistribute spatial simulation requiring frequent synchronization

37

Acknowledgements Dr. Jui-Fa Chen (陳瑞發老師 ) Tsu-Han Chen (鄭子涵 ) Members of the Alpha Lab, TKU CS

Dr. Chin-Kun Hu (胡進錕老師 ) Guan-Ming Liao (廖冠名 ) LSCP, Institute of Physics, Academia Sinica

Joaquin Keller (France Tele. R&D, Solipsis) Jon Watte (there.com) Kuan-Ta Chen (陳寬達 , NTU)

38

Q&A

VON: A Scalable Peer-to-Peer Network for Virtual EnvironmentsIEEE Network, vol. 20, no. 4, Jul./Aug. 2006

Thank you!

[email protected]://vast.sourceforget.net

(http://vast.sf.net)

39

Issues for Creating NVE

Consistency (events/states)

Responsiveness multiplayer Security

Scalability Persistency massively multiplayer Reliability (Fault-tolerance)

40

Issues for Creating P2P NVE

Consistency (events/states)

Responsiveness multiplayer Security

Scalability Persistency massively multiplayer Reliability (Fault-tolerance)

Consistency (topology) P2P NVE

41

Server-cluster issues

Insufficient total resourceHardware provisioning over-provision!

High user density (crowding)User limits limits scale & realism!

Excessive inter-server communicationsLess load balancing difficult balance!

42

Related Work (1):

DHT-based: SimMUD

[Knutsson et al. 2004] (UPenn)

Pastry + Scribe Regions Coordinators

(super-nodes)

Fixed-size region Relay overhead

43

Related Work (2):

Neighbor-list Exchange

[Kawahara et al. 2004] (Univ. of Tokyo)

Fully-distributed Nearest-neighbors List exchange

High transmission Overlay partition

44

Related Work (3):

Mutual Notification: Solipsis

[Keller & Simon 2003] (France Telecomm R&D)

Links with AOI neighbor Mutual cooperation Inside convex hull

Potentially slow discovery Inconsistent topology

45

Consistency Metrics

Topology Consistency [Kawahara, 2004]

observed AOI neighbors

actual AOI neighbors

Drift Distance [Diot, 1999]Distance between observed position and actual position

(average over all nodes)

46

Consistency: Topology Consistency

99.90

99.91

99.92

99.93

99.94

99.95

99.96

99.97

99.98

99.99

100.00

0 500 1000 1500 2000Number of Nodes

Topolo

gy C

onsis

tency (%

)

basic

dAOI

47

Consistency: Drift Distance

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0 400 800 1200 1600 2000

Number of Nodes

Ave

rage

Drift D

ista

nce basic

dAOI

48

Problems of Voronoi Approach

Message traffic Circular round-up of nodes Redundant message sending

(inherent to fully-distributed design)

Incomplete neighbor discovery Can happen with inconsistent / incorrect neighbor list Fast moving node

Limited AOI Direct connections

49

P2P NVE Comparisons

DHT-based

Neighbor-list exchange

Solipsis VON

Consistency (topology)

DHT & Supernode

(consistent)

Neighbor list-exchange

(partitioning)

Neighbor notify&query

(undiscovery)

Neighbor notify

(consistent)

Responsive-ness

two to many

One hop One hop One hop

Scalability O(n) on supernode

Constant in crowding

Constant if fixed density

Constant in crowding

Con Latency too high

Overlay partitioning

Occasional undiscovery

Circular node line-up

50

Future Perspectives

Short-term Distributed event/state consistency Customizable AOI (Heterogeneity in P2P) Recovery from overlay partition and fast-moving nodes

Long-term Persistency issue (P2P-based database) Security issue (protection from malicious nodes) 3D content distribution (3D streaming on P2P)

Massive, persistent 3D environment sharable by all!

51

NVE: A Shared Space

von: a scalable peer-to-peer network for virtual environments

Documents