Download - Gunrock: A High-Performance, Data-Centric Abstraction for ...jowens/tmp/nvidia-webinar-160426.pdf · Gunrock: A High-Performance, Data-Centric Abstraction for GPU Graph Computation

Gunrock: A High-Performance, Data-Centric Abstraction for GPU Graph Computation

John Owens Child Family Professor of Engineering and Entrepreneurship

Department of Electrical and Computer Engineering UC Davis

w/ Yangzihao Wang, Yuechao Pan, Yuduo Wu, Carl Yang, Leyuan Wang, Mohamed Ebeida, Chenshan Shari Yuan, Weitang Liu

Slides at http://preview.tinyurl.com/owens-nv-webinar-160426

Graphs

The Ninja Problem

“I believe that in the datacenter, one question is critical: If you can’t get to peak performance on GPUs, they basically lose all their value proposition. So how can you get close to peak without becoming an architecture expert and programming/performance wizard?” —Anonymous, Large Internet Company, 27 May 2014

Gunrock Genesis

• Summer 2013, DARPA XDATA summer camp

• Focus: to-the-metal GPU graph implementations

• 8 weeks to write (port) betweenness centrality

• Not a sustainable model!

Gunrock: Goals

• Bottom-up: To leverage the highest-performing GPU computing primitives for efficiency.

• Top-down: To be expressive enough to represent a wide variety of graph computations for usability.

Gunrock Status

• Open-source release (Apache 2.0), currently version 0.3

• http://gunrock.github.io/

• Fastest programmable GPU library for graph analytics

• Superior load-balancing/work distribution

• More powerful abstractionYangzihao Wang, Andrew Davidson, Yuechao Pan,

Yuduo Wu, Andy Riffel, and John D. Owens. Gunrock: A High-Performance Graph Processing Library on the GPU. ACM PPoPP 2016. Distinguished Paper.

http://escholarship.org/uc/item/6xz7z9k0

Other programmable GPU frameworks …

• … leverage a bulk-synchronous model

• … use CPU abstractions:

• Pregel (Medusa)

• GAS (VertexAPI2, CuSha, MapGraph)

• … organize steps of computation, with two significant disadvantages:

• Programming models are not very general

• Kernels are small and miss opportunities for producer-consumer locality

• Graph represented as CSR (~ sparse matrix)

• Bulk-synchronous: series of parallel steps (operations) separated by global barriers

• Data-centric: All operations are on one or more frontiers of active vertices/edges

• Advance: generates a new frontier through visiting the neighbor vertices/edges of elements in the current frontier. Key: Work distribution/load balancing

• Filter: removes elements from frontier via validation test

• Compute: user-defined vertex-centric or edge-centric computations that run in parallel

Gunrock: Programming Model

• Graph represented as CSR (~ sparse matrix)

• Bulk-synchronous: series of parallel steps (operations) separated by global barriers

• Data-centric: All operations are on one or more frontiers of active vertices/edges

• Advance: generates a new frontier through visiting the neighbor vertices/edges of elements in the current frontier. Key: Work distribution/load balancing

• Filter: removes elements from frontier via validation test

• Compute: user-defined vertex-centric or edge-centric computations that run in parallel

Considering new operators:

• Global

• Neighborhood

• Sampling

• Frontier-frontier intersection

Gunrock: Programming Model

Gunrock’s Data-Centric Abstraction & Bulk-Synchronous Programming

• Data-centric abstraction– Operations are defined on a group

of vertices or edges = a frontier– Operations = manipulations of

one or more frontiers

• Bulk-synchronous programming– Operations are done one by one,

in order– Within a single operation,

computing on multiple elements can

be done in parallel, without order

Loop until convergence

A group of V or E

Do something

Resulting group of V or E

Do something

Another resulting group of V or E

A generic graph algorithm:

Using Gunrock

• As a programmer …

• Write your own Gunrock primitives (using advance, compute, filter)

• Write your own Gunrock operators!

• As an end-user …

• Write an executable that runs Gunrock primitives

• Link against a Gunrock library that provides Gunrock primitives(C linkage)

• python

• Julia

Graph challenges on GPUs

• Efficient parallel algorithms

• Different balance between brute-force and elegant than on CPUs (next slide)

• Load-balancing due to irregularity

• Moving beyond simple algorithms

• Graph representations

• Scalability (memory constraints)

Algorithm example: SSSP

Andrew Davidson, Sean Baxter, Michael Garland, and John D. Owens. Work-Efficient Parallel GPU Methods for Single Source Shortest Paths. In Proceedings of the 28th IEEE International Parallel and Distributed Processing Symposium, May 2014.

Currently Supported Primitives

• Currently have over 10 graph primitives including:

• Traversal-based (e.g., BFS, DOBFS, SSSP)

• Node-ranking (e.g., HITS, SALSA, PageRank)

• Global (e.g., connected component, MST, triangle-counting)

• LOC under 300 for each primitive, under 10 to use a primitive

• In progress:

• Graph coloring, Maximal Independent Set

• Community Detection

• Subgraph Matching

Industry interest examples• Twitter: “Who To Follow” service

• Historically: SALSA (“Stochastic Approach for Link-Structure Analysis”)

• Personalized PageRank generates circle of trust

• Hubs & authorities, random walks

• Facebook

• PageRank & Personalized PageRank

• Label propagation

• Graph embeddings into Rn so similar nodes are close

Industry interest examples• Twitter: “Who To Follow” service

• Historically: SALSA (“Stochastic Approach for Link-Structure Analysis”)

• Personalized PageRank generates circle of trust

• Hubs & authorities, random walks

• Facebook

• PageRank & Personalized PageRank

• Label propagation

• Graph embeddings into Rn so similar nodes are close

Afton Geil, Yangzihao Wang, and John D. Owens. WTF, GPU! Computing

Twitter's Who-To-Follow on the GPU. In Proceedings of the Second ACM

Conference on Online Social Networks, COSN '14, pages 63–68, October 2014.

Load-Balanced Traversal• Problem: Lots of parallelism

across vertices, but each vertex has a different number of neighbors

• Merrill: Depending on size of worklist, vertex work mapped to one {thread, warp, block}

• Davidson: Instead of allocating vertices to threads, allocate edges to threads

• Requires sorted search to find start and endpts of edges

• Gunrock advantage: Best load-balancing (2–20x over Medusa)

tn

tn

Block 0 t0 t1 tn t0 t1 tn t0 t1

t0 t1 tn t0 t1 tn t0 t1

t0 t1 tn t0 t1

Block 1

Block 255

t0 t1 tn t0 t1 tn t0 t1 tn t0 t1

Block cooperative Advance of large neighbor lists;

t0 t31 t0 t0 t31 t0 t31t1 t0 t31

Warp cooperative Advance of medium neighbor lists;

t0 t1 t2 tn

Warp 31Warp 1Warp 0

Per-thread Advance of small neighbor lists.

Merrill’s per-{thread,warp,CTA} load balance

Davidson’s load-balanced partitioning

Push vs. Pull (Direction-Optimized Breadth-First Search)

• Normal operation (“push”): vertex frontier visits every outbound edge, generates list of connected unvisited vertices

• Works great when you’re expanding: more new vertices than old

• Works poorly with few unvisited vertices

Input queue: v1,v2,v3

0

1 2 3

4 5

label=0

label=1

label=?

0 4 4 3 4 5

from v1 from v2 from v3

4 5

Explored edges (gray ones are failures)

Final output queue

Push-basedEdge Mapping

6 7

Push vs. Pull• We also support pull: Start with unvisited vertices, check which have

inbound edges from frontier

• Difficult to express in compute-focused APIs

• Gunrock: Frontier is unvisited vertices; pull from that frontier

0

1 2 3

4 5

label=0

label=1

label=?

Pull-basedEdge Mapping

6 7

Input queue: v4 v5 v6 v7

1 3 4 5

from v4 from v5 from v6

4 5

Explored edges (blue ones are valid in bitmap)

Final output queue

5

from v7

0 1 1 1 0 0 0 0working queue bitmap

Supporting Priority Queues• Our SSSP (standalone) implementation compares three data

structures for work queues:

• Workfront: All active vertices only (big improvement over Bellman-Ford)

• Bucketing: Closest to delta-stepping: vertices sorted by distance from source, placed into buckets

• Near-far: 2 buckets, “near” and “far”

• Near-far is best: cost of multisplit was too high on GPU (reorganizational overhead: 82% of runtime). (We published a paper on multisplit at PPoPP 2016.)

• Difficult on compute-focused APIs, but in Gunrock we can just have multiple active frontiers, one per bucket

• Bipartite graphs (SALSA, matching, link prediction, personalized PageRank)

• Streaming graphs

• Mutable graphs

• Graphs that change as a result of the computation (Borůvka minimum-spanning-tree, Delaunay triangulation)

• Graphs that require modifying the graph to compute (Karger’s mincut)

• In general, significant data structure challenges.

• Is CSR the right format?

Research Directions: Broader Graph Types

Gunrock vs. nvGRAPH• Native graph

representation

• Custom (but good!) load-balancing

• Open-source

• Write your own primitives

• Which primitives fit into the Gunrock model?

• Matrix-based representation

• Leverages extensive sparse-matrix infrastructure (sparse vector: a challenge!)

• API access only

• Limited set of primitives

• Which primitives fit into the Graph BLAS?

https://developer.nvidia.com/nvgraph

1-GPU Performance Comparison

• Each row: single engine on certain dataset, vs. Gunrock

• Black dots/right: Gunrock faster. White dots/left: Gunrock slower


• 10+x faster than single-core CPU (Boost), or PowerGraph


• On par with fastest 2-socket CPU (Ligra)(Gunrock 16 wins, Ligra 8 wins)


• Fastest of all GPU programmable frameworks(CuSha, MapGraph, Medusa)


• Competitive with hardwired GPU implementations

Research Directions: Scalability

• Largest memory on a CPU: 5 TB

• On an NVIDIA GPU: 12 GB (gp100: 16 GB)

• Today: Multi-GPU, single node (next!)

• Tomorrow:

• Out of core?

• Multi-node?

• Long term?: heterogeneous single-chip processors

Multi-GPU Framework (for programmers)

Iterate until convergence

Input frontier

Output frontier

Single GPU

Associated data (label, parent, etc.)

Recap: Gunrock on single GPU


Iterate until convergence

Input frontier

Output frontier

GPU 0


Input frontier

Output frontier

GPU 1


Dream: just duplicate the single GPU implementation Reality: it won’t work, but good try!

unchanged single-GPU Gunrock


Specify (1) how to combine frontiers, (2) what data to communicate, (3) global convergence condition


Local input frontier

Local output frontier

GPU 0


GPU 1


Remote output frontier

Remote input frontier

Remote input frontier

Local input frontier

Remote output frontier

Local output frontier

Partition

Iterate until all GPUs converge

Yuechao Pan, Yangzihao Wang, Yuduo Wu, Carl Yang, and John D. Owens. Multi-

GPU Graph Analytics. arxiv, abs/1504.04804(1504.04804v2), April 2016.



Results: Multi-GPU Scaling

• Primitives (except DOBFS) get good speedups (averaged over 16 datasets of various types)BFS: 2.74x, SSSP: 2.92x, CC: 2.39x, BC: 2.22x, PR: 4.03x using 6 GPUs

• Peak DOBFS performance: 514 GTEPS with rmat_n20_512

• Gunrock can process a graph with 3.6B edges (full-friendster graph, undirected, DOBFS in 339ms, 10.7 GTEPS using 4 K40s); 50 PR iterations on the directed version (2.6B edges) took ~51 seconds

BFS: Multi-GPU Gunrock vs. Others

• Gunrock generally outperforms other implementations on GPU clusters with 4–64 GPUs on both the real and generated graphs cited in their publications

• Gunrock’s “just-enough” memory allocation: critical!

• 2–5 times faster than Enterprise (Liu and Huang, SC15), a dedicated multi-GPU DOBFS implementation

group name |V | |E| D group name |V | |E| D group name |V | |E| D

soc soc-LiveJournal1 4.85M 85.7M 13 web indochina-2004 7.41M 302M 24 rmat rmat n20 512 1.05M 728M 6.26⇤soc hollywood-2009 1.14M 113M 8 web uk-2002 18.5M 524M 25 rmat rmat n21 256 2.10M 839M 7.22⇤soc soc-orkut 3.00M 213M 7 web arabic-2005 22.7M 1.11B 28 rmat rmat n22 128 4.19M 925M 7.56⇤soc soc-sinaweibo 58.7M 523M 5 web uk-2005 39.5M 1.57B 23 rmat rmat n23 64 8.39M 985M 8.32⇤soc soc-twitter-2010 21.3M 530M 15 web webbase-2001 118M 1.71B 379 rmat rmat n24 32 16.8M 1.02B 8.61⇤

rmat rmat n25 16 33.6M 1.05B 9.06⇤

TABLE II: Datasets we used to evaluate our work. |V | and |E| are vertex and edge counts; d is the graph diameter, ⇤ indicatesan approximated diameter computed by multiple run of random-sourced BFS.

graph algo ref. ref. hw. ref. perf. our hw. our perf. comp.

com-orkut (3M, 117M, UD) BFS Bisson [5] 1⇥K20X⇥4 2.67 GTEPS 4⇥K40 14.22 GTEPS 5.33Xcom-Friendster (66M, 1.81B, UD) BFS Bisson [5] 1⇥K20X⇥64 15.68 GTEPS 4⇥K40 14.1 GTEPS 0.90Xkron n23 16 (8M, 256M, UD) BFS Bernaschi [4] 1⇥K20X⇥4 ⇠1.3 GTEPS 4⇥K40 30.8 GTEPS 23.7Xkron n25 16 (32M, 1.07G, UD) BFS Bernaschi [4] 1⇥K20X⇥16 ⇠3.2 GTEPS 6⇥K40 31.0 GTEPS 9.69Xkron n25 32 (32M, 1.07G, D) BFS Fu [15] 2⇥K20⇥32 22.7 GTEPS 4⇥K40 32.0 GTEPS 1.41Xkron n23 32 (8M, 256M, D) BFS Fu [15] 2⇥K20⇥2 6.3 GTEPS 4⇥K40 27.9 GTEPS 4.43Xkron n24 32 (16.8M, 1.07G, UD) BFS Liu [24] 2⇥K40⇥1 15 GTEPS 2⇥K40 77.7 GTEPS 5.18Xkron n24 32 (16.8M, 1.07G, UD) BFS Liu [24] 4⇥K40⇥1 18 GTEPS 4⇥K40 67.7 GTEPS 3.76Xkron n24 32 (16.8M, 1.07G, UD) BFS Liu [24] 8⇥K40⇥1 18.4 GTEPS 4⇥K80 40.2 GTEPS 2.18Xtwitter-mpi (52.6M, 1.96G, D) BFS Bebee [3] 1⇥K40⇥16 0.2242 sec 3⇥K40 94.31 ms 2.38Xrmat n21 64 (2M, 128M, D) BFS Merrill [29] 4⇥C2050⇥1 8.3 GTEPS 4⇥K40 23.7 GTEPS 2.86X

TABLE III: Comparison with previous in-core GPU graph processing work. Ref. hardware is denoted by intra-node GPUcount⇥GPU model⇥node count. We use the same number of GPUs whenever possible within the constraints of a single node.

graph algo ref. ref. perf. our hw. our perf.

uk-2002 {BFS, SSSP, PR, CC} Sengupta [33], 1⇥K40 {49, 80, 153, 162} sec 1⇥K40 {59, 762, 1991, 1848} mstwitter-rv {SSSP, CC, PR} Lee [23], 4 cores⇥21 nodes {126, 304, 149} sec {2, 3, 2}⇥K40 {2203, 304, 149} mstwitter-rv {BFS, SSSP, CC, PR} Shi [35], 1⇥K20m {58.8, 153, 35, 145} sec {1, 2, 3, 1}⇥K40 {98, 2203, 1712, 49700} msLiveJournal1 BFS Shi [34], 1⇥K20m 7.48 ms 1⇥K40 12.17 ms

TABLE IV: Comparison with previous out-of-core GPU or CPU graph processing work. Our framework can process the largestdatasets that were reported by any of this previous work, on all reported primitives, using much less processing time. {uk-2002,twitter-rv, LiveJournal1} are directed graphs with {18.5M, 42M, 5M} vertices and {298.1M, 1.5B, 68M} edges.

● ● ● ● ● ● ● ●

0

100

200

300

1 2 3 4 5 6 7 8Number of GPUsBi

llion

Trav

erse

d Ed

ges

per S

econ

d (G

TEPS

)

● Strong_Scaling Weak_Edge_Scaling Weak_Vertex_Scaling

(a) DOBFS scalability

●●

●●

●● ● ●

0

10

20

30

40

50

1 2 3 4 5 6 7 8Number of GPUsBi

llion

Trav

erse

d Ed

ges

per S

econ

d (G

TEPS

) ● Strong_Scaling Weak_Edge_Scaling Weak_Vertex_Scaling

(b) BFS scalability

● ● ● ● ●● ● ●

0

10

20

30

40

1 2 3 4 5 6 7 8Number of GPUs

Billio

n Tr

aver

sed

Edge

s pe

r Sec

ond

(GTE

PS)

● Strong_Scaling Weak_Edge_Scaling Weak_Vertex_Scaling

(c) PR scalability

Fig. 6: Scalability of DOBFS, BFS, and PR. {Strong, weak edge, weak vertex} scaling use rmat graphs with {224, 219,219 ⇥ |GPUs|} vertices and edge factor {32, 256⇥|GPUs|, 256} accordingly.

we feel is the fairest comparison—systems with the samenumber of GPUs—we also note that our system featuresall of those GPUs on the same node, whereas some of thecomparison systems instead allocate one GPU per node. Inter-GPU bandwidth on a single node is larger than inter-nodebandwidth, so our comparisons must be considered in thislight; however, as we noted in the introduction, we believethat our results motivate a future focus on scaling up (fewerbut more powerful nodes, with more GPUs) in preference toscaling out (more nodes). The datasets we choose for com-

parison against each system are those specifically highlightedby the authors of the comparison systems in their results,presumably the datasets where their systems show the bestresults. Finally, most of the systems that we compare againstare designed to run only a single graph primitive, and areoptimized specifically for that primitive, whereas our systemruns a wide range of graph primitives in a programmableframework without a primitive-specific focus.

Enterprise [24] is a hardwired DOBFS implementation withvarious optimizations, and our DOBFS outperforms it by 2 to

NVIDIA “Pascal” (2016)• How to scale beyond one node?

• Scale-out: multiple nodes?

• Scale-up: out-of-core?

• gp100 has:

• Stacked memory (720 GB/s)

• NVLink high-speedCPU-GPU connection(160 GB/s bidirectional)

• CUDA 8’s unified virtual memory

• On current hardware, we contend mGPU on 1 node is the right building block

Graph Matching

© All Rights Reserved 2014 | Neo Technology, Inc.

Cypher: Basic Example

• Declarative query langue with SQL-like clause syntax

• Visual graph patterns

• Tabular results

// get node MATCH (a:Person {id: 0}) RETURN a

// return friends of friends MATCH (a:Person {id: 0})--()--(c) RETURN c

// return friends MATCH (a:Person {id: 0})-->(b) RETURN b


Core industries

& Use Cases

WEB / ISV Financial Services

Tele-communications

Health Care & Life

Sciences

Web Social, HR & Recruiting

Media & Publishing

Energy, Services, Automotive, Gov’t,

Logistics, Education, Gaming, Other

Network &Data

Center Manageme

ntMaster Data

Management

Social

GEO

Recommend-ations

Identity & Access Mgmt

Content Manageme

ntBI, CRM, Impact

Analysis, Fraud

Detection,

Finance

Neo4j Adoption SnapshotSelect Commercial Customers


Fraud Detection

Research Directions: Long Term• Efficiency

• Raw peak performance

• Achieving peak performance with smaller graphs

• More and higher-level algorithms

• More customers!

• Asynchronous execution

• Graph coloring

• Rich data on vertices and edges

• What goes above Gunrock? GraphX, TinkerPop, etc.

• What goes below Gunrock?

• Beyond CSR

• Graph BLAS

• Dynamic graphs

Frog: Asynchronous Graph Processing … http://grid.hust.edu.cn/xhshi/projects/frog.html

Thanks to …• Yangzihao Wang, Yuechao Pan, Yuduo Wu, Carl Yang, Leyuan Wang,

Mohamed Ebeida, Chenshan Shari Yuan, Weitang Liu (UC Davis)

• Nikolai Sakharnykh, Rob Zuppert, Joe Eaton, Doug Holt, Tom Reed, Ujval Kapasi, Cliff Woolley, Mark Harris, Duane Merrill, Michael Garland, David Luebke, Chandra Cheĳ (NVIDIA), and the CUDA Fellows program

• Vishal V, Erich Elsen, Guha Jayachandran (Onu)

• DARPA XDATA program & program managers Christopher White and Wade Shen, and Gabriela Araujo

• NSF awards ccf-1017399, oci-1032859

• UC Lab Fees Research Program Award 12-lr-238449

• Adobe and Grainger Foundation grants

• NVIDIA hardware donations & cluster access

Next steps!

• Feel free to send us questions! [email protected]

• Even better, file Gunrock issues!https://github.com/gunrock/gunrock/issues

• Slides from this talk athttp://preview.tinyurl.com/owens-nv-webinar-160426

Download - Gunrock: A High-Performance, Data-Centric Abstraction for ...jowens/tmp/nvidia-webinar-160426.pdf · Gunrock: A High-Performance, Data-Centric Abstraction for GPU Graph Computation

Top Related