![Page 1: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/1.jpg)
Large-Scale GraphProcessing
@doryokujinHadoop Conference Japan 2011 Fall
~Introduction~
![Page 2: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/2.jpg)
・井上 敬浩(26歳)
・twitter: doryokujin
・データマイニングエンジニア
・MongoDB JP 代表
・Hadoop, MongoDB, GraphDB に関心
・マラソン2時間33分
自己紹介
![Page 3: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/3.jpg)
@IT さんの9/15付のインタビュー記事でカッティング氏がGiraphについて言及
Hadoop MapReduce デザインパターン——MapReduceによる大規模テキストデータ処理
1 Jimmy Lin, Chris Dyer�著、神林 飛志、野村 直之�監修、玉川 竜司�訳
2 2011年10月01日 発売予定3 210ページ4 定価2,940円
MapReduceデザインパターン本の5章でグラフアルゴリズムが取り上げられている
Motivation: Why Graph?
![Page 4: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/4.jpg)
Map Reduce
ではアカン?
![Page 5: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/5.jpg)
Map Reduce
ではアカン?
1. Iteration の問題
2. Graph のデータ構造の問題
![Page 6: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/6.jpg)
MR: Good For Simple Problems
Map
Map
Map
Reduce
Reduce
Map
HDFS
![Page 7: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/7.jpg)
MR: Bad for Iterative Problems
Map
Map
Map
Reduce
Reduce
Map
HDFS
Map
Map
Map
Reduce
Reduce
Map
HDFS
Shuffle & barrier
job start/shutdown
イテレーション毎のデータロード
i i+1
![Page 8: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/8.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
Is MR Fit for Graph Data?
![Page 9: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/9.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
3
5
B
C
D
E
F
GA
5
1
3
5
42
41
3
5!4
3
3 2
min(6,4)
Graph Processing = “Vertex Based Approach”
i
i+1
隣接するノード間のメッセージパッシングをMRでどう記述する?
Is MR Fit for Graph Data?
![Page 10: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/10.jpg)
MapReduce 以外にええ
のんあるんでっか?
![Page 11: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/11.jpg)
MapReduce 以外にええ
のんあるんでっか?
BSP: Bulk Synchronous Parallel
![Page 12: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/12.jpg)
BSP: Bulk Synchronous Parallel
a super step
http://en.wikipedia.org/wiki/Bulk_Synchronous_Parallel
![Page 13: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/13.jpg)
BSP: Bulk Synchronous Parallel
1. Local Computation: 各Processorがローカルデータに対して独立した処理を行う
2. Communication: Processor 間でメッセージパッシングを行う
3. Barrier Synchronisation:全Processorのメッセージパッシングが完了するまで待機。
1.~3. の “super step” のイテレーション
...
![Page 14: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/14.jpg)
Relation: MR and BSP
Local Computation= Map Phase
Communication + Barrier = Shuffle and Sort Phase
Aggregation or (next) Local Computation
= Reduce Phase
a super step
![Page 15: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/15.jpg)
BSP Iterative MR
MR
Relation: MR and BSP
![Page 16: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/16.jpg)
Graph, Matrix, MachineLearning
BSP Iterative MR
MRGraph Processing
Matrix Computation
Machine Leaning ※1
※1 多くのMachine Learning ModelはMapReduceで記述可能なことが証明されている
![Page 17: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/17.jpg)
・2009年6月に Google が発表
- BSP を Graph Processing に応用
- 大規模データの80%をMapReduceで、20%をPregelで
- 10億node, 800億edgeのグラフをPC480台で並列処理、最短経路問題を200秒で解く
- YouTube の Graph-Based Recommendations で使われてるれているらしい
- 論文も入手可能
Google Pregel
![Page 18: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/18.jpg)
Map: BSP to Graph ProcessingLocal Computation
-> [頂点へのユーザ定義関数] Compute()
Communication -> 隣接するノードへメッセー
ジパッシング
Barrier synchronisation-> 全ノードのメッセージパッシングが終了するまで待機
a super step
![Page 19: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/19.jpg)
どっちがええのか比較せなあかんがな
SSSP: Single Source Shortest Paths
![Page 20: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/20.jpg)
SSSP: Parallel BFS
MapReduce & Pregel
※ SSSP: Single Source Shortest Paths, BFS: Breadth First Search
![Page 21: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/21.jpg)
SSSP: Parallel BFS
MapReduce & Pregel
※ SSSP: Single Source Shortest Paths, BFS: Breadth First Search
![Page 22: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/22.jpg)
SSSP: MapReduce Model
B
C
D
E
F
GA
5
1
3
5
42
41
3
initialize
・Load: Adjacency ListA: <(B,5),(D,3)>
B: <(E,1)>
C: <(F,5)>
D: <(B,1),(C,3),(E,4),(F,2)>
E: <>
F: <(G,4)>
G: <>
Source
![Page 23: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/23.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
+∞
1
+∞
+∞
+∞+∞
+∞0
・Map Input: [Graph Structure]- <A: <0, (B,5),(D,3)>>
- <B: <∞, (E,1)>>
- <C: <∞, (F,5)>>
- <D: <∞, (B,1),(C,3),(E,4),(F,2)>>
- <E: <∞>>
- <F: <∞, (G,4)>>
- <G: <∞>>
SSSP: MapReduce Model
![Page 24: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/24.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
+∞
1
+∞
+∞
+∞+∞
+∞0
・Map Output:- (B,5),(D,3), <A: <0, (B,5),(D,3)>>
- (E,∞), <B: <∞, (E,1)>>
- (F,∞), <C: <∞, (F,5)>>
- (B,∞),(C,∞),(E,∞),(F,∞),
<D: <∞, (B,1),(C,3),(E,4),(F,2)>>
- <E: <∞>>
- (G,∞), <F: <∞, (G,4)>>
- <G: <∞>>
SSSP: MapReduce ModelGraph 構造もReducerに送信
Local Disk のFlush
![Page 25: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/25.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
+∞
1
+∞
+∞
+∞+∞
+∞0
・Reduce Input:[A] - <A: <0, (B,5),(D,3)>>
[B] - (B,5),(B,∞), <B: <∞, (E,1)>>
[C] - (C,∞), <C: <∞, (F,5)>>
[D] - (D,3),
<D: <∞, (B,1),(C,3),(E,4),(F,2)>>
[E] - (E,∞),(E,∞), <E,<∞>>
[F] - (F,∞),(F,∞), <F: <∞, (G,4)>>
[G] - (G,∞), <G: <∞>>
SSSP: MapReduce Model
![Page 26: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/26.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
5
1
+∞
+∞
+∞+∞
30
SSSP: MapReduce Model
・Reduce Process:[A] - <A: <0, (B,5),(D,3)>>
[B] - (B,5),(B,∞), <B: <∞, (E,1)>>
[C] - (C,∞), <C: <∞, (F,5)>>
[D] - (D,3),
<D: <∞, (B,1),(C,3),(E,4),(F,2)>>
[E] - (E,∞),(E,∞), <E,<∞>>
[F] - (F,∞),(F,∞), <F: <∞, (G,4)>>
[G] - (G,∞), <G: <∞>>Reduce後、HDFS のフラッシュ
![Page 27: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/27.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
5
2
+∞
+∞
+∞+∞
30
SSSP: MapReduce Model
・Map Input (Reduce Output):- <A: <0, (B,5),(D,3)>>
- <B: <5, (E,1)>>
- <C: <∞, (F,5)>>
- <D: <3, (B,1),(C,3),(E,4),(F,2)>>
- <E,<∞>>
- <F: <∞, (G,4)>>
- <G: <∞>>
![Page 28: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/28.jpg)
SSSP: MapReduce Model
B
C
D
E
F
GA
5
1
3
5
42
41
3
5
2
+∞
+∞
+∞+∞
30
・Map Output:- (B,5),(D,3), <A: <0, (B,5),(D,3)>>
- (E,6), <B: <5, (E,1)>>
- (F,∞), <C: <∞, (F,5)>>
- (B,4),(C,6),(E,7),(F,5), <D: <3, (B,1),(C,3),(E,4),(F,2)>>
- <E,<∞>>
- (G,∞), <F: <∞, (G,4)>>
- <G: <∞>>Local Disk のFlush
![Page 29: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/29.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
4
2
6
+∞
56
30
・Reduce Process:[A] - <A: <0, (B,5),(D,3)>>
[B] - (B,5),(B,4), <B: <5, (E,1)>>
[C] - (C,6), <C: <∞, (F,5)>>
[D] - (D,3),
<D: <3, (B,1),(C,3),(E,4),(F,2)>>
[E] - (E,6),(E,7), <E, <∞>>
[F] - (F,∞),(F,5), <F: <∞, (G,4)>>
[G] - (G,∞), <G: <∞>>
SSSP: MapReduce Model
Reduce後、HDFS のフラッシュ
![Page 30: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/30.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
4
3
6
+∞
56
30
・Map Input (Reduce Output):- <A: <0, (B,5),(D,3)>>
- <B: <4, (E,1)>>
- <C: <6, (F,5)>>
- <D: <3, (B,1),(C,3),(E,4),(F,2)>>
- <E: <6>>
- <F: <5, (G,4)>>
- <G: <∞>>
SSSP: MapReduce Model
![Page 31: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/31.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
4
3
6
+∞
56
30
・Map Output:- (B,5),(D,3), <A: <0, (B,5),(D,3)>>
- (E,5), <B: <4, (E,1)>>
- (F,11), <C: <6, (F,5)>>
- (B,4),(C,6),(E,7),(F,5),
<D: <3, (B,1),(C,3),(E,4),(F,2)>>
- <E: <6>>
- (G,9), <F: <5, (G,4)>>
- <G: <∞>>
SSSP: MapReduce Model
Local Disk のFlush
![Page 32: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/32.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
4
3
5
9
56
30
・Reduce Process:[A] - <A: <0, (B,5),(D,3)>>
[B] - (B,5),(B,4), <B: <4, (E,1)>>
[C] - (C,6), <C: <6, (F,5)>>
[D] - (D,3),
<D: <3, (B,1),(C,3),(E,4),(F,2)>>
[E] - (E,5), (E,7), <E, <6>>
[F] - (F,5),(F,11), <F: <5, (G,4)>>
[G] - (G,9), <G: <∞>>
SSSP: MapReduce Model
![Page 33: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/33.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
4
end
5
9
56
30
SSSP: MapReduce Model
![Page 34: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/34.jpg)
SSSP: Parallel BFS
MapReduce & Pregel
※ SSSP: Single Source Shortest Paths, BFS: Breadth First Search
![Page 35: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/35.jpg)
SSSP: Pregel Model
B
C
D
E
F
GA
5
1
3
5
42
41
3
+∞
1
+∞
+∞
+∞+∞
+∞0
P1 P2 P1 P2Compute()
Comupte():Vertexの値と前回のステップからの
メッセージを元に計算
![Page 36: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/36.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
+∞
1
+∞
+∞
+∞+∞
+∞0
P1 P2 P1 P2Compute()
Communicate
SSSP: Pregel Model
Communicate:更新した値を矢線の出る方のノード
へメッセージパッシング
![Page 37: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/37.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
+∞
1
+∞
+∞
+∞+∞
+∞0
P1 P2 P1 P2Compute()
CommunicateBarrier
SSSP: Pregel Model
Barrier:全てのノードへメッセージパッシン
グが終了するまで待機
![Page 38: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/38.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
5
2
+∞
+∞
+∞+∞
30
P1 P2 P1 P2Compute()
CommunicateBarrier
Compute()
SSSP: Pregel Model
Compute():Vertexの値と前回のステップからの
メッセージを元に計算
![Page 39: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/39.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
5
2
+∞
+∞
+∞+∞
30
P1 P2 P1 P2Compute()
CommunicateBarrier
Compute()Communicate
Barrier
SSSP: Pregel Model
Communicate & Barrier:更新した値を矢線の出る方のノードへメッセージパッシング、待機
![Page 40: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/40.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
4
3
6
+∞
56
30
P1 P2 P1 P2Compute()
CommunicateBarrier
Compute()Communicate
BarrierCompute()
SSSP: Pregel Model
![Page 41: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/41.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
4
3
6
+∞
56
30
P1 P2 P1 P2Compute()
CommunicateBarrier
Compute()Communicate
BarrierCompute()
CommunicateBarrier
SSSP: Pregel Model
![Page 42: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/42.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
4
4
5
9
56
30
P1 P2 P1 P2Compute()
CommunicateBarrier
Compute()Communicate
BarrierCompute()
CommunicateBarrier
Compute()
SSSP: Pregel Model
![Page 43: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/43.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
4
4
5
9
56
30
P1 P2 P1 P2Compute()
CommunicateBarrier
Compute()Communicate
BarrierCompute()
CommunicateBarrier
Compute()Communicate
Barrier
SSSP: Pregel Model
![Page 44: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/44.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
4
5
5
9
56
30
P1 P2 P1 P2Compute()
CommunicateBarrier
Compute()Communicate
BarrierCompute()
CommunicateBarrier
Compute()Communicate
Barrier
SSSP: Pregel Model
Compute()
![Page 45: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/45.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
4
5
5
9
56
30
P1 P2 P1 P2Compute()
CommunicateBarrier
Compute()Communicate
BarrierCompute()
CommunicateBarrier
Compute()Communicate
Barrier
SSSP: Pregel Model
Compute()Communicate
Barrier
![Page 46: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/46.jpg)
B
C
D
E
F
GA
5
1
3
5
42
41
3
4
end
5
9
56
30
P1 P2 P1 P2Compute()
CommunicateBarrier
Compute()Communicate
BarrierCompute()
CommunicateBarrier
Compute()Communicate
Barrier
SSSP: Pregel Model
Compute()Communicate
BarrierTerminate
![Page 47: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/47.jpg)
・MapReduce:- “Dence” なグラフに対する処理は苦手- ネットワーク通信は状態とグラフ構造- (基本的な問題に関しては最適化可能)
・Pregel:- シンプルなアルゴリズム- ネットワーク通信はメッセージのみ
MapReduce v.s. Pregel大規模グラフデータの場合のネットワークコストは莫大
シンプルさは重要:そもそもアルゴリズムを考案、実
装できるか
![Page 48: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/48.jpg)
現在使用できるライブラリはあるのん?
1. Apache Hama2. Apache Giraph3. GoldenOrb
![Page 49: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/49.jpg)
Hama GoldenOrb Giraph
Logo
API BSP Pregel(Graph) Pregel(Graph)
NextGen MR 対応 ? 対応
Lincense Apache Apache Apache
Infrastructure 必要 必要 不要(on Hadoop)
Hama, GoldenOrb, Giraph
![Page 50: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/50.jpg)
Hama GoldenOrb Giraph
Logo
API BSP Pregel(Graph) Pregel(Graph)
NextGen MR 対応 ? 対応
Lincense Apache Apache Apache
Infrastructure 必要 必要 不要(on Hadoop)
Hama, GoldenOrb, Giraph
YARN 対応!
HamaはBSP全般を扱う
Hadoop上でのMapのイテレーション
Pregelに準拠したGraphAPI
![Page 51: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/51.jpg)
データの多様性の増大と共に計算モデルの多様化も
進む
データを見る目と、適切なツール・モデルを選択する能力が求められる時代に!
![Page 52: Large-Scale Graph Processing〜Introduction〜(LT版)](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b72cc84a795903318b45f0/html5/thumbnails/52.jpg)
ありがとうございました
@doryokujinHadoop Conference Japan 2011 Fall