![Page 1: Minimal MapReduce Algorithms Yufei Tao Chinese University of Hong Kong, Hong Kong](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649d6f5503460f94a51a8b/html5/thumbnails/1.jpg)
Minimal MapReduce Algori
thms
Yufei Tao
Chinese University of Ho
ng Kong, Hong Kong
![Page 2: Minimal MapReduce Algorithms Yufei Tao Chinese University of Hong Kong, Hong Kong](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649d6f5503460f94a51a8b/html5/thumbnails/2.jpg)
outline
• INTRODUCTION• PRELIMINARY AND RELATED WORK• SORTING• BASIC MINIMAL ALGORITHMS IN DATABAS
ES• SLIDING AGGREGATION• EXPERIMENTS• CONCLUSIONS
![Page 3: Minimal MapReduce Algorithms Yufei Tao Chinese University of Hong Kong, Hong Kong](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649d6f5503460f94a51a8b/html5/thumbnails/3.jpg)
introduction• Motivation Although these principles have guided th
e design of MapReduce algorithms, the previous practices have mostly been on a best-effort basis, paying relatively less attention to enforcing serious constraints on different performance metrics.
![Page 4: Minimal MapReduce Algorithms Yufei Tao Chinese University of Hong Kong, Hong Kong](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649d6f5503460f94a51a8b/html5/thumbnails/4.jpg)
introduction• Minimal MapReduce Algorithms
Minimum footprint.Minimum footprint.Bounded net-trafficBounded net-trafficConstant roundConstant roundOptimal computationOptimal computation
![Page 5: Minimal MapReduce Algorithms Yufei Tao Chinese University of Hong Kong, Hong Kong](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649d6f5503460f94a51a8b/html5/thumbnails/5.jpg)
introduction• Contributions
The core of this work comprises of neat mini
mal algorithms for two problems:
SortingSortingSliding AggregationSliding Aggregation
![Page 6: Minimal MapReduce Algorithms Yufei Tao Chinese University of Hong Kong, Hong Kong](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649d6f5503460f94a51a8b/html5/thumbnails/6.jpg)
introductionSortingSortingSliding AggregationSliding Aggregation
![Page 7: Minimal MapReduce Algorithms Yufei Tao Chinese University of Hong Kong, Hong Kong](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649d6f5503460f94a51a8b/html5/thumbnails/7.jpg)
related work
MapReduceMapReduceTeraSortTeraSortAlgorithms on MapReduceAlgorithms on MapReduceRelevance to Minimal AlgorithmsRelevance to Minimal Algorithms
![Page 8: Minimal MapReduce Algorithms Yufei Tao Chinese University of Hong Kong, Hong Kong](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649d6f5503460f94a51a8b/html5/thumbnails/8.jpg)
related work-MR
Statelessness for Fault ToleranceStatelessness for Fault Tolerance
Some MapReduce implementations (e.g., Hadoop) place the requirement that, at the end of a round, each machine should send all the data in its storage to a distributed file system.
![Page 9: Minimal MapReduce Algorithms Yufei Tao Chinese University of Hong Kong, Hong Kong](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649d6f5503460f94a51a8b/html5/thumbnails/9.jpg)
related work-TS
What's TeraSort?What's TeraSort?
![Page 10: Minimal MapReduce Algorithms Yufei Tao Chinese University of Hong Kong, Hong Kong](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649d6f5503460f94a51a8b/html5/thumbnails/10.jpg)
sorting-TS
![Page 11: Minimal MapReduce Algorithms Yufei Tao Chinese University of Hong Kong, Hong Kong](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649d6f5503460f94a51a8b/html5/thumbnails/11.jpg)
sortingDefine Si = S ∩(bi−1, bi], for 1 ≤ i ≤ t. In Round 2, all the objects in Si are gathered by Mi, which sorts them in the reducephase. For TeraSort to be minimal, it must hold:P1. s = O(m).P1. s = O(m).P2. |Si| = O(m) for all 1 ≤ i ≤ tP2. |Si| = O(m) for all 1 ≤ i ≤ t
![Page 12: Minimal MapReduce Algorithms Yufei Tao Chinese University of Hong Kong, Hong Kong](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649d6f5503460f94a51a8b/html5/thumbnails/12.jpg)
sortingPr
Discussion
Minimality
![Page 13: Minimal MapReduce Algorithms Yufei Tao Chinese University of Hong Kong, Hong Kong](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649d6f5503460f94a51a8b/html5/thumbnails/13.jpg)
sorting
Removing the Broadcast Assumption
(by changing round 1)
![Page 14: Minimal MapReduce Algorithms Yufei Tao Chinese University of Hong Kong, Hong Kong](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649d6f5503460f94a51a8b/html5/thumbnails/14.jpg)
in databases
Ranking & Skyline
Group by
Semi-Join
![Page 15: Minimal MapReduce Algorithms Yufei Tao Chinese University of Hong Kong, Hong Kong](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649d6f5503460f94a51a8b/html5/thumbnails/15.jpg)
in databasesGroup by
example
![Page 16: Minimal MapReduce Algorithms Yufei Tao Chinese University of Hong Kong, Hong Kong](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649d6f5503460f94a51a8b/html5/thumbnails/16.jpg)
sliding aggregation
,
,
( )
( ) ( )o window o
win sum o w o
The window sum of o equal:
![Page 17: Minimal MapReduce Algorithms Yufei Tao Chinese University of Hong Kong, Hong Kong](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649d6f5503460f94a51a8b/html5/thumbnails/17.jpg)
sliding aggregation
Sorting with Perfect Balance
![Page 18: Minimal MapReduce Algorithms Yufei Tao Chinese University of Hong Kong, Hong Kong](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649d6f5503460f94a51a8b/html5/thumbnails/18.jpg)
sliding aggregation
Sliding Aggregate Computation
![Page 19: Minimal MapReduce Algorithms Yufei Tao Chinese University of Hong Kong, Hong Kong](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649d6f5503460f94a51a8b/html5/thumbnails/19.jpg)
experiments-sorting
![Page 20: Minimal MapReduce Algorithms Yufei Tao Chinese University of Hong Kong, Hong Kong](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649d6f5503460f94a51a8b/html5/thumbnails/20.jpg)
experiments-sorting
![Page 21: Minimal MapReduce Algorithms Yufei Tao Chinese University of Hong Kong, Hong Kong](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649d6f5503460f94a51a8b/html5/thumbnails/21.jpg)
experiments-skyline
![Page 22: Minimal MapReduce Algorithms Yufei Tao Chinese University of Hong Kong, Hong Kong](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649d6f5503460f94a51a8b/html5/thumbnails/22.jpg)
本篇论文的主要贡献是填充了
最小 MR 算法概念一个空隙。。
thx @hh's