lecture 6 objectives communication complexity analysis collective operations –reduction...
Post on 20-Dec-2015
223 views
TRANSCRIPT
![Page 1: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/1.jpg)
Lecture 6
• Objectives
• Communication Complexity Analysis
• Collective Operations– Reduction– Binomial Trees– Gather and Scatter Operations
• Review Communication Analysis of Floyd’s Algorithm
![Page 2: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/2.jpg)
Parallel Reduction Evolution
![Page 3: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/3.jpg)
Binomial Trees
Subgraph of hypercube
![Page 4: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/4.jpg)
Finding Global Sum
4 2 0 7
-3 5 -6 -3
8 1 2 3
-4 4 6 -1
![Page 5: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/5.jpg)
Finding Global Sum
1 7 -6 4
4 5 8 2
![Page 6: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/6.jpg)
Finding Global Sum
8 -2
9 10
![Page 7: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/7.jpg)
Finding Global Sum
17 8
![Page 8: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/8.jpg)
Finding Global Sum
25
Binomial Tree
![Page 9: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/9.jpg)
Agglomeration
![Page 10: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/10.jpg)
Agglomeration
sum
sum sum
sum
![Page 11: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/11.jpg)
Gather
![Page 12: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/12.jpg)
All-gather
![Page 13: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/13.jpg)
Complete Graph for All-gather
![Page 14: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/14.jpg)
Hypercube for All-gather
![Page 15: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/15.jpg)
Analysis of Communication
n
• Lambda is latency
= message delay
= overhead to send 1 message
• Beta is bandwidth
= number of data items per unit time
= bytes per message
Sending a message with n data items costs
![Page 16: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/16.jpg)
Communication Time for All-Gather
p
pnp
p
np
i
)1(
log2
log
1
1-i
Hypercube
Complete graph
p
pnp
pnp
)1(
)1()/
)(1(
![Page 17: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/17.jpg)
Adding Data Input
![Page 18: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/18.jpg)
Scatter
![Page 19: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/19.jpg)
Scatter in log p Steps
12345678 56781234 56 12
7834
![Page 20: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/20.jpg)
Communication Time for Scatter
p
pnp
p
np
i
)1(
log2
log
1
1-i
Hypercube
Complete graph
p
pnp
pnp
)1(
)1()/
)(1(
![Page 21: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/21.jpg)
Recall Parallel Floyd’s Computational Complexity
• Innermost loop has complexity (n)
• Middle loop executed at most n/p times
• Outer loop executed n times
• Overall computation complexity (n3/p)
![Page 22: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/22.jpg)
Floyd’s Communication Complexity
• No communication in inner loop• No communication in middle loop• Broadcast in outer loop — complexity is
• Executed n times
)/(log np
![Page 23: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/23.jpg)
Execution Time Expression (1)
)/4(log/ npnnpnn
Iterations of outer loopIterations of middle loop
Cell update time
Iterations of outer loop
Messages per broadcastMessage-passing time bytes/msg
Iterations of inner loop
![Page 24: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/24.jpg)
Accounting for Computation/communication Overlap
Note that after the 1st broadcast all the wait times overlap the computation time of Process 0.
![Page 25: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/25.jpg)
Execution Time Expression (2)
Iterations of outer loopIterations of middle loop
Cell update time
Iterations of outer loop
Messages per broadcastMessage-passing time
Iterations of inner loop
/4loglog/ nppnnpnn Message transmission
![Page 26: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/26.jpg)
Predicted vs. Actual Performance
Execution Time (sec)
Processes Predicted Actual
1 25.54 25.54
2 13.02 13.89
3 9.01 9.60
4 6.89 7.29
5 5.86 5.99
6 5.01 5.16
7 4.40 4.50
8 3.94 3.98
X=25.5 nsec
L = 250 usecs
B = 10MB/sec
N = 1000
![Page 27: Lecture 6 Objectives Communication Complexity Analysis Collective Operations –Reduction –Binomial Trees –Gather and Scatter Operations Review Communication](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649d415503460f94a1b34d/html5/thumbnails/27.jpg)
Summary
• Two matrix decompositions– Rowwise block striped– Columnwise block striped
• Blocking send/receive functions– MPI_Send– MPI_Recv
• Overlapping communications with computations