computer organization: a programmer's...
TRANSCRIPT
Computer Organization:A Programmer's Perspective
Optimizing for Cache Performance
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 2
Cache Performance MetricsCache Performance MetricsMiss Rate● Fraction (percent) of memory references not found in cache ● Typical numbers (misses/references):
● 3-10% for L1● can be quite small (e.g., < 1%) for L2, depending on size, etc.
Hit Time● Time to deliver a line in the cache to the processor
● includes time to determine whether the line is in the cache● Typical numbers:
● 4 clock cycles for L1● 10 clock cycles for L2
Miss Penalty● Additional time required because of a miss● Typically 50-200 cycles for main memory (trend: increasing!)
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 3
Let’s think about those numbers
Huge difference between a hit and a miss Could be 100x, if just L1 and main memory
Would you believe 99% hits is twice as good as 97%? Consider:
cache hit time of 1 cyclemiss penalty of 100 cycles
Average access time:
97% hits: 1 cycle + 0.03 * 100 cycles = 4 cycles
99% hits: 1 cycle + 0.01 * 100 cycles = 2 cycles
This is why “miss rate” is used instead of “hit rate”
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 4
Writing Cache Friendly CodeWriting Cache Friendly Code
Repeated references to variables are good (temporal locality)
Stride-1 reference patterns are good (spatial locality)
Examples:cold cache, 4-byte words, 4-word cache blocks
int sumarrayrows(int a[M][N]){ int i, j, sum = 0;
for (i = 0; i < M; i++) for (j = 0; j < N; j++) sum += a[i][j]; return sum;}
int sumarraycols(int a[M][N]){ int i, j, sum = 0;
for (j = 0; j < N; j++) for (i = 0; i < M; i++) sum += a[i][j]; return sum;}
Miss rate = Miss rate = 1/4 = 25% 100%
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 5
The Memory MountainThe Memory Mountain
Read throughput (read bandwidth)Number of bytes read from memory per second (MB/s)
Memory mountainMeasured read throughput as a function of spatial and temporal
locality.Compact way to characterize memory system performance.
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 6
The Memory MountainThe Memory Mountain
s1
s3
s5
s7
s9
s11
s13
s15
8m
2m 512k 12
8k 32k 8k
2k
0
200
400
600
800
1000
1200
read
th
rou
gh
pu
t (M
B/s
)
stride (words) working set size (bytes)
Pentium III Xeon550 MHz16 KB on-chip L1 d-cache16 KB on-chip L1 i-cache512 KB off-chip unifiedL2 cache
Ridges ofTemporalLocality
L1
L2
mem
Slopes ofSpatialLocality
xe
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 7
Ridges of Temporal LocalityRidges of Temporal Locality
Slice through the memory mountain with stride=1illuminates read throughputs of different caches and memory
0
200
400
600
800
1000
1200
8m
4m
2m
10
24
k
51
2k
25
6k
12
8k
64
k
32
k
16
k
8k
4k
2k
1k
working set size (bytes)
rea
d t
hro
ug
pu
t (M
B/s
)
L1 cacheregion
L2 cacheregion
main memoryregion
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 8
A Slope of Spatial LocalityA Slope of Spatial Locality
Slice through memory mountain with size=256KBshows cache block size.
0
100
200
300
400
500
600
700
800
s1 s2 s3 s4 s5 s6 s7 s8 s9 s10 s11 s12 s13 s14 s15 s16
stride (words)
rea
d t
hro
ug
hp
ut
(MB
/s)
one access per cache line
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 9
Matrix Multiplication ExampleMatrix Multiplication Example
Major Cache Effects to Consider Total cache size
Exploit temporal locality and keep the working set small (e.g., by using blocking) Block size
Exploit spatial locality
Description: Multiply N x N matrices O(N^3) total operations Accesses
N reads per source element N values summed per destination
but may be able to hold in register
/* ijk */for (i=0; i<n; i++) { for (j=0; j<n; j++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum; }}
/* ijk */for (i=0; i<n; i++) { for (j=0; j<n; j++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum; }}
Variable sumheld in register
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 10
Miss Rate Analysis for Matrix MultiplyMiss Rate Analysis for Matrix Multiply
Assume: Line size = 32B (big enough for 4 64-bit words) Matrix dimension (N) is very large
Approximate 1/N as 0.0 Cache is not even big enough to hold multiple rows
Analysis Method: Look at access pattern of inner loop
CA
k
i
B
k
j
i
j
=x
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 11
Layout of C Arrays in Memory Layout of C Arrays in Memory C arrays allocated in row-major order
each row in contiguous memory locations
Stepping through columns in one row: for (i = 0; i < N; i++)
sum += a[0][i]; accesses successive elements if block size (B) > sizeof(a[i][j]), exploit spatial locality
miss rate = sizeof(a[i][j]) / B
Stepping through rows in one column: for (i = 0; i < n; i++)
sum += a[i][0]; accesses distant elements no spatial locality!
miss rate = 1 (i.e. 100%)
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 12
Matrix Multiplication (ijk)Matrix Multiplication (ijk)
/* ijk */for (i=0; i<n; i++) { for (j=0; j<n; j++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum; }}
/* ijk */for (i=0; i<n; i++) { for (j=0; j<n; j++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum; }}
A B C
(i,*)
(*,j)
(i,j)
Inner loop:
Column-wise
Row-wise Fixed
Misses per Inner Loop Iteration:A B C
0.25 1.0 0.0
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 13
Matrix Multiplication (jik)Matrix Multiplication (jik)
/* jik */for (j=0; j<n; j++) { for (i=0; i<n; i++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum }}
/* jik */for (j=0; j<n; j++) { for (i=0; i<n; i++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum }}
A B C
(i,*)
(*,j)
(i,j)
Inner loop:
Row-wise Column-wise
Fixed
Misses per Inner Loop Iteration:A B C
0.25 1.0 0.0
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 14
Matrix Multiplication (kij)Matrix Multiplication (kij)
/* kij */for (k=0; k<n; k++) { for (i=0; i<n; i++) { r = a[i][k]; for (j=0; j<n; j++) c[i][j] += r * b[k][j]; }}
/* kij */for (k=0; k<n; k++) { for (i=0; i<n; i++) { r = a[i][k]; for (j=0; j<n; j++) c[i][j] += r * b[k][j]; }}
A B C
(i,*)(i,k) (k,*)
Inner loop:
Row-wise Row-wiseFixed
Misses per Inner Loop Iteration:A B C
0.0 0.25 0.25
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 15
Matrix Multiplication (ikj)Matrix Multiplication (ikj)
/* ikj */for (i=0; i<n; i++) { for (k=0; k<n; k++) { r = a[i][k]; for (j=0; j<n; j++) c[i][j] += r * b[k][j]; }}
/* ikj */for (i=0; i<n; i++) { for (k=0; k<n; k++) { r = a[i][k]; for (j=0; j<n; j++) c[i][j] += r * b[k][j]; }}
A B C
(i,*)(i,k) (k,*)
Inner loop:
Row-wise Row-wiseFixed
Misses per Inner Loop Iteration:A B C
0.0 0.25 0.25
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 16
Matrix Multiplication (jki)Matrix Multiplication (jki)
/* jki */for (j=0; j<n; j++) { for (k=0; k<n; k++) { r = b[k][j]; for (i=0; i<n; i++) c[i][j] += a[i][k] * r; }}
/* jki */for (j=0; j<n; j++) { for (k=0; k<n; k++) { r = b[k][j]; for (i=0; i<n; i++) c[i][j] += a[i][k] * r; }}
A B C
(*,j)
(k,j)
Inner loop:
(*,k)
Column -wise
Column-wise
Fixed
Misses per Inner Loop Iteration:A B C
1.0 0.0 1.0
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 17
Matrix Multiplication (kji)Matrix Multiplication (kji)
/* kji */for (k=0; k<n; k++) { for (j=0; j<n; j++) { r = b[k][j]; for (i=0; i<n; i++) c[i][j] += a[i][k] * r; }}
/* kji */for (k=0; k<n; k++) { for (j=0; j<n; j++) { r = b[k][j]; for (i=0; i<n; i++) c[i][j] += a[i][k] * r; }}
A B C
(*,j)
(k,j)
Inner loop:
(*,k)
FixedColumn-wise
Column-wise
Misses per Inner Loop Iteration:A B C
1.0 0.0 1.0
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 18
Summary of Matrix MultiplicationSummary of Matrix Multiplication
for (i=0; i<n; i++) {
for (j=0; j<n; j++) {
sum = 0.0;
for (k=0; k<n; k++)
sum += a[i][k] * b[k][j];
c[i][j] = sum;
}
}
ijk (& jik): 2 loads, 0 stores misses/iter = 1.25
for (k=0; k<n; k++) {
for (i=0; i<n; i++) {
r = a[i][k];
for (j=0; j<n; j++)
c[i][j] += r * b[k][j];
}
}
for (j=0; j<n; j++) {
for (k=0; k<n; k++) {
r = b[k][j];
for (i=0; i<n; i++)
c[i][j] += a[i][k] * r;
}
}
kij (& ikj): 2 loads, 1 store misses/iter = 0.5
jki (& kji): 2 loads, 1 store misses/iter = 2.0
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 19
Core i7 Matrix Multiply Performance
50 100 150 200 250 300 350 400 450 500 550 600 650 7001
10
100
jkikjiijkjikkijikj
Array size (n)
Cyc
les
per
in
ner
lo
op
ite
rati
on
ijk / jik
jki / kji
kij / ikj
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 20
Improving Temporal Locality by BlockingImproving Temporal Locality by BlockingExample: Blocked matrix multiplication
“block” (in this context) does not mean “cache block”.Instead, it mean a sub-block within the matrix.Example: N = 8; sub-block size = 4
C11 = A11B11 + A12B21 C12 = A11B12 + A12B22
C21 = A21B11 + A22B21 C22 = A21B12 + A22B22
A11 A12
A21 A22
B11 B12
B21 B22X =
C11 C12
C21 C22
Key idea: Sub-blocks (i.e., Axy) can be treated just like scalars.
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 21
Blocked Matrix Multiply (bijk)Blocked Matrix Multiply (bijk)
for (jj=0; jj<n; jj+=bsize) { for (i=0; i<n; i++) for (j=jj; j < min(jj+bsize,n); j++) c[i][j] = 0.0; for (kk=0; kk<n; kk+=bsize) { for (i=0; i<n; i++) { for (j=jj; j < min(jj+bsize,n); j++) { sum = 0.0 for (k=kk; k < min(kk+bsize,n); k++) { sum += a[i][k] * b[k][j]; } c[i][j] += sum; } } }}
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 22
Blocked Matrix Multiply AnalysisBlocked Matrix Multiply Analysis
Innermost loop pair multiplies a 1 X bsize sliver of A by a bsize X bsize block of B and accumulates into 1 X bsize sliver of C
Loop over i steps through n row slivers of A & C, using same B
A B C
block reused n times in succession
row sliver accessedbsize times
Update successiveelements of sliver
i ikk
kk jjjj
for (i=0; i<n; i++) { for (j=jj; j < min(jj+bsize,n); j++) { sum = 0.0 for (k=kk; k < min(kk+bsize,n); k++) { sum += a[i][k] * b[k][j]; } c[i][j] += sum; }Innermost
Loop Pair
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 23
Pentium Blocked Matrix Multiply PerformancePentium Blocked Matrix Multiply PerformanceBlocking (bijk and bikj) improves performance by a factor
of two over unblocked versions (ijk and jik)relatively insensitive to array size.
0
10
20
30
40
50
60
Array size (n)
Cy
cle
s/it
era
tio
n
kji
jki
kij
ikj
jik
ijk
bijk (bsize = 25)
bikj (bsize = 25)
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 24
Concluding ObservationsConcluding Observations Programmer can optimize for cache performance
How data structures are organized How data are accessed
Nested loop structure Blocking is a general technique
All systems favor “cache friendly code” Getting absolute optimum performance is very platform specific
Cache sizes, line sizes, associativities, etc. Can get most of the advantage with generic code
Keep working set reasonably small (temporal locality) Use small strides (spatial locality)
Computer Organization: A Programmer's Perspective Based on class notes by Bryant and O'Hallaron 25
?שאלות