level-3 blas and lu factorization on a matrix processor

Vol. 49 No. SIG 2(ACS 21) IPSJ Transactions on Advanced Computing Systems Mar. 2008

Regular Paper

Level-3 BLAS and LU Factorization on a Matrix Processor

Ahmed S. Zekri†1 and Stanislav G. Sedukhin†1

As increasing clock frequency approaches its physical limits, a good approach to enhanceperformance is to increase parallelism by integrating more cores as coprocessors to general-purpose processors in order to handle the different workloads in scientific, engineering, andsignal processing applications. In this paper, we propose a many-core matrix processor modelconsisting of a scalar unit augmented with b×b simple cores tightly connected in a 2D torusmatrix unit to accelerate matrix-based kernels. Data load/store is overlapped with computingusing a decoupled data access unit that moves b×b blocks of data between memory and thetwo scalar and matrix processing units. The operation of the matrix unit is mainly processingfine-grained b×b matrix multiply-add (MMA) operations. We formulate the data alignmentoperations including matrix transposition and skewing as MMA operations in order to overlapthem with data load/store. Two fundamental linear algebra algorithms are designed and an-alytically evaluated on the proposed matrix processor: the Level-3 BLAS kernel, GEMM, andthe LU factorization with partial pivoting, the main step in solving linear systems of equa-tions. For the GEMM kernel, the maximum speed of computing measured in FLOPs/cycle isapproached for different matrix sizes, n, and block sizes, b. The speed of the LU factorizationfor relatively large values of n ranges from around 50–90% of the maximum speed dependingon the model parameters. Overall, the analytical results show the merits of using the matrixunit for accelerating the matrix-based applications.

1. Introduction

The idea of attaching accelerators or co-processors to general-purpose processors forenhancing the compute-intensive parts of ap-plications has been used and is now gettingmore attention due to approaching the phys-ical limits of VLSI technology. One recentexample is the STI Cell/BE processor whichhas eight special-purpose Synergistic Process-ing Elements (SPEs) augmented to a PowerPCgeneral-purpose microprocessor to enhance vec-tor operations in graphics and scientific ap-plications 1). Other example accelerators in-clude: vector co-processors 2),3) that enhancethe floating-point workloads found in scientificapplications, and Graphics Processing Units(GPUs) that can efficiently deal with graphicsand streaming applications.

The basic linear algebra subprograms (BLAS)were introduced to make the performance ofdense linear algebra algorithms portable onhigh performance computers 4)–6). There arethree levels of BLAS according to the complex-ity of computations. The most important ofthese levels is Level-3 BLAS, and the most im-portant of Level-3 BLAS is the GEneral MatrixMultiply operation (GEMM) since it is pos-

†1 Department of Information Systems, The Universityof Aizu

sible to express all Level-3 BLAS kernels interms of the highly optimized GEMM and asmall percentage of Level-1 and Level-2 of theBLAS 7)–9).

Solving linear systems of equations is oneof the most important computations in scien-tific computing. The most compute-intensivepart of the solution process is the factorizationphase. The standard factorization algorithmusing Gaussian elimination can be expressed us-ing levels 1 and 2 of the BLAS. However, theselevels have a low degree of data reuse and hencecouldn’t reach the potential speed on high per-formance processors 10). To use the GEMM-based Level-3 BLAS, the matrices and vectorsare segmented into blocks that can reside in theupper levels of the storage hierarchy (registers,cache, local memory) and be maximally reusedbefore moving to lower levels of storage (off-chip memories and disks) 11),12). In so doing thespeed gap between processing and memory canbe hidden and consequently the performance ofthe LU factorization increases.

Our motivation in this research is that ma-trix multiplication is a pervasive operation, andmany scientific and signal processing applica-tions can be formulated in terms of matrix-matrix multiplication 11),13). Moreover, thetrend in chip-manufacturing is to put moreprocessing units or cores together to handlethe diverse demands of current and future ap-

37

38 IPSJ Transactions on Advanced Computing Systems Mar. 2008

plications. However, many problems remainunsolved such as the interconnection topol-ogy that guarantees scalability, the applicationsthat could benefit from all of these parallel re-sources, the memory organization that keepspace with the powerful processing speeds, etc.

In this paper, we propose a model of a many-core matrix processor consisting of a scalar unitaugmented with b×b simple cores tightly con-nected into a 2D torus SIMD matrix unit to ac-celerate matrix-based applications. The SIMDmatrix unit uses local communications betweenPEs and avoids global connections (the latencyof the long wrap-around connections can beovercome by folding the torus 14)), and henceit is scalable. The matrix unit is optimizedfor executing fine-grained b×b MMA opera-tions in the minimal computing time. A keypoint that distinguishes our proposed acceler-ator from others such as ClearSpeed CSX-600processor, Cell processor and GPUs is that lo-cal communications between PEs are mergedand overlapped with computing. Specifically,each PE performs a scalar ‘multiply-add-roll’as one operation. In this operation the scalarmultiply-add c←c+a×b, is executed while bothscalars a and b, or a (or b) and c are rolledor circularly shifted to the appropriate neigh-bor PEs. The rolling of a or b is done simulta-neously with computing c while in the case ofrolling the result c, it must be done at the endof the multiply-add-roll operation due to datadependency. Using software pipelining and loopunrolling optimization techniques, the through-put of the multiply-add-roll operation can reachone multiply-add-roll operation every clock cy-cle. Consequently, the theoretical peak perfor-mance of the proposed b×b matrix unit canreach 2b2 FLOPs/cycle.

We present three optimal ways to performthe b×b MMA on the matrix unit 15),16). Toimplement these ways, we propose using threeforms of the multiply-add-roll operation differ-ing in the directions of data rolling (see Sec-tion 2). The three optimal ways that per-form the b×b MMA have different data dis-tributions and require alignment overhead be-fore and may be after computing. One simplesolution to the alignment problem is to storedata in the aligned form. Indeed, this is notpractical since alignment is also needed dur-ing all the processing time. Our approach isto hide this alignment overhead by formulatingthe alignment operations, including the matrix

transpose, as MMA operations executed on thematrix unit. Using a decoupled load/store unit,we could overlap the alignment overhead withthe load/store operations.

To evaluate the merits of the proposed ma-trix processor, we designed our algorithms toimplement the Level-3 BLAS, represented bythe GEMM kernel, and the LU factorizationwith partial pivoting. It was recently recog-nized that storing matrices by square blocksenhances the performance of memory hierarchyand consequently on dense linear algebra appli-cations 17),18). Therefore, the idea behind ouralgorithm design is that the matrices are storedas b×b blocks rather than row- or column-major formats. This required the load/storeunit to move contiguous b×b blocks to boththe scalar and matrix units. Therefore, anyblock loaded into the matrix registers and/ordata cache must be maximally reused in orderto reduce the cost of the load/store operations.Our analytical evaluation of the GEMM kernelshowed that the maximum speed of computingmeasured in FLOPs/cycle for different matrixsizes is approached.

The LU factorization with partial pivotingis an example of an application that includesscalar, vector and matrix operations. Basedon storing the coefficient matrix as squareb×b blocks, we designed a fine-grained blockedright-looking algorithm to run on both the ma-trix and scalar processing units. We did notconsider execution overlapping between the twoprocessing units. We scheduled the primitivesof the algorithm so that only the matrix mul-tiplication part is executed on the matrix unitand all other parts on the scalar unit. This re-quired returning results to the main memorybefore switching computing between the twoprocessing units. We included this data move-ment overhead in our timing model. Our anal-ysis showed that the fraction of matrix mul-tiplication in this algorithm dominates as thesize of the matrix increases and consequently,the speed of computing is asymptotically deter-mined by the speed of the matrix co-processoron executing the MMA kernel. The analyticalresults showed the speed of the LU factoriza-tion with partial pivoting for large matrix size32 K ranges from around 50–90% of the maxi-mum speed depending on the model parameterswhich we will discuss.

In Section 2, a description of the proposedmatrix processor and its timing model is pre-

Vol. 49 No. SIG 2(ACS 21) Level-3 BLAS and LU Factorization on a Matrix Processor 39

sented. In Section 3, three optimal data distri-butions to execute the b×b MMA operation onthe matrix unit are presented. The formulationof the alignment operations as matrix multiply-add operations is introduced, and an algorithmof one variant of the GEMM operation is de-scribed and its predicted execution time is cal-culated. In Section 4, the standard LU factor-ization algorithm is discussed, and we designeda blocked right-looking algorithm and show howits primitives are scheduled between the scalarand matrix units. In addition, an algorithm foreach primitive is described using software pre-fetching techniques, and the expected times arecomputed. Performance evaluation of the pro-posed algorithms is demonstrated in Section 5.Section 6 concludes the paper.

2. The Matrix Processor

2.1 Architecture ModelThe proposed matrix processor is regarded

as a 2D matrix register file augmented to ageneral-purpose scalar processor to acceleratematrix-based applications. The matrix registerfile is partitioned into b×b banks where eachbank is directly connected to a simple core orPE. This partitioning of the matrix register fileis called element-partitioned 19) since a matrix(or vector) is partitioned into b×b blocks (or b-element segments) so that each element residesin a different bank. The PEs are tightly coupledto form a 2D b×b torus matrix unit where thelatency of the long wrap-around connectionscan be overcomed by folding the torus such thatall connections between PEs become equal 14).All PEs are working synchronously in a Sin-gle Instruction Multiple Data (SIMD) mode. Amatrix instruction causes the PEs to processthe elements in the same position at all registerfile banks simultaneously. Therefore, the ma-trix register file is viewed as a number of layersor matrix registers (see Fig. 1). Communica-tions between PEs are local, and each PE cansend and receive data to/from its four NEWSneighbors (North, East, West, and South) atthe same time. For example, it can simulta-neously receive data from South and East andsend data and/or partial results to North andWest, respectively.

The basic operation of the matrix unit is ex-ecuting fine-grained b×b MMA operations inb multiply-add-roll time-steps where the scalarmultiply-add c←c+a×b is performed while theappropriate data is rolled to neighbor PEs. Our

Fig. 1 The proposed matrix processor block diagram.

idea is to merge and overlap the rolling of datawith computing. That is, the rolling of c isdone at the end of the multiply-add-roll opera-tion while the rolling of a and b is done simul-taneously with computing c. We propose threeforms of the multiply-add-roll operation corre-sponding to the three optimal ways (see Section3) to execute the b×b MMA operation on thematrix processor. The difference between thethree forms is the direction of data rolling. Inone form, both b and a are rolled northward andwestward, respectively. In the other two forms,b and c, and c and a are rolled northward andwestward, respectively. (Other forms of datarolling may be useful in other data-parallel al-gorithms not considered in this paper. In ad-dition, element-wise operations such as matrixaddition/subtraction and register copying donot require data rolling. Combining all thesemultiply-add-roll forms in one generalized formis a matter of further investigations.)

The scalar unit of the matrix processor is afull-fledged core capable of executing the se-quential part of an application beside control-ling the matrix unit. The program instructionsare loaded into the instruction cache for execu-tion. The required data for the scalar executionis loaded into a software-controlled data cachenear to the scalar registers. To reduce data misspenalty, we applied software pre-fetching tech-niques 20) where pre-fetch or pre-load instruc-tions are inserted automatically by the com-piler or manually by the programmer to bringdata ahead of its use. The pre-load instructioncauses a matrix block to be brought from themain memory to the data cache. This pre-loadinstruction looks like a load instruction exceptno register is specified 20).

To preserve the integrity of data betweenthe scalar unit data cache and the main mem-ory, the altered blocks in the data cache mustbe written back (or post-stored) into the mainmemory before switching the computing to the


matrix unit. The cache replacement policy isassumed to be software-controlled. While us-ing software-controlled memories increases thecomplexity of programming, it has been demon-strated that this approach is effective in hidingmemory latency using smaller cache sizes thanconventional hardware-controlled caches 21).

The movement of data/results between themain memory and the data cache of the scalarunit and between the main memory and the ma-trix register file of the matrix unit is decoupledfrom processing by using a separate data access(or load/store) unit. Therefore, computing anddata load/store can be partially or fully over-lapped, which is required to hide the memorylatency. The main components of the proposedmatrix processor are shown in Fig. 1.

2.2 Timing ModelIn order to evaluate the effectiveness of the

proposed matrix processor especially the co-processor unit we need to predict the execu-tion times of the designed algorithms. We de-velop an execution time model for our proposedmatrix processor. This model is simple, deter-ministic, and predictable due to using software-controlled data cache.

We decompose the total execution time ofan algorithm into: 1) computing time, 2)load/store time, and 3) overhead time. The firstterm is the time spent in processing on both thematrix and scalar units. The second term is thetime elapsed in the load/store operations. Thisterm also includes the time of data pre-fetchingand post-storing which are needed at the begin-ning and the end of the algorithm, and beforeswitching computing between the matrix andscalar units. The third term of the executiontime represents the overhead of data alignmentneeded for the processing on the matrix unit.

The MMA operations executed on the ma-trix unit will have the following times. As-suming each multiply-add-roll time-step takesτ clock cycles, the time of a b×b MMA is bτcycles 16) (see also next section). Other matrix-formulated operations such as matrix skewingand transposition discussed in Subsection 3.2will be quantified as the matrix multiply-addoperation. The matrix unit operates on b×bblocks of matrices, and we assume the matri-ces are stored in memory in a b×b block datalayout 17),18). Moving a b×b block betweenmemory and the matrix registers or the scalardata cache takes tbls clock cycles. Since we as-sume a decoupled load/store unit, whenever a

load/store operation is overlapped with com-puting or data alignment, the maximum execu-tion time is taken.

In order to predict the execution time of thehigh-level language constructs used in our algo-rithms, we applied a prediction method sim-ilar to the one introduced in Ref. 22) as fol-lows. The predicted execution time of the as-signment statement variable=expression is thetime of calculating the expression plus the timeof loading its variables if they are not in reg-isters. For example, the statement y=z+1 hasthe execution time 2tls+tadd clock cycles, wheretls is the number of cycles to load/store a scalarvalue from cache to registers and tadd is the timeof an addition in clock cycles. However, if z andy are in registers and will be used for next com-putations the time will be tadd cycles.

The execution time of the if-then statementhas two possible values depending on its logicalcondition. Hence, a time interval [T1,T2] willrepresent the lower and upper bounds of execu-tion time. We will mean by adding an executiontime T0 to an interval [T1,T2] that T0 is addedto both bounds. The same convention appliesalso to multiplying a time by an interval. Forexample, the execution time of the statement ‘if(c) then s1’ is the interval [tcd+(tjp,ts1)], wheretcd is the time to check the condition c, tjp isthe number of cycles to jump around s1 to theend of the if statement, and ts1 is the numberof cycles to calculate the statement s1.

The execution time of the for-loop statementequals the total number of the loop iterationsmultiplied by the sum of the execution timesof the statements within the loop body and theloop overhead time, tloop, in clock cycles.

3. Level-3 BLAS

Level-3 BLAS are targeted to the basic oper-ations of O(n3) such as matrix-matrix multipli-cation, rank-k update and the solution of tri-angular systems of equations 23). The most im-portant of these operations is the general MMA(GEMM) C←αC+βop(A)·op(B) where op(X)is the matrix X or its transpose and α and βare scalars. In this section, we present our im-plementation of the GEMM operation on ourproposed matrix processor.

3.1 Matrix Multiply-add OperationConsider the MMA operation C←C+A·B

where A=[a(i, k)], B=[b(k, j)], and C=[c(i, j)]are b×b dense matrices. The goal is to find op-timal data distributions of the three matrices


Fig. 2 Three data allocations to perform the b×b MMA operationC←C+A·B on the b×b torus matrix unit, b=4.

to compute matrix C on the b×b torus matrixunit in the minimum number of steps.

We showed in prior works 15),16) that the in-dex space of the b×b MMA operation can berepresented as a 3D b×b×b torus where mod-ular scheduling is used to describe the distribu-tion of the active index points at each schedul-ing step. All optimal modular scheduling func-tions are determined 15). At each schedulingstep, b2 points are active from the total b3

points of the index space. To increase the ef-ficiency of computing, we used the linear pro-jection method to map the scheduled compu-tations at each step to the 2D processor spacewhere a b×b torus array processor (or matrixunit) is used to perform the matrix multiply-add operation in the minimal time, b time-steps of multiply-add-roll. For each optimal 3Dscheduling function, three 2D data distributionsare obtained by projection along i, j, and k axes(see Ref. 15), 16) for all possible projections).All the optimal 2D distributions are classifiedinto three categories. In one category matrixC elements remain stationary during process-ing while the other two matrices are rolled.This category results from projection along thek-axis and the well-known Cannon’s distribu-tion 24) belongs to this category. The other twocategories keep either A or B stationary (i.e.,projection along j or i-axis, respectively) whilerotating the other two matrices.

In this paper, we will use one optimal 2Ddata distribution from each category. The threeselected optimal data allocations are shownin Fig. 2. In Allocation 1, the rows of Aand the columns of B are initially skewed us-ing circular shifts. Then at each step, allPEs perform simultaneously the multiply-addc(i, j)=c(i, j)+a(i, k)·b(k, j) while the elementsa(i, k) and b(k, j) are rolled westward andnorthward, respectively, to neighbor PEs. Af-

ter b=4 multiply-add-roll time-steps, matrix Cis computed and data returns to the initial stateshown in the figure. In Allocation 2, after align-ing B and C by skewing, matrix A remainswhile matrices B and C are rolled northwardand westward, respectively, at each step of com-puting. In Allocation 3, matrices A and C areskewed as shown in the figure. During comput-ing, matrix B remains while matrices A andC are rolled westward and northward, respec-tively.

3.2 AlignmentGiven the three matrices A, B, and C in

the canonical (conventional) layout, some align-ment overhead is required before using the threeoptimal allocations in Fig. 2. In addition, ma-trices B and A need transposition in Alloca-tions 2 and 3, respectively. Our aim is tohide this overhead. We do so by formulat-ing both the skewing and transposition oper-ations as b×b matrix multiply-add operationsso that they can be overlapped with data load-ing/storing, as we will show in the algorithms.

The Skew Operation. To skew therows/columns of an b×b matrix horizon-tally/vertically, the ith row/column, i =0,1,. . . ,b-1, is rolled i times using circular shifts.We formulate the skewing operation as anMMA operation on the 2D torus matrix unitwhere C is initially set to zero and either A orB is set to some predefined 0-1 matrix register.For example, to skew the columns of the b×bmatrix B to the North, we perform the MMAoperation C← C+RN·B, where C is initially setto zero. During computing, the matrix registerRN, which has ones on the leftmost column andzeros elsewhere, is rolled westward while matrixC is rolled northward. After b=4 multiply-add-roll time-steps, C is replaced with the requiredresult. To skew the rows of matrix A to theWest, we perform C←C+A·RW, where C is ini-


tially set to zero and the matrix register RW,which has ones on the topmost row and zeroselsewhere, is rolled northward while matrix Cis rolled westward.

The two registers RN and RW can alsobe used to return, respectively, a northwardskewed matrix X and a westward skewed ma-trix Y to their canonical forms.

The Transpose Operation. To transposea b×b matrix X on the torus matrix unit,three MMA operations are performed in se-quence. First, matrix X is skewed to theNorth by performing C←C+RN·X, C is ini-tially zero, as explained above. Second, theMMA X←X+RI·C, X is initially zeroed, isperformed. During computing, the matrix reg-ister RI, which hold the identity matrix, is keptstationary while matrices X and C are rolledwestward and northward, respectively. Now,matrix X holds the westward skewed form ofmatrix XT . To get XT in canonical form, ma-trix X is skewed eastward by performing an ap-propriate MMA operation.

For aligning matrix B in Allocation 2, weneed to only apply the first two steps of theaforementioned transpose algorithm and thistakes 2b multiply-add-roll time-steps. A sim-ilar procedure can be done to align matrix Ain Allocation 3. For more details on the align-ment operations on the torus matrix unit referto Ref. 25).

3.3 GEMM Implementation on theMatrix Unit

Using the alignment operations discussedabove, the four variants of the GEMM opera-tion, C ←C+A·B, C←C+AT ·B, C←C+A·BT ,and C←C+AT ·BT , can be performed on thetorus matrix unit. In Ref. 16) we described thefour variants for different initial data layouts ofA and B inside the torus matrix unit. However,the sizes of matrices were the same size as thematrix unit, b.

In this paper, we design a blocked fine-grained algorithm to implement the main vari-ant C←C+A·B, where A is an n1×n3 matrix,B is an n3×n2 matrix, and n1, n2, n3�b, onthe matrix unit. The other three variants canbe treated in a similar way.

Assume that the three matrices are parti-tioned and stored by square blocks as the ma-trix unit size, b, and that d blocks from A, B orC can be loaded into the matrix register file andreused many times. Then, we have two differentways to calculate the blocks of C on the matrix

unit. One way is to update d blocks from C us-ing d blocks from A (or B) and one block fromB (or A). Updating the C blocks in this wayis called blocked saxpy MMA 11). The secondway is to compute one block of C using a dot-product of one block row of A and one block col-umn of B. This is called a blocked dot-productMMA 11). Choosing which way to implementdepends mainly on the sizes of involved matri-ces. We will describe our implementation of afine-grained blocked saxpy algorithm when dis-cussing the LU factorization. Here we show ourimplementation of a fine-grained blocked dot-product algorithm on the matrix unit assumingfor simplicity that one block row of A or oneblock column of B can be loaded into the ma-trix register file and reused. This assumptionwill be changed when discussing the saxpy ver-sion in Subsection 4.3.

Assume the b×b blocks are stored in thecolumn-major order. Initially, the matrixblocks are stored into the main memory. Be-fore computing on the matrix unit, the neededblocks should be loaded into matrix registersby the load/store unit. Results after comput-ing must be stored back to memory before itcan be used on the scalar unit. We includethese initial and final data movement opera-tions in our timing model. Algorithm 1 belowdescribes our implementation of a jik-version ofthe blocked dot-product of the GEMM kernelC ←C+A·B on the matrix unit. We appliedAllocation 2, where K1=�n1/b�, K2=�n2/b�,and K3=�n3/b�.

Since the matrix unit and the load/store unitoverlap execution, we applied a loop splittingtechnique called loop peeling 26) to correctly in-sert the load/store instructions within the algo-rithm. In this technique, a loop is divided intothree sections: a prologue with the first itera-tion (and may be the second iteration too), amain body, and an epilogue with the last iter-ation. The statements, which are marked withthe symbols ‘�’ or ‘∗’ at the beginning of the al-gorithm lines, will overlap execution as in lines4 and 5, and lines 7 and 8 of Algorithm 1.Algorithm 1.01. for j=1 to K2

02. Load B1,j {tbls}03. for k=2 to K3

{(K3-1)· (tloop+max(tbls,bτ)+bτ)}04.� Load Bk,j {tbls}05.� Transpose & Skew Bk−1,j northward {2bτ}06. end for07.∗ Transpose & Skew BK3−1,j northward {2bτ}


% Prologue08.∗ Load C1,j {tbls}09.� Load A1,1 {tbls}10.� Skew C1,j westward {bτ}11. for k=1 to K3 − 1

{(K3-1)· (tloop+max(tbls,bτ)}12.� Compute C1,j+=A1,k·Bk,j {bτ}13.� Load A1,k+1 {tbls}14. end for15.� Compute C1,j+=A1,K3 ·BK3,j {bτ}16.� Load C2,j {tbls}

% Main Body17. for i=2 to K1-1

{(K1-2)(tloop+2max(tbls,bτ)+time of line 11)}18.� Store Ci−1,j {tbls}19.� Load Ai,1 {tbls}20.� Skew Ci,j westward {bτ}21. repeat lines 11-14 where X1,.=Xi,.

22.� Compute Ci,j+=Ai,K3 ·BK3,j {bτ}23.� Load Ci+1,j {tbls}24. end for

% Epilogue25.� Store CK1−1,j {tbls}26.� Load AK1,1 {tbls}27.� Skew CK1,j westward {bτ}28. repeat lines 11-14 where X1,.=XK1,.

{time of line 11}29.� Compute CK1,j+=AK1,K1 ·BK1,j {bτ}30.� Store CK1,j {tbls}31. end for

Note that, w±=1 means w=w±1. In lines 5and 7, transposing the given block and skewingit northward requires performing two MMA op-erations as discussed before where, one of themwill be overlapped with the loading at lines 4and 8. As we see in Algorithm 1 expected ex-ecution times of each statement are calculated.Also, we can see that the alignment overhead isalmost hidden with the load/store operations.In Section 5, we will evaluate the performanceof the GEMM operation with different param-eter values .

4. LU Factorization

In solving the linear system of equationsAx=b where A is a real dense n×n matrix andboth x and b are real vectors of length n, firstthe factorization of matrix A into a unit lowertriangular matrix L and an upper triangularmatrix U is done. Then, the two triangularsystems Ly=b and Ux=y are solved to find x.

In this paper, we consider the Gaussian Elim-ination with Partial Pivoting (GEPP) to de-compose matrix A into PA=LU where P is ann×n rows permutation matrix. Using partial

pivoting guarantees the stability of the solutionof the system but complicates the implementa-tion, see Ref. 27) for more details. Algorithm 2is the standard GEPP algorithm assuming A isnon-singular 11),27). We used MATLAB colonnotation to describe ranges of array indices. Inthis notation an entire for-loop is written incompact form as in line 6, while line 7 expressestwo nested for-loops. After the completion ofAlgorithm 2, the lower triangular part of A willbe overwritten by matrix L with unit diagonals(which are not stored in the diagonals of A),and the upper triangular part will be overwrit-ten by matrix U .Algorithm 2.01. for k=1 to n-102. find and record γ, |a(γ, k)|=maxk≤j≤n |a(j, k)|03. if γ>k04. swap a(k, 1 : n), a(γ, 1 : n)05. end if06. a(k + 1 : n, k)=a(k + 1 : n, k)/a(k, k)07. a(k + 1 : n, k + 1 : n)-=

a(k + 1 : n, k)·a(k, k + 1 : n)08. end for

The amount of work (floating-point opera-tions, FLOPs) in the standard GEPP algorithmis 2n3/3+O(n2)≈2n3/3 11) where one FLOPmeans a scalar floating-point addition, subtrac-tion, multiplication and/or division operation.The O(n2) term includes the cost of pivot-ing which can be neglected compared with theO(n3) arithmetic cost of the algorithm. How-ever, the pivoting cost may not be negligiblewhen the data are not sequentially stored inmemory.

4.1 Blocked LU FactorizationIn Algorithm 2, line 6 represents a vector-

scaling operation (i.e., Level-1 BLAS) whileline 7 is a rank-one update of the trailing sub-matrix (Level-2 BLAS). In order to increase theamount of FLOPs per memory reference, i.e.,the degree of data reuse, matrix-matrix multi-plication (Level-3 BLAS) should be used.

There are three variants of the blocked LUfactorization known as the i,j,k variants rela-tive to the outermost loop index 10),28). Thek variant or the so-called right-looking algo-rithm 10) is generally described as follows. Then×n matrix A is segmented into four blockswhere A11 is an nb×nb block (nb is the block-ing size) and A22 is an (n−nb)×(n−nb) block.First, the unblocked GEPP algorithm is ap-plied to the current block column of A. Thismeans we compute P1·A1:2,1=L1:2,1·U11 where


L11 and U11 are nb×nb lower and upper trian-gle matrices, respectively, and P1 is a permuta-tion matrix representing the effects of pivoting.Second, the permutation P1 is also applied tothe right (and to the left when factorizing thenext block columns) of the current block col-umn of A to produce A1:2,2. Third, the trianglesystem with multiple right-hand-sides (RHS)L11·U12=A12 is solved for U12. In practice, thesub-matrices of L and U are stored on the cor-responding blocks of A to save space and there-fore, A12 ← U12 and A21←L21. Fourth, ma-trix A22 is updated using the matrix multiply-subtract operation A22←A22-A21×A12. Next,A22 is segmented into blocks and the abovesteps are repeated to factorize all matrix A.

We will implement the above k variant al-gorithm assuming that matrix A is stored asb×b blocks into memory. We selected this vari-ant since it can be scheduled to run on boththe matrix and scalar units provided that theydon’t overlap execution. Suppose that a partialfactorization of A has been obtained so thatthe first �-1 block columns of L and the first �-1block rows of U have been evaluated and storedinto A (see Fig. 3). Algorithm 3 describes theblocked GEPP algorithm assuming for simplic-ity that n=b·m.Algorithm 3.1. for �=1 to m-12. Factor P�·A�:m,�=L�:m,�·U�,�

3. Pivot A�:m,�+1:m=P�·A�:m,�+1:m

4. Pivot A�:m,1:�−1=P�·A�:m,1:�−1, % � > 15. Solve L�,�·A�,�+1:m=A�,�+1:m

6. Update A�+1:m,�+1:m-=A�+1:m,�·A�,�+1:m

7. end for8. Factor Pm·Am,m=Lm,m·Um,m

Fig. 3 Matrix A after �-1 iterations of Algorithm 3;the dark grey area is already factorized. TheFactor primitive is applied to the white dottedarea, the Solve primitive is applied to the lightgrey area, the Update primitive is applied tothe white area.

9. Pivot Am,1:m−1=Pm·Am,1:m−1

In line 2 the unblocked factorization is ap-plied to a rectangular matrix of size (n-(�-1)·b)×b. In this case, Algorithm 2 should beadjusted so that the last value of the columnsindex k is b-1, and the last value of the rowsindex will be n as it is. Then, after k=b-1, onemore iteration at k=b is applied without theupdating in line 7 of Algorithm 2.

In solving the m-� lower triangle systems eachwith b RHS in line 5, we changed the columnversion of the forward substitution algorithmgiven in Ref. 11) so that it can handle b multi-ple RHS. Algorithm 4 describes the Solve prim-itive applied to block A�,k at Fig. 3. Note thatL�,� is the lower triangular part of A�,� but withdiagonal elements equals to one.Algorithm 4.1. fori=1 to b-12. a(i, :)=a(i, :)/l(i, i)3. a(i + 1 : b, :)-=l(i + 1 : b, i)·a(i, :)4. end for5. a(b, :)=a(b, :)/l(b, b)

In general Algorithm 4 can be applied to anylower triangle system with multiple RHS. How-ever, when the triangular matrix has unit values(as in Algorithm 3), we can omit lines 2 and 5in Algorithm 3 since division by the constantvalue ‘one’ gives the same result. We will de-scribe our implementation of all primitives ofAlgorithm 3 in Subsection 4.3.

4.2 Level-3 BLAS FractionIn this subsection, we will calculate the cost

of Algorithm 3 in terms of the arithmetic oper-ations. Then we will analyze the impact of thepercentage of the matrix-matrix multiplication(i.e., the Update primitive) on the performanceof the blocked Gaussian elimination algorithm.For simplicity, we will not consider the overheadof pivoting, however, in Subsection 4.3 we willpredict its execution time.

Table 1 demonstrates the amount of arith-metic operations in the primitives of Algo-rithm 3. In line 2 of Algorithm 3, the Fac-tor primitive is applied to a rectangular sub-matrix whereas in line 8 the last square diag-

Table 1 The arithmetic cost of Algorithm 3.

primitive #FMA #mult. #div.

Factor b−14

n2− 3b2+3b−212

n 12n2+ 4b2−3

6n n

Solve b−14

n2 − b2−b4

n - -

Update 13n3 − b

2n2 + b2

6n - -


onal block of matrix A is factorized applyingAlgorithm 2 directly. As we see, the overallFLOPs count (2×#FMA+#mult.+#div.) ofthe Factor primitive on Algorithm 2 is O(n2)because the operations can be implemented us-ing Level-2 and Level-1 BLAS. The primitiveSolve in line 4 solves a b×b triangle systemwith (m− �)·b multiple RHS. The overall com-plexity of this primitive is O(n2) arithmetic op-erations because it can be implemented usingLevel-2 and Level-1 BLAS (see Algorithm 4).The third primitive Update is the GEMM op-eration C←C-A·B. The amount of FLOPs con-tributed by this primitive is O(n3). From Ta-ble 1 we can see that the Update primitive isthe most intensive computational part. There-fore, it is expected that any enhancement to thisprimitive will enhance the total performance ofthe LU decomposition algorithm.

Table 1 also shows the amount of the fusedmultiply-add/subtract (FMA) included in eachprimitive. We can see that the Solve and Up-date primitives are entirely composed of FMAswhere each FMA equals two FLOPs. Morethan half the FLOPs of the Factor primitiveare FMA and this percentage increases as b in-creases. This large percentage of FMA on Al-gorithm 3 suggests that using an FMA instruc-tion will have a significant advantage over usinga floating-point multiply followed by a floating-point add. This is because using separate mul-tiply and add instructions costs 1.7 times morethan an FMA 29).

Now, we perform a simplified timing anal-ysis of Algorithm 3 disregarding the pivotingprocess since it has less work than the updat-ing of the trailing sub-matrix. We also assumethat an FMA operation (c=c+a×b) takes μ cy-cles which will also be the same time for mul-tiplication (c=1+a×b), addition/subtraction(c=a±b×1) operations. Since multiplication isfaster than division, the number of divisions inline 6 of Algorithm 2 can be reduced by com-puting the reciprocal of the pivot element sep-arately and then multiplying the result by theelements of matrix A at line 6.

The time of executing Algorithm 3 on thematrix processor is the total time of the threeprimitives. The time to execute a b×b matrixmultiply-add operation on the b×b torus ma-trix unit is b·τ clock cycles (without the over-head of data alignment). Therefore, the ex-pected execution time of the Update primitiveon the matrix unit is approximately:

(13n3 − b

2n2 + b2

6 n)· 1

b2 τ. (1)

The approximate execution time of Algo-rithm 3 (ignoring pivoting) on the matrix pro-cessor is:(

b2n2 + b2−2

6 n)· μ + n · λ +

(13n3 − b

2n2 + b2

6 n)· 1

b2 τ. (2)

Now, the speed of computing Algorithm 3 onthe matrix processor measured in the amountof FLOPs/cycle can be calculated as the ra-tio of the amount of FLOPs in the sequentialalgorithm, 2n3/3, and the parallel time of Al-gorithm 3 given in Eq. (2). The speed dependsmainly on τ and if operations of the matrix unitcan be scheduled so that we get one multiply-add-roll each clock cycle, then the maximumspeed of computing asymptotically becomes thespeed of matrix-matrix multiplication (or theUpdate primitive), i.e., 2b2 FLOPs/cycle.

In the next section we will implement the fourprimitives of Algorithm 3 on the scalar and ma-trix units in order to predict the overall execu-tion time of Algorithm 3 and hence evaluate themerits of adding the matrix unit as an acceler-ator for matrix kernels.

4.3 Primitives ImplementationIn this subsection we design our algorithms

to implement the four primitives, Factor, Pivot,Solve and Update, of the blocked LU factoriza-tion algorithm (Algorithm 3) on the proposedmatrix processor. The idea behind our designis that the matrices are stored as b×b blocksrather than row- or column-major formats, andany block loaded into registers and data cachemust be maximally reused before returning tomemory. We scheduled each iteration of Algo-rithm 3’s main loop so that the three primi-tives Factor, Pivot, and Solve are executed onthe scalar core and then switching to the ma-trix unit to execute the Update primitive. Notethat, before switching computing, partial re-sults should return to memory.

To compute on the scalar core, the requiredblocks are moved by the load/store unit frommemory to the data cache one b×b block ata time. Then, during processing, the scalardata is loaded into scalar registers and storedback to the data cache. We assume the datacache is controlled by software so that a pre-load operation causes the load/store unit topre-fetch one block from main memory to thedata cache. Pre-fetching data ahead of its use


reduces the stall time which happens when datais not in cache. The pre-fetch operations arehand-inserted into the algorithms in order topre-fetch the data before it is actually needed.In addition, we applied a post-store operationto control the replacement of blocks in the datacache. Note that, if some of the statementswhich are overlapped with the pre-load andpost-store instructions should be serialized, weput the symbol ‘◦’ at the beginning of their cor-responding lines as in lines 2 and 3 of Algorithm5. (The global constant c is the reciprocal of b.)

The Factor Primitive. Algorithm 5 belowis the implementation of Algorithm 2 (GEPP)to factorize the �th block column of matrix A(dotted area in Fig. 3) on the scalar core. InAlgorithm 5, the processed block column is di-vided into two parts. The first one is the blockA�,� and the second is the rest of the blocks. Weapply the pre-fetching technique to the secondpart to correctly insert the pre-load and post-store instructions. Each iteration of the mainloop at line 4 takes two passes on the blocks.The first pass finds the pivot row in the blockcolumn while the second computes the multi-pliers (elements of matrix L) and stores themin the current column. Then the trailing sub-matrix of the �th block column of A is updated.Note that altered blocks are written back tomemory at the end of Algorithm 5.Algorithm 5.01.� Pre-load A�,� {tbls}02.�◦ s=b·(� − 1)+1 {2 μ}03.�◦e=b·� {μ}04. for j=s to e-1

% Find the pivot element in column j05. amax = abs(a(j, j)) {tls+tabs}06. pivot(j)=0 {tmv}

% Prologue07.� Pre-load A�+1,� {tbls}08.� for kk=j+1 to e

{(e-j)·(tls+tabs+tcd+tloop+(tjp, 2tmv))}09. t=abs(a(kk, j)) {tls+tabs}10. if t > amax {tcd+(tjp, 2tmv)}11. amax=t {tmv}12. pivot(j)=j {tmv}13. end if14. end for15.∗ Pre-load A�+2,� {tbls}16.∗ for kk=b·� + 1 to b·(�+1)

{b·(tls+tabs+tcd+tloop+(tjp, 2tmv))}17. repeat lines 9-1318. end for

% Main Body19. for i=�+2 to m-1

{(m-�-2)·(tloop+max(tbls, time of line 16))}

20.� Post-store Ai−1,� {tbls}21.� Pre-load Ai+1,� {tbls}22.� for kk=b·(i-1)+1 to b·i {time of line 16}23. repeat lines 9-1324. end for25. end for

% Epilogue26.∗ Post-store Am−1,� {tbls}27.∗ Pre-load Apivot(j)·c+1,� {tbls}28.∗ for kk=b·(m-1)+1 to b·m {time of line 16}29. repeat lines 9-1330. end for

% Swap the jth row and the pivot row31.� Post-store Am,� {tbls}32.� if pivot(j) >0

{tcd+(tjp, b·(tloop+4tls+3tmv))}33. for k=s to e {b·(tloop+4tls+3tmv)}34. t=a(j, k) {tls + tmv}35. a(j, k)=a(pivot(j), k) {2tls+tmv}36. a(pivot(j), k)=t {tmv+tls}37. end for38. end if

% Compute the multipliers and update% the trailing sub-matrix

39.∗ Post-store Apivot(j)·c+1,� {tbls}40.∗ Pre-load A�+1,� {tbls}41.∗◦ piv=1/a(j, j) {tls+λ}42.∗◦ for ii=j+1 to e {(e-j)2(tloop+3tls+μ)+

(e-j)(tloop+2tls+μ+tmv)}43. a(ii, j)=piv·a(ii, j) {tls+μ}44. t=a(ii, j) {tmv+tls}45. for kk=j+1 to e {(e-j)(tloop+3tls+μ)}46. a(ii, kk)-=t·a(j, kk) {3tls+μ}47. end for48. end for49.� Pre-load A�+2,� {tbls}50.� repeat lines 42-48 where ii=b·�+1 to b·(�+1)

{b(e-j)(tloop+3tls+μ)+b(tloop+2tls+μ+tmv)}51. for i = �+2 to m-1

{(m-�-2)·(tloop+max(tbls, time of line 50))}52.� Post-store Ai−1,� {tbls}53.� Pre-load Ai+1,� {tbls}54.� repeat lines 42-48 where ii=b·(i-1)+1 to b·i

{time of line 50}55. end for56.� Post-store Am−1,� {tbls}57.� repeat lines 42-48 where ii=b·(m-1)+1 to b·m

{time of line 50}58. Post-store Am,� {tbls}59. end for60. repeat lines 4-58 where j=e,

except the for-loops starting with j+161. Post-store A�,� {tbls}

The Pivot Primitive. After factoring thecurrent block column of A, the pivoting effectshould be applied to all the left and right col-


umn blocks, too. Algorithm 6 below describesthe pivoting of a number of adjacent blockcolumns of matrix A on the scalar unit. Notethat, the execution time of line 10 is the sameas the lines 33–37 of Algorithm 5.Algorithm 6.01. for k=k1 to k202.� Pre-load A�,k {tbls}03.�◦ s1=b·(k-1)+1 {2μ}04.�◦ e1=b·k {μ}05.�◦ piv=pivot(s1)·c+1 {μ}06.∗ Pre-load Apiv,k {tbls}07.∗ piv1=pivot(s1+1)·c+1 {2 μ}08.∗ Pre-load Apiv1,k {tbls}09.� if pivot(s1) > 0

{tcd+(tjp, b · (tloop+4tls+3tmv))}10. swap rows s1 and pivot(s1)

{b·(tloop+4tls+3tmv)}11. end if12. for i=s1+1 to e1-1

{(b-1)·(tloop+4 μ+(tcd+(tjp,tbls))+time of line 9)}

13. piv1=pivot(i − 1)·c+1 {2 μ}14. piv2=pivot(i + 1)·c+1 {2 μ}15. if piv1 <> piv216.� Post-store Apiv1,k {tbls}17.� Pre-load Apiv2,k {tbls}18. end if19. repeat lines 9-11 where s1 = i

{time of line 9}20. end for19. piv=pivot(e1 − 1)·c+1 {2 μ}20.� Post-store Apiv,k {tbls}21.� repeat lines 9-11 where s1=e1

{time of line 9}22. Post-store Apiv2,k {tbls}23. Post-store A�,k {tbls}24. end for

The Solve Primitive. In Algorithm 4, wehave described the solution of a b×b triangularsystem with b multiple RHS. However, in line5 of Algorithm 3, we need to solve the samesystem but with b×(m− �)·b multiple RHS. InAlgorithm 7 below, we describe how the lattertriangular system is implemented on the scalarunit of the matrix processor. See Fig. 3 for theworkspace of the algorithm.Algorithm 7.01.� Pre-load A�,� {tbls}02.�◦ s1=s+b {μ}03.�◦ e1=e+b {μ}04. Pre-load A�,�+1 {tbls}05. for i=s to e-1 {∑e−1

i=s(tloop+time of line 6)}

06. for ii=i+1 to e{(e-i)·(tloop+tmv+tls+b·(tloop+3tls+μ))}

07. t=a(ii, i) {tmv+tls}08. for j=s1 to e1 {b·(tloop+3tls+μ)}09. a(ii, j)-=t·a(i, j) {3tls+μ}10. end for11. end for12. end for13. Pre-load A�,�+2 {tbls}14. for k=�+2 to m-1 {(m-�-2)·(tloop+2·μ+

max(tbls, time of line 5))}15. s1=s1+b {μ}16. e1=e1+b {μ}17.� Post-store A�,k−1 {tbls}18.� Pre-load A�,k+1 {tbls}19.� repeat lines 5-12 {time of line 5}20. end for21. s1=s1+b {μ}22. e1=e1+b {μ}23.� Post-store A�,m−1 {tbls}24.� repeat lines 5-12 {time of line 5}25. Post-store A�,m {tbls}

The Update Primitive. This primitive willbe executed on the matrix unit. Algorithm 8below is our implementation of a fine-grainedblocked saxpy version of Algorithm 1. We usedAllocation 1 to find the blocks of C. To re-duce the loading time of blocks from main mem-ory to the matrix registers, we assume at leastd blocks from A are loaded and reused manytimes. In this case, the updating of the trailingsub-matrix at each iteration of � may be dividedinto two parts if the number of its block rows isnot an exact multiple of d. We apply Algorithm8 to update the main part. To update the re-maining part (where the number of block rowsare not multiple of d), we also apply Algorithm8 but the loop index i will start from �+((m-�)/d)·d+1 to m and step equals to ‘one’ whilei+d-1 is replaced with m. It should be notedthat when � approaches m, m-� becomes lessthan d. In this case, d must be reset to thevalue m-� to correctly apply Algorithm 8.Algorithm 8.01. for i=�+1 to �+�(m-�)/d�·d step d02. Load Ai,� {tbls}03. for j=i+1 to i+d-104.� Skew Aj−1,� westward {bτ}05.� Load Aj,� {tbls}06. end for07.� Skew Ai+d−1,� westward {bτ}08.� Load A�,�+1 {tbls}09. for k=�+1 to m-1

% Prologue10.� Skew A�,k northward {bτ}11.� Load Ai,k {tbls}12.∗ Ai,k-=Ai,�·A�,k {bτ}


13.∗ Load Ai+1,k {tbls}% Main Body

14. for j=i+1 to i+d-215.� Aj,k-=Aj,�·A�,k {bτ}16.� Load Aj+1,k {tbls}17.� Store Aj−1,k {tbls}18. end for

% Epilogue19.� Ai+d−1,k-=Ai+d−1,�·A�,k {bτ}20.� Store Ai+d−2,k {tbls}21.∗ Store Ai+d−1,k {tbls}22.∗ Load A�,�+1 {tbls}23. end for24. repeat lines 10-21 where k=m25. end for

Modifications. We should mention that be-cause of using software pre-fetching techniquesin designing the previous algorithms, some mi-nor modifications should be done to the algo-rithms for proper working at iterations m-1 andm of Algorithm 3. This is due to the limitednumber of blocks which doesn’t allow applyingthe pre-fetching. We mention here the changesto the algorithms.

To apply Algorithm 5 at iteration �=m-1, line7 can be moved outside the main loop. Lines15, 19–31, 39–40, 49, and 51–58 shouldn’t beapplied. At the end of the algorithm the blockA�+1,� should be post-stored to memory. Atiteration �=m of Algorithm 5, i.e., the factoringof block Am,m, the algorithm is applied withoutthe lines 7, 15–31, 39–40, and 49–60.

In algorithm 6, at each iteration k, the twoblocks A�,� and A�,�+1 are pre-loaded. Then,the middle loop at line 12 runs from s1 to e1for rows interchange. After that, the two blocksare post-stored. A similar change is done when�=m, but the middle loop runs from s1 to e1-1.

At iteration �=m-1 of Algorithm 3, only onetriangular system with b RHS is solved. In thiscase, Algorithm 7 is applied without the lines13–24. Also, in Algorithm 8, only the block up-date Am,m-=Am,m−1·Am−1,m is needed whichrequires three block loads and one store. Theskewing of Am,m−1 and Am−1,m is overlappedwith two of the three block loads.

5. Performance Evaluation

In Section 2, we have developed a predictableexecution time model for the proposed matrixprocessor. In this model, we have decomposedthe total execution time of an algorithm intothe sum of: computing time, load/store time,

and overhead time. In Sections 3 and 4, wehave predicted the execution times of the state-ments of the designed algorithms so that thetotal execution times can be easily calculated.In this section, we analyze the performanceof the GEMM operation and the blocked LUfactorization with partial pivoting on the pro-posed matrix processor. Also, we analyse theeffect of changing the memory bandwidth onthe GEMM operation, which is the basic oper-ation in the LU factorization algorithm.

We measure the performance of Level-3BLAS and the blocked LU factorization on theproposed matrix processor by the speed of com-puting in FLOPs/cycle. For the parameters re-lated to architectural implementation, we as-sume the time of the operations tls, tmv, tjp, tcd,and tabs is one clock cycle. Since most compu-tations of the algorithms are FMA, we assumethe floating-point functional units are fully-pipelined and the compiler is able to schedulethe independent operations using loop unrollingand software pipelining techniques. Hence, thethroughput of the floating-point FMA, μ, andthe multiply-add-roll, τ , time-step is assumedone FMA (or two FLOPs) each cycle and onemultiply-add-roll operation each cycle, respec-tively. However, this is not the case with thefloating-point division since its amount in thealgorithms is small and we assume its execu-tion time, λ, is twenty clock cycles.

Level-3 BLAS (GEMM). We discuss thecase that all matrices A, B, and C are n × n.This means K1=K2=K3=�n/b� in Algorithm1. For other cases, we can select between ap-plying Allocations 2 and 3 in Algorithm 1 orapplying Allocation 1 in Algorithm 8.

An important parameter for the efficiencyof the matrix processor is the time of load-ing/storing a b×b matrix block, tbls. Assum-ing the load/store unit can access ω contiguouselements in one clock cycle, then tbls is equalto b2/ω clock cycles. Figure 4 (a) shows theperformance of the GEMM operation for dif-ferent matrix sizes n and different block sizesb, where ω=b and the number of blocks thatcan be loaded and reused, d, is equal to m. Itis clear from the figure that the speed of com-puting increases as b increases. Meantime, as nincreases the maximum speed 2b2 FLOPs/cycleis approached. This means that the alignmentoverhead associated with the b×b MMA oper-ations is almost hidden. The main reason forthat is the overlap between the skewing oper-


Fig. 4 Performance of the GEMM operation C←C+A·B on the b×b matrix unit.

ations and the loading of data. However, de-creasing d will degrade the performance rela-tively as we expect.

Figure 4 (b) shows the effect of changing ω onthe performance of the n×n GEMM operationon the torus matrix unit, b=8. We see fromthe figure that the speed approaches the max-imum value when ω≥b. Otherwise, the speeddecreases till 2b FLOPs/cycle when the blocksare sequentially loaded/stored, i.e., ω=1.

LU Factorization. In the primitives suchas Factor and Pivot, there are two bounds ofthe execution time depending on the values ofmatrix A. The lower execution time will bethe case when no swapping of the matrix rowsoccurs. This means all lower bound execu-tion time of all if-then statements are taken.This lower bound of execution time is the up-per bound of the speed of computing. Also,the upper bound of the execution time (whenall comparisons to find the pivot hold) is thelower bound of the speed of computing. Fig-ures 5 (b,d) and 6 (b,d) show the lower boundsof speed while (a,c) in both figures show theupper bounds. We see in Figs. 5 and 6 that forrelatively large size matrix, n=32 K, the differ-ence between the speed bounds is small. How-ever, a noticeable difference exists for small sizematrices, for example when n=1K which is dueto the pivoting overhead.

Both Figs. 5 and 6 show the increase of thespeed of computing by increasing the matrixsize, n, for different values of b. This is dueto the increase in matrix multiplication por-tion as n increases. For the selected values of n(≤32K), the most appropriate values of b seemto be b=4 and b=8. b=4 is better in terms ofthe percentage of the maximum speed attained,

i.e., the efficiency of the matrix processor. How-ever, b=8 can be chosen if we favor speed withmoderate efficiency. At small values of n, wesee that the speed and efficiency are not highbecause the time of the Factor and Solve prim-itives became apparent. In addition, our modelassumes a scalar unit with two FLOPs/cycleand no overlap between computing and regis-ters load/store. Since the two primitives aremainly Level-1 and Level-2 BLAS, we expectthat using a scalar unit with a SIMD extensionwill enhance the performance of the blocked LUfactorization at small matrix sizes.

In Figs. 5 and 6, the effect of the loop over-head, tloop, is shown. In Fig. 5, d is fixed at thebest case d=m while in Fig. 6 it is fixed on theleast number we assumed, d=8. Both figuresshow that a large percentage of speed is gainedby decreasing tloop from 15 cycles going down tozero cycles, the case of complete loop unrolling.In addition, the two figures show that increas-ing the value of d enhances the speed due toincreasing the amount of data reuse.

6. Conclusions

Augmenting scalar cores with specialized co-processors for enhancing compute-intensive ker-nels have gained the attention of chip manufac-turers due to approaching the physical limits ofincreasing clock frequency. We have proposeda matrix processor model consisting of a scalarunit and a SIMD matrix unit of b×b simplecores tightly connected in a 2D torus topologyto process b×b MMA operations. The matrixunit is scalable since there are no global con-nections between PEs. An execution model forthe proposed matrix processor is developed. Wepresented three optimal data distributions to


Fig. 5 Performance of the blocked LU factorization with different matrix sizesand block sizes where d=m. The numbers on top of bars at n=32Kare the percentages of the maximum speed attained.

execute the MMA operation in the minimumtime. We formulated the alignment operationsincluding matrix transpose as MMA operationsand could overlap them with the load/store op-erations to hide their overhead.

We have designed a fine-grained blocked algo-rithm for the GEMM operation (Level-3 BLAS)as an application or kernel that is run com-pletely on the matrix unit. Our analytical re-sults showed that the maximum speed of com-puting is attained. Increasing the matrix unitsize, b, increases the speed of computing. Wehave also designed a blocked right-looking al-gorithm for the LU factorization with partialpivoting, which represents an example of an ap-plication that needs to run on both scalar andmatrix units. Only the matrix updating prim-itive of the algorithm is scheduled to executeon the matrix unit while the other primitivesrun on the scalar unit. We didn’t consider ex-ecution overlap between the matrix and scalarunits.

Since the percentage of the MMA increases as

the matrix size increases, around 50–90% of themaximum computing speed is attained by vary-ing the model parameters. Getting high perfor-mance requires increasing the amount of datareuse, i.e., the number of blocks, d, that canbe loaded into registers and reused before re-placement. Moreover, decreasing the loop over-head (using loop unrolling techniques) greatlyenhances the performance.

In our algorithms, we have used softwarepre-fetching techniques and software-controlleddata cache replacement policy in order to hidememory latency by exploiting the decouplingbetween the load/store unit and the process-ing (scalar and matrix) units. Although thismay complicate programming and increasesprogram sizes, it may be useful in hiding mem-ory latency and using smaller cache sizes (sinceonly three b×b blocks may exist at the sametime inside the data cache as shown in the al-gorithms).

In our future works, we will consider imple-menting and evaluating other factorization al-


Fig. 6 Performance of the blocked LU factorization with d=8.

gorithms on the proposed matrix processor in-cluding Cholesky, QR, and singular value de-compositions.

References

1) Williams, S., et al.: The potential of thecell processor for scientific computing, CF ’06:Proc. 3rd conference on Computing frontiers,New York, NY, USA, pp.9–20, ACM Press(2006).

2) Gustafson, J.L. and Greer, B.S.: A hard-ware accelerator for the Intel Math Kernel,White paper, ClearSpeed Technology Inc., athttp://www.clearspeed.com (2006).

3) The Berkeley Intelligent RAM (IRAM)Project: http://iram.cs.berkeley.edu.

4) Lawson, C.L., Hanson, R.J., Kincaid, R.J. andKrogh, F.T.: Basic linear algebra subprogramsfor FORTRAN usage, ACM Trans. Math. Soft-ware, Vol.5, pp.308–323 (1979).

5) Dongarra, J.J., Croz, J.D., Hammarling, S.and Hanson, R.J.: An extended set of FOR-TRAN basic linear algebra subprograms, ACMTrans. Math. Software, Vol.14, pp.1–17 (1988).

6) Dongarra, J.J., Croz, J.D., Duff, I. and

Hammarling, S.: A set of Level 3 Basic Lin-ear Algebra Subprograms, ACM Trans. Math.Software, Vol.16, pp.1–17 (1990).

7) Kagstrom, B., Ling, P. and Loan, C.V.:High Performance GEMM-based level 3 BLAS:Sample Routines for Double Precision RealData, High Performance Computing II, North-Holland, pp.269–281 (1991).

8) Kagstrom, B., Ling, P. and Loan, C.V.:GEMM-based level 3 BLAS: high-performancemodel implementations and performance eval-uation benchmark, ACM Trans. Math. Soft-ware, Vol.24, No.3, pp.268–302 (1998).

9) Gustavson, F.G., Henriksson, A., Jonsson,I., Kagstrom, B. and Ling, P.: SuperscalarGEMM-based Level 3 BLAS — The On-going Evolution of a Portable and High-Performance Library, PARA ’98: Proc. 4thInternational Workshop on Applied ParallelComputing, Large Scale Scientific and In-dustrial Problems, London, UK, pp.207–215,Springer-Verlag (1998).

10) Dongarra, J.J., Duff, I.S., Sorensen, D.C., andvan der Vorst, H.A.: Numerical Linear Alge-bra for High-Performance Computers, SIAM,


Philadelphia, Pennsylvania (1998).11) Golub, G.H. and Loan, C.F.V.: Matrix Com-

putations, John Hopkins, Baltimore, Maryland(1996).

12) Gallivan, K.A., Plemmons, R.J. and Sameh,A.H.: Parallel algorithms for dense linear al-gebra computations, SIAM Rev., Vol.32, No.1,pp.54–135 (1990).

13) Kung, S.: VLSI Array Processors, PrenticeHall (1988).

14) Dally, W. and B.Towles: Principles and Prac-tices of Interconnection Networkks, Elsevier(2004).

15) Zekri, A.S. and Sedukhin, S.G.: Computa-tionally Efficient Parallel Matrix-Matrix Mul-tiplication on the Torus, the 6th Int. Symp.on High Performance Computing, Nara City,Japan, (being published as LNCS4759) (2005).

16) Zekri, A.S. and Sedukhin, S.G.: The Gen-eral Matrix Multiply-Add Operation on 2DTorus, the 20th IEEE IPDPS Symp., PDSEC-06 Workshop, Rhodes Island, Greece, IEEEComputer Society (2006).

17) Gustavson, F.G.: High-performance linear al-gebra algorithms using new generalized datastructures for matrices, IBM J. Res. Dev.,Vol.47, No.1, pp.31–55 (2003).

18) Park, N., Hong, B. and Prasanna, V.K.:Analysis of Memory Hierarchy Performance ofBlock Data Layout, ICPP ’02: Proc. 2002 In-ternational Conference on Parallel Processing(ICPP’02), p.35 (2002).

19) Asanovic, K.: Vector Microprocessors, PhDThesis, Univ. of California at Berkeley (1998).

20) Callahan, D., Kennedy, K. and Porterfield,A.: Software prefetching, ASPLOS-IV: Proc.4th Int. Conf. on Architectural support forprogramming languages and operating systems,pp.40–52 (1991).

21) Kondo, M., Okawara, H. and Nakamura, H., etal.: SCIMA: A novel processor architecture forhigh performance computing, 4th Internationalconference on HPC in the Asia Pacific Region,Vol.1, pp.355–360 (2000).

22) Shaw, A.C.: Reasoning About Time in Higher-Level Language Software, IEEE Trans. Softw.Eng., Vol.15, No.7, pp.875–889 (1989).

23) Dongarra, J.J., Croz, J.D., Duff, I. andHammarling, S.: A set of Level 3 Basic Lin-ear Algebra Subprograms, ACM Trans. Math.Software, Vol.16, pp.1–17 (1990).

24) Cannon, L.: A Cellular Computer to Imple-ment the Kalman Filter Algorithm, Ph.D. The-

sis, Montana State University (1969).25) Zekri, A.S. and Sedukhin, S.G.: Matrix Trans-

pose on 2D Torus Array Processor., The 6th

IEEE International Conference on Computerand Information Technology, Seoul, Korea,p.45 IEEE Computer Society (2006).

26) Wolfe, M.J.: High Performance Compilers forParallel Computing, Addison-Wesley LongmanPublishing Co., Inc., Boston, MA, USA (1995).

27) Demmel, J.W.: Applied numerical linear alge-bra, SIAM, Philadelphia, PA (1997).

28) Dongarra, J.J., Gustafson, F.G. and Karp,A.: Implementing linear algebra algorithms fordense matrices on a vector pipeline machine,SIAM Review, Vol.26, No.3, pp.91–112 (1984).

29) Montoye, R.K., Hokenek, E. and Runyon,S.L.: Design of the IBM RISC System/6000Floating-Point Execution Unit., IBM J. of Res.and Develop., Vol.34, No.1, pp.59–70 (1990).

(Received July 23, 2007)(Accepted October 31, 2007)

Ahmed S. Zekri received hisM.Sc. degree in computer sci-ence from Alexandria Univer-sity, Egypt in 1999. He is an as-sistant lecturer with the depart-ment of Mathematics and Com-puter Science, Faculty of Sci-

ence, Alexandria University. Currently, he is aPh.D. candidate at the University of Aizu. Hiscurrent research interests include parallel anddistributed computing, vector/matrix process-ing, and parallel algorithm design.

Stanislav G. Sedukhin isa professor and Director of theComputer Software Departmentat the University of Aizu. Hereceived his Ph.D. in ComputerScience and Dr.Sci. (Physicsand Mathematics) from the Rus-

sian Academy of Sciences in 1982 and 1993, re-spectively. His research interests are in paralleland distributed computing, architectural syn-thesis of VLSI-oriented processors, and designof highly-parallel algorithms and application-specific array processors. Dr. Sedukhin is amember of ACM, IEEE Computer Society, andIEICE.

level-3 blas and lu factorization on a matrix processor

Documents