OBJECT-ORIENTED PROGRAMMING FOR LINEAR ALGEBRA

by

John Alan McDonald

TECHNICAL REPORT No. 175

August 1989

Department of Statistics, GN-22

University of Washington

Seattle, Washington 98195 USA

algebra.

ALAN LVH"J LJU

Dept. of Statistics,

July 10,

2 Abstractions

low high mathematical abstractions,losing the level of performance inproblems provided by traditionaltine libraries.

To the results of this experiment (sofar), this paper compares thetation, and of onstandard numerical linear algebra problems withLinpack[4], a classic high-quality Fortran packagefor solving system of linear equations and relatedproblems,

Abstract

numerical analysts and Lisp jSmalltalk pro­share the assumption that languages

"'("...,.t"''''1"1 are more appropriate for traditional,scientific programming than object-

To show how straightfor­ward application of object-oriented design to stan­dard algorithms in numerical linear algebra im­proves clarity and expressiveness, without sacri­ficing speed or accuracy, I describe parts of Cac-

a system for numerical linear algebra and con­strained optimization, implemented in Common

and CLOS,

Keywords: Numerical Analysis, ScientificComputing, Common Lisp Object System

1 Introduction

Choosing appropriate abstractions is an essentialpart both good programming and good mathe-matics. purpose of this paper is to give an ex-

of number-crunching problemsabstraction mechanisms provided

Undergraduate mathematics courses in linear alge­bra tend to come in two styles: a relatively con­crete version often called "Matrix Algebra" a,nd amore abstract version, designed for math majors,that might be called "Finite Dimensional VectorSpaces"[6].

The basic entities in a matrix algebra course arematrices: n x m arrays of numbers. oper-ation is matrLx multiply: = L,kBik * Ckj.

isting numericalcompiled subroutine libraI'iesteractive svstems

dnnelllsliomu vector space course,coaeents are vector spaces

vector space, is a coJle<:tlcmtwo +

aestracnon is to

in

••

• (make-element Abstract-Space &restoptions)

• (element? Object Abstract-Space)

4 Spaces

A space is represented in Cactus by an instance ofa subclass of Abstract-Space. Each space class,therefore, represents a family of spaces (its in­stances). An inheritance graph for some importantspace classes is given in figure l.

The basic Abstract-Space protocol provides away to test whether a given object is an element ofthe space and a way to create a new object c~Jpa.l~le

of representing an element of the space:

tation isSI. Second, C2 may extendresentation that's validthird, C2 may C1

its instances to represent of a set difijoin.tfrom SI. If class inheritance is used to reT)reSelrtsubsetting, then only the first possibility is

Cactus provides two distinct, roughly l:'a.'<lde''''~'

class hierarchies, described in the next two sec­tions. The classes described in section 4 n1'",.,.v,I1'"

representations for spaces; classes in section 5provide representations for points. There is no sim­ple correspondence between classes in thetwo hier­archies, because a given space may use many pointclasses to represent its elements and a given pointclass may be used by many spaces.

many

introduction to

concepts, encouraging appro­nrovrdinz a com-

of abstraction is in appuca-because it is more the same concepts

can to study infinite dimensional spacesapplications to differential equations, signal

processing, etc.

3 Representations

Cactus is designed to closely model the abstrac­tions used a course on finite dimensional (vec­tor) spaces[12]. It provides representations for avariety of spaces, points (elements of some space),and mappings between spaces. The linear algebracore of Cactus deals with vector spaces, vectors,and linear transformations; more general spaces,points, and mappings are used are used to sup­port constrained optimization at a similar level ofabstraction[lO].

By space, I mean essentially a set of elements,possibly with a set of operations defined on thoseelements. A representation for a space must pro­vide a way to tell if a given object is an elementof the space (or, more precisely, if a given objectis a representation of an element of the space). Itshould also provide a way to create new elements ofthe space. In addition, we need to be able to tell if

81 a subset (or subspace, in an appropriateof S2.

makes a space sound suspiciously aa natural first thought is to represent spaces

the space opera­in-

Figure 1: Inheritance graph of some important Cactus space classes (from top to bottom).

1

represents a particularlyfamily of subspaces of euclideanA block

abuse of notation-ego one generic add function isused by all vector spaces. (This restriction impliesthat if a given object is an element of two

one space a linear subspacefar, this has not been a senous llmlta-

tion.)key idea in many fast algorithms is to

problems into small ones that;;)Ulev", ..... separareiy. In numerical linear algebra,

temporarily restricting attention to

Euclideart-Vector-Space)

euclidean vector spaces includes:

should be define mul1tiplespaces over a given set of objects; the vee­

adlJJtJlOn and

section 2, we defined a vector space as a set

• (canonical-bas is-elementEuclideart-Vector-Space Integer)

•

• (make-linear-trartsformationVector-Space Vector-Space).

spaces of the home space.in arbitrary subspaces of the home spaceexpensive and usually avoided.

For the purposes of this paper, we are concernedthe subclasses of

stances represent vectors.protocol includes:

• (add Abstract-Vector Abstract-Vector&:key result)

• (scale Number Abstract-Vector &:keyresult)

Figure 2: Inheritance graph of some important ele­ment representation classes (from top to bottom).

are Lvectors and Block-Lvectors. Lvectors isthe "standard" representation in Cactus for Rnfamily of vector spaces; Block-Lvectors representthe block subspaces of Rn.

• (linear-mix NumberNumber Abstract-Vector &:key

Linear-mix computes a general linear ....UUllUU1(l,­

tion of two vectors.The Euclidean-Vector protocol includes:

• (inner-product Euclidean-VectorEuclidean-Vector)

• (12-norm Euclidean-Vector)

• (11-norm Euclidean-Vector)

spaces.

• (sup-norm Euclidean-Vector)

The only two instantiable classes shownure 2 are Lvector and Block-Lvector.

Lvector instances are the "standard" represen­tation for vectors in Rn, i.e., the elementsLvectors space. An Lvector has aholding its coordinates in a standardsystem.

A Block-Lvector is a compactresentation for an element of itshome space. A Block-Lvectorholding an array justzero coordinates,and

a read-

5 Points

• (element? Abstract-Point

• (home-space Abstract-Point)

Elements of spaces are represented by instancesof subclasses of Abstract-Point. An inheritancegraph for some point classes is in figure 2. Theprotocel for Abstract-Point's includes:

Q 't'~.-" I '''''''''', ~ .nap P l ns : prun" (app"r:d (a 11-"ub"1a"""" 'linear-tran~fornation)

fl

//' <:--- "Lfli

a: 1

Dynamic Liso Listener 2

Figure 3: Inheritance graph of abstract Mapping classes (from left to right) .

6 Mappings for linear transformation classes where aODrc)J:)I-l­ate.

between spaces are represented by in­stances of subclasses of Mapping, some of whichare shown in figure 3. Each Mapping. has domainand codomain spaces. (The slightly pedantic"codomain" is used here, rather than "range," be­cause, strictly speaking, the range of a transfor­mation is the image of it's domain, which mightnot cover the codomain space.) The protocol forMapping's includes:

• (transform Mapping Abstract-Point &keyresult)

The most important subclassLinear-Transformation is MatriX, whichsents the transformation by a two-dimensionalray and implements the protocol functions inusual way (matrix multiply). An importantrelated to Matrix is Restricted-Matrix, whosedomain and codomain are required to be block sub­spaces. A Restricted-Matrix is represented bythe appropriate part of a larger two-dimensionalarray; it is used to focus attention on a smallproblem.

point argument must be an element ofthe mapping's domain; the result, if supplied,must be an element of the codomain.

• (compose Mapping Mapping &key result)

There are a fairly large number ofcates that determine whether ahas some property (eg, unitary?)the choosing an algorithm to use, forin solving a vector equation. For some

swercan beIdentity-Transformation is always U.l.<:t.!5''';U<:1..l.:

In cases a Matrixto

transformations h~1twf~n twospaces are a vector

Linear-Transformation

eqto the codomain of the second. The optional

if must be a m,t.PiHl1·j;arguments domain to

f!:UmEmt's codomain.

earspaces,

(9raph-~ubcla~~e~ 'linear-transfornation :prune '(la-annihilator»

Figure 4: Inheritance graph of Linear Transformation classes (from left to right) .

Another important class of transformations isProduct. A product transformation contains a listof other transformation objects-its factors-andrepresents the composition of the factors. Producttransformations arise from matrix decompositionsdiscussed below.

7: Comparison with matrix alge­bra

point, the apparent simplicity of matrixal!?;e()ra systems may seem to be a significant ad­vant<},.e;e over Cactus.

vector space course).

Further, the apparent simplicity of the matrixgebra system is misleading. The apparent simplic­ity is actually due to the failure to make explicitthe abstractions essential to the underlying algo­rithms. Multiple representations of linear transfor­mations are, in fact, an unavoidable part of any nu­mericallinear algebra system. Linpack representslinear transformations by arrays, parts of arrays,and groups of scalar variables in at least a dozen

ways. Linpack is excellent Fortran, but,because it is Fortran, the representations and therules for their use are not in the code.

#'inverse

to

The LU decomposition is based onit is easy to "iT,v",r-t-" a non-singular trIangUlartrix,

In theof a matrix, T-l, is aninstance of either Inverse-Upper-Triangular orInverse-Lo>1er-Triangular. In either case,has a slot holding the same two-dimensional arrayas T (or a copy). The difference between T-lT is the transform method. T, of course, usesmatrix multiply. T-l uses either back substitutionor forward elimination, depending on whether T isupper or lower triangular. Back substitution andforward elimination are comparable in speed andnumerical stability to matrix multiplication[5].

(Back substitution is based on the following:Suppose U is a non-singular upper triangular ma­trix. Given b, we can find an x such that U-1b = xor, equivalently, Ux = b, by first solving Un,nxn =bn for X n, then substituting X n into Un-1,n-1Xn-1 +Un-1,nXn = bn - 1 to get X n-1, and so on. Forwardelimination is essentially the same.

Back substitution and forward elimination areexamples where thinking in terms of systems ofequations is useful. However, this is not a seriousviolation of the Cactus paradigm, it's en­capsulated within the definition of the two inversetriangular classes.)

The issue that remains is how to A. Theactual factoring is more complicated than simply Land U. vVe actually produce A = (G1G2'"

where U is upper triangular the G, aretriangular. G1 is chosen so that GIl. A = Al haszeros in the first column, below the first element.Then we restrict attention to the (n - 1) x (n - 1)

corner A and recurseG, are

LirJ;eaI'-Pl~odtlct;)inverse (

'Linear-Product:factors

To aP1Dreciate

tors

8 An example

we a Matrix.a Linear-Product

factors toare easy to invert.

Linear-Product we invoke a siI1n.pJ,e IIlet.b.o<l:

advantages of Cactus over matrixconsider a standard algorithm,

LU decomposition.Suppose we have A : nn H> nn, a linear trans­

and b E nn, and we want to solvex E nn such that: Ax = b. If-to make

easy-we assume that A is square and in­vertible, then we need to invert A and apply it tob: x =A-lb.

The thing to notice is that it's a bad idea totry to represent by a matrix, The best floatingpoint matrix approximation to A-I, }I-I, may notbe good enough[5}; for some A's and some b's, theapproximate x = }I-1b maybe be far from the truesolution z.

This isn't a problem for Cactus, where manyother representations are available, but it causestrouble for traditional systems, where the matrixis the only abstraction around. The recommenda­tion of standard numerical linear algebra texts isfor users to change how they think-to give up theelegance of a single vector equation and, instead,think about how to "analyze and solve various sys­tems of simultaneous (scalar] linear equations"[4].

The basic idea underlying many algorithms innumerical linear algebra is to decompose the ma­trix, A = A1 A 2 ... An, in such a way that the fac­tors properties that are convenient for the

at hand. To solve Ax = b, we find fac­whose inverses can be computed and ap­

to vectors etticientlvA - I•. , 1 •

is exactlyit to Dr,:>d.l1ce

a1t;erl.la.ti,re representations, to improve space,or accuracy of is a of curre~~n~lt;r~,e==-•....................................................................................... 'LrJ;lJl..l;r;se (factol;'.s.tGL}»))7 >.

sparse rna-

10 Conclusion

9 Performance

spaces.

Runninz times for Cactus Iu-decomposeo

qr-decompose, and the most closely comparaoreLinpack subroutines, are given in figuresComparing Lisp and Fortran on a :::Jymt)011CSmachine might seem unfair to Fortran; hOWE~ver.

believe similar results would hold on COjo.Vl~nt;t()l(l.;:tl

architectures. For all but the smallest pTClol,ems,the overhead due to the use ofgramming is negligible compared to thein a set of simple floating point array nnp"'U.lIlf,,,

(eg. computing dot products) essentially theas the Fortran BLAS [9J. The relative performanceof Cactus to Linpack on a given machine will be de­termined by how efficiently BLAS operations callbe implemented in Common Lisp versus Fortran OIl

that machine. Cactus includes a protocolr.n.0n Lisp BLAS, to permit specializedimplementations for different Common Lisps.ple array loops execute at close to the samein Symbolics Fortran and Symbolics Commonif the Lisp versions are carefully coded

Limited experience with Commonon standard architectures suggests

appropriate

5:Iu-decompose with SGEFA, on aics 3620 FPA, Symbolics Commonera 7.2 and PCL (AAAI'88 versionFortran 77. times are in seconds.

decomposition, a sta-A PIG1 · · • PnGnU.

For

Hc)us:enolcler vector

a twc-mmensionar

implementation in is almost aabove. The factors

are by instances of the obvious GaussPivot classes. Because it's a little easier, Cac­

tus actually produces a factoring of the inverse:= U-1G;;1 Pn ... GI" PI, where Pi-

l = Pi and= I - gi ® ei.

the Linpack routine SGEFAas a vector of integers, P and

R. vector of integersthe ith ele-

P the ith pivot, taking advan-taze of the fact, that, in this particular case, theoith pivot must be between i and something else. R

packs together the Gi'S and U in one array. Theupper triangle of R is U; theith sub-diagonal col­umn of R holds the non-zero elements of the gi vec­tors, taking advantage of the fact that the first ielements of gi must be zero. To use the results ofSGEFA to solve a system of equations, one passes

right side a vector to SGESL. Toresults for other purpose is not practi-

cal.

The QR decomposition is another standard al­gorithm in numerical linear algebra, similar in out­line LU. Orthogonal Householder transfer-

of Pivot. InLinear-P:rO<1U<:'t transfer­

in the same way as thereturned by LU decomposi-

of routines W,,"U.\..JU'V

[6J P.R. Halmos, Finite-dimensional VectorSpaces. Van Nostrand, Princeton, NJ,

[3J Tony D. DeRose. A coordinate-free approach

to geometric programming (a geometricbra and its implementation).port, Dept. of Computer Science, U.ington, 1989. course notes for ,,","',,",1''''-'''

Spring

[4J J.J. Dongarra, C.B. Moler, J.R. Bunch,G.W. Stewart. LINPACK Users'SIAM, Philiadelphia, 1979.

[5J G.H. Golub and C.F. Van Loan. MatrixComputations. The Johns Hopkins UniversityPress, Baltimore, 1983.

[2] Tony D. DeRose. Geometric programrmng;coordinate-free approach. ACM ."'.~~~i\."·U.

Tutorial Course No. 25, 1988.

[7J Richard D. A primer: 11new SCRATCHPAD. In John Fitch,iter, EUROSAM'84 , Internationalsium on Symbolic and Algebraiction, Proceedings, pages 123-147, ....,"'...1..4>.•,

Springer-Verlag. Lecture Notes,,",J-.l'''U\'''''', voiume 174.

References

6: Performance comparison of Cactusqz--decompcae with Linpack SQRSL, on a Symbol­

with FPA, Symbolics Common Lisp, Gen­PCL (AAA1'88 version) and Symbolics

r ortran 77. times are given in seconds.

n;n~a.IJl.e and to reuse in con-symptom that error messages

semantic content: "The vector V notthe domain of the transformation TO" as

opposed to "Attempting to as e t beyond thearray bounds of #<art-q 30711>."

Better evidence of the advantages of this levelof abstraction will be obtained as Cactus is ex­tended to be a toolkit for problems in constrainedoptimization, which is an area intrinsically morecomplicated than linear algebra. Constrained opti­mization requires careful treatment of the un­

the primaryen:rphasis hastranstormations rather than on spaces).

will be similar to work in computer graphicsby DeRose (2,3] and otners.

An.otl1er compelling aspect that deserves morecomoimng abstract aigeera

symbolic

11 . Acknowledgments

An outline of arizona.'j.'t:!r;hJl1c:al Report 131, Dept. of Statistics, U.

vvaJ.tE~r Noll. Finite-Dimensional Spaces (Al­Geometry, and Analysis), volume 1.,

Steele. Common Lisp, The Language.jJJ.l::'J. ".c;w. Press, 1984.

Wolfram. Mathematica: A Sys­tem for Doing Mathematics by Computer.Addison-Wesley, Redwood City, CA, 1988.