Matrix Algebra From a Statistician’s Perspective

David A. Harville

Matrix AlgebraFrom a Statistician’sPerspective

123

David A. Harville IBM T.J. Watson Research Center Mathematical Sciences Department Yorktown Heights, NY 10598-0218 USA harville@us.ibm.com

Library of Congress Cataloging-in-Publication Data Harville, David A.

Matrix algebra from a statistician's perspective I David A. Harville.

p. em. Includes bibliographical references and index. ISBN 0-387-94978-X (he: alk. paper) I. Algebras, Linear. 2. Linear models (Statistics) I. Title.

QA184.H375 1997 512'.5-dc21 97-9854

Printed on acid-free paper.

© 1997 Springer-Verlag New York, Inc. All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer-Verlag New York, Inc., 175 Fifth Avenue, New York, NY 10010, USA}, except for brief excerpts in connection with reviews or scholarly analy­sis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use of general descriptive names, trade names, trademarks, etc., in this publica­tion, even if the former are not especially identified, is not to be taken as a sign that such names, as understood by the Trade Marks and Merchandise Marks Act, may accordingly be used freely by anyone.

987654 SPIN 11008286

Springer-Verlag is a part of Springer Science+ Business Media

springeron/ine.com

Preface

Matrix algebra plays a very important role in statistics and in many other disci-plines. In many areas of statistics, it has become routine to use matrix algebra inthe presentation and the derivation or verification of results. One such area is linearstatistical models; another is multivariate analysis. In these areas, a knowledge ofmatrix algebra is needed in applying important concepts, as well as in studying theunderlying theory, and is even needed to use various software packages (if theyare to be used with confidence and competence).

On many occasions, I have taught graduate-level courses in linear statisticalmodels. Typically, the prerequisites for such courses include an introductory (un-dergraduate) course in matrix (or linear) algebra. Also typically, the preparationprovided by this prerequisite course is not fully adequate. There are several rea-sons for this. The level of abstraction or generality in the matrix (or linear) algebracourse may have been so high that it did not lead to a “working knowledge” of thesubject, or, at the other extreme, the course may have emphasized computations atthe expense of fundamental concepts. Further, the content of introductory courseson matrix (or linear) algebra varies widely from institution to institution and frominstructor to instructor. Topics such as quadratic forms, partitioned matrices, andgeneralized inverses that play an important role in the study of linear statisticalmodels may be covered inadequately if at all. An additional difficulty is that sev-eral years may have elapsed between the completion of the prerequisite courseon matrix (or linear) algebra and the beginning of the course on linear statisticalmodels.

This book� is about matrix algebra. A distinguishing feature is that the content,the ordering of topics, and the level of generality are ones that I consider appro-priate for someone with an interest in linear statistical models and perhaps also

� The content of the paperback version is essentially the same as that of the earlier, hard-cover version. The paperback version differs from the earlier version in that a numberof (mostly minor) corrections and alterations have been incorporated. In addition, thetypography has been improved—as a side effect, the content and the numbering of theindividual pages differ somewhat from those in the earlier version.

vi Preface

for someone with an interest in another area of statistics or in a related discipline.I have tried to keep the presentation at a level that is suitable for anyone whohas had an introductory course in matrix (or linear) algebra. In fact, the book isessentially self-contained, and it is hoped that much, if not all, of the material maybe comprehensible to a determined reader with relatively little previous exposureto matrix algebra. To make the material readable for as broad an audience as pos-sible, I have avoided the use of abbreviations and acronyms and have sometimesadopted terminology and notation that may seem more meaningful and familiar tothe non-mathematician than those favored by mathematicians. Proofs are providedfor essentially all of the results in the book. The book includes a number of resultsand proofs that are not readily available from standard sources and many othersthat can be found only in relatively high-level books or in journal articles.

The book can be used as a companion to the textbook in a course on linearstatistical models or on a related topic—it can be used to supplement whateverresults on matrices may be included in the textbook and as a source of proofs.And, it can be used as a primary or supplementary text in a second course onmatrices, including a course designed to enhance the preparation of the studentsfor a course or courses on linear statistical models and/or related topics. Above all,it can serve as a convenient reference book for statisticians and for various otherprofessionals.

While the motivation for the writing of the book came from the statistical ap-plications of matrix algebra, the book itself does not include any appreciable dis-cussion of statistical applications. It is assumed that the book is being read becausethe reader is aware of the applications (or at least of the potential for applications)or because the material is of intrinsic interest—this assumption is consistent withthe uses discussed in the previous paragraph. (In any case, I have found that thediscussions of applications that are sometimes interjected into treatises on matrixalgebra tend to be meaningful only to those who are already knowledgeable aboutthe applications and can be more of a distraction than a help.)

The book has a number of features that combine to set it apart from the more tra-ditional books on matrix algebra—it also differs in significant respects from thosematrix-algebra books that share its (statistical) orientation, such as the books ofSearle (1982), Graybill (1983), and Basilevsky (1983). The coverage is restricted toreal matrices (i.e., matrices whose elements are real numbers)—complex matrices(i.e., matrices whose elements are complex numbers) are typically not encounteredin statistical applications, and their exclusion leads to simplifications in terminol-ogy, notation, and results. The coverage includes linear spaces, but only linearspaces whose members are (real) matrices—the inclusion of linear spaces facili-tates a deeper understanding of various matrix concepts (e.g., rank) that are veryrelevant in applications to linear statistical models, while the restriction to linearspaces whose members are matrices makes the presentation more appropriate forthe intended audience.

The book features an extensive discussion of generalized inverses and makesheavy use of generalized inverses in the discussion of such standard topics as thesolution of linear systems and the rank of a matrix. The discussion of eigenvalues

Preface vii

and eigenvectors is deferred until the next-to-last chapter of the book—I have foundit unnecessary to use results on eigenvalues and eigenvectors in teaching a firstcourse on linear statistical models and, in any case, find it aesthetically displeasingto use results on eigenvalues and eigenvectors to prove more elementary matrixresults. And the discussion of linear transformations is deferred until the very lastchapter—in more advanced presentations, matrices are regarded as subservient tolinear transformations.

The book provides rather extensive coverage of some nonstandard topics thathave important applications in statistics and in many other disciplines. These in-clude matrix differentiation (Chapter 15), the vec and vech operators (Chapter 16),the minimization of a second-degree polynomial (in n variables) subject to linearconstraints (Chapter 19), and the ranks, determinants, and ordinary and general-ized inverses of partitioned matrices and of sums of matrices (Chapter 18 and partsof Chapters 8, 9, 13 16, 17, and 19). An attempt has been made to write the bookin such a way that the presentation is coherent and non-redundant but, at the sametime, is conducive to using the various parts of the book selectively.

With the obvious exception of certain of their parts, Chapters 12 through 22(which comprise approximately three-quarters of the book’s pages) can be read inarbitrary order. The ordering of Chapters 1 through 11 (both relative to each otherand relative to Chapters 12 through 22) is much more critical. Nevertheless, evenChapters 1 through 11 include sections or subsections that are prerequisites foronly a small part of the subsequent material. More often than not, the less essentialsections or subsections are deferred until the end of the chapter or section.

The book does not address the computational aspects of matrix algebra inany systematic way, however it does include descriptions and discussion of certaincomputational strategies and covers a number of results that can be useful in dealingwith computational issues. Matrix norms are discussed, but only to a limited extent.In particular, the coverage of matrix norms is restricted to those norms that aredefined in terms of inner products.

In writing the book, I was influenced to a considerable extent by Halmos’s(1958) book on finite-dimensional vector spaces, by Marsaglia and Styan’s (1974)paper on ranks, by Henderson and Searle’s (1979, 1981b) papers on the vec andvech operators, by Magnus and Neudecker’s (1988) book on matrix differentialcalculus, and by Rao and Mitra’s (1971) book on generalized inverses. And I ben-efited from conversations with Oscar Kempthorne and from reading some notes(on linear systems, determinants, matrices, and quadratic forms) that he had pre-pared for a course (on linear statistical models) at Iowa State University. I alsobenefited from reading the first two chapters (pertaining to linear algebra) of notesprepared by Justus F. Seely for a course (on linear statistical models) at OregonState University.

The book contains many numbered exercises. The exercises are located at (ornear) the ends of the chapters and are grouped by section—some exercises mayrequire the use of results covered in previous sections, chapters, or exercises. Manyof the exercises consist of verifying results supplementary to those included in thebody of the chapter. By breaking some of the more difficult exercises into parts

viii Preface

and/or by providing hints, I have attempted to make all of the exercises appropriatefor the intended audience. I have prepared solutions to all of the exercises, and itis my intention to make them available on at least a limited basis.�

The origin and historical development of many of the results covered in thebook are difficult (if not impossible) to discern, and I have not made any systematicattempt to do so. However, each of Chapters 15 through 21 ends with a short sectionentitled Bibliographic and Supplementary Notes. Sources that I have drawn onmore-or-less directly for an extensive amount of material are identified in thatsection. Sources that trace the historical development of various ideas, results, andterminology are also identified. And, for certain of the sections in a chapter, someindication may be given of whether that section is a prerequisite for various othersections (or vice versa).

The book is divided into (22) numbered chapters, the chapters into numberedsections, and (in some cases) the sections into lettered subsections. Sections areidentified by two numbers (chapter and section within chapter) separated by adecimal point—thus, the third section of Chapter 9 is referred to as Section 9.3.Within a section, a subsection is referred to by letter alone. A subsection in adifferent chapter or in a different section of the same chapter is referred to byreferring to the section and by appending a letter to the section number—forexample, in Section 9.3, Subsection b of Section 9.1 is referred to as Section9.1b. An exercise in a different chapter is referred to by the number obtained byinserting the chapter number (and a decimal point) in front of the exercise number.

Certain of the displayed “equations” are numbered. An equation number com-prises two parts (corresponding to section within chapter and equation withinsection) separated by a decimal point (and is enclosed in parentheses). An equa-tion in a different chapter is referred to by the “number” obtained by starting withthe chapter number and appending a decimal point and the equation number—forexample, in Chapter 6, result (2.5) of Chapter 5 is referred to as result (5.2.5). Forpurposes of numbering (and referring to) equations in the exercises, the exercisesin each chapter are to be regarded as forming Section E of that chapter.

Preliminary work on the book dates back to the 1982–1983 academic year,which I spent as a visiting professor in the Department of Mathematics at theUniversity of Texas at Austin (on a faculty improvement leave from my thenposition as a professor of statistics at Iowa State University). The actual writingbegan after my return to Iowa State and continued on a sporadic basis (as timepermitted) until my departure in December 1995. The work was completed duringthe first part of my tenure in the Mathematical Sciences Department of the IBMThomas J. Watson Research Center.

I am indebted to Betty Flehinger, Emmanuel Yashchin, Claude Greengard, andBill Pulleyblank (all of whom are or were managers at the Research Center) for thetime and support they provided for this activity. The most valuable of that support(by far) came in the form of the secretarial help of Peggy Cargiulo, who entered

� The solutions were published by Springer-Verlag in 2001 in a volume entitled MatrixAlgebra: Exercises and Solutions (ISBN 0-387-95318-3).

Preface ix

the last six chapters of the book in LATEX and was of immense help in getting themanuscript into final form. I am also indebted to Darlene Wicks (of Iowa StateUniversity), who entered Chapters 1 through 16 in LATEX.

I wish to thank John Kimmel, who has been my editor at Springer-Verlag. Hehas been everything an author could hope for. I also wish to thank Paul Nikolai(formerly of the Air Force Flight Dynamics Laboratory of Wright-Patterson AirForce Base, Ohio) and Dale Zimmerman (of the Department of Statistics andActuarial Science of the University of Iowa), whose careful reading and markingof the manuscript led to a number of corrections and improvements. These changeswere in addition to ones stimulated by the earlier comments of two anonymousreviewers (and by the comments of the editor).

Contents

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v

1 Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Basic Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Basic Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Some Basic Types of Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2 Submatrices and Partitioned Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . 132.1 Some Terminology and Basic Results . . . . . . . . . . . . . . . . . . . . . . . 132.2 Scalar Multiples, Transposes, Sums, and Products of Partitioned

Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.3 Some Results on the Product of a Matrix and a Column Vector . . 192.4 Expansion of a Matrix in Terms of Its Rows, Columns, or

Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3 Linear Dependence and Independence . . . . . . . . . . . . . . . . . . . . . . . . . . 233.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233.2 Some Basic Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

4 Linear Spaces: Row and Column Spaces . . . . . . . . . . . . . . . . . . . . . . . . 274.1 Some Definitions, Notation, and Basic Relationships and

Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274.2 Subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294.3 Bases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314.4 Rank of a Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364.5 Some Basic Results on Partitioned Matrices and on Sums of

Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

xii Contents

5 Trace of a (Square) Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 495.1 Definition and Basic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . 495.2 Trace of a Product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505.3 Some Equivalent Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

6 Geometrical Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556.1 Definitions: Norm, Distance, Angle, Inner Product, and

Orthogonality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556.2 Orthogonal and Orthonormal Sets . . . . . . . . . . . . . . . . . . . . . . . . . . 616.3 Schwarz Inequality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 626.4 Orthonormal Bases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

7 Linear Systems: Consistency and Compatibility . . . . . . . . . . . . . . . . . 717.1 Some Basic Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 717.2 Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 727.3 Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 737.4 Linear Systems of the Form A0AX D A0B . . . . . . . . . . . . . . . . . . . 74

Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

8 Inverse Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 798.1 Some Definitions and Basic Results . . . . . . . . . . . . . . . . . . . . . . . . 798.2 Properties of Inverse Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 818.3 Premultiplication or Postmultiplication by a Matrix of Full

Column or Row Rank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 828.4 Orthogonal Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 848.5 Some Basic Results on the Ranks and Inverses of Partitioned

Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

9 Generalized Inverses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1079.1 Definition, Existence, and a Connection to the Solution of

Linear Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1079.2 Some Alternative Characterizations . . . . . . . . . . . . . . . . . . . . . . . . . 1099.3 Some Elementary Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1179.4 Invariance to the Choice of a Generalized Inverse . . . . . . . . . . . . . 1199.5 A Necessary and Sufficient Condition for the Consistency of a

Linear System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1209.6 Some Results on the Ranks and Generalized Inverses of

Partitioned Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1219.7 Extension of Some Results on Systems of the Form AX D B to

Systems of the Form AXC D B . . . . . . . . . . . . . . . . . . . . . . . . . . . 126Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

Contents xiii

10 Idempotent Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13310.1 Definition and Some Basic Properties . . . . . . . . . . . . . . . . . . . . . . . 13310.2 Some Basic Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134

Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

11 Linear Systems: Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13911.1 Some Terminology, Notation, and Basic Results . . . . . . . . . . . . . . 13911.2 General Form of a Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14011.3 Number of Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14211.4 A Basic Result on Null Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14411.5 An Alternative Expression for the General Form of a Solution . . 14411.6 Equivalent Linear Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14511.7 Null and Column Spaces of Idempotent Matrices . . . . . . . . . . . . . 14611.8 Linear Systems With Nonsingular Triangular or

Block-Triangular Coefficient Matrices . . . . . . . . . . . . . . . . . . . . . . 14611.9 A Computational Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14911.10 Linear Combinations of the Unknowns . . . . . . . . . . . . . . . . . . . . . . 15011.11 Absorption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15211.12 Extensions to Systems of the Form AXC D B . . . . . . . . . . . . . . . 157

Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158

12 Projections and Projection Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16112.1 Some General Results, Terminology, and Notation . . . . . . . . . . . . 16112.2 Projection of a Column Vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16312.3 Projection Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16612.4 Least Squares Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17012.5 Orthogonal Complements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172

Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177

13 Determinants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17913.1 Some Definitions, Notation, and Special Cases . . . . . . . . . . . . . . . 17913.2 Some Basic Properties of Determinants . . . . . . . . . . . . . . . . . . . . . 18313.3 Partitioned Matrices, Products of Matrices, and Inverse Matrices 18713.4 A Computational Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19113.5 Cofactors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19113.6 Vandermonde Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19513.7 Some Results on the Determinant of the Sum of Two Matrices . . 19713.8 Laplace’s Theorem and the Binet-Cauchy Formula . . . . . . . . . . . . 200

Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205

14 Linear, Bilinear, and Quadratic Forms . . . . . . . . . . . . . . . . . . . . . . . . . . 20914.1 Some Terminology and Basic Results . . . . . . . . . . . . . . . . . . . . . . . 20914.2 Nonnegative Definite Quadratic Forms and Matrices . . . . . . . . . . 21214.3 Decomposition of Symmetric and Symmetric Nonnegative

Definite Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218

xiv Contents

14.4 Generalized Inverses of Symmetric Nonnegative DefiniteMatrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222

14.5 LDU, U0DU, and Cholesky Decompositions . . . . . . . . . . . . . . . . . 22314.6 Skew-Symmetric Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23914.7 Trace of a Nonnegative Definite Matrix . . . . . . . . . . . . . . . . . . . . . 24014.8 Partitioned Nonnegative Definite Matrices . . . . . . . . . . . . . . . . . . . 24314.9 Some Results on Determinants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24714.10 Geometrical Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25514.11 Some Results on Ranks and Row and Column Spaces and on

Linear Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25914.12 Projections, Projection Matrices, and Orthogonal Complements . 260

Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277

15 Matrix Differentiation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28915.1 Definitions, Notation, and Other Preliminaries . . . . . . . . . . . . . . . . 29015.2 Differentiation of (Scalar-Valued) Functions: Some Elementary

Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29615.3 Differentiation of Linear and Quadratic Forms . . . . . . . . . . . . . . . 29815.4 Differentiation of Matrix Sums, Products, and Transposes (and

of Matrices of Constants) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30015.5 Differentiation of a Vector or (Unrestricted or Symmetric)

Matrix With Respect to Its Elements . . . . . . . . . . . . . . . . . . . . . . . . 30315.6 Differentiation of a Trace of a Matrix . . . . . . . . . . . . . . . . . . . . . . . 30415.7 The Chain Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30615.8 First-Order Partial Derivatives of Determinants and Inverse and

Adjoint Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30815.9 Second-Order Partial Derivatives of Determinants and Inverse

Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31215.10 Differentiation of Generalized Inverses . . . . . . . . . . . . . . . . . . . . . . 31415.11 Differentiation of Projection Matrices . . . . . . . . . . . . . . . . . . . . . . . 31915.12 Evaluation of Some Multiple Integrals . . . . . . . . . . . . . . . . . . . . . . 324

Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327Bibliographic and Supplementary Notes . . . . . . . . . . . . . . . . . . . . . 335

16 Kronecker Products and the Vec and Vech Operators . . . . . . . . . . . . 33716.1 The Kronecker Product of Two or More Matrices: Definition

and Some Basic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33716.2 The Vec Operator: Definition and Some Basic Properties . . . . . . . 34316.3 Vec-Permutation Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34716.4 The Vech Operator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35416.5 Reformulation of a Linear System . . . . . . . . . . . . . . . . . . . . . . . . . . 36716.6 Some Results on Jacobian Matrices . . . . . . . . . . . . . . . . . . . . . . . . . 368

Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371Bibliographic and Supplementary Notes . . . . . . . . . . . . . . . . . . . . . 377

Contents xv

17 Intersections and Sums of Subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . 37917.1 Definitions and Some Basic Properties . . . . . . . . . . . . . . . . . . . . . . 37917.2 Some Results on Row and Column Spaces and on the Ranks of

Partitioned Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38517.3 Some Results on Linear Systems and on Generalized Inverses

of Partitioned Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39217.4 Subspaces: Sum of Their Dimensions Versus Dimension of

Their Sum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39617.5 Some Results on the Rank of a Product of Matrices . . . . . . . . . . . 39817.6 Projections Along a Subspace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40217.7 Some Further Results on the Essential Disjointness and

Orthogonality of Subspaces and on Projections and ProjectionMatrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411Bibliographic and Supplementary Notes . . . . . . . . . . . . . . . . . . . . . 417

18 Sums (and Differences) of Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41918.1 Some Results on Determinants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41918.2 Some Results on Inverses and Generalized Inverses and on

Linear Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42318.3 Some Results on Positive (and Nonnegative) Definiteness . . . . . . 43718.4 Some Results on Idempotency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43918.5 Some Results on Ranks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444

Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 450Bibliographic and Supplementary Notes . . . . . . . . . . . . . . . . . . . . . 458

19 Minimization of a Second-Degree Polynomial (in n Variables)Subject to Linear Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45919.1 Unconstrained Minimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46019.2 Constrained Minimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46319.3 Explicit Expressions for Solutions to the Constrained

Minimization Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46819.4 Some Results on Generalized Inverses of Partitioned Matrices . . 47619.5 Some Additional Results on the Form of Solutions to the

Constrained Minimization Problem . . . . . . . . . . . . . . . . . . . . . . . . . 48319.6 Transformation of the Constrained Minimization Problem to an

Unconstrained Minimization Problem . . . . . . . . . . . . . . . . . . . . . . . 48919.7 The Effect of Constraints on the Generalized Least Squares

Problem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 491Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492Bibliographic and Supplementary Notes . . . . . . . . . . . . . . . . . . . . . 495

20 The Moore-Penrose Inverse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49720.1 Definition, Existence, and Uniqueness (of the Moore-Penrose

Inverse) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49720.2 Some Special Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 499

xvi Contents

20.3 Special Types of Generalized Inverses . . . . . . . . . . . . . . . . . . . . . . 50020.4 Some Alternative Representations and Characterizations . . . . . . . 50720.5 Some Basic Properties and Relationships . . . . . . . . . . . . . . . . . . . . 50820.6 Minimum Norm Solution to the Least Squares Problem . . . . . . . . 51220.7 Expression of the Moore-Penrose Inverse as a Limit . . . . . . . . . . . 51220.8 Differentiation of the Moore-Penrose Inverse . . . . . . . . . . . . . . . . . 514

Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517Bibliographic and Supplementary Notes . . . . . . . . . . . . . . . . . . . . . 519

21 Eigenvalues and Eigenvectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52121.1 Definitions, Terminology, and Some Basic Results . . . . . . . . . . . . 52221.2 Eigenvalues of Triangular or Block-Triangular Matrices and of

Diagonal or Block-Diagonal Matrices . . . . . . . . . . . . . . . . . . . . . . . 52821.3 Similar Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53021.4 Linear Independence of Eigenvectors . . . . . . . . . . . . . . . . . . . . . . . 53421.5 Diagonalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53721.6 Expressions for the Trace and Determinant of a Matrix . . . . . . . . 54521.7 Some Results on the Moore-Penrose Inverse of a Symmetric

Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54621.8 Eigenvalues of Orthogonal, Idempotent, and Nonnegative

Definite Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54821.9 Square Root of a Symmetric Nonnegative Definite Matrix . . . . . . 55021.10 Some Relationships . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55121.11 Eigenvalues and Eigenvectors of Kronecker Products . . . . . . . . . . 55421.12 Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55621.13 Simultaneous Diagonalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56621.14 Generalized Eigenvalue Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . 56921.15 Differentiation of Eigenvalues and Eigenvectors . . . . . . . . . . . . . . 57121.16 An Equivalence (Involving Determinants and Polynomials) . . . . . 574

Appendix: Some Properties of Polynomials . . . . . . . . . . . . . . . . . . 580Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 582Bibliographic and Supplementary Notes . . . . . . . . . . . . . . . . . . . . . 588

22 Linear Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58922.1 Some Definitions, Terminology, and Basic Results . . . . . . . . . . . . 58922.2 Scalar Multiples, Sums, and Products of Linear Transformations 59522.3 Inverse Transformations and Isomorphic Linear Spaces . . . . . . . . 59822.4 Matrix Representation of a Linear Transformation . . . . . . . . . . . . 60122.5 Terminology and Properties Shared by a Linear Transformation

and Its Matrix Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60922.6 Linear Functionals and Dual Transformations . . . . . . . . . . . . . . . . 612

Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 616

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 621

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625

Errata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E1