JOSEPH A. W O L F *

F I N I T E N E S S OF O R B I T S T R U C T U R E

F O R R E A L F L A G M A N I F O L D S

ABSTRACT. Let G be a reductive real Lie group, a an involutive automorphism of (7, and L = G u the fixed point set of a. It is shown that G has only finitely many L-conjugacy classes of parabolic subgroups, so if P is a parabolic subgroup of G then there are only finitely many L-orbits on the real flag manifold G/P. This is done by showing that G has only finitely many L-conjugacy classes of a-stable Cartan subgroups. These results extend known facts for the case where G is a complex group and L is a real form of G.

Key words and phrases: flag manifold, reductive Lie group, semisimple Lie group, para- bolic subgroup, Caftan subgroup, Cartan subalgebra.

AMS Subject Classification (1970) Primary: 22-50, 22-70, 53-66 Secondary: 14-45, 57-47

1. I N T R O D U C T I O N

One knows [4] that there are only finitely many conjugacy classes of Cartan subalgebras in a reductive real Lie algebra. If G is a complex reductive Lie group and L is a real form, it follows [8] that there are only finitely many L-eonjugacy classes of parabolic subgroups of G. In particular, if X= G/P is a complex flag manifold of G, then [8] there are only finitely many L-orbits on X. Here we extend these results to the case where G is a reductive real Lie group and L is the fixed point set of an involutive automorphism, and we indicate the scope of applicability of the extension.

2. C O N J U G A C Y OF CARTAN SUBALGEBRAS

If G is a Lie group then Go denotes its identity component, g denotes its Lie algebra, and Int (g) denotes the inner automorphism group {Ad (g):g e Go) of g.

THEOREM 1. Let g be a reductive real Lie algebra, a an involutive auto- morphism of g, and 1 = g~ its fixed point set. Let L o denote the analytic sub- group oflnt (g ) for I. Then there are only finitely many Lo-conjugacy classes of a-stable Cartan subalgebras of g.

Proof. It suffices to consider the case where g is semisimple and has no proper a-stable ideal. We now assume that, and we fix the notation g = I +m where rrt= {x~g:tr(x)= -x} .

* Research partially supported by NSF Grant GP-16651.

Geometriae Dedicata 3 (1974) 377-384. All Rights Reserved Copyright © 1974 by D. Reidel Publishing Company, Dordrecht-Holland

378 JOSEPH A. W O L F

Case 1: Int(g) is compact. This case follows directly from Kostant 's proof [4] that g* = I +i ra has only finitely many conjugacy classes of Cartan subal- gebras. Kostant shows that every Cartan subalgebra of g* is conjugate to a a-stable one and that g* has only finitely many Lo-conjugacy classes of a-stable Cartan subalgebras D*. Then ~)* ~ D = ([9* n I) + i(D* n ira) gives our finiteness assertion.

Case 2: Int(g) is complex and a is conjugate-linear. Then I is a real form of g, and our assertion is Kostant's result [4] on finiteness of the number of conjugacy classes of Cartan subalgebras of I.

Case 3: Int (g) is complex and a is complex-linear. This is the nontrivial case. Fix a Cartan involution 8 of g that commutes with a. Its fixed point set go is a a-stable compact real form of g, so

g o = i O + m 0 where 1 ° = I n g ° and m ° = m c ~ g ° .

Let Lg denote the analytic subgroup of Lo for [o. Case 1 tells us that gO has only finitely many Lo°-conjugacy classes of a-stable Caftan subalgebras. Since gO is a real form of g, this says that g has only finitely many Lo°-conjugacy classes of 0-stable a-stable Cartan subalgebras. L~-conjugacy implies Lo- conjugacy. Now we need only show that every Lo-conjugacy class of a-stable Caftan subalgebras of g contains a 0-stable algebra.

Let ~ be an Lo-conjugacy class of a-stable Cartan subalgebras of g. We first consider the situation

if D ~ ¢ ' then D : m i.e. [ ) c ~ I = 0 .

Fix t ) ~ and let [)~ be its real form spanned by the roots. If ~b is an b-root then a*q~= -~b because Dcrrt. Thus g has a a-stable compact real form g' with i D # = g'. Let 0' denote the Cartan involution of g that is complex con- jugation over g'. Evidently 0' (t))= t). Thus, to exhibit a 8-stable element of

it suffices to prove that 0 and 0' are L0-conjugate, and for this we need only check that g '= gO' is Lo-conjugate to gO.

As g' is a-stable, I °' = I c~ g' is a compact real form of I, hence is Lo-con- jugate to I °. Thus we may assume I°'= I °. That assumption made, g ' = = I ° + m °' just as gO= i0 + m o. Since I ° is a real form of I, I ° c 0 °r and 00' is complex linear, we now have I ~ goo'.

( = 8 ' -1 .8 is a complex linear automorphism of g and trivial on [, so (Ira commutes with every element of Loire. If ~1,~ = + 1 then 0 = + 0' on m, so m °' = rrt e or m °" = i m °. If ra °' = m ° then g' = go as desired. If m °" = ira ° then g '= I ° +ira °, which is noncompact, contradicting the fact that 8' is a Cartan involution. Now we assume ~1,,~ + 1. g is simple because it has no proper

O R B I T S T R U C T U R E F O R R E A L F L A G M A N I F O L D S 379

a-stable ideal, so now (go, p) is an irreducible hermitian symmetric pair, and r e=m+ +m_ direct sum of I-invariant I-irreducible subspaces that are interchanged by 0 and also by 8'. Thus we have a complex number g with

~1~+=~ and ~ l ~ . = ~ .

Let x± era± and compute

x+ + x _ = 02(x+ + x _ )

= (o'¢)'(x+ + x_) = O'~(O'~x_ + O'~x+) = +

Thus ~ = 1 and ~ = Ad (z) where z is central in L~. Now Om °" = O' Ad (z) m °' = • = 0'11l °" = 1110', SO

m e' = (m"" n m °) + (m °' c~ ira").

The second summand vanishes because g' is compact. We conclude g '= ga. In summary: if JY is an Lo-conjugacy class of a-stable Caftan subalgebras

of g, and if b e . ~ implies I~=m, we have shown that ~Y' contains a P-stable algebra.

Now let .Yg be any Lo-conjugacy class of a-stable Cartan subalgebras of g. Let ~)~ ~¢g and split [~ = (b n I )+ (I~ nm) . Passing to the derived algebra of the g-centralizer of ~ n I, we reduce to the case just considered, obtaining a P-stable Lo-conjugate of ~). This completes the argument for Case 3.

Case 4: the general case. Let n=dimg, r=rank g and N = ( n ) - 1. Use

Pliicker coordinates to view the Grassmannian of r-dimensional subspaces of gc as a subvariety of the complex projective space PN(C). Let Lo c denote the analytic subgroup of Int(g c) with Lie algebra I c. Now every LC-eon - jugacy class of Cartan subalgebras of gc is an LC-orbit on pN (C) under the rational representation A' (Adln t (gc)lLc). If O is such an orbit, then [6, Lemma 1.1 ] ~ c~ P N (R) is a finite union of Lo-orbits. Case 4 now follows from Cases 2 and 3. Q.E.D.

3. C O N J U G A C Y O F P A R A B O L I C S U B G R O U P S

If G is a connected complex semisimple Lie group with Lie algebra g, and if P is a complex Lie subgroup with Lie algebra p, then one has equivalent conditions

(i) G/P is compact, (ii) G/P is a compact simply connected kaehler manifold,

380 JOSEPH A. WOLF

(iii) G/P is a complex projective variety, (iv) G/P is a closed G-orbit in a projective representation. Under those circumstances, P is a parabolic subgroup of G, p is a parabolic

subalgebra of g, and one can see that P = {#~G:Ad(g) p =p}. Let G be a complex Lie group, g its Lie algebra, ~ the solvable radical of

fl, and n:g ~ fl/~ the projection. The parabolic subalgebras of g are the n -1 (q) where q is a parabolic subalgebra of g[~. The parabolic subgroups of G are the normalizers P = {g~G:Ad(g) p =p} where p is a parabolic subal- gebra of g; then p is the Lie algebra of P.

Let G be a real Lie group and fl its Lie algebra. Then parabolic subalgebra of g means a subalgebra p = g c~ q where q is a parabolic subalgebra of gc stable under complex conjugation over ft. The parabolic subgroups of G are the normalizers P = {g ~ G: Ad (g) p = p } where p is a parabolic subalgebra of fl, and then P has Lie algebra p.

THEOREM 2. Let G be a real Lie group, tr an involutive automorphism of G, and L= G ° its fixed point set. Then there are only finitely many Lo-con- jugacy classes of parabolic subgroups of G.

Proof. We will show that g has only finitely many Lo-eonjugacy classes of parabolic subalgebras. For this we may assume G connected and simply connected. Now G has ¢-stable Levi decomposition GI" S where S is the solvable radical and we may replace G by its semisimple quotient G/S ~- Gt. Thus we may assume g semisimple.

Let p be a parabolic subalgebra of g. Then p c~ a (p) is a-stable and con- tains a Cartan subalgebra of g, so it contains a a-stable Cartan subalgebra D of g. There are only finitely many parabolic subalgebras of g containing any given Caftan subalgebra, and Theorem 1 says that there are only finitely many possibilities for D up to Lo-conjugacy. Thus there are only finitely many possibilities for p up to L0-conjugacy. Q.E.D.

4. APPLICATION TO REAL FLAG MANIFOLDS

By real flag manifold we mean a homogeneous space X= G/P where G is a real Lie group and P is a parabolic subgroup. Since P is its own normalizer in G, we may view Xas the set of all G-conjugates of P under x ~ {g e G: g (x) = = x } . I f S is a subgroup of G, then the orbit S(x) corresponds to the S-con-

jugacy class of {g~G:g(x)=x}. Now Theorem 2 gives us

THEOREM 3. Let G be a real Lie group, P a parabolic subgroup, and X=G/P the corresponding real flag manifold. Let ~ be an involutive auto- morphism of G and L= G ¢ its fixed point set. Then there are only finitely many

O R B I T S T R U C T U R E F O R R E A L F L A G M A N I F O L D S 381

Lo-orbits on X. In particular, L and Lo each has both open and closed orbits on X, and each L.orbit is a finite union of Lo-orbits.

A trivial consequence: X has only finitely many topological components. Here note that X is connected if and only if every G-conjugate of P is Go- conjugate to P.

In the complex case (G is a complex Lie group, P is a complex parabolic subgroup and L is a real form of G) there is just one dosed Lo-orbit on every topological component of X= G/P [8, Theorem 3.3] Here is an example in the real case where there are two closed orbits. Let G= SL (2; R) and

a_ 1 :a, beR with a G O

so that X= G/P is the circle R u {~ } bounding the upper half plane, the action being

: x t - ~ - - . c x + d

Let tr be conjugation by

('0 then L = G ~ is the Cartan subgroup

{(0 a 0 t ) : 0 # a ~ R l

with Lo given by a>0. As

(0 0) a-1 : x~-~a2x,

L and Lo have the same orbits on X, which are

two dosed orbits {0} and {~}, and two open orbits {xeR:x>O} and {xeR:x<O}.

This is a variation on the ~ilov boundary example in 5 below.

5. A P P L I C A T I O N TO S Y M M E T R I C R - S P A C E S

A symmetric R-space is a compact Riemannian symmetric space X=K/V which admits a connected effective Lie transformation group G such that (i) K = G and (ii) if Y is a G-stable Riemannian factor of ,Y then G acts on Y as a noncompact group. Then [5] G is semisimple, K is a maximal corn-

382 JOSEPH A. W O L F

pact subgroup of G, and, up to finite covering, X is a real flag manifold G/P. It often happens that G contains an interesting noneompact form L of K.

EXAMPLE: Grassmann Manifolds. Let F denote the field of real, complex or quaternion numbers. If n>0 then U(n; F) denotes its unitary group, which is the orthogonal group O(n), the complex unitary group U(n) or the unitary symplectic group Sp(n). The grassmannian of k-planes in F k+~ is the compact Riemannian symmetric space

X = K/V = U(k + l; F)/U(k; F) x U(l; F).

Under the action of the general linear group G=GL(k+I ; F) it is a real flag G]P where

{ /*l*'fl P = g~G:g hasmatrix \ ~ [ , j j .

Let L be the F-unitary group of any of the nondegenerate F-Hermitian forms h(x, y)= (xlYl + ' " +Xp~p)- (xp+ lyp+ ~ +-.. +xp+q~p+~) where p +q=k+l . Then Witt's Theorem (over F) gives us the L-orbit structure of X.

EXAMPLE: Hermitian Symmetric Spaces. Here X = K / V is an Hermitian symmetric space of compact type and G = K c. The dual bounded symmetric domain D = K*/V where K* is a certain noncompact real form of G, and the Borel embedding realizes D as a certain open K*-orbit on X. One knows the K*-orbit structure of X ([7-9]). Let L be an arbitrary real form of G.

EXAMPLE: ~ilov Boundaries of Tube Domains. Let X= K/V be an hermitian symmetric space of compact type and D = K*/V the dual bounded symmetric domain. Suppose that D is holomorphically equivalent to a tube domain. Then its Bergman-~ilov boundary is a symmetric R-space S = VIE= K*]W; see [3]. Let L be an isotropy subgroup of K* on an open K*-orbit in X. If one expresses the open orbit as K* (c~x0) where x0 is the base point and ez is a partial Cayley transform then L = K* n Ad (c~) V c, a possibly-noncom- pact form of V.

These examples all fit into the following pattern, and the result allows us to apply Theorem 3 to symmetric R-spaces. Here note that any reductive subalgebra of g is invariant under a Cartan involution.

THEOREM 4. Let fl be a semisimple Lie algebra, 0 a Cartan involution, and g =~ + s the eigenspace decomposition under O, so that ~ = go is the maximal compactly embedded subalgebra. Suppose that g has no proper O-stable ideal,

ORBIT S T R U C T U R E FOR REAL FLA G M A N I F O L D S 383

i.e. that (g, [) is an irreducible Riemannian symmetric pair of noncompact type. Let I be a O-stable subalgebra of g such that I c'~[c.

(1) There is a complex linear automorphism ~ of g c that preserves the com- pact form ~ + is and sends I c to ~c.

(2) g is stable under the involutive automorphism a=~-10~, and I = g ~. Remark. I suspect that Theorem 4 is true for g reductive and (g, ~) assumed

irreducible. But this does not affect the application to symmetric R-spaces. Proof. We first check that (1) implies (2). Since ~ ( I c ) = [ c, the fixed point

set (gc) ,=~-l . (gc)o=~-l(~c)=[c which is stable under 0. Thus 0 and a commute. Let m c denote the ( - 1)-eigenspace of a, so gc = ic + m c under a. Then

gc = ([c c~ [) + ([c c~ is) + (lrt c n [) + (rrt c c~ i~) + + (I c n i l ) + (ic c~ ~) + (m c n i I ) + (m c c~ is).

Each of these eight summands is o-stable because a preserves ~c, sc and ~+i~. That gives a-stability of g=(ICc~)+(ICc~s)+(mCc~)+(mCc~s), and now go = g c~ (gc)~ = g c~ [c = I.

We prove (1). Let g * = t + i s and I * = ( I c ~ l ) + i ( I n s ) . If g is not absolutely simple, then g* ~ ~@l with ~ embedded diagonally and

0 interchanging the two summands by 0 (x, y) = (y, x). Let vj: g* ~ Ij denote projection to the j th summand. I* is in neither summand because 0I*= I*, so vj:I* ~_~1. That gives p~Aut (~) with I* = {(x,/~x):xe~). Notice that/~ 2 = 1 because I*--0I*= {(BY, Y):Y~) . Now ~= 1 x/~ is an automorphism of g* that interchanges I* and the diagonally embedded t.

I f g is absolutely simple and rank ~=rank ~, then g* is simple and its subalgebra ~ is a maximal subalgebra of maximal rank. The Borel-de Sieben- thal classification [1 ] shows that any two subalgebras of g* isomorphic to must be Aut(g*)-conjugate.

Now suppose g absolutely simple with rank t < rank ft. If g* is of classical type then the inclusion ~ ~ g* is one of

so (n) ~ stt (n), sp (n) ~ stt (2n), so (p) • so (q) ~ so (p + q).

Looking at degrees of representations we see that [ q g* and l* ~g* are equivalent representations. If ~/is the equivalence we take a=r / [ 0," If g* is of exceptional type then [ ~ g* is one of

s p ( 4 ) ~ e 6 or f 4 ~ e 6

and Dynkin's results [2] show that any two subalgebras of g* isomorphic to must be Aut(g*)-conjugate. Q.E.D.

384 JOSEPH A. WOLF

B I B L I O G R A P H Y

1. Borel, A. and de Siebenthal, J,: 'Les sous-groupes ferm6s de rang maximum des groupes de Lie dos', Comm. Math. Helv. 23 (1949), 200-221.

2. Dynkin, E. B.: 'Semisimple Subalgebras of Semisimple Lie Algebras', Math. Sb. 30 (72), 349--462; also in Am. Math. Soc. Transl. Set. 2, 6, 111-244.

3. KoHmyi, A. and Wolf, J. A.: 'Realization of Hermitian Symmetric Spaces as Gener- alized Half-Planes', Ann. Math. 81 (1965), 265-288.

4. Kostant, B.: 'On the Conjugacy of Real Cartan Subalgebras', Proc. Nat. Acad. Sci. U.S.A. 41 (1955), 967-970.

5. Nagano, T.: 'Transformation Groups on Compact Symmetric Spaces', Trans. Am. Math. Soc. 118 (1965), 428-453.

6. Richardson, R. W.: 'Principal Orbit Types for Real-Analytic Transformation Groups', Am. J. Math. 95 (1973), 193-203.

7. Takeuchi, M.: 'On Orbits in a Compact Hermitian Symmetric Space', Am. J. Math. 90 (1968), 657-680.

8. Wolf, J. A.: 'The Action of a Real Semisimple Group on a Complex Flag Manifold, I: Orbit Structure and Holomorphic Arc Components', Bull. Am. Math. Soc. 75 (1969), 1121-1237.

9. Wolf, J. A.: 'Fine Structure of Hermitian Symmetric Spaces', in Symmetric Spaces: Short Courses Presented at Washington University (ed. by Boothby and Weiss), Mareell Dekker, New York, 1972.

Author's address:

Joseph A. Wolf,

Universi ty o f California,

Berkeley,

Calif. 94720, U.S.A.

(Received May 3, 1974)