Proceedings of Symposia in Pure Mathematics Volume 37, 1980

REPRESENTATIONS IN CHARACTERISTIQ

LEONARD L. SCOTT’

We will be discussing here topics in three related areas: maximal subgroups, irreducible representations, and group cohomology.

1. Maximal subgroups. I will assume in this section that the finite simple groups can be classified and even that their irreducible representations in characteristicp can someday be determined. What I wish to demonstrate is that this will carry us a long way toward the determination of their maximal subgroups.

It should not come as a total surprise that such a problem is tractible, since the corresponding question for complex Lie algebras and connected Lie groups was solved some time ago by E. B. Dynkin [8]. Let us review Dynkin’s solution.

First of all, Dynkin treated the exceptional types separately, and I surely expect that the same will be necessary in the finite group case, requiring internal classification theory type arguments rather than representation theory. This reduced Dynkin essentially to studying the maximal subgroups of SL(,, C), O(n, C), and Sp(n, C). Next, he reduced to the case of an irreducible subgroup by simply noting that any irreducible subspace has an obvious stabilizer (when there is a form around, the irreducibility forces the subspace to be either nonsingular or totally isotropic). This had the pleasant by-product for Dynkin of also reducing to the case of a semisimple subgroup, since any connected abelian normal subgroup would have to act by scalar multiplications; in the finite groups case the maximal local subgroups associated with primes distinct from the characteristic would still have to be determined.

Next Dynkin reduced the problem from the semisimple to the simple’ case (i.e., the problem of finding all maximal connected subgroups which were irreducible and simple) by using the tensor product decomposition for irreduci- ble representations of a direct product. Any product of two or more terms would

1980 Mathematics Subject Classification. Primary 2OGO5; Secondary 20B15,2OB35,2OC20, 2OC30,

2OG10,2OJO6,2OG40, 18699, 14F05, 14L17.

‘Supported by the National Science Foundation. 2Read as “quasisimple”. Dynkin’s “simple” groups can have finite centers. In the finite case also

one needs irreducible representations for central extensions as well as for the simple groups

themselves. 0 American Mathematical Society 1980

319

320 L. L. SCOTT

have to be contained in SL(s) x X.(t) with s + I = n and 1 < s < n; the analogous subgroup for Sp(n) is Sp(s) x O(t) and for O(n) there are two possibilities: O(s) x O(t) and Sp(s) X Sp(t). This reduction goes through in spirit for the finite case, though the list of possibilities is larger-e.g., X(s) wr S,, 3s = n. (Incidentally, Dynkin checks as he goes along that each group in his list of obvious possibilities is indeed maximal.) Also one reduces not to a quasisim- ple group but to an automorphism group containing it.

Finally in the simple case Dynkin has available a complete classification of the simple groups in his category together with a complete determination of their irreducible representations. He also has available for each irreducible represen- tation the knowledge of whether such a representation has a symmetric or alternating bilinear invariant, or none at all (denoted + 1, -1, or 0 in his table). This settles the question as to whether our subgroup can be put in O(n) or Sp(n) or neither, which up until now had been hanging. The analogous information will certainly be required in the finite case (even more because of the two types of quadratic forms) and will just have to be considered part of the representation theory problem. The corresponding information for hermitian forms is also required, though this would at least follow from complete knowledge of the characters.

Having come this far, Dynkin now decides, in a brilliant application of the law of excluded middle, to decide which of his (theoretically classified) irreduci- ble simple groups is not maximal (in O(n), Sp(n), or SL(n)). This makes it possible to list at least some version of the results on a single page (see the table at the end of this paper). There are just four infinite families and fourteen individual exceptions. More detailed information on the exceptions is available in longer tables [8] I have not given here. For another exposition of Dynkin’s work, see Tits [%I.

Now let us assume that we have adapted Dynkin’s program to successfully find all maximal subgroups of the finite simple groups of Lie type and their containing automorphism groups, and somehow manage to treat the same problem for the finitely many sporadic groups. The question arises now as to what we do with the symmetric and alternating groups, perhaps the most interesting case of all. The answer, happily, is that we may almost be done at this point! I have listed the general forms for possible maximal subgroups in an appendix. As one might expect, the only undetermined possibilities (assuming a classification of the finite simple groups) involve a primitive embedding of an automorphism group of a simple group. If said simple group is not an altemat- ing group of lower degree, then we would know all of its primitive permutation representations at this point. Thus we would be in a good position to inductively determine all possible exceptions to maximality, completing our classification.

2. Irreducible representations. We will be discussing here representations of groups of Lie type in their natural characteristic p. Little has been done with representations in fields of characteristic I #p, though Alperin has suggested that this theory should parallel the modular theory of the symmetric groups, which has a substantial literature [23].

In characteristic p the irreducible representations all are restrictions of ra- tional (polynomial) representations of the ambient simply connected algebraic

REPRESENTATIONS IN CHARACTERISTIC p 321

group G over the algebraic closure k of GF(p) (Steinberg). To keep an example in mind, the group G is SL(n, k) if the original group was SL(,, p’), This is quite a serious class of examples, with the irreducible representations (or even their degrees) of SL(5,p) still unknown. (To my knowledge no one in the field has actually sat down and tried directly to construct representations. It would be a good problem (suggested originally by M. O’Nan) to try and find a general recipe for the irreducible representations of SL(,, 2) simply from a combina- torial point of view. Something like Young diagrams is what I have in mind.)

Very recently (just a few weeks before this conference) George Lusztig made the first serious conjecture regarding the characters of the irreducible representa- tions of G. To give the “character” of a representation of an algebraic group is to describe it on a maximal torus (the diagonal matrices in SL(n, k), in that example). The representation on such a torus T is a direct sum of l-dimensional representations or “weights”; from this point of view the “roots” are the representations associated with the root groups, positive roots corresponding to root groups in a fixed Bore1 subgroup B, e.g.,

)/I in ((.

The Weyl group W = No(T)/ T obviously acts on the weights and permutes the roots. Each irreducible representation has a unique highest weight with respect to the ordering A > p iff A - lo is a sum of positive roots. The abelian group of weights carries a natural positive definite symmetric bilinear form ( , ) and the high weight h of each irreducible representation is “dominant” in the sense that (A, C.X) > 0 for each positive root (Y. In this way the dominant weights completely parameterize or label the irreducible representations of G, even though little more is known about them. To describe Lusztig’s conjecture and its context we shall go back first to the classical theory of Weyl and Kostant which successfully describes the character of the irreducible module VA with high weight A when k is replaced by the complex numbers.

First of all, let p be any weight of T and regard p as a l-dimensional representation of B. The Lie algebra b of B correspondingly acts on CL, and we may consider the corresponding induced representation ZP = 91(e) Bwt,) p for the Lie algebra g of G. Here Q(g) denotes the universal enveloping algebra. The module ZP is also a rational T- (and even B-) module and is freely generated by

1 C3 p over a(n), the universal enveloping algebra for the group U- generated

by the negative root groups. In the example this group is

1

1 I- * 1

* * 1

Because of this, the multiplicity of any T-weight v in ZP is just p(v - CL) = the number of ways v - p can be written as a sum of positive roots. The module Z,, is called the Vet-ma module associated with CL. The formulas of Weyl and Kostant

322 L. L. SCOTT

describe V,, as a simple alternating sum

[ VA] = 2, (- l)““‘[ GA]

in the Grothendieck group of T-modules and thus give its character. Here w + A = w(A + p) - p where p is half the sum of the positive roots, and (- 1)““’ is the determinant of w in its action on the weights ( expressed in terms of the length function I(w) of reflection group theory). The proof of this result in Humphreys’ book [lo] shows quite clearly the importance of showing that the only Z,, in the same “block” as VA are of the form Z,,,, (Ha&h-Chandra)? (Each Z, is indecomposable, and even has a unique irreducible quotient module.) This gets one to the point where there must be at least some expression

[VA1 = ~.wEW c(w)[Z,,+]. Because of this, Humphreys and Verma pushed a corresponding block-theoretic investigation in the characteristic p case using the affine Weyl group W, (the semi-direct product of W withpZ@, the latter acting by translation on the weights, so that W, + A = We h + pZ@; here Z@ denotes the root lattice).

Recently [l] H. H. Andersen (sharpening results of Humphreys, Kac- Weisfeiler, and Jantzen) has proved that the high weights of any two irreducible representations of G in the same block must also belong to the same orbit under the affine Weyl group W,.” This implies that there must be at least some expression Z w E w c( w)[ Z,+,.,] for the irreducible representation in characteristic p (the simplest wa$ to make sense of this in characteristic p is to think of the character of Z,, in terms of the partition function p we discussed earlier). Essentially Lusztig’s conjecture gives the coefficients c(w) for the most im- portant weights when the prime p is large relative to the root system. I have given a precise statement in an appendix. Implicit in its philosophy are results of J. C. Jantzen [12] which allow one to obtain formulas for all weights from just a few well placed ones, also parameterized by elements of Wp.

Lusztig’s conjecture is analogous to an earlier conjecture [15] of Kazhdan and Lusztig in characteristic 0 regarding irreducible modules which are the quotients of Verma modules associated with nondominant weights. The main ingredients for the characteristic 0 (resp., characteristicp) conjecture are certain polynomials P ,,,,,, defined for each pair of elements w, w’ of the Weyl group (resp., affine Weyl group), the values of these polynomials at 1 giving the requisite doubly parameterized system of coefficients (the various c(w)‘s above). The polynomials P w,w, were apparently first found in the representation theory of generic Hecke algebras, arising naturally in lifting Springer’s Weyl group representations to these algebras. Since this conference took place, they have been shown to be Poincart polynomials for a new geometric cohomology theory of Goreski,

3T~o indecomposable modules are in the same block if they can be joined by a chain of

indecomposable modules with a nonzero homomorphism (either direction) between successive terms. ‘Recently S. Donkin has completed the determination of the blocks [Zs]. They are described as

orbits of WP or its analogues for higher powers of p, depending on the power (plus one) of p dividing h + p.

REPRESENTATIONS IN CHARACTERISTICp 323

MacPherson, and Deligne, applied to Schubert varieties. (See Lusztig’s article in these PROCEEDINGS, where he also notes that the characteristic p conjecture

implies the characteristic 0 conjecture, through a translation principle of Jant- Zen.)

Previously a Poincare polynomial interpretation in terms of group cohomol-

ogy had been given by David Vogan (cf. [2] and 83), but with coefficients involving the unknown irreducible modules, and assuming a conjecture he says is equivalent. In spite of these drawbacks Vogan’s interpretation does formally imply the characteristic 0 conjecture, and conceivably could be instrumental in its proof. I have described a characteristicp analogue in $3.

The theory is too young to say definitely where proofs might come from, so I will just survey some of the other approaches that have emerged so far. One really untried possibility I have already mentioned to you: directly construct representations. Another major avenue of attack is to decompose known non- irreducible representations. The main possibilities here are the Weyl modules, which may be described by a suitable reduction modp from a characteristic 0 module (k 8 z (Q u + in terms of the Kostant Z-form) and a number of results

in this direction have been obtained by Jantzen. Nowadays these modules may be described as the duals of certain O-dimensional cohomology groups (H’(B, -X @ R(G)) or H’(G/B, C(-X)), where R(G) is the affine coordinate ring of G and C( -A) is the line bundle on G/B associated with A), and H. Andersen has already demonstrated the usefulness of considering the higher dimensional cohomology groups. (They are used heavily in the proof of his result on blocks cited above.) Another simple description of the Weyl module associated with X is as the universal module with high weight A, the dual of the “induced” module - AJG = Morph,(G, -A) in the sense of algebraic groups. The equivalence of all these definitions depends on the vanishing theorem first

proved by George Kempf [16] and more recently by Andersen [2] and Haboush [27]. In any event the known results on the structure of these modules (as opposed to just knowing their composition factors) are very meager, the only complete results being for type A, (the group SL(2, k)), due to Carter and Cline, cf. [24]. (Carter and Cline actually give the lattice structure, though in general one might be content with well-understood filtrations.) To give the reader some appreciation of where the structure theory of these modules is today, I mention the following open problem:

Let G be SL(3, k) and V its standard 3-dimensional module, with V* the dual. Describe the structure of the tensor product Sm( V) C3 Sn( V*) of symmetric powers for all integers m, n > 0.

The problem is open even with a reasonable bound on m and n (say m + n + 2

<P2).

Still another approach is to look for modules in nature, meaning algebraic geometry. Aside from the higher cohomology groups mentioned above, the most interesting phenomena of this kind to my mind are the B-filtrations of line bundles arising in George Kempf’s study of Schubert varieties [la. One might hope to use these results in conjunction with the extension theory [q, which mostly reduces the question of constructing G-modules to B. The theory of [U]

324 L. L. scol-r

gives explicit constructions for bases of (duals of) Weyl modules, natural in terms of algebraic geometry, for all classical groups. For type A Carter and Lusztig [25] have used somewhat similar bases to obtain some partial results. Further material in this direction may be found in Towber [29] and James [23].

The Lie algebra of L(G) of G also plays some role in this theory; indeed, it was recognized very early that it was enough to construct the irreducible representations of L(G). (Here “representations” are meant in the sense of restricted Lie algebras.) One surprising connection with the group case is that

the projective indecomposable modules for L(G) lift to G modules for p large relative to the root system (unknown for small p). This was proved by Ballard [3]; the context here is a theory of Humphreys [ll], partly inspired by work of Jeyakumar. Recently Jantzen has shown for large primes that these modules are filtered by Weyl modules. The ultimate role of these modules is difficult to assess, though it is at least clear we want to know more about them.

The restricted enveloping algebra of the Lie algebra has higher order ana- logues, the “hyperalgebras”, introduced in the present context by Humphreys, developing suggestions of Verma. These have nice interpretations (cf. [S] for an exposition) in terms of the theory of infinitesimal groups. I will not say anything about this theory beyond the fact that if G, is the group scheme corresponding to a hyperalgebra (e.g., the restricted enveloping algebra of L(G)) then the category of modules for G, coincides with that of the hyperalgebra. The advantage of this point of view is that many group theoretic considerations suggest themselves when we think of G,., which is a normal subgroup scheme of G (for example, the Hochschild-Serre sequence, or the study of BG,-modules). Doubtless you thought G had no normal subgroups! Never fear, since the infinitesimal scheme G, has only one element. But in spite of this it is useful for representation theory.

I think on this note I will end this part of my exposition. There is one other approach, involving decomposing certain characteristic 0 representations of finite groups, which is discussed by Roger Carter in his lecture at this con- ference. The general role of characteristic 0 representations, aside from analogy, is not understood. For another instance where characteristic 0 representations enter, see Green [9] where it is shown that the (infinite dimensional) injective indecomposable modules for the algebraic group G can be “lifted” to character- istic 0.

Addenda. Another approach was found by this author during the conference, partly inspired by some remarks of Humphreys. It offers a well-defined pro- gram, but seems a bit slow in a time of such dramatic events. An unpublished’ result of mine asserts the injectives above have a filtration whose sections are induced modules -XI G. One has [22] ExtL(L,,, -AIG) = 0 for any irreducible module Lp whose high weight p does not satisfy - wop > A. Consequently we can form a submodule Zi of an indecomposable injective Q consisting of all “sections” - plG with p < A. Next one shows that

dim Extk( - hlG, Ii) = dim Extk(Soc( -hlG), Ii) = multiplicity of -hlG

5Au&d in proof. Recently S. Donkin has also found this filtration. His work will appear in Math.

Z.

REPRESENTATIONS IN CHARACTERISTIC p 325

as a section of Q. One shows also this is the characteristic 0 multiplicity, which is in turn the multiplicity of the irreducible socle of Q in - AIG as a composition factor (Green). Now it might be possible to inductively determine Extb( - AIG, Z.J, using the cocycles from one answer to construct the “next largest” Zi as a maximal essential extension.

3. Cohomology. I will try to give a brief overview of some problems close to my own interests and the theory of $2.

For algebraic group cohomology, it is important to study the structure of the induced modules -XIG (or equivalently their duals, the Weyl modules) and to study how the injectives are built from these (ef. the addenda to $2). The vanishing result on Exth(L,, -hlG) mentioned in the addenda is actually valid for all Ext”, n > 1, yielding a powerful dimension shift for computing algebraic group cohomology when the structure of the relevant Weyl modules is known.

Results of Cline, Parshall, Scott, van der Kallen [22] indicate how to compute finite Chevalley group cohomology in terms of algebraic group cohomology for

large fields. Recently Bill Dwyer, using “split buildings” constructed by Ruth Charney, has obtained some extremely promising stability results with respect to the rank of the group [7]. At the moment, his results are stated for GL,, and modules related to the standard module, but they should easily generalize. I leave the most definitive formulation as an open problem. Though there is still work to be done,6 it is now likely that entire families of finite group cohomology problems can be reduced to a finite number of cases by general methods.

Turning to David Vogan’s work on the Kazhdan-Lusztig conjecture in char- acteristic 0, we can express his Poincare polynomial interpretation as follows, in terms of algebraic group cohomology:

P,,,(q) = x qi dim Ext’$“)-‘cy)-2i( -y . h, L( - we A)) i>O

(*)

where y, w E W and A is any dominant weight. Here we agree that any Ext group of negative degree is 0. The starting point of Vogan’s investigation in [21] seems to be the observation that (*) implies the characteristic 0 conjecture through the application of an Euler characteristic formula. The analogue of the

latter in characteristicp is

[L(-we A)] = zh T (-l)“dimExti(-y.X,L(-w.A))[ V-,.x].

Here X is again any dominant weight and L( - w * A) is the irreducibile module with high weight - w * h, but w, y come from Wp, and we assume - we h, -y . A are both dominant. The formula is easily proved by appealing to the fact [22] that - plG 8 - v is B-acyclic for p, v dominant and replacing L( - we A) by an induced module.

6E.g., the stability results of [22] need to be treated for the twisted groups and one needs better stability theorems for growth of the characteristic p. The latter problem at least reduces, using

methods of [22], to algebraic group cohomology for a module twisted by the Frobenius endomor-

phism. Added in proof. The stability results of [22] have now been treated for twisted groups by G.

Avrunin, Trans. Amer. Math. Sot. (to appear).

326 L. L. SCOTT

PROPOSITION. Assume X is in the “bottom alcove” C, and - w * X is a dominant weight in the “bottom p2-alcove” C, (see the Appendix; these are just the

hypotheses of the Lusztig conjecture). Zf (*) hoI& for ally E Wp, then so does the formula for [L _ ,++,I conjectured by Lusztig.

This follows easily by just comparing coefficients. It would be interesting to know if other of Vogan’s results have analogues in

characteristicp. It would also be interesting to have some general calculations of B-cohomology. As far as I know, complete results on H”(B, CL) with p an arbitrary weight do not exist even for n = 2. Such calculations would also be extremely helpful for specific computations in the finite Chevalley group case mentioned earlier.

To complete this exposition, I would like to come back once more to maximal subgroups of finite groups. It is a theory of Bob Griess that interesting or “sporadic” 1-cocycles should lead to interesting subgroups by considering the elements of the group on which the cocycle is zero (the stabilizer of a vector in the usual extension module corresponding to the cocycle). This is supported by a number of theoretical results [19], [%I], but no one has yet made an attempt to

systematically look at examples. A good starting point would be

H’(SU(n, 2’) A3V) where V is the standard module, which Wayne Jones has shown to be l-dimensional for n > 7.7 The same cohomology for larger fields is 0, so that in some sense these cohomology groups are all sporadic. (The stable behavior with respect to the rank is an instance of what one should be able to prove by generalizing Dwyer’s results.) For some recent cohomology calcula-

tions, see [14] and [4]. I would like to thank H. Andersen, J. Humphreys and G. Lusztig for several

conversations and my colleagues Ed Cline and Brian Parshall for numerous contributions to this lecture.

Appendix: Statement of the Lwztig Conjecture (adapted from a lecture by H. Andersen). Let W be any Coxeter group and S its set of simple reflections. Let < denote the usual partial order on S in which y < w iffy has some reduced expression which is a subsequence of a reduced expression for w (well-defined, cf., Bourbaki). Let Z(w) denote the length of a reduced expression for w.

We are going to inductively define some polynomials P,,,, in q for y, w E W.* We will have P,,,, = 0 unless y < w, and that the degree in q of PY,” is at most i(l(w) - Z(y) - 1). Set ~(y, w) equal to the coefficient in PY,, of this largest

possible degree ~(Z(w) - Z(y) - 1) w en h the latter is a nonnegative integer. Define P,,, = 1 and PY,w = 0 if y 4 w. If y < w and P,,+, has been defined for ally with smaller w, choose s with ws < w. Put

‘The sporadic behavior actually occurs already for n = 6, where the same cohomology group is 24imensional. One regards V as n-dimensional, so that dim A’V = 20 in this case.

s1 am indebted to Roger Carter for catching an error in my original description of P,,,,Y and

apologize if any inaccuracies remain.

REPRESENTATIONS IN CHARACTERISTIC p 327

DEFINITION.

P,,,(q) = 4’-cp,,,ws + 4=py,ws

-q 2 p(z, ws)q(‘(ws)-‘(“)-‘)/2p,,=. y<z<ws

z.7 <r

In Kazhdan-Lusztig one finds the useful fact that y < w, SW < w, sy > y together imply P,+, = P,,,, (s E S; y, w E W). (We will just use this for giving an alternate form of the conjecture.)

Now suppose W is a Weyl group or affine Weyl group and L,, denotes the irreducible module with high weight y. (I also write L(p) sometimes.) The characteristic may be 0 orp. Regard W in its usual reflection action, centered at - p, where p = half the sum of the positive roots, except regard the elements of S as acting as reflections in the alcove containing -2p (rather than the “opposite” alcove containing 0). We will assume throughout in the characteristic p case that p is at least the Coxeter number, which amounts to saying there is at least some integral dominant weight interior to the lowest alcove C,. To be explicit C, = { ~1 p + S is dominant and (p, a”) <p for all positive roots a}. Our reflections S are acting through the walls of - C, in the characteristic p case. We shall also need the “lowest p2-alcove” C,, which is defined by replacing p byp’ in the definition of C,.

We can now simultaneously describe the Kazhdan-Lusztig conjecture in

characteristic 0 and Lusztig’s adaptation for characteristic p. Let X be a domi- nant weight and w E W (or W, in characteristic p). If the characteristic is p, assume X E C, and that - w * A is a dominant weight in C,. The conjecture is

[LA] = y’) ‘(w)-~ti)PY,w(l)[ z+].

In characteristicp this can also be written in the alternate form

[LA] = -y,h~~ant(-l)‘(w)-~opy.v(l)[ v-,.A].

Formulas for all the other weights follow from work of Jantzen: a weight p is in the upper closure of an alcove - w * C, if no reflection in the walls of that alcove - closer to the origin -p. Every weight is in the upper closure of some alcove. Jantzen’s theorem tells us the above formulas hold without change if we just replace X by - w - ’ . p. Thus for largep the conjecture gives formulas for all the irreducible modules in the restricted range from which all others are constructed via the tensor product theorem of Steinberg. The situation for small p is unclear. According to an example of Jantzen, one cannot just use the same formula for smallp. (The example is G, in characteristic 3.)

Appendix on maximal subgroups. Call a subgroup H of a direct product II,,, Gi diagonal if each projectionpi: Ilie Gi + Gj is injective on H. If each pi is in fact an isomorphism (which means in particular that all Gj are isomorphic) we shall call H a fuZZ diagonal subgroup.

-

328 L.L.SCOTT

LEMMA. Let H be a subgroup of a finite direct product IIiEl Gi of nonubelian simple groups, and assume each projection pi is subjective on H. Then H is the direct product n Hi of full diagonal subgroups of subproducts II,,$ Gi where the 4 form a partition of I.

PROOF. Certainly we may assume H # 1, in which case there is a minimal subset S G I with H n IIiEs Gi # 1. Set D = H n II,,s Gi and note that D is normal in H, hence any projection of D is normal in any projection of H. Clearly, pi(D) # 1 for j E S by minimality, so pi(D) = Gj. Minimality also impliespi is injective on D. Thus D is a full diagonal subgroup of IIj,s Gj.

Now let E be the projection of H on IIj,, Gi. Then D 4 E as noted above, since D is its own projection on this subproduct. But obviously a full diagonal subgroup of a product of nonabelian simple groups is self-normalizing. Thus D = E and H is the direct product D X D’ where D’ = H n II,,,-s Gi. The result now follows by induction on the cardinality of the index set I.

COROLLARY. Let H be as in the previous lemma. Then every subgroup K of II,,, Gi containing H has the form

K = n ps(H) SE9

where 9 is a partition of I refining the partition we associated with H, andp, is the

projection of G on IliEs Gi.

PROOF. By the lemma applied to K we have K = n, EJps( K) for some partition ‘3 of I, with p,(K) full diagonal in IIics Gi. Since H C K we have ps(H) c ps(K). Alsop, = &or. psn,.(H) where {Ii} is the partition associ- ated with H. Since this product co&ins a full diagonal subgroup of IIiEs Gi, it

must consist of only one term and coincide withp,(K). Q.E.D. We can use the above results to get a very useful picture of the general

primitive finite permutation group. Recall that such a group has a socle which is either elementary abelian (an irreducible representation for the point stabilizer) or a direct product of isomorphic nonabelian simple groups.’ Any nontrivial normal subgroup of a primitive group of course acts transitively.

THEOREM. Let H be a subgroup of a finite direct product G = IIiEl Gi of isomorphic nonabelian simple groups. Then the transitive permutation representa- tion of G on G/H extencis to a primitive permutation representation of some group in which G is the socle if and onIy if either

(a) there is a partition 9 of I into subsets of equal prime cardinali@ with H the direct product n s E9 As of full diagonal subgroups As of the subproducts n,,s Gi, or

(b) the subgroup H is a dir-et product II,=, Hi where Hi ,+ a subgroup of Gi which Q an intersection Gi n Hi for some maximal subgroup Hi of a group ei with Gi c Gi c Aut Gi. Also, for each pair i, j of indices there must be an isomorphism of Gi with Gj carving Hi to Hi.

9The only case of a nonunique minimal normal subgroup N occurs when N is regular and the socle is N X N (nonabelian).

REPRESENTATIONS IN CHARACTERISTIC p 329

The details of the proof are fairly straightforward from the previous results, and we leave them to the reader. We now aim at describing the possible maximal subgroups of the symmetric and alternating groups.

We recall that if A is an abstract group and B is a group acting on a set P,

then the wreath product A wr B is the semidirect product (TI, A) - B obtained from the action of B given by fb(y) = f(yb-‘) for f E II, A (functions from I7 to A), b E B, and y E I?. If A acts on a set !2, the usual set one has the wreath product act on is the disjoint union 2, !2 of III’1 copies of 0. However, there is another natural action, namely, on the product II, 9. This action is given by

( &xY) = dYP9 gb(u> = g( Y b-‘)

for g E II, Q2, f E &.A, y E I’, b E B. We shall call this the product action of the

wreath product.

PROPOSITION. Zf A, B are transitive in their respective actions, and A acts primitively but not just as a regular group of prime order, then A wr B is primitive in the product action.

PROOF. Suppose g, g’ are distinct elements of a block A of imprimitivity. Let

y E P be a coordinate where g and g’ differ. Applying the stabilizer of g(y) in the yth copy of A to g’ yields at least one more element g” in A agreeing with g’ everywhere but at y. Primitivity of A now implies A contains all such g”. Transitivity of B now forces A to be all of Iir G. Thus A wr B is primitive. Q.E.D.

THEOREM.” Let M P 62 be a subgroup of the symmetric group S,,. Then some conjugate of M is contained in one of the subgroups listed below. Here 1 < m < n and p is prime.

(a) S,,, X S,, m + k = n (intransitive), (b) S, wr S,, mk = n (imprimitive), (c) S,,, wr S,, mk = n, m > 5 (product action), (d) I’- GL(V),p” = n = 1 V(; Vu vector space over GF(p), (e) (G wr S’) Out G, 1 GIP-’ = n, G a nonabelian simpIe group, (f) an automorphism group of a nonabelian simple group G < a”.,, containing G

and acting primitively (the full normalizer of G in S,,).

The group in (e) is the extension of G wr S, by the outer automorphism group

Out G obtained from the natural extension Aut G of a diagonal copy of G; the isotropy group is Aut G x S,.

The proof is easy from the preceding results: if M is intransitive, imprimitive,

or primitive but local we have cases (a), (b), or (d). Otherwise M is primitive and its socle is a direct product of isomorphic nonabelian simple groups. Case (b) of the previous theorem leads to cases (c) and (f) here, while case (a) there yields either (e) or (c) again.

‘@This theorem has been independently obtained by Mike O’Nan.

330 L. L. SCOTT

Type of the group 0

%Jl n>3

%)2 n22

(B Zn+l k ) nzl k>l

%)k k>l

%)

w3

$1

(C3)1

tc3)2

(4)

@)a)1

@a)2

E6)l

@6)2

@7)1

(E7)2

@7)3

@I)4

Table 1”

Scheme of 8

k o---o--o...-

CT&

6 0

I 1

N

3 q2)

3 (n+3) 4

In (k+y-z)

II s=l (k+s

k )

2k+S k+4) s( 4

7

189

128

90

350

560

495

4928

351

17550

1539

27664

365 750

3 792 0%

0

0

(-I)@+1 )k

I

1

1

1

1

-1

0

1

0

0

0

1

-1

1

-1

“Reprinted from E. B. Dynkin, Maximal subgroups of the ckssical groqs, Amer. Math. See. Transl. Ser. 2 6 (1957), p. 364.

REPRESENTATIONS IN CHARACTERISTIC p 331

BIBLIOGRAPHY

1. H. H. Andersen, The strong linkageprinciple, J. Reine Angew. Math. 315 (1980), 53-59.

2. -> The Frobenius morphism on the cohomologv of homogeneous actor burudles on G/B,

preprint (Aarhus). 3. J. Ballard, Injectiw modules for restricted mueloping algebrcls, Math. Z. 163 (1978), 57-63. 4. G. Bell, On the cohomology of the finite special linear grotqs. I, II, J. Algebra 54 (1978),

216-238,239-259. 5. E. Cline, B. Parshall and L. Scott, Cohomology, hyperalgebrar and representations, J. Algebra 63

(1980), 93-123. 6. -, Induced modules and extensions of representations. I, II, Invent. Math. 47 (1978),

41-51; J. London Math. Sot. (2) 20 (1979), 403-414. 7. W. Dwyer, Twisted homological stability for general linear groups,, Ann. of Math. 111 (1980),

239-251.

8. E. B. Dynkin, The maximal subgroups of the classical groups, Amer. Math. Sot. Transl. Ser. 2 6 (1957), 245-378.

9. J. A. Green, Local&finite representations, J. Algebra 41 (1976), 137-171. 10. J. E. Hum&eys, Introduction to Lie algebras and rqwesentation theory, Grad. Texts in Math.,

vol. 9, Springer-Verlag, Berlin and New York, 1972. 11 . -7 OraVnaty ana’ modular representations of Chew&y groups, Lecture Notes in Math., vol.

528, Springer-Verlag, Berlin and New York, 1976. 12. J. C. Jantzen, 2% Charakterformel gewisser Darstelhmgen halbeinfacher Gruppen und Lie-Alge-

bren, Math. Z. 140 (1974), 127-149.

13. -, Darstelhmgen halbeinfacher Gryppen und ihren Frobenius Keme, preprint (Bonn). 14. W. Jones and B. Parshall, On the I-cohomology of finite groqs of Lie type, Proc. Ccrnf. on

Finite Groups (Park City, 1975), Academic Press, New York, 1976. 15. D. Kazhdan and G. Lusztig, Representations of Coxeter groups and Hecke algebras, Invent.

Math. 53 (1979), 165-184.

16. G. Kempf, Linear systems on homogeneous spaces, Ann. of Math. (2) 103 (1976), 557-591. The Grothendieck-Cousin complex of an induced representation, Advances in Math. 29

(l&)~b6.

18. V. Lakshmibai, C. Musili and C. S. Seshadri, Geometry of G/B, Bull. Amer. Math. Sot. (N.S.) 1(1979), 432-433.

19. L. Scott, Permutation modules and I-co/aomology, Arch. Math. 27 (1976), 362-368.

20. -, Matrices andcohomology, Ann. of Math. (2) 105 (1977), 473-492.

21. D. Vogan, The Kazhaan-Lwztig conjectures, preprint (MIT). 22. E. Cline, B. Parshall, L. Scott and W. van der Kallen, Rational and generic cohomology, Invent.

Math. 39 (1977), 143-163.

23. G. James, The representation theory of the ymmetric grotq, Lecture Notes in Math., vol. 682, Springer-Verlag, Berlin and New York, 1978.

24. E. Cline, A second look at the Weyl modules for S&, preprint (Clark). 25. R. Carter and G. Lusztig, Modular representations of finite groups of Lie type, Proc. London

Math. Sot. 32 (1976), 347-384. 26. J. Tits, Sous-algebres des algebres de Lie semi-simple, Sdminaire Bourbaki 1954/1955, expod

119, Paris, 1959. 27. W. Haboush, A short proof of the Kempf vanishing theorem, Invent. Math. 56 (1980), 109-l 12. 28. S. Donkin, The blocks of a semisimple algebraic groqn, preprint (Warwick). 29. J. Towber, Young symmetry, the flag manifold, and representations of GL(n), J. Algebra 61

(1979), 414-462.

U~~v~asrrv OF VIRGINIA