Arch. Math., Vol. 57, 417-423 (1991) 0003-889X/91/5705-0417 $ 2.90/0 �9 1991 Birkh~user Verlag, Basel

Permutation polynomials and primitive permutation groups

By

STEPSEN D. COHEN

1. Introduction. We present here progress that has been made through the application of the theory of primitive permutation groups to the conjecture of L. Carlitz (1966) on the non-existence of permutation polynomials of even degree over a finite field Fr of odd order q = p~. Featured recently as unsolved problem P9 in [9], the conjecture can be stated as follows.

C o nj e c t u r e C.. Given an even positive integer n, there exists a constant c. such that, ifq is odd and q > c,, there does not exist a permutation polynomial of degree n over

The weaker version of C, that has the added condition p ~e n was proved by Hayes [8] but polynomials whose degree is divisible by p are mucli more difficult to handle. Here, as far as published results ha this direction are concerned, C, has been established for all even n < 16, the significant contributions being that of Hayes himself who proved Clo and, twenty years later, that of Wan [14] who showed that Clz and C14 are true.

Nevertheless, very recently, in response to a preprint of the present paper, Daqing Wan has kindly sent me a copy of his (as yet) unpublished note [15] in which C, is shown for n = 2 r, r an odd prime.

As in the work mentioned above, the conjecture of Carlitz will be attacked by consid- ering the class of exceptiona! polynomials whose definition we now recall. In it, and throughout, we shall assume, without loss, that a polynomial f (y) in Fq[y] is separable, i.e. is not of the form f l (YP), where 7"1 (Y)~ Fq[y] (for otherwise, we could deal with f~ instead). Further we shall let q~s(x, y) denote the two variable polynomial ( f ( y ) - f ( x ) ) / ( y - x). Then a separable polynomial f ( y ) of degree n > 1 in Fq[y] is called exceptional over Fq if every irreducible factor of q~y (x, y) in F~ [x, y] is not absolutely irreducible, which means that it becomes reducible in Fq [x, y], where Fq denotes the algebraic closure of Fq. The merit of introducing exceptional polynomials is, of course, the fact that for each even n, Conjecture C, is implied by the following statement.

E,. There are no exceptional polynomials of degree n over F~ for any odd q.

We shall prove that the set of even integers n for which E, (and hence C,) is true is infinite and includes all small values by establishing the following theorem.

Archiv der Mathematik 57 27

418 S.D. Cox-ms ARCH. MATH.

Theorem. Suppose n is even. Then E , and hence C, hold whenever either (i) n < 1000, or

(ii) n = 2 p l , Pl a prime.

Apart from the result itself, the hope is that our proof of the Theorem will serve to indicate the value of the approach by means of primitive permutation group theory. While I have not at this stage managed to settle completely the conjecture of Carlitz by these means, I am aware of other constraints they impose on the nature of exceptional polynomials and would expect them to yield more eventually. We remark that Wan's independent proof of part (ii) of the Theorem is by different methods, geometric rather than algebraic in flavour.

The link between polynomials and permutation groups has already been exploited in the treatment of other problems (see [7] for example) but to end this initial section we give an outline relevant to the present context and introduce some notation.

For a separable polynomial f over a field F denote by G (f~, F) the Galois group of f (y) - z over F (z), z an indeterminate, regarded as a transitive group of permutations on (2 I , the set of zeros of f (y) - z. In particular, given the finite field Fq, abbreviate G (fz, Fq) to G = G ( f ) and G(fz , Fq) to G = G(f ) . As far as possible we reserve x for a member of t2 I , in which sense the stabilizer G x (fz, F) of x in G (fz, F) can be interpreted as the Galois group of ~os(x, y) over F(x) acting on (2i\{x }. We note that evidently G ( f ) is normal in G ( f ) and G/G ~- Gx/CJ x ~- Gal F~,/Fq (for some t), a cyclic group of order t. Finally, we highlight the facts (to be amplified later) that f , regarded as a composition of polynomials over F, is actually indecomposable precisely when G (fz, F) is a primitive group on f~s and that in any discussion of E, attention may be restricted to indecompos- able polynomials.

2. Permutation polynomials and exceptional polynomials. For the first two lemmas we refer to [10], w and [14] w and the references given there. All polynomials are assumed to be separable.

Lemma 2.1. Every exceptional polynomial over Fq is a permutation polynomial of Fq.

Lemma 2.2. There exists a sequence of positive numbers c2, c3,. . , such that if f is a permutation polynomial of Fq of degree n(> 2) and q > c,, then f is an exceptional polynomial over F~.

Corollary 2.3. For any even integer n, statement E, implies Conjecture C,.

It is very easy to see that, if f = g (h) is a composition of polynomials with 9 (Y), h (y) in Fq [y], then f is a permutation polynomial of Fq if and only if both 9 and h are. We claim that the analogous statement for exceptional polynomials over Fq is valid. In the first place, since q~h divides q~I, it is evident that, if f = O (h) is exceptional over Fq, then so is h. To justify the rest, we employ the criterion for a polynomial to be exceptional taken from Lemma 6 of [2].

Lemma 2.4. A separable polynomial f(y) in •q[y] is exceptional over Fq if and only if any a in G = G( f ) such that (7G generates G/G actually lies in G x for some x in f2f.

Lemma 2.5. A composition f = g (h), where g ( y), h (y) e Fq [y], is exceptional over ~:q if and only if both g and h are exceptional over Fq.

P r o o f. Let G O = G (g), G O = G(9) acting on f2g = {h (x), x E t2y }. We find easily that the map 0 : GIG -~ Go/G o determined by O(crG) = a o Go, where (70 is the restriction of tz (as an automorphism

Vol. 57, 1991 Permutation polynomials and groups 419

of Fq (f2:)) to F 4 (O~), is a well-defined homomorphism of cyclic groups. Of course, any extension a in G of a given cr o in G O satisfies 0 (a G) = a o Go; hence 0 is an epimorphism. Thus, if GIG and Go/G o are cyclic of orders t, t o, respectively (where to I t) and ~rG is one generator of G/G, then the set of generators of GIG is {a'G, 1 < i < t, (i, t) = 1} and that of Go/Go is {cr~o Go, 1 < i < t o, (i, to) = 1), o o being the restriction of a as before.

Now suppose f is exceptional over Fq and that 0 o ~ G O is such that a o G O generates Go/G o. By the above, there is an extension a of 0 o to G for which cr G generates GIG. From Lemma 2.4, ~r fixes x (say) in O: and accordingly a (and so 0o) fixes h (x) in f2 o. Another application of Lemma 2.4 yields the conclusion that O is exceptional over Fq.

Conversely, suppose that both 9 and h are exceptional over Fq and that ~rG generates GIG. Then ao Go generates Go/UX o. Because g is exceptional, a fixes u in f2g. It follows that a ~ G(h,,, Fq) = G 1, say, and indeed that aG~ generates G~/Ga. We deduce from the fact that h is exceptional, that cr fixes x where h(x) = u and so f ( x ) = z. This completes the proof.

Corollary 2.6. For any even integer n, E, holds whenever statement E', holds, where E', is as follows. E',: There are no indecomposable exceptional polynomials of degree an even divisor of n over ~:q for

any odd q.

P r o o f. If f = g (h) has even degree, then either g or h has even degree and the result follows from Lemma 2.5.

F o r the r e m a i n d e r of this section, suppose f is an excep t iona l p o l y n o m i a l of even degree n = p~ m, where m is even a n d no t divis ible by p (odd) and s > 1. The inves t iga t ions of [8] and [14] were based on the fact t ha t the h o m o g e n e o u s pa r t of q~: of highest degree is divis ible by y - x to the precise p o w e r p~ - 1 and y + x to the p o w e r p~. Moreove r , an i r reduc ib le fac tor ~ol (x, y) of ~0: (x, y) in Fq [x, y] is such tha t in the h o m o g e n e o u s pa r t s of all its i r reduc ib le factors in ~ [x, y], y - x and, s imilar ly, y + x occur to the same power . Fu r the r , by e l emen ta ry p roper t i e s of finite fields, q~l factor izes as a p r o d u c t of t~ irre- ducib le p o l y n o m i a l s in F~ [x, y] of the same degree, where t~ ( > 1) divides t and F~, is the explici t a lgebra ic c losure of Fq in Fq (f2:). Re ta in ing this last n o t a t i o n we summar i ze some consequences .

Lemma 2.7. Suppose f is an exceptional polynomial over Fq o f even degree and the factorization o f ~pf into irreducible polynomials over Fq is given by

~p: (x, y) = ~p 1 (x, y) " " ~o k (x, y),

where deg q~i = dl, i = 1 . . . . k. Then for each i = 1 , . . . , k, there exist integers e~, ti with tl ( > 1) It such that d i = e i t i and the highest common factor (t 1 . . . . , tk) = 1. In particular

O) k > 1, i.e. ~p: is reducible over ~Zq (ii) t is not a prime power.

P r o o f. F o r each i, q~ spli ts over Fq as a p r o d u c t of (say) t~(> 1) i r reduc ib le p o l y n o - mia l s of degree (say) e~. By the p reced ing remarks , (tl . . . . , tk) divides each of pS and pS _ 1 and so mus t be 1. Then (i) is i m m e d i a t e and (ii) fol lows f rom (i).

3. Polynomials and permutation groups. The co r r e spondence be tween i n d e c o m p o s a b l e p o l y n o m i a l s and pr imi t ive g roups was obse rved by F r i e d [5] bu t is of sufficient i m p o r - tance to w a r r a n t a qu ick p r o o f here.

27*

420 S .D . COHEN ARCH. MATH.

Lemma 3.1. Let f be a separable polynomial over a field F. Then f is indecomposable over F if and only if G = G (fz, F) is primitive on f2 I.

P r o o f. Suppose G is imprimitive. Then by [16], Theorem 8.2, for any x in f2 I, Gx is not a maximal subgroup of G. Hence there exists a field strictly between F (z) and F (x), necessarily of the form F (u) (by Liiroth's theorem). Thus u = h (x) and z = g (u) for rational functions g, h (not linear fractional transformations) over E Because f = g (h) is a polynomial, it is easily arranged that g and h are also (non-linear) polynomials over F. Thus f is decomposable over F. Conversely, if f = g (h) is decomposable, it is evident that, for any x in f2 s, the subset {x 1 ~ f2i: h(xl) = h(x)} is a set of imprimitivity for G. This completes the proof.

Continuing with G (f~, F), we can interpret via Galois theory, properties of G~ acting on f2i\(x } (as described in [16] Chapter 3 and [13] Chapter 3) in terms of the irreducible factors of ~o s (x, y) in F [x, y]. There is an obvious correspondence between the orbits of G~ and these factors in which the length of an orbit (its subdegree) agrees with the degree of the associated irreducible factor. (We note, incidentally, that the polynomials corre- sponding to "paired orbits" simply have x and y interchanged.) The rank of G is the total number of orbits + 1 (for the trivial orbit (x} when G~ acts on f2i). In particular, G has rank 2 if and only if G is doubly transitive, or, equivalently, ~01 is irreducible over F. When f is indecomposable much more can be said because then G is primitive. Sample conse- quences of results in [16], w w [13] w include the following.

Lemma 3.2. Suppose that f is a separable indecomposable polynomial over F and ~os (x, y) = tp I (x, y) "" q~k(X, y), where for each i = 1 . . . . , k, q)i(x, y) is an irreducible poly- nomial of degree di in F [x, y] with I < d 1 < d E < . . . ~ d k. Then, for each i = 1 . . . . . k, the following hold.

(i) di < d l d i - 1 ( i>1) . (ii) (di, dk) ~= 1.

(iii) None of the prime factors of di exceed dl. (iv) Every composition factor of Gx is also a composition factor of a subgroup of G~, the

Galois group of ~oi(x, y) over F (x) acting on the zeros of r In particular, IGxl and I Gel possess the same prime factors, etc.

Naturally, we apply Lemma 3.2 when F = Fq and f is indecomposable over F~. Indeed, provided such an f remains indecomposable over Fq, it also describes G( f ) and the factorization of rp s over Fq. It could be that this is always the case but, as far as I know, it has not been generally proved that, if f ( y ) in Fq [y] is decomposable over IF~, then it is decomposable over F~ (see, for instance [6], Example 2, for an illustration of a polyno- mial f (y) in Fq[y] with non-trivial alternative decompositions f = 9 (h) = 91 (hi), where g (y), h (y) ~ Fq [y], gl (Y), hi (y) ~ Fq [y]\F~ [y]). As a final comment on Lemma 3.2, com- pare the data it yields with that from Lemma 2.7 when f is an exceptional polynomial of even degree.

Staying with the hypothesis that f is an indecomposable exceptional polynomial of even degree over F~, we proceed to some remarks about the socle of G ( f ) (denoted by soc G) which is the subgroup of G generated by all the minimal normal subgroups of the

Vol. 57, 1991 Permutation polynomials and groups 421

primitive group G. It is a s tandard result ([11], pp. 168-169) that soc G is a characteristic subgroup of G and is a direct product of simple groups, a much fuller description being provided by the Scot t -O 'Nan theorem, see [1]. We note, however, that soc G is not necessarily primitive.

Lemma 3.3. Suppose f is an indecomposable exceptional polynomial of even degree n over Fq (q odd). Then soc G is a non-abelian subgroup of G.

P r o o f. Suppose soc G d: ~ and that H is a minimal normal subgroup for which H d: G. Since Gc~ H is normal in G then H c~ G = {1}.We deduce from the first isomor- phism theorem o f group theory that H ~- HG/G and so is cyclic because G/G is cyclic. By [11] Proposi t ion 4.9 we may conclude that H = soc G and, more significantly, that n is a prime power. This yields a contradiction because the even integer n is also divisible by p which is odd. Hence soc G ___ G. Again the fact that n is not a prime power disallows soc G from being abelian (by the same result). The proof is now complete.

4. Exceptional polynomials of even degree < 1000. There are none as we show with the aid of the complete list of primitive groups of degree < 1000 compiled by Dixon and Mort imer [4] partly, at least, on the basis of the classification of simple groups. All primitive groups G (on s say) within this range with non-abelian socle lie in some cohort, each member of which has the same socle H (with normalizer N in S~) and H _~ G _ N. The cohorts themselves are distinguished (by the nature of their socles) into types; types A - H comprise those with simple socles, while the socles of types I, J are direct products of r (_-> 2) isomorphic simple groups T, where r = 2 or 3 whenever n is even. (Of course, we are interested only in examples with even degree.)

Let f be an indecomposable exceptional polynomial of even degree n < 1000 over Fq (q odd). By Lemma 3.3 and the above we have

H = s o c G ~ G ~ _ G ~ _ N .

By Lemma 2.7 the order of the cyclic group G/G is not a prime power. Accordingly G cannot belong to a cohort for which N/H is (trivial or) a p0-group for any prime Po. This eliminates from consideration all but ten cohorts with simple socle, namely those with degrees (i n order of appearance in Table 2) 520, 378, 730, 344 (type B); 280, 960, 336, 456 (type C); 126, 672 (type D). All cohorts with composite socle are excluded on the same grounds for the reasons we now sketch. By Lemma 5 of [4], the socles of type I have H = T x T or T x T x T and N = M wr St, r = 2 or 3 and M, the normalizer of T, is an extension of T by its outer automorphisms. Similarly, for type J Lemma 6 of [4] yields a description of N as an extension of H by the outer automorphisms of T. For the groups in question, T i s one of the simple groups Ak(5 < k < 9), L2(Q) (Q = 7, 8, 11), $6(2 ) or a Mathieu group; thus the outer au tomorphism group has order at most 3. Hence N/H, if non-trivial, is a 2-group or a 3-group.

We now briefly dispose of the ten remaining cohorts. When n = 730, 344, 126 the rank of N is 2 - indeed H is one of the familiar doubly transitive groups L 2 (36), L2 (73) or U3 (5). Hence ~py is irreducible (even over F~). If n = 456 or 672, N/H = S 3 yielding 2 or 3 as the only possible orders for the cyclic groups G/G. This contradicts Lemma 2.7(ii). In the

422 S.D. COHEN ARCH. MATH.

r ema in ing cases N / H is the cyclic group of order 6 or the d ihedra l g roup of order 12. Nevertheless, for each, there is an i rreducible c o m p o n e n t of the pe rm u ta t i on representa- t ion of N whose restr ict ion to H is also i rreducible which implies the existence of absolutely i rreducible factors of r of degrees 9, 26, 9, 7, 5 when n = 520, 378, 280, 960, 336, respectively. Thus E'~ is val idated for all even n < 1000 and par t (i) of the Theo rem follows.

5. Exceptional polynomials of degree 2p. We proceed to par t (ii) of the Theorem in which we m a y assume n = 2 p, the odd pr ime p being the characterist ic of Fo. The key is an appl ica t ion of a celebrated theorem on the r ank of a pr imit ive p e r m u t a t i o n g roup of degree 2 p to G, once the lat ter g roup has been shown to be primitive. F o r the first stage of the plan, we quote some consequences of the results of [3].

Lemma 5.1 ([3], Theorem 1.2). Suppose that the separable polynomial f (y) of (arbitrary) degree n in Fq[y] is such that q~y(x, y) has a linear factor L(x, y) in F~ [x, y]. Then f = 9(h), where g(y), h (y) ~ F~ [x, y] and L(x, y) divides tph (x, y), the latter being a product of linear factors in F~[x, y].

Lemma 5.2 ([3]), Theorem 1.1). Suppose that the separable polynomial f (y) of degree 2 p in Fq [y] is such that qgy factorizes into a product of linear factors in Fq Ix, y]. Then

f ( y ) = ~ ( y p + fly)2 + ~, ct, f l , ~ F q .

Lemma 5.3 ([3], Lemma 2.3). The following statement holds for monic separable polynomials h~ (y), i = O, 1, 2, 3, in Fq [y]. Suppose that P (x, y) in Fq [x, y] divides 9ho (x, y) for some h o. Then there exists h 1 such that P(x, y) divides ~Oh2(X, y) if and only if h 2 = h3(hl) for some h 3.

Lemma 5.4. Suppose that f is a separable indecomposable polynomial of degree 2 p over Fq. Then f is indecomposable over Fq.

P r o o f. Suppose that f decomposes non-trivially over ~q as f = g (h). It is easily seen that h can be assumed to be monic with h (0) = 0. Also h must have degree 2 or p and, of course, 9h divides ~0 I (in ~q Ix, y]).

First, let deg h = 2. Then q~h is linear and we deduce from Lemma 5.1 that r itself is a product of linear factors in ~q [x, y] (because f is indecomposable over Fq). But then Lemma 5.2 operates to force the contradiction that f is decomposable over F~.

We may therefore assume that deg h = p. Writing h ~ for the polynomial over Fq for which h'(y) = (h(yl/~)) q, etc., we observe that, since f~ = f, then q~y is divisible also by ~Dh~. Let q~h and q~h~ have a common factor P(x, y). From Lemma 5.3 we obtain the existence of (monic) hi, hz, h 3 over Fq such that P (x, y) divides ~Ph~ and h --- h 2 (ha) , h ~ = h 3 (hi). If P (and hence h~) is non-constant, it follows that deg hx = deg h --- p. Indeed we may take h = h~ and conclude h"(y) = c~h(y) + ~, for some ~,/~ in Fq. However, the assumptions that h is monic and h (0) = 0 mean that, in fact, h" = h or, equivalently, h(y)~Fq[y] contradicting our hy_pothesis. The outcome is that q~h and q% are co-prime divisors of q~I, where for some L(x, y) in Fq [x, y]

~(x , y) = ~(x , y) ~o(x, y)L(x, y).

Here since deg q~y = 2 p - 1 and deg q~k = deg q~hr = p - 1. L (x, y) must be a linear polynomial. We may therefore apply Lemmas 5.1 and 5.2 exactly as before to obtain a contradiction and so reach the desired goal.

We m a y now quickly complete the proof of our Theorem. Let f be a n except ional po lynomia l of degree 2 p over Fq which we ma y assume to be indecomposab le over F~ and even over Fq, by Coro l la ry 2.6 and L e m m a 5.4. Then, on the one hand , because f is

Vol. 57, 1991 Permutation polynomials and groups 423

exceptional, certainly q~: is reducible over Fq possessing at least two irreducible factors of the same degree. On the other hand, because G ( f ) is primitive, Theorem 31.2 of [16] permits only two possibilities for the factorization of ~o: over Fq, each of which conflicts with the above. The first is that G has rank 2, i.e. q~: is irreducible over Fq. Alternatively, 2p = (2 s + 1) z + 1 for some integer s, G has rank 3 and q~: is a product of two irreducible factors over Fq of (different) degrees s(2s + 1), (s + 1) (2s + 1). Thus no exceptional polynomial of degree 2 p can exist and the Theorem is proved.

It remains to make a few concluding remarks. In the search for (indecomposable) exceptional polynomials f of even degree over fields of odd order, it can be assumed from [3] that ~0: has no linear factors over Fq. Indeed, from a more general study of exceptional polynomials being undertaken by the author, it is likely that neither can ~0: have factors of degree 2 or 3 over F~. Moreover, for all the exceptional polynomials (of any degree) I know to exist, soc G ( f ) is abelian and it would be interesting to discover whether there are any examples with non-abelian socle.

References

[1] M. ASCHBACrlER and L. SCOTT, Maximal subgroups of finite groups. J. Algebra 92, 44-80 (1985).

[2] S. D. COHEN, The distribution of polynomials over finite fields. Acta Arith. 17, 255-271 (1970). [3] S.D. COEN, The factorable core of polynomials over finite fields. J. Austral. Math. Soc.

Ser. A, 49, 309-318 (1990). [4] J.D. DIXON and B. MORTIMER, The primitive permutation groups of degree less than 1000.

Math Proc. Cambridge Philos. Soc. 103, 213-238 (1988). [5] M. FRmo, On a conjecture of Schur. Michigan Math. J. 17, 41-55 (1970). [6] M. FRIED, Arithmetical properties of function fields II. The generalized Schur problem. Acta

Arith. 25, 225-258 (1974). [7] M. FRIED, Exposition on an arithmetic-group theoretic connection via Riemann's existence

theorem. Proc. Sympos. Pure Math. 37, 571-602 (1980). [8] D. R. HAYES, A geometric approach to permutation polynomials over a finite field. Duke Math

J. 34, 293-305 (1967). [9] R. LIDL and G. L. MULLEN, When does a polynomial over a finite field permute the elements

of the field? Amer. Math. Monthly 95, 243-246 (1988). [10] R. LIDL and H. NIEDERREITER, Finite fields. Reading, Mass. 1983. [11] D. PASSMAN, Permutation groups. New York 1968. [12] W R. SCOTT, Group Theory. Englewood Cliffs 1964. [13] T. TSUZUKU, Finite groups and finite geometries. Cambridge 1976. [14] D. WAN, On a conjecture of Carlitz. J. Austral Math. Soc. Ser. A 43, 375-384 (1987). [15] D. WAN, Permutation polynomials and resolution of singularities. Proc. Amer. Math. Soc., to

appear. [16] H. WIELANDT, Finite permutation groups. New York-London 1964.

Anschrift des Autors:

Stephen D. Cohen Department of Mathematics University of Glasgow Glasgow G12 8QW Scotland

Eingegangen am15.3.1990