kazhdan constants and the dual space topology

14
Math. Ann. 293, 495-508 (1992) Springer-Verlag 1992 Kazhdan constants and the dual space topology Eberhard Kaniuth 1'* and Keith F. Taylor 2'*'** 1Fachbereich Mathematik/Informatik, Universit/it Paderborn, W-4790 Paderborn, Federal Republic of Germany 2 Department of Mathematics, University of Saskatchewan, Saskatoon, Saskatchewan S7N OWO, Canada Received October 18, 1991; in revised form December 19, 1991 Mathematics Subject Classification (1991): 22D10, 22D15, 22D30 Introduction Let G be a locally compact group, and let t~ denote the dual space consisting of all equivalence classes of irreducible unitary representations of G. For a compact subset S of G and g e G, define ~Cs(ZO= inf (max ],n(x)~-~,,), ~, \ ~s where Jr is the unit sphere in ~'f~, the Hilbert space of ~. For finitely generated discrete groups, with S a finite set of generators, ~Cs has been introduced by de la Harpe et al. [10] to provide a quantitative version of property (T) of Kazhdan, and they refer to ~s(~) as the Kazhdan constant for ~ and S. Explicit and intricate computations giving estimates of Kazhdan constants for some representations and certain sets of generators of SL(3, 7/) have been carried out by Burger [3] (see also the appendix of [11]). The purpose of this paper is to study Ks as a function on G for locally compact groups G. In particular, we are interested in the behaviour of ~Cs with respect to the topology on G. More precisely, our main emphasis will be on the question to what extent Xs can be continuous on G. Let G be an almost connected amenable group. For such G, requiring that Ks be continuous on G for at least one (and hence for all) compact generating subsets S of G turns out to be equivalent to that G has a relatively compact commutator subgroup (Theorem 1). Theorem 1 indicates that we cannot expect ~s to be continuous on all of G very often. However, ~Cs is always continuous at the trivial representation 1G, and we * Supported by NATO collaborative research grant CRG 900029 ** Partly supported by an operating grant from NSERC Canada

Upload: independent

Post on 11-Nov-2023

0 views

Category:

Documents


0 download

TRANSCRIPT

Math. Ann. 293, 495-508 (1992)

�9 Springer-Verlag 1992

Kazhdan constants and the dual space topology

Eberhard Kaniuth 1'* and Keith F. Taylor 2'*'** 1 Fachbereich Mathematik/Informatik, Universit/it Paderborn, W-4790 Paderborn, Federal Republic of Germany 2 Department of Mathematics, University of Saskatchewan, Saskatoon, Saskatchewan S7N OWO, Canada

Received October 18, 1991; in revised form December 19, 1991

Mathematics Subject Classification (1991): 22D10, 22D15, 22D30

Introduction

Let G be a locally compact group, and let t~ denote the dual space consisting of all equivalence classes of irreducible unitary representations of G. For a compact subset S of G and g e G, define

~Cs(ZO= inf (max ] , n ( x ) ~ - ~ , , ) , ~ , \ ~ s

where Jr is the unit sphere in ~'f~, the Hilbert space of ~. For finitely generated discrete groups, with S a finite set of generators, ~Cs has been introduced by de la Harpe et al. [10] to provide a quantitative version of property (T) of Kazhdan, and they refer to ~s(~) as the Kazhdan constant for ~ and S. Explicit and intricate computations giving estimates of Kazhdan constants for some representations and certain sets of generators of SL(3, 7/) have been carried out by Burger [3] (see also the appendix of [11]). The purpose of this paper is to study Ks as a function on G for locally compact groups G. In particular, we are interested in the behaviour of ~Cs with respect to the topology on G. More precisely, our main emphasis will be on the question to what extent Xs can be continuous on G.

Let G be an almost connected amenable group. For such G, requiring that Ks be continuous on G for at least one (and hence for all) compact generating subsets S of G turns out to be equivalent to that G has a relatively compact commutator subgroup (Theorem 1).

Theorem 1 indicates that we cannot expect ~s to be continuous on all of G very often. However, ~Cs is always continuous at the trivial representation 1G, and we

* Supported by NATO collaborative research grant CRG 900029 ** Partly supported by an operating grant from NSERC Canada

496 E. Kaniuth and K.F. Taylor

establish a technical result (Theorem 2) which asserts that in certain situations ~c s varies continuously under inducing representat ion. This result can often be used to show that ~Cs is continuous on large subsets of G. For instance, if G is a motion group or a group of which all irreducible representa t io~ are finite dimensional, then ~c s is continuous on some dense open subset of G that can be described explicitly. From Theorem 2 we also deduce that if G is a connected semi-simple Lie group which has a finite center and is acceptable, then ~Cs is continuous on the irreducible principal series representations of G. In particular, if G is an acceptable complex connected semi-simple Lie group, then Ks is continuous on the reduced dual of G. We also present some illustrating examples.

I Preliminary results

For a compact subset S of a locally compact group G and any unitary representa- tion rc of G on the Hilbert space ~ , let

~Cs(n)= inf (max ,,n(x)~ - ~,,) .

Also, for Z c (~, let

tCs(Z) = inf ~Cs(a) .

In Proposition 1 below we collect together a number of elementary properties of ~Cs. Since most of these involve the topology on (~, we first recall its definition. Unitary representations of G are identified with ,-representations of the_group C*-algebra C*(G) in the usual way. For 22 ~_ G = C*(G) ̂ and n~G, r r~Z if

ker rc ___ n {ker ~; ~ ~ _r } ,

where ker a denotes the kernel of a as a representation of C*(G). This closure operation determines the topology on G. Then rc ~ 2T is equivalent to the following: for any tl ~ ~ff~, ]t q ]l = 1, e > 0 and a compact subset C of G, there exist a ~ I; and

~ ~r 11 ~ ][ -- 1, such that

1(~(x),7,,7) - (~ (x )C ~)1 =

for all x ~ C [5, Proposit ion 18.1.51. This topology on G is in general far away from being Hausdorff, and a set of particularinterest, in this regard, is the cortex cor(G) of G which consists of all elements in G that cannot be Hausdorff separated from the trivial representation la (see [1]). We also remind the reader that G is said to have Kazhdan 's property (T) if 1G is an isolated point in G.

Moreover, we introduce the following notations. For any unitary representa- tion g of G the support of g, supp re, consists of all p e G such that ker p ~_ ker g. Two sets S and T of representations are weakly equivalent (S ~ T) if

N k e r a = ('] ke r~ . aES I:~T

Proposition 1. Let G be a locally compact group and S a compact subset of G. Then the following conclusions hold:

(i) if22 ~_ G and 1 ~ ~, then Ks(S,) = O. (ii) I f (n~)~t is a net in G such that n~ --* 1~, then Ks(nz) ~ O.

Kazhdan constants and the dual space topology 497

(iii) I f S generates G and X ~_ G is such that Xs(Z) = O, then 1~fi17. (iv) I f S generates G, then G has property (T) if and only i f k s ( G \ { l a } ) > O. (v) I f S generates G and n~G, then n e c o r ( G ) if and only if there exists a net

(n,),~1 in G such that n, --, n and ~ (n , ) ~ 0. (vi) For Z ~ G and n~G, n e Z implies SCs(Z ) < Sos(n). (vii) I f n, a ~ G are weakly equivalent, then ~Cs(n ) = ~Cs(a).

Proof. (i) If las17, then given e > 0, there exist a e Z and ~ ocg~, 11411 = 1, such 8 2

that ](a(x)~, ~) - iI _-< ~- for all x e S . Then, for x e S ,

II a(x)~ - ~ II 2 -- 2(1 - Re@(x)~, 4) ) < 82 �9

Thus ~Cs(a) < 8, for some a e S, and since e > 0 was arbitrary, this shows ~Cs(2;) = 0. (ii) Suppose that Xs(n,) does not converge to 0. By passing to a subnet if

necessary we can assume Xs(n,) > 6 > 0 for some ~ > 0 and all t. Let Z = {n,; ~ e I }. Then ~Cs(Z) > ~ and l o e Z which contradicts (i).

(iii) Suppose S generates G, and let 2 ;__G satisfy X s ( Z ) = 0 . Since Xs = ~Cs-, = Xs~,s-,, we can assume S = S -~. To show l o e Z, let C be any compact subset of G and 8 > 0. Then C _ S n for some n e N . As Xs(Z) = 0, there

such that II a (x )~ - ~ II < -~ for all x e S . For y e C n

x~ . . . x,. It follows as in [10, p roof of Lemma 4]

~=L ~(x~ . . . x j - 1 ) ( ~ r ( x j ) 4 - ~)

_-< ~ II a (x j ) r - ~ tl --< 8 . j = l

Hence I (a(y)~ , ~) - 11 < 8, and this implies 1Ge 27. (iv) Fo l lows immedia te ly from (i) and (iii). (v) For n e G , n e c o r ( G ) is equivalent to the existence of some net (n,),~, in

(; such that n, --* n and n, ~ la. Therefore it suffices to show that n, ~ 1G if and only if Ks(n,) ~ 0. The "only if" part is (ii) above. To verify the "if" part, consider any subnet (7~/,t)),~A of (n,),~1. For 2 o e A , let Sx o = {n,/ 2 > 20}. Then Xs(2;Xo) = 0 as Xs(n,~) --* O, and hence 1~ ~ 17Zo by (iii). Thus (nZ)~a has a further subnet converging to 1 G. This is true for every subnet of (n,), ~ t, thus the entire net (n,), ~ ~ must converge to la.

(vi) By the definition of Xs(n), for any 8 > 0, there exist ~ e ~ , II ~ II = 1, such that 11 r~(x) ~ - ~ II 2 < ~Cs(n)2 + 8 for all x e S. Since n e I7 and S is compact , there are a s Z and t /s Yf,, II t/II -- 1, such that

[(r~(x)~, ~5 - (~(x) t / , u ) l < 8

for all x e S. Therefore

I l a ( x ) t / - r/ll 2 = 2(l - Re (a(x)t/, t/))

2(1 - Re(n(x)4, ~)) + 28

= IIn(x)~ - ~ll 2 -t- 28 < Xs(n) 2 + 38

for all x~S . Thus Xs(a) 2 < Xs(n) 2 + 38 for some a e I ; . This proves Xs(Z) < ~Cs(n).

exist a e S and ~ e~'/g~, II~ll = 1,

we find xt . . . . . xneS with y = that

I1 a ( y ) ~ - ~ II =

498 E. Kaniuth and K.F. Taylor

(vii) This follows from (vi) since n and a are weakly equivalent exactly when

n � 9 1 4 9 []

Remarks. a) Due to (vii) in Proposition 1 ~Cs can be considered as a function on Prim(G), the primitive ideal space of C* (G), which is equipped with the quotient topology from the map n ~ kern of G onto Prim(G).

b) If S is a compact set generating G and necor (G) is such that n does not weakly contain 1G, then ~Cs fails to be continuous at n by (i) and (v). This simple fact will provide the key to the proof of one of our main results (Theorem 1).

c) By (vi) ~:s is a lwaysupper semicontinuous on G. It is natural to attempt to identify large subsets of G on which Xs is continuous.

d) The functions Xs for different generating compact subsets S of G are compar- able in the sense that if S and T are two compact subsets of G, each of which generates G, then there are positive constants c~ and ca, depending on S and T, such that

c~ ~cs(n) <= ~c~(n) < c~c~(n)

for all representations n of G. To see this note, for example, that

To_. ( S u S -1 w {e}) ~

for some natural number n. Reasoning as in proof of (iii) in Proposition 1, we see that for y �9 T and ~ e ~ , I[ 4 II = 1,

[In(y)4 - 411 < nmax [In(x)4 - ~11. X E S

Thus xr(n) < n~cs(n) and we can take c2 = n. Interchanging the roles of S and T yields the appropriate c~.

2 The main results

If N is a closed normal subgroup of G, then every representation of G/N can be considered as a representation of G, by composition with the quotient map. In this way G~'N is identified with a closed subset of G.

Proposition 2. Let G be a locally compact group, N a closed normal subgroup such that G/N is abelian arid S a compact subset of G. Then

(i) 2 ~ ~s(n | 2) is continuous on G/N, for any representation n of G. (ii) ~s is continuous on G provided that N is compact.

Proof. (i) For x �9 G, ~ �9 ~tt~ with II 4 II = 1 and 2, # ~ G ~ we have

II (n | 2)(x)~ - (n | II = II (~(x) - 14x))n(x)4 II = 12(x) - / 4 x ) l �9

Hence, if e > 0 and I)~(x) - #(x)l < e for all x � 9 then

IIl(n | 2) (x)4 - ~11 - II(n | ~ ) (x)4 - ~111 <

II(n @ 2)(x)4 - (n | # ) (x)~ II _-<

for x �9 S, and therefore

I max ll(n | 2)(x)~ - 411 - max II(n @/~)(x)4 - 4 II < ~. x e S x ~ S ~

Kazhdan constants and the dual space topology 499

This estimate is uniform in ~. Thus, taking the infimum over all ~ with ]] ~ II = 1, we obtain

IKs(n | ,~) - Ks(~ | #)1 < e .

(ii) As noted in the remark following Proposition 1, Ks can be considered as a function on Prim(G), and it suffices to show that as such Ks is continuous. Now, since N is compact, Prim(G) is a disjoint union of sets of the form

K~ = {ker(~ | 2); 2 e G~'N}, r c e G ,

and each K~ is open in Prim(G) [12, Theorem 2]. Moreover, the mapping

2 ~ ker(n | 2), G ' ~ --* K~

is open [12, Proposition]. It follows that Ks is continuous on each K~, and hence on Prim(G). []

Before proceeding to Theorem 1 mentioned in the introduction we insert a simple fact which will be useful at several points in the proof of Theorem 1.

Lemma 1. Let H be a locally compact group possessing a compact generating set S such that Ks is continuous on H. Then every factor group has the same property.

Proof If N is a closed normal subgroup of H and q: H ~ H / N the quotient map, then q(S) is a compact generating subset of H/N, and it is easy to check that for every representation n of H / N one has Ks(n ~ q) = Kq~s)(n). []

In what follows, for a closed subgroup H of G and any representation a of H, i n d i a will denote the representation of G induced by a.

Theorem 1. Let G be an almost connected amenable group. Then the following assertions are equivalent.

(i) Ks is continuous on G, for every compact subset S of G. (ii) Ks is continuous on G, for at least one compact subset S of G which

generates G. (iii) G contains a compact normal subgroup K such that G/K is abelian.

Proof (i) ~ (ii) is obvious, and (iii) ~ (i) is a special case of Proposition 2(ii). To prove (ii) =~ (iii), notice first that we can assume G to be a Lie group. Indeed, this follows from Lemma 1 and the fact that every almost connected group is a projec- tive limit of Lie groups 1-16, p. 175]. The connected component Go of G has finite index in G. Let R denote the radical of Go, that is, the maximal solvable normal subgroup of R. Again, by appealing to the lemma, we may assume that R has no non-trivial compact normal subgroup. Amenability of G implies that G/R is compact (compare [18, Theorem (3.8)]).

We now argue by induction on the dimension of the connected Lie group R. If dim R = 0, then G is compact and (iii) holds. Assume (ii) implies (iii) for all such groups G for which dim R < n. If dim R = n + 1, then R contains a non-trivial vector group V which is normal in G. In fact, denoting by

R = R o 2 . . . ~_ R a + l ={e} the commutator series of R, every Rj is connected and normal in G. Hence Re is the direct product of a vector group V and a compact subgroup C, which of course is

500 E. Kaniuth and K.F. Taylor

normal in G and hence trivial. By Lemma 1 and the induction hypothesis, G/V has a compact normal subgroup with abelian quotient.

We claim that Visa contained in the center of G. For that we have to show that for each 2 ~ V, the stability subgroup Gz of 2 in G equals G. By Lemma 1.7 of [1] we know that

supp(ind~ 1~,) _ cor(G).

On the other hand, no n~G/V, n 4: 1~, weakly contains 1~ as Prim(G/V) is a T1 space. Therefore, since Ks is continuous,

supp(indaa~ 1G~) = {1G} ,

i.e. Gz = G, by Remarks, b). Now there exists a normal subgroup N of G such that V ~_ N, N / V is compact

and G/N is abelian. Namely, N can be chosen as the inverse image in G of the closed commutator subgroup of G/V. Being almost connected, G/N is of the form W• C where C is compact. By enlarging N we can assume that G/N is a vector group. V is contained in the center of N, and N/V is compact. This implies that the commutator subgroup of N is relatively compact. More precisely, N = V x D where D is a compact group [9, Theorem 4.6]. Clearly, D is normal in G, and by using Lemma 1 again, we can move to the quotient G/D. Hence we may assume that N = V and G/V is a vector group. Now take any x E G \ V and consider the subgroup H of G defined by H ___ V and H/V is the 1-dimensional subspace of G~ V spanned by x V. Since V is central and V and G/V are vector groups, H is a normal vector subgroup of G. By the same argument as was used on V, we see that H is contained in the center of G. This shows that G must be abelian and completes the proof of Theorem 1. []

Theorem 1 shows that we cannot expect Xs to be continuous on all of G very often. However, it turns out that for various classes of locally compact groups G, Xs is continuous on large subsets of G. The following theorem is a key to such results.

Theorem 2. Let G be a separable locally compact group and H a closed subgroup of G such that G/H is compact. Suppose Z is a set of representations of H all of which act in the same finite dimensional Hilbert space 9r ~. Then, for each compact subset S of G,

a ~ xs(ind~a)

is a continuous function on S,.

Proof. Since G is separable and G/H is compact, there exists a Borel cross-section

c: G/H ~ G of the left H-cosets in G such that c(G/H) is compact 1-15]. Thus, denoting by p: G --, G/H the quotient map and by x . t the natural action of x e G on t e G/H, we have x . t = p(xc(t)). Choose a continuous rho-function p on G, and let /t be the corresponding quasi-invariant measure on G/H [7, Chap. III. Sect. 14]. As G/H is compact, /~ can be assumed to be a probability measure. Its Radon- Nikodym derivative

d(x#) r l ( x , t ) -

dt~ is given by

rl(X, t) = p(x- lc(t))p(c(t))-I ,

Kazhdan constants and the dual space topology 501

x ~ G, t ~ G/H. Then for any uni tary representa t ion a of H in the Hi lber t space a f t , the induced representa t ion indGno - can be realized in Lz(G/H, p, aft), with the opera to r i n d i a ( x ) acting o n ~LZ(G/H, #, 3r by

ind~a(x)~( t ) = tl(x, t) 1/2 a ( c ( t ) - l x c ( x - 1 t ) ) ~ ( X - ' ' t)

(compare [7, Chap. XI, Sect. 10]). N o w let S be a fixed compac t subset of G and set

C = {c ( t ) - l x c ( x - l ' t ) ; x 6 S , t e a ~ H } - ,

which is a compac t subset of H. If(a,),~, is a net in Z converging to a e Z, then, since the a, and a all act in the same Hilber t space a f , this means that for each ~o ~ a f , one has a,(y)~o--+ a(y)q~ uniformly in y on every compac t subset of H. The space a f being finite dimensional , this implies I] a,(y) - a(y)]] -+ 0 uniformly on compac t subset of H. Thus, for any given e > 0, there exists lo e I such that

I l a , (Y) -a (Y) t [ =<~ f o r y e C a n d l > z o .

Setting h(x, t) = c( t ) - l x c ( x - 1. t), we obtain for ~ ~ L 2 (G/H, l~, Jr) with ]J ~ [] = 1, x 6 S and t > lo

[[ ind~ a,(x){ - i n d i a ( x ) ~ I[ 2

-- ~ ~/(x, t)11 {a,(h(x, t)) - a(h(x, t ) )}g(x -1 .t)ll2dl~(t) G/H

<= ~ q(x, t)llal(h(x, t)) - ~(h(x, t))[I z II ~(x -1 "t)HZdl~(t) G/H

__< aa ~ ~(x , t ) l lg (x -1.t)H2d/~(t) = e2 ~ ]lg(t)ll2dt~(t) = e2. G/~I G/H

Hence, for each x e S , { e L 2 ( G / H , I*, a f ) with tl~ll = 1 and ~ > to

[t1 ind~a,(x)~ - ~ 1t - [[ ind~a(x)~ - ~ Ill <

11 ind~a,(x)~ - ind~a(x)~ [t < e .

This implies, for t > to and every { e L 2 (G/H, I~, aft) with [1 ~ II = 1,

maxims Ilind~a,(x)~ - ~l[ - maxims Il ind~a(x)~ - 411 _-< e .

If t > to is fixed and { is chosen so that

max l[ ind~a,(x){ - { l[ _-< ~cs(ind~a,) + ~, x~S

then the same { gives

max [lind~a(x)~ - ~ ]l ~ ~cs(indga,) + 2e . x e S

Thus ~cs(indt~a) < ~cs(ind~a,) + 2e for t > to, and interchanging a, and a yields [~s ( ind~a , ) -~cs ( ind~a) ] < 2 e for t > to. This proves that ~s is cont inuous at a. []

502 E. Kaniuth and K.F. Taylor

The separability assumption in Theorem 2 has only been used to guarantee the existence of a Borel cross-section. Thus, Theorem 2 also holds when H has finite index in G, where G need not be separable anymore.

3 Applications

We are going to apply Theorem 2 to groups with finite dimensional irreducible representations, to motion groups and to real rank one semisimple Lie groups.

Thus let G be a locally compact group all of whose irreducible representations are finite dimensional. Such groups have been completely characterized by Moore [17] and are therefore now usually referred to as Moore groups. By [17, Theorem 2 and Theorem 3] a locally compact group G is a Moore group if and only if G is a projective limit of groups each of which is a finite extension of a group with cocompact center. As a result we have the following structural properties. Let GF denote the subgroup of G consisting of all elements with relatively compact conjugacy classes. Then GF has finite index in G, and the commuta tor subgroup of Gv has compact closure.

Lemma 2. Let G be a Moore group, and let Z denote the set o f all a e Gv^such that the stability subgroup o f cr in G equals GF. Then S is dense and open in GF.

Proof Recall that the action of G on (~v is written as ( a , a ) ~ a " , where aa(x) = a (a- l xa), a e G, x e G v, O- e G v. Let A denote a representative system for the non-trivial cosets of GF in G. It follows from

s = U o" = a~A

and from the continuity of a ~ o -a, that Z is open in GF. Let K denote the closed commuta tor subgroup of GF. Then, by [12, Theorem 2] and since GF is type I, the set

r | (GF/K) ̂ = {~ | 2; ), e (GF/K) ^ }

is open in (~F for every o e d r . It therefore suffices to show that Z c~ [o- | (Gv /K) ^ ] is dense in O- | (Gv/K) ^. Fix O- and suppose there exist a e A and an open subset W#: ~ of O- | ^ such that r " = r for all f e W . Since p | ^ = ~r | (Gp/K) ^ for each p e O- | (Ge /K) ^, we can assume O- e W. Let

V = { 2 e ( G e / K ) ^ ; a | ),e W} .

Then Vis open in (GF/K) ^, and for 2e V

0" | = (0" | a = O-a| = O-|

SO that a | = O-. Now, let F = { •e (Gr /K)^ ; ~r| = a}. Then F is a closed subgroup of (GF/K) ̂ , hence of the form F = (Gv/H) A for some closed subgroup H of Gr containing K. It follows from the definition of F that

a ~ i n d ~ ( a l g ) ,

and since a is finite-dimensional, this implies that H has finite index in GF, and hence in G. Denote by N the largest normal subgroup of G contained in H. Then

Kazhdan constants and the dual space topology 503

N has finite index in G. By what we have seen above, ~ ) - a ~ / ' ~ and hence Aal N = 21N for all 2 t V. Let A be the subgroup of N~'K generated by the open set {).IN; ).~ V}. Then A = N"~ foAr some closed subgroup L of N, and Oa = ~ for all 6 e A. The subgroup A of N~/s being open, L / K must be compact. Thus L is compact and normal in Gr and since Gv has finite index in G, the closed normal subgroup C of G generated by L is also compact. Now, ]A a = [2 for all # t N'~'C. Equivalently,

[n, a] = nan- l a - 1 t C

for every n t N. We claim that this yields the contradiction a t Gv. Indeed, if B is a finite coset representative system for N in G, then for arbitrary x = bn, b ~ B, n t N ,

x a x - 1 = b In, a] ab - 1 t bab - 1 C ,

so that the conjugacy class of a is contained in the compact set U bab- lC" This b e B

finishes the proof of Lemma 2. []

Theorem 3. Let G be a Moore group, Z as in Lemma 2 and

1-1 . G . z } . = {mdG~a, a t

Then 1I is open and dense in G, and ~:s is continuous on 11for every compact subset S o f G .

Proo f By Mackey's theory, i n d ~ a is irreducible for every a t Z. Since inducing is continuous and Z is dense in Gv by Lemma 2, 11 is dense in the reduced dual Gr of G, the support of the left regular representation of G. However, (~r = G as G is amenable. To show that 11 is open, let n = ind~Fat11, a t Z , and consider a net (n,)~z converging to n in G. Then ~,[Gv ~ ~[Gr, and for every t we can choose a, t Gv such that re, ] Gr is a direct sum of representations in the finite G-orbit of a,. Thus we can assume a, ~ a in Gr. As 2 is open in Gr, this implies a, ~ Z and hence n, = i n d ~ : , t 1 1 for t __> Zo.

Let now S be any compact subset of G and denote by K the closed commutator subgroup of Gp. For z t Z, let

Z~ = Z c~ [z | (Gv/K) ^] .

Z~ is open in G, and all representations in Z~ act in the Hilbert space of z. It follows from Theorem 2 that a --* xs(ind~Fa) is continuous on each Z, and hence on Z. On the other hand, this function is constant on every G-orbit G(a), and therefore can be regarded as a continuous function on the quotient space Z/G of G-orbits G(a), a t Z. Finally, notice that the mapping

H ~ Z/G, n ~ supp(Tz[Gr)

is continuous (in fact, a homeomorphism). This proves that Ks is continuous on H. []

We now turn to motion groups. Here by a motion group G we mean a semi- direct product G = K t,< V where V is a vector group and K a compact connected Lie group. By [2, Theorem 3.1] there exists a so-called principal stability subgroup for the action of K on 1,3, i.e. there are a dense open subset U of V and a subgroup

504 E. Kaniuth and K.F. Taylor

H of K such that for every 2 E U the stability group K~ of 2 is conjugate to H. Of course we can assume that U is invariant under the action of K. Moreover, by a result of Glimm [8, Theorem 3] there exists a dense, open, K-invariant subset Wof Vsuch that the stability groups vary continuousl~ on W. That is, the mapping 2 ~ Ka from W into the set 5"(K) of all closed subgroups of K is continuous, where 5e(K) carries Fell's compact-open topology [.see [6]). Intersecting U and W we obtain a dense, open, K-invariant subset of V having both of these properties.

For 2 E V and aEK~., let a 2 E ( K ~ , < V) ̂ denote the representation (a, x) --* 2(x)a(a), aEKx, xE V, and

zt~,~ = i n d ~ v(~r2) ,

the representation of G obtained by inducing a2 up to G. Recall that by Mackey's theory these nz,,, 2E 12, rrEKz, are irreducible and exhaust G.

Theorem 4. Let G = K t,< V be a motion group as above. Let H be a closed subgroup of K and A a dense, open, K-invariant subset of V with the following properties:

(i) For 2 E A, the stability group Kz of 2 is conjugate to H. (ii) The stability groups Kz vary continuously on A.

Let 1I denote the set of all representations

nz,~ = i n d ~ v ( a 2 ) ,

; teA, a E I( a. Then 11 is dense and open in G, and for any compact subset S of G, Ks is continuous on 1I.

Proof If rtE(;, then supp(n lV)=K( ; t ) for some 2ff 17" and rcr if and only if K(;t) n A = ~ . This implies that G i l l is closed in G. To see that 11 is dense in G, observe that

{a2; r ~ i n d ~ v 2 , and therefore

{ i n d ~ v ( a 2 ) ; aE/s ~ indv~

Since A is dense in 19, we obtain that 11 is weakly equivalent to the left regular representation of G. G being amenable, this proves t ha t /7 is dense in G.

Now let S be a compact subset of G and put An = {;tEA; K~ = H}. B y Theorem 2, for each rrEH, the.function ;t --, ~Cs(rCz,,) is continuous on An. As H is discrete, it follows that (;t, a) --* Xs(n~,~) is continuous on A n x H. Finally, denote by N the normalizer of H in K. Then, for ( ; t~ ,a3EAnxH, i = 1, 2, we have rt~,.~ = r ~ , ~ if and only if (al;tl)~ = az22 for some aEN, and inducing defines a homeomorphism between the quotient space (An x H)/N of N-orbits in An x H and 17 [4, Lemma 2.2]. Hence

l t z , , ---, n(~r;t) ---, ~ : s (~z , s )

is continuous on/7 . []

Corollary 1. Let G = SO(n)e,<R", nEIN, where as usual SO(n) acts on IR" by rotations, and let S be a compact subset of G. Then Xs is continuous on GkSO(n) ^, that is, on the set of infinite dimensional representations of G.

Proof Identifying ~ " with R", it is easy to see that the set A in Theorem 4 can be chosen as lR"\{0}. []

Kazhdan constants and the dual space topology 505

Finally, we apply Theorem 2 to connected semi-simple Lie groups with finite center. For such a group G let G = K A N be an Iwasawa decomposition, M the centralizer of A in K and P = M A N the corresponding minimal parabolic sub- group. Associated to each a s M and c~EA is the unitary representation n,, , of G obtained by inducing the finite dimensional representation

man ~ ~(a)a(m), m~M, a~A, n ~ N ,

of P to the group G. These representations constitute the principal series of G (see [23, Chap. V]). In general, ~.~ need not be irreducible. However, it is if either G is complex or a = 1M (see E22, Theorem 4.1-] and [23, Theorems 5.5.2.3 and 5.5.2.4]). On the other hand, for SL(n, 1R) all the principal series representations are irredu- cible if and only if n is odd [22, Theorem 5.1]. Let Gp_ G denote the set of irreducible principal series representations. For the sake of completeness we recall Harish-Chandra's notion of acceptability. Let Gc be a complexification of G and [ a Cartan subalgebra of g, the Lie algebra of G. Furthermore, let P denote the set of positive roots of (go, be) with respect to some ordering and set p = �89 ~, c~. Then

0tEP

G is said to be acceptable if G admits a Gc such that there exists an analytic homomorphism 3o of Hc, the Cartan subgroup associated with b~, into 112 \ {0} such that ~p (exp h )= e pth) for all h~l)~. Note that G is acceptable if G~ is simply connected and that every connected semi-simple Lie group with finite center possesses a finite covering group which is acceptable [23, Sect. 8.1.1-].

Theorem 5. Let G be an acceptable connected semi-simple Lie group with finite center. Then, for any compact subset S of G, the function Xs is continuous on Gp.

Proof Since )~ is discrete it follows from Theorem 2 that the function

f" (~, ~) ~ ' Ks(~,o) is continuous on A • M. Let W denote the Weyl group, that is, the quotient of the normalizer of A in K modulo M. The group W is finite and acts on M and A in the natural way. By a fundamental result of Bruhat (see Lemma 4.1 of [14]) rc~,, and ha. ~ are unitarily equivalent if and only if there exists w ~ W such that w(~) =/~ and w(a) = r. This shows that f is well defined and continuous on the quotient space (A • ~-I)/W. Now let 2; be the set of all (~, a)~A • A~ such that n~., is irreducible. Then ,2 is W-invariant and W(~, a) ~ r~.~ is a bijection between Z /W and Gp. On the other hand, Lipsman [14, Theorem 6.2] proved that this bijection is a homeo- morphism. Thus rc ~ Ks(n) is continuous on Gp. []

For a locally compact group G let G, denote reduced dual of G, which consists of all those elements of G that are weakly contained in the regular representation. Since for a connected complex semi-simple Lie group G~ coincides with Gp (com- pare [14, Remark (2) on p. 412]), we obtain the following corollary to Theorem 5.

Corollary 2. I f G is an acceptable complex connected semi-simple Lie group and S is a compact subset of G, then Ks is continuous on the reduced dual G~ of G.

4 Examples

In this final section we present some examples illustrating and complementing our previous results.

506 E. Kaniuth and K.F. Taylor

Example 1. If G is a compact group and S is taken to be all of G, then the following

argument shows that xs(n) > ~ for any representation n of G which does not contain the trivial representation. To see this note that

(~(x)~, ~ ) a x = o G

for every ~ ~ ~ , by the orthogonality relations. Therefore, if ~ ~ Yg. with II r II = 1, there exists x~G such that Re(rc(x)r ~) < 0, and hence

II rc(x)r - ~ [I = (2(1 - Re(Tz(x)~, ~)))x/2 > x/~.

There would be no special value in trying to improve the lower bound of ~ in general. But, if G is a connected compact group, it may be possible that ~c~(rc) = 2 for every non-trivial irreducible representation rc of G (at least, we know of no counterexample), xG(z) = 2 is equivalent to the following interesting property: For any ~ Y ~ , there exists x~G such that rc(x)~--- - 4 . Note however that, if for

example G is the cyclic group of order 3, then toG(?) = xf3 for each non-trivial character 7 of G.

Recall that for an almost connected amenable group G, continuity of ~:s on all of G (for some compact generating subset S of G) forces G to have a relatively compact commutator subgroup (Theorem 1). Our next example shows that there is no similar result for discrete groups. In fact, there exists a (however, not finitely generated) discrete group G such that Ks is identically zero for every finite subset SofG.

Example 2. Let F be a finite field and ff its algebraic closure. Let G = SL( ~ , if) be the discrete group consisting of all infinite matrices over/7 with determinant one, ones on the diagonal and only finitely many non-zero entries outside the diagonal. Since the algebraic closure of a finite field is an ascending union of finite fields, G is locally finite and hence amenable. We claim that Xs(rC) = 0 for every finite subset S of G and all 7~ ~ G. This will be clear once we have shown that every z E G weakly contains lo.

To that end we employ the theory of characters in the sense of Thoma [21 ]. Let G be an arbitrary discrete group, and denote by K(G) the set of all normalized positive definite functions ? on G that satisfy the equation y(x-lyx) = 7(Y) for all x, y~ G. The set K(G) is convex and w*-compact, and hence the closed convex hull of the set E(G) of extreme points in K(G). Let n be any irreducible representation of G, and choose a normalized positive definite function ~o annihilating I = ker ft. Let C denote the w*-closed convex hull of all functions q~, x~G, where q~(y) = qg(x-lyx). Then every r ~ C annihilates I. Now suppose that G is amen- able. Since the action

(x ,~k )~k ~, G•

is continuous, C n K(G) ~ ~ by the fixed point theorem for amenable groups [18, Theorem 2.24]. Thus some 7~K(G) annihilates I.

The crucial fact now is that in case G = SL(ov, fi), by a result of Skudlarek [20, KoroUar 5], E(G) consists only of the trivial character 1 and the regular character 6~, e the unit element in G. Hence ~ is a convex sum of 1 and 6~. Since, due to the amenability of G, the regular representation weakly contains lo, we conclude that 1 annihilates I, and therefore ker~ _ ker IG.

Kazhdan constants and the dual space topology 507

Example 3. Let G - IF2 , the free group on two generators, and let S be a finite generating subset of IFz. We claim that Ks is continuous at ~ ~ IF2 of and only if Ks(n) = 0, that is, g weakly contains the trivial representation. To verify this we use a remarkable and only recently fully recognized result due to Yoshizawa [24] which states the existence of some p e1172 that weakly contains all g elF2. In particular ~Cs(p) = 0, and if ~s is continuous at ~, then ~s(n) = 0. Conversely, if ~s(rC) = 0 and g , ~ r c such that ~Cs(g,) does not converge to zero, then Ks(~,~) > 6 > 0 for some 6 > 0 and some subnet (~,~) of (~,), and hence ~Cs(r~) > 6. Moreover, it is worth mentioning that ~s is constant and non-zero on the reduced dual (IFz)2 of IF2 since the reduced group C*-algebra C*(IF2) is simple [19] and therefore any two representations in (IFz) 2 are weakly equivalent. One can obtain estimates for this constant as follows. Let IFz be generated by a and b, and let

S = {a ,b ,a - t , b - t } and h =�88 ~ 6x. x ~ S

Fix rr e (IF2)2, and denote by 2 the left regular representation of IF 2 . Then, since rc is a faithful representation of C* (IF2), re(h) and 2(h) have the same spectrum. Now, by intricate computations Kesten [13, Theorem 3] (see also [18, 4.31(iii)]) has shown

SpJ.(h) = [ - � 8 9 a x e ] .

Thus, using [10, Proposition 2 and the remark following it] we conclude

x / 2 - x/3 < ~Cs(~) < x/2(2 - x/~) .

One of the crucial points in the proof of Theorem 1 was the fact that, for compact generating subsets S of G, ~s fails to be continuous at points in the cortex of G which do not weakly contain 1 a. Looking at SO(2)~< IR 2 or IF 2 one observes that Ks is continuous outside the cortex. Indeed, cor(lF2) = 1132 since there is a dense point in IF2, and c o r ( S O ( 2 ) x IR 2) = SO(2)^. Therefore, one might conjecture that, in general, ~s is continuous on G\cor(G). However, this is not true even for almost abelian groups. We give an example of a discrete group G with abelian normal subgroup of index four which not only has discontinuities of ~c s at non-cortex points but also illustrates the calculation of ~s in a simple case and the dense open subset of G to which Theorem 3 refers.

Example 4. Let H = 7/2 ~< 7/, where 7l 2 acts on 7/ in the unique non-trivial way, and G = H x H. The dual space H is well known. For 0 < t < 1, let ~, denote the 2-dimensional representation of H induced by the character n ~ e ~"" of 7/. The representation ~, is irreducible for 0 < t < 1, ~zo decomposes into the two charac- ters n~- = la and =o of 7Z.2 = G/7/, and n~ decomposes into the two other charac- ters of G/27/, rV~ and a~-, say. Then

= 0 < t < 1} n o , } .

The set of 2-dimensional representations ~, carries its parameter space topology, and if t ~ 0 (resp. t ~ 1), then ~z,~ rc~-, % (resp. ~,+~+1, =i-). In particular, cor(H) = {~6 ~ , So }. Clearly, G = / 4 x H and cor(G) = cor(H) x cor(H). Now let

S = {((I, 1), (1,0)), ((1, 1), (0, 1)), ( ( - 1 , 1), (0, 0)), ( ( 1 , - 1), (0,0))} ~_ G ,

and consider the behaviour of tCs on (~. We leave the computations to the reader who should easily verify that

508 E. Kaniuth and K.F. Taylor

(i) Xs(n, x n~) = m a x { x / 2 ( 1 - cos nt), x/2(1 - cosns)}, 0 < t, s < 1. (ii) Xs(nt x no ) = 2 for 0 < t < 1.

This shows that Ks is no t con t inuous at every nt x n o , 0 < t < 1.

One might also expect that Xs is con t inuous on every open Hausdor f f subset of G. Yet, the g roup H = Z2 ~,< Z occurr ing in Example 4 shows that this is no t t rue in general. In fact, H \ { n +, n'~, n~} is open and Hausdor f f and nevertheless Ks is d i scon t inuous at n o .

Acknowledgement. The authors are indebted to the referee for valuable comments and for pointing out some misprints.

References

1. Bekka, M.B., Kaniuth, E.: Irreducible representations that cannot be Hausdorff separated from the identity representation. J. Reine Angew. Math. 385, 203-220 (1988)

2. Bredon, G.E.: Introduction to compact transformation groups. New York London: Academic Press 1972

3. Burger, M.: Kazhdan constants for SL(3, Z). J. Reine Angew. Math. 413, 36-67 (1991) 4. Carey, A.L., Kaniuth, E., Moran, W.: The Pompeiu problem for groups. Math. Proc. Camb.

Philos. Soc. 109, 45-58 (1991) 5. Dixmier, J.: Les C*-alg6bres et leurs repr6sentations. Paris: Gaflthier-Villars 1964 6. Fell, J.M.G.: Weak containment and induced representations, II. Trans. Am. Math. Soc. 110,

424-447 (1964) 7. Fell, J.M.G., Doran, R.S.: Representations of *-algebras, locally compact groups, and Banach

*-algebraic bundles, vols. 1 and 2. Boston. MA: Academic Press 1988 8. Glimm, J.: Locally compact transformation groups. Trans. Am. Math. Soc. 101, 124-138 (1961) 9. Grosser, S., Moskowitz, M.: Compactness conditions in topological groups. J. Reine Angew.

Math. 246, 140 (1971) 10. de la Harpe, P., Robertson, G., Valette, A.: On the structure of the sum of the generators of

a finitely generated group. Isr. J. Math. to appear 11. de la Harpe, P., Valette, A.: La propri6t6 (T) de Kazhdan pour le groupes localement

compacts. (Asterisque vol. 175) Paris: Soc. Math. J., 1989 12. Kaniuth, E.: Primitive ideal spaces of groups with relatively compact conjugacy classes. Arch.

Math. 32, 16-24 (1979) 13. Kesten, H.: Symmetric random walks on groups. Trans. Am, Math. Soc. 92, 336-354 (1959) 14. Lipsman, R.L.: The dual topology for the principal and discrete series on semisimple groups.

Trans. Am. Math. Soc. 152, 399-417 (1970) 15. Mackey, G.W.: Induced representations of locally compact groups I. Ann. Math. 55, 101-139 (1952) 16. Montgomery, D., Zippin, L.: Topological transformation groups. New York: Interscience 1955 17. Moore, C.C.: Groups with finite dimensional irreducible representations. Trans. Am. Math.

Soc. 166, 401-410 (1972) 18. Paterson, A.L.T.: Amenability. (Math. Surv. Monogr., vol. 29) Providence, RI: Am. Math.

Soc. 1988 19. Powers, R.T.: Simplicity of the C*-algebra associated with the free groups on two generators.

Duke Math. J. 42, 151-156 (1975) 20. Skudlarek, H.L.: Die unzerlegbaren Charaktere einiger diskreter Gruppen. Math. Ann. 223,

213-231 (1976) 21. Thoma, E.: Uber unit/ire Darstellungen abz/ihlbarer diskreter Gruppen. Math. Ann. 153,

111-138 (1964) 22. Wallach, N.: Cyclic vectors and irreducibility for principal series representations. Trans. Am.

Math. Soc. 158, 107-113 (1971) 23. Warner, G.: Harmonic analysis on semi-simple Lie groups I, II. Berlin Heidelberg New York:

Springer 1972 24. Yoshizawa, H.: Some remarks on unitary representations of the free group. Osaka Math. J. 3,

55-63 (1951)