Representations of the universal covering group of the group su(1,q) Текст научной статьи по специальности «Математика»

REPRESENTATIONS OF THE UNIVERSAL COVERING GROUP OF THE GROUP SU(l,q) 1

V. F. Molchanov, Yu. A. Sharshov

G. R. Derzhavin Tambov State University, Russia University of Leiden, Leiden, The Netherlands

In this paper we study a family of representations of the group SU (1, q), the universal covering group of the group SU(1, q), induced by characters of a maximal parabolic subgroup. These representations can be realized on spaces of homogeneous functions on a cone in C1+9, so we call them representations associated with a cone. They are labelled by two complex parameters a. r.

We determine when these representations Ta<T are irreducible and describe completely the structure of invariant subspaces and irreducible subfactors (composition series) in the reducible case. We find out all intertwining operators and write down them both in a ” matrix” and an integral form. We determine when an invariant sesqui-linear form exists for the pair Tff,T,Ta 1)T1 (or their subfactors) and find out all unitarizable representations TaT or their subfactors. There are the following series of unitarizable representations: continuous series (2 real parameters), complementary series (2 real parameters), several ’’thin” series (which are ’’long” and ’’short”) (1 real parameter), ’’discrete” series (discrete infinite set), ’’exceptional” series (discrete infinite set, the set of weights has lower ’’dimension”).

The detailed description of these representations is necessary for requirements of harmonic analysis and quantization on the complex hyperbolic spaces SU(l,q)/U(l,q — 1).

For the groups SU(p,q), the study of representations of class 1 with respect to H = S(U(p,q — 1) x C/(l)) = U(p,q — 1) was undertaken in [5]. A description of representations of SU(p,q),p > 1, of the class u with respect to the same H was given in [8], partially it was done in [3]. The case of another H was considered in [7] for SU(n,n), H — SL(n,C) • R*. See also [7] for references.

Remark that for the pair SU(p,q)/U(p,q — 1) the case p = 1 has some peculiarities in comparing with the case p > 1: another weight lattice, 2 complex parameters against 1 complex and 1 integer.

1 The group SU(l,q) and its universal covering group

The group G = 577(1, g) consists of matrices g e SL(n, C), n — 1 + q, preserving the Hermitian form in C":

[x,y\ = -xiyl + x2y2 H---------t- xnyn.

'Supported by the Russian Foundation for Basic Research (grants No. 05-01-00074a, No. 05-01-00001a ), the Scientific Programs ’’Universities of Russia” (grant No. ur.04.01.465) and ’’Devel. Sci. Potent. High. School”, (Templan, No. 1.2.02).

We shall assume q > 1 (i.e. n > 2). We shall consider that G acts on Cn from the right and so we shall write vectors in the row form. Thus we have

where I = diag {—1,1,...,1}, the prime denotes matrix transposition. Let us write g € G in the block form with respect to the partition n = 1 + q:

so that a is a number.

The Lie algebra g of the group G consists of complex n x n trace zero matrices X satisfying the condition

where £ € M, ( € Herm (g), £ + tr ( = 0, r\ is a row-vector in Cg.

Let us consider the two following commuting automorphisms 9 and a of G (for the corresponding automorphisms of q we preserve notation 9 and a):

where J = diag {1,..., 1, —1}. Notice that both are inner, indeed, for example, 9(g) = TgT-1, where T = diag {A, —A,..., —A}, A = exp (inq/n). Let K and H denote the fixed point subgroups of 9 and cr, and t and t) their Lie algebras respectively. The coset spaces G/K and G/H are semisimple symmetric spaces belonging to the class of complex hyperbolic spaces. See, for example, [6] in connection with that. The Lie algebra decomposes into the direct sums of +1, — 1-eigenspaces of 9 and a respectively:

where u 6 R, ( £ Herm (q), u + tr £ = 0. It has one-dimensional centre Z(t) spanned by

g'lg = i,

(l.i)

x'l + IX = 0.

Let Herm (r) denote the space of Hermitian r x r matrices. A matrix X in 0 has the following block form (as (1.1)):

9{g) = Igl, a{g) = JgJ,

Q = t + p = fj + q.

The subgroup K consists of block diagonal matrices

(1.3)

where \^p\ — 1 and ip-det ip = 1. It is isomorphic to U(q) and is a maximal compact subgroup of G. The Lie algebra t consists of block diagonal matrices

(1.4)

(1.5)

Thus, G/K is a Hermitian symmetric space. The semisimple part = [6, 6] consists of matrices (1.4) with u = 0, so that tr£ = 0. The algebra 6 decomposes into the direct sum

l = Z{l)+№. (1.6)

Let be the subgroup of K consisting of matrices (1.3) with (p = 1 so that det t(j = I and Kis isomorphic to SU(q). Its Lie algebra is The subspace p consists of matrices

x_ ( 0 V

x~{n o

where r] is a row in C9.

Similarly the subgroup H consists of block diagonal matrices which are split into blocks according to the partition n = q + 1. Its Lie algebra f) has one-dimensional centre with a basis diag {i,... ,i, —qi}. So that G/H is a semi-Kahlerian symmetric space (in terminology of Berger [1], it means that it has G'-invariant complex structure and G-invariant, pseudo-Hermitian metric). The rank of G/H is equal to 1. It means that Cartan subspaces in q have dimension 1. As this one let us take the subspace a of q with the basis:

A) = , (1-7)

Here and further a matrix 3x3 denotes a matrix n x n written in the block form according to the partition n = 1 + (n — 2) + 1.

The Lie algebra 0 decomposes into the direct sum of root spaces of the pair (g, a):

0 = 0—2a + S—a + 00 + 0a + 02a-

The subspace go is the direct sum m + a where the subalgebra m consists of matrices

iu 0 0 \

0 iv ° ’ (1.8)

0 0 iu )

where u G R, v € Herm (n — 2), 2u + tr v = 0. We see that m lies in 6 and is the direct sum of two commuting subalgebras:

m = mo + mi,

where mo consists of matrices (1.8) with u — 0 (so that mo C 6^) and mi is the onedimensional centre RYo of m where

f i 0 0

Yq = 0 (-2i/(n - 2))E 0

\ 0 0 i

The nilpotent subalgebra 3 = 0-a + 0-2a corresponding to negative roots consists of matrices:

it £ it

t 0 t —it — £ —it

where t £ R, £ is a row in Cn~2.

Let Y denote the cone [y, y] = 0, y ^ 0. The group G acts on Y by translations: y yg.

Let S be the submanifold of Y defined by the equation |yi| = 1. It is the direct product of

the circle and the sphere: S — S1 x S2q~l. The group G acts on S as follows:

(1'9>

In particular, the group K acts by translations: s h» s ■ k = sk.

Let Lx be the corresponding action of g:

/>(•) = 5

f(s-exptX), Xeq. (1.10)

t=o

In particular, we have

Lx(sg) i = (sX)i,

I(«5)11 = Re (sX,s), (1.11)

where (x, y) denotes the following Hermitian form on C”:

(x,y)=xiyl. (1.12)

Let us take as a basic point of S the point

s° = (1,0,...,0,1).

The stabilizer of s° in g for the action (1.9) is mo + a + 3.

Let A, M, Mq , Mi, Z denote the analytic subgroups of G with the Lie algebras a, m, mo, mi, 3 respectively. For the action (1.9) the stabilizer of the point ,s° in G is M0AZ and in K is Mq, so that S = G/MqAZ and

S = K/M0.

Since Mi commutes with Mq, it acts on S from the left:

s°k 1 ^ mis°k = s°mik, (1-13)

in fact, it is the multiplication of a vector s by a number exp(w) if mi = exp uYq.

Introduce on S the coordinate a:

Sl = eia.

Let us take on S the Euclidean measure ds:

ds = dadv, (1-14)

where dv is the Euclidean measure on the unit sphere 52?_1, so that the volume of the whole S is equal

47rn

VOl S = 2n n2q = TT,--------7T,

r(n-l)

recall that the volume of the unit sphere in Rm is equal to

2tT™/2

0m = i>72) ■ (L15)

Under the action (1.9) the measure ds is transformed as follows:

ds = \(sg)i\~2q ds. (1-16)

In particular, this measure is invariant with respect to if. In section 5 we shall need the following formula of an ” integration by parts”:

ijj(s) (Lx<p){s) ds = 2q f Re (sX, s) ip(s) (p(s) ds — f (Lxifi)(s) ■ <p(s)ds (1-17) Js Js

s

for functions ip, ip invariant under the left action of M\.

Let G denote the universal covering group for the group G. Let n : G —)■ G be the natural projection. For a subgroup^of G we denote by the same letter with widetilde a corresponding analytic subgroup of G (i.e. having the same Lie algebra). In virtue of their simple connectedness the subgroups K^sS>,A7Z,Mq of G are isomorphic to the subgroups K(s\A,Z,Mo of G and can be identified with them respectively. We shall assume that.

The centre Z(G) of the group G is isomorphic to Z and isjsxp 2ttZXo, see (1.5). The kernel of the projection it is the subgroup D = exp 2nnZXo of G.

Lemma 1.1 The centre Z(G) contains in the subgroup M ofG:

Z(G) C M.

Proof: Let X be an arbitrary element of m, see (1.8). Decompose it according to (1.6):

X = aX0 + y, (1.18)

where Y € 6^. We have a = —(n/q)u and

/ 0 0 0 \

Y = | 0 iv + (iu/q)E 0 I .

\ 0 0 iun/q )

Let us take u so that a = 2nl, I 6 Z, i.e. u = —2irlq/n, then

/ 0 0 0 \

Y = I 0 iv — (2iril/n)E 0 I .

V 0 0 - 27Til )

For this element, we have in G:

/1 0 0

exp Y — I 0 exp (iv) ■ exp (—2iril/n) 0

\ 0 0 1

We see that^exp Y £ M0. But M0 = M0, hence exp Y G M0. Apply to (1.18) the map

exp : g —> G. We obtain

exp aXo = exp X ■ exp (—Y). (1-19)

If we take a as above, then both factors at the right hand side of (1.19) belong to M, so the left hand side belongs to M too. □

Let us denote S — K/Mq = K/Mq. It follows from the lemma that

K/M = K/M. (1.20)

The Iwasawa type decompositions of G and G are:

G = AZK, G = AZK = AZK,

i.e. any element g 6 G and any element g £ G can be written as

g = azk = exp tAo ■ zk (1-21)

g = azk = exp tAo • zk (1.22)

In each of these decompositions the elements a £ A are determined by g and g uniquely. Moreover, if g = 7r(g), then the parameter t for both decompositions (1.21), (1.22) is the same. This parameter can be found as follows. Apply (1.21) to s°:

s°g = e4 • s°k

whence using |si| = 1 we get

e^KAhl

and, therefore, for t from (1.22) we have

et = |(s‘MsO)1| (1-23)

2 Representations associated with the cone

Let ij,T £ C. Define the following characters (one-dimensional representations) of the subgroups A,K,MAZ:

uja(a) = eat,

uT(k) = eiTU,

uatT(fhaz) = uT(fh) u>a(a)

= eirueat,

where a = exp £ A, k = exp X £ K with X given by (1.4), m = exp X £ M with X

given by (1.8) (recall M C K). The subgroup MAZ of G is a maximal parabolic subgroup.

Let Ta<r denote the representation of G induced by the character u>atT of MAZ. It acts on the space Vcrj(G) of functions tp in C°°(G) satisfying the condition

ip(fhazg) = uja>T(fhaz) ip(g)

by right translations

Ta,r(g) <p(gi) = <p(gig)

Let us show the realization of TC<T on functions on S. For a function <p 6 Var(G), let us consider the following function ф on K\

ф(к) = <p(k) u)T(k)~l.

It satisfies the condition

ф(fhk) = ф(к)

so, by (1.20), defines a function on К/M. But it is more convenient instead of this function on K/M to consider the corresponding function on 5 = K/Mq which is invariant with respect to the left action of Mi, see (1.13). Namely, let us consider the following function f on S defined by

f(s) = f(s°k) = ф(к) = tp(k) u>T(fc)-1, where к = 7г(к). It satisfies the condition

f(Xs) = f(s), AeC, |A| = 1. (2.1)

The space of functions / in 'D(S) (for a manifold M, V(M) denote the Schwartz space of compactly supported C°°-functions with a usual topology) satisfying (2.1) will be denoted by V. Thus, Va>T(G) is isomorphic to V. In this realization the representation Ta>T looks as follows: _

(2V.rffl/)W=/(7^-) IMir5^, (2.2)

\{s9)iy uT(k)

where к is an element in К such that s = s°k, further, к is an element in К such that 7Г(k) = k, and at last g = 7r(g); the element k\ is determined by the Iwasawa decomposition

kg = aizih,

we used (1.23). It can be checked that the right hand side of (2.2) is well-defined, i.e. it does not depend on the choice of к and к for given s.

It is much more convenient to deal with the representation Ta<T of the Lie algebra g corresponding to Ta,T of the group G (we use the same symbol):

cZVlTpO/)(e) - (Lxf)(s) + \{{a + T)(sX, s) + (a- r)(s, SX)} f(s), (2.3)

where Xgg, see (1.2), Lx is the operator (1.10), the form (. , .) is given by (1.12).

The Hermitian form

(/i,/2) = I (2-4)

s

where ds is the measure (1.14) is invariant with respect to the pair Ta>T,T^*tr'

(Tff,T(X)f b/2) = -(h,Tf*,r(X)h); (2.5)

here and further

a* = 2 - 2n - a = -2q - cr (2.6)

(formula (2.5) is proved by means of (1.11), (1.16)).

3 Decomposition of the space V

In this section we decompose the space V into irreducible subspaces with respect to the action of the group K — S(U(1) x U(q)) = U(g) by translations:

R(k)ip(s) = <p(sk). (3.1)

We follow [5].

First consider the unit sphere S? — S2q~1 consisting of points (s2,--.,sn) 6 C9, satisfying the condition S2S2 + ... + snsn = 1, see section 1. As it is known [9], the space T>(S2) decomposes into the direct sum of the subspaces m £ N = {0,1,2,The

subspace V.(m) consists of the restrictions to S2 of homogeneous harmonic polynomials in x2,x2,. ■ •, xn,xn of degree m with complex coefficients. Denote by the same symbol T-L(m) the space of those polynomials. Let V (r), A(r), A(r) denote the spaces of all, analytic and antianalytic homogeneous polynomials of degree r respectively. Under the natural action of U(q), the spaces A(r) and .A(r) are irreducible, and the space 'H(m) decomposes into the subspaces

%(m, v) = [A(m — v) <g> .A(v)] D H(m),

v = 0,1,..., m. Let us denote by D(r), D(r),D(m, v) the corresponding representations of U(q) on A(r),A(r),'H(m, v) respectively.

Lemma 3.1 The representations D(m,v) of the group U(g) are irreducible and pairwise non-equivalent. The dimension of7i(m,v) is equal to

, , ^ T{m-v + q- 1) T(m + q- 1)

{m + q~1}

Proof. The highest weights of D(r) and D(r) are (r, 0,..., 0), (0,0,..., — r), respectively. The highest component D'(m,v) in D(m — v) ® D(v) has the highest weight

(m — w,0,..., 0, — v). (3.2)

As it is known [9],

V{m) = T-L(m) + Q ■ T-L(m — 2) where Q = x2x2 + ... + xnxn. Therefore,

A(m — v)® A(v) = Him, v) + Q ■ [A(m — v — 1) ® A(v — 1)]. (3.3)

The highest weight of the second term in the right hand side is (m — v - 1, 0,..., 0, — v + 1), it is lower than (3.2). Therefore, D'(m,v) C D(m,v). But the dimensions of both latter representations are equal to each other: one can check this statement calculating these dimensions by means of (3.3) and the Weyl formula, respectively. Thus, D(m,v) = D'(m,v) whence D(m,v) is irreducible. The non-equivalence follows from (3.2). □

Lemma 3.2 The restriction of the representations D(m,v) to the group SU(g) are irreducible. For q > 2, they are pairwise non-equivalent; for q = 2, the restrictions with the same m are equivalent (to the representation with the highest weight (m, 0)).

Indeed, the restriction of D(m, v) to the group S U(q) has the highest weight (m, v,...,v, 0).

□

Lemma 3.3 The space of functions in 'H(m,v) depending only on sn is one-dimensional. The normalized by 1 at the point (0,..., 0,1) basis function is ip(m, v; sn), where for m ^ 2v:

v; t) = c(m, v) tm~2v F(—v, m — v + q — 1; m — 2v -f 1; it)

and for m ^ 2v:

t) — ip(m,m — v;t), here F is the Gauss hypergeometric function ([2], Ch. 2),

T(m — v + 1) T(g — 1)

c(m,v) = (—1)”

r(m — 2v + 1) T(m + q — 1) ’

Proof. Let m ^ 2v. Then a polynomial <p(x) in T-l(rn: v) depending only on xn can be written as x™~2v f(a), where f(a) is a polynomial in a = xnxn of degree v. Since ip is harmonic, / has to satisfy the equation (hypergeometric):

a( 1 - a)^-4 + \m - 2v + 1 - (m - 2v + q)a] ~ + v(m -v + q- 1)/ = 0 daz da

and be regular at a = 0. □

As to the circle S\ = 51, see Section 1, the space V(S\) decomposes into the sum of one-dimensional spaces H(l), I € Z, spanned by s[.

Going to S — Si x 52, we have got that the space V(S) decomposes into the sum of the spaces %(l) <S> 'H(m,v). They are irreducible under K = U(q) and even K^ = SU(g).

Now let us decompose V (see Section 2). The condition (2.1) gives

I + m — 2v, (3.4)

whence

I = m (mod 2) (3-5)

and

|i| < m. (3.6)

Let us denote by A the set of pairs z = (l,m), l,m £ Z, which satisfy (3.5) and (3.6). These pairs will be called weights. Thus, if-irreducible subspaces which occur in the decomposition

of V can be labelled by weights z £ A. We shall write T-i{z) for T-L(l) ® 'H(m, v) with (3.4).

The corresponding representation of K and £ on T~L(z) will be denoted by R(z).

For each 'H(z), the subspace of functions in 'H(z) depending only on si and sn is onedimensional, the basis function ipz(si,sn) normalized by 1 at s° is for I ^ 0:

-ml + m + 2q-2

-------------;l + l;aia„ (3.7)

‘Ipzisi: $n) — c(l,m) SnF^ 2

and for I < 0:

sn) '*/’(—i;m)(sl> sn) (3-8)

where for I > 0:

c(l m) = (-l)(™-')/2__r((/ + m+.?.)/g)_E(lzl)___. (3 g)

^ j 1 j r(I + l)r(H + m + 2g-2)/2)' [ j

4 The structure of representations associated with the cone

For the study of the structure of the representations TCT)T (irreducibility, composition series etc.) we use the restriction to 6. Let 16!, see (1.4). Then by (2.3) we have

Ta,r(X) = Lx + iru. (4.1)

Remember the decomposition (1.6) and consider (4.1) when X belongs to one of two terms of (1.6).

For X = Xo, see (1.5), we have

Ta,T(X0) = LXo +

so that

T„,T(Xo) = -^(qr + nl) on H{z). (4.2)

For X € we have T„iT(X) = Lx, so that the restriction of Ta,T of G to the subgroup r(s) ^ if (s) » SU(q) does not depend on a, r and is the representation R of SU(ry) on V by translations, see (3.1).

Lemma 4.1 The restriction of the representation Ta>T to the Lie subalgebra t decomposes into the direct sum of irreducible pairwise non-equivalent representations R(z) on the spaces 7i(z), see Section 3.

The lemma follows from Lemma 3.2 and (4.2).

Since A0 commutes with m, see (1.7) and (1.8), the operator Ta!T(A0) preserves the space of functions depending on s\ = x,sn = y only. On such functions this operator has the following expression:

d d _d _d n ( d d _d _d\ !/to+I^+5» + ^“R<,(a:5) \xTx+y-Fy+xai + vW +

+ ^{(cr + r)xy + (a - r)xy}.

Apply it to i/)2, see (3.7), (3.8), (3.9). Omitting rather cumbersome calculations with hypergeometric functions, we write down the result.

Introduce the following 4 vectors on the plane R2:

ei = (l,l), e2 = (l,-l), e3 = (-l,l), 64 = (-1,-1)

and the following 4 linear functions /% of z:

Pi{ct,t;z) = a - t - I - m, (4.3)

/32 (cr,T-,z) = a-T~l+m + 2q-2, (4.4)

/?3 (a, t; z) = a + T + l + m, (4.5)

/34(a,r;z) = a + r + l + m + 2q —2. (4.6)

For i = 1, 2,3,4 we shall denote i' = 5 — i, so that e# = — e*. Notice that

/3i(a,T-z) + Pi>(a*,T-z) = -2.

Theorem 4.2 PFe have

4

2V,r(4)) 'Yi(z)Pi(criT; Z^z+ei (4.7)

2=1

where

7i(z) = (-1)<;

4(m + ^ — 1) ’

Remark that 71 (z) and 7.3(2) have no zeroes on Л (for Л, see Section 3) and 72 (z) and 74 (z) vanish on the rays I = rri and m — —I in Л, respectively.

We say that the representation Та>т links a weight z with a weight z1 if there exist X Є fl and f E'H(z) such that TatT(X)f is non-zero and belongs to 'H(z').

Lemma 4.3 The representation TaiT links z and z' if and only if z' = z + e{ with Pi(a, t; z) ф

0 for some i.

The lemma is proved similarly to the corresponding lemma from [5].

Let us call the line /3^ (cr, r; z) = 0 on the plane of z = (I, m) a barrier for Ta>T if it intersects

with Л for г = 1,3 and with Л fl {m > |£|} for і = 2,4. The barrier Д = 0 divides Л into

two parts: the set /Зі ^ 0 (the interior of the barrier) and the set /3* < 0 (the exterior of the barrier). If /Зі = 0 is a barrier, we shall sometimes speak for brevity: there is the barrier i. Let us indicate sets of pairs (a, r) for which there is the barrier і (see Fig. 1): barrier 1: a — r = 0, 2,4,...

barrier 2: a — т = —2q, —2q — 2,...

barrier 3: a + r = 0,2,4,...

barrier 4: a + т = —2</, —2q — 2,...

So we see that for given cr, r at most 2 barriers may happen.

If /Зі = 0 is a barrier then we denote by Vi the subspace of V which is the direct sum of 7i(z) with /Зі ^ 0.

The following theorem is a quick implication of Lemma 4.3.

Theorem 4.4 The subspaces V{ are invariant under Ta>T. Any irreducible subfactor ofTa<r is obtained by means of subspaces V% (i.e. any irreducible subfactor is V, or Vi, or Vi П Vj factorized over a sum of several V\)- If both a + r and a — т do not belong to 2Z; then Та>т is irreducible.

Fig. 1

In order to see the structure of invariant subspaces, irreducible subfactors, composition series etc., it is sufficient to draw barriers on the plane of z = (/, m) and to endow each barrier by a bristle (dashes) oriented inside of the barrier (where /3j ^ 0), see Fig. 2-5 for the cases with two barriers (Fig. 4, 5 are given for a > 1 — q).

Summarizing, we can write that when there are 2 barriers i,j the irreducible subfactors

Vi nvy, Vi/Vj', Vf/Vi, V/iVi + Vf), (4.8)

for brevity we write Vi/Vj instead of Vi/{Vi fl Vj), the sum in (4.8) means the arithmetic sum. In fact, some subfactors indicated in (4.8) can be absent (be trivial). It happens for the cases 11' and 22': then either the first, or the last one in (4.8) is trivial, and if, moreover, a = —q,

a = -q, t = q,q + 2,..., (4.9)

or

a = -q, t = -q, -q - 2,..., (4.10)

then both are trivial, so that V decomposes into the direct sum

V = Vx + Vv or V = V2 + V2> (4.11)

respectively to (4.9) or (4.10).

The irreducible subspace Vt n Vv for a = 1 - q has weights z lying on one ray, i.e. on coinciding barriers i,i'. Similarly it goes for a = -1 - q for the subfactor V/(Vt + V^). Let us call such subfactors exceptional.

The subfactors VinVy and V/^+Vy) are finite-dimensional. In particular, for a — r = 0 and a* = t = 0 they realize the unit representation of G (and G).

О

Fig. 2

О

Fig. 3

Fig. 4 Fig. 5

For the subfactor W, let A(W) denote the set of weights z G A occuring in W. For an invariant subspace U under Tff,r, denote by its orthogonal complement with respect to form (2.4). Then A(?7X) = A \ A(U) and UL is invariant under T?*if. For a subfactor W = UjZ of Ta<T the subspace W* = Z^/U1- is a subfactor of We have A(PF*) = A(W).

Let us call W* the dual subfactor. The dual subfactors for Vi and VjV1 are V/Vti and V# respectively, and for the subfactors (4.8) are V/(Vii +V3), Vj/Vi/Vj. Vt> D Vj respectively.

5 Intertwining operators

A continuous operator A on V is said to intertwine representations Ta T and Tai iTl if for any X 6 0 we have

Taun{X)A = AT„tT{X). (5.1)

A similar definition is given for the case when an operator A maps a subfactor W of TCiT in a subfactor W\ of Tar ,n •

In this section we find out all such operators and show ’’the main of them” in an integral form.

Theorem 5.1 A non-zero intertwining operator A as above exists only in the following cases:

(а) o-i = a, T\ — t;

(б) CTi = a*, n = r.

In the case (a) such an operator is a scalar operator (the multiplication by a number) on

V except of (4.9), (4.10), then A is a scalar operator on each term in (4.11).

In the case (b) for any irreducible subfactor W ofTa/r there exists a unique up to the factor non-zero operator mapping W onto the dual subfactor W* and intertwining the subfactor of T^t on W with the subfactor of TCT* >T on W*.

Proof. Let A be a continuous operator on V satisfying (5.1). It follows from Lemma 4.1, that A preserves each T-L(z) and its restriction to T-L(z) is the multiplication by a number a(z). In particular, we have

A^z = a{z)tpz. (5.2)

The numbers a(z) depend on a, r, oi,t\. Take in (5.1) X = Xo, see (1.5), and use (4.2), we just obtain that t\ = t. Now take in (5.1) X = Aq, see (1.7), apply to and use (4.7), then we obtain relations for a(z):

/3i(a, r; z) a(z + e*) = $(<71, r; z) a(z),

(5.3)

i = l,2, 3,4. These relations have to be co-ordinated, i.e. if one applies to a(z + et) the same relation with i replaced by i\ then one has to get a(z) again. It gives the conditions:

Pi (a, t-z + ev) Pv (cr, r; z) a{z) = Pi (<7i , t; z + e*») Pi' (cri, r; z) a(z).

(5.4)

Since A ^ 0, there exists at least one z such that a(z) ^ 0. Omitting this a(z) from (5.4) and substituting the explicit expressions (4.3) — (4.6), we obtain that a square trinomial (in a) whose roots give the sum —2q has the same values at <7 and <7i. Therefore, we have a\ = a or <71 = —a — 2q (= cr*).

First let a\ = a. The relations (5.3) shows that a(z + e2) = a(z) for that z and i for which Pi(a, r; z) 7^ 0. This fact together with the same one for i' gives that a(z) should be constant along any line z + tet and, therefore, on the whole A, except the case (4.9), (4.10) when two barriers 11' or 22' split A into two non-linked subsets, so that we have decompositions (4.11).

Now let <7i = a*. Then (5.3) is

Pi(a,r;z) a(z + e*) = Pi{a*,r;z) a(z).

(5.5)

Let W — U/Z be some irreducible subfactor of Ta<r. Similarly to [4] we can show that equations (5.5) have a unique up to the factor solution on A(U) which is the intersection of the interiors of some barriers and, moreover, a(z) = 0 on A(U) fl A(Z). This solution a(z) defines an operator on U vanishing on U fl Z so that it maps U/Z onto Z^/UL and intertwines corresponding subfactors of Ta>T and Ta* jT. Similarly to [4] it is shown that this operator is continuous. □

Denote by z^,T where z, X £ C, r 6 M, the following function of z: ii z = reia, r > 0, 0 ^ cl < 2ir, then

If z = z(u) is some curve, then

d_

du

r\,T

u=0

|zX’T (

_/ „/

A Re----------1- ir Im —

z z

(5.6)

u=0

Now let us consider the operator AatT on V which is defined by integral:

(Aa,Tf)(s) = [ Js

f s t

Is Sl tx_

f(t) dt.

(5.7)

The integral converges absolutely for Re <7 < —n + 1 and is extended to other cr, r by the analyticity.

Theorem 5.2 The operator A<jtT intertwines Ta^T and Ta*tT. On the subspaces H(z) it is the multiplication by the numbers

a(a,T-,z) =4{-l)mirnei

T(1 — n — cr)T(2 — n — (a + t)/2)T(2 — n — (a — r)/2)

El r(—(1/2)Pi{<r,T\z)) i-1

(5.8)

Proof. First we show that Aa,T is an intertwining operator. Denote by A(s, t) the kernel (a function) of this operator.

Let X £ 0. According to (1.9) we write s = s ■ exp uX, t = t ■ exp uX. Denote a = (sX,s), b = (tX,t), see (1.12). Apply (5.1) with o\ = a*, t\ = r to a function / € V. Then the left hand side is:

J ~ A(s, t)f(t) [(a* + r)a + (a* - r)a] j A(s, t) ip(t) dt and the right hand side is:

J A{s,t) j (£*/)(£) + ^ [{a + r)b + {a - r)b\ /(t)| dt.

By (1.17) the difference between (5.9) and (5.10) is equal to

A(s,t)+ A(s,t) [—2gRe6+

i

+ ^((a* + r)a + (a* -r)a) - i((a + T)b + (a - r) 6)] J f(t)dt

(5.9)

(5.10)

Hi

u=0

(5.11)

since

d_

du

u—0

(A(s,t) + A{s,t)) = —

A{s,t).

By (5.6) we have

du

u=0

Ap’r) = Tu

[s,t]

jf=o { (sg)i(tg)l J = A(s, t) {-cr*Re(a + b) — irlm(a + 6)}.

Substituting it in (5.11), we obtain that the integrand of (5.11) is A(s,t)f(t) multiplied by

-a* Re (a + b) - irlm(a + b) - 2qReb + ^ [(a* + r)a + (a* - r)a] - ^ [(<7 + r)b + [a - r)6]

which is equal to 0 in virtue of (2.6). It proves that our operator is intertwining.

Therefore, as it was said in the proof of Theorem 5.1, the operator Aa^ on the subspace H(z) is the multiplication by a number a(cr, r; z), or a(z) for brevity.

First we calculate a(0). Set in (5.2) A = ACjT, z = 0 (so that tpz = 1) and take it at s = s°. Then

fl(°) = j {-i + tn/hY''T dt.

s

Taking t\ = eta. tn = re1' and integrating in other variables, we obtain

27r 2ir 1

a(0) =V.2q-2 I j J (—1 + eiare~ll3y ,T (I — r2)q~2 r dr dadf3

ooo

2tt 1

= 27rJ72g-2 j j ( —1 + relvy ,r (1 - r2)q~2 r dr dv.

0 0

The latter integral is calculated by means of the change

reiv - 1 = pe^

(so that | < 7 < and formula [2] 1.5 (30). Finally we obtain

a(0) =47xn eiT7T ______________r(l - n - a)--------

( ) F((—a — r)/2) T((—cr + t)/2)

Now, starting from a(0) and applying the formula

Pi W*,t;z)

a(z + ег) = a(z)

Pi (<*, t; z)

(which is (5.3) with — a*) first (/ + m)/2 times with i = 1 and then (m —1/2 times with

i = 3, we obtain (5.8). □

By means of operator (5.7) we can write down for each irreducible subfactor W = U/Z an intertwining operator on the dual subfactor W*, see Theorem 5.1. Namely, it is the first non-zero Laurent, or Taylor, coefficient at A = a of the operator A\)T on U considered as a function of A.

6 Unitarity

A sesqui-linear form on V is called invariant with respect to the pair T<r)T,T„liTl, if

for any X € g and any f,fiEV we have

tfCzvpo/,/!) +H(f,Tai,T1(x)f1) = o. (6.i)

Similar we define the invariance of a form on a pair of invariant subfactors W/U and Wi/U\\ then / G WJi € W! and = 0 if / G U or h e Ux.

Theorem 6.1 A non-zero sesqui-linear form H(f,fi) invariant with respect to the pair

TCT,r, Tai )Tl on V or on a pair of invariant subfactors exists only in the following cases:

(а) o\ = a*, ri = r;

(б) (ji =a, T\ = r.

In the case (a) the form H coincides up to the factor with the form (2.4) except (4.9) and

(4.10). In the latter cases the form H coincides up to the factor with (2.4) on each of sub spaces

(4.11).

In the case (b) the form H can be expressed in terms of the form (2.4) and the operator A intertwining TaT and Tff- jT or their subfactors, see Theorem 5.1, namely

H(f,f1) = (AfJ1).

Proof. First set in (6.1) X = Xq and take /, f\ £ H(z), then by (4.2) we obtain ri = r. Now let X range №. Then (6.1) gives that H is invariant with respect to the representation R of K= SU(q). Therefore the subspaces ^.(z) are orthogonal with respect to the form H (for q = 2, one has else to remember Xq and (4.2) and to obtain the orthogonality for different I).

Therefore, H has a ” diagonal form”:

#(/, fi) = h(z) (/,/i) (6.2)

if both / and /i belong to 'H(z) and H(f. f \) = 0 if /, f\ belong to different 'H(z). Further the study of h(z) goes similarly to the study of a(z) in the proof of Theorem 5.1, so we omit it. □

Now determine when representations Ta T or their subfactors are unitarizable. For that we have to set — a and T\ — t in Theorem 6.1 and the form H(f, f\) has to be Hermitian (H(/, /) real) and positively definite. For r we obtain r€l. For a we obtain two possibilities: a = a* and a — a. In the first case (a = a*) we have Re o = 1 — n = —q. So we obtain a series of unitarizable representations Ta>r of G (or jj), the invariant inner product is (2.5), so that the unitary completion acts on the space of functions / in L2(S) satisfying (2.1). Let us call this series the continuous series. Representations of this series are irreducible except the split case (4.9), (4.10).

Let now consider the second case: a = a, i.e. First consider the irreducible case.

Theorem 6.2 An irreducible representation Ta>T with a, r G R is unitarizable for points on the plane (a, r) which fill in one "big” square jcr + gr| H- |r) < q (with the diagonal of length 2q) and a family of ’’small” squares: \a + q\ + \t ± (q + 2k + 1)| < 1, k G N (with the diagonal of length 2). The invariant inner product is c(a,r) (j4<t,t/, /1), where Aa^T is the operator from section 5, (.,.) is the form (2.4) and c(a,r) = a(a, r;0)_1.

Let us call the family of representations pointed out in this theorem the complementary series.

Proof. As well as a(z) in section 5, the factor h(z) from (6.2) satisfies the equation:

(3i(cr,T;z) h(z + ej) = Pi(a*,T-,z) h(z). (6.3)

So we have to learn when equations (6.3) have a positive solution h(z). For that it is necessary and sufficient that for each i both functions /32(cr. r; z) and /32(a*, t; z) considered as functions of z = (I, m) are of one sign on A. In turn, for that it is necessary and sufficient that both intervals (cr — r, a* — r) and (a + r, a* + r) on R contain no points of 2N. Hence we obtain the cases indicated in the theorem. □

Now let us turn to the reducible case: o ± r £ 2Z. Prom (6.3) we see that if h(z) is defined outside some barrier then h(z) = 0 inside of it. Therefore, we have to look for invariant positively definite Hermitian forms on irreducible subfactors. Omitting rather tiresome treatments, let us give the result.

Theorem 6.3 Unitarizable representations on irreducible subfactors of Ta,T form the following series:

(a) long thin series (labelled by one parameter ranging a ray on the real line) on invariant subspaces:

V\ for a — r = 0, a < 0 and for a — t = 2,4, 6,..,, a < — q + 1;

Vs for cr + r = 0, <t < 0 and for a + r = 2,4, 6,..., a < — q + 1; and, in the duality, on invariant factor-spaces:

V/V4 for a* — t = 0, cr* < 0 and for a* — r = 2,4,..., a* < —q + 1;

V/V2 for a* + r = 0, a* < 0 and for a* + t = 2,4,..., a* < — q + 1;

(b) short thin series (labelled by one parameter ranging an interval on the real line) on invariant subspaces:

V2 for a* + r = 0,2,4,..., — q — 1 < a* < r;

V4 for a* — t — 0,2,4,..., — q — 1 < a* < —t; and, in the duality, on invariant factor-spaces:

V/Vj for a + t = 0, 2,4,..., — q — 1 < a < t;

V/V1 for a — t = 0,2,4,..., — q — 1 < a < —t;

(c) ’’discrete” series: these representations correspond to integer even points (a,r £ 2Z) lying on the right and the left angles (i.e. a ^ |r| and a* ^ |r| respectively):

V/(Vi + Vz) for a ^ \t\, and, in the duality,

V2 H V4 for a* ^ |t|.

(d) ’’exceptional” series: these representations correspond to integer even points

(a,r £ 2Z) at the ends of the thin series (the weights z for each of these representations

lie on a ray), namely, in the right angle (a ^ |r| ):

onV1/V3 fora = r, on V3/V1 for a = —t, in the left angle (a* ^ \t\) (dual to the right angle): on V2/V4 for a* = t, on V4/V2 for a* = —t, in the upper angle (t ^ \a + q\ + q):

onV2 (~l V3 for a = — q + 1,

on V/(V2 + V3) for a = — q — 1 (dual to the preceding case);

in the lower angle (t ^ — \a + q\ — q):

on V\ fl V4 for a = —q + 1,

on V/(Vi + V4) for a — — q — 1 (dual to the preceding case);

(e) the unit representation - in the right and the left vertex of the big square:

on V\ D V3 for <7 = 0, r = 0,

on V/(V2 + Vi) for a* = 0, r = 0.

REFERENCES

1. M. Berger. Les espaces symetriques non-compacts. Ann. Sci. Ec. Norm. Super., 1957, tome 74, 85-177.

2. A. Erdelyi et al. Higher Transcendental Functions, vol. I. McGraw-Hill, New York, 1953.

3. A. U. Klimyk, B. Gruber. Structure and matrix elements of the degenerate series representations of U(p+q) and U(p,q) in a U(p)xU(q) basis. J. Math. Phys., 1982, vol. 23, No. 8, 1399-1408.

4. V. F. Molchanov. Representations of a pseudo-orthogonal group associated with a cone. Matem. Sbornik, 1970, tom 81, No. 3, 358-375. Engl, transl.: Mat. USSR Sbornik, 1970, vol. 10, No. 3, 333-347.

5. V. F. Molchanov. Representations of a pseudo-unitary group associated with a cone. In: Functional analysis, Spectral Theory, Ulyanovsk State Pedagogical Institute, Ulyanovsk, 1984, 55-66.

6. V. F. Molchanov. Harmonic analysis on homogeneous spaces. In: Itogi nauki i tekhn. Sovr. probl. matem. Fundam. napr., tom 59, VINITI, 1990, 5-144. Engl, transl.: Encycl. Math. Sci., Springer , Berlin etc., vol. 59, 1995, 3-136.

7. V. F. Molchanov. Maximal degenerate series representations of the universal covering of the group SU(n,n). Inst. Mittag-Leffler, 1995/96, Report 16, 21 p.

8. Yu. A. Sharshov. Representations of the group SU(j3, q) associated with a cone. Math. Inst. Univ. of Leiden, Leiden, The Netherlands, 1996, Report W 96-03, 14 p.

9. N. Ja. Vilenkin. Special Functions and the Theory of Group Representations. Moscow: Nauka, 1965. Engl, transl.: Transl. Math. Monographs, vol. 22, Amer. Math. Soc., Providence, R.I., 1968.