lecture 2. tensor unfoldings

From Matrix to Tensor:The Transition to Numerical Multilinear Algebra

Lecture 2. Tensor Unfoldings

Charles F. Van Loan

Cornell University

The Gene Golub SIAM Summer School 2010Selva di Fasano, Brindisi, Italy

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 2. Tensor Unfoldings

What is this Lecture About?

A Common Framework for Tensor Computations...

1. Turn tensor A into a matrix A.

2. Through matrix computations, discover things about A.

3. Draw conclusions about tensor A based on what is learnedabout matrix A.

Let us begin to get ready for this...



Recurring themes...

Given tensor A ∈ IRn1×···×nd , there are many ways to assemble itsentries into a matrix A ∈ IRN1×N2 where N1N2 = n1 · · · nd .

For example, A could be a block matrix whose entries are A-slices.

A facility with block matrices and tensor indexing is required tounderstand the layout possibilities.

Computations with the unfolded tensor frequently involve theKronecker product. A portion of Lecture 3 is devoted to thisimportant “bridging the gap” matrix operation.



Example: Gradients of Multilinear Forms

If f :IRn1 × IRn2 × IRn3 → IR is defined by

f (u, v ,w) =n1∑

i1=1

n2∑i2=1

n3∑i3=1

A(i1, i2, i3)u(i1)v(i2)w(i3)

and A ∈ IRn1×n2×n3 , then its gradient is given by

∇f (u, v ,w) =

A(1) w ⊗ v

A(2) w ⊗ u

A(3) v ⊗ u

where A(1), A(2), and A(3) are matrix unfoldings of A.



Tensor Unfoldings of A ∈ IR4×3×2

A(1) =

2664a111 a121 a131 a112 a122 a132

a211 a221 a231 a212 a222 a232

a311 a321 a331 a312 a322 a332

a411 a421 a431 a412 a422 a432

3775A(2) =

24 a111 a211 a311 a411 a112 a212 a312 a412

a121 a221 a321 a421 a122 a222 a322 a422

a131 a231 a331 a431 a132 a232 a332 a432

35A(3) =

»a111 a211 a311 a411 a121 a221 a321 a421 a131 a231 a331 a431

a112 a212 a312 a412 a122 a222 a322 a422 a132 a232 a332 a432

–


Where We Are

Lecture 1. Introduction to Tensor Computations

Lecture 2. Tensor Unfoldings

Lecture 3. Transpositions, Kronecker Products, Contractions

Lecture 4. Tensor-Related Singular Value Decompositions

Lecture 5. The CP Representation and Rank

Lecture 6. The Tucker Representation

Lecture 7. Other Decompositions and Nearness Problems

Lecture 8. Multilinear Rayleigh Quotients

Lecture 9. The Curse of Dimensionality

Lecture 10. Special Topics


Block Matrices

Definition

A block matrix is a matrix whose entries are matrices.

A 3-by-2 Block Matrix...

A =

A11 A12

A21 A22

A31 A32

Aij ∈ IR50×70

Regarded as a matrix of scalars, A is 150-by-140.


Block Matrices

The blocks in a block matrix do not have to have uniform size.

A =

A11 A12

A21 A22

A31 A32

=

50-by-70 50-by-10

50-by-70 50-by-10

20-by-70 20-by-10


Block Matrices: Special Cases

A Block Column Vector...

A =

A1

A2

A3

Ai ∈ IR100×50

A Block Row Vector...

A =[

A1 A2 A3

]Ai ∈ IR100×50


Block Matrices: Special Cases

A Column-Partitioned Matrix...

A =[

a1 a2 a3

]ai ∈ IR100

A Row-Partitioned Matrix

A =

aT1

aT2

aT3

ai ∈ IR100


Block Matrices: Addition

A 3-by-2 Example...

B11 B12

B21 B22

B31 B32

+

C11 C12

C21 C22

C31 C32

= B11 + C11 B12 + C12

B21 + C21 B22 + C22

b31 + C31 B32 + C32

The matrices must be partitioned conformably.


Block Matrices: Multiplication

An Example... B11 B12

B21 B22

B31 B32

[C11 C12

C21 C22

]

= B11C11 + B12C21 B11C12 + B12C22

B21C11 + B22C21 B21C12 + B22C22

B31C11 + B32C21 B31C12 + B32C22

The matrices must be partitioned conformably.


Block Matrices: Transposition

An Example... B11 B12

B21 B22

B31 B32

T

=

[BT

11 BT21 BT

31

BT12 BT

22 BT32

]

Different from Block Transposition... B11 B12

B21 B22

B31 B32

[T ]

=

[B11 B21 B31

B12 B22 B32

]

(More on this later.)


Recursive Block Structure

A block matrix can have block matrix entries.

A =

A11 A12

A21 A22

A11 =

A1111 A1112

A1121 A1122

A12 =

A1211 A1212

A1221 A1222

A21 =

A2111 A2112

A2121 A2122

A22 =

A2211 A2212

A2221 A2222

Example: The Discrete Fourier Transform Matrix Fn

F2mP =

[Fm Ω · Fm

Fm −Ω · Fm

]P = permutation, Ω = diagonal


The vec Operation

Turns matrices into vectors by stacking columns...

X =

1 102 203 30

⇒ vec(X ) =

123102030


Matlab: Vec of a matrix using the reshape function.

% If A is a matrix , then the following

% assigns vec(A) to a ...

[n1 ,n2] = size(A);

a = reshape(A,n1*n2 ,1);

It can be shown that

a(i1 + (i2 − 1)n1)

=

A(i1, i2)


The vec Operation

Turns tensors into vectors by stacking mode-1 fibers...

A ∈ IR2×3×2 ⇒ vec(A) =

A(:, 1, 1)

A(:, 2, 1)

A(:, 3, 1)

A(:, 1, 2)

A(:, 2, 2)

A(:, 3, 2)

=

a111

a211

a121

a221

a131

a231

a112

a212

a122

a222

a132

a232

We need to specify the order of the stacking...


Tensor Notation: Subscript Vectors

Reference

If A ∈ IRn1×···×nd and i = (i1, . . . , id) with 1 ≤ ik ≤ nk for k = 1:d ,then

A(i) ≡ A(i1, . . . , ik)

We say that i is a subscript vector. Bold font will be useddesignate subscript vectors.

Bounds

If L and R are subscript vectors having the same dimension, thenL ≤ R means that Lk ≤ Rk for all k.

Special Cases

A subscript vector of all ones is denoted by 1. (Dimension clearfrom context.) If N is an integer, then N = N · 1.


The Index-Mapping Function col

Definition

If i and n are length-d subscript vectors, then the integer-valuedfunction col(i,n) is defined by

col(i,n) = i1 + (i2− 1)n1 + (i3− 1)n1n2 + · · ·+ (id − 1)n1 · · · nd−1

The Formal Specification of Vec

If A ∈ IRn1×···×nd and v = vec(A), then

a (col(i,n)) = A(i) 1 ≤ i ≤ n


Matlab: Vec of a tensor using the reshape function

% If A is a n(1) x...x n(d), then the

% following assigns vec(A) to a ...

n = size(A);

a = reshape(A,prod(n),1);

If A ∈ IRn1×···×nd and a = vec(A) then

a(i1 + (i2 − 1)n1 + (i3 − 1)n1n2 + ... + (id − 1)n1 · · · ... · · · nd−1)

=

A(i1, i2..., id)


Parts of a Tensor

Fibers

A fiber of a tensor A is a vector obtained by fixing all but one A’sindices. For example, if A = A(1:3, 1:5, 1:4, 1:7), then

A(2, :, 4, 6) = A(2, 1:5, 4, 6) =

A(2, 1, 4, 6)A(2, 2, 4, 6)A(2, 3, 4, 6)A(2, 4, 4, 6)A(2, 5, 4, 6)

is a fiber. We adopt the convention that a fiber is a column-vector.

Problem 2.1. How many fibers are there in the tensor A ∈ IRn1×···×nd?


Matlab: Fiber Extraction

n = [3 5 4 7];

A = randn(n);

v0 = A(2,:,4,6); % v0(1,j,1,1) = A(2,j,4,6)

v1 = squeeze(v0); % v1(j) = A(2,j,4,6)

[r,c] = size(v1);

if r==1

v = v1 ’;

else

v = v1;

end

The function squeeze removes “singleton dimensions”. In the above, v0

is a fourth-order tensor that has dimension 1 in modes 1, 3, and 4.

Sometimes squeeze produces a row vector. The if-else is necessary

because (by convention) we insist that fibers are column vectors.


Problem 2.2. Write a recursive Matlab function alpha =

Tensor1norm(A) that returns the maximum value of ‖ f ‖1 where f is afiber of the input tensor A ∈ IRn1×···×nd .


Parts of a Tensor

Slices

A slice of a tensor A is a matrix obtained by fixing all but two ofA’s indices. For example, if A = A(1:3, 1:5, 1:4, 1:7), then

A(:, 3, :, 6) =

A(1, 3, 1, 6) A(1, 3, 2, 6) A(1, 3, 3, 6) A(1, 3, 4, 6)A(2, 3, 1, 6) A(2, 3, 2, 6) A(2, 3, 3, 6) A(2, 3, 4, 6)A(3, 3, 1, 6) A(3, 3, 2, 6) A(3, 3, 3, 6) A(3, 3, 4, 6)

is a slice. We adopt the convention that the first unfixed index inthe tensor is the row index of the slice and the second unfixedindex in the tensor is the column index of the slice.

Problem 2.3. How many slices are there in the tensor A ∈ IRn1×···×nd ifn1 = · · · = nd = N?


Matlab: Slice Extraction

n = [3 5 4 7];

A = randn(n);

C0 = A(2,:,4,:); % C0(2,i,4,j) = A(2,i,4,j)

C = squeeze(C0); % C(i,j) = A(2,i,4,j)

In the above, C0 is a fourth-order tensor that has dimension 1 in modes 1

and 3. C is a 5-by-7 matrix, a.k.a., 2nd order tensor.

Problem 2.4. Write a recursive Matlab function alpha = MaxFnorm(A)

that returns the maximum value of ‖ B ‖F where B is a slice of the inputtensor A ∈ IRn1×···×nd .


Subtensors

Subscript Vectors

If i = [ i1, . . . , id ] and j = [ j1, . . . , jd ] are integer vectors, theni ≤ j means that ik ≤ jk for k = 1:d .

Suppose A ∈ IRn1×···×nd with n = [ n1, . . . , nd ]. If 1 ≤ i ≤ n, then

A(i) = A(i1, . . . , id).

Specification of Subtensors

Suppose A ∈ IRn1×···×nd with n = [ n1, . . . , nd ]. If1 ≤ L ≤ R ≤ n, then A(L:R) denotes the subtensor

B = A(L1:R1, . . . , Ld :Rd)


Matlab: Assignment to Fibers, Slices, and Subtensors

n = [3 5 4 7];

A = zeros(n);

A(2,5,:,6) = ones (4 ,1);

A(1,:,3,:) = rand (5 ,7);

A(2:3 ,4:5 ,1:2 ,5:7) = zeros (2,2,2,3);

Problem 2.5. Write a Matlab function A = Sym4(N) that returns arandom 4th order tensor A ∈ IRN×N×N×N with the property thatA(i) = A(j) whenever j(1:2) is a permutation of i(1:2) and j(3:4) is apermutation of i(3:4).


The Matlab Tensor Toolbox

Why?

It provides an excellent environment for learning about tensorcomputations and for building a facility with multi-index reasoning.

Who?

Bruce W. Bader and Tammy G. Kolda, Sandia Laboratories

Where?

http://csmr.ca.sandia.gov/ tildetkolda/TensorToolbox/


Matlab Tensor Toolbox: A Tensor is a Structure

>> n = [3 5 4 7];

>> A_array = randn(n);

>> A = tensor(A_array );

>> fieldnames(A)

ans =

’data’

’size’

The .data field is a multi-dimensional array. The .size field is an

integer vector that specifies the modal dimensions. In the above, A.data

and A array have same value and A.size and n have the same value.


A First Example

The Frobenius norm of a 3-tensor

function sigma = NormFro3(A)n = A.size;s = 0;for i1=1:n(1)

for i2=1:n(2)for i3=1:n(3)

s = s + A(i1,i2,i3)^2;end

endends = sqrt(s);


Matlab Tensor Toolbox: Order and Dimension

>> n = [3 5 4 7];

>> A_array = randn(n);

>> A = tensor(A_array)

>> d = ndims(A)

d =

4

>> n = size(A)

n =

3 5 4 7


Problem 2.6. If A is a d-dimensional array in Matlab, then what issum(A)? Explain why the following function returns the Frobenius norm ofthe input tensor.

function sigma = NormFro(A)

% A is a tensor.

% sigma is the square root of the sum of the

% squares of its entries.

B = A.data.^2;

for k=1:ndims(A)

B = sum(B);

end

sigma = sqrt(B);


Matlab Tensor Toolbox: Tensor Operations

n = [3 5 4 7];

% Extracting Fibers and Slices

a_Fiber = A(2,:,3,6); a_Slice = A(3,:,2,:);

% Familiar initializations ...

X = tenzeros(n); Y = tenones(n);

A = tenrandn(n); B = tenrandn(n);

% These operations are legal ...

C = 3*A; C = -A; C = A+1; C = A.^2;

C = A + B; C = A./B; C = A.*B; C = A.^B;

% Applying a Function to Each Entry ...

F = tenfun(@sin ,A);

G = tenfun(@(x) sin(x)./exp(x),A);


Problem 2.7. Suppose A = A(1:n1, 1:n2, 1:n3). We say A(i) is an interiorentry if 1 < i < n. Otherwise, we say A(i) is an edge entry. If A(i1, i2, i3) isan interior entry, then it has six neighbors: A(i1 ± 1, i2, i3),A(i1, i2 ± 1, i3),and A(i1, i2, i3 ± 1). Implement the following function so that it performsas specified

function B = Smooth(A)

% A is a third order tensor

% B is a third order tensor with the property that each

% interior entry B(i) is the average of A(i)’s six

% neighbors. If B(i) is an edge entry, then B(i) = A(i).

Strive for an implementation that does not have any loops.

Problem 2.8. Formulate and solve an order-d version of Problem 2.7.


All Tensors Secretly Wish that They Were Matrices!

Example 1. A = A(1:n1, 1:n2, 1:n3)

ima matrix =[A(1, : , : ) · · · A(n1, : , : )

]ima matrix =

[A( : , 1, : ) · · · A( : , n2, : )

]ima matrix =

[A( : , : , 1) · · · A( : , : , n3)

]



Example 2. A = A(1:n1, 1:n2, 1:2, 1:2, 1:2)

ima matrix =

A( : , : , 1, 1, 1)

A( : , : , 2, 1, 1)

A( : , : , 1, 2, 1)

A( : , : , 2, 2, 1)

A( : , : , 1, 1, 2)

A( : , : , 2, 1, 2)

A( : , : , 1, 2, 2)

A( : , : , 2, 2, 2)



Example 3. A = A(1:n1, 1:n2, 1:n3, 1:n4)

ima matrix =

A( : , : , 1, 1) A( : , : , 1, 2) · · · A( : , : , 1, n4)

A( : , : , 2, 1) A( : , : , 2, 2) · · · A( : , : , 2, n4)...

.... . .

...

A( : , : , n3, 1) A( : , : , n3, 2) · · · A( : , : , n3, n4)

A(i , j , k, `) is the (i , j) entry of block (k, `)


Tensor Unfoldings

What are they?

A tensor unfolding of A ∈ IRn1×···×nd is obtained by assemblingA’s entries into a matrix A ∈ IRN1×N2 where N1N2 = n1 · · · nd .

Obviously, there are many ways to unfold a tensor.

An important family of tensor unfoldings are the mode-kunfoldings.

In a mode-k unfolding, the mode-k fibers are assembled to producean nk -by-(N/nk) matrix where N = n1 · · · nd .

The tensor toolbox function tenmat can be used to produce modalunfoldings and other, more general unfoldings.


Modal Unfoldings Using tenmat

Example: A Mode-1 Unfolding of A ∈ IR4×3×2

tenmat(A,1) sets upa111 a121 a131 a112 a122 a132

a211 a221 a231 a212 a222 a232

a311 a321 a331 a312 a322 a332

a411 a421 a431 a412 a422 a432

(1,1) (2,1) (3,1) (1,2) (2,2) (3,2)

Notice how the fibers are ordered.




tenmat(A,2) sets up a111 a211 a311 a411 a112 a212 a312 a412

a121 a221 a321 a421 a122 a222 a322 a422

a131 a231 a331 a431 a132 a232 a332 a432

(1,1) (2,1) (3,1) (4,1) (1,2) (2,2) (3,2) (4,2)





tenmat(A,3) sets up[a111 a211 a311 a411 a121 a221 a321 a421 a131 a231 a331 a431

a112 a212 a312 a412 a122 a222 a322 a422 a132 a232 a332 a432

](1,1) (2,1) (3,1) (4,1) (1,2) (2,2) (3,2) (4,2) (1,3) (2,3) (3,3) (4,3)




Precisely how are the fibers ordered?

If A ∈ IRn1×···×nd , N = n1 · · · nd , and B = tenmat(A,k), then Bis the matrix A(k) ∈ IRnk×(N/nk ) with

A(k)(ik , col(ik, n)) = A(i)where

ik = [i1, . . . , ik−1, ik+1, . . . , id ]

nk = [n1, . . . , nk−1, nk+1, . . . ,md ]

Recall that the col function maps multi-indices to integers. For example,if n = [2 3 2] then

i (1, 1, 1) (2, 1, 1) 1, 2, 1) (2, 2, 1) (1, 3, 1) (2, 3, 1) (1, 1, 2) (2, 1, 2) ...col(i, n) 1 2 3 4 5 6 7 8 ...


Matlab Tensor Toolbox: A Tenmat Is a Structure

>> n = [3 5 4 7];

>> A = tenrand(n);

>> A2 = tenmat(A,2);

>> fieldnames(A2)

ans =

’tsize ’

’rindices ’

’cindices ’

’data’

A2.tsize = size of the unfolded tensor = [3 5 4 7]A2.rindices = mode indices that define A2’s rows = [2]A2.cindices = mode indices that define A2’s columns = [1 3 4]A2.data = the matrix that is the unfolded tensor.


Other Modal Unfoldings Using tenmat

Mode-k Unfoldings of A ∈ IRn1×···×nd

A particular mode-k unfolding is defined by a permutation v of[1:k − 1 k + 1:d ]. Tensor entries get mapped to matrix entries asfollows

A(i) → A(ik , col(i(v),n(v)))

Name v How to get it...

A(k) (Default) [1:k−1 k+1:d ] tenmat(A,k)

Forward Cyclic [k + 1:d 1:k−1] tenmat(A,k,’fc’)

Backward Cyclic [k − 1:− 1:1 d :−1:k+1] tenmat(A,k,’bc’)

Arbitrary v tenmat(A,k,v)


Problem 2.9. Given that A = A(1:n1, 1:n2, 1:n3), show how tenmat canbe used to compute the following unfoldings:

A1 =Â(1, : , : ) · · · A(n1, : , : )

Ã2 =

Â( : , 1, : ) · · · A( : , n2, : )

Ã3 =

Â( : , : , 1) · · · A( : , : , n3)

˜

Problem 2.10. How can tenmat be used to compute the vec of a tensor?


A Block Matrix is an Unfolded 4th Order Tensor

Some Natural Identifications

A block matrix with uniformly sized blocks

A =

A11 · · · A1,c2

.... . .

...Ar2,1 · · · Ar2,c2

Ai2,j2 ∈ IRr1×c1

has several natural reshapings:

Ai2,j2(i1, j1) ↔ A(i1, i2, j1, j2) A ∈ IRr1×r2×c1×c2

Ai2,j2(i1, j1) ↔ A(i1, j1, i2, j2) A ∈ IRr1×c1×r2×c2

The function tenmat can be used to set up these unfoldings...


A 4th Order Tensor is a Block Matrix

Example 1.

If A ∈ IRn1×n2×n3×n4 , then tenmat(A,[1 2],[3 4] sets up A(1, 1, :, :) · · · A(1, n2, :, :)...

. . ....

A(n1, 1, :, :) · · · A(n1, n2, :, :)

Example 2.

If A ∈ IRn1×n2×n3×n4 , then tenmat(A,[1 3],[2 4] sets up A(1, :, 1, :) · · · A(1, :, n3, :)...

. . ....

A(n1, :, 1, :) · · · A(n1, :, n3, :)


Even More General Unfoldings

Example: A = A(1:2, 1:3, 1:2, 1:2, 1:3)

B =

(1,1) (2,1) (1,2) (2,2) (1,3) (2,3)2666666666666666666666664

a11111 a11121 a11112 a11122 a11113 a11123

a21111 a21121 a21112 a21122 a21113 a21123

a12111 a12121 a12112 a12122 a12113 a12123

a22111 a22121 a22112 a22122 a22113 a22123

a13111 a13121 a13112 a13122 a13113 a13123

a23111 a23121 a23112 a23122 a23113 a23123

a11211 a11221 a11212 a11222 a11213 a11223

a21211 a21221 a21212 a21222 a21213 a21223

a12211 a12221 a12212 a12222 a12213 a12223

a22211 a22221 a22212 a22222 a22213 a22223

a13211 a13221 a13212 a13222 a13213 a13223

a23211 a23221 a23212 a23222 a23213 a23223

3777777777777777777777775

(1,1,1)

(2,1,1)

(1,2,1)

(2,2,1)

(1,3,1)

(2,3,1)

(1,1,2)

(2,1,2)

(1,2,2)

(2,2,2)

(1,3,2)

(2,3,2)

B = tenmat(A,[1 2 3],[4 5])


Matlab Tensor Toolbox: General Unfoldings Using tenmat

function A_unfolded = Unfold(A,rIdx ,cIdx)

% A = A(1:n(1) ,... ,1:n(d)) is a tensor.

% rIdx and cIdx are integer vectors with the

% property that [rIdx cIdx] is a

% permutation of 1:d.

%

% A_unfolded is an nRows -by -nCols matrix where

% nRows = prod(n(rIdx))

% nCols = prod(n(cIdx))

% if 1<=i<=n, then

% A(i) = A_unfolded(r,c)

% with

% r = col(i(rIdx),n(rIdx))

% c = col(i(cIdx),n(cIdx))

A_unfolded = tenmat(A,rIdx ,cIdx)


Summary of Lecture 2.

Key Words

A block matrix is a matrix whose entries are matrices.

A fiber of a tensor is a column vector defined by fixing all butone index and varying what’s left. A slice of a tensor is amatrix defined by fixing all but two indices and varying what’sleft.

A mode-k unfolding of a tensor is obtained by assembling allthe mode-k fibers into a matrix.

tensor is a Tensor Toolbox command used to constructtensors.

tenmat is a Tensor Toolbox command that is used toconstruct unfoldings of a tensor.


lecture 2. tensor unfoldings

Documents