tirgul no. 7

Tirgul no. 7Tirgul no. 7

Topics covered:Topics covered:

- More Sorting and Searching + Complexity analysis- More Sorting and Searching + Complexity analysis

- Matrices: Definitions and basic operations.- Matrices: Definitions and basic operations.

- Gram-Schmidt Orthonormalization algorithm- Gram-Schmidt Orthonormalization algorithm

SearchingSearching

Searching for a value in an unsorted list requires - linear scan Searching for a value in an unsorted list requires - linear scan of all elements.of all elements.

However, when the list is sorted we can use this fact to However, when the list is sorted we can use this fact to significantly improve our search complexity.significantly improve our search complexity.

This is done using the Binary Search algorithm which is a This is done using the Binary Search algorithm which is a simple example of the “simple example of the “Divide and ConquerDivide and Conquer” approach.” approach.

Divide and ConquerDivide and Conquer: given a problem – divide it into sub-: given a problem – divide it into sub-problems and solve each one of them separately.problems and solve each one of them separately.

Binary SearchBinary Search//assume A is an array with n values//return the index in which the value key is located, //or –1 if not found in the arrayBinarySearch(A,key) {

(1) lower 1, upper n (2) middle (lower+upper)/2

(3) while( (lower<=upper) and (A[middle]< >key))

(3.1) if (key < A[middle]) then upper middle-1

else lower middle+1

(3.2) middle (lower+upper)/2

(4) if (A[middle] = key) then return(middle)else return(-1)

}

Binary Search – complexity Binary Search – complexity analysisanalysis In each round the algorithm divides the search space into 2 equal parts and

uses the assumption that the numbers are sorted to choose one out of the 2 parts in to continue searching in.

Why is the sorting assumption important?

In order to calculate the running time, we have to count the number of comparisons that the algorithm makes – which is equivalent to the number of values that the index middle is given:

Worst case analysis: Let us assume that we are searching for the smallest value in the sorted array

A: For an array of size n: middle= (n/2, n/4, n/8,n/16,n/32…,1)

- Note that this is equivalent to searching for a non existing value in the array!

TTherefore for an array of size n: the runtime will be: T(n)= O(log n)

In the average case we may find the key sooner.

Insertion Sort using Binary Insertion Sort using Binary SearchSearch Insertion sort - each time insert the next value into an already sorted array. Insertion sort - each time insert the next value into an already sorted array. In order to find its insertion index – use BinarySearch! – In order to find its insertion index – use BinarySearch! –

using a small modification in the BinarySearch algorithm we can have it return the using a small modification in the BinarySearch algorithm we can have it return the expected position of a value in a sorted list:expected position of a value in a sorted list:

// when insertFlag = true, we are searching for the insertion index of key into the list –// i.e. we are not interested to know whether key is in the list or not, but rather where he// should be positioned in order to keep the list sorted!BinarySearch (A,key,insertFlag) {

(1) lower 1, upper n (2) middle (lower+upper)/2 (3) while( (lower<=upper) and (A[middle]< >key)) (3.1) if (key < A[middle]) then

upper middle-1 else

lower middle+1(3.2) middle (lower+upper)/2

(4) if (A[middle] = key) or (insertFlag) then return(middle)else return(-1)

}

Insertion Sort using Binary Insertion Sort using Binary SearchSearchpublic static void insertionSort (int[] numbers){

boolean insertFlag= true; for (int index = 1; index < numbers.length; index++){

int key = numbers[index]; int insertionPosition = binarySearch(numbers,key,insertFlag,0,index); int currPosition = index;

// shift larger values to the right while (position > insertionPosition)) { numbers[position] = numbers[position-1]; position--; } numbers[insertionPosition] = key; } }

Shell’s SortShell’s Sort Shell sort is the fastest O(nShell sort is the fastest O(n22) sorting algorithm. ) sorting algorithm. The algorithms for shell sort can be defined in two steps:

Step 1: divide the original list into smaller lists.

Step 2: sort individual sub lists using any known sorting algorithm (like bubble sort, insertion sort, selection sort, etc).

Many questions arise:- how should I divide the list?- Which sorting algorithm to use? - How many times I will have to execute steps 1 and 2? - And the most puzzling question: If I am anyway using bubble,

insertion or selection sort then how I can- achieve improvement in efficiency?

Shell’s SortShell’s Sort For dividing the original list into smaller lists, we choose a

value K, which is known as increment. Based on the value of K, we split the list into K sub lists.

For example: if our original list is x[0], x[1], x[2], x[3], x[4]….x[99] and we choose K=5 as the increment value then we get the following sub lists:

first_list = x[0], x[5], x[10], x[15]…….x[95]

second_list =x[1], x[6], x[11], x[16]…….x[96]

third_list =x[2], x[7], x[12], x[17]…….x[97]

forth_list =x[3], x[8], x[13], x[18]…….x[98]

fifth_list =x[4], x[9], x[14], x[19] …….x[99]

Shell’s SortShell’s Sort For dividing the original list into smaller lists, we choose a

value K, which is known as increment. Based on the value of K, we split the list into K sub lists. For example: if our original list is x[0], x[1], x[2], x[3], x[4]

….x[99] and we choose K=5 as the increment value then we get the following sub lists:

first_list = x[0], x[5], x[10], x[15]…….x[95]second_list =x[1], x[6], x[11], x[16]…….x[96]third_list =x[2], x[7], x[12], x[17]…….x[97]forth_list =x[3], x[8], x[13], x[18]…….x[98]fifth_list =x[4], x[9], x[14], x[19] …….x[99]

So the ith sub list will contain every Kth element of the original list starting from index i-1.

Shell’s SortShell’s Sort According to the algorithm mentioned above, for each

iteration, the list is divided and then sorted. If we use the same value of K, we will get the same sub lists and every time we will sort the same sub lists, which will not result in the ordered final list.

Note also that sorting the five sub lists independently do not ensure that the full list is sorted! So we need to change the value of K.

We also know that number of sub lists we get are equal to the value of K.

if we decide to reduce the value of K after every iteration we will reduce the number of sub-lists also in every iteration. Eventually, when K will be set to 1, we will have only one sub list.

Shell’s Sort – why does it perform better Shell’s Sort – why does it perform better – intution– intution Let’s assume that we will use insertion-sort to sort the

sublists. If the list is either small or almost sorted then insertion sort is efficient since a smaller number of elements will be shifted.

In Shell Sort the original list is divided into smaller lists based on the value of increment and each list is sorted. Initially value of K (increment) is fairly large giving a large number of small sub lists.

As K reduces its value, the length of the sub lists increases. But since the earlier sub lists have been sorted, we except the full list to become more and more sorted.

Thus the inefficiency arising out of working with larger lists is partly compensated by lists being increasingly sorted.

Choosing the best value of increments (Choosing the best value of increments (k k )) Up to date there is no optimal sequence of increments. There have

been a few suggestions that have been shown to be efficient (both theoretically and empirically)

It has been shown empirically that using a larger number of increments gives more efficient results.

It is advisable not to use empirical sequences like 1,2,4,8… or 1,3,6,9… because they may result in having almost the same elements compared for different increment values which will result in poor performance.

Knuth’s suggestion:• h1 = 1

• ht+1 = ht* 3 +1 and stops at ht, When ht+2 >= N.

Which gives the increment series as 1, 4, 13….. ht.

Shell’s Sort – implementation:Shell’s Sort – implementation:public void ShellSort(int[] numbers) {

int[] increments= generateIncrementSeq();

// start with the largest increment and proceed till 1.

for(int k=increments.length-1; k>=0; k--) {

int currIncrement=increments[k];

// here the list is divided into ’currIncrement’ subarrays

for (int sublistid=0; sublistid < currIncrement; sublistid++){

//perform insertion sort with each sublist

//note that you can use any sorting algorithm here

insertionSort(currIncrement, sublistid,numbers);

}//for k

}

Shell’s Sort – implementation:Shell’s Sort – implementation:public int[] generateIncrementSeq(int arrayLength){

//count how many increments need to be generated.numOfInc=0; for(int h=1;h<arrayLength;) {

if(h > arrayLength) numOfInc= numOfInc –2;h=3*h + 1;

}//generate increment sequence:increments= new int[numOfIncs];for ( int i=1;i< numOfIncs; i++ ){ increments[i]= h;

h= 3*h + 1;}return(increments);

}

Shell’s Sort – exampleShell’s Sort – exampleInt[ ] numbers=

{8, 2, 84, 6, 12, 5, 76, 9, 13, 67, 54, 0, 43, 20, -4, 78, 32, 62, 75, 106, 16, 4, 77, 45, 23, 31, 92, 7, 18};

When increment is 13, we get the following 13 sub lists and the elements are divided among the sub lists in the following ways i.e. every K’th element of the list starting with the index i will be in the sub list I:

Shell’s Sort – exampleShell’s Sort – exampleAfter sorting the sub lists, the resulting list we get is as

follows:

{8, -4, 18, 6, 12, 5, 76, 9, 4, 67, 45, 0, 31, 20, 2, 78, 32,

62, 75, 106, 16, 13, 77, 54, 23, 43, 92, 7,84}

Compare this to:

{8, 2, 84, 6, 12, 5, 76, 9, 13, 67, 54, 0, 43, 20, -4, 78, 32,

62, 75, 106, 16, 4, 77, 45, 23, 31, 92, 7, 18};

Shell’s Sort – example (cont.)Shell’s Sort – example (cont.)We now, reduce the increment value to 4 and get the

following 4 sub lists.

Shell’s Sort – example (cont.)Shell’s Sort – example (cont.)After sorting these sub lists, the resulting list is as follows:

{ 4, -4, 2, 0, 8, 5, 18, 6, 12, 13, 45, 7, 16, 20, 75, 9, 23,

43, 76, 54, 31, 62, 77, 78, 32, 67, 92, 106, 84 }

We now repeat for increment= 1 (which is simply insertion sort) and get the sorted array!

MatricesMatrices

An m*n dimensional matrix is a mathematical object that is An m*n dimensional matrix is a mathematical object that is defined by a table of numbers with N rows and M columns. For defined by a table of numbers with N rows and M columns. For example, a Matrix A 2*3 and a matrix B 3*2 are defined asexample, a Matrix A 2*3 and a matrix B 3*2 are defined as::

23

3231

2221

1211

32

232221

131211 ,

,

,

,

,,,,

,,

B

bb

bb

bb

BAaaa

aaaA

The element in the i'th row and j’th column in the The element in the i'th row and j’th column in the matrix a is denoted by Amatrix a is denoted by A

i, ji, j

Basic Operations on MatricesBasic Operations on Matrices

Sum - We can define the sum of 2 matrices as the sum of each of Sum - We can define the sum of 2 matrices as the sum of each of the corresponding elements, i.e. (susbstraction is same)the corresponding elements, i.e. (susbstraction is same)

nm

jijiji

mnmm

n

CBA

BACji

BA

ccc

ccc

C

,,

, ,,,

,2,1

1,12,11

5.12,1.5,1.5

4.16,3.9,7.4

5.50.7,0.41.1,1.30.2

5.79.8,8.45.4,5.32.1

5.5,0.4,1.3

5.7,8.4,5.3,

0.7,1.1,0.2

9.8,5.4,2.1

BAC

thenBA

Matrix MultiplicationMatrix Multiplication

Matrix Multiplication – is defined by multiplication of the Matrix Multiplication – is defined by multiplication of the corresponding elements in the 2 matrices, and results in a new corresponding elements in the 2 matrices, and results in a new matrix, i.e.matrix, i.e.

n

l

kmjlliji

kn

nknn

n

nm

mnmm

n

CbaCwhereABC

thenB

bbb

bbb

BA

aaa

aaa

A

1,,,

21

11211

21

11211

*,

,

,

,

,

,

,

If the number of columns in If the number of columns in AA is not equal to the number of is not equal to the number of rows in rows in BB, matrix multiplication is undefined., matrix multiplication is undefined.

Matrix Multiplication - ExampleMatrix Multiplication - Example

20)1*25*33*1(*,60,38

22,20

6,1

2,5

4,3

,7,5,2

2,3,1

3

11,,11,1

l

ll BACABC

thenBA

Multiplication By A VectorMultiplication By A Vector

right multiplication of a matrix by a vector is defined if the right multiplication of a matrix by a vector is defined if the dimension of the vector equals the number of columns in the dimension of the vector equals the number of columns in the matrix, i.e.matrix, i.e.

n

l

nllii

n

n

nm

mnmm

n

ybaywhereAxy

thenx

x

x

xA

aaa

aaa

A

1,

1

21

11211

*,

,,

,

,

Multiplication By A Vector - Multiplication By A Vector - exampleexample

20)1*25*33*1(*,38

20

1

5

3

,7,5,2

2,3,1

3

1,11

l

ll xAyAxy

thenxA

Gram-Scmidt Gram-Scmidt OrthonormalizationOrthonormalization

If we have a set of linearly-independent but non-orthogonal If we have a set of linearly-independent but non-orthogonal vectors vectors xx11 … … xxnn we often wish to turn them into an alternative we often wish to turn them into an alternative

set of vectors, set of vectors, qq11 … … qqn n that are mutually orthonormal:that are mutually orthonormal:

Achieving this orthonormalization is the purpose of the Gram-Achieving this orthonormalization is the purpose of the Gram-Schimdt procedureSchimdt procedure

jifor

jiforqq j

Ti 0

1


Let's address the particular example where:Let's address the particular example where:

The first is simple, involving a simple normalization; The first is simple, involving a simple normalization;

0

11

7

3

4

12

6

1

4

3

2

1

321 xxx

4

3

2

1

30

111

11

1 xxx

qT


The second vector will be the original second vector, The second vector will be the original second vector, xx22 minus minus

its projection on the first orthonormal basis vector its projection on the first orthonormal basis vector qq11 , that is: , that is:

28

33

10

7

6

1

52

39

26

13

6

1

24

72

36

6

6

1

4

3

2

1

30

65

4

12

6

1

4

3

2

1

30

1

4

3

2

1

30

141261

4

12

6

1

2

111222

q

qqxxq T

Gram-Schmidt: NormalizationGram-Schmidt: Normalization

We now noramlize We now noramlize qq22 to unit norm: to unit norm:

623.0

734.0

222.0

156.0

28

33

10

7

6

1

12 q

q

Gram-Schmidt (cont.)Gram-Schmidt (cont.)

To construct the last basis vector, we need to subtract from To construct the last basis vector, we need to subtract from qq33

its projections on its projections on bothboth qq11 and and qq22 : :

Sparing some tedious math:Sparing some tedious math:

2231133 3 qqxqqxxq TT

281.0

211.0

476.0

806.0

3q

Gram-Schmidt (cont.)Gram-Schmidt (cont.)

In conclusion, we have transformed our original, non-In conclusion, we have transformed our original, non-orthonormal, basis set, to an entirely equivalent set, whose orthonormal, basis set, to an entirely equivalent set, whose vectors are all mutually orthogonal and have unit norms. vectors are all mutually orthogonal and have unit norms.

Now, when we express an arbitrary vector from the space Now, when we express an arbitrary vector from the space spanned by these 2 alternative sets as a linear combination of spanned by these 2 alternative sets as a linear combination of the orthonormal vectors the orthonormal vectors qq11 qq22 and q and q3.3.

The projections (the weights) are entirely independent of one The projections (the weights) are entirely independent of one another, and their sum is exactly the norm of the represented another, and their sum is exactly the norm of the represented vector, and not more. vector, and not more.

Gram-Schmidt algorithm:Gram-Schmidt algorithm:

InputInput: Matrix A: Matrix Anxnnxn = ( a = ( a00 a a11 ... a ... ann ) whose columns form a basis of R ) whose columns form a basis of Rnn (are linearly (are linearly independent). independent).

OutputOutput: Matrix B: Matrix Bnxnnxn whose columns are an orthonormal basis of R whose columns are an orthonormal basis of Rnn , i.e. for all , i.e. for all columns bcolumns bii,b,bjj we have <b we have <bii,b,bjj> = 0 , <b> = 0 , <bii,b,bii> = 1> = 1

(1)(1)

(2) for i=1 to n do:(2) for i=1 to n do:

(2.1) (2.1)

0

00 a

ab

bbaa

bbaab

k

iikk

i

k

iikk

k

1

1

tirgul no. 7

Documents