introduction to algorithms and data structures - usth moodle

Introduction to Algorithms and Data Structures

Doan Nhat Quang

[email protected] of Science and Technology of Hanoi

ICT department

2018-2019

Doan Nhat Quang Introduction to Algorithms and Data Structures 1 / 46

Objectives

Course objectives:

I Provide basic knowledges about algorithms and datastructures.

I Be able to choose appropriate data structures for a specifiqueproblem.

I Approach different algorithms and solve a problem ininformatics.


Textbooks

Recommended Textbook: Data Structures and Algorithms in C++4th edition, Adam Drozdek


Development Tools

I Dev-C++: a free, portable, fast and simple C/C++ IDE

I Visual C++ Express: a free set of tools

I Online: http://cpp.sh/ for simple programs


Why C++?

I C++ is the prodigy of C, comparable to C.

I C++ has almost everything that C has to offer but in a betterway.

I C++ provides support for object-oriented programming


Objectives

Why Study AlgorithmsI Many problems can be solved by using a computer

I want it to go faster? Process more data?I want it to do something that would otherwise be impossible?

I Technology improves many aspects butI it might be costlyI good algorithmic design can do much better and might be

cheaperI super-computers cannot handle a bad algorithm


Example

Suppose that a sequence of a1, a2, ... an (n ≥ 2) is available, findthe maximum?

1 Step 1: Given that Max = a1and the index i = 2,

2 Step 2: If i > n then go toStep 6

3 Step 3: If ai > Max thenMax = ai

4 Step 4: Increment index i

5 Step 5: Go to Step 2

6 Step 6: Return Max


Example

Strategy to solve a problem:


Algorithms and approaches

I Do you know any algorithm?

I The travelling salesman problem (Shortest Paths): ”Given alist of cities and the distances between each pair of cities,what is the shortest possible route that visits each city andreturns to the origin city?”

I Job scheduling: assign different tasks at particular times withvarious constraints.

I How to determine the best move for chess/go?



I Do you know any algorithm?

I Google search algorithm: PageRank

I Euclidean’s algorithm for finding the greatest common divisor

I Algorithms to operate evelators in a building

I and more


From Algorithms to Artificial Intelligence and MachineLearning

I Algorithms are the theory/core concepts of AI and MachineLearning

I Applications are widely large:I Banking: Fraud detection, Stock market analysisI Business: Online customer support, Virtual personal assisstantsI Health: Medical diagnosis, Medical image processingI Transport: Intelligent/smart vehicles, Transport optimizationI etc.


Algorithms

Concept

From the original problem, we have to :

I identify the needs, the ideas of the problem

I find an approach, an appropriate algorithm to solve theproblem

Data Structure

In this course, we will study basic strutures:

I Arrays, Pointers

I Linked Lists, Stacks, Queues

I Tree, Binary Tree


Data structure applications

Data Structure

I List of items in cart when you visit online shop

I List of possible actions (undo/redo) in word editor

I Bitmap (array 2D) to store image pixels

I Graph to represent a group of persons and their relationship(Graph Theory, Graph Mining)

I Tree to arrange and index data like web pages, images, etc.


Definition

Computer program

A computer program is a collection of instructions that can beexecuted by a computer to perform a specific task.

Algorithm

An algorithm is a finite sequence of well-defined instructions tosolve a specific problem or to perform a computation.

I 3 constructs of algorithm: Input → Process → Output.

Data structure

A data structure is a data organization, management, and storageformat that enables efficient access and computation.


Definition

Algorithm vs Program

I A program can implement one or more algorithms;

I A program there is always the idea that it will be executed bya computer while an algorithm could be executed by a person;

I A program is written in programming language, while analgorithm is conceptual and can be described using language(including programming languages), flowcharts or pseudocode.

Program = Algorithm + Data structure.


Algorithms

Flowcharts

Flowcharts are used in designing and documenting simple processesor programs. Flowcharts help visualize and understand a process


Algorithms

Pseudo-code

I Pseudo-code is a high-level description

I Pseudo-code is concrete and easy for human comprehension

I Pseudo-code is ususal used to describe algorithms

Syntax

I Control flow:I if .... then.... (if .... else....)I while (...) ... doI repeat .... until (...)I for .... do....

I Method declarationI InputI Output

findMax (a)

1: Max = a12: for i = 2→ n do3: if ai > Max then4: Max = ai5: end if6: end for7: return Max



A good algorithm must possess the following properties:

I Finiteness The algorithm must always terminate after a finitenumber of steps. Stopping condition should be asserted.

I Definiteness: Each instruction must be precisely defined andconcise. Repetition should be avoided at all cost.

I Input/Output: An algorithm may or may not have inputs butan algorithm has one or more ouputs available when thealgorithm terminates

I Effectiveness: All operations to be performed must besufficiently basic that they can be done exactly and in finitelength.



Criteria are often used to evaluate an algorithm:

I Correctness : does the algorithm always produce correctoutput for each given input? This asserts that the algorithmproducts expected results.

I Efficiency/Complexity: An algorithm is always optimized tohave low running time as possible (time complexity). Duringthe programming, variables are declared and located inmemory, an algorithm is always optimized to have lowallocation memory as possible (space complexity).


Algorithm Design Paradigm

Algorithm types we will consider include:

I Simple recursive algorithms

I Divide and conquer algorithms

I Dynamic programming algorithms

I Greedy algorithms

I Backtracking algorithms

I Randomized algorithms


Recursive Algorithms

A simple recursive algorithm:

I convert the main problem to sub-problems

I solve the base cases directly

I recur with a simpler sub-problem



The picture shows that the solu-tion computes solutions to the sub-problems more than once for noreason:

7 6 5 4 3 2 1 01 1 2 3 5 8 13 21

→ Complexity is exponential,O(2n)



For Hanoi Tower problem, moving plates are done recursively. Wesuppose that n-1-plate problem is done then we solve n-plateproblem


Divide and Conquer Algorithms

A divide and conquer algorithm:

I given a problem to be solved, split this into several smallersub-problems.

I solve each of them recursively and then combine thesub-problem solutions so as to product a solution for theoriginal problem.

Traditionally, an algorithm is “divide and conquer” if it contains atleast two recursive calls



Example: Fibonacci numbers 0, 1, 1, 2, 3, 5, 8, ...

1 i n t f i b o ( n )2 i f ( ( n == 0) | | ( n == 1 ) )3 r e t u r n n ;4 r e t u r n f i b o ( n=1) + f i b o ( n=2);5



I Quicksort:I Partition the array into two parts (smaller numbers in one part,

larger numbers in the other part)I Quicksort each of the parts

I Mergesort:I Cut the array in half, and mergesort each halfI Combine the two sorted arrays into a single sorted array by

merging them


Dynamic Programming

A dynamic programming algorithm remembers past results anduses them to find new results:

I cut the main problem into a set of simpler sub-problems

I solve each of those sub-problems just once, and store theirsolutions which can be used later.

This differs from Divide and Conquer, where sub-problemsgenerally need not overlap.


Dynamic Programming

Example: Fibonacci numbers 0, 1, 1, 2, 3, 5, 8, ...

1 i n t f i b o ( n )2 i f ( ( n == 0) | | ( n == 1 ) )3 r e t u r n n ;4 i f f i b o ( n ) != 05 r e t u r n f i b o ( n ) ;6 r e t u r n f i b o ( n=1) + f i b o ( n=2);7

Since finding the i th Fibonacci number involves finding all smallerFibonacci numbers which are already calculated, the secondrecursive call has little work to do.


Greedy Algorithm

A greedy algorithm is an algorithm that follows the problem solvingheuristic:

I take the best that we can get right now, without regard forfuture consequences.

I choose a local optimum at each step with the hope of findinga global optimum.

Greedy algorithms sometimes work well for optimization problems.


Greedy Algorithm

Example: Suppose you want to count out a certain amount ofmoney, using the fewest possible bills and coins. A greedyalgorithm would do this would be:

I At each step, take the largest possible bill or coin that doesnot overshoot.

I To make 151,000 VND, how many bills as few as possible?which should be the best solution?:

I a 100,000 VND billI a 50,000 VND billI a 1,000 VND bill


Greedy Algorithm

Example: Suppose you want to count out a certain amount ofmoney, using the fewest possible bills and coins. A greedyalgorithm would do this would be:

I At each step, take the largest possible bill or coin that doesnot overshoot.

I To make 151,000 VND, how many bills as few as possible?which should be the best solution?:I a 100,000 VND billI a 50,000 VND billI a 1,000 VND bill


Greedy Algorithm

Example: Suppose that in a certain currency system, we have 1coin, 7 coins and 10 coins pieces.

I Using a greedy algorithm to count out 15 coins, you wouldget:I A 10 coins pieceI Five 1 coin pieces, for a total of 15 coinsI This requires six coins

I A better solution would be to use:I Two 7 coins pieces and one 1 coin pieceI This only requires three coins


Greedy Algorithm

Example: Suppose that in a certain currency system, we have 1coin, 7 coins and 10 coins pieces.I Using a greedy algorithm to count out 15 coins, you would

get:I A 10 coins pieceI Five 1 coin pieces, for a total of 15 coinsI This requires six coins

I A better solution would be to use:I Two 7 coins pieces and one 1 coin pieceI This only requires three coins


Greedy Algorithm

Label new data sample according to a number of data neighbors(k)

I Find k nearest neighbors of data sample

I Among these k data, if the number of data from any class ismore common, the data sample is assigned to this class.


Backtracking Algorithm

A backtracking algorithm bases on recursion:

I starting with one possible move out of many available moves

I find next move from the starting point

I if this satisfies given constraints, continue next moves; elsereturn to previous move

Sometimes, backtracking algorithms don’t have solutions due tothe constraints.


Backtracking Algorithm


Randomized Algorithms

A randomized algorithm is an algorithm that employs a degree ofrandomness as part of its logic.

I In Quicksort, using a random number to choose a pivot

I In Machine Learning, some techniques are “randomized” suchas K-means, Self-organized Map, Random Forest, etc.


Algorithm complexity

Complexity

A theoretical evaluation to measure how good an algorithm is interm of running time and computational memory.

Assume that the running time of an algorithm is T (n) with nobjects.

I Big Θ: T (n) = Θ(f (n)) if ∃k1, k2, n0 ∈ N+, ∀n ≥ n0:k1f (n) ≤ T (n) ≤ k2f (n)

I Big O: T (n) = O(f (n)) if ∃k , n0 ∈ N+, ∀n ≥ n0:T (n) ≤ kf (n)

I Big Ω: T (n) = Ω(f (n)) if ∃k , n0 ∈ N+, ∀n ≥ n0:T (n) ≥ kf (n)



Big Θ notation, an asymptotically tight bound on the runningtime, T (n) = Θ(f (n)) if ∃k1, k2, n0 ∈ N+, ∀n ≥ n0:k1f (n) ≤ T (n) ≤ k2f (n)



Big O notation, the asymptotic upper bounds of a running time,T (n) = O(f (n)) if ∃k , n0 ∈ N+, ∀n ≥ n0: T (n) ≤ kf (n)



Big Ω notation, the asymptotic lower bounds of a running time,T (n) = Ω(f (n)) if ∃k, n0 ∈ N+, ∀n ≥ n0: T (n) ≥ kf (n)



Step 1 if Max = a1 and i = 2, 1 operation → O(1)

Step 2 if i > n then go to Step 6

n-1 operations → O(n)Step 3 If ai > a1 then Max = aiStep 4 Increment iStep 5 Go to Step 2Step 6 Return Max 1 operation → O(1)

Complexity: O(n + 1 + 1) = O(n)



Step 1 if Max = a1 and i = 2, 1 operation → O(1)Step 2 if i > n then go to Step 6

n-1 operations → O(n)Step 3 If ai > a1 then Max = aiStep 4 Increment iStep 5 Go to Step 2

Step 6 Return Max 1 operation → O(1)




Step 1 if Max = a1 and i = 2, 1 operation → O(1)Step 2 if i > n then go to Step 6

n-1 operations → O(n)Step 3 If ai > a1 then Max = aiStep 4 Increment iStep 5 Go to Step 2Step 6 Return Max 1 operation → O(1)




Some complexity examples:

I Any operation, statement, instruction: S1, ...,Sk → O(1)

I fori = 1; i <= n; i + +Si → O(n)

I fori = 1; i <= n; i + +forj = 1; j <= n; j + +Si → O(n2)



Several properties for complexity computation:

I f (n) = O(h(n))→ nf (n) = O(nh(n))

I f (n) = O(h(n))→ kf (n) = O(h(n)) where k is a constant

I f (n) = O(h(n))→ g(n) = O(y(n))→ f (n)g(n) =O(h(n)y(n))

I f (n) + g(n) = max(O(f (n)),O(y(n)))



I O(1): Accessing any single element in an array takes constant timeas only one operation has to be performed to locate it; or anyarithmeric operations between two numbers, only one operation hasto be done.

I O(ln(n)): Algorithms taking logarithmic time are commonly foundin operations on binary trees or when using binary search.

I O(n): This linear complexity means that for large enough inputsizes the running time increases linearly with the size of the input.

I O(n2):This quadratic complexity can be seen in algorithms forexample bubble sort, insertion sort consisting of nested loops (aloop inside another loop) with the size of n.

I O(n3): This running time often requires from the multiplication oftwo nxn matrices

I O(2n): An exponential running time is used to found in thetravelling salesman problem.



log n√n n nlogn n(logn)2 n2

3 3 10 33 110 1007 10 100 664 4,414 10,000

10 32 1,000 9,966 99,317 1,000,00013 100 10,000 132,877 1,765,633 100,000,00017 316 100,000 1,660,964 27,588,016 10,000,000,000


Example

1 f o r ( i n t i = 1 ; i <n ; i ++)2 f o r ( i n t j = 1 ; j <n ; j ++)3 a [ i ] [ j ] = 0 ;

1 f o r ( i n t i = 1 ; i <n ; i ++)2 f o r ( i n t j = 1 ; j <n ; j ++)3 a [ i ] [ j ] = 0 ;4 f o r ( i n t k = 1 ; k < n ; k++)5 a [ k ] [ k ] = 1 ;


Example

1 i n t sum = 0 ;2 i n t x , y ;3 f o r ( i n t i = 1 ; i <n ; i ++)4 f o r ( i n t j = 1 ; j <n ; j ++)5 f o r ( i n t k = 1 ; k < 1 0 0 ; k++)6 y = k ;7 x = 2*y ;8 9 sum = sum + i * j *k ;

10


introduction to algorithms and data structures - usth moodle

Documents