combinatorial algorithms

Combinatorial Algorithms

INTRODUCTIONJessica

What are Combinatorial Algorithms?

• Algorithms to investigate combinatorial structures• Generation: Construct all the combinatorial structures

of a particular type– Ex: subsets, permutations, partitions, trees, Catalan families

• Enumeration: Compute the number of different structures of a particular type– Ex: find number of k-subsets of an n element set

• Search: Find at least one example of a structure of particular type (if it exists)– Ex: find a clique of a specified size in a given graph

Optimization Problems

• Suppose we are trying to solve an optimization problem

• There exist backtracking solutions for this reason

Backtracking Algorithms

• Generate all possible solutions in a certain lexicographic order

• Advantages:– Useful for finding one optimal solution and for counting or

enumerating all optimal solutions• Disadvantages:

– May be inefficient for finding just one solution– For optimal verification, may need to examine large part of

the state space tree, even with pruning– Sometimes need to “back up” several levels when partial

solution cannot be extended

Heuristic Algorithm

• Definition: A heuristic algorithm tries to find a certain combinatorial structure or solve an optimization problem by the use of heuristics.

• Heuristic: “serving or helping to find out or discover; proceeding by trial and error”

• Types of problems can be applied to:– Find 1 optimal solution– Find “close to” optimal solution

Heuristic Characteristics and Methods

Characteristics• The state space is not fully

explored• Randomization is often

employed• There is a concept of

neighborhood search

Methods• Hill-climbing• Simulated annealing• Tabu search• Genetic algorithms

General framework for heuristic search

Generic Optimization Problem (maximization):

Instance: A finite set X An objective function P : X Z m feasibility functions g : Xj Z, 1 ≤ j ≤ m

Find: the maximum value of P(X) subject to Xj X and g (X) ≥ 0, for 1 ≤ j ≤ m

Designing a heuristic search

• Define a neighborhood function N : X 2X

– Ex: N(X) = {X1 , X2 , X3 , X4 , X5 }

• Design a neighborhood search:– Algorithm that finds a feasible solution on the

neighborhood of a feasible solution X.– 2 types of neighborhood searches:

• Exhaustive (chooses best profit among neighbor points)• Randomized (picks a random point among the neighbor

points)

Defining a neighborhood functionN : X 2X

So, N(X) is a subset of X.• N(X) should contain elements that are similar or “close to”

X.• N(X) may contain infeasible elements of X.• To be useful, want to get to Xopt from X0 via a number of

applications of N(▪)• ie. Graph G with V(G) = X and E(G) = {{X, Y} : Y N(X)}

should ideally be connected, or at least have one optimal solution in each of its connected components

• Computing N(X) should be fast, and in particular |N(X)| shouldn’t be too large

HILL CLIMBING ALGORITHMSumanaruban, Yogesh and Lahiru

Hill-Climbing• Idea: Go up the hill continuously, stop when

stuck.

Hill Climbing Algorithm

1. Pick a random point in the search space2. Consider all the neighbors of the current

state3. Choose the neighbor with the best quality

and move to that state4. Repeat 2 thru 4 until all the neighboring

states are of lower quality5. Return the current state as the solution state

• Problem: it can get stuck in a local optimum

• Simple (often effective) solution– Multiple random restarts

Uniform graph partition• Instance

A complete graph on 2n vertices,A cost function, cost :

• Find the minimum value of

),( VG

}0{E

xxvutC

vuEvu

10,},{10

),(cos]),([

Example• n = 4;

– cost(1, 2) = 1, cost(1, 3) = 2,– cost(1, 4) = 5, cost(2, 3) = 0,– cost(2, 4) = 5, cost(3, 4) = 1

x1

X2

x3x4

1

1

2

5 05

• Only 3 feasible solutions:• X0 = {1, 2}, X1 = {3, 4}

• C([X0, X1]) = 12

x1

X2

x3x4

1

1

2

5 05

• X0 = {1, 4}, X1 = {2, 3}

• C([X0, X1]) = 9

x1

X2

x3x4

1

1

2

5 05

• X0 = {1, 3}, X1 = {2, 4}

• C([X0, X1]) = 7 (optimal)

x1

X2

x3x4

1

1

2

5 05

Neighbourhood function: exchange x X0 and y X1.∈ ∈

Algorithm UGP(Cmax)X = [X0, X1] ← SelectRandomPartitionc ← 1

while (c ≤ Cmax) do[Y0, Y1] ← Ascend(X)if not fail then{X0 ← Y0; X1 ← Y1; }else returnc ← c + 1

Algorithm Ascend([X0, X1])g ← 0for each i X∈ 0 dofor each j X∈ 1 dot ← G[X0,X1](i, j) (gain obtained in exchange)if (t > g) {x ← i; y ← j; g ← t}if (g > 0) thenY0 ← (X0 {y}) \ {x}∪Y0 ← (X1 {x}) \ {y}∪fail ← falsereturn [Y0, Y1]else {fail ← true; return [X0, X1]}

Other drawbacks

• Ridge = sequence of local maxima difficult for greedy algorithms to navigate

• Plateau = an area of the state space where the evaluation function is flat.

Image from: http://classes.yale.edu/fractals/CA/GA/Fitness/Fitness.html

local maximum

ridge

plateau

Hill-climbing variations

• How do we make hill climbing less greedy? • Stochastic hill-climbing

– Random selection among the better neighbors.– The selection probability can vary with the steepness of the uphill

move.

• What if the neighborhood is too large to enumerate?• First-choice hill-climbing

– Randomly generate neighbors, one at a time – If better, take the move

• Random-restart hill-climbing– Tries to avoid getting stuck in local maxima.

Hill-Climbing AlgorithmforSteiner Triple SystemsStinson's Algorithm

Steiner Triple System

Examples STS(3), v = 3 (trivial case)

V = {1,2,3} B = {123}


V = {1,2,3} B = {123}

STS(7), v = 7 V = {1,2,3,4,5,6,7} B = {1,2,3}, {1,4,5}, {1,6,7}, {2,4,6}, {2,5,7}, {3,4,7}, {3,5,6}


V = {1,2,3} B = {123}

STS(7), v = 7 V = {1,2,3,4,5,6,7} B = {1,2,3}, {1,4,5}, {1,6,7}, {2,4,6}, {2,5,7}, {3,4,7}, {3,5,6}

STS(9), v = 9 V = {1,2,3,4,5,6,7,8,9} B = {1,2,3}, {4,5,6}, {7,8,9}, {1,4,7}, {2,5,8}, {3,6,9}, {1,5,9}, {2,6,7}, {3,4,8}, {1,6,8}, {2,4,9}, {3,5,7}.

A Graph Theoretic ViewAnother way to look at Steiner Triple Systems

Image source - http://mathworld.wolfram.com/SteinerTripleSystem.html

A Graph Theoretic View

Consider the complete graph on v vertices, Kv.A decomposition of Kv into edge disjoint triangles (K3's) is equivalent to a Steiner Triple System.

Another way to look at Steiner Triple Systems

Image source - http://mathworld.wolfram.com/SteinerTripleSystem.html

Fano Plane

The Fano plane is an STS(7) Steiner triple system. The blocks are the 7 lines, each containing 3 points. Every pair of points belongs to a unique line.

Fano Plane

The Fano plane is an STS(7) Steiner triple system. The blocks are the 7 lines, each containing 3 points. Every pair of points belongs to a unique line.

Fano Plane

Has application like factoring integers via quadratic forms

Kirkman's Schoolgirl ProblemA problem in combinatorics proposed by Rev. Thomas Penyngton Kirkman in 1850 as Query VI in The Lady's and Gentleman's Diary (pg.48).

Image source - http://delphiforfun.org/programs/kirkman.htm

Kirkman's Schoolgirl Problem

Fifteen young ladies in a school walk out three abreast for seven days in succession:it is required to arrange them daily so that no two shall walk twice abreast.

A problem in combinatorics proposed by Rev. Thomas Penyngton Kirkman in 1850 as Query VI in The Lady's and Gentleman's Diary (pg.48).

Image source - http://delphiforfun.org/programs/kirkman.htm

Kirkman SystemsThe solution to Kirkman’s 15 schoolgirl problem is a resolution of a block design on 15 points with blocks of size 3

Steiner Triple System on 15 points


Steiner Triple System on 15 points Solution not unique


Steiner Triple System on 15 points Solution not unique

7 possible solutions for STS(15)

1

1

If an STS(n) exists, so too does an STS(2n + 1).2


PROOF: 1.1 r = number of blocks in which any point appearv = total number of points

1


• Any point x must appear in some block with each of all other (v - 1) points

1


• Any point x must appear in some block with each of all other (v - 1) points• Point x occurs with 2 other points in each of the r blocks it appears in

1


• Any point x must appear in some block with each of all other (v - 1) points• Point x occurs with 2 other points in each of the r blocks it appears in• Therefore,

1

PROOF: 1.2 |B| = number of blocks in set Bv = total number of points

1


• Let b = |B| be the total number of blocks

1


• Let b = |B| be the total number of blocks• We count T, the number of points with their replications appearingon B, in two ways:

1



• T = 3 X b

1



• T = 3 X b• T = v X r (r = (v-1)/2 using lemma 1.1)

1



• T = 3 X b• T = v X r (r = (v-1)/2 using lemma 1.1)

• Therefore,

1



Reference - http://www.math.mun.ca/~davidm/4341/sts.pdf

1

Theorem 1.1 STS(v) ⇒ v ≡ 1, 3 (mod 6)Theorem 1.2 v ≡ 1, 3 (mod 6) ⇒ STS(v)

PROOF: Let (V,B) be an STS(v)• For any x ∈ V, the triples containing x partition V-{x} into pairs,

Theorem 1.1 STS(v) ⇒ v ≡ 1, 3 (mod 6)


• thus, v-1 is even



• thus, v-1 is even• and v is odd




• Therefore, v≡1,3 or 5(mod 6)




• Therefore, v≡1,3 or 5(mod 6)• From lemma 1.2 (b-number of blocks)

b=v(v-1)/6


PROOF: Let (V,B) be an STS(v)• However, if v = 6k+5, computing |B| gives:



|B| = (6k+5)(6k+4)/6|B| = (36k2 + 54k + 20)/6



|B| = (6k+5)(6k+4)/6|B| = (36k2 + 54k + 20)/6 which is not an integer.



|B| = (6k+5)(6k+4)/6|B| = (36k2 + 54k + 20)/6 which is not an integer.

Therefore, v≡1,3(mod 6)


PROOF (Using Mathematical Induction)• Base Case :

• The base cases are v = 1, v = 3, v = 7 and v = 9, which have been illustrated

Theorem 1.2 v ≡ 1, 3 (mod 6) ⇒ STS(v)


V = {1,2,3} B = {123}

STS(7), v = 7 V = {1,2,3,4,5,6,7} B = {1,2,3}, {1,4,5}, {1,6,7}, {2,4,6}, {2,5,7}, {3,4,7}, {3,5,6}

STS(9), v = 9 V = {1,2,3,4,5,6,7,8,9} B = {1,2,3}, {4,5,6}, {7,8,9}, {1,4,7}, {2,5,8}, {3,6,9}, {1,5,9}, {2,6,7}, {3,4,8}, {1,6,8}, {2,4,9}, {3,5,7}.

PROOF (Using Mathematical Induction)• Base Case :

• The base cases are v = 1, v = 3, v = 7 and v = 9, which have been illustrated• Induction Hypothesis :

• Let v = k, (k ≡ 1, 3 (mod 6))• Assume that there exist an STS(l) for all l < k (l ≡ 1, 3 (mod 6))

Theorem 1.2 v ≡ 1, 3 (mod 6) ⇒ STS(v)

Theorem 1.2 v ≡ 1, 3 (mod 6) ⇒ STS(v) PROOF (Using Mathematical Induction)• Induction Step : (Prove STS(k) exist)

• We consider four cases, depending on the congruence of k modulo 12• k = 12m + 1• k = 12m + 3• k = 12m + 7• k = 12m + 9


Case 1 : k = 12m + 1• 6m-3 < k• Therefore, by the induction hypothesis

• STS(6m - 3) exist• k = 12m + 1 = 2(6m - 3) + 7• Lemma 3 says : If an STS(n) exists, so too does an STS(2n + 7)• Using Lemma 3 there is an STS(k)


Case 2 : k = 12m + 3• 6m+1 < k• Therefore, by the induction hypothesis

• STS(6m + 1) exist• k = 12m + 3 = 2(6m + 1) + 1• Lemma 2 says : If an STS(n) exists, so too does an STS(2n + 1)• Using Lemma 2 there is an STS(k)

Partial Steiner Triple System

Is a set system ),( BV

In which every block has size V

3

)(vPSTS

Every pair of points in is contained in at most one block

6/)1(|| vvB

If 6/)1(|| vvB )()( vSTSvPSTS

Construct Steiner Triple System

Stinson’s Algorithm

6

Switch Heuristic

Remarks

No termination guaranteeIn practice, it always terminatesif the choices made by the heuristic are random

()Switch

Usually runs very quickly

Results

)ln( bbOThe average number of iterations is

GENETIC ALGORITHMShalinda , Ramanathan and Pham Nam

Genetic Algorithm• The knowledge that each specie gains is encoded in its

chromosomes and they undergo transformation when reproduction occurs.

• Genes from `good‘ parents will produce offspring that are better than either parent.

• Thus each successive generation will become more suited to their environment.

Developed by John Holland, University of Michigan (1970’s)• Provide efficient, effective techniques for optimization and

machine learning applications

Generic Genetic AlgorithmIncorporates a heuristic hN based on a neighborhood function N and

a recombination operationrec: x → x

Chromosomes could be:Bit strings (0101 ... 1100)Real numbers (43.2 -33.1 ... 0.0 89.2) Permutations of element (E11 E3 E7 ... E1 E15)Lists of rules (R1 R2 R3 ... R22 R23)Program elements (genetic programming)... any data structure …

Generic Genetic AlgorithmIncorporates a heuristic hN based on a neighborhood function N and

a recombination operationrec: x → x

Find members that are similar or close to X

Generic Genetic AlgorithmIncorporates a heuristic hN based on a neighborhood function N

and a recombination operationrec: x → x

Reproduction

1. Number of generations2. Until the algorithm

converged

Parents are selected to mate, on the basis of their fitness



Operator types are:• Crossover (recombination)• Mutation

• In biology a crossover happens when chromosomes from differentparents are split and recombine; the new cell has partial strands from both parents

• Prime distinguished factor of GA from other optimization techniques

Crossovers

Partially matched crossover = {1,….,n}Two parents are α and β, two children are γ and δTwo crossover points j and k where 1≤j≤k≤n

Example…Assume thatα =[3,1,4,7,6,5,2,8] and β=[8,6,4,3,7,1,2,5] Crossover points are j=3 and k=6

i=3 No effectsi=4 γ=[7,1,4,3,6,5,2,8] and δ=[8,6,4,7,3,1,2,5]i=5 γ=[6,1,4,3,7,5,2,8] and δ=[8,7,4,6,3,1,2,5]i=6 γ=[6,5,4,3,7,1,2,8] and δ=[8,7,4,6,3,5,2,1]

• Adopting terminology from molecular biology, changes to chromosomes are called mutations

• Mutation alters one or more gene values in a chromosome from its initial state

• Purposes: • Inhibit premature convergence.

• User-definable mutation probability. • Should be a low probability.

* *Before: (6 5 4 3 7 1 2 8)After: (6 5 2 3 7 1 4 8)

Mutations



Evaluate population• Determined by an objective function

Genetic TSP

A Genetic Algorithm for TSP

Ramanathan

Travelling Salesman ProblemInstance: a complete graph Kn

a cost function C(X)

Find: a Hamiltonian circuit [x0, x1, … , xn-1] that minimizes C(X) = c(x0, x1) + c(x1, x2) + ... + c(xn-1, x0)

Universe: X = set of all (n-1)!/2 permutations For 10 cities, 181440 tours are possible

IdeaWhen we find a good tour, try to improve it using mutation and recombination

The recombined path could be better than either parents’ path

Substring from parent 1’s path

Substring from parent 2’s path

Preliminaries

Fitness: Overall cost of the tour

Selection: Initial population drawn from a random set of tours

Recombination: Partially matched crossover

Mutation: Steepest ascent 2-opt heuristic

2-opt move

Once a tour has been generated by some tour construction heuristic, we need to improve that solution

2-opt move: 1. Remove two edges from the tour2. Reconnect the two paths created to form a new tour

We complete the move only if the new tour will be shorter

2-opt move

xi

xi+1

xj

xj+1

xi

xi+1

xj

xj+1

The gain in applying a 2-opt move is the decrease in cost.G(X,i,j) = C(X) – C(Xij)

= c(xi,xi+1) + c(xj,xj+1) – c(xi+1,xj+1) – c(xi,xj)

N(x) = all Y ∈ X that can be obtained from x by a 2-opt move

Mutation heuristic consists of a sequence of 2-opt moves

Using a steepest ascent strategy, iteratively apply 2-opt moves until no pair of edges can be found to yield a positive gain

Steepest Ascent method

Input: Given tour XOutput: Optimal tour gainmax = 0;while (gainmax > 0) do

gainmax = 0;for i = 0 to (n-1) do

for j = (i+2) to (n-1) dog = G(X, i, j);if (g > gainmax) then

gainmax = g; imax = i; jmax = j;if (gainmax > 0) then

X = Ximax,jmax;

Algorithm

Example

Fitness function : Euclidean distance

Example

Random tourTour length = 1.7938

Example

Tour length = 1.7938

Example


Example

Tour length = 1.006

Example


Example

Optimal tour length = 0.9678

Partially Matched Crossover Algorithm

Input: Parent tours A, B Minimum bound on the substring length, h

Output: Offspring tours C, D

j = random number (0, n-1); (start of the substring)k = random number (h, n/2); (length of the substring)(C, D) = PartiallyMatchedCrossover(A, B, j, (j+k) mod n)SteepestAscentTwoOpt(C);SteepestAscentTwoOpt(D);return (C, D);

Select initial population

Unranking algorithm: intuition

Unranking algorithm: intuition vs algorithm

Π(1)

dn-1r

Π

Π(2)

dn-2

Π(n-1)

d1

Π(n)

1

r’

intuition

algorithm

Unroll recursive

Unranking permutation generating j

d

r

Π

0

0

15

1

1

1

14

2

2

1

12

231

3

2

0

3241

Genetic TSP

Applications

Puneet

Applications

• Speech Recognition• Mobile Robotics• Singapore MRT• Computer Aided Design• Economics

Speech Recognition• Language Model• Acoustic Model• Initially a simple language model and an acoustic

model is used for speech recognition and results are stored in a speech lattice.

• However it will be error prone.• Rescoring of the speech lattice is done using the

sophisticated LM.• Neighborhoods are created using edit distance metrics. • Hill Climbing is applied to find the path with the highest

probability (which is the final output).

Speech Recognition

• Edit Distance is used for defining the neighbourhood.

• Hill Climbing is used to find the highest probable path in the lattice after rescoring.

Edit Distance

Artificial Intelligence

Mobile Robotics

• Gradients are used to determine most likely pose of a robot given sensor measurements.

• The computation of gradient gave real time performance using hill climbing algorithm.

Computer Aided Design

Computer Aided Design

• Genetic Algorithm takes feedback from evaluation process to select fitter designs.

• Generates new design through the recombination of selected parts of the design

• Results in a population of high performance designs.

Satellite Truss Design

• Genetic Algorithms optimized satellite truss designs for NASA to maximize vibration control.

• Evolving: Design structures

• Fitness: Vibration resistance

• Evolutionary “creativity”

Optimized Structure

Real Time Systems

Mass Rapid Transit

• Genetic Algorithm was used for TimeTable Synchronization in Singapore MRT.

Economics

• Game Theory

Game Theory

• Study of multi person decision problems.• Each rival player has to consider what the

others will do.• All players play rationally to maximize their

reward• Genetic algorithms find the most optimal

strategy for a player in a given situation.

Conclusion

• Introduction• Hill Climbing

– Uniform Graph Partition– STS

• Genetic Algorithm– TSP

• Applications

combinatorial algorithms

Documents

subset of x

finite set x

infeasible elements

solution state problem

particular nx shouldnt

feasible solution

neighborhood functionn

partial solution