cs 461 – aug. 24

CS 461 – Aug. 24

• Course overview

• Section 1.1– Definition of finite automaton (FA)– How to draw FAs

Overview

• What is Computational Theory?– Not so concerned with specific HW/SW– Search for general truths, models, imagination

Ex. What is possible vs. impossible?

• Facets– Automata & Formal Languages (Ch. 1 – 3)

There are 3 important models of computation.– Computability (Ch. 3 – 5)

“Some problems can’t be solved.”– Complexity (Ch. 7)

“Some problems take too long to solve.”

Theory

• What is theory?• You have already dealt with abstract ideas in

earlier courses– Difference between void, null, zero– Different kinds of zeros (int, float, positive, negative)– Infinity versus NaN– Correctness of an algorithm– Number of iterations in a loop– Does recursion really work?

Computation Models

• Purpose of a model is to study what computers and programs can do.

• Models range from simple to complete:– Model #1: Finite Automaton– Model #2: Pushdown Automaton– Model #3: Turing Machine

• Assume programming essentially boils down to input outputs.

• For simplicity / convenience, – input = any binary string– output = 0/1, T/F, accept/reject

Finite Automaton

• Our first model – Chapter 1• Purpose of FA is to take (binary) input string and

either “accept” or “reject” it.

• The set of all strings accepted by an FA is called the language of the FA.

• “Language” simply means a set of strings, e.g. the set of all binary strings starting with “0”.

Example

• Start state• Accept/happy state(s)

– If we are in one of these when input is done, we accept string.

• Transitions• “finite” refers to # of

states. More on limitations later.

s1 s2

1 0, 1

0

Example

• The FA could also be expressed in a table.

• The table tells us where to go on each possible input bit.

From Input 0

Input 1

s1 s2 s1

s2 s2 s2

s1 s2

1 0, 1

0

Example

• Let’s try sample input like 101.

• Can you figure out the language of this machine?– i.e. How do we get to

the happy state?

s1 s2

1 0, 1

0

Example #2

• What is language of this FA?• Note: if we change which state is the accept

state, we’d have a different language!

s1 s2 s3

0

0

1

0, 11

CS 44 – Aug. 26

• Finish section 1.1– Definition of FA– Examples– Regular languages– Union of 2 regular languages (handout)

Formal Definition

A finite automaton has 5 things:1. A set of states

2. An alphabet, for input

3. A start state

4. A set of accept states

5. Transition function: δ(state, input) = state

• When we create/define an FA, must specify all 5 things. A drawing does a pretty good job.

Example

s1 s2 s3

0

0

1

0, 11

From state Input 0 Input 1

s1 s2 s1

s2 s1 s3

s3 s3 s3

Let’s make FA’s

1. L = bit strings that have exactly two 1’s

2. L = starts with 01

3. L = ends with 01

4. L = has an even number of 1’s

5. L = starts and ends with same symbol

*** Very good idea to give meaningful names to your states.

Since L is a set… how would we create an FA for the complement of L?

Regular Language

• A regular language is a set of strings accepted by some FA.

• Examples– Starting with 01; ending with 01, containing 01.– Strings of length 2.– {0, 11}

• Any finite set is regular. Infinite sets are more interesting. Yet, # states always finite!

Operations on sets• We can create new regular sets from existing

ones, rather than starting from scratch.

• Binary operations– Union– Intersection

• Unary operations– Complement – Star: This is the concatenation of 0 or more

elements. For example, if A = {0, 11}, then A* is { ε, 0, 11, 00, 1111, 011, 110, …}

• “Closure property”: you always get a regular set when you use the above operations.

Union

• Book shows construction (see handout)

• We want to union two languages A1 and A2: M1 accepts A1 and M2 accepts A2. The combined machine is called M.

• M pretends it’s running M1 and M2 at same time!

δ((r1,r2),a) = (δ1(r1,a),δ2(r2,a))

• # states increases because it’s the Cartesian product of M1 and M2’s states.

• Next section will show easier way to do union.

Wolf, Goat, Cabbage

• A man would like to cross a river with his wolf, goat and head of cabbage.

• Can only ferry 1 of 3 items at a time. Plus:– Leaving wolf & goat alone: wolf will eat goat.– Leaving goat & cabbage alone: goat will eat

cabbage.

• Yes, we can solve this problem using an FA!– Think about possible states and transitions.

CS 44 – Aug. 29

• Regular operations– Union construction

• Section 1.2 - Nondeterminism– 2 kinds of FAs– How to trace input – NFA design makes “union” operation easier – Equivalence of NFAs and DFAs

Wolf, Goat, Cabbage

• A man would like to cross a river with his wolf, goat and head of cabbage.

• Can only ferry 1 of 3 items at a time. Plus:– Leaving wolf & goat alone: wolf will eat goat.– Leaving goat & cabbage alone: goat will eat

cabbage.

• Yes, we can solve this problem using an FA!– Think about possible states and transitions.

Operations on sets• We can create new regular sets from existing

ones, rather than starting from scratch.

• Binary operations– Union– Intersection

• Unary operations– Complement – Star: This is the concatenation of 0 or more

elements. For example, if A = {0, 11}, then A* is { ε, 0, 11, 00, 1111, 011, 110, …}

• “Closure property”: you always get a regular set when you use the above operations.

Union

• Book shows construction (see handout)

• We want to union two languages A1 and A2: M1 accepts A1 and M2 accepts A2. The combined machine is called M.

• M pretends it’s running M1 and M2 at same time!

δ((r1,r2),a) = (δ1(r1,a),δ2(r2,a))

• # states increases because it’s the Cartesian product of M1 and M2’s states.

• Next section will show easier way to do union.

Non-determinism

• There are 2 kinds of FA’s– DFA: deterministic FA– NFA: non-deterministic FA

• NFA is like DFA except:– A state may have any # of arrows per input symbol– You can have ε-transitions. With this kind of transition,

you can go to another state “for free”.

• Non-determinism can make machine construction more flexible. At first the theory seems more complex, but NFA’s will come in handy.

Example

s1 s2 s3 s41 0, ε 1

0, 10, 1

From state Input 0 Input 1 Input ε

s1 s1 s1, s2

s2 s3 s3

s3 s4

s4 s4 s4

continued

• See the non-determinism?• Remember, any time you reach a state that has

ε-transitions coming out, take it! It’s free.• Let’s trace input 010110.

s1 s2 s3 s41 0, ε 1

0, 10, 1

s1 s2 s3 s41 0, ε 1

0, 10, 1

Start Read 0 Read 1 Read 0 Read 1 Read 1 Read 0

s1 s1

s1 s1

s1

s1 s1

s2 s3

s3 [dies]

s2 [dies]

s3 s4 s4

s2 s3 s4

s3 [dies]

Moral

• When tracing a word like 010110, we just want to know if there is any way to get to the accept state.

• Language is anything containing 11 or 101.

• Corresponding DFA would have more states.

CS 461 – Aug. 31

Section 1.2 – Nondeterministic FAs

• How to trace input √

• NFA design makes “union” operation easier

• Equivalence of NFAs and DFAs

NFA’s using “or”

• Can you draw NFA for:

{ begin with 0 or end with 1 } ?

New start

Old start 1

Old start 2

ε

ε

Amazing fact

• NFA = DFA• In other words, the two kinds of machines have

the same power.• Proof idea: we can always convert a DFA into

an NFA, or vice versa. Which do you think is easier to do?

Formal NFA def’n

• The essential difference with DFA is in the transition function:

DFA δ: Q x Σ Q

NFA δ: Q x Σε P(Q)

• Thus, converting DFA NFA is easy. We already satisfy the definition!

NFA DFA construction

1. When creating DFA, states will be all possible subsets of states from NFA.

– This takes care of “all possible destinations.”– In practice we won’t need whole subset: only create

states as you need them.– “empty set” can be our dead state.

2. DFA start state = NFA’s start state or anywhere you can begin for free. Happy state will be any subset containing NFA’s happy state.

3. Transitions: Please write as a table. Drawing would be too cluttered. When finished, can eliminate useless states.

Example #1

• NFA transition table given to the right.

• DFA start state is {1, 3}, or more simply 13.

• DFA accept state would be anything containing 1. Could be 1, 12, 13, 123, but we may not need all these states.

inputs

state a b ε

1 - 2 3

2 2,3 3 -

3 1 - -

continued

• The resulting DFA could require 2n states, but we should only create states as we need them.

inputs

state a b ε

1 - 2 3

2 2,3 3 -

3 1 - -

Let’s begin:

If we’re in state 1 or 3, where do we go if we read an ‘a’ or a ‘b’?

δ(13, a) = 1, but we can get to 3 for free.

δ(13, b) = 2. We need to create a new state “2”.

Continue the construction by considering transitions from state 2.

answer

NFA

DFA

inputs

state a b ε

1 - 2 3

2 2,3 3 -

3 1 - -

inputs

state a b

13 13 2

2 23 3

23 123 3

123 123 23

3 13

Notice that the DFA is in fact deterministic: it has exactly one destination per transition. Also there is no column for ε.

Example #2


• DFA start state is A.

• DFA accept state would be anything containing D.

inputs

State 0 1 ε

A A A,C -

B D - C

C - B -

D B D -

continued

Let’s begin.

δ(A, 0) = A

δ(A, 1) = AC

We need new state AC.

δ(AC, 0) = A

δ(AC, 1) = ABC

Continue from ABC…

inputs

State 0 1 ε

A A A,C -

B D - C

C - B -

D B D -

answer

NFA DFA

inputs

State 0 1 ε

A A A,C -

B D - C

C - B -

D B D -

inputs

State 0 1

A A AC

AC A ABC

ABC AD ABC

AD ABC ACD

ACD ABC ABCD

ABCD ABCD ABCD

final thoughts

• NFAs and DFAs have same computational power.

• NFAs often have fewer states than corresponding DFA.

• Typically, we want to design a DFA, but NFAs are good for combining 2+ DFAs.

• After doing NFA DFA construction, we may see that some states can be combined.– Later in chapter, we’ll see how to simplify FAs.

CS 461 – Sept. 2

• Review NFA DFA• Combining FAs to create new languages

– union, intersection, concatenation, star– We can basically understand how these work by

drawing a picture.

• Section 1.3 – Regular expressions– A compact way to define a regular set, rather than

drawing an FA or writing transition table.

Example #2


• DFA start state is A.

• DFA accept state would be anything containing D.

inputs

State 0 1 ε

A A A,C -

B D - C

C - B -

D B D -

continued

Let’s begin.

δ(A, 0) = A

δ(A, 1) = AC

We need new state AC.

δ(AC, 0) = A

δ(AC, 1) = ABC

Continue from ABC…

inputs

State 0 1 ε

A A A,C -

B D - C

C - B -

D B D -

answer

NFA DFA

inputs

State 0 1 ε

A A A,C -

B D - C

C - B -

D B D -

inputs

State 0 1

A A AC

AC A ABC

ABC AD ABC

AD ABC ACD

ACD ABC ABCD

ABCD ABCD ABCD

Moral

• NFAs and DFAs have same computational power.

• NFAs often have fewer states than corresponding DFA.

• Typically, we want to design a DFA, but NFAs are good for combining 2+ DFAs.

• After doing NFA DFA construction, we may see that some states can be combined.– Later in chapter, we’ll see how to simplify FAs.

U and ∩

• Suppose M1 and M2 are DFAs. We want to combine their languages.

• Union: We create new start state. √– Do you understand formalism p. 60 ?

• How can we also do intersection?

Hint: A ∩ B = (A’ U B’)’

Concatenation

• Concat: For each happy state in M1, turn it into a reject state and add ε-trans to M2’s start.

• ExampleL1 = { does not contain 00 }

L2 = { has even # of 1’s }

Let’s draw NFA for L1L2.

• Let’s decipher formal definition of δ on p. 61.

Star

• We want to concat the language with itself 0+ times.

• Create new start state, and make it happy.• Add ε-transitions from other happy states to the

start state.• Example

L = { begins with 1 and ends with 0 }

Let’s draw NFA for L*.

• Formal definition of δ on p. 63.

Regular expression

• A concise way to describe a language– Text representation, straightforward to input into

computer programs.

• Use alphabet symbols along with operators+ means “or”

* means repetition

Concatenation

• There is no “and” or complement.

Examples

• What do you think these regular expressions mean?

0* + 1*

0*1*

00*11* (a shorthand would be 0+1+ )

(0 + 1)*• What’s the difference between 10*1 and

1(0+1)*1 ? Does this mean “anything with 1 at beginning and end?”

Practice

• Words with 2 or more 0’s. What’s wrong with this answer: 1*01*01 ?

• Words containing substring 110.• Every even numbered symbol is 0.

– What’s wrong with: ((0 + 1)*0)* ?

• Words of even length.• The last 2 symbols are the same.

• What is the shortest word not in: 1*(01)*0* ?• True or false: (111*) = (11 + 111)*

CS 461 – Sept. 7

• Section 1.3 – Regular expressionsExamples

Equivalent to FA’s.

regular expression FA (straightforward)

FA regular expression (* major skill)

Regular expression

• A concise way to describe a language– Text representation, straightforward to input into

computer programs.

• Use alphabet symbols along with operators+ means “or”

* means repetition

Concatenation

• There is no “and” or complement.

Examples

• What do you think these regular expressions mean?

0* + 1*

0*1*

00*11* (a shorthand would be 0+1+ )

(0 + 1)*• What’s the difference between 10*1 and

1(0+1)*1 ? Does this mean “anything with 1 at beginning and end?”

Practice

• Words with 2 or more 0’s. What’s wrong with this answer: 1*01*01 ?

• Words containing substring 110.• Every even numbered symbol is 0.

– What’s wrong with: ((0 + 1)*0)* ?

• Words of even length.• The last 2 symbols are the same.

• What is the shortest word not in: 1*(01)*0* ?• True or false: (111*) = (11 + 111)*

Conversions

• It’s useful to be able to convert among:– English description of language– FA drawing– Regular expression

• Also good practice to consider complement of a language.

• Let’s practice writing regular expressions.– General technique: think about how to get to happy

state. Sometimes difficult, so we’ll look at general algorithm later.

Reg exp FA

• Just build the FA in pieces.– We already know how to combine FA’s using union,

concatenation, star – these are the basic operations in regular expressions.

– You may even be able to construct the FA intuitively.

• Let’s try examples– 0*10– 0* + 1*

FA reg exp

• If the FA doesn’t have too many “loops,” it’s not hard to write down the regular expression.

• Try this one. Also think about what would happen if a different state is the accept state.

s1 s2 s3

s4

1 1

0

0, 1

0, 1

0

In other words

s1 s2 s3

s4

1 1

0

0, 1

0, 1

0


s1 s4 s2

s2 s2 s3

s3 s4 s4

s4 s4 s4

CS 461 – Sept. 9

• Section 1.3 – Regular expressionsExamples Equivalent to FA’s. regular expression FA (straightforward)

FA regular expression (* major skill)

0, 1

0

1

0

0, 1

0

1

1

0, 1

0

This one is more ambitious…

Think about L and L’.

1

State 0 1

s1 s2 s2

s2 s3 s7

s3 s7 s4

s4 s5 s7

s5 s6 s6

s6 s1 s7

s7 s7 s7

Need help

• Often an FA is too complex to just figure out the language just by looking at it.

• Need a general procedure!

s1 s2

0

0

1 1

01* (01*0) ?????

Nope!

Actually, it should be

1*0(1* + 01*0)*

State 0 1

s1 s2 s1

s2 s1 s2

Procedure

• Step-by-step transformation of FA, taking away one state at a time until we just have one start and one happy.

• We’ll label edges with regular expressions.– So our FA will be a “generalized” NFA

• Step 1: Add a new start and new happy state. • Step 2: For each of the other states:

– Pick one to be ripped out.– Recalculate edges (except to start or from happy)

taking this removal into account.

Basic idea

Suppose we are getting rid of state X.

To go from A to B, you can either get their directly, or via X. We would label edge from A to B as:

δ(A,B) = δ(A,B) + δ(A,X) δ(X,X)* δ(X,B)

A B

X

Notes:1. If X has no edge to itself,

Just say ε.2. If no edge AX or XB,

we say Ø and thusno need to recalculate AB.

Example

• First create new start and happy states.

• New machine below…

s1 s2

0

0

1 1

s1 s2

0

0

1 1

start end

ε ε

In other words

s1 s2

0

0

1 1

s1 s20

0

1 1

start end

ε ε

State 0 1

s1 s2 s1

s2 s1 s2

State 0 1 ε

start s1

s1 s2 s1

s2 s1 s2 end

end

Get rid of s1

• Need to compute δ(start,s2) and δ(s2,s2).δ(start,s2) = Ø + ε1*0, which simplifies to 1*0

δ(s2, s2) = 1 + 01*0

s1 s2

0

0

11

start end

ε ε

s2???

???

start endε

In other words

State 1*0 1+01*0 ε

start s2

s2 s2 end

end

State 0 1 ε

start s1

s1 s2 s1

s2 s1 s2 end

end

Becomes:

Get rid of s2

• Calculate δ(start, end) = Ø + 1*0 (1+01*0)* ε

which simplifies to 1*0 (1 + 01*0)*.

s21*0

1 + 01*0

start endε

???start end

Example #2

• Create new start & happy states.• What happens when we get rid of state 1?

1 2

b

aa, b

1 2

b

a

start end

a, b

ε ε

In other words

1 2b

aa, b

1 2

b

a

start end

a, b

ε ε

State a b ε

start 1

1 1 2

2 2 2 end

end

Get rid of state 1δ(start, 2) = Ø + ε a*b = a*b

δ(start, ) = Ø + ε a* Ø = Ø

δ(2, 2) = a + b + Ø… = a + b

δ(2, ) = ε

1 2

b

a

start end

a, b

ε ε

continued

Finally we can get rid of state 2 to obtain: a*b (a + b)*

2a*b

start end

a + b

ε

CS 461 – Sept. 12

• Simplifying FA’s.– Let’s explore the meaning of “state”. After all if we

need to simplify, this means we have too many states.

– Myhill-Nerode theorem

– Handout on simplifying

What is a state?

• No matter what next input symbol is, all words in same state act alike.

x, y same state z, xz same state as yz• A state is a collection of words that react to input

in the same way.

need 1 1 1

0 0 0, 1

Example: 0*10*

state = 0* state = 0*10* state = 0*10*1 (0 + 1)*

Equivalent states

• Whether we’ve seen even or odd number of 0’s shouldn’t matter. Only concern is ultimate outcome: will string be accepted or not?

• Words x and y should be in same state if z, xz and yz have the same outcome.

even 1 1

0 0, 1

odd

0 01

In other words

• The 2 machines are equivalent.


Need 1 Need 1 Good

Good Good Bad

Bad Bad Bad


Even Odd Good

Odd Even Good

Good Good Bad

Bad Bad Bad

Myhill-Nerode theorem

• Basis for simplification algorithm.• Also gives condition for a set to be regular.

– i.e. infinite # states not allowed.

3 parts to theorem:1. For any language L, we have equivalence relation R:

xRy if z, xz and yz same outcome.

2. If L is regular, # of equivalences classes is finite.

3. If # equivalences classes finite, language is regular.

Proof (1)

For any language L, we have equivalence relation R:xRy if z, xz and yz same outcome.

• To show a relation is an equivalence relation, must show it is reflexive, symmetric and transitive.

• Reflexive: xz and xz have same outcome. (i.e. both are accepted, or both are rejected.)

• Symmetric. If xz has same outcome as yz, then yz has same outcome as xz.

• Transitive. If xz has same outcome as yz, and yz has same outcome as tz, then xz has same outcome as tz.

All 3 are obviously correct.

Proof (2)

If L is regular, # of equivalences classes is finite.• Regular means L is recognized by some FA.• Thus, # of states is finite.• It turns out that (# equiv classes) <= (# states)

Why? Because 2 states may be “equivalent.”More importantly: # classes can’t exceed # states. Proof:

Assume # classes > # states. Then we have a state representing 2 classes. In other words, x and y in the same state but x not related to y. Meaning that z where xz, yz don’t have same fate but travel thru the same states! This makes no sense, so we have a contradiction.

• Since (# equiv classes) <= (# states) and # states is finite, we have that # equiv classes is finite.

Proof (3)

If # equivalences classes finite, language is regular.• We prove regularity by describing how to construct an

FA.

• Class containing ε would be our start state.• For each class: consider 2 words x and y.

– x0 and y0 have to be in the same class.– x1 and y1 have to be in the same class.– From this observation, we can draw appropriate transitions.– Since states are being derived from classes, the number of

states is also finite.

• Accept states are classes containing words in L.

Example

• L = { all bit strings with exactly two 1’s } has 4 equivalence classes

[ ε ], [ 1 ], [ 11 ], [ 111 ]

• Let’s consider the class [ 11 ]. This includes words such as 11 and 0101.– If we append a 0, we get 110 and 01010. These words

also belong in [ 11 ].

– If we append a 1, we get 111 and 01011. These words belong in the class [ 111 ].

• This is the same thought process we use when creating FA’s transitions anyway.

Learning from theorem

• There exist non-regular languages! It happens if # of equivalence classes is infinite.– Soon we’ll discuss another method for determining

non-regularity that’s a little easier.

• Part 3 of proof tells us that there is an FA with the minimum number of states (states = equivalence classes).– See simplification algorithm handout.

Example

• Should x=0 and y=00 be in the same class? No!

Consider z = 1. Then xz = 01 and yz = 001. Different outcome!

• ∞ # classes Can’t draw FA.• Equivalence classes are usually hard to conceive of, so

we’ll rely more on a different way to show languages not regular.

no 0’s one 0 two 0’s Etc.

c1 = { ε } c2 = { 0 } c3 = { 00 } c4 = { 000 }

Is { 0n1n } regular? This is the language ε, 01, 0011, 000111, etc.

0 0 0

CS 461 – Sept. 14

• Practice simplifying FA’s

• Section 1.4 – Pumping lemma

– Purpose is to show that a language is not regular– Proof– Examples– Can also help to determine if a language is

finite/infinite.

Non-regular languages• Means there is no such FA for language.• They do exist! How can a language be non-

regular?– Not because language is too big, because the largest

possible language (0 + 1)* is regular.– Non-regular means the “shape” of the language is

defective. It requires an infinite number of states. Finite # of states can’t precisely define the language. You’ll discover that a finite # of states winds up defining something else.

finite regular

Essential idea

The pumping lemma basically says this:

• If L is regular, you should be able take almost any word in L, and repeat some of its characters in order to produce longer words in L.

– Incidentally, you should also be able to take away some of its characters to produce a shorter word.

– Degenerate case: finite sets.– “Almost any word”? Some words are not interesting

like ε and 0.

Motivation• Consider language { 0n1n }. • Suppose it’s regular. Then there must be some

FA for it.– How many states does it have? Let’s pick a number

like 95. – Then, how would we process the word 097197? While

reading the 0s, we have to visit the same state twice! In other words, we go back to a previous state, for example from s80 s81 s82 s80, and then continue on to accept state.

– There must be another word that leaves out the s81 and s82 steps, such as 094197, or does them repeatedly. But those words are not in L! Contradiction implies that L is not regular.

More generally…

• Suppose L is regular and M is FA recognizing it. Let p = # of states.

• Consider a word s, |s| p. We take at least p transitions, so we visit at least p+1 states. So, we must visit some state twice while reading the first p symbols! Suppose we call this state q.

• Let s = xyz, where x is part of string up to first time we visit q, and y is the part of string between our 2 visits.

• Other words of the form xyiz are also accepted by M. In other words, we can loop as many times as we want, even 0 times.

A litmus test

• The pumping lemma is a guaranteed property of any regular language.

• Analogy…– If it’s a duck, then it will quack.– If it’s a cow, then it will moo.

• Contrapositive proves an imposter!– If it doesn’t quack, it’s not a duck.– If it doesn’t moo, it’s not a cow.

• However, can’t be used to prove regularity.

Pumping lemma

• See handout– Formal statement (if it’s a duck…)– Contrapositive used to show nonregularity

• If language is regular, then there is some part of any word we can “pump.”

• If we have a language we suspect is non-regular, use contrapositive!

Proving non-regularity

• It’s like a 2-player game…– Adversary picks secret number p.– We select any string we want, in terms of p

(e.g. 0p1p)– Adversary will break up s into xyz subject to

constraints.• The place to pump has length at least 1.• The place to pump appears in the first p positions.

– Be ready to show that xyiz won’t be in language for some i.

CS 461 – Sept. 16

• Review Pumping lemma– How do we show language non-regular?

• Applications of FA: – Scanning (part of compilation)– Searching for text, using regular expression

Proving non-regularity

• It’s like a 2-player game…– Adversary picks secret number p.– We select any string we want, in terms of p

(e.g. 0p1p)– Adversary will break up s into xyz subject to

constraints.• The place to pump has length at least 1.• The place to pump appears in the first p positions.

– Be ready to show that xyiz won’t be in language for some i.

Example languages

• Bit strings that are palindromes• Bit strings with equal # of 0s and 1s.• More 0s than 1s.• { 0n : n is a perfect square }• { 0n : n is prime }• { 03n + 2 : n 0 }• { 0i 1j : i is even or i j }

Notice that regular sets can’t handle counting or nonlinear patterns.

Consider complement

Show L = { 0i 1j : i is even or i j } is not regular.• Let’s use some set theory…

– L regular iff L’ regular

• What is L’ ?– Hint: Need to consider (0 + 1)* - 0*1*.– Language L’ has “and” instead of “or”, so easier to

produce a word not in language.

continued

L = { 0i 1j : i is even or i j } Then L’ is all words with:• The definition turned around.

Let A = { i is odd and i < j }.• Plus all words not of the form 0*1*.

Let B = (0 + 1)* - 0*1* = 0*11*0(0 + 1)*.Then, L’ = A union B.

Let s = 02p+1 12p+2 and i = 2 and see how it works in pumping lemma…

CS 461 – Sept. 19

• Last word on finite automata…– Scanning tokens in a compiler– How do we implement a “state” ?

• Chapter 2 introduces the 2nd model of computation

Application of FA

• Compilers! When a compiler looks at your program, it does the following:– Lexical analysis (scanning: break up into tokens)– Syntax analysis– Semantic analysis– Code generation– Optimization

Nextchar

Read/

Comment

Found “/=“

Div by number

/

*=

num

Chapter 2

• How can we represent a language?– English description √– FA drawing (state diagram) √– Transition table √– Regular expression √– Recursive definition!

• A context-free grammar (CFG) is a compact way of writing a recursive definition.

Expr vs. grammar

• Basic idea to creating a grammar:– Base case: what is smallest word in language?

(Sometimes we need > 1 base case.)– Recursive case: how to create bigger words?

What do we append to the beginning or end of word to create more words?

• Ex. 101* in the form of a grammar:– Shortest word is 10.– Create more words by appending 1s to the right end.

S 10

S S1

example

• Let’s try (00)*10 as a grammar:– Shortest word is 10.– Bigger words? Append 00 to the front.

S 10

S 00S– These rules can be combined to say S 10 | 00S

examples

• Let’s try 01*0(10)*.– Uh-oh, how do you pump in the middle? If you need

concatenation, define the language in parts. The first part will handle 01* and the second part 0(10)*.

S AB

A 0 | A1

B 0 | B10

• (10*1)* could be written this way:

S ε | SAB

A 1 | A0

B 1

Formal definition

• A context-free grammar has 3 things– Set of terminal symbols (alphabet plus ε)– Set of nonterminal symbols (i.e. variables in grammar)– Set of rules or “productions” for our recursive

definitions

• The rules have the following format:

nonterminal concat of 1+ terminals/nonterminals

• Often we have several rules for the same nonterminal, so they are joined by “|” meaning “or.”

More power

• CFGs are more powerful than FAs or regular expressions because they allow us to define non-regular languages!

• For example, 0*1* and { 0n 1n }

S AB S ε | 0S1

A ε | A0

B ε | B1• Wow, maybe non-regular languages are easier!• With CFGs, we now have a 2nd class of

languages, the context-free languages.

Models

# 1 # 2

Model name Finite automaton (FA)

Pushdown automaton (PDA)

Recognizes what?

Regular languages

Context-free languages

Text rep’n Regular expression

Context-free grammar

2nd model

• Encompasses regular & non-regular languages.• PDA is like FA, but also has a stack.

– More on PDAs in section 2.2 later.

• Section 2.1: CFGs– Important because this is how programming

languages are defined.– Goals:

1. Given a grammar, can we generate words;

2. Given a set, can we write a grammar for it.

Deriving words

• Here is a CFG. What is it defining?

S AB

A ε | 1A

B ε | 1B0

More practice

• { 0i 1j : i > j }• 0*11*0(0 + 1)*• (000)*11• 1*0 (1 + 01*0)*• (a + ba) b*a• [bb + (a + ba) b*a]*

CS 461 – Sept. 21

Context-free grammars• Handouts

– Hints on how to create grammar– Examples of generating words from a grammar

• Writing a grammar for a language

• Derivations

Formal definition

• A context-free grammar has 3 things– Set of terminal symbols (alphabet plus ε)– Set of nonterminal symbols (i.e. variables in grammar)– Set of rules or “productions” for our recursive

definitions

• The rules have the following format:

nonterminal concat of 1+ terminals/nonterminals

• Often we have several rules for the same nonterminal, so they are joined by “|” meaning “or.”

More power

• CFGs are more powerful than FAs or regular expressions because they allow us to define non-regular languages!

• For example, 0*1* and { 0n 1n }

S AB S ε | 0S1

A ε | A0

B ε | B1• Wow, maybe non-regular languages are easier!• With CFGs, we now have a 2nd class of

languages, the context-free languages.

Models

# 1 # 2

Model name Finite automaton (FA)

Pushdown automaton (PDA)

Recognizes what?

Regular languages

Context-free languages

Text rep’n Regular expression

Context-free grammar

2nd model

• Encompasses regular & non-regular languages.• PDA is like FA, but also has a stack.

– More on PDAs in section 2.2 later.

• Section 2.1: CFGs– Important because this is how programming

languages are defined.– Goals:

1. Given a grammar, can we generate words;

2. Given a set, can we write a grammar for it.

Practice

Should be able to write a grammar for any regular set, and some simple non-regular cases:

• { 0i 1j : i > j }• 0*11*0(0 + 1)*• (000)*11• 1*0 (1 + 01*0)*• (a + ba) b*a• [bb + (a + ba) b*a]*

Deriving words

• Here is a CFG. What is it defining?

S AB

A ε | 1A

B ε | 1B0

• Handouts: Generating random strings using a grammar.

Old Mother Hubbard

She went to the ____P____

to buy him a […….T……..

but when she same back,

he was _____D_____ the …....V…….]

T and V have to rhyme,So select them at the same time.

CS 461 – Sept. 23

Context-free grammars

• Derivations• Ambiguity• Proving correctness

Derivations

• A sequence of steps showing how a word is generated using grammar rules.

• Drawing a diagram or tree can help.• Example: derivation of 1110 from this grammar:

S AB S

A ε | 1A A B

B ε | 1B0 1A 1B0

11A 1ε0

11ε 10

11

Derivations, con’d

• There are several ways to express a derivation– Tree – Leftmost derivation = as you create string, replace

leftmost variable. Example:

S AB 1AB 11AB 11B 111B0 1110– Rightmost derivation

• Often, grammars are ambiguous– For some string, 2+ derivation trees. Or,

equivalently: 2+ leftmost derivations.– Example?

More examples

• Even number of 0’s

• Words of form { 0i 1j } where j is constrained.i j 2i (not to be confused with { 0n 12n } )

i j 2i + 3

i j 1.5i

• Next topic: how do we show that our grammar is correct?

Correctness

• Given a language L and a grammar G, how do we know if L = L(G)? Must show:

1. L L(G). All words in L can be derived by grammar. Tell how to derive the words.

2. L(G) L. All words derived by grammar are in L. Use induction.

First example

• Show that S ε | 0S1 is a CFG for the language { 0n1n }

• Step #1: Any word in 0n1n can be generated by the grammar. We explain the derivation.– Apply the rule 0S1 n times– Finally, apply the ε rule.

• Step #2: All words generated by the grammar are of the form 0n1n.– Induction on the number of times we use a rule. – S ε for the basis, and S 0S1 for the inductive step.

Second example

L is { even number of 0s } and G is

S ε | S1 | S0S0

Step #1. Suppose w L. How can G generate w?

If w = ε, we’re done.

Do the following until w is ε:• If w ends with 1s, take them off by virtue of “S1”.• Now w ends with 0, so take 2 0’s away by virtue of

“S0S0”. We know there must be 2 0’s in w because w had an even number to start with.

Illustration

S ε | S1 | S0S0

Let’s see how we can derive “010100”:• The last 2 0s: S S0S0 ε0ε0 = 00• 0101 is 010 + 1

010 is S S0S0 ε0S10 ε0ε10 = 010

1 is S S1 ε1 = 1

• We can reconstruct derivation from beginning in form of a tree.

L(G) is in L

S ε | S1 | S0S0

Step #2: Need to show all generated words are in L.

Base case: ε is in L. It has even number of 0s. Recursive case. Let w be a word generated by the

grammar. Assume it has even # 0s. We can create a larger word by applying rule “S1” or “S0S0”. S1 adds no more 0s. S0S0 adds two 0s. Either way the # of 0s stays even.

Extra example

• Let’s show that this grammar G

S ε | 0S1 | 00S111

generates the language L = { 0i 1j | i j 1.5i }

• Step #1: any word in L can be generated– This is the hard part, so let’s motivate the proof with

an example or two.

• Step #2: all words generated are in L– Just induction again.

L is in L(G)

S ε | 0S1 | 00S111

• How could we generate 011 116?– We have 5 extra 1s. So we use rule 00S111 five

times.– Undoing these steps, # 0s = 11-5*2 = 1 and # 1s =

16-5*3 = 1. Then we use 0S1 once and we are left with ε.

• Okay, how about 012 116?– We have 4 extra 1s, so use rule 00S111 four times.– Undoing these steps, # 0s = 12-4*2 = 4 and # 1s =

16-4*3 = 4. They match! So use “0S1” 4 times.

Thinking out loud…

S ε | 0S1 | 00S111

Let w = 0i 1j L. In other words, i j 1.5i

Consider the number j – i. This is the number of times to apply rule #3 (00S111).

Note that using rule #3 (j – i) times will account for 2(j – i) zeros.

Then apply rule #2 (0S1) the “appropriate number” of times. How many? Well, we want i 0’s and we’ve already got 2(j – i), so we need the difference: i – 2(j – i) = 3i – 2j.

Finishing step 1

S ε | 0S1 | 00S111


Use rule #1 once to start with empty word.

Use rule #3 (j – i) times.

Use rule #2 (3i – 2j) times.

Total # 0s = 3i – 2j + 2(j – i) = i

Total # 1s = 3i – 2j + 3(j – i) = j

Thus, the word can be generated.

L(G) is in L

S ε | 0S1 | 00S111

Now the easy part. Need to show that when we apply any rule, we preserve: i j 1.5i

Base case: ε has no 0s or 1s. 0 0 1.5*0 √

Recursive case. Let w be generated by the grammar with i 0s and j 1s satisfying i j 1.5i .

If we apply either rule 0S1 or 00S111, we can show (i + 1) (j + 1) 1.5(i + 1)

(i + 2) (j + 3) 1.5(i + 2)

(Need to work out arithmetic.)

Think about…

• Can you write a CFG for

{ 0i 1j | 0.5i j 1.5i } ?

in other words, the ratio of 1s to 0s must be between ½ and 3/2.

CS 461 – Sept. 26

• CFG correctness

• Section 2.2 – Pushdown Automata

Correctness

• Given a language L and a grammar G, how do we know if L = L(G)? Must show:

1. L L(G). All words in L can be derived by grammar. Tell how to derive the words.

2. L(G) L. All words derived by grammar are in L. Use induction.

First example

• Show that S ε | 0S1 is a CFG for the language { 0n1n }

• Step #1: Any word in 0n1n can be generated by the grammar. We explain the derivation.– Apply the rule 0S1 n times– Finally, apply the ε rule.

• Step #2: All words generated by the grammar are of the form 0n1n.– Induction on the number of times we use a rule. – S ε for the basis, and S 0S1 for the inductive step.

Second example

L is { even number of 0s } and G is

S ε | S1 | S0S0

Step #1. Suppose w L. How can G generate w?

If w = ε, we’re done.

Do the following until w is ε:• If w ends with 1s, take them off by virtue of “S1”.• Now w ends with 0, so take 2 0’s away by virtue of

“S0S0”. We know there must be 2 0’s in w because w had an even number to start with.

Illustration

S ε | S1 | S0S0

Let’s see how we can derive “010100”:• The last 2 0s: S S0S0 ε0ε0 = 00• 0101 is 010 + 1

010 is S S0S0 ε0S10 ε0ε10 = 010

1 is S S1 ε1 = 1

• We can reconstruct derivation from beginning in form of a tree.

L(G) is in L

S ε | S1 | S0S0

Step #2: Need to show all generated words are in L.

Base case: ε is in L. It has even number of 0s. Recursive case. Let w be a word generated by the

grammar. Assume it has even # 0s. We can create a larger word by applying rule “S1” or “S0S0”. S1 adds no more 0s. S0S0 adds two 0s. Either way the # of 0s stays even.

Extra example

• Let’s show that this grammar G

S ε | 0S1 | 00S111

generates the language L = { 0i 1j | i j 1.5i }

• Step #1: any word in L can be generated– This is the hard part, so let’s motivate the proof with

an example or two.

• Step #2: all words generated are in L– Just induction again.

L is in L(G)

S ε | 0S1 | 00S111

• How could we generate 011 116?– We have 5 extra 1s. So we use rule 00S111 five

times.– Undoing these steps, # 0s = 11-5*2 = 1 and # 1s =

16-5*3 = 1. Then we use 0S1 once and we are left with ε.

• Okay, how about 012 116?– We have 4 extra 1s, so use rule 00S111 four times.– Undoing these steps, # 0s = 12-4*2 = 4 and # 1s =

16-4*3 = 4. They match! So use “0S1” 4 times.

Thinking out loud…

S ε | 0S1 | 00S111


Consider the number j – i. This is the number of times to apply rule #3 (00S111).

Note that using rule #3 (j – i) times will account for 2(j – i) zeros.

Then apply rule #2 (0S1) the “appropriate number” of times. How many? Well, we want i 0’s and we’ve already got 2(j – i), so we need the difference: i – 2(j – i) = 3i – 2j.

Finishing step 1

S ε | 0S1 | 00S111


Use rule #1 once to start with empty word.

Use rule #3 (j – i) times.

Use rule #2 (3i – 2j) times.

Total # 0s = 3i – 2j + 2(j – i) = i

Total # 1s = 3i – 2j + 3(j – i) = j

Thus, the word can be generated.

L(G) is in L

S ε | 0S1 | 00S111

Now the easy part. Need to show that when we apply any rule, we preserve: i j 1.5i

Base case: ε has no 0s or 1s. 0 0 1.5*0 √

Recursive case. Let w be generated by the grammar with i 0s and j 1s satisfying i j 1.5i .

If we apply either rule 0S1 or 00S111, we can show (i + 1) (j + 1) 1.5(i + 1)

(i + 2) (j + 3) 1.5(i + 2)

(Need to work out arithmetic.)

Think about…

• Can you write a CFG for

{ 0i 1j | 0.5i j 1.5i } ?

in other words, the ratio of 1s to 0s must be between ½ and 3/2.

PDAs

What is a Pushdown Automaton?• Like an FA, but PDA has a stack

– Finite number of states– Stack can grow to unlimited depth

• Transition function often non-deterministic• 2 flavors

– Accept by happy state– Accept by empty stack (this one has fewer states)

• Can be tedious to draw, so instead give a table.– What’s new is saying what to do with stack.

Formal definition

• A PDA has 6 things.– Same five from FA, plus:– Stack alphabet (can be different from Σ)

• Transition function

δ(state, tos, input) = (push/pop, new state)

both are optional

{ 0n 1n }

• While reading 0s, push them.

• If you read 1, change state. Then, as you read 1s, pop 0s off the stack.

• Watch out for bad input!– Unspecified transition crash (reject)

• Now, let’s write this in the form of a table.

PDA δ for { 0n 1n }

State Reading 0 Reading 1

Tos (don’t care) 0 Empty

Input 0 1 0 1 0 1

Action Push 0

Go to state

“reading 1”

Crash Pop Crash Crash

Notes:

Action depends on input symbol AND what’s on top of stack.

Action includes manipulating stack AND/OR changing state.

CS 461 – Sept. 28

• Section 2.2 – Pushdown Automata– { 0n 1n }– Palindromes– Equal

• Next: Converting CFG PDA

{ 0n 1n }

• While reading 0s, push them.

• If you read 1, change state. Then, as you read 1s, pop 0s off the stack.

• Watch out for bad input!– Unspecified transition crash (reject)

• Now, let’s write this in the form of a table.

PDA δ for { 0n 1n }

State Reading 0 Reading 1

Tos (don’t care) 0 Empty

Input 0 1 0 1 0 1

Action Push 0

POP & Go to state “reading

1”

Crash Pop Crash Crash

Notes:

Action depends on input symbol AND what’s on top of stack.

Action includes manipulating stack AND/OR changing state.

Example #2

• { w # wR } (easier form of palindrome)

• Let’s design a PDA that accepts its input by empty stack, as before.

• Think about: state, top-of-stack, input, action

PDA δ for { w # wR }

State Before # After #

Tos (don’t care) 0 1

Input 0 1 # 0 1 # 0 1 #

Action Push 0

Push 1

Go to “after

#”

Pop Crash Crash Crash Pop Crash

Palindrome PDA?

State Before # After #

Tos (don’t care) 0 1

Input 0 1 # 0 1 # 0 1 #

Action Push 0

Push 1

Go to “after

#”

Pop Crash Crash Crash Pop Crash

Changes needed:

Non-deterministically go to “after #” when you push 0 or 1.

Also, non-deterministically don’t push, in case we are dealing with odd-length palindrome!

Other examples

Think about these

• “equal” language– How many states do we need? …

• More 1s than 0s.

• Twice as many 1s as 0s.– Hint: think of the 0s as counting double.

Equal PDA

State (Just one state)

Tos ε 0 1

Input 0 1 0 1 0 1

Action Push 0

Push 1

Push 0

Pop Pop Push 1

CS 461 – Oct. 3

• Converting a CFG into a PDA

• Union, intersection, complement of CFL’s

• Pumping lemma for CFL’s

CFG PDAAlgorithm described in book• Have state 1 push S on stack, and go to state 2.• State 2 actions:

– (Note that any symbol in grammar could be tos, and any terminal could be input.)

– When tos is variable, ignore input. Action is to replace variable with its rules on stack.

• Multiple rules non-determinism • Push symbols out of order.• Push ε means leave stack alone.

– When tos is terminal, need action for when input=tos: just pop it. Don’t need action for !=, just let non-deterministic instance disappear.

Example: S AB

A ε | 1A2

B ε | B3

State 1 2

Tos ignore S A B 1 2 3

Input ignore ignore ignore ignore 1 2 3

Action Push S goto

2

Pop S

Push AB

Pop A Pop A

Push 1A2

Pop B Pop B

Push B3

Pop Pop Pop

Notice how the grammar rules are encoded into the machine!

Let’s trace 1122333

State 1 2

Tos ignore S A B 1 2 3

Input ignore ignore ignore ignore 1 2 3

Action Push S goto

2

Pop S

Push AB

Pop A Pop A

Push 1A2

Pop B Pop B

Push B3

Pop Pop Pop

S

A

B

1

A

2

B

A

2

B

1

A

2

2

B

A

2

2

B

2

2

B

2

B B

B

3

B

3

3

B

3

3

3

3

3

3

3

3 3

Combining CFLs

• Let’s write grammars for

L1 = { 1i 2i 3j } and L2 = { 1i 2j 3j }

• Grammar for L1 U L2 ? Simple technique.

• How about ∩ ? L1 ∩ L2 = { 1n 2n 3n }

Soon, we’ll prove this is not CFL.

What can we conclude?

• Complement? Remember: A ∩ B = (A’ U B’)’

Pumping lemma

L is a CFL implies:

There is a p,

such that for any string s, |s| >= p

We can break up s into 5 parts uvxyz:

| v y | >= 1

| v x y | <= p

and uvixyiz L.

• How does this differ from the first pumping lemma?

In other words

• If L is a CFL, then for practically any word w in L, we should be able to find 2 substrings within w located near each other that you can simultaneously pump in order to create more words in L.

• Contrapositive: If w is a word in L, and we can always create a word not in L by simultaneously pumping any two substrings in w that are near each other, then L is not a CFL.

Gist

• A string can grow in 2 places.– When we write recursive rules, we can add symbols

to left or right of a variable.• Ex. S #A#

A ε | 1A2

Look at derivation of: #111…222#• Even a complex rule like S S1S1S2 can be

“simplified”, and still only need 2 places to grow the string.– This is because any CFG can be written in such a way that there

are no more than 2 symbols on the right side of .

Example• L = { 1n 2n } should satisfy the pumping lemma.

Let p = 2. Strings in L of length >= 2 are:

{ 12, 1122, 111222, … }. All these words have both a 1 and a 2, which we can pump.

Just make sure the 2 places to pump are within p symbols of each other. So let’s say they are in the middle.

u v x y z

11111…1 1 ε 2 22222….2

Could p = 1 work?

CS 461 – Oct. 5

• Pumping lemma #2– Understanding– Use to show a language is not a CFL

• Next: Applications of CFLs– Expression grammars and Compiling

Pumping lemma

L is a CFL implies:

There is a p,

such that for any string s, |s| >= p

We can break up s into 5 parts uvxyz:

| v y | >= 1

| v x y | <= p

and uvixyiz L.

• How does this differ from the first pumping lemma?

In other words

• If L is a CFL, then for practically any word w in L, we should be able to find 2 substrings within w located near each other that you can simultaneously pump in order to create more words in L.

• Contrapositive: If w is a word in L, and we can always create a word not in L by simultaneously pumping any two substrings in w that are near each other, then L is not a CFL.

Gist

• A string can grow in 2 places.– When we write recursive rules, we can add symbols

to left or right of a variable.• Ex. S #A#

A ε | 1A2

Look at derivation of: #111…222#• Even a complex rule like S S1S1S2 can be

“simplified”, and still only need 2 places to grow the string.– This is because any CFG can be written in such a way that there

are no more than 2 symbols on the right side of .

Example• L = { 1n 2n } should satisfy the pumping lemma.

Let p = 2. Strings in L of length >= 2 are:

{ 12, 1122, 111222, … }. All these words have both a 1 and a 2, which we can pump.

Just make sure the 2 places to pump are within p symbols of each other. So let’s say they are in the middle.

u v x y z

11111…1 1 ε 2 22222….2

Could p = 1 work?

Example 2

• { w # wR }. Think about where we can grow.

Let p = 3. All strings of length 3 or more look like

w # wR where | w | >= 1.

As a technicality, would other values of p work?

Non-CFLs

• It’s possible for a language to be non-CFL! Typically this will be a language that must simultaneously pump in 3+ places.

• Ex. How could we design a PDA for { 1n2n3n }?

Adversary’s Revenge

• In Game #1, how did we win? Ex. { 0n 1n }.– Adversary chose p.– We chose s.– Adversary broke up s = xyz subject to constraints.– We were always able to pump and find words outside

L.

• Game #2 strategy– Adversary’s constraints looser. The middle 3 parts

have to be within p of each other. Can be anywhere in the string, not just in first p symbols of word.

{ 1n 2n 3n }

• Let p be any number.• Choose s = 1p2p3p.• Let s = uvxyz such that |vxy| <= p and |vy| >= 1.

Where can v and y be?– All 1’s– All 2’s– All 3’s– Straddling 1’s and 2’s– Straddling 2’s and 3’s

• In every case, can we find a word not in L?

More examples

• { 1i 2j 3k : i < j < k }– What should we chose as s?– How can the word be split up?

• { w w }– Let s = 0p1p0p1p. Where can v & y be?– Same section of 0’s and 1’s.– In neighboring sections– Either v or y straddles a border.

CS 461 – Oct. 7

• Applications of CFLs: Compiling• Scanning vs. parsing• Expression grammars

– Associativity– Precedence

• Programming language (handout)

Compiling

• Grammars are used to define programming language and check syntax.

• Phases of a compiler

sourcecode

scannerstream of

tokensparser parse

tree

Scanning

• Scanner needs to know what to expect when eating your program.– identifiers– numbers– strings– comments

• Specifications for tokens can be expressed by regular expression (or regular grammar).

• While scanning, we can be in different states, such as inside a comment.

Parser

• Purpose is to understand structure of program.• All programming structures can be expressed as

CFG.• Simple example for + and –

expr expr + digit | expr – digit | digit

digit 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

How would we derive the string 9 – 5 + 2 ?

9 – 5 + 2

expr expr + digit | expr – digit | digitdigit 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

expr

expr + digit expr - digit 2 digit 5

9

Leftmost derivation: expr expr + digit expr – digit + digit digit – digit + digit 9 – digit + digit 9 – 5 + digit 9 – 5 + 2

“parse tree”

Left & right recursion

• What is the difference between these 2 grammars? Which one is better?

expr expr + digit | expr – digit | digitdigit 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

expr digit + expr | digit – expr | digitdigit 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

• Let’s try 9 – 5 + 2 on both of these. The grammar must convey the order of operations!

• Operators may be left associative or right associative.

+ - * /

Question:• How do we write grammar for all 4 operators?

Can we do it this way…expr expr + digit | expr – digit |

expr * digit | expr / digit |digit

digit 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

• BTW, we can ignore “digits” and from now on just replace them with “num”, and understand it’s any single number.

Precedence

• (* /) bind stronger than (+ -)• (+ -) separate better than (* /)• Need to break up expression into terms

– Ex. 9 – 8 * 2 + 4 / 5– We want to say that an expression consists of “terms”

separated by + and –– And each term consists of numbers separated by *

and /– But which should we define first, expr or term?

Moral

• If a grammar is defining something hierarchical, like an expression, define large groupings first.

• Lower precedence operators appear first in grammar. (They separate better)– Ex. * appears lower in parse tree than + because it

gets evaluated first.

• In a real programming language, there can be more than 10 levels of precedence. C has ~15!

C language

• Handout

– How does the grammar begin?– Where are the mathematical expressions?– Do you agree with the precedence?– Do you see associativity?– What else is defined in grammar?– Where are the terminals?

CS 461 – Oct. 10

• Review PL grammar as needed

• How to tell if a word is in a CFL?– Convert to PDA and run it. – CYK algorithm– Modern parsing techniques

Accepting input

• How can we tell if a given source file (input stream of tokens) is a valid program?Language defined by CFG, so …– Can see if there is some derivation from grammar?– Can convert CFG to PDA?

• Exponential performance not acceptable. (e.g. doubling every time we add token)

• Two improvements:– CYK algorithm, runs in O(n3)– Bottom-up parsing, generally linear, but restrictions on

grammar.

CYK algorithm

• In 1965-67, discovered independently by Cocke, Younger, Kasami.

• Given any CFG and any string, can tell if grammar generates string.

• The grammar needs to be in CNF first.– This ensures that the rules are simple. Rules are of

the form X a or X YZ

• Consider all substrings of len 1 first. See if these are in language. Next try all len 2, len 3, …. up to length n.

continued

• Maintain results in an NxN table. Top right portion not used.– Example on right is for

testing word of length 3.

• Start at bottom; work your way up.

• For length 1, just look for “unit rules” in grammar, e.g. Xa.

1..3

X X1..2 2..3

X1..1 2..2 3..3

continued

• For general case i..j– Think of all possible

ways this string can be broken into 2 pieces.

– Ex. 1..3 = 1..2 + 3..3or 1..1 +

2..3– We want to know if

both pieces L. This handles rules of form A BC.

• Let’s try example from 3+7+. (in CNF)

1..3

X X1..2 2..3

X1..1 2..2 3..3

337 3+7+ ?

S AB

A 3 | AC

B 7 | BD

C 3

D 7

For each len 1 string, which variables generate it?

1..1 is 3. Rules A and C.

2..2 is 3. Rules A and C.

3..3 is 7. Rules B and D.

1..3

X X1..2 2..3

X1..1

A, C

2..2

A, C

3..3

B, D

337 3+7+ ?

S AB

A 3 | AC

B 7 | BD

C 3

D 7

Length 2:

1..2 = 1..1 + 2..2 =

(A or C)(A or C) = rule A

2..3 = 2..2 + 3..3 =

(A or C)(B or D) = rule S

1..3

X X1..2

A

2..3

S X1..1

A, C

2..2

A, C

3..3

B, D

337 3+7+ ?

S AB

A 3 | AC

B 7 | BD

C 3

D 7

Length 3: 2 cases for 1..3:

1..2 + 3..3: (A)(B or D) = S

1..1 + 2..3: (A or C)(S) no!

We only need one case to work.

1..3

S X X1..2

A

2..3

S X1..1

A, C

2..2

A, C

3..3

B, D

CYK example #2

Let’s test the word baabS AB | BCA BA | aB CC | bC AB | a

Length 1:‘a’ generated by A, C‘b’ generated by B

1..4X X X

1..3 2..4X X

1..2 2..3 3..4X

1..1

B

2..2

A, C

3..3

A, C

4..4

B

baab

S AB | BC

A BA | a

B CC | b

C AB | a

Length 2:

1..2 = 1..1 + 2..2 = (B)(A, C) = S,A

2..3 = 2..2 + 3..3 = (A,C)(A,C) = B

3..4 = 3..3 + 3..4 = (A,C)(B) = S,C

1..4X X X

1..3 2..4X X

1..2

S, A

2..3

B

3..4

S, C X1..1

B

2..2

A, C

3..3

A, C

4..4

B

baab

S AB | BC

A BA | a

B CC | b

C AB | a

Length 3: [ each has 2 chances! ]

1..3 = 1..2 + 3..3 = (S,A)(A,C) = Ø

1..3 = 1..1 + 2..3 = (B)(B) = Ø

2..4 = 2..3 + 4..4 = (B)(B) = Ø

2..4 = 2..2 + 3..4 = (A,C)(S,C) = B

1..4X X X

1..3

Ø

2..4

B X X1..2

S, A

2..3

B

3..4

S, C X1..1

B

2..2

A, C

3..3

A, C

4..4

B

Finally…

S AB | BCA BA | aB CC | bC AB | aLength 4 [has 3 chances!]1..4 = 1..3 + 4..4 = (Ø)(B) = Ø1..4 = 1..2 + 3..4 = (S,A)(S,C) = Ø1..4 = 1..1 + 2..4 = (B)(B) = Ø

Ø means we lose!baab L.

However, in general don’t give up if you encounter Ø in the middle of the process.

1..4

Ø X X X1..3

Ø

2..4

B X X1..2

S, A

2..3

B

3..4

S, C X1..1

B

2..2

A, C

3..3

A, C

4..4

B

CS 461 – Oct. 12

Parsing• Running a parse machine

– “Goto” (or shift) actions– Reduce actions: backtrack to earlier state– Maintain stack of visited states

• Creating a parse machine– Find the states: sets of items– Find transitions between states, including reduce.– If many states, write table instead of drawing

Parsing

• CYK algorithm still too slow• Better technique: bottom-up parsing• Basic idea

S AB

A aaa

B bb

At any point in time, think about where we could be while parsing the string “aaabb”.

When we arrive at aaabb. We can reduce the “aaa” to A.

When we arrive at Abb, we can reduce the “bb” to B.

Knowing that we’ve just read AB, we can reduce this to S.

• See handouts for details.

Sets of items

• We’re creating states.• We start with a grammar.

First step is to augment it with the rule S’ S.

• The first state I0 will contain S’ S

• Important rule: Any time you write before a variable, you must “expand” that variable. So, we add items from the rules of S to I0.

Example: { 0n 1n+1 }

S 1 | 0S1

We add new start rule

S’ S

State 0 has these 3 items:

I0: S’ S

S 1S 0S1

Expand S

continued

• Next, determine transitions out of state 0.

δ(0, S) = 1

δ(0, 1) = 2

δ(0, 0) = 3

I’ve written destinations along the right side.

• Now we’re ready for state 1. Move cursor to right to become S’ S

State 0 has these 3 items:

I0: S’ S 1

S 1 2

S 0S1 3

I1: S’ S

continued

• Any time an item ends with , this represents a reduce, not a goto.

• Now, we’re ready for state 2. The item S 1 moves its cursor to the right: S 1 This also become a reduce.

I0: S’ S 1

S 1 2

S 0S1 3

I1: S’ S r

I2: S 1 r

continued

• Next is state 3. From

S 0S1, move cursor.

Notice that now the is in front of a variable, so we need to expand.

• Once we’ve written the items, fill in the transitions. Create new state only if needed.

δ(3, S) = 4 (a new state)

δ(3, 1) = 2 (as before)

δ(3, 0) = 3 (as before)

I0: S’ S 1

S 1 2

S 0S1 3

I1: S’ S r

I2: S 1 r

I3: S 0 S1 4

S 1 2

S 0S1 3

continued

• Next is state 4. From item S 0 S1, move cursor.

• Determine transition.

δ(4, 1) = 5

Notice we need new state since we’ve never seen “0 S 1” before.

I0: S’ S 1

S 1 2

S 0S1 3

I1: S’ S r

I2: S 1 r

I3: S 0 S1 4

S 1 2

S 0S1 3

I4: S 0S 1 5

Last state!

• Our last state is #5.

Since the cursor is at the end of the item, our transition is a reduce.

• Now, we are done finding states and transitions!

• One question remains, concerning the reduce transitions: On what input should we reduce?

I0: S’ S 1

S 1 2

S 0S1 3

I1: S’ S r

I2: S 1 r

I3: S 0 S1 4

S 1 2

S 0S1 3

I4: S 0S 1 5

I5: S 0S1 r

CS 461 – Oct. 17

• Creating parse machine – Convert grammar into sets of items– Determine goto and reduce actions

• On what input do we reduce?– Whatever “follows” the nonterminal we’re reducing to.

• Declaration grammar

0n 1n+1

• There are 5 states.

When the cursor is at the end of the item, our transition is a reduce.

• Now, we are done finding states and transitions!

• One question remains, concerning the reduce transitions: On what input should we reduce?

I0: S’ S 1

S 1 2

S 0S1 3

I1: S’ S r

I2: S 1 r

I3: S 0 S1 4

S 1 2

S 0S1 3

I4: S 0S 1 5

I5: S 0S1 r

When to reduce

If you are at the end of an item such as S 1 , there is no symbol after the telling us what input to wait for. – The next symbol should be whatever “follows” the

variable we are reducing. In this case, what follows S. We need to look at the original grammar to find out.

– For example, if you were reducing A, and you saw a rule S A1B, you would say that 1 follows A.

– Since S is start symbol, $ (end of input) follows S.

• For more info, see parser worksheet.– New skill: for each grammar variable, what follows?

First( )

To calculate first(A), look at A’s rules.

• If you see A c…, add c to first(A)• If you see A B…, add first(B) to first(A).

Note: don’t put $ in first( ).

Follow( )

What should be included in follow(A) ?

• If A is start symbol, add $.• If you see Q …Ac…, add c.• If you see Q …AB…, add first(B).• If you see Q …A, add follow(Q).

Note: don’t put ε in follow( ).

CS 461 – Oct. 19

• Examples– Calculator grammar– Review first( ) and follow( )– Declaration grammar

• How to handle ε in grammar– Need to change how we find first( ) and follow( ).

First( )

To calculate first(A), look at A’s rules.

• If you see A c…, add c to first(A)• If you see A B…, add first(B) to first(A).

– If B can yield ε, continue to next symbol in rule until you reach a symbol that can represent a terminal.

• If A can yield ε, add ε to first(A).

Note: don’t put $ in first( ).

Follow( )

What should be included in follow(A) ?

• If A is start symbol, add $.• If you see Q …Ac…, add c.• If you see Q …AB…, add first(B).• If you see Q …A, add follow(Q).

• If you see Q …ABC, and B yields ε, add first (C).• If you see Q …AB, and B yields ε, add follow(Q).

Note: don’t put ε in follow( ).

Example

Try this grammar:

S AB

A ε | 1A2

B ε | 3B

First(B) = ε, 3

First(A) = ε, 1

First(S) = ε, 1, 3

(note in this case A ε)

Follow(S) = $

since S is start symbol

Follow(A) = 2, 3, $

we need first(B)

since B ε, we need $

Follow(B) = $

Try this one

Rules First Follow

S’ S

S ε | SABC

A 2 | 1A

B ε | 3B | 4B

C ε | 56C

Let’s try the language ((1*2(3+4)*(56)*)*

CS 461 – Oct. 21

Begin chapter 3

• We need a better (more encompassing) model of computation.

• Ex. { 1n 2n 3n } couldn’t be accepted by PDA. – How could any machine accept this language?– Somehow need to check 1s, 2s and 3s at same time.– We should also be able to handle 1n 2n 3n 4n , etc.

Turing Machine

• 3rd model of computation– Alan Turing, 1936– Use “tape” instead of stack– Can go left or right over the input!

• Every TM has:– Set of states, including 1 start and 1 accept state– Input alphabet– Tape alphabet, including special “blank” symbol– Transition function

δ (state, input) = (state, output, direction)

The tape and δ

• Tape is infinite in both directions– Blanks straddle the input on the tape.

• Begin reading at leftmost input symbol.• As soon as you enter accept state, halt.

– You don’t have to read all the input.

δ (state, input) = (state, output, direction)– You may change state.– You may change the symbol that’s on the tape.– The “direction” is either L or R.

States

• A TM has at least 3 states– One accept state– One reject state– Ordinary states that we describe in our transition

table, beginning with a start state.

• We don’t need transitions for the accept or reject states.

• Sometimes a transition “doesn’t matter” because we can’t be in that situation.– Ex. If the leftmost symbol isn’t blank, then the

rightmost symbol can’t be blank. You can just go to reject state.

Example

• A TM to accept the language 0(0+1)*1.– Make sure first symbol is 0. Else reject.– Keep moving right until you see blank.– After blank, go left. If 1, accept. Else reject.

State Input 0 Input 1 Input _

s1 s2, 0, R reject reject

s2 s2, 0, R s2, 1, R s3, _, L

s3 reject Doesn’t matter

Let’s trace 011


s1 s2, 0, R reject reject

s2 s2, 0, R s2, 1, R s3, _, L

s3 reject Doesn’t matter

Tape Transition Effect

[s1]011 s2,0,R 0[s2]11

0[s2]11 s2,1,R 01[s2]1

01[s2]1 s2,1,R 011[s2]

011[s2]_ s3,_,L 01[s3]1_

01[s3]1

Input output

• TMs can easily modify the input on the tape. Thus we can have useful output!

• Example: let’s flip the bits of our binary input, and automatically accept.– Note: the reject state is unreachable.


s1 s1, 1, R s1, 0, R

Infinite loop

• It’s possible for a TM to have an infinite loop. We say that the TM “loops.” In this case, it doesn’t accept or reject its input.

• Simple example: From start state, go left on any input!


s1 s1, 0, L s1, 1, L s1, _, L

Mystery #1

• What does this TM do? Assume s1 is the start state, and unspecified transitions are crash.


s1 s2, 1, R

s2 s1, 0, R

Hint: notice we never move left. This means it doesn’t matter what we write on tape.

Mystery #2

• Can you figure out what this machine does?– See something wrong?


s1 s1, 0, R s2, 1, R s1, _, R

s2 s2, 1, R s2, _, R

CS 461 – Oct. 24

• Turing machines √– Can go left or right– Can write on tape– Can halt early or go on forever!

• More examples of TMs.– Going in both directions along the tape.– Practice!– Optional features– Doing arithmetic on a TM.

Mystery #2

• Can you figure out what this machine does?– See something wrong?


s1 s1, 0, R s2, 1, R s1, _, R

s2 s2, 1, R s2, _, R

Review:

Back & forth

• Often it’s useful for a TM to scan its input in both directions.

• Ex: Cut and paste, as in 001 ___#001.– First, put a # at the end of the input.– Copy first input symbol onto the end, and continue for

all other symbols.

– How do we know we’re “at the end”?– How do we know a given symbol has already been

duplicated?– How will we know we are finished?

algorithm

1. Go right until you see _. Upon blank, change it to # and head left. Go to 2.

2. Keep going left until you see _. Upon blank, go right and go to state 3.

3. If “0”, change it to _, head right and go to state 4.

If “1”, change it to _, head right and go to state 6.

If “#”, accept.

4. Go right until you see _. At the _, change it to 0 and head left in state 5.

5. Keep left until _. At _, go right. Go to state 3.

6. Just like state 4, but change _ to 1.

δ and trace[s1]01 [s5]_1#0

0[s1]1 _[s3]1#0

01[s1]_ __[s6]#0

0[s2]1# __#[s6]0

[s2]01# __#0[s6]_

[s2]_01# __#[s5]01

[s3]01# __[s5]#01

_[s4]1# _[s5]_#01

_1[s4]# __[s3]#01

_1#[s4]_ _1[s5]#0

_[s5]1#0

State 0 1 # _

1 1,0,R 1,1,R 2,#,L

2 2,0,L 2,1,L 2,#,L 3,_,R

3 4,_,R 6,_,R

4 4,0,R 4,1,R 4,#,R 5,0,L

5 5,0,L 5,1,L 5,#,L 3,_,R

6 6,0,R 6,1,R 6,#,R 5,1,L

Q: How can you tell if a transition basically says “do nothing” and move L/R?

Some practice

• See handout for some more examples.

• Let’s design TMs that accept these languages.

0*1*

Palindrome

Palindrome

1. If “0”/”1”, change to _ and go to 2/4. If _, accept.

2. Continue right until _. Then go left and go to 3.

3. If “0”, change to _, go left. Go to 6. If “1” reject

4. (states 4,5 are analogous to 2,3)

5. If “1”, change to _, go left. Go to 6. If “0” reject.

6. Continue left until _. Turn right. Start over at 1.

Closer look at δ

[s1]101

_[s4]01

_0[s4]1

_01[s4]_

_0[s5]1

_[s6]0_

[s6]_0_

_[s1]0_

__[s2]_

_[s3]__

At this point we realize the missing δ should be also.

State 0 1 _

1 2, _, R 4, _, R

2 2, 0, R 2, 1, R 3, _, L

3 6, _, L Reject

4 4, 0, R 4, 1, R 5, _, L

5 Reject 6, _, L

6 6, 0, L 6, 1, L 1, _, R

CS 461 – Oct. 26

• TM practice– 0*1*

– Twice as many 0’s as 1’s

– Palindrome…

– Adding register(s) to simplify programming

• Arithmetic on a TM

• TM variants (handout)– Adding more features doesn’t add power.

If a TM can’t do it, nothing can.

Palindrome

1. If “0”/”1”, change to _ and go to 2/4. If _, accept.


3. If “0”, change to _, go left. Go to 6. If “1” reject

4. (states 4,5 are analogous to 2,3)

5. If “1”, change to _, go left. Go to 6. If “0” reject.


Closer look at δ

[s1]101

_[s4]01

_0[s4]1

_01[s4]_

_0[s5]1

_[s6]0_

[s6]_0_

_[s1]0_

__[s2]_

_[s3]__

transitions have been added.

State 0 1 _

1 2, _, R 4, _, R

2 2, 0, R 2, 1, R 3, _, L

3 6, _, L Reject

4 4, 0, R 4, 1, R 5, _, L

5 Reject 6, _, L

6 6, 0, L 6, 1, L 1, _, R

Registers

• In palindrome example, we needed 2 extra states because leftmost input symbol could be 0 or 1.

• What if alphabet were { a, b, c, … z } ? Do we literally want 2*25 more states?

• In place of those extra states, store a finite amount of information in TM’s “control unit”, just like registers in CPU. With a register, our state is just an ordered pair like (2, ‘t’).

• More and more, a TM is looking like a computer!

New palindrome

1. If _, accept. Otherwise copy first symbol to register. Change tape to _ and go to 2.


3. If input matches register, change to _, go left. Go to 6. If no match, reject. (if _, accept)


Arithmetic

• Can do binary addition on a TM. – Method #1: Unary programming

Input is of the form 1101111 for 2 + 4.

– Method #2: With binary notation: Modify our definition of TM so it can have 3 parallel tracks on the tape. The third track can hold the answer.

• Can do other operations analogously.

Unary +

1. Make sure input is of the form 1*01*. If not, reject. If so, put = at end and go back to beginning of word.

10. Copy all the 1s on left side of 0 to the right end of word.

20. Copy all the 1s between the 0 and = to the right end of word.

• This is just an outline, and I didn’t attempt to count exact number of states needed.

• If we didn’t care about changing the input, we could just get rid of the 0 and squeeze the 1s together!

TM Variants

• Based on additional features.– Can save a lot of effort.– Multi-tape probably most useful.

• We can show that a new feature can be emulated by the ordinary TM model.

Nothing can beat a TM. The extra bells and whistles just make programming easier.

Some variants

• Allowing a TM to “stay put” instead of having to move left or right.

Multi-tape (not to be confused with multi-track tape)– Makes use of stay-put– Scanning the “virtual heads”

Non-deterministic– Makes use of multi-tape

Non-determinism

• The non-deterministic computations can be arranged in a tree. – If we want to deterministically search this tree, DFS or

BFS?– P. 151: Tape 3 will contain a list of transitions to

experiment on. Copy tape 1 to tape 2, and use tape 2 for scratch work. If reject, increment tape 3 to next possible set of transitions.

• Ex. Accept the union of 2+ languages

• Ex. For some language L, design a TM that will accept substrings (of length 3) of these words.

Computer

• A TM with multiple tapes can simulate a computer.– One tape contains data and instruction memory.

Unfortunately, need to label each with address.– One tape can store the address of current instruction.– Another tape can store address of data operand.– Finally, a tape for scratch work.

CS 461 – Oct. 28

• TM applications– Recognize a language √– Arithmetic √– Enumerate a set– Encode a data structure or problem to solve

• Two kinds of TMs– Decider: will always halt for any input

This one is useful for encoding an algorithm. – Non-decider: for some input, may loop forever

Enumerators

• List all the words in the language– In order of length– Within each length, canonical order such as

alphabetical

• Technique– Ignore input– Use multiple parallel tapes. One for output, the rest

for scratch work.

Example

• Let’s enumerate { 0n 1n }.

– One tape will hold value of n.– One tape will hold for i = 1 to n.– One tape will hold the words we write.– (need to separate words by special symbol such as #)

– The machine will go on forever.

• Similarly, we could enumerate all positive integers, all prime numbers, etc. (They’re not algorithms.)

TM input encoding

• Often we use TMs to solve problems or answer mathematical questions.

• For input, we need a specific encoding or rep’n.– Ex. Given an equation, can we solve for x?– Ex. Given a graph, can we tell if it’s connected? – If “yes,” there is an algorithm… so we can design a

TM (or program). We encode the input to feed to TM because it can’t see the graph like we can.

TMString inputYes

No

Encode these

If an object doesn’t have a text or numerical representation, need to create one.

How would you encode…?• Graph• Binary tree• finite automaton• context-free grammar• Polynomial• Function (or one-to-many relation)

Algorithm

• A sequence of steps that always halts ( or ) no matter what the input is.

• Decider: TM will always enter or state, even if it takes a long time.

• If we can create a decider (i.e. algorithm), we say the problem is “decidable”.

• 3 ways we specify a TM– List all transitions formally (as in a table)– Write pseudocode for each state– High-level description – making it clear algorithm will

always halt, but doesn’t spell out states.

Examples

• Let’s look at informal TM descriptions.– Must clearly define the input, the overall steps, and

how to accept/reject.

• Graph connectedness– See how book writes it on p. 158

• Solving an equation for “x”– What is wrong with TM description on p. 156?

• Determine if a number is prime• Does a given FA accept its input

CS 461 – Oct. 31

• Chapter 4 – Decidable questions

“Is it decidable to determine if …”We want a TM that will always halt.

• A closer look at ∞

Question about FAs

• Given any DFA and any input string, is there an algorithm to determine if the DFA will accept the string?

• This is called the “acceptance problem” for DFAs: ADFA.

• Approach (can visualize as diagram)– Encode the DFA and input string.– Feed into TM, and have TM run the DFA on input.– If DFA accepts, say “yes.” If DFA rejects, say “no.”– Note that this TM will always halt!

Other FA questions

• Once we can solve one problem about FAs, try to use similar technique…

• ANFA

• For 2 FA’s, L(M1) = L(M2) ?

• L(M) = Ø ?• Is L(M) infinite?• Does L(M) contain a word of length 5?• Does L(M) contain all words of length 5?• Does L(M) contain all words of the form *01?• Are all words in L(M) of the form *01?

Questions about CFG

• ACFG

– How would we phrase this question?– Is it decidable? (i.e. can we do it in finite time?)

• Also decidable– L(G) = Ø ?– L(G) contains some word or finite subset.– L(G) is itself infinite.

• The “=“ question is harder than for FA’s– For CFLs, intersection isn’t closed.– For complicated reason, not decidable anyway.– Questions about infinite intersections undecidable.

Question about TM

• ATM is an important question– Why can’t we run a TM just like we did for FAs?– Does this prove that it’s impossible?

• To resolve the question of ATM

– We’ll prove undecidable by contradiction.– First, need to understand something about ∞– What do the words “finite” and “infinite” mean?

CS 461 – Nov. 2

• Sets– finite vs. infinite – Infinite sets

• Countable• Uncountable

• Prepare for ATM

– Proving undecidable is similar to proving a set uncountable.

– Please be sure you understand reasoning.

Finite/infinite

• Finite set – means that its elements can be numbered. Formally: its elements can be put into a 1-1 correspondence with the sequence 1, 2, 3, … n, where n is some positive integer.

• An infinite set is one that’s not finite.

• However, there are 2 kinds of infinity!

Countable

• Countable set: An infinite set whose elements can be put into a 1-1 correspondence with the positive integers.– Examples we can show are countable:

Even numbers, all integers, rational numbers, ordered pairs.

• Uncountable set = ∞ set that’s not countable. – Examples we can show are uncountable:

Real numbers, functions of integers, infinite-length strings

Examples

• The set of even numbers is countable.

• The set of integers is countable.

2 4 6 8 10 12 …

1 2 3 4 5 6 …

0 1 -1 2 -2 3 -3 4 -4 …

1 2 3 4 5 6 7 8 9

Ordered pairs

j = 1 j = 2 j = 3 j = 4

i = 1 1 2 4 7 …

i = 2 3 5 8 …

i = 3 6 9 …

i = 4 10 …

…

The number assigned to (i , j) is (i + j – 1)*(i + j – 2)/2 + i

Real numbers

• Suppose real numbers were countable. The numbering might go something like this:

• The problem is that we can create a value that has no place in the correspondence!

Real # Value

1 2.71828…

2 3.14159…

3 0.55555…

4 -1.23456…

5 5.676767...

… …

X ? . 85667…

Infinite bit strings

# Value

1 00000…

2 100000…

3 011010…

4 0010001…

5 11111001011…

…

? 11010…

Universal TM

Let’s design “U” – the Universal TM:• Input consists of <M> and w:

– <M> is the encoding of some TM – w is any (binary) string.

• Assume: U is a decider (i.e. ATM is decidable.)

<M>,w

U

no

yesw

Myes

no*

ATM solution

• Start with U, the Universal Turing Machine

• Suppose U decides ATM. Let’s build new TM D.

– D takes in a Turing machine, and returns opposite of U’s answer.

<M>

<M>,<M>

D

U

yes

no

no

yes

If M accepts its own string rep’n, D rejects <M>.

If M doesn’t accept <M>, D accepts <M>.

What does D do with <D> as input?

For example

<M1> <M2> <M3> <M4> … <D>

M1 Yes No No Yes

M2 No No Yes Yes

M3 Yes No Yes No

M4 Yes No No No

…

D No Yes No Yes Uh-oh

Contradiction The TM D can’t exist So U is not a decider.

In other words

• Let U = universal TM. – Its input is a TM description <M> and a word <w>. – Determines if M accepts w.– Assume U halts for all inputs. (is a decider)

• Create 2nd TM called D. – Its input is a TM description <M>. – Gives <M> to U as the TM to run as well as the input.– D returns the opposite of what U returns.

• What happens when the input to D is <D>?– According to U, if D accepts <D>, U accepts, so D must reject!– According to U, if D rejects <D>, U rejects, so D must accept!– Both cases give a contradiction.

– Thus, U is not a decider. ATM is undecidable.

CS 461 – Nov. 7

• Decidability concepts– Countable = can number the elements – Uncountable = numbering scheme impossible – ATM undecidable

– Language classes

• Next– { languages } uncountable, but { TMs } countable

There are more languages than TMs! …– Be on the lookout for ∞ rep’n.

Universal TM

Let’s design “U” – the Universal TM:• Input consists of <M> and w:

– <M> is the encoding of some TM – w is any (binary) string.

• Assume: U is a decider (i.e. ATM is decidable.)

<M>,w

U

no

yesw

Myes

no*

ATM solution

• Start with U, the Universal Turing Machine

• Suppose U decides ATM. Let’s build new TM D.

– D takes in a Turing machine, and returns opposite of U.

<M>

<M>,<M>

D

U

yes

no

no

yes

If M accepts its own string rep’n, D rejects <M>.

If M doesn’t accept <M>, D accepts <M>.

What does D do with <D> as input?

For example

<M1> <M2> <M3> <M4> … <D>

M1 Yes No No Yes

M2 No No Yes Yes

M3 Yes No Yes No

M4 Yes No No No

…

D No Yes No Yes Uh-oh

Contradiction The TM D can’t exist So U is not a decider.

In other words

• Let U = universal TM. – Its input is a TM description <M> and a word <w>. – Determines if M accepts w.– Assume U halts for all inputs. (is a decider)

• Create 2nd TM called D. – Its input is a TM description <M>. – Gives <M> to U as the TM to run as well as the input.– D returns the opposite of what U returns.

• What happens when the input to D is <D>?– According to U, if D accepts <D>, U accepts, so D must reject!– According to U, if D rejects <D>, U rejects, so D must accept!– Both cases give a contradiction.

– Thus, U is not a decider. ATM is undecidable.

Language classes

Working from the inside out:• Finite set• Regular• CFL (deterministic)• CFL (non-deterministic)• Decidable• Turing-recognizable• Outer space!

– Yes – it’s possible for a language not to be recognized by any TM whatsoever

• Note: all languages are countable (or finite).

Language beyond TM

• The set of all TM’s is countable.– Finite representation

• The set of all languages is uncountable.– Infinite representation

• Not enough TM’s to go around There must be a language unrecognized by any TM.

• Let’s find one!

Other properties

• 2 kinds of TMs 2 kinds of languages.– Turing-recognizable (a.k.a. recursively enumerable)

• Example: ATM

– Decidable (a.k.a. recursive)• Example: 0*

• If L is decidable, then L’ is decidable.• If L and L’ are both Turing-recognizable,

then L is decidable. (since either L or L’ must accept)

• Therefore, the complement of ATM is not even Turing recognizable.

CS 461 – Nov. 9,11

• Chomsky hierarchy of language classes– Review– Let’s find a language outside the TM world!– Hints: languages and TM are countable, but the set

of all languages is uncountable

• Undecidability– We have seen that ATM is undecidable.– Let’s look at some more undecidable problems.

TM properties

• 2 kinds of TMs 2 kinds of languages.– Turing-recognizable (a.k.a. recursively enumerable)

• Example: ATM

– Decidable (a.k.a. recursive)• Example: 0*

• If L is decidable, then L’ is decidable.• If L and L’ are both Turing-recognizable,

then L is decidable. (since either L or L’ must accept)

• Therefore, the complement of ATM is not even Turing recognizable.

Decidable?

for n = 3 .. maxint:

for each combo of (x, y, z) starting at 1:

if (xn + yn == zn)

print (“hello, world”);

• Are we able to tell if this program will print “hello, world” or not?– Only since 1993 . For 300 years we didn’t know.– Solving this problem in general (for any program)

must be hard… but what if it could be done?

“hello, world” tester

• Suppose H can decide the question.• Then we can create similar algorithm H2.

– Returns “hello, world” instead of “no”.– Let’s run H2 on itself!

H

program & inputPrint “yes”

Print “no”

H2

H2Print “yes”

Print “hello, world”

Regular?

• REG = “Is there an algorithm to determine if a TM recognizes a regular language?”

• Proof by contradiction, p. 191• Assume REG is decidable.

– Then, TM “R” that can tell if an input <TM> recognizes a regular language.

– We’ll use R to solve the ATM problem by creating a decider S.

– In other words:

Machine R solves REG -- which implies existence of ….

Machine S which solves ATM.

– Do we understand the I/O of R and S?

Regular, continued

• Here is how to build machine S, which will “solve” ATM.

• Input to S is <T, w>.

• Create a new TM “M2”– M2 sees if its input, x, is of the form 0n1n. If so, M2 accepts x.– If x is not of the form 0n1n,

• M2 runs T on w.• If T accepts w, M2 accepts x.

– Thus, L(M2) is either Σ*, if T accepts w, or 0n1n.– The only way for M2 to accept Σ* is for T to accept w.

• Feed M2 into R.– If R says that M2 accepts a regular language, we can conclude T

accepts w, so S can say “yes”, i.e. accept. Otherwise S rejects.– But S can’t exist. So we have a contradiction. REG undecidable.

Rice’s Theorem

• Any question Q about a TM language is undecidable.• Proof by contradiction:

• Assume Q is decidable. Then TM “QD” that – always halts, and – answers Q for any input TM it receives.

• Using QD, we can build a machine AD to decide ATM.

• (But we know that AD can’t exist, therefore QD can’t exist. That is the contradiction.)

Building TM “AD”• AD will do the following:

– Its input is <T, w>.– Build a TM “Q1” that satisfies Q.– Build a TM “M2” whose input is x.

• M2 runs T on w. If T loops/rejects, M2 will loop/reject. • If T accepts w, next run Q1 on x. M2 accepts if Q1 accepts.• Note that L(M2) = L(Q1) if T accepts w. L(M2) = if T does

not accept w.• In other words, if T accepts, then M2 satisfies Q.

– Feed M2 into QD. If QD accepts, AD accepts. Otherwise AD rejects.

• Thus: QD is a decider AD is a decider.

Glossary to proof

• Q = some question or property about a TM language• QD = a TM that can decide Q• Q1 = a TM for which Q is true, built inside AD

• ATM = the acceptance question for TM’s, already known to be undecidable

• AD = a hypothetical decider for ATM

• T = the TM input to AD• w = the corresponding word also input to AD• M2 = a second TM built inside AD

• Rice’s theorem is a little complex because 5 TM’s!

Examples

• Let Q = “Is the language infinite?” or ….• “Does the language include 001?”• “Is the language finite?”• “Is the language nonregular?”

• Can also prove by contradiction:– Is it decidable to determine if one TM language is the

subset of another?

CS 461 – Nov. 14

• Section 5.2 – Undecidable problems starting from ATM:

Matching sets of strings

Ambiguity of CFGs

String matching

• Formal name is the Post Correspondence Problem, (by mathematician Emil Post)

• Given a set of dominoes– Each contains a string on the top and bottom– Use the dominoes so that the strings on the top and

bottom match.– You may use each domino as many times as you like.

But there must be one domino. – The solution is the sequence of dominoes (e.g. 1,2,3)

11111

100001

11111

Another PCP

• Can you find a solution to this one?

1111

1011110

100

10101

01111

101011

Or this one?

Undecidability

• To show the problem is undecidable in general, we draw a parallel with ATM, which we know is undecidable.

• Take any TM with an input word. If we follow the TM’s steps, there is a parallel sequence of dominoes! Accept there is a match, and reject there is no match.

• For each possible TM action, we specify what the dominoes look like.

Example

• This is best seen via an example. Let’s say we have a TM that accepts the language 1*00*1(0+1)*, and we feed it the word 01.

State 0 1 _

s1 s2, 0, R s1, 1, R rej, _, R

s2 s2, 0, R acc, 1, r rej, _, R

[s1] 0 1

0 [s2] 1

0 1 [acc]

Create dominoes!

• There are 7 kinds of dominoes we need.

– The first domino represents the initial configuration of the TM, which is

[s1] 0 1

0 [s2] 1

0 1 [acc]

##[s1] 0 1 #

Dominoes (2,3)

• Second type of domino represents moves to the right. In our case we see

[s1] 0 1

0 [s2] 1

0 1 [acc]

[s1] 00 [s2]

[s2] 11 [acc]

The third type of domino would represent moves going left, but we don’t have any of these.

In general, right moves are qa/br and left moves are cqa/rcb.

Dominoes (4, 5)

• Allow ourselves to do nothing to a tape symbol away from the read-write head.

[s1] 0 1

0 [s2] 1

0 1 [acc]

00

11

__

##

## _

#_ #

Also, allow use to put a delimiter (#) between steps, and put blanks on either side of the input as needed.

Dominoes (6, 7)

• Allow ourselves to eat the tape once we accept! There will be 6 dominoes like this:

[s1] 0 1

0 [s2] 1

0 1 [acc]

0 [acc][acc]

[acc] 0[acc]

[acc] # ##

And finally, we need a special case domino to “eat” the accept state notation on the tape.

CS 461 – Nov. 16

• Decidable vs. undecidable √– The goal was to create a TM that always halts.– Last undecidable problem: ambiguity of CFG

• Tractable vs. intractable– Now, not only should it halt, we want an algorithm that

won’t take too long!

– Two kinds of problems “P” and “NP”

Ambiguous grammars

• If we can’t tell if PCP has a solution, then we can’t tell if a CFG is ambiguous.

• The idea is to convert a domino game into a grammar that has to be ambiguous exactly when PCP has a solution.

• First, begin with an instance of a PCP problem, like we saw before.

• Notice that we have a top & bottom. And we’re wondering if we can get them to match! Create a grammar for the top (A) and bottom (B) and say

S A | B

continued

• Our grammar will look like

S A | B

A ______________________

B ______________________

In our case, we have:

A 1Ai1 | 10111Ai2 | 10Ai3 |

1i1 | 10111i2 | 10i3B 111Bi1 | 10Bi2 | 0Bi3 |

111i1 | 10i2 | 0i3

Derivations

• Let’s see how our solution 10111 1 1 10 is derived from our A and B grammars.

• We wind up with these strings

10111 1 1 10 i3 i1 i1 i2and

10 111 111 0 i3 i1 i1 i2

These strings are equal, so there are 2 derivations ambiguous grammar. Note that within A and B it couldn’t be ambiguous, because each time we lay a domino (and choose a grammar rule) we are writing down a distinct “i” number.

Complexity

• Algorithms are often classified according to complexity.

• Big-O notation is pretty common. General worst-case behavior.

• Example: Sorting– Radix sort, O(n)– Merge sort, O(n log n)– Insertion sort, O(n2)– Stooge sort, O(2.7n)– Monkey sort, O(n!)

Good

Bad

Non-determinism

• An algorithm should be deterministic!• Non-determinism is “cheating.”

– For sorting, non-determinism means pick the right arrangement of elements, and verify it’s in order!

• Ex. Spell checking– Deterministic, O(log n)– Non-deterministic, O(1)

• Looks like non-determinism makes things easier, but the trick is there’s no way to know what data to pick. Must try them all, so deterministic algorithm often exponential.

P and NP

• P = Problems where deterministic polynomial-time algorithm.– “can quickly decide” (in the TM sense)– The run time is O(n2) or O(n3), etc.

• NP = Problems where non-deterministic polynomial-time algorithm.– “can quickly verify”– A deterministic algorithm would require exponential

time, which isn’t too helpful.– (NP – P) consists of problems where we don’t know of

any deterministic polynomial algorithm.

CS 461 – Nov. 18

Section 7.1• Overview of complexity issues

– “Can quickly decide” vs. “Can quickly verify”

• Measuring complexity• Dividing decidable languages into complexity classes.

• Next, please finish sections 7.1 – 7.2:– Algorithm complexity depends on what kind of TM you

use– Formal definition of P, NP, NP-complete

P and NP

• P = Problems where deterministic polynomial-time algorithm.– “can quickly decide” (in the TM sense)– The run time is O(n2) or O(n3), etc.

• NP = Problems where non-deterministic polynomial-time algorithm.– “can quickly verify”– A deterministic algorithm would require exponential

time, which isn’t too helpful.– (NP – P) consists of problems where we don’t know of

any deterministic polynomial algorithm.

Revisit:

Conjecture

• P and NP are distinct.– Meaning that some NP problems are not in P.– There are some problems that seem inherently

exponential.

• Major unsolved question! – For each NP problem, try to find a deterministic

polynomial algorithm, so it can be reclassed as P.– Or, prove that such an algorithm can’t exist. We don’t

know how to do this. Therefore, it’s still possible that P = NP.

• Ex. Primality was recently shown to be in P.

Example

• Consider this problem: subset-sum. Given a set S of integers and a number n, is there a subset of S that adds up to n?– If we’re given the subset, easy to check. NP

– Nobody knows of a deterministic polynomial algorithm.

• What about the complement?– In other words, there is no subset with that sum.

– Seems even harder. Nobody knows of a non-deterministic algorithm to check. Seems like we need to check all subsets and verify none adds up to n.

– Another general unsolved problem: are complements of NP problems also NP?

Graph examples

• The “clique” problem. Given a graph, does it contain a subgraph that’s complete?– Non-deterministically, we would be “given” the clique,

then verify that it’s complete. – What is the complexity?– (Complement of Clique: not known to be in NP.)

• Hamiltonian path: Given a graph, can we visit each vertex exactly once?– Non-deterministically, we’d be given the itinerary.

Inside NP

• There are generally 2 kinds of NP problems.– Smaller category:

Problems where a deterministic polynomial algorithm is lurking out there, and we’ll eventually find it.

– Larger category:

Problems that seem hopelessly exponential. When you distill these problems, they all have the same structure. If a polynomial solution exists for one, they would all be solvable!

These problems are called NP-complete.

Examples

• Some graph problems– Finding the shortest path– Finding the cheapest network (spanning tree)– Hamiltonian and Euler cycles– Traveling salesman problem

• Why do similar sounding problems have vastly different complexity?

CS 461 – Nov. 21

Sections 7.1 – 7.2• Measuring complexity• Dividing decidable languages into complexity classes.• Algorithm complexity depends on what kind of TM you

use.• Formal definition of “P”

• Next: Concepts of NP and NP-complete

Examples

• Some graph problems– Finding the shortest path– Finding the cheapest network (spanning tree)– Hamiltonian and Euler cycles– Traveling salesman problem

• Why do similar sounding problems have vastly different complexity?

“O” review

• Order of magnitude upper bound• Rationale: 1000 n2 is fundamentally less than n3, and is in the

same family as n2.• Definition: f(n) is O(g(n)) if there exist integer constants c and

n0 such that f(n) c g(n) for all n n0.– In other words: in the long run, f(n) can be bounded by g(n) times a

constant.– e.g. 7n2 + 1 is O(n2) and also O(n3) but not O(n). O(n2) would be a

tight or more useful upper bound.– e.g. Technically, 2n is O(3n) but 3n is not O(2n).– log(n) is between 1 and n.

• Ordering of common complexities:

O(1), O(log n), O(n), O(n log n), O(n2), O(n3), O(2n), O(n!)

Measuring complexity

• Complexity can be defined:– as a function of n, the input length– It’s the number of Turing machine moves needed.– We’re interested in order analysis, not exact count.

• E.g. About how many moves would we need to recognize { 0n1n } ?– Repeatedly cross of outermost 0 and 1.– Traverse n 0’s, n 1’s twice, (n-1) 0’s twice, (n-1) 1’s twice, etc.– The total number of moves is approximately:

3n + 4((n-1)+(n-2)+(n-3)+…+1) = 3n + 2n(n-1) = 2n2 + n ~ 2n2

– 2n2 steps for input size 2n O(n2).

Complexity class

• We can classify the decidable languages.• TIME(t(n)) = set of all languages that can be decided by a

TM with running time O(t(n)).– { 0n 1n } TIME(n2).– 1*00*1(0+1)* TIME(n).– { 1n 2n 3n } TIME(n2).– CYK algorithm TIME(n3).– { ε, 0, 1101 } TIME(1).

• Technically, you can also belong to a “worse” time complexity class. L TIME(n) L TIME(n2) .

It turns out that { 0n 1n } TIME(n log n). (p. 252)

TM variants

• It turns out that the complexity depends on the variant!– Ordinary one-tape TM. (for definition of TIME(t(n))– Multi-tape TM. (useful for programming)– Non-deterministic TM.

• E.g. { 0n 1n } can be decided in O(n) on a multi-tape TM.– See page 253.

• In general, a multi-tape TM that runs in O(t(n)) can be converted to a single-tape TM that runs in O((t(n)2).– See page 254.

• In general, a non-deterministic TM that runs in O(t(n)) can be converted to an ordinary TM running in 2 O(t(n)).– See page 256.– This is why we say that NP problems are exponential.

Multi-tape { 0n 1n }

• We have seen an O(n2) algorithm on a single-tape TM.• And a better algorithm that runs in O(n log n), also with

single tape.• With multiple tapes, we can do it in O(n).

– Scan input to make sure of the form 0*1*.– Return to beginning.– Read 0’s and copy them to 2nd tape.– When you reach the first 1, start the 2nd tape’s head at the first 0.

– Now we match 0’s and 1’s simultaneously.

• So, the problem is O(n), although technically it’s still in TIME(n log n). Not a big deal in the long run.

Single vs. multi tape

O(t(n)) multi-tape O(t(n)2) single tape. Proof:• We simulate the multi-tape operation on single tape TM.

– Each of the k tapes is performing a computation that runs in t(n) time. In other words, we take up to t(n) steps.

• Scan the tape, taking note of where virtual heads are.• How long is the tape contents?

– Each virtual section of the tape could be t(n) symbols long, because each step could write a symbol on the tape.

• Final calculation:– Scanning the entire tape takes O(t(n)) time.– And we have to scan the tape t(n) times!– Multiply together to obtain O(t(n)2).

DTM vs. NTM

O(t(n)) NTM 2 O(t(n)) DTM. Proof:• The NTM has a tree of possible computational futures.• For input length n, the depth of tree is up to t(n).• Each node of the tree can have up to c children.

– Total number of nodes in the tree is on the order of O(c t(n)).

• The time it takes to reach the appropriate node in the tree is t(n)– Multiply this by the number of nodes to obtain the upper bound on

the complexity: O(t(n) c t(n)). – Claim: this is equivalent to 2 O(t(n)). Actually it’s greater but ok.

• Finally, need to convert multi-tape to single tape. But squaring an exponential is still just an exponential.

Definition of P

• The union of all language classes of the form:

TIME(1) TIME(n) TIME(n2) TIME(n3) …

• Thus, it’s all algorithms that can be decided by an ordinary (one-tape, deterministic) TM in polynomial time.– Even if we allowed multiple tapes, the time complexity would still by

polynomial, by our earlier theorem.– We also generalize “TM” to computer program. When we speak of a

O(f(n)) algorithm, this is generally assumed. But it’s still deterministic! So, it’s ok to say { 0n 1n } is O(n) problem.

• Many algorithms are in P. This is where we like to be.– No recursion– Nested loops ok.

CS 461 – Nov. 28

Sections 7.3 – 7.4

• Classes of problems “NP” and “NP-complete”

• We don’t know if P = NP

• Proving that an NP-complete problem.– And then showing that we have other problems of similar

complexity.

Beyond P

• We classify decidable problems by their complexity. √• Some problems apparently have no polynomial-time

algorithm. In other words, they appear not to be in P.• Most of these are in “NP”. They can be verified by DTM

in polynomial time. (Also: decided by NTM)– Nobody knows if “Hamiltonian Path” problem is in P.– But, given a path in a graph, we can easily verifying if it’s

Hamiltonian.

• Worse, some problems are apparently even outside NP.– “Does this graph not have a Hamiltonian path?” What given info

could we use to verify? Brute force

NP

• Up to 3 types of problems in NP– P (the easy ones)– NP-complete (the hard ones)– NP-intermediate (we don’t know if any exist!)

• But, if P = NP, then the above 3 types collapse into 1• NP-complete:

– Many NP problems are intrinsically related to one another. We can prove so by constructing a polynomial time reduction. Thus, if one of them turns out to be in P, they all are.

• Examples of NP problems:– Hamiltonian, Clique & subset sum (p. 268)– Isomorphic

NP-complete

• Formal definition. Q is NP-complete if:– The problem is in NP– Every other problem in NP can be reduced to Q in polynomial

time.

• Step 1 is usually straightforward.• For step 2, we have to select some existing problem

already known to be NP-complete and show that it corresponds to Q.

• But, from first principles, we have to show that first of all there exists one NP-complete problem.– Analogous to a base case in an inductive proof.

The SAT problem

• Given a boolean formula, is there some assignment of truth values to the variables that makes the formula true?– e.g. (p q’ r’ s) (p’ s) (q r’ s’)– In general, there could be a lot of different variables used in a

boolean formula, so we give our variables subscripted names x1, x2, x3, … And if there are lots of possible variables, we can even see multiple subscripts like x16,40,1.

• Cook-Levin theorem: SAT is NP-complete– We need to show that it’s NP, and that any other NP problem can

be reduced (in polynomial time) to SAT.– SAT is in NP because if we are given a truth assignment of the

variables, we can determine if the boolean formula is true.– The polynomial reduction is very interesting! (pp. 277-281)

Any NP SAT

• We’ll show that any NP language/problem can be reduced to SAT, and this reduction can be performed in polynomial time.

• If A is any NP problem, then there exists a NTM that decides A.– Furthermore, it runs in polynomial time, so the number of steps in its

execution can be bounded by nk.

• One possible execution of this NTM can be expressed in the form of a (nk x nk) table. – Rows correspond to individual steps of the NTM computation, and

literally represent the ID’s/configurations of the state & tape, along with blanks to fill up nk columns.

– Columns are for individual tape & state symbols on tape. First and last columns are # delimiters.

NTM execution table

• The table corresponds to one possible run of the NTM. It may represent an accepting computation or not.

• Although the table has nk rows, we may accept the input before running that many steps.

• Each cell in the table contains a symbol in an ID, which could be:– Delimiter # on the far left & right ends of the table.– State– Tape symbol

• The NTM accepts w if there exists a table showing an accepting configuration.

Creating • We now convert the table into a corresponding boolean

formula.

• The possible variables of our formula are of the form xi,j,s

where– i is a row number in the table– j is a column number in the table– s is a symbol that may appear in cell [ i, j ], such as a state, tape

symbol or #.

• What does this variable mean? xi,j,s is “true” if cell [ i, j ] contains the symbol s.

is created in such a way that is true when the corresponding execution table shows that the NTM accepts w.

= cell start accept move

• In order for the table to represent an “accept”, 4 conditions must be satisfied.

cell says that for all cells in the table, there can only be one symbol in that cell. For all i and j, there is exactly one s for which xi,j,s is “true”.

start says that the first row of the table contains the initial configuration of the NTM, including the # delimiters and blanks.

accept says that at some point we must reach an accept state. One cell must contain qaccept. For some i and j, xi,j,q accept

is true.

continued

move says that the rows of the table represent legal moves of the NTM.– We specify this by examining 2x3 windows of the table.

– Reminiscent of our PCP dominoes.

– Some windows center on activity around the state/cursor; some are far away from the state and don’t change from one row to the next.

– All windows must be “legal” according to and proper move behavior.

• This reduction from the NTM to can be done in polynomial time. We have a polynomial number of variables and terms to create. At most we need O(n2k) terms for .

Other problems

• We now have 1 problem, “SAT”, proved to be

NP-complete by definition.• To show that some other problem is NP-complete, all we

need to do is create a polynomial-time correspondence with SAT, or other problem already proved to be NP-complete.– Not necessary to start over with arbitrary NTM.

• Examples:– 3SAT: the satisfiability question assuming the boolean formula

is of the form (_ _ _) (_ _ _) (_ _ _) …i.e. CNF in which each factor has exactly 3 terms.

– Clique, subset-sum, isomorphic, Hamiltonian path, etc.

CS 461 – Nov. 30

Section 7.5• How to show a problem is NP-complete

– Show it’s in NP.– Show that it corresponds to another problem already known to

be NP-complete.

• Example NP-complete problems– 3SAT, special case of SAT– Clique– Subset sum

Other problems

• We now have 1 problem, “SAT”, proved to be

NP-complete by definition.• To show that some other problem is NP-complete, all we

need to do is create a polynomial-time correspondence with SAT, or other problem already proved to be NP-complete.– Not necessary to start over with arbitrary NTM.

• Examples:– 3SAT: the satisfiability question assuming the boolean formula

is of the form (_ _ _) (_ _ _) (_ _ _) …i.e. CNF in which each factor has exactly 3 terms.

– Clique, subset-sum, isomorphic, Hamiltonian path, etc.

Review:

3SAT

• Want to show this is NP-complete.• Note that we can’t simply say “3SAT is a special case of

SAT, therefore it’s NP-complete.” – Consider this special case of SAT: all boolean formulas

containing just one variable and no operators. Now the problem is trivially in P.

– It’s possible that a “special case” of an NP-problem could become simple enough to be in P. But in the case of 3SAT we show it’s still NP-complete.

• Simple approach: re-do our proof of SAT, but this time make sure each formula we write is expressed:– in CNF (conjunctive normal form)

– with exactly 3 terms per factor

3SAT in CNF

• The we created for SAT is almost already in CNF.– CNF means an AND of OR’s. has 4 factors. See book for precise definitions.

– 3 of the 4 factors are in CNF. Only 4th factor move needs modifying.

move is more complex: an AND of OR’s of AND’s.

We can use DeMorgan’s Law to convert OR’s of AND’s into AND’s of OR’s. Now we have CNF.

continued

• We also need exactly 3 terms per factor (_ _ _) .– If fewer than 3, just repeat a term. e.g. (p q q)

– To reduce 4 terms to 3, we would do this:

(x1 x2 x3 x4) = (x1 x2 d) (d’ x3 x4)

where “d” is a new dummy variable.

See why it works? Consider all xi false. Consider one xi true.

– For more than 4 terms the technique generalizes:

(x1 x2 x3 … xn) =

(x1 x2 d1) (d1’ x3 d2) (d2’ x4 d3)…

(dn-3’ xn-1 xn)

Clique problem

• Given a graph G and a number k, does G contain a complete subgraph of k vertices?– Alternatively, given a graph G, find the size (k) of the largest

complete subgraph.

• This problem is in NP, because we could be given a subgraph to consider, and we can verify in O(k2) time that it’s complete by checking that all required edges are present.

• To complete the proof that “Clique” is NP-complete, we will give a polynomial-time reduction from 3SAT to clique.– We already know that 3SAT is NP-complete.

– We must convert a boolean formula into a graph G, such that the is true iff G has a clique.

3SAT Clique

• We’re given , which has k factors, 3 terms per factor.• Create the graph G as follows.• The vertices are arranged in groups of 3.

– Each “triple” of vertices corresponds to a factor in – Label each vertex with corresponding term (e.g. x1’). Note that

the same label may appear in other triples.

• Edges connect all vertices except:– Vertices within the same triple

– Vertices that are opposites of each other (as x1 and x1’)

• Why does this construction work?– We’ll show: is “true” G has a clique with k vertices.

True clique

• Suppose = true.– One of the terms in each triple must be true.

– Looking at G, the vertices are grouped in k triples.

– There are lots of edges between triples. The only type of edge between triples we don’t have is between contradictory terms like x1 and x1’.

– A “true term” in a triple can always be connected to a “true term” in another triple.

– Therefore, there is a clique of size k.

• Suppose G has a clique of size k.– Each vertex in the clique must be in a different triple.

– Let’s assign truth values to each vertex to make each clique vertex true. There will be no contradictions.

– Therefore, each triple will have one term that is true, so is true.

Summary

• Thus far we have shown:– SAT is NP-complete– Therefore, 3SAT is NP-complete– Therefore, Clique is NP-complete

• Please watch this video lecture by Ron Graham:

RonaldLG1988.mpeg on the class Web site

• Next:– Wrap up section 7.5– Review

CS 461 – Dec. 2

Major results• Automata: 3 models of computation

– Definitions, properties, corresponding language classes, text representations

• Decidability• Complexity: classes of decidable languages

– P: polynomial algorithm– NP-complete: exponential algorithm & polynomial verifier– NP-hard: exponential algorithm, no known polynomial verifier

Graham’s talk

• Minimum spanning tree vs. minimum Steiner tree– Helpful lemmas don’t reduce exponential complexity– NP-complete: “I can’t find an efficient algorithm, but neither can all

these famous people!”

• One common feature of NP-complete problems:– A solution that’s almost right is no good.– TSP tour that is 2 miles too long.– Subset sum that might be off by 1.

• It may be good enough to – approximate solution

• e.g. Unfortunately, sum of square roots can be difficult to compare!

– Restricted problem – special case

• Sometimes even the verifying requires exponential time.

Major results

• NFA = DFA• FA = regular expression• Can simplify FA’s (Myhill-

Nerode theorem)• Pumping lemmas• PDA = CFG• CYK algorithm O(n3) time• Bottom-up parsing (linear

time)• Variants of TM

• Countable, uncountable

• ATM undecidable

non Turing recognizable language

• Rice’s theorem• PCP undecidable• SAT is NP-complete• Many other NP-complete

problems• We don’t know if P = NP

CS 461 – Dec. 5

Endgame• Review questions today• Study day (Wednesday)

– Conference room food/drinks/fun at 2:00– I’m available afterward if you have questions

• As Carol Burnett would say, …I’m so glad we had this time together

Just to have a laugh or sing a song.

Seems we just get started and before you know it

Comes the time we have to say so long.

cs 461 – aug. 24

Documents