CSCI 4325 / 6339Theory of ComputationChapter One

Zhixiang Chen

Department of Computer Science

University of Texas-Pan American

Background

Experience in at least one high-level programming language

Data structures Discrete math Others …

Topics

What is computation? Or what is an algorithm?

Formal languagesand computing devices? Computation vs. Formal language understanding?

What can be computed? Or what cannot be computed?

What can be efficiently computed? Or what cannot be efficiently computed?

Reading Materials

Chapter 0, Introduction

The World of Theoretical Foundations of Computation

…

PARTIALLY COMPUTABLE

COMPUTABLE

PSPACE

NEXP

EXP

NP

P

CHAPTER 0Introduction

Automata, Computability, and Complexity Want to know

What are the fundamental capabilities and limitations of computers?

Complexity

What makes some problem computationally hard and others easy?

How to classify problems according to their computational difficulties?

How to understand the computational difficulties in the worst case, randomized, or approximate settings?

How to harness the computational difficulties for applications like cryptography?

Computability

What are limitations of Computers? Want to know what problems are solvable by

computers, and what are not? Note: In complexity, we are interested in

“solvable” problem, but want to know how difficult to solve.

Automata

Automata theory deals with the definitions of properties of mathematical models of computation. These models play a role in several applied areas of computer science. Finite automata = regular expression: for text processing Context-free grammar: for programming language design and

implementation, AI, natural language processing Context-sensitive grammar: for natural language processing

Sets

What is a Set? A set is a collection of objects

Ex’s? Notes:

We use capital letters to denote sets An object contained in a set is called a set element. We use lower-

case letters to denote set elements. Notations

Given a set A and an object x, if x is an element of A, we write

Otherwise we write

Ax

Ax

Finite Sets and Infinite Sets

A set is finite if it has a finite number of elements Ex’s?

A set is infinite if it has an infinite number of elements Ex’s?

Set Cardinality

Given a set A, the cardinality of A, denoted by |A|, is the number of elements

in A. Ex’s?

Empty set

The empty set is a set without elements. The empty is denoted as Ø.

Set Representation

By enumeration: List all elements. Only good for finite sets Ex’s.

A = {1, 2, 3, 4, 5}, B = {a, b, c, d, e} C = { Joe, Jose, Chen_1, Chen_ 2} D = { Black_Cat, White_Cat}

By Property: Give property for set elements. Good for infinite sets (and

finite sets as well) Ex’s.

A = {x: x is an integer} B = {y: y is a student at UTPA} C = {z: z is a student in the CS department} D = {w: and w is even}Aw

Set Operations

Union A U B is a set of elements in either A or B A U B = {x: }

Intersection is a set of elements in both A and B.

Difference A – B is a set of elements in A but not in B.

BxAx

BA

}:{ BxAxxBA

}:{ BxAxxBA

Set Operations

Subset if any implies

Proper Subset if and

Equivalence A = B if

Want example’s?

BxAxBA

BA BA BA

ABBA

Properties of Set Operation Idempotency

A U A = A Commutativity

Associativity

ABBA

ABBA

)()(

)()(

CBACBA

CBACBA

Properties of Set Operation Distributivity

Absorption

DeMorgan’s Law

)()()(

)()()(

CBCACBA

CBCACBA

AABA

AABA

)(

)(

)()()(

)()()(

CABACBA

CABACBA

Can You See DeMorgan’s Law?

A

B

C

)()()(

)()()(

CABACBA

CABACBA

Disjoint Sets and Power Sets Disjoint Sets

A and B are disjoint if Power Sets

Given a set A, the power set of A, denoted by is the collection of all subsets of A.

Ex’s.

A = {1, 2, 3} = {, {1}, {2}, {3}, {1, 2}, {1, 3}, {2, 3}, {1, 2, 3}}

.BA

A2

}.:{2 AXXA

A2

Two Notations

Let S be a set such that any element in S is also a set, define

Ex’s

What is ? What is ?

},:{

},:{

PxSPxPS

PxSPxPS

SP

SP

A

A

2

2

Set Partition

Given a nonempty set A, we say is a partition of A if Any two elements of are disjoint.

AXX

A

}{2

Set Partition Examples

Ex’s Let S = {a, b, c, d} What partition of S has the fewest members? The

most members? List all partitions of S with exactly two members?

Relations and Functions- Ordered Pairs Given any two objects a, b, let (a, b) denote the

ordered pair of a and b. The ordered pair of a and b tells that a and b is

related under some relationship The differences between (a, b) and {a, b}:

(a, b) (b, a), while {a, b} = {b, a} (a, a) is a good ordered pair, while {a, a} is not.

The equivalence (a, b) = (c, d) if and only if a = c, b = d.

Ordered Tuples

Given any n objects is an n-ary ordered tuple.

Note: The objects may not be distinct.

Ordered tuple examples?

,,,, 21 naaa ),,,( 21 naaa

,,,, 21 naaa

Cartesian Products

Given two sets A and B, the Cartesian Product of A and B is

Given n sets , the n-fold Cartesian Product of these sets is

Ex’s?

}:),{( BbAabaBA

}1,:),,,{( 2121 niAaaaaAAA iinn

nAAA ,,, 21

Relations

A binary relation is defined as

A n-ary relation is defined as

BAR :BAR

BAAAR n 21:

BAAAR n 21

Functions

A function is a binary relation on A and B such that there is exactly one such that A is called the domain of f.

For any , we denote y as f(x). f(x) is called the image (or value) of x.

The range of f is defined as f [A] = { y: y = f(x) for some }

BAf :

,Ax By .),( fyx

Ax

fyx ),(

Function Mapping

A Bf

One-to-One, Onto, Bijection

One-to-One A function is one-to-one if for any two distinct

elements Onto

A function is onto B, if

Bijection A function is a bijection if f is one-to-one and

onto

BAf :

).()(,, bfafAba

BAf :

).(,, xfyAxBy

BAf :

Inverse

The inverse of a binary relation is

The inverse of a function may not be a function any more. Why?

If a function is bijection then its inverse is still a function. Why?

BAR :}),(:),{(1 RbaabR

Composition

Let Q and R to be two relations

Then RQ is also a relation

,: BAQ CBR :

CARQ :

}),(),(,:),{( RzyQyxByzxRQ

A B C

Q R

Composition Examples

Let R = {(a, b), (a, c), (c, d), (a, a), (b, a)}, What is RR? What is ? Is R, RR or a function?

1R1R

Graph Representation of a Binary Relation Given a binary relation

for each draw a node to represent a, draw another node to represent b, draw an edge from a to b. Then, we obtain a directed graph for R.

,: AAR

,),( Rba

An Graph Representation Example Let R = {(a, b), (b, a), (a, d), (d, c), (c, c),

(c, a)}. The Graph is

a b

cd

Reflexive

A relation R: A→A is reflexive, if for any

For a reflexive relation R, the directed graph for R has a self-circle for each node

Graph example? Other examples?

The less-than-or-equal-to relation over integers

.),(, RaaAa

Symmetric

A relation R: A→A is symmetric if for any

In the directed graph of a symmetric relation, if there is an edge from node a to node b, then there must be another edge from node b to node a.

Graph examples? Other Examples?

The friend relation over a group of people.

.),(,),( RabRba

Antisymmetric

A relation R: A A is antisymmetric if for any

In the direct graph of a antisymmetric relation, if there is an edge from a to b, then there must be no edge from b to a.

Some graph examples? Other examples?

The parent relation over a group of people.

.),(,),( RabRba

Transitive

A relation R:A A is transitive if (a, b) and(a, c) are in R, then (a, c) is also in R.

In the directed graph of a transitive relation, if there a path from a to b, then there is an edge from a to b.

Some graph examples? Other examples?

The ancestor relation over a group of people. The less-than relation over integers.

Equivalence Relation

A relation R: A A is equivalent if R is reflexive, symmetric and transitive.

Let R be an equivalence relation, for any a in A, we define the equivalence class containing the element a as [a] = {x: (a, x) in R} That is, [a] is the set of all elements related to a.

Examples? The same-family relation The x=y (mod p) relation for the given p.

A Equivalence Relation Theorem Theorem. Let R: A→A to be an equivalence

relation, then the equivalence classes of R is a partition of A. That is {[a]: a in A} is a partition of A.

Proof.

AaAa

][

],[2,1 aSSAa

.2121 SSorSSeither

Partial Order A relation R: A→A is a partial order if R is

reflexive, antisymmetric, and transitive. Ex’s

Some graph examples? The boss-employee relation, assuming any

person is the boss of him/herself. Minimal elements of a partial order R: A→A

An element a in A is minimal if for any b in A, (b, a) in R implies b = a.

Ex’s

Total Order, Path, Length, Cycles Total Order

A relation R: A A is a total order, if R is a partial order, and for any a, b in A, either (a, b) in R or

(b, a) in R. Path, Length, Cycles

Easy to see from the directed graph of a relation R.

Examples?

Countable and Uncountable Sets Countable

A set S is countable if one of the following is true S is finite S is infinite but there is a bijection f: SN, where N is

the set of natural numbers. In other words, S is countable, if elements in S are

enumerable. Countable set examples? Uncountable

Sets that are not countable

Unions of Countable Sets

The union of a finite collection of countable sets is countable Proof. By dovetailing

The Union of a countably infinite collection of countable sets is countable Proof. By dovetailing

There Are Uncountable Sets

Proof. By diagonalization. Will give a proof later.

Definitions

Definitions describe the objects and notions that we use. A definition may be simple, or complex. Precision is essential to any mathematical definition. When defining some object we must make clear what constitutes that object and what does not.

Theorems

A theorem is a mathematical statement proved true. Generally a theorem is used to refer to a mathematical statement of special interest. Sometimes we prove statements that are interesting only because they assist in the proof of another, more significant statement. Such statements are called lemmas. Occasionally a theorem or its proof may allow us to conclude easily that other, related statements are true. These statements are called corollaries of the theorem.

Proofs

A proof is a convincing logical argument that a statement is true. In mathematics an argument must be airtight, that is, convincing in an absolute sense. This is rather different from the notion of proof that we use in everyday life or in the law. A mathematician demands proof beyond any doubt.

Five Fundamental Proof Techniques Proof by Mathematical Induction Proof by Construction Proof by Contradiction Proof by Pigeon Hole Principle Proof by Diagonalization

Mathematical Induction

The Principle of Math Induction: Let A be a set of natural numbers such that

and For each natural number , if then ,

then, A = N

Math Induction can be used to prove the following assertion For all natural numbers n, property P(n) is true.

A00n An 1An

Three Steps of Math Induction To show property P(n) is true for all natural

numbers, follow the following three steps: Base case: show P(0) is true. Induction hypothesis: Assume P(n) is true Induction: show P(n+1) is true.

Math Induction Example

Show that for any

Proof. (Prove in class.)

,0n

2

)1(21

nnn

Proof by Construction

Many theorems state that a particular type of object exists. One way to prove such a theorem is by demonstrating how to construct the object.

Theorem. For any even number n greater than 2, there exists a graph with n nodes such that every node has exactly 3 edges connecting to it.

Proof. By Construction.

Proof by Contradiction

We assume that the theorem is false and then show that this assumption leads to an obviously false consequence, called a contradiction.

Theorem. is irrational. Proof. By contradiction. (do it in class.)

2

The Pigeonhole Principle

The Principle. Let m pigeons to sit in n holes, if m > n then there is at least one hole with at least 2 pigeons.

Ex: Let R to be a binary relation on a finite set A, and let a, b be in A. If there is a path from a to b in R, then there is a path of length at most |A|.

Proof . By Pigeonhole Principle.

Diagonalization

The Principle. Let R to be a binary relation on a set A, define D = {a: a in A and (a, a) not in R}

For any a in A, define = {b: b in A and (a, b) in R}

Then, for any a in A,

aD

aDD

Applications of Diagonalizations

Theorem 1. Let [0, 1] denote the set of all real numbers between 0 and 1. Then it is uncountable.

Proof. By diagonalization. (Do it in class.)

Theorem 2. is uncountable, where N is the set of all nature numbers.

Proof. By diagonalization. (Do it in class.)

N2

Closures and Algorithms

The idea of reflexive transitive closure of a relation R is to find a smallest reflexive and transitive relation to cover R.

Ex.

R:

a1 O O a4

a2 O O a3

R*: a1 a4

a2 a3

R* is reflexive and transitive closure of R.

Reflexive Transitive Closure

Definition: Let R: A X A be a relation. The transitive closure of R is the relation R* = {(a, b): a, b are in A and there is a path from

a to b in R or a = b} Give R, how to find the transitive closure R*

of R? Suppose A = {a1, a2, . . . , an}

Closure Algorithm 1

Algorithm 1 Initially R* = For i = 1, . . . , n do For each i-tuple (b1, . . ., bi) in do

if (b1, . . ., bi) is a path in R then add (b1, bi) to R

iA

Closure Algorithm 2

Algorithm 2 Initially R* = R {(ai, ai) : ai in A} For j = 1, 2, . . ., n do

For i = 1, 2, . . ., n and k = 1, 2, . . ., n do If (ai, aj), (aj, ak) are in R* but (ai, ak) are not in R* then

add (ai, ak) to R*

Algorithm Analysis

How to show the correctness of the two closure algorithms? Do they halt on any given input? If halt, Do they output the right result? What are the performances of the two algorithms

Time complexity Space complexity

How to analyze the performance?

Big-O Notation

Let f, g be two functions from N to N. f(n) = O(g(n)) if there are two constants c and such that for all

Ex. But

More Ex’s ?

0nn)()( ncgnf

0n

)(16100 525 nOnn )(16100 225 nOnn

Time Complexity of the Two Closure Algorithms

Algorithm 1

Algorithm 2

Which algorithm is better?

)( 1nnO

)( 3nO

Alphabets and Languages

Consider Pascal Consider English Observation:

Language = Alphabet + Syntax + Semantics Alphabet

A finite set of symbols Ex’s?

{0, 1}, {a, . . . , z, A, . . . , Z}

Strings

A string of an alphabet A sequence of symbols from the alphabet.

Ex’s 1010 of {0, 1} ILovetheory of {a, . . . , z, A, . . . , Z}

Empty string a string without symbols denoted by e.

String Length

The length of a string w the number of occurrences of symbols in w,

denoted by |w| Ex’s

|1010|= 4 |ILoveTheory|= 11 |e|=0

String Concatenation

Concatenation of two strings x, y: xy is the string of x followed by y. |xy|=|x| +|y|

Substring, Prefix, Suffix

z is a substring of w if there are strings x, y such that w = xyz Ex. Find substrings of “Iiovetheory”

z is a prefix of w if there is a string x such that w = zx Ex. Find prefixes of “Iiovetheory”

z is a suffix of w if there is a string x such that w = xz Ex. Find suffixes of “Iiovetheory”

Power of a String

The power of a string w Given any natural number i,

.0

,0,1 iifww

iifew ii

Reversal of a String

The reversal of a string w

where a is some letter in the alphabet

Property

,

,0||,

auwifau

wifew RR

RRR xyxy )(

Σ, Σ*, and Languages

Let Σ be any finite alphabet {a1, a2, . . . , an} Σ* denote the set of all strings of Σ Any subset L of Σ* is called a language over Σ. Ex’s: Pascal, English, binary numbers, decimal

numbers, etc.

Note: Languages are defined as sets. Thus, all set operations such as union, intersection, difference, complement are all applicable to languages.

Concatenation and Kleene Star Concatenation of Languages: Given two

languages and , the concatenation

of and is

Kleene Star: the Kleene star of a language L is

Ex’s = {a, b}, = {11}

},,,,0:{ 2121* LwwwkwwwL kk

}:{ 22112121 LwLwwwLL

2L1L

1L

1L

2L

2L

The closure of a language

The closure of a language L:

Ex. Find the closures of the following languages L1 = {a, b}, L2 = {11}

},,,,1:{ 2121* LwwwkwwwLLL kk

Finite Representations of Languages The Large Picture

Given an alphabet Σ, the collection of all languages over Σ is

when , Σ* is countably infinite,

is uncountably infinite.

1||

*

2

*

2

Finite Representation of a Language A finite representation of a language

is a finite string of some other finite alphabet .

The total number of languages represented by strings of is .

is countably infinite.

Can you get some conclusion?

*

2L

|| *

*

The Language World

All languages

Finitely representable languages

Regular Expressions

Definition. Regular expressions over a given alphabet Σ are defined as follows: (1) Φ and any symbol of Σ are regular exp. (2) If α and β are regular expressions, then so are

(α β), (α U β ), and α* (3) Only those derived from (1) and (2) are regular

expressions Ex’s. Σ = {a, b}

Φ, a, b, (ab), (aa)*, (aabb)*, (aba)*(bb)*

Regular Languages

Languages defined by regular expressions Given a regular expression α, let L(α) denote the

language represented by α. We have L(Φ) = Φ, L(a) = {a} for a Σ L(α U β ) = L(α) U L(β) L(αβ) = L(α) L(β) L(α*) = L(α)*

Ex’s (give examples in class.) A language L is regular if there is a regular

expression α such that L = L(α).

Non-Regular Languages

There are languages which are not regular

Ex’s

}:{

}0:{*

wwwB

nbaAR

nn

Recognizable & Generatable

A and B are recognizable Ex.

We can write a C++ program which, given any string w Σ*, can decide whether w in A or not

A and B are generatable Ex. We can write a C++ program to enumerate all

elements of A, provided we have infinite supply of paper and computing time and space.

A New World of Languages

All languages

Finitely representable languages

Regular languages