finite-state automata shallow processing techniques for nlp ling570 october 5, 2011

Finite-State Automata

Shallow Processing Techniques for NLPLing570

October 5, 2011

RoadmapFinite-state automata variants

Non-deterministic Finite-State Automata (NFA)

Probabilistic Finite-State Automata (PFSA)

Finite-State Transducers (FST)Intro

CARMEL

FSAs FormallyA Finite-State Automaton (FSA) is a 5-tuple:

A set of states Q { q0,q1,q2,q3,q4}

A finite alphabet Σ {b,a,!}A state state q0

A set of accepting states {q4}

A transition function Q x Σ Q

Non-deterministic Finite-State Automata

Deterministic FSA:Given current state x input, at most one legal

transition



transition

Sources of non-determinism in NFA



transition

Sources of non-determinism in NFAMore than one transition for qi x a



transition

Sources of non-determinism in NFAMore than one transition for qi x a

ε-transitionsTransitions that consume no input


Deterministic FSA: Given current state x input, at most one legal transition

Sources of non-determinism in NFA More than one transition for qi x a

ε-transitionsTransitions that consume no input

Multiple start states

Transitions: Q x (Σ U ε) 2S

Sheeptalk NFAMultiple choice:

Sheeptalk NFAMultiple choice:

ε-transitions

NFA & DFAWhat is the relationship b/t NFAs and DFAs?


NFAs and DFAs are equivalentAccept same languages



Standard transformation from NFA to DFA:Create new state for each equivalence class in NFA



Standard transformation from NFA to DFA:Create new state for each equivalence class in NFA If NFA has N states, up to 2N states in DFA



Standard transformation from NFA to DFA:Create new state for each equivalence class in NFA If NFA has N states, up to 2N states in DFA

Why use both?

Managing Non-Determinism

Approaches to non-determinism


Approaches to non-determinismBackup/backtracking:

At choice points, mark state, inputIf fail, return and try alternative




Look-ahead:Look ahead to see which choice to take




Look-ahead:Look ahead to see which choice to take

Parallelism:At choice points, consider all paths in parallel

Basically the NFA DFA conversion process

Recognition in NFAsTwo standard approaches:


Transformation:Convert NFA DFAPerform standard DFA recognition


Transformation:Convert NFA DFAPerform standard DFA recognition

Search:Explicitly model recognition as state-space searchPerform backtracking as necessary

Recognition in NFAFor strings in the language:


There exists some state sequence leading to a final state



Not all paths necessarily lead to an accept state

For strings not in the language:



Not all paths necessarily lead to an accept state

For strings not in the language:No paths lead to an accept state

NFA Recognition as SearchSearch problem:


Start search-state(start node, beginning of input)



Goal test(final node, end of input)




Successor function Transitions





Path cost





Path cost

Use standard search algorithmse.g. Depth-first, breadth first, A*

Example

Breadth-first Search

Equivalences and Operations

FSAs equivalent to Regular Languages (Regex, etc)Can be shown by inductionDemonstrate primitive regex ops imitated by

automata


FSAs equivalent to Regular Languages (Regex, etc)Can be shown by inductionDemonstrate primitive regex ops imitated by

automata

E.g. Concatenation


Closure:

Union:

Finite-State ToolsToolkits:

AT&T Finite-State Toolkit:http://www.research.att.com/~fsmtools/fsm

NLTK: www.nltk.orgCARMEL (ISI): /NLP_TOOLS/ml_tools/FST/carmel

Finite-State ToolsToolkits:

AT&T Finite-State Toolkit:http://www.research.att.com/~fsmtools/fsm

NLTK: www.nltk.orgCARMEL (ISI): /NLP_TOOLS/ml_tools/FST/carmel

Functions:Convert Regex to FSAConvert FSA to RegexConvert NFA to DFAPerform operations: concatenation, disjunction, *, etc

Probabilistic Finite-State Automata (PFA)

PFA DefinitionA Probabilistic Finite-State Automaton is a 7-

tuple:A set of states QAn alphabet ΣA set of transitions: δsubset Q x Σ x Q


tuple:A set of states QAn alphabet ΣA set of transitions: δsubset Q x Σ x Q Initial state probabilities: Q R+



Transition probabilities: δ R+




Final state probabilities: Q R+




Final state probabilities: Q R+

s.t.

PFA ExampleExample

Example due to F. Xia

PFA ExampleExample

I(q0)=1

I(q1)=0


PFA ExampleExample

I(q0)=1

I(q1)=0

F(q0)=0

F(q1)=0.2


PFA ExampleExample

I(q0)=1

I(q1)=0

F(q0)=0

F(q1)=0.2

P(q0,a,q1)=1; P(q1,b,q1) =0.8


PFA ExampleExample

I(q0)=1

I(q1)=0

F(q0)=0

F(q1)=0.2

P(q0,a,q1)=1; P(q1,b,q1) =0.8

P(abn)


PFA ExampleExample

I(q0)=1

I(q1)=0

F(q0)=0

F(q1)=0.2

P(q0,a,q1)=1; P(q1,b,q1) =0.8

P(abn) = I(q0)*P(q0,a,q1)*P(q1,b,q1)n*F(q1)


PFA ExampleExample

I(q0)=1

I(q1)=0

F(q0)=0

F(q1)=0.2

P(q0,a,q1)=1; P(q1,b,q1) =0.8

P(abn) = I(q0)*P(q0,a,q1)*P(q1,b,q1)n*F(q1)

= 0.8n*0.2


PFAPFA augments NFAs with probabilities on arcs


Probability of a path through PFA is product of transition probabilities (and initial/final state probs)



Probability of a string is the sum of accepting paths.




Tech. issue: Numerical underflow:

PFA PFA augments NFAs with probabilities on arcs



Tech. issue: Numerical underflow: Replace product of probs with sum of log probs

Tasks: Given an input, find most probable state sequence Given an input, find its probability etc (We’ll do these when we generalize to HMMs)

Weighted FSASimilar to PFA

Arcs associated with weight (not probability)

Finite-State Transducers

FSTsFinite automaton that maps between two strings

Automaton with two labels/arcinput:output

FST ApplicationsTokenization

Segmentation

Morphological analysis

Transliteration

Translation

Speech recognition

Spoken language understanding

Approaches to FSTsFST as recognizer:

Takes pair of input:output stringsAccepts if in language, o.w. rejects



FST as generator:Outputs pairs of strings in languages




FST as translator:Reads an input string and prints output string




FST as translator:Reads an input string and prints output string

FST as set relator:Computes relations between sets

FSTs, Formally

FSTs & Regular RelationsFSAs: equivalent to regular languages


FSTs: equivalent to regular relationsSets of pairs of strings


FSTs: equivalent to regular relationsSets of pairs of strings

Regular relations:For all (x,y) in Σ1 x Σ2, {(x,y)} is a regular relation

The empty set is a regular relation If R1,R2 are regular relations,

R1R2 , R1 U R2 and R1* are regular relations

Regular Relation Operations

Regular relations are closed under: Inversion: Switching input and output labels

If T maps from I to O, T-1 maps from O to !

Regular Relation Operations

Regular relations are closed under: Inversion: Switching input and output labels

If T maps from I to O, T-1 maps from O to !

Composition:If T1 is a transducer from I1 to O2 and T2 is a

transducer from O2 to O3, then T1 T2 is a transducer from I1 to O3

CARMEL

Defining an FSA/FSTfinal-state

(from-state (to-state “input-sym” “outputsym”? weight?)*)

(from-state (to-state “input-sym” “outputsym”? weight?)*)

…..

First line is final state

Lines define state transitions input-sym (in quotes) required output-sym, weight optional

*e* is the representation for ε

patas:/NLP_TOOLS/ml_tools/fsm/carmel/latest/bin/carmel

FSA Exampleq4

(q0 (q1 “b”))

(q1 (q2 “a”))

(q2 (q3 “a”))

(q3 (q2 *e*))

(q3 (q4 “!”))

FST Exampleq1

(q0 (q0 “a” “A”))

(q0 (q1 “b” “b”))

(q1 (q1 “b” “B”))

WFST ExampleS

(S (S “they” “PRO” 1.0))

(S (S “can” “AUX” 0.99))

(S (S “can” “VERB” 0.01))

(S (S “fish” “NOUN” 0.7))

(S (S “fish” “VERB” 0.3))

Basic Functionscarmel –k N mywfst

Prints top N most probable paths through mywfst

carmel –Ok N mywsft Print top N most probable outputs from mywfst

carmel fst1 fst2 Composes fst1 and fst2

cat input_file | carmel –sli mywfst Converts 1st line of input_file to fst & composes (left) -b flag: batch processes & prints best path

HW #2Finite-State Automata

Goals:Build Finite-State Automata to represent Regex

Employ modern FST package to build FSA acceptor

Implement FSA acceptor yourselves

finite-state automata shallow processing techniques for nlp ling570 october 5, 2011

Documents

state state q

nfa slide

nondeterminism slide

dfa slide

input slide

mark state

new state

current state x input