compiler components and their generators - lr parsing

Challenge the future

DelftUniversity ofTechnology

Course IN4303Compiler Construction

Guido Wachsmuth

Compiler Components & GeneratorsTraditional Parsing Algorithms

lessons learned

LR Parsing

Recap: Traditional Parsing Algorithms

How can we parse context-free languages effectively?

• predictive parsing

Which grammar classes are supported by these algorithms?

• LL(k) grammars, LL(k) languages

How can we generate compiler tools from that?

• implement automaton

• generate parse tables

2

today’s lecture

Lexical Analysis

Overview

3

today’s lecture

Lexical Analysis

Overview

efficient parsing algorithms

• LR parsing

• LR parse table generation

• SLR & LALR parse tables

3

LR Parsing

LR parsing

I

4

idea

LR Parsing

LR parsing

problems with LL parsing

• predicting right rule

• left recursion

LR parsing

• see whole left-hand side of a rule

• look ahead

• shift or reduce

5

example

LR Parsing

LR parsing

6

* 3 $

input

$

stack

7* 37 +

example

LR Parsing

LR parsing

6

* 3 $$ 7* 37 +

example

LR Parsing

LR parsing

6

* 3 $$ 7* 37 +

* 3 $7+$ E 3*

example

LR Parsing

LR parsing

6

* 3 $$ 7* 37 +

* 3 $7+$ E 3*

* 3 $7+$ E * E

example

LR Parsing

LR parsing

6

* 3 $$ 7* 37 +

* 3 $7+$ E 3*

* 3 $7+$ E * E

* 3 $7+$ E

example

LR Parsing

LR parsing

6

* 3 $$ 7* 37 +

* 3 $7+$ E 3*

* 3 $7+$ E * E

* 3 $7+$ E

$ E + E 3* $

example

LR Parsing

LR parsing

6

* 3 $$ 7* 37 +

* 3 $7+$ E 3*

* 3 $7+$ E * E

* 3 $7+$ E

$ E + E 3* $

$ E + E * E $

example

LR Parsing

LR parsing

6

* 3 $$ 7* 37 +

* 3 $7+$ E 3*

* 3 $7+$ E * E

* 3 $7+$ E

$ E + E 3* $

$ E + E

$ E + E * E $

$

example

LR Parsing

LR parsing

6

* 3 $$ 7* 37 +

* 3 $7+$ E 3*

* 3 $7+$ E * E

* 3 $7+$ E

$ E + E 3* $

$ E + E

$ E + E * E $

$

$ E $

example

LR Parsing

LR parsing

6

* 3 $$ 7* 37 +

* 3 $7+$ E 3*

* 3 $7+$ E * E

* 3 $7+$ E

$ E + E 3* $

$ E + E

$ E + E * E $

$

$ E $

$ S

LR Parsing

Grammar classes

7

context-free grammars

LL(k)

LL(1)

LL(0)

LR Parsing

Grammar classes

7


LR(0)

LL(k)

LL(1)

LL(0)

LR Parsing

Grammar classes

7


LR(1)

LR(0)

LL(k)

LL(1)

LL(0)

LR Parsing

Grammar classes

7


LR(k)

LR(1)

LR(0)

LL(k)

LL(1)

LL(0)

LR Parsing

Grammar classes

7


LR(k)

LR(1)

SLR

LR(0)

LL(k)

LL(1)

LL(0)

LR Parsing

Grammar classes

7


LR(k)

LR(1)

LALR(1)

SLR

LR(0)

LL(k)

LL(1)

LL(0)

LR Parsing

LR parse tables

II

8

parse table

LR Parsing 9

rows

• states of a DFA

columns

• topmost stack symbol

• Σ, N

entries

• reduce, goto state

• shift, goto state

• goto state

LR parsing

T1 ... N1 ...

1 s 3

2 g 5

3 r 1

4 r 2

5

6 g 1

7 s 1

8

...

items, closure & goto

LR Parsing

LR(0) parse tables

10

S → x S → ( L ) L → SL → L , S

S’ → . S $


LR Parsing

LR(0) parse tables

10

S → x S → ( L ) L → SL → L , S

S’ → . S $


LR Parsing

LR(0) parse tables

10

item

S → x S → ( L ) L → SL → L , S

closure

• for every item A → α . X β

• for every rule X → γ

• add item X → . γ

S’ → . S $


LR Parsing

LR(0) parse tables

10

item

S → x S → ( L ) L → SL → L , S

S’ → . S $S → . x S → . ( L )

closure

• for every item A → α . X β

• for every rule X → γ

• add item X → . γ


LR Parsing

LR(0) parse tables

10

S → x S → ( L ) L → SL → L , S

S’ → . S $S → . x S → . ( L )


LR Parsing

LR(0) parse tables

10

S → x S → ( L ) L → SL → L , S

S’ → . S $S → . x S → . ( L )


LR Parsing

LR(0) parse tables

10

S’ → S . $

S S → x S → ( L ) L → SL → L , S

S’ → . S $S → . x S → . ( L )


LR Parsing

LR(0) parse tables

10

S’ → S . $

S

x S → x .

S → x S → ( L ) L → SL → L , S

S’ → . S $S → . x S → . ( L )


LR Parsing

LR(0) parse tables

10

S’ → S . $

S

x S → x .

( S → ( . L )

S → x S → ( L ) L → SL → L , S

S’ → . S $S → . x S → . ( L )


LR Parsing

LR(0) parse tables

10

S’ → S . $

S

x S → x .

( S → ( . L ) L → . SL → . L , SS → . x S → . ( L )

S → x S → ( L ) L → SL → L , S

S’ → . S $S → . x S → . ( L )


LR Parsing

LR(0) parse tables

10

S’ → S . $

S

x S → x .

( S → ( . L ) L → . SL → . L , SS → . x S → . ( L )

x

S → x S → ( L ) L → SL → L , S

S’ → . S $S → . x S → . ( L )


LR Parsing

LR(0) parse tables

10

S’ → S . $

S

x S → x .

( S → ( . L ) L → . SL → . L , SS → . x S → . ( L )

x

( S → x S → ( L ) L → SL → L , S

S’ → . S $S → . x S → . ( L )


LR Parsing

LR(0) parse tables

10

S’ → S . $

S

x S → x .

( S → ( . L ) L → . SL → . L , SS → . x S → . ( L )

x

(

S

L → S .

S → x S → ( L ) L → SL → L , S

S’ → . S $S → . x S → . ( L )


LR Parsing

LR(0) parse tables

10

S’ → S . $

S

x S → x .

( S → ( . L ) L → . SL → . L , SS → . x S → . ( L )

x

(

S

L → S .

L S → ( L . ) L → L . , S

S → x S → ( L ) L → SL → L , S

S’ → . S $S → . x S → . ( L )


LR Parsing

LR(0) parse tables

10

S’ → S . $

S

x S → x .

( S → ( . L ) L → . SL → . L , SS → . x S → . ( L )

x

(

S

L → S .

L S → ( L . ) L → L . , S

)

S → ( L ) .

S → x S → ( L ) L → SL → L , S

S’ → . S $S → . x S → . ( L )


LR Parsing

LR(0) parse tables

10

S’ → S . $

S

x S → x .

( S → ( . L ) L → . SL → . L , SS → . x S → . ( L )

x

(

S

L → S .

L S → ( L . ) L → L . , S

)

S → ( L ) .

,L → L , . S

S → x S → ( L ) L → SL → L , S

S’ → . S $S → . x S → . ( L )

L → L , . S S → . x S → . ( L )


LR Parsing

LR(0) parse tables

10

S’ → S . $

S

x S → x .

( S → ( . L ) L → . SL → . L , SS → . x S → . ( L )

x

(

S

L → S .

L S → ( L . ) L → L . , S

)

S → ( L ) .

,

S → x S → ( L ) L → SL → L , S

S’ → . S $S → . x S → . ( L )

L → L , . S S → . x S → . ( L )


LR Parsing

LR(0) parse tables

10

S’ → S . $

S

x S → x .

( S → ( . L ) L → . SL → . L , SS → . x S → . ( L )

x

(

S

L → S .

L S → ( L . ) L → L . , S

)

S → ( L ) .

,

x

S → x S → ( L ) L → SL → L , S

S’ → . S $S → . x S → . ( L )

L → L , . S S → . x S → . ( L )


LR Parsing

LR(0) parse tables

10

S’ → S . $

S

x S → x .

( S → ( . L ) L → . SL → . L , SS → . x S → . ( L )

x

(

S

L → S .

L S → ( L . ) L → L . , S

)

S → ( L ) .

,

x

(

S → x S → ( L ) L → SL → L , S

S’ → . S $S → . x S → . ( L )

L → L , . S S → . x S → . ( L )


LR Parsing

LR(0) parse tables

10

S’ → S . $

S

x S → x .

( S → ( . L ) L → . SL → . L , SS → . x S → . ( L )

x

(

S

L → S .

L S → ( L . ) L → L . , S

)

S → ( L ) .

,

x

(

SL → L , S .

S → x S → ( L ) L → SL → L , S

S’ → . S $S → . x S → . ( L )

L → L , . S S → . x S → . ( L )


LR Parsing

LR(0) parse tables

10

S’ → S . $

S

x S → x .

( S → ( . L ) L → . SL → . L , SS → . x S → . ( L )

x

(

S

L → S .

L S → ( L . ) L → L . , S

)

S → ( L ) .

,

x

(

SL → L , S .

S → x S → ( L ) L → SL → L , S

12

3

4 6 7

5

8

9

( ) x , $ S L

1 s 3 s 2 g 4

2 r 1 r 1 r 1 r 1 r 1

3 s 3 s 2 g 6 g 5

4 a

5 s 7 s 8

6 r 3 r 3 r 3 r 3 r 3

7 r 2 r 2 r 2 r 2 r 2

8 s 3 s 2 g 9

9 r 4 r 4 r 4 r 4 r 4

result

LR Parsing 11

LR(0) parse tables

S → x S → ( L ) L → SL → L , S

( ) x , $ S L

1 s 3 s 2 g 4

2 r 1 r 1 r 1 r 1 r 1

3 s 3 s 2 g 6 g 5

4 a

5 s 7 s 8

6 r 3 r 3 r 3 r 3 r 3

7 r 2 r 2 r 2 r 2 r 2

8 s 3 s 2 g 9

9 r 4 r 4 r 4 r 4 r 4

result

LR Parsing 11

LR(0) parse tables

S → x S → ( L ) L → SL → L , S

$)x,x(

1

( ) x , $ S L

1 s 3 s 2 g 4

2 r 1 r 1 r 1 r 1 r 1

3 s 3 s 2 g 6 g 5

4 a

5 s 7 s 8

6 r 3 r 3 r 3 r 3 r 3

7 r 2 r 2 r 2 r 2 r 2

8 s 3 s 2 g 9

9 r 4 r 4 r 4 r 4 r 4

result

LR Parsing 11

LR(0) parse tables

S → x S → ( L ) L → SL → L , S

$)x,x

13

( ) x , $ S L

1 s 3 s 2 g 4

2 r 1 r 1 r 1 r 1 r 1

3 s 3 s 2 g 6 g 5

4 a

5 s 7 s 8

6 r 3 r 3 r 3 r 3 r 3

7 r 2 r 2 r 2 r 2 r 2

8 s 3 s 2 g 9

9 r 4 r 4 r 4 r 4 r 4

result

LR Parsing 11

LR(0) parse tables

S → x S → ( L ) L → SL → L , S

$)x,

132

( ) x , $ S L

1 s 3 s 2 g 4

2 r 1 r 1 r 1 r 1 r 1

3 s 3 s 2 g 6 g 5

4 a

5 s 7 s 8

6 r 3 r 3 r 3 r 3 r 3

7 r 2 r 2 r 2 r 2 r 2

8 s 3 s 2 g 9

9 r 4 r 4 r 4 r 4 r 4

result

LR Parsing 11

LR(0) parse tables

S → x S → ( L ) L → SL → L , S

$)x,

13

( ) x , $ S L

1 s 3 s 2 g 4

2 r 1 r 1 r 1 r 1 r 1

3 s 3 s 2 g 6 g 5

4 a

5 s 7 s 8

6 r 3 r 3 r 3 r 3 r 3

7 r 2 r 2 r 2 r 2 r 2

8 s 3 s 2 g 9

9 r 4 r 4 r 4 r 4 r 4

result

LR Parsing 11

LR(0) parse tables

S → x S → ( L ) L → SL → L , S

$)x,

136

( ) x , $ S L

1 s 3 s 2 g 4

2 r 1 r 1 r 1 r 1 r 1

3 s 3 s 2 g 6 g 5

4 a

5 s 7 s 8

6 r 3 r 3 r 3 r 3 r 3

7 r 2 r 2 r 2 r 2 r 2

8 s 3 s 2 g 9

9 r 4 r 4 r 4 r 4 r 4

result

LR Parsing 11

LR(0) parse tables

S → x S → ( L ) L → SL → L , S

$)x,

13

( ) x , $ S L

1 s 3 s 2 g 4

2 r 1 r 1 r 1 r 1 r 1

3 s 3 s 2 g 6 g 5

4 a

5 s 7 s 8

6 r 3 r 3 r 3 r 3 r 3

7 r 2 r 2 r 2 r 2 r 2

8 s 3 s 2 g 9

9 r 4 r 4 r 4 r 4 r 4

result

LR Parsing 11

LR(0) parse tables

S → x S → ( L ) L → SL → L , S

$)x,

135

( ) x , $ S L

1 s 3 s 2 g 4

2 r 1 r 1 r 1 r 1 r 1

3 s 3 s 2 g 6 g 5

4 a

5 s 7 s 8

6 r 3 r 3 r 3 r 3 r 3

7 r 2 r 2 r 2 r 2 r 2

8 s 3 s 2 g 9

9 r 4 r 4 r 4 r 4 r 4

result

LR Parsing 11

LR(0) parse tables

S → x S → ( L ) L → SL → L , S

$)x

1358

( ) x , $ S L

1 s 3 s 2 g 4

2 r 1 r 1 r 1 r 1 r 1

3 s 3 s 2 g 6 g 5

4 a

5 s 7 s 8

6 r 3 r 3 r 3 r 3 r 3

7 r 2 r 2 r 2 r 2 r 2

8 s 3 s 2 g 9

9 r 4 r 4 r 4 r 4 r 4

result

LR Parsing 11

LR(0) parse tables

S → x S → ( L ) L → SL → L , S

$)

13582

( ) x , $ S L

1 s 3 s 2 g 4

2 r 1 r 1 r 1 r 1 r 1

3 s 3 s 2 g 6 g 5

4 a

5 s 7 s 8

6 r 3 r 3 r 3 r 3 r 3

7 r 2 r 2 r 2 r 2 r 2

8 s 3 s 2 g 9

9 r 4 r 4 r 4 r 4 r 4

result

LR Parsing 11

LR(0) parse tables

S → x S → ( L ) L → SL → L , S

$)

1358

( ) x , $ S L

1 s 3 s 2 g 4

2 r 1 r 1 r 1 r 1 r 1

3 s 3 s 2 g 6 g 5

4 a

5 s 7 s 8

6 r 3 r 3 r 3 r 3 r 3

7 r 2 r 2 r 2 r 2 r 2

8 s 3 s 2 g 9

9 r 4 r 4 r 4 r 4 r 4

result

LR Parsing 11

LR(0) parse tables

S → x S → ( L ) L → SL → L , S

$)

13589

( ) x , $ S L

1 s 3 s 2 g 4

2 r 1 r 1 r 1 r 1 r 1

3 s 3 s 2 g 6 g 5

4 a

5 s 7 s 8

6 r 3 r 3 r 3 r 3 r 3

7 r 2 r 2 r 2 r 2 r 2

8 s 3 s 2 g 9

9 r 4 r 4 r 4 r 4 r 4

result

LR Parsing 11

LR(0) parse tables

S → x S → ( L ) L → SL → L , S

$)

13

( ) x , $ S L

1 s 3 s 2 g 4

2 r 1 r 1 r 1 r 1 r 1

3 s 3 s 2 g 6 g 5

4 a

5 s 7 s 8

6 r 3 r 3 r 3 r 3 r 3

7 r 2 r 2 r 2 r 2 r 2

8 s 3 s 2 g 9

9 r 4 r 4 r 4 r 4 r 4

result

LR Parsing 11

LR(0) parse tables

S → x S → ( L ) L → SL → L , S

$)

135

( ) x , $ S L

1 s 3 s 2 g 4

2 r 1 r 1 r 1 r 1 r 1

3 s 3 s 2 g 6 g 5

4 a

5 s 7 s 8

6 r 3 r 3 r 3 r 3 r 3

7 r 2 r 2 r 2 r 2 r 2

8 s 3 s 2 g 9

9 r 4 r 4 r 4 r 4 r 4

result

LR Parsing 11

LR(0) parse tables

S → x S → ( L ) L → SL → L , S

$

1357

( ) x , $ S L

1 s 3 s 2 g 4

2 r 1 r 1 r 1 r 1 r 1

3 s 3 s 2 g 6 g 5

4 a

5 s 7 s 8

6 r 3 r 3 r 3 r 3 r 3

7 r 2 r 2 r 2 r 2 r 2

8 s 3 s 2 g 9

9 r 4 r 4 r 4 r 4 r 4

result

LR Parsing 11

LR(0) parse tables

S → x S → ( L ) L → SL → L , S

$

1

( ) x , $ S L

1 s 3 s 2 g 4

2 r 1 r 1 r 1 r 1 r 1

3 s 3 s 2 g 6 g 5

4 a

5 s 7 s 8

6 r 3 r 3 r 3 r 3 r 3

7 r 2 r 2 r 2 r 2 r 2

8 s 3 s 2 g 9

9 r 4 r 4 r 4 r 4 r 4

result

LR Parsing 11

LR(0) parse tables

S → x S → ( L ) L → SL → L , S

$

14

LR Parsing

conflict resolution

III

12

E → T + . E E → . T + EE → . TT → . x

S → . E $E → . T + E E → . T T → . x

shift-reduce conflicts

LR Parsing

SLR parse tables

13

T → x .

x

E S → E . $

T

T +

E

E → T . + E E → T .

E → T + E .

E → T + E E → T T → xx

E → T + . E E → . T + EE → . TT → . x

S → . E $E → . T + E E → . T T → . x


LR Parsing

SLR parse tables

13

T → x .

x

E S → E . $

T

T +

E

E → T . + E E → T .

E → T + E .

E → T + E E → T T → x

1

2

3

5 4 6x

E → T + . E E → . T + EE → . TT → . x

S → . E $E → . T + E E → . T T → . x


LR Parsing

SLR parse tables

13

T → x .

x

E S → E . $

T

T +

E

E → T . + E E → T .

E → T + E .

E → T + E E → T T → x

1

2

3

5 4 6x

x + $ E T1 s 5 g 2 g 32 a3 r 2 ? r 24 s 5 g 6 g 35 r 3 r 3 r 36 r 1 r 1 r 1

E → T + . E E → . T + EE → . TT → . x

S → . E $E → . T + E E → . T T → . x


LR Parsing

SLR parse tables

13

T → x .

x

E S → E . $

T

T +

E

E → T . + E E → T .

E → T + E .

E → T + E E → T T → x

1

2

3

5 4 6x

x + $ E T1 s 5 g 2 g 32 a3 s 4 r 24 s 5 g 6 g 35 r 3 r 36 r 1

look-ahead

LR Parsing

LR(1) parse tables

14

E → T + E E → T T → x

S → . E $E → . T + E E → . T T → . x

?$$+ $

E S → E . $ ?

T E → T . + E E → T .

$$

T → x .

x

+ $

T +

x

E → T + . E E → . T + EE → . TT → . x

$$$+ $ E E → T + E . $

look-ahead

LR Parsing

LR(1) parse tables

14

E → T + E E → T T → x

S → . E $E → . T + E E → . T T → . x

?$$+ $

E S → E . $ ?

T E → T . + E E → T .

$$

T → x .

x

+ $

T +

x

E → T + . E E → . T + EE → . TT → . x

$$$+ $ E E → T + E . $

x + $ E T1 s 5 g 2 g 32 a3 s 4 r 24 s 5 g 6 g 35 r 3 r 36 r 1

state space reduction

LR Parsing

LALR(1) parse tables

unify states

• with same items

• and same outgoing transitions

• but different look-ahead sets

might introduce new conflicts

15

LR Parsing

summary

IV

16

lessons learned

LR Parsing

Summary

17

lessons learned

LR Parsing

Summary

How can we generate LR parse tables?

• items, closure, goto

17

lessons learned

LR Parsing

Summary



How can we improve LR(0) parse table generation?

• SLR: consider FOLLOW sets to avoid shift-reduce conflicts

• LR(1): consider look-ahead in states

• LALR(1): unify LR(1) states to reduce state space

17

lessons learned

LR Parsing

Summary



How can we improve LR(0) parse table generation?

• SLR: consider FOLLOW sets to avoid shift-reduce conflicts

• LR(1): consider look-ahead in states

• LALR(1): unify LR(1) states to reduce state space

Why are efficient parsing algorithms problematic?

• not longer pure, declarative, beautiful

• paradise lost: seven plagues

• paradise regained: scannerless generalised parsing

17

learn more

LR Parsing

Literature

18

learn more

LR Parsing

Literature

syntactical analysis

Andrew W. Appel, Jens Palsberg: Modern Compiler Implementation in Java, 2nd edition. 2002

Alfred V. Aho, Ravi Sethi, Jeffrey D. Ullman, Monica S. Lam: Compilers: Principles, Techniques, and Tools, 2nd edition. 2006

18

learn more

LR Parsing

Literature

syntactical analysis

Andrew W. Appel, Jens Palsberg: Modern Compiler Implementation in Java, 2nd edition. 2002

Alfred V. Aho, Ravi Sethi, Jeffrey D. Ullman, Monica S. Lam: Compilers: Principles, Techniques, and Tools, 2nd edition. 2006

generalised parsing

Eelco Visser: Syntax Definition for Language Prototyping. PhD thesis 1997

M.G.J. van den Brand, J. Scheerder, J.J. Vinju, and E. Visser: Disambiguation Filters for Scannerless Generalized LR Parsers. CC 2002

Lennart C. L. Kats, Eelco Visser, Guido Wachsmuth: Pure and Declarative Syntax Definition - Paradise Lost and Regained. SPLASH 2010

18

coming next

LR Parsing

Outlook

lectures

• guest lecture: DSLs

Question & Answer Jan 08

• 10 questions, submit & vote

Lab Dec 14

• translate fields & variables

• challenge: variable ranges

19

LR Parsing

copyrights

20

LR Parsing 21

http://creativecommons.org/licenses/by-nc-sa/3.0/




copyrights

LR Parsing

Pictures

Slide 1: Book Scanner by Ben Woosley, some rights reserved

Slide 19: Ostsee by Mario Thiel, some rights reserved

22

http://www.flickr.com/photos/18713399@N00/1026799114/

http://www.flickr.com/photos/18713399@N00/1026799114/

http://www.flickr.com/people/empact/

http://www.flickr.com/people/empact/

http://www.flickr.com/photos/thiemadotcom/4290480618/

http://www.flickr.com/photos/thiemadotcom/4290480618/

http://www.flickr.com/people/thiemadotcom/

http://www.flickr.com/people/thiemadotcom/

compiler components and their generators - lr parsing

Education