Welcome

ASSALAM-0-ALAIKUM!!

Topics:

Lexical Analyzer Specification of tokens Recognition of Tokens Data structures Involved in Lexical

Analysis.

Lexical Analysis

It involves……… The Role of the Lexical Analyzer

Tokens , Patterns, and LexemeAttributes for Tokens

Role of Lexical Analyzer

As the first phase of a compiler, the main task of the lexical analyzer is to read the input characters of the source program, group them into lexemes, and produce as output a sequence of tokens for each lexeme in the source program.

The stream of tokens is sent to the parser for syntax analysis. It is common for the lexical analyzer to interact with the symbol table as well. When the lexical analyzer discovers a lexeme constituting an identifier, it needs to enter that lexeme into the symbol table

The call, suggested by the getNextToken command, causes the lexical analyzer to read characters from its input until it can identify the next lexeme and produce for it the next token, which it returns to the parser.

Sometimes, lexical analyzers are divided into a cascade of two processes: a) Scanning consists of the simple

processes that do not require tokenization of the input, such as deletion of comments and compaction of consecutive whitespace characters into one.

b) Lexical analysis proper is the more complex portion, where the scanner produces the sequence of tokens as output.

Token :When discussing lexical analysis, we use three

related but distinct terms: A token is a pair consisting of a token name and

an optional attribute value. The token name is an abstract symbol representing a kind of lexical unit,

e.g., a particular keyword, or a sequence of input characters denoting an identifier. The token names are the input symbols that the parser processes. In what follows, we shall generally write the name of a token in boldface. We will often refer to a token by its token name.

Patterns

A pattern is a description of the form that the lexemes of a token may take. In the case of a keyword as a token, the pattern is just the sequence of characters that form the keyword. For identifiers and some other tokens, the pattern is a more complex structure that is matched by many strings.

1. One token for each keyword. The pattern for a keyword is the same as the keyword itself.

2. One token representing all identifiers.

3. One or more tokens representing constants, such as numbers and literal strings .

4. Tokens for each punctuation symbol, such as left and right parentheses, comma, and semicolon.

Lexemes:

A lexeme is a sequence of characters in the source program that matches the pattern for a token and is identified by the lexical analyzer as an instance of that token.

Strings and Languages . Operations on Languages Regular Expressions . . . Regular Definitions . . . . Extensions of Regular Expressions

SPECIFICATION OF TOKENS

Specification of Tokens

Regular expressions are an important notation for specifying lexeme patterns.

While they cannot express all possible patterns those types of patterns that we actually need for tokens.

expressions to automata that perform the recognition of the specified token patterns, they are very effective in specifying those types of patterns that we actually need for tokens.

we shall see how these expressions are used in a lexical-analyzer generator.

how to build the lexical analyzer by converting regular expressions to automata that perform the recognition of the specified tokens.

Strings and Languages An alphabet is any finite set of symbols.

Typical examples of symbols are letters, digits, and punctuation. The set {a, 1 } is the binary alphabet. ASCII is an important example of an alphabet ; it is used in many software systems.Uni-code, which includes approximately 100,000 characters from alphabets around the world, is another important example of an alphabet.

A string over an alphabet is a finite sequence of symbols drawn from that alphabet. In language theory, the terms "sentence" and "word" are often used as synonyms for "string."

The length of a string s, usually written | s| , is the number of occurrences of symbols in s. For example, banana is a string of length six.

The empty string, denoted t, is the string of length zero.

A language is any countable set of strings over some fixed alphabet.

Terms for Parts of Strings A prefix of string S is any string obtained

by removing zero or more symbols from the end of s. For example, ban, banana, and E are prefixes of banana.

A suffix of string s is any string obtained by removing zero or more symbols from the beginning of s. For example, nana, banana, and E are suffixes of banana.

A substring of s is obtained by deleting any prefix and any suffix from s. For instance, banana, nan, and E are substrings of banana.

The proper prefixes, suffixes, and substrings of a string s are those, prefixes, suffixes, and substrings, respectively, of s that are not E or not equal to s itself.

A subsequence of s is any string formed by deleting zero or more not necessarily consecutive positions of s. For example, baan is a subsequence of banana.

If x and y are strings, then the concatenation of x and y, denoted xy, is the string formed by appending y to x. For example, if x = dog and y = house, then xy = doghouse. The empty string is the identity under concatenation;

This definition is very broad. Abstract languages like 0, the empty set, or {t} , the set containing only the empty string, are languages under this definition.

Operations on Languages

In lexical analysis, the most important operations on languages are union, concatenation, and closure, which are defined formally. Union is the familiar operation on sets. The concatenation of languages is all strings formed by taking a string from the first language and a string from the second language, in all possible ways , and concatenating them.

The (Kleene) closure of a language L, denoted L*, is the set of strings you get by concatenating L zero or more times. Note that L0 , the "concatenation of L zero times," is defined to be {ɛ} , and inductively, L 7 is L 7 ˉ ˡ L.

Finally, the positive closure, denoted L + ,is the same as the Kleene closure, but without the term L0 .

That is, ɛ will not be in L + unless it is in L itself.

 Regular Language

The set of regular languages over an alphabet  is defined recursively as below. Any language belonging to this set is a regular language over.

 

Definition of Set of Regular Languages : Basis Clause:  Ø, {ʌ} and {a} for any symbol a ϵ

  are regular languages. 

Inductive Clause: If Lr and Ls are regular languages, then Lr   Ls , LrLs and Lr

* are regular languages. 

Extremal Clause: Nothing is a regular language unless it is obtained from the above two clauses.

Regular expression Regular expressions are used to denote

regular languages. They can represent regular languages and operations on them succinctly. The set of regular expressions over an alphabet ∑ is defined recursively as below. Any element of that set is a regular expression. 

Basis Clause: Ø, ʌ , and a are regular expressions corresponding to languages Ø , {ʌ} and {a}, respectively, where a is an element of 

Inductive Clause: 

If r and s are regular expressions corresponding to languages Lr and Ls , then ( r + s ) , ( rs ) and ( r*) are regular expressions corresponding to languages Lr  Ls , LrLs and Lr

* , respectively. 

Extremal Clause: Nothing is a regular expression unless it is

obtained from the above two clauses.

Rules is a regular expression that denotes {}, the set

containing empty string. If a is a symbol in , then a is a regular expression

that denotes {a}, the set containing the string a. Suppose r and s are regular expressions denoting

the language L(r) and L(s), then (r) |(s) is a regular expression denoting L(r)L(s).(r)(s) is regular expression denoting L (r) L(s).(r) * is a regular expression denoting (L (r) )*.(r) is a regular expression denoting L (r).

Example of Regular Expressions Example: Let = { a, b }

The regular expression a|b denotes the set {a, b}

The regular expression (a|b)(a|b) denotes {aa, ab, ba, bb} the set of all strings of a’s and b’s of length two. Another regular expression for this same set is aa| ab| ba| bb.

The regular language a* denotes that the set

of all strings of zero or more a’s i.e… {ϵ,a, aa, aaa, aaaa, ……..}

Precedence Conventions

The unary operator * has the highest precedence and is left associative.

Concatenation has the second highest precedence and is left associative.

| has the lowest precedence and is left associative.

(a)|(b)*(c)a|b*c

The regular expression (a|b)* denotes the set of all strings containing zero or more instances of an a or b, that is the set of all strings of a’s and b’s. Another regular expression for this set is (a*b*)*

The regular expression a |a*b denotes the set containing the string a and all strings consisting of zero or more a’s followed by a b.

Properties of Regular Expression

where each di is a distinct name, and each ri

is a regular expression over the symbols in {d1,d2,…,di-1}, i.e.,

Note that each di can depend on all the previous d's.

Note also that each di can not depend on following d's. This is an important difference between regular definitions and productions.

Regular Definitions

These look like the productions of a context free grammar we saw previously, but there are differences. Let Σ be an alphabet, then a regular definition is a sequence of definitions

d1 → r1

d2 → r2

...

dn → rn

Example: C identifiers can be described by the following regular definition

letter_ → A | B | ... | Z | a | b | ... | z | _

digit → 0 | 1 | ... | 9

CId → letter_ ( letter_ | digit)*

Example: Unsigned numbers digit → 0| 1 | 2 | … | 9

digit → digit digit*optional_fractional → digit | ϵ

optional_exponent → (E ( + | - | ϵ ) digit ) | ϵ

num → digit optional_fraction optional_exponent

Extensions of Regular Expressions Many extensions have been added to regular expressions

to enhance their ability to specify string patterns. Here we mention a few notational extensions that were first incorporated into unix utilities such as Lex that are particularly useful in the specification lexical analyzers .

One or more instances. The unary, postfix operator + represents the positive closure of a regular expression and its language. That is , if r is a regular expression, then (r)+denotes the language (L( r) ) +. The operator + has the same precedence and associativity as the operator *. Two useful algebraic laws, r* = r+ I ϵ and r+ = rr* = r*r relate the Kleene closure and positive closure.

Zero or one instance. The unary postfix

operator ? means "zero or one occurrence ." That is , r? is equivalent to r lϵ , or put another way, L(r?) = L(r) U {ϵ}. The ? operator has the same precedence and associativity as * and +.

Example

digit → 0| 1 | 2 | … | 9

digit → digit digit+

optional_fractional → ( . digit ) ?

optional_exponent → (E ( + | - ) ? digit ) ?

num → digit optional_fraction optional_exponent

Character classes. A regular expression a1 la2|

···| an , where the ai ' s are each symbols of the alphabet , can be replaced by the shorthand [a la2 ··· an] . More importantly, when a 1 , a2 ,.·· ,an form a logical se quence, e.g., consecutive uppercase letters, lowercase letters, or digits, we can replace them by a I -an , that is , just the first and last separated by a hyphen. Thus, [abc] is shorthand for al b i c, and [a- z] is shorthand for al b l··· l z .

Example

[A-Z a-z][A-Z a-z 0-9]*

RECOGNITION OF TOKEN

Lexical analyzer

Also called a scanner or tokenizer Converts stream of characters into a

stream of tokens Tokens are:

Keywords such as for, while, and class.

Special characters such as +, -, (, and <Variable name occurrencesConstant occurrences such as 1, 0, true.

Phase Input Output

Lexer Sequence of characters

Sequence of tokens

Parser Sequence of tokens

Parse tree

Comparison with Lexical Analysis

The role of lexical analyzer

Lexical Analyzer ParserSourceprogram

token

getNextToken

Symboltable

To semanticanalysis

What are Tokens For? Parser relies on token classification

e.g., How to handle reserved keywords? As an identifier or a separate keyword for each?

Output of the lexer is a stream of tokens which is input to the parser.

The lexer usually discards “uninteresting” tokens that do not contribute to parsing Examples: white space, comments

Recognition of TokensTask of recognition of token in a lexical analyzer

Isolate the lexeme for the next token in the input buffer

Produce as output a pair consisting of the appropriate token and attribute-value, such as <id,pointer to table entry> , using the translation table given in the Fig in next page

Recognition of Tokens

Task of recognition of token in a lexical analyzer

Regular expression

Token Attribute-value

if if -id id Pointer to

table entry< relop LT

Recognition of tokens Starting point is the language grammar to

understand the tokens:stmt -> if expr then stmt

| if expr then stmt else stmt

| Ɛ

expr -> term relop term

| term

term -> id

| number

Recognition of tokens (cont.) The next step is to formalize the patterns:

digit -> [0-9]

Digits -> digit+

number -> digit(.digits)? (E[+-]? Digit)?

letter -> [A-Za-z_]

id -> letter (letter|digit)*

If -> if

Then -> then

Else -> else

Relop -> < | > | <= | >= | = | <> We also need to handle whitespaces:

ws -> (blank | tab | newline)+

The terminals of the grammar, which are if, then, else, relop, id, and number, are the names of tokens as used by the lexical analyzer.

The lexical analyzer also has the job of stripping out whitespace, by recognizing the "token" ws .

Compiler Construction

Tokens, their patterns, and attribute values

LEXICAL ANALYSIS Recognition of Tokens

Method to recognize the token Use Transition Diagram

Diagrams for Tokens Transition Diagrams (TD) are used to represent the

tokens

Each Transition Diagram has:

States: Represented by Circles

Actions: Represented by Arrows between states

Start State: Beginning of a pattern (Arrowhead)

Final State(s): End of pattern (Concentric Circles)

Deterministic - No need to choose between 2 different actions

Recognition of TokensTransition Diagram(Stylized flowchart)

Depict the actions that take place when a lexical analyzer is called by the parser to get the next token

start 0 6 7

8

return(relop,GE)

return(relop,GT)*

> =

otherStart state

Accepting state

Notes: Here we use ‘*’ to indicate states on which input

retraction must take place

if it is necessary to retract the forward pointer one position ( i.e., the lexeme does not include the symbol that got us to the accepting state) , then we shall additionally place a *near that accepting state.

Ex :RELOP = < | <= | = | <> | > | >=We begin in state 0, the start state < as the first input symbol, then among the lexemes that match the pattern for relop we can only be looking at <, <>, or <=.

Compiler Construction

Recognition of Identifiers

Ex2: ID = letter(letter | digit) *

9 10 11start letter

return(id)

# indicates input retraction

other#

letter or digitTransition Diagram:

Install the reserved words in the symbol table initially. A field of the symbol-table entry indicates that these strings are never ordinary identi fiers, and tells which token they represent . We have supposed that this method is in use in Fig . When we find an identifier, a call to install ID places it in the symbol table if it is not already there and returns a pointer to the symbol-table entry for the lexeme found Of course, any identifier not in the symbol table during lexical analysis cannot be a reserved word, so its token is id.

COMPLETION OF THE RUNNING EXAMPLE

So far we have seen the transition diagrams for identifiers and the relational operators.

What remains are: WhitespaceNumbers

Recognizing white spaces:We also want the laxer to remove

whitespace so we define a new token .

{ ws → ( blank | tab | newline ) + }

where blank, tab, and newline are symbols used to represent the corresponding ascii characters.

Transition diagram for white space:

The “delim” in the diagram represents any of the whitespace characters, say space, tab, and newline.

The final star is there because we needed to find a non-whitespace character in order to know when the whitespace ends and this character begins the next token.

There is no action performed at the accepting state. Indeed the lexer does not return to the parser, but starts again from its beginning as it still must find the next token.

Accepting integer e.g. 12

Accepting float e.g. 12.31

Accepting float e.g. 12.31E4

Transition Diagram for Numbers :

Explaination with example : Beginning in state 12, if we see a digit, we go

to state 13. In that state, we can read any number of additional digits . However, if we see anything but a digit or a dot , we have seen a number in the form of an integer; 12 is an example .

Accepting integere.g. 12

If we instead see a dot in state 13, then we have an "optional fraction ."

State 14 is entered, and we look for one or more additional digits; state 15 is used for that purpose.

If we see an E, then we have an "optional exponent ," whose recognition is the job of states 16 through 19.

Architecture of a Transition-Diagram- Based Lexical Analyzer : Each state is represented by a piece of

code . We may imagine a variable state holding the number of the current state for a transition diagram. A switch based on the value of state takes us to code for each of the possible states, where we find the action of that state. Often, the code for a state is itself a switch statement or multi way branch that determines the next state by reading and examining the next input character.

Transition diagram for relop:

Sketch of implementation of relop transition diagram

getRelop ( ), a C++ function whose job is to simulate the transition diagram and return an object of type TOKEN , that is , a pair consisting of the token name (which must be relop in this case) and an attribute value .

getRelop ( ) first creates a new object ret Token and initializes its first component to RELOP , the symbolic code for token relop

We see the typical behavior of a state in case 0, the case where the current state is 0. A function nextChar ( ) obtains the next character from the input and assigns it to local variable c. We then check c for the three characters we expect to find , making the state transition dictated by the transition diagram .

For example, if the next input character is =, we go to state 5.

If the next input character is not one that can begin a comparison operator, then a function fail ( ) is called, and It should reset the forward pointer to lexemeBegin.

Because state 8 bears a *, we must retract the input pointer one position (i.e. , put c back on the input stream) . That task is accomplished by the function retract ( ). Since state 8 represents the recognition of lexeme >=, we set the second component of the returned object , which we suppose is named attribute, to GT, the code for this operator.

DATA STRUCTURES INVOLVED IN LEXICAL ANALYSIS

DEFINITION

“The symbol table is a data structure used by all phases of the compiler to keep track of user defined symbols and keywords.”

In computer science, a symbol table is a data structure used by a language translator such as a compiler or interpreter, where each identifier in a program's source code is associated with information relating to its declaration or appearance in the source, such as its type, scope level and sometimes its location.

During early phases (lexical and syntax analysis) symbols are discovered and put into the symbol table.

During later phases symbols are looked up to validate their usage.

Uses An object file will contain a symbol table of the

identifiers it contains that are externally visible. During the linking of different object files, a linker will use these symbol tables to resolve any unresolved references.

A symbol table may only exist during the translation process, or it may be embedded in the output of that process for later exploitation, for example, during an interactive debugging session, or as a resource for formatting a diagnostic report during or after execution of a program.

While reverse engineering an executable, many tools refer to the symbol table to check what address has been assigned to global variables & Non-functions if the symbol table has been stripped or cleaned out before being converted into executable tools will find it harder to determining address or understand anything about the program.

At that time of accessing variable & allocating memory dynamically a compiler should perform many works & as such, the extended stack model requires the symbol table.

Typical symbol table activities:

Add a new name Add information for a name Access information for a name determine if a name is present in the table remove a name revert to a previous usage for a name (close a

scope).

Many possible Implementations

Linear List Sorted List Hash Table Tree Structure

Typical information fields:

print value kind

(e.g. reserved, type_id, var_id, func_id, etc.) block number/level number type initial value base address etc.