Lecture 6Lecture 6Lecture 7Lecture 7

Regular ExpressionsRegular Expressionsgrepgrep

Why Regular Expressions?Why Regular Expressions? Regular expressionsRegular expressions are used to describe text are used to describe text

patterns/filterspatterns/filters Unix commands/utilities that support regular Unix commands/utilities that support regular

expressions:expressions: grepgrep(fgrep, egrep) - (fgrep, egrep) - searchsearch a file for a string or a file for a string or

regular expressionregular expression sed - sed - stream stream editoreditor awk awk (nawk) - (nawk) - pattern pattern scanningscanning and and processingprocessing

languagelanguage There are some minor differences between the There are some minor differences between the

regular expressions supported by these programsregular expressions supported by these programs We will cover the general matching operators first.We will cover the general matching operators first.

Character ClassCharacter Class

[] matches any of the enclosed chars[] matches any of the enclosed chars [abc] [abc] matches a singlematches a single a b a b oror c c [a-z] [a-z] matches any of matches any of abcdef…xyzabcdef…xyz [^A-Za-z] [^A-Za-z] matches a single character as long as it matches a single character as long as it

is not a letter.is not a letter.

Example: [Dd][Aa][Vv][Ee]Example: [Dd][Aa][Vv][Ee] Matches "Dave" or "dave" or "dAVE",Matches "Dave" or "dave" or "dAVE", Does not match "ave" or "da"Does not match "ave" or "da"

Regular Expression OperatorsRegular Expression Operators Any character (except a metacharacter!) matches itself.Any character (except a metacharacter!) matches itself. .. Matches any single character except newline.Matches any single character except newline. ** Matches 0 or more of the immediately preceding R.E.Matches 0 or more of the immediately preceding R.E. ?? Matches 0 or 1 instances of the immediately preceding Matches 0 or 1 instances of the immediately preceding

R.E.R.E. + + Matches 1 or more instances of immediately preceding Matches 1 or more instances of immediately preceding

R.E.R.E. ^̂ Matches the preceding R.E. at the beginning of the lineMatches the preceding R.E. at the beginning of the line $$ Matches the preceding R.E. at the end of the lineMatches the preceding R.E. at the end of the line || Matches the R.E. specified before or after this symbolMatches the R.E. specified before or after this symbol \\ Turn off the special meaningTurn off the special meaning

Examples of R.E.Examples of R.E.

x[abc]?xx[abc]?x matches "xax" or "xx“matches "xax" or "xx“

[abc]* [abc]* matches "aaaaa" or "acbca"matches "aaaaa" or "acbca"

0*10 0*10 matches "010" or "0000010"or "10" matches "010" or "0000010"or "10"

^(dog)$^(dog)$ matches lines starting and ending matches lines starting and ending with dogwith dog

[\t ]*[\t ]*

(A|a)+b*c?(A|a)+b*c?

Grouping with parensGrouping with parens

If you put a subpattern inside parens you If you put a subpattern inside parens you can use + * and ? to the entire subpattern.can use + * and ? to the entire subpattern.

a(bc)*d a(bc)*d matches "ad" and "abcbcd"matches "ad" and "abcbcd"

does not match "abcxd" or "bcbcd"does not match "abcxd" or "bcbcd"

ExampleExample

1.1. Christian Scott lives here and will put on a Christmas partyChristian Scott lives here and will put on a Christmas party

2.2. There are around 30 to 35 people invited.There are around 30 to 35 people invited.

3.3. They are:They are:

4.4. TomTom

5.5. DanDan

6.6. Rhonda SavageRhonda Savage

7.7. Nicky and Kimberly.Nicky and Kimberly.

8.8. Steve, Suzanne, Ginger and LarrySteve, Suzanne, Ginger and Larry

^[A-Z]..$^[A-Z]..$ ^[A-Z][a-z]*3[0-5]^[A-Z][a-z]*3[0-5] [a-z]*\.[a-z]*\.

^ *[A-Z][a-z][a-z]$^ *[A-Z][a-z][a-z]$ ^[A-Z][a-z]*[^,][A-Za-z]*$^[A-Z][a-z]*[^,][A-Za-z]*$

Review: Metacharacters Review: Metacharacters for filename abbreviationfor filename abbreviation

* * Matches anything: Matches anything: ls Test*.docls Test*.doc ?? Matches any single characterMatches any single character

ls Test?.docls Test?.doc [abc…][abc…] Matches any of the enclosed Matches any of the enclosed

characters: characters: ls T[eE][sS][tT].docls T[eE][sS][tT].doc [a-z] matches any character in a range[a-z] matches any character in a rangels [a-zA-Z]*ls [a-zA-Z]*

[!abc…][!abc…] matches any character except those matches any character except those listed: listed: ls [!0-9]*ls [!0-9]*

Difference !!Difference !! Although there are similarities to the metacharacters Although there are similarities to the metacharacters

used in filename expansion – we are talking about used in filename expansion – we are talking about something different!something different! Filename expansion is done by the shell.Filename expansion is done by the shell. Regular expressions are used by commands (programs).Regular expressions are used by commands (programs).

However, be careful about specifying RE on the However, be careful about specifying RE on the command line as a result of this overlapcommand line as a result of this overlap Good idea to always quote RE with special chars (‘’or “”)on Good idea to always quote RE with special chars (‘’or “”)on

the command linethe command line Example:Example:

% grep ‘[a-z]*’ chap[12]*% grep ‘[a-z]*’ chap[12]*Note: filename mask expanded by shell w/o ``Note: filename mask expanded by shell w/o ``

grep - search for a stringgrep - search for a string grep [-bchilnsvw] PATTERN [filename...]grep [-bchilnsvw] PATTERN [filename...]

Read files or standard /redirected input Read files or standard /redirected input Search for specified pattern in each lineSearch for specified pattern in each line Send results to the standard outputSend results to the standard output

Examples:Examples:%grep ‘^X11’%grep ‘^X11’ *- search all files for lines starting with *- search all files for lines starting with

the string “X11”the string “X11”

%grep -v text file - print lines that do not match “text”%grep -v text file - print lines that do not match “text”

Regular expressions for grepRegular expressions for grep

cc any non special characterany non special character\c\c turn off any special meaning of character cturn off any special meaning of character c^̂ beginning of linebeginning of line$$ end of lineend of line.. any single characterany single character[...][...] any of characters in range .…any of characters in range .…[^....][^....] any single character not in range .…any single character not in range .…r*r* zero or more occurrences of rzero or more occurrences of r

Regular Expressions for grepRegular Expressions for grep

\<\< beginning of word anchorbeginning of word anchor\<abc matches “abcd” but not “dabc”\<abc matches “abcd” but not “dabc”

\>\> end of work anchorend of work anchorabc\> matches “dabc” but not “abcd”abc\> matches “dabc” but not “abcd”

\(…\)\(…\) stores the pattern …stores the pattern …\(abc\)def matches “abcdef” and stores\(abc\)def matches “abcdef” and storesabc in \1. So \(abc\)def\1 matches abc in \1. So \(abc\)def\1 matches

“abcdefabc”. Can store up to 9 matches“abcdefabc”. Can store up to 9 matches

grep - optionsgrep - options

Some useful optionsSome useful options--cc count number of linescount number of lines-h-h do not display filenamedo not display filename-l-l list only the files with matching lineslist only the files with matching lines-v-v display lines that do not matchdisplay lines that do not match--nn print line numbersprint line numbers

File dbFile db

northwestnorthwest NWNW Charles Main Charles Main 3.03.0 .98.98 33 3434

westernwestern WEWE Sharon GraySharon Gray 5.35.3 .97.97 55 2323

southwestsouthwest SWSW Lewis DalsassLewis Dalsass 2.72.7 .8.8 22 1818

southernsouthern SOSO Suan ChinSuan Chin 5.15.1 .95.95 44 1515

southeastsoutheast SESE Patricia HemePatricia Heme 4.04.0 .7.7 44 1717

easterneastern EAEA TB SavageTB Savage 4.44.4 .84.84 55 2020

northeastnortheast NENE AM Main Jr.AM Main Jr. 5.15.1 .94.94 33 1313

northnorth NONO Margot WebberMargot Webber 4.54.5 .89.89 55 99

centralcentral CTCT Ann StephensAnn Stephens 5.75.7 .94.94 55 1313

grep with pipesgrep with pipes

Remember, we can use pipes when a file Remember, we can use pipes when a file is expectedis expected

ls –l | grep ‘\<Feb.*3\>’ls –l | grep ‘\<Feb.*3\>’

egrepegrep

Extended grepExtended grep allows for more kinds of regular expressionsallows for more kinds of regular expressions unfortunately, egrep regular expressions are unfortunately, egrep regular expressions are

not a superset of grep regular expressionsnot a superset of grep regular expressions• some of grep’s regular expressions are not some of grep’s regular expressions are not

available in egrepavailable in egrep

grep vs. egrepgrep vs. egrep

new to egrepnew to egrep f+f+ matches one or more occurrences of fmatches one or more occurrences of f f?f? matches zero or one occurrences of fmatches zero or one occurrences of f f|gf|g matches f or gmatches f or g (ab)(ab) groups characters a and b togethergroups characters a and b together

only in greponly in grep \( … \), \<, \>\( … \), \<, \>

Final Note: Different versions of grep/egrep may Final Note: Different versions of grep/egrep may support different expressions. Make sure to support different expressions. Make sure to check the man pages.check the man pages.

Recommended ReadingRecommended Reading

Chapter 3Chapter 3 Chapter 4, sections 4.1 – 4.5 Chapter 4, sections 4.1 – 4.5