Unix - Shell Scripting

Unix Shell Programming - Advanced

Vinodh K Nair

8/23/2012 7:52:52 AM 5864_ER_FED_ALT 1

Introduction: Name Short ID Role Responsibility Experience Area of interest Any previous experience in shell scripting Current rating on shell scripting knowledge (1-10) Expectations from the program?

8/23/2012 7:52:52 AM 5864_ER_FED_ALT 2

Ground Rules In order to ensure the productivity of our training, we will need to be.Pagers and mobile phones off Full participation One speaker at a time Respect the views of others Silence indicates agreement Keep to the break times agreed

8/23/2012 7:52:52 AM 5864_ER_FED_ALT 3

Course Agenda:Shell Basics1. Introduction to Shell and types of Shells 2. Variables and Keywords 3. Metacharacters 4. Regular Expressions 5. Using grep and egrep 6. Using Quotes

Introduction Shell Programming1. Shell Programming Step 1 2. Expression Handling 3. Positional Parameters 4. Conditional Statement 5. Case Statement 6. Loops in Shell 7. IO Redirection

Advance Shell Programming1. Shell Functions 2. Advanced tools a) The find utility b) Cutting the outputs with cut 3. Fundamentals of awk 4. Fundamentals of sed 5. Miscellaneous tools 6. Debugging the Shell scripts 7. Dealing with Signals8/23/2012 7:52:52 AM 5864_ER_FED_ALT 4

Pre Quiz Write your name and Employee ID without fail. 25 questions, multiple choices in 15 minutes Use the pre-quiz columns for writing your answers Do not discuss A question can have more than one correct answer, but choose the best one. Please encircle your guessed answers to find how good you are at guessing Chocolates will be distributed for: Pre-quiz highest scorer Post-quiz highest scorer One with maximum difference8/23/2012 7:52:52 AM 5864_ER_FED_ALT 5

1. 2. 3.

Unix BasicsStructure of Unix

Applications

Shell Kernel

8/23/2012 7:52:52 AM 5864_ER_FED_ALT 6

ShellShell is a command interpreter and acts as an interface between a user and kernel

Types of Shells

Bourne Shell /usr/bin/sh Korn Shell /usr/bin/ksh C Shell /usr/bin/csh bash /bin/sh POSIX /usr/bin/sh

8/23/2012 7:52:52 AM 5864_ER_FED_ALT 7

8/23/2012 7:52:52 AM 5864_ER_FED_ALT 8

HPUX Directory Structure / (root)

newconfig

8/23/2012 7:52:52 AM 5864_ER_FED_ALT 9

Unix Shell ProgrammingBREAK

8/23/2012 7:52:52 AM 5864_ER_FED_ALT 10

Permissions --- --- ---

others

Permission r w x

Description read write execute

Weight 4 2 1

Ex: - rw- r- -r- - 1 user1 group1 12 Oct 10:45 file1

Use chmod to change the attributes

8/23/2012 7:52:52 AM 5864_ER_FED_ALT 11

Absolute path

Absolute path represents the location of a file or directory starting from the root directory

Ex: /> /home/user1/scripts/check.sh

Relative pathAccessing files and directories from the current directory

Ex: /home/user/scripts> ./check.sh

8/23/2012 7:52:52 AM 5864_ER_FED_ALT 12

VI Editor

bottom8/23/2012 7:52:52 AM 5864_ER_FED_ALT 13

8/23/2012 7:52:52 AM 5864_ER_FED_ALT 14

Shell BasicsVariablesVariables are the general words which are used to store and manipulate information within a shell program. Variables are fully under user control. These can be created and destroyed at anytime. Ex: = TOWN=Delhi ; TOTAL=0 ; DATE= 28Nov05

Local Variables Local Variables are the variables that are presented within the current instance of the Shell. These are not available to the any child processes that are started by the current shell. Local variables are defined by users. Global Variables Global Variables are the Variables that are presented within the current shell and also available to any child processes that are started by the current shell.

8/23/2012 7:52:52 AM 5864_ER_FED_ALT 15

Keywords

Keywords are the words which have standard predefined meaning in the shell already. Keywords can not be used as variable names. Keywords are also called as Reserve words.

List of Keywords

echo export done continue sleep

read if for exit test

set else until return

unset fi case trap

readonly while esac wait

shift do break exec

8/23/2012 7:52:52 AM 5864_ER_FED_ALT 16

Environment VariablesEnvironment Variables are the special variables that are set by the shell and is required by the shell in order to function correctly. These variables are also called as shell variables.

List of Environment Variables

PATH HOME LOGNAME SHELL TERM PWD PS1 IFS

Defines the path which the shell must search in Stores the default working directory for the user Stores the login name of the user Defines the name of default working shell Defines the name of the terminal Stores the Present working Directory Defines the system prompt Defines the Internal Field Separator

8/23/2012 7:52:52 AM 5864_ER_FED_ALT 17

8/23/2012 7:52:52 AM 5864_ER_FED_ALT 18

Shell Meta Characters. * ? [] [!...] Matches any one character Represents any combination of any characters Represents any one character Matches any one character from the enclosed list Matches any one character except those in the list

Ex: $ ls a* $ ls a?b? $ ls [aeiou]* $ ls [A-Fv-z]* $ ls [!A-Z]* Lists all files beginning with character a Lists all 4 character filenames whose first character is a and third character is b Lists all the files whose first character is a, e, i, o, u Lists all files whose first character is in the range A to F or v to z Lists all files whose first character is anything other than in the range of A to Z

8/23/2012 7:52:52 AM 5864_ER_FED_ALT 19

Regular ExpressionsA regular expression is a string that can be used to describe several sequences of characters. ^ $ \ ^$ \< \> \ Matches the beginning of a line Matches the end of a line Used to specify patterns that contains wild cards Matches a blank line Matches at the beginning of the word Matches at the end of the word Matches a complete word

8/23/2012 7:52:52 AM 5864_ER_FED_ALT 20

Basic Shell commands man - Very important command known in UNIX cp [-ir] file1 file2 cp [-ir] file-list directory i for interactive. prompt use whenever a file will be overwritten r for recursive. copy a directory tree

ls [-alRF] file-list a for listing all files including the dot files l for long format R for recrusive. list the all subdirectories. F for listing directories with a trailing /

date [+format] %date +%h %d, 19%y Oct 1, 19968/23/2012 7:52:53 AM 5864_ER_FED_ALT 21

Basic Shell commands (Cont) wc file-listDisplay the number of lines, words and characters

more file-listBrowse through text files one page at a time.

head [-n ] file-listDisplay the first n lines of the files (default=10)

tail [+n|-n| -f| ] Display the last few lines in the files (default = 10) Example: # tail +5 foo # display the last parf of foo starting from line 5 # tail -5 foo # display the last five lines of foo # tail +30 foo | head -15 | more #display line 30-45 of foo # tail -f foo # wait and display the new lines appended to foo

8/23/2012 7:52:53 AM 5864_ER_FED_ALT 22

Basic Shell commands (Cont) cut -c list filecut -f list [-dChar] file Cut out selected charatcers or fields from each line of a file Examples: # cut -c 1-5,15-20 foo # cut -f 1,3 -d moo

# extract chars 1-5 and 15-20 from each line of foo. # extract field 1 and 3 from each line of moo.

paste file1 file2 Concatenate corresponding lines of the given input files Example (reverse two fields of the file abc) # cut -f1 abc > abc1 # cut -f2 abc > abc2 # paste abc2 abc1 > xyz

8/23/2012 7:52:53 AM 5864_ER_FED_ALT 23

Basic Shell commands (Cont)sort [-tC] [-o outfile] [field-list] [file-list]sort the files # sort +1 list1 # sort +2 -3 list2 # sort -n -o list4 list3 # sort list 1 starting from field 2 to the end of the line # sort list2 on the third field # sort list3 numerically and place the output in list4

diff file1 file 2 Display different lines that are found when comparing two files It prints a message that users ed-line notation (a - append, c - change, d -delete) to describe how a group of lines has changed. It also describes what changes need to be made to the first file to make it the same as the second file. Example file1 file2 file3 apples apples oranges oranges oranges bananas bananas kumquats peaches8/23/2012 7:52:53 AM 5864_ER_FED_ALT 24

Basic Shell commands (Cont)Tr Command Translate input character to output character based on the input and output patterns Example # tr [A-Z] [a-z] out # translate all letters to lower case. # tr -s \012\011\040 \012\012\012 < in > out # translate blank, tab and new line chars to new line chars and squeeze (-s) consecutive newline char into one # tr -cs [a-z][A-Z] [\012*] < in > out # change all non-alphabetic (-c) chars to new line chars and squeeze consecutive new line char into one. # tr -d \040 < in > out # delete all blanks.

8/23/2012 7:52:53 AM 5864_ER_FED_ALT 25

8/23/2012 7:52:53 AM 5864_ER_FED_ALT 26

Basic Shell commands (Cont)Uniq Display a file, removing all successive repeated lines

Example:file1: apple banana banana apple banana # sort fruit | uniq -c apple 2 banana 3 # tr -cs [a-z][A-Z] [\012*] < fileA | sort | uniq # show a list of distinct words in fileA.8/23/2012 7:52:53 AM 5864_ER_FED_ALT 27

# uniq file1 apple banana apple banana

Basic Shell commands (Cont)Find Recursively search the directory tree rooted at and find all files whose names satisfy There are many details in the expression. Examples: # find . -name \*.doc -print # list all files names ending with .doc # find /etc/source -atime 2 -print # print the names of the files under /etc/source whose lst access time was 2 days ago. # find . -name [a-z]* -exec rm {} \; # remove the files under the current directory whose names begin lower case letter. # find / $-name a.out -o -name *.o $ -atime +7 -exec {} \; # remove the object and binary executable files under the root directory which have not be accessed more than 7 days.

with a

8/23/2012 7:52:53 AM 5864_ER_FED_ALT 28

Basic Shell commands (Cont) du [-as.] [dir list] [file list] Reports the allocated dispace for each file and/or directory specified

-a lists all files, -s lists the grand total of each dir given Examples: # du -s print the total disk space used by the files in and under the current dir.

# du -s *print the disk space used by each file and dir in the current dir.

8/23/2012 7:52:53 AM 5864_ER_FED_ALT 29

The grepThe grep stands for globally regular expression print. It lets us to search particular patterns in a file. The general form of grep is shown below grep < word > < file_name > Grep options -i -n -v -l Ex: $ grep ^unix read.txt $ grep system$ read.txt $ grep -l hello * $ grep -v unix read.txt Lists the lines beginning with word unix Lists the lines ending with word system Lists the file name which contains word hello Lists all the lines except those which contains the word unix Ignores the case Lists the line numbers along with matching patters Lists all the lines that do not match the patters Lists only filenames that contain match

8/23/2012 7:52:53 AM 5864_ER_FED_ALT 30

Quotes\ `` Ex: $ echo Hello\;Welcome $ echo $PWD $ echo $PWD $ echo Working Directory is : `pwd` $ OS=HP-UX $ VER=11i $ echo $OS $VER $ echo $OS $VER `hostname` $ echo $OS $VER $ echo $OS $VER `hostname` Used to specify patterns that contains wild cards Used to take string as-is Used when command substitution is needed Used when command evaluation is needed

8/23/2012 7:52:53 AM 5864_ER_FED_ALT 31

8/23/2012 7:52:53 AM 5864_ER_FED_ALT 32

What is a shell script? A text file containing UNIX commands A text file with READ and EXECUTE permissions A program executed/interpreted by the shell A tool for automating UNIX tasks A program that doesnt need to be compiled A program that is easy to create and modify

8/23/2012 7:52:53 AM 5864_ER_FED_ALT 33

Introduction to Shell ProgrammingShell Programming Step1A shell program can be defined as a series of commands to be executed to obtain a desired results. It can also be called as a shell script. Structure of a script

Shell InitializationScript Description Variable Declaration Function Definition Main Program Function calling Exit with Return Status

8/23/2012 7:52:53 AM 5864_ER_FED_ALT 34

Planning to write a shell script:Planning: What is the script required to do? What program components are required in the script? What logical testing conditions are required? Will user input be required? Will any special variables be required?

Preparation: Organize the components into logical order Keep everything simple to start with Write the components in plain English, and it to shell syntax

8/23/2012 7:52:53 AM 5864_ER_FED_ALT 35

Guidelines for writing Unix Commands/Tools/Scripts Standard command format Recognize meta-characters (handle multiple files) Standard I/O (stdin,stdout, stderr. If file arg is absent use std) Keep messages and prompts to a minimum.

Provide verbose options Input/output data should be text whenever possible. Use dot files and environment variables for frequently used info. Use standard library and tools to save coding effort.

8/23/2012 7:52:53 AM 5864_ER_FED_ALT 36

Simple ProgramConsider the following script and observe the output

### Script #1 welcome.sh echo Hello! Welcome to SHELL World echo Shell Scripts are very handy to system admins echo Enjoy the Course $ chmod u+x welcome.sh

$ ./welcome.sh Hello! Welcome to SHELL World Shell Scripts are very handy to system admins Enjoy the Course

8/23/2012 7:52:53 AM 5864_ER_FED_ALT 37

Interactive ScriptScript that runs interactively. It takes input from user while execution ###Script #2 interactive.sh echo Enter your Name : read name echo Enter your Designation : read design echo Enter your Organization name : read org echo Hello $name!! echo Your Position in $org is $design

8/23/2012 7:52:53 AM 5864_ER_FED_ALT 38

Here DocumentHere Document provides STDIN to Unix command from lines that follow until delimiter is found at start line. General syntax command 0 True if length of string is 0 True if string is not null

Operator ! -a -o

Description Condition is False Logical AND Logical OR

File Test Operator -f -s -d -r -w -x

Description True if file exists True if file exists True if file exists True if file exists True if file exists True if file exists

and is not a directory and size is greater than 0 and is a directory and has a read permission and has a write permission and has an execute permission

8/23/2012 7:52:54 AM 5864_ER_FED_ALT 53

if [ ! condition ] if [ -f $FILENAME ] if [ -d $DIRNAME ] if [ -x $FILENAME ] if [ -w $FILENAME ] if [ -r $FILENAME ] if [ -s $FILENAME ] if [ $VAR1 -ne $VAR2 ] if [ $VAR1 -gt $VAR2 ] if [ $VAR1 -lt $VAR2 ] if [ $VAR1 -eq $VAR2 ] if [ $VAR1 -ge $VAR2 ] if [ $VAR1 -le $VAR2 ] if ["$VAR1"="$VAR2"] if ["$VAR1"!="$VAR2"] if [ ! "$VAR1" ] if [ "$VAR1" ]

If condition is not true If file exists called $FILENAME If directory exists called $DIRNAME If file $FILENAME is executable If file $FILENAME is writable If file $FILENAME is readable If file $FILENAME exists and is of greater than zero size. If VAR1 does not equal VAR2 If VAR1 is greater than VAR2 If VAR1 is less than VAR2 If VAR1 is equal to VAR2 If VAR1 is greater than or equal to VAR2 if VAR1 is less than or equal to VAR2 If VAR1 is equal to VAR2 (for strings) If VAR1 does not equal VAR2 (strings) If VAR1 is null If VAR1 is not null

8/23/2012 7:52:54 AM 5864_ER_FED_ALT 54

8/23/2012 7:52:54 AM 5864_ER_FED_ALT 55

The case statementThe case statement in shell is used for flow control. It allows us to select one out of multiple choices available. It avoids nesting if-else-fi statements. General syntax case value in choice1) choice2) choice3) *) command choice 1 ;; command choice 2 ;; command choice 3 ;; command choice 4 to execute for no match ;;

8/23/2012 7:52:54 AM 5864_ER_FED_ALT 56

###Script #3 makechoice.sh echo Hello!! Welcome to the address tool echo 1. CSC Indore echo 2. CSC Noida echo 3. CSC Hyderabad echo \n Choice: \c read choice case $ choice in 1) echo CSC Indore echo Building #, Indore, MP ;; 2) echo CSC Noida echo C-24/25, Sector- 56, Noida, UP ;; 3) echo CSC Hyderabad echo Building 7, Mindspace, Hyderabad, AP ;; *) echo Invalid choice selected ;; esac8/23/2012 7:52:54 AM 5864_ER_FED_ALT 57

LoopsLoops enable us to execute series of commands multiple times. There are various loops available in shell to perform the repeated execution. The while loop The until loop The for loop While and until loops works in a very similar fashion. While loop executes as long as the test condition remains true whereas the until loop executes as long as the test condition remains false. For loop works slightly different from the while and until. It executes set of commands repeatedly for each item listed in the list. The most common use of for loop is to perform same set of commands for large number of files.

8/23/2012 7:52:54 AM 5864_ER_FED_ALT 58

The while loopGeneral syntax while do command set done ###Script #4 printnum1.sh i=0 while [ $i lt 10 ] do echo $i i=`expr $i + 1` done

8/23/2012 7:52:54 AM 5864_ER_FED_ALT 59

8/23/2012 7:52:54 AM 5864_ER_FED_ALT 60

The until loopGeneral syntax until do command set done ###Script #5 printnum2.sh i=10 until [ $i eq 0 ] do echo $i i=`expr $i - 1` done

8/23/2012 7:52:54 AM 5864_ER_FED_ALT 61

8/23/2012 7:52:54 AM 5864_ER_FED_ALT 62

The for loopGeneral syntax for item in item1 item2 item3 do command set done

### Script #6 printnum3.shfor num in 1 2 3 4 5 6 7 8 9 do echo $num done

8/23/2012 7:52:54 AM 5864_ER_FED_ALT 63

8/23/2012 7:52:54 AM 5864_ER_FED_ALT 64

8/23/2012 7:52:54 AM 5864_ER_FED_ALT 65

IO redirectionWhen a command is executed, the result produced is printed to a terminal which is called STDOUT. And the error messages occurred are printed to a special terminal called STDERR. Most of the commands will have STDOUT as their STDERR. Using the IO redirection operators, we can redirect the output or error messages to any files instead of STDOUT or STDERR. All these IO devices are identified by numbers as follows 0 1 2 2>&1 &> < > >>

Unix - Shell Scripting

Documents