unix scripting - a quick reference

8/8/2019 UNIX Scripting - A Quick Reference

1/12

Wipro Technologies; E-Enabling; Data Warehousing

Introduction to UNIX

History of UNIXGE, Bell Labs, and MIT: Network OS; Multiplex Information and Computing Services(Multics)Dennis Ritchie, Ken Thompson, and Brian KerninghamUnix code development language:Assembly Code Machine dependant and not portableB: Ken ThompsonC: Dennis RitchieOpen source development has lead to the availability of various flavors of UNIX.

Features of UNIXMulti-userThis feature is facilitated by approaches like Dumb Terminals, Terminal Emulation, andDial-up Connection

Discussion on SSH (Terminal Emulation)>ssh l informat app360.sea.wamu.net>ssh [email protected]

Shells in UNIXOrganization of the Unix SystemApplications - Shell - Kernel - HardwareBourne Shell (sh), C Shell (csh), Korn Shell (ksh), and Bash Shell (bash)>echo $SHELLThe default shell is set by the admin while creating the user and is given by the abovecommand.Every time a user logs in, a shell process is invoked which acts as the parent process forother processes created and serves as the command interpreter.

Login and Password resetting/bin/passwd contains details about all the users of the system./etc/passwd executable that allows us to change our passwords.

Internal and External commandsThe essential UNIX commands for general use are located in /usr/bin. All the commands,which have an independent existence in the above directory, are called externalcommands, while the commands that are built-in and are not stored as separate files arecalled internal commands.>type The above command will give you the location of an external executable command.

I/O redirection and PipingCombine commands by using the semi colon

>cd /informat/scripts; ls lrtls > dir.list puts the output of ls into a file named dir.list. ls | sort pipes the output of lsinto the sort command. The final output on the terminal will now be sorted.UNIX I/O revolves around an integer number called a file descriptor. When a file or otherdevice is opened, a file descriptor number is assigned and used for all the I/O. Byconvention, UNIX programs write their normal output to unit 1, and expect input on unit 0.Error/diagnostic output goes to unit 2.

Unix commands and their help files.Pager pg, more, less

- 1 -


2/12


>export $PAGER=moreNavigation is through f and b.Discussion on search facility, using the forward slash (/) and how to quit (:q).To get a one line description of the command, useman k, apropos, man f, whatis

Introduction to Visual Editor (vi) (Part I)

Modes of OperationIn the Command mode, all the keys pressed by the user are interpreted as commands.In the Insert mode, one is permitted to insert new text.In the ex Command mode, the commands are given at the command line.

Adding, deleting and overwriting textHands on

Create a new file.Move into the inset modeType in text.Save data from buffer into the file without quitting.Navigate through the text in the file. (h,j,k,l)

Append data (within line; before current line; after current line)Delete lines.Overwrite data.Join Lines.

Undoing changes.u/U Latest change.[1-9]p If you want to undo changes (deletions) which were done earlier.

Quitting vi. (Temporary and Permanent):wq, ZZ, :x, :q! Permanent:sh Temporary

Block commands.

All executed in the ex command mode.Line numbers needed. (se nu, se nonu)Deletion :m,n dMove :m,n mo pCopy :m,n co pAll the above commands make changes within the same file.To copy lines to another file :m,n w filename:m,n w >> filename

- 2 -


3/12


Visual Editor (vi) (Part II): Advanced Features

SearchMechanism Command Function/pattern Searches forward?pattern Searches backwardN Repeats the last search commandN Repeats the last search command in the

backward direction.

Characters to build patterns Character Meaning* Zero or more characters

[] A set or range of characters$ End of line^ Beginning of line\< Beginning of word\> End of word. Matches any single character except the

newline character

How to build patterns?Pattern Meaning^part All lines which begin with the word partpart$ All lines which end with the word part\ All strings which end with the word part.e.g., depart

\ The whole word part[m-s]ore All strings which contain any character in the

range m tos and is followed by ore.[^p]art All strings which contain the characters art,

with art being preceded by any characterother than p.

Wing* Wing followed any sequence of characters

Note: If any special character is being searched then they must be preceded with abackslash (\)Search and ReplaceMechanism

Command Function:s/str1/str2 Replace first occurrence of str1 with str2 in

current line:s/str1/str2/g Replace all occurrences of str1 with str2 in

current line:m,n s/str1/str2/g Replace all occurrences of str1 with str2

from lines m to n.:1,$ s/str1/str2/g Replace all occurrences of str1 with str2

- 3 -


4/12


from 1st line to end of file.:1,. s/str1/str2/g Replace all occurrences of str1 with str2

from 1st line to current line.:.,$ s/str1/str2/g Replace all occurrences of str1 with str2

from current line to end of file.

abbrabbr command in vi stands for abbreviation. It can be used to enhance the usersconvenience.(Esc):abbr pr printf pr now becomes the shorthand for printf(Esc):abbr Display the entire list of macros you have set.(Esc):una pr The pr abbreviation is forgotten.

Set Commands and customizing the vi environment(Esc):set all Displays the list of all options available with set.

Some commonly used options Command Function:se nu Display of line numbers on:se eb Beep the speaker when an error occurs

:se ai Auto indent on:se ic Ignore case while searching a pattern:se showmode Display mode in which we are working

Whenever vi is invoked, it searches for the .exrc file in the users default workingdirectory ($HOME). This file is optional, i.e., vi does not give an error if it is not able tolocate this file. However, if the file is present any set commands or abbreviations that arestored in this file automatically become effective for all vi sessions.

Multiple file editing Command line options in vi

Command

Function Command Function

$vi file1

file2 file3

3 files are loaded into vi

buffer for editing

$vi + 10 file Loads file and places cursor

on the 10

th

line of the file:n Edit next file in buffer $vi +/pattern file Loads file and places cursoron first occurrence ofmatching pattern

:n! Edit next file withoutsaving current file

$view file Opens file in read-onlymode

:rew Edit 1st file in buffer:rew! Edit 1st file without

saving current file:args Display names of all

files in buffer, withcurrent file nameenclosed in []

:f Displays name of

current file

- 4 -


5/12


Shell Scripting: Introduction & Decision making statements

User defined and System defined variablesSystem variables are always accessible. The shell provides the values for these variables.These variables are usually used by the system itself and govern the environment wework under. If we so desire we can change the values of these variables as per ourpreferences and customize the system environment. For e.g.,PS1 : System Prompt 1.PATH : The path, which the shell must search in order to execute any command or file.HOME : The default working directory of the user.SHELL : Default working shell.MAIL : The file where the mail of the user is stored.

MAILCHECK : Duration after which the shell checks whether the user has received anymail.LOGNAME : Login name of the userThe value of the system variables can be displayed by typing the following command atthe prompt echo $.

User defined variables are used most extensively in shell programming. To access thevalue of any kind of variable, we need to precede the variable name with a dollar ($) sign.

1. All shell variables are string variables. We cannot carry out arithmetic calculations onthem unless we use a command called expr.

2. While using the assignment operator to assign values to variables, there should be nospaces on either side of =. During assignment, if the variable does not exist it will becreated.

3. If the variable needs to contain more than one word then the assignment should bemade using double quotes.

4. We can carry out more than one assignment in a line. So also we can echo more thanone variables value at a time.

5. All variables defined inside a script die the moment the execution of the script is over.6. A constant variable can be created by saying $>readonly .7. Variables can be made to cease existing by using the unset command. $>unset

. This command cannot be used on system variables.8. The entire user defined variables and system defined variables can be displayed by

using the command set at the command prompt.

Positional parameters (setting and shifting)The shell script uses positional parameters to access the command line arguments whichare passed to it. Positional parameters are 9 in number, named $1 through $9.The set command can be used to assign values to the positional parameters.$>set We are now trying to understand the setting of positional parameters withoutcalling a script$>echo $1 $2 $3 $4 $5 $6 $7 $8 $9$>echo Total number of arguments: $#$>echo All positional parameters: $@$>echo After shifting by 5 positions$>shift 5; echo 1 $2 $3 $4 $5 $6 $7 $8 $9

Exit status of a command

- 5 -


6/12


Every UNIX command/script returns an exit status of zero, if it has been successfulexecuted, otherwise, a value greater than zero is returned.$>echo $?$? stores the exit status of the last executed command.

Things to be noted while using arithmetic in shell scripts.$>echo `expr $a \* $b`

1. The multiplication symbol should be preceded by a \. Otherwise, the shell treats it as awildcard character for all files in the current directory.

2. Terms of the expression provided to expr must be separated by blanks.

3. expr is only capable of carrying out integer arithmetic. To carry out arithmetic on realnumbers it is necessary to use the bc command.

Numerical Test Operators

Operator Meaning-gt Greater than-lt Less than-ge Greater than or equal

to-le Less than or equal to

-ne Not equal to-eq Equal to

File Tests

Option TRUE if -s File exists and has size greater

than zero-f File exists and is not a directory-d File exists and is a directory-r File exists and user has read

permission on it-w File exists and user has write

permission on it

-x File exists and user hasexecute permission on it

String Tests

Operator TRUE if string1 = string2 Strings are same-n string Length of string > 0-z string Length of string = 0string String is not a null

string

Logical operators

Operator Meaning

-a AND-o OR! NOT

The Case Control Structure

- 6 -

case value inchoice 1)

....;;

choice 2)....;;

*)..;;

esac


7/12


Example Problems and Solutions.A. Code Snippets

Interaction with the user Command line arguments.

B. Program Example 1. The file /etc/passwd contains information about all users. However it is difficult to

decipher the information stored in it. Write a shell script which would receive thelogname during execution, obtain information about it from /etc/passwd and displaythis information on the screen in easily understandable format.

2. If the cost price and selling price of an item is input through the keyboard, write a

program to find out whether the seller has made profit or incurred loss. Alsodetermine how much profit was made or loss incurred.

Code Snippet 1: Interaction with theuser

script_1.ksh

Code Snippet 2: Command linearguments. script_2.ksh

Program Example 1. script_3.ksh


- 7 -


8/12


9/12


Code Snippet 1: Interaction with theuser

script_1.ksh


Shell Scripting: Miscellaneous Information

Difference between executing a script normally and with a .The preceding dot (.) while executing a script ensures that the script gets executed in thecurrent shell. A space between the . And the script name is mandatory. If it is absent theshell considers the script as a hidden file.

When do we need to export?If we want variables to be available to all sub-shells we must export them from the

current shell.$>export a. A variable once exported from the parent shell becomes available to the sub shell or

any other shells launched from this shell.b. If we are to unexport a variable, we have to first unset it and then recreate it.c. A variable can be exported from the parent shell to its sub shell, but never the other

way round.d. If the sub-shell changes the value of an exported variable the value of this variable in

the parent shell remains unchanged since the sub-shell always works only the copy ofthe variable in the parent shell.

e. If an exported variable is modified in a sub shell then to make this modified valueavailable to a sub-sub-shell we must once again export the variable in the sub-shell.

Trapping signals and Terminal settings.

A few examples on trappingtrap 123 Does nothing on generation of signal numbers 1,2 or 3trap ls;who 2Displays file listing and current user listing on generation of signal number2.

Terminal Settings

- 9 -

Signal Signal NumberExit 0Ctrl d 1Del 2Kill 15

Setting Actionecho Echoes back every character typed-echo Does not echo back every character typedsane Resets all modes to some reasonable values.

By trapping the program terminating signalsshown in the side table, we can use them toachieve some other task totally unconnected withtheir original duties, and hence ignore the taskthey were initially instructed to do.


10/12


Here documentThe HERE document signifies that the data is here instead of in a separate file. Anycommand using statndard input can also take input from a here document.Example (Executing Oracle procedures from Unix scripts)

# ---------------------------------------------------------------# Calling Database procedures from a UNIX script

# ---------------------------------------------------------------# Script Name call_sortbydisplayorder.sh# Usage sh call_sortbydisplayorder.sh# ---------------------------------------------------------------(sqlplus s user/password@hoststring


11/12


Shell Scripting: Coding Standards

Goals1. Make the least efforts when moving scripts from one environment to another.2. Make the least errors when testing scripts.3. Enable running the same version of scripts simultaneously in different environments

in the same machine (in case the test/dev/prod environments reside in the samemachine).

Standards1. Script naming & Coding style

A script name should have a suffix indicating the shell on which it has been designedto be executed (.sh/.ksh/.csh). The script name should have a mixture of cases andthere should be an avoidance of the use of underscore.

The following sections should be a part of the script in the sequence mentioned

a. Mandatory Shell specificationThe shebang line should be the first line of the code to indicate which shell shouldbe used to execute the script.#!/bin/ksh

b. Mandatory header sectionThis section should contain all details like the script name, created by, creation date,updated by, description, etc.Example:# -------------------------------------------------------------# Script name :# Created by :# Created on :

# Description :# -------------------------------------------------------------# Updated by :# Updated at :# Reason(s) for update # -------------------------------------------------------------

c. Mandatory environment setting To ensure smooth transition between the various environments available to you,initialize all commonly used variables in a single file (for ETL Services, the file is

- 11 -


12/12


.etl.env). After this is done, put the following line of code in your script before anyprocessing is done . /To differentiate between your various environments (especially if all of them lie on thesame machine), name the envFile in such a manner that it indicates the environmentto which it belongs and just change the above line of code while moving from oneenvironment to another.

Note:For any script that will not be directly called but will be called by a parent script, and ifit references any environment variables and its parent script has already invoked theenvironment variable file, then the child script does not have to re-invoke theenvironment variables file.

d. Mandatory usage check sectionExample:if [ $# -ne 2 ]then

echo "Improper Usage! Correct Usage: $0 "exit 1

fi

e. Optional variable definition sectionUse this section to replace all hard coded values which are used more than once inthe script.

f. Mandatory comments section for major code blocksEvery major code block should be preceded by an explanatory comment.

g. SQL FileScripts involving calling a SQL file should pass the parameter to the SQL file instead ofhard coding the value of the parameter in the SQL script.Example:Replace the statement below in the SQL file spool /wamu/informat/logFileswith spool &1and pass the value from the Unix script as parameter like below sqlplus user/passwd@connectString /wamu/informat/logFiles

2. Script log file naming

If a script generates a log file, the log file should be named according to the followingconvention:

a. The log file should have the script name (without scripts suffix) as the 1 st part of filename.

b. The log file should have a timestamp (YYYYMMDD:HH:MM:SS) as the 2 nd part of filename.

c. The log file should have .log suffix as the last part of the file name.

Example:Script name : MyScript.shServer Time : 2005/04/05 10:30:56Log File : MyScript20050405:10:30:56.log

3. Script migration

During migration of a script from development to production check the file forexecution privilege. File must have proper execution privilege.

- 12 -

unix scripting - a quick reference

Documents