14 december 2015 unix and shell scripting fundamentals cedric zoppolo [email protected]
TRANSCRIPT
2April 21, 2023
Knowledge resources• Bibliography
– UNIX Overview training– Tutorial y Guía Breve del Comando Sed para Unix y Linux
V0.2 - AmericaTI.com– Linux HowTo’s and manual pages [ linux’s man ]
• Resources in Internet– UNIX - Linux Bourne - Bash Shell Scripting Tutorial [ http://steve-parker.org ]– Wikipedia [ www.wikipedia.org ]
• Software used– Ubuntu Linux – Cygwin Linux-like environment for Windows [ www.cygwin.com ]
3April 21, 2023
Agenda• UNIX enviroment
– Components– Log in & shutdown– Users and Groups– File System– File system commands– Editing files (using vi)– sed and awk– Job control and process management
• Shell Scripting– Basic Shell Scripting (Hello world script)– Scripting philosophy– Variables ($# $1 …)– External programs– Functions – Escape Characters– Loops – for loops and while loops– Test – if statements– Case– Using awk and sed– Quick reference
April 21, 2023
UNIX environment
5April 21, 2023
Components• Kernel — source code in /usr/sys, composed of several sub-components:
– conf — configuration and machine-dependent parts, including boot code – dev — device drivers for control of hardware (and some pseudo-hardware) – sys — operating system "kernel", handling memory management, process
scheduling, system calls, etc. – h — header files, defining key structures within the system and important system-
specific invariables• Documentation — Unix was the first operating system to include all of its documentation
online in machine-readable form. The documentation included: – man — manual pages for each command, library component, system call, header file,
etc. – doc — longer documents detailing major subsystems, such as the C language and
troff
6April 21, 2023
Components• Commands —• Major categories are:
– sh — The "shell" programmable command-line interpreter, the primary user interface on Unix before window systems appeared, and even afterward (within a "command window").
– Utilities — the core tool kit of the Unix command set, including cp, ls, grep, find and many others. Subcategories include:
– System utilities — administrative tools such as mkfs, fsck, and many others – User utilities — environment management tools such as passwd, kill, and others. – Document formatting —nroff, troughs, tbl, eqn, refer, and pic. Modern Unix
systems also include packages such as TeX and GhostScript. – Graphics —. Modern Unix systems generally include X11 as a standard windowing
system and GUI, and many support OpenGL. – Communications — Early Unix systems contained no inter-system communication,
but did include the inter-user communication programs mail and write. V7 introduced the early inter-system communication system UUCP, and systems beginning with BSD release 4.1c included TCP/IP utilities.
7April 21, 2023
Log in and shutdown the systemCommand Function
halt Brings the system down immediately.
init 0 Powers off the system using predefined scripts to synchronize and clean up the system prior to shutdown. (Not available on all Unix systems.)
init 6 Reboots the system by shutting it down completely and then bringing it completely back up. (Not available on all systems.)
poweroff Shuts down the system by powering off.
reboot Reboots the system.
shutdown Shuts down the system.
8April 21, 2023
Users and Groups• There are three primary types of accounts on a Unix system: the root user (or superuser)
account, system accounts, and user accounts. Almost all accounts fall into one of those categories.
• Root account– The root account's user has complete and unfettered control of the system.The root
user (also called root) can do absolutely anything on the system, with no restrictions on files that can be accessed, removed, and modified
• System accounts– System accounts are those needed for the operation of system-specific components.
They include, for example, the mail account (for electronic mail functions) and the sshd account (for ssh functionality). System accounts are generally provided by the operating system during installation. They generally assist in the running of services or programs that the users require.
9April 21, 2023
Users and Groups• Users accounts
– User accounts provide interactive access to the system for users and groups of users. General users are typically assigned to these accounts and usually have limited access to critical system files and directories.
• Groups accounts– Group accounts add the capability to assemble other accounts into logical
arrangements for simplification of privilege (permission) management.– One of the strengths of groups is that an account can belong to many groups, based
on access requirements.• There are three main user administration files:
– /etc/passwd — Identifies the authorized accounts for the system.– /etc/shadow — Holds the encrypted password of the corresponding account. Most
Unix systems have this file.– /etc/group — Contains information on group accounts.
10April 21, 2023
Users and Groups• Commands
• Manually
– Modify /etc/passwd to add or remove the account line.
– Modify /etc/shadow to add or remove the account line.
– Modify /etc/group to add or remove the account references.
– Add or remove the account's home directory (if not shared, which by default it should not be).
Command Description
useradd Adds accounts to the system.
usermod Modifies account attributes.
userdel Deletes accounts from the system.
groupadd Adds groups to the system.
groupmod Modifies group attributes.
groupdel Removes groups from the system.
11April 21, 2023
Users and Groups• Becoming another user
– If you are logged in as jdoe, for example, and want to log in as jsmith, type:
su jsmith
– When using su, you continue to use your own environment variables and profile. If you want to use the account's user environment, put a dash (-) between the su and the account name:
su - jsmith
•who,w and whoami•groups command
12April 21, 2023
File system• A file system is a logical collection of files on a partition or disk. A partition is a container
for information and can span an entire hard drive if desired. An apple pie, for example, can be eaten whole or it can be cut into slices, which is similar to how a hard drive or other physical storage device can be manipulated. A slice of pie corresponds to a partition on a drive, and the whole pie could represent a single partition that takes up a whole disk.
13April 21, 2023
Directory Description
/ Root should contain only the directories needed at the top level of the file structure (or that come already installed in it). Unnecessary subdirectories under root can clutter your system, making administration more difficult and, depending on the system, filling up the space allocated for /.
bin Usually contains binary (executable) files critical for system use, and often contains essential system programs, such as vi (for editing files), passwd (for changing passwords), and sh (the Bourne shell).
boot Contains files for booting the system.
devdevices
Either or both of these will exist. They contain device files, often including cdrom(CD-ROM drive), eth0 (Ethernet interface), and fd0 (floppy drive). (The devices are often named differently in the different Unix systems.)
etc Contains system configuration files such as passwd (holds user account information and is not to be confused with /bin/passwd); hosts (contains information about host resolution); and shadow (contains encrypted passwords).
export Often contains remote file systems (those external to the physical system), such as home directories exported from another system to save space and centralize home directories.
home Contains the home directory for users and other accounts (specified in /etc/passwd, for example).
kernel Contains kernel files.
lib Contains shared library files and sometimes other kernel-related files.
mnt Used to mount other temporary file systems, such as cdrom and floppy for the CD-ROM drive and floppy diskette drive, respectively.
proc Contains all processes marked as a file by process number or other information that is dynamic to the system.
sbin Contains binary (executable) files, usually for system administration. Examples include fdisk (for partitioning physical disks) and ifconfig (for configuring network interfaces).
tmp Holds temporary files used between system boots (some Unix systems do not delete the contents of the tmp directory between boots).
usr Used for miscellaneous purposes, or can be used by many users (such as for man pages). Can include administrative commands, shared files, library files, and others.
var Typically contains variable-length files such as log and print files and any other type of file that may contain a variable amount of data. For instance, the log files (typically in /var/log) range in size from very small to very large, depending on the system configuration.
14April 21, 2023
File system commandsCommand Description
cat Concatenate: displays a file.
cd Change directory: moves you to the directory identified.
cp Copy: copies one file/directory to specified location. Option –R (recirsive) to copy directories
file Identifies the file type (binary, text, etc).
find Finds a file/directory.
head Shows the beginning of a file.
less Browses through a file from end or beginning.
ls List: shows the contents of the directory specified.
mkdir Make directory: creates the specified directory.
more Browses through a file from beginning to end.
mv Move: moves the location of or renames a file/directory.
pwd Print working directory: shows the current directory the user is in.
rm Remove: removes a file.
rmdir Remove directory: removes a directory.
tail Shows the end of a file. With option –f follows changes in files
touch Creates a blank file or modifies an existing file's attributes.
whereis Shows the location of a file.
which Shows the location of a file if it is in your PATH.
15April 21, 2023
File types
File Type Description
- Regular file, such as an ASCII text file, binary executable, or hard link (links are discussed in the following section)
b Block special file (block input/output device file used for transferring data from or to a device such as a physical hard drive)
c Character special file (raw input/output device file used for transferring data from or to a device such as a physical hard drive)
d Directory file (file that contains a listing of other files and/or directories contained within the directory)
l Symbolic link file (discussed in the following section)
p Named pipe (a mechanism for interprocess communications)
s Socket (used for interprocess communication)
16April 21, 2023
File and directory permissions
Permission Applied to a Directory Applied to Any Other Type of File
read (r) Grants the capability to readthe contents of the directory or subdirectories.
Grants the capability to view the file.
write (w) Grants the capability to create, modify, or remove files or subdirectories.
Grants write permissions, allowing an authorized entity to modify the file, such as by adding text to a text file, or deleting the file.
execute (x) Grants the capability to enter the directory. Allows the user to "run" the program.
- No permission. No permission.
• Example– $ ls -l /home/mikec
-rwxr-xr-- 1 mikec users 1024 Nov 2 00:10 myfile drwxr-xr--- 1 mikec users 1024 Nov 2 00:10 mydir
17April 21, 2023
File and directory permissionsCharacters Apply to Definition
rwx (characters 2–4) The owner (known as user in Unix) of the file
The owner of the file (mikec) has read (or view), write, and execute permission to the file.
r-x (characters 5–7) The group to which the file belongs
The users in the owning group (users) can read the file and execute the file if it has executable components (commands, and so forth). The group does not have write permission — notice that the - character fills the space of a denied permission.
r-- (characters 8–10) Everyone else (others)
Anyone else with a valid login to the system can only read the file — write and execute permissions are denied (--).
18April 21, 2023
Changing permissions•chmod command
– With symbolic permissions you can add, delete, or specify the permission set you want by using the operators in the following table. The example file, testfile, has original permissions of rwxrwxr--.
•Chown command
– This command lets you change the owner of the file / directory
chmod operator
Meaning Example Result
+ Adds the designated permission(s) to a file or directory.
chmod o+wx testfile
Adds write and execute permissions for others (permission character set 9–10) on testfile.
- Removes the designated permission(s) from a file or directory.
chmod u-x testfile
Removes the file owner's capability to execute testfile (u = user or owner).
= Sets the designated permission(s).
chmod g=r-x testfile
Sets permissions for the group to read and execute on testfile (no write).
19April 21, 2023
Changing permissionsNumber Binary Permission
Reference Octal Permission Representation
0 000 --- No permission
1 001 ---x Execute permission
2 010 -w- Write permission
3 011 -wx Execute and write permission: 1 (execute) + 2 (write) = 3
4 100 r-- Read permission
5 101 r-x Read and execute permission: 4 (read) + 1 (execute) = 5
6 110 rw- Read and write permission: 4 (read) + 2 (write) = 6
7 111 rwx All permissions: 4 (read) + 2 (write) + 1 (execute) = 7
20April 21, 2023
File System• The du (disk usage) command enables you to specify directories to
show disk space usage on a particular directory– $ du /etc
10 /etc/cron.d 126 /etc/default 6 /etc/dfs
• Options –ks are useful to show a summary in kB of the usage
Column Description
Filesystem The physical file system (fdX (X=floppy drive number) = floppy drive, /dev/dsk/c0t0d0s0 represents a partition on a disk drive, and so forth).
kbytes Total kilobytes of space available on the storage medium.
used Total kilobytes of space used (by files).
avail Total kilobytes available for use.
capacity Percentage of total space used by files.
Mounted on What the file system is mounted on. In Figure 4-6, the / (root) file system is mounted on /dev/dsk/c0d0s0 and has only 26% of its total allocated space available. Mounts are discussed later in the chapter.
21April 21, 2023
Understanding Path• The PATH environment variable contains a list of directories where executable files might
be located. If a directory is listed in the PATH variable's value, that directory's name does not need to be typed to invoke an executable file that resides in it. For example, if the mail program is stored in /usr/bin/ and /usr/bin/ is part of PATH's value, you can simply type mail at the command prompt to invoke the program, instead of using the complete path /usr/bin/mail.
• Multiple directories can be added as a colon-separated list:PATH=$PATH:/home/joe:/home/joe/bin
• You can edit the PATH variable in your .profile shell file
• Remember that ~/ is the user’s home directory
22April 21, 2023
Help on commands•man
– Unix manual pages (or man pages) are the best way to learn about any given command. Find a particular manual page with the command man commandname
•apropos– Looking for a specific file or a manual page that doesn't come up when you think
you've typed the right command name? The apropos command may be useful to you. It is used with keywords to find related files.
•info– Some programmers include an additional set of help documents, known as info
pages, with their packages or programs.
23April 21, 2023
Editing files• There are many ways to edit files in Unix, including the use of early line-based text editors
such as ed (EDitor) and ex (EXtended) and the use of screen-oriented text editors such as Emacs (Editor MACroS) and vi (VIsual editor). In Linux there is an improved version of vi called vim.
•vi is generally considered the “de facto” standard in Unix editors because:– It's usually available on any Unix system. Once you have learned how to use it, there
is no learning curve when you're working on another Unix system, which allows for faster recovery or adjustments of files.
– Vi implementations are very similar across the board (from Linux, to Mac OS X, to Sun Solaris, and so on).
– It requires very few resources.– It is more user friendly than ed or ex.
24April 21, 2023
Editing files • There are two modes in vi
– Command mode — Enables you to perform administrative tasks such as saving files, executing commands, moving the cursor, cutting (yanking) and pasting lines or words, and finding and replacing. In this mode, anything that's typed — even a single character — is interpreted as a command.
– Insert mode — Enables you to insert text into the file. Everything that's typed in this mode is interpreted as input.
Command Description Results
vi Start vi without any filename arguments.
Vi starts with an empty palette. You must save to a new file before exiting.
vi filename • Use an existing filename as the argument.
• Use a new filename as the argument (file doesn't exist until saved in vi).
• The existing file opens in vi. When you save it, the file is updated with the changes you make.
• When you save this file, you create a new file with the filename specified in the argument.
vi -R filename or view filename
Open the file in read-only mode. The file is read-only (changes can't be saved); great for practicing your vi commands on a file.
25April 21, 2023
Moving in VICommand Description Result
h Lowercase h left (also: CTRL-H, <BS>, or <Left> key)
j Uppercase j down (also: CTRL-J, <Down>)
k Lowercase k up (also: CTRL-P and <Up>)
l Uppercase l right (also: <Space> or <Right> key)
1G 1 Uppercase G Goes to line number 1
G Uppercase G Goes to last line of file
26April 21, 2023
Editing and deleting in VICommand Description Result
i Lowercase i Inserts text before current cursor location
I Uppercase I Inserts text at beginning of current line
a Lowercase a Inserts text after current cursor location
A Uppercase A Inserts text at end of current line
o Lowercase o Creates a new line for text entry below cursor location
O Uppercase O Creates a new line for text entry above cursor location
27April 21, 2023
Editing and deleting in VICommand Description Result
r Lowercase r Replaces current character with the next character typed
cw Lowercase cw Replaces next word with the characters typed afterwards. Press ESQ to finish.Instead of cw it could be cw$ (chnge till end of line) or c2w (change 2 words), etc
x Lowercase x Deletes the character under the cursor location (or starting with the character under the cursor if the command is preceded by a number)
X Uppercase X Deletes the character before the cursor location (or starting with the cursor if the command is preceded by a number)
dw Lowercase dw Deletes from the current cursor location to the next word (or multiple words if preceded by a number). E.g. d2w would delete 2 words
D Uppercase D Deletes from the cursor position to the end of the current line
dd Lowercase dd Deletes the line the cursor is on
28April 21, 2023
Searching in VI (in no edit mode)Command Result
/word Searches for next “word” in text
?word Searches for previuos “word” in text
n Repeats last search
N Repeats last search in reverse way
* Searches for next word corresponding to the current one over the cursor
29April 21, 2023
Sed• The sed utility reads the specified files and/or the standard input and modifies the input as directed by
a list of commands. The input is then written to the standard output, which can be redirected if needed.
• In this example, the sed command is searching for all instances of the name Paul in the two files provided in the command-line argument and replacing them with the name Pablo. After the search and replace has been completed, the output is redirected from standard out to a new file called output.txt. Notice the trailing g in the command, s/Paul/Pablo/g:
– $ sed s/Paul/Pablo/g input1.txt input2.txt > output.txt
This specifies that sed should look globally. Without that trailing g, if the name Paul happened to be on the same line twice, only the first would be substituted.
Note that while only one line from each file was affected by substitution, all the lines from both files are displayed, in the order they are processed, in the output from sed. The original files are unchanged (i.e. imput1.txt and input2.txt); only the output, or in this example the file created from the output (output.txt), contains the substitution of Pablo for Paul.
30April 21, 2023
AWK•awk takes two inputs: a command, set of commands, or a command file and a data or
data file. As with sed the command or command file contains pattern-matching instructions for which AWK is to use as a guideline for processing the data or data file.
wc -l test.txt | awk '{ print $1 }'
• In the example the first data retrieved from the wc command output is retrieved
$ wc -l test.txt
6 test.txt
• In some cases cut can also be used for such purposes. For this example you could do:
$ wc -l test.txt | cut -f1 -d' '
6
31April 21, 2023
Job control and process management• A process, in simple terms, is an instance of a running program.
• Behind the scenes, a fork library call or an execve system call is used to start the new program. A fork is produced when the current running program is copied to make a child, an exact copy of the running program. The forked program has a new PID and a different parent process ID (of course), and the child's resource utilizations are all reset. For example, by default, the forked child and its parent share file descriptors and can share open files.
• In contrast to forking a process, you can replace the current running process with a new process. The Unix shell includes a built-in command called exec that replaces the running shell with a new program.
32April 21, 2023
Job control and process management• Process running
– ps command is used to view the process running
– ps –ef is used to see all the process.
– Each process has an environment with various attributes such as command-line arguments, user environment variables, file descriptors, working directory, file creation mask, controlling terminal (console), resource limitations, and a lot more. Many of the attributes are shared with the parent process.
Field Definition
user Effective user ID of the process
pid Process ID
ppid Process ID of the parent
pcpu Percentage of CPU time used
rss Real memory size in kilobytes
pmem Percentage of rss to physical memory
vsz Kilobytes of the process in virtual memory
tty Controlling terminal name
state (or s) Process state
stime Time started
time Accumulated user and system CPU time
command (or comm)
Command name
33April 21, 2023
Job control and process management•kill command
– Technically, the kill command does not kill a command, but sends a special signal to the process. Signals are used for simple communication between processes.
– The default signal for the kill command is SIGTERM (for terminate). It is possible that the software you're trying to stop is written to cleanly back up files or close down its work when it receives this TERM signal, but don't count on it
– To list the possible signal names, run kill with the -l switch (also check the man for kill):$ kill -l
– To kill a process you have to know the PID. See– A small script to kill a known application could be
killps
#!/bin/shps -ef | grep $1 | grep -v grep | awk '{print $2}' | kill
You would call it using killps app_name Be careful this may kill any process with the name app_name within its name
34April 21, 2023
Job control and process management•pstree
– A process tree displays the lineage of your different processes, placing a child process with its parent. Note that only one parent process exists per child process but that each parent can have multiple children.
$ pstree
init-+3*[agetty]
|-events/0-+aio/0
| |-ata/0
| |-kacpid
| |-kblockd/0
| |-khelper
| `-2*[pdflush]
|-gconfd-2
|-kdeinit-+artsd
| |-firefox-bin---firefox-bin---3*[firefox-bin]
| |-3*[kdeinit]
| `-kdeinit---sh---ssh
|-8*[kdeinit]
|-khpsbpkt
35April 21, 2023
Job control and process management•top command
– The top command is a very useful tool for quickly showing processes sorted by various criteria. It is an interactive diagnostic tool that updates frequently and shows information about physical and virtual memory, CPU usage, load averages, and your busy processes.
load averages: 0.19, 0.24, 0.12; up 3+19:01:58 06:14:02
37 processes: 36 sleeping, 1 on cpu
CPU states: 99.7% idle, 0.0% user, 0.2% kernel, 0.0% iowait, 0.0% swap
Memory: 512M real, 381M free, 26M swap in use, 859M swap free
PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
1 root 1 59 0 1232K 368K sleep 0:17 0.00% init
358 root 1 59 0 4520K 2696K sleep 0:13 0.00% sshd
307 root 2 59 0 2648K 2000K sleep 0:01 0.00% vold
366 jreed 1 59 0 2672K 1984K sleep 0:01 0.00% bash
16492 jreed 1 59 0 1760K 1184K cpu 0:00 0.00% top
318 root 1 59 0 122M 11M sleep 0:00 0.00% Xsun
335 root 1 59 0 7912K 4864K sleep 0:00 0.00% dtgreet
36April 21, 2023
Run programs at specified times• To put the system clock to good use and to automate tasks, all Unix systems provide two
key tools — cron and at — for running commands, applications, and scripts at specified times.
•crontab command– crontab –l– FILES
•/etc/cron.d main cron directory•/etc/cron.d/FIFO used as a lock file•/etc/default/cron contains cron default settings•/var/cron/log cron history information•/etc/cron.d/logchecker moves log file to /var/cron/olog
if log file exceeds system ulimit.•/etc/cron.d/queuedefs queue description file for at, batch, and cron.
37April 21, 2023
Run programs at specified times - Crontab example•# Use the hash sign to prefix a comment# +---------------- minute (0 - 59)# | +------------- hour (0 - 23)# | | +---------- day of month (1 - 31)# | | | +------- month (1 - 12)# | | | | +---- day of week (0 - 7) (Sunday=0 or 7)# | | | | |# * * * * * command to be executed
•0,20,40 8-17 * * 1-5 /usr/lib/sa/sa1 1200 3 & 0 * * * 0,6 /usr/lib/sa/sa1 & 0 18-7 * * 1-5 /usr/lib/sa/sa1 & 5 18 * * 1-5 /usr/lib/sa/sa2 -s 8:00 -e 18:01 -i 3600 -ubcwyaqvm &
April 21, 2023
Shell Scripting
39April 21, 2023
Basic Shell Scripting• Shell programs are called scripts• A script is compiled on the fly, at running time• We have some tools as in a programming language
– If-then statement– Case statement– While and until statement– Functions
• Example
#!/bin/bash
pwd
ls
40April 21, 2023
Basic Shell Scripting - Hello world script• Code segments and script output will be displayed as preformatted text.
Command-line entries will be preceded by the Dollar sign ($). If your prompt is different, enter the command:
PS1="$ " ; export PS1
• Then your interactions should match the examples given (such as $ ./my-script.sh below). Script output (such as "Hello World" below) is displayed at the start of the line.
$ echo '#!/bin/sh' > my-script.sh$ echo 'echo Hello World' >> my-script.sh$ chmod 755 my-script.sh$ ./my-script.shHello World$
41April 21, 2023
Basic Shell Scripting - Hello world script• The first script hello world could be then:
first.sh
#!/bin/sh# This is a comment!echo Hello World
• Remember to add read and execute permissions to the script file. Note that to make a file executable, you must set the eXecutable bit, and for a shell script, the Readable bit must also be set:
$ chmod a+rx first.sh
42April 21, 2023
Scripting philosophy• The script must have a clear, readable layout.
• Avoid unnecessary commands. One of the major weaknesses in many shell scripts is lines such as:cat /tmp/myfile | grep "mystring"which would run much faster as:grep "mystring" /tmp/myfile
• indentation is critical for understanding what a script does.
• If you supply a customer with a shell script, s/he can inspect it quite easily. So you might as well accept that it will be inspected by anyone you pass it to
43April 21, 2023
Variables• variables - a symbolic name for a chunk of memory to which we can assign values, read
and manipulate its contents. • Note that there must be no spaces around the "=" sign: VAR=value works; VAR = value
doesn't work.
var.sh
#!/bin/shMY_MESSAGE="Hello World"echo $MY_MESSAGE
• The shell does not care about types of variables; they may store strings, integers, real numbers - anything you like.
• If you assign a string to a variable then try to add 1 to it, you will not get away with it: $ x="hello"$ y=`expr $x + 1`expr: non-numeric argument$
44April 21, 2023
Variables• We can interactively set variable names using the read command; the following script asks
you for your name then greets you personally:
var2.sh
#!/bin/shecho What is your name?read MY_NAMEecho "Hello $MY_NAME - hope you're well."
• In order to receive environment changes back from the script, we must source the script - this effectively runs the script within our own interactive shell, instead of spawning another shell to run it. We can source a script via the “ . “ Another way to achieve this is using the export command
45April 21, 2023
Variablesuser.sh
#!/bin/shecho "What is your name?" read USER_NAME
echo "Hello $USER_NAME"echo "I will create you a file called ${USER_NAME}_file"touch "${USER_NAME}_file"
46April 21, 2023
Variables - $# $1 …• There are a set of variables which are set for you already, and most of these cannot have
values assigned to them.
• These can contain useful information, which can be used by the script to know about the environment in which it is running.
• The first set of variables we will look at are $0 .. $9 and $#. – The variable $0 is the basename of the program as it was called.– $1 .. $9 are the first 9 additional parameters the script was called with.– The variable $@ is all parameters $1 .. whatever. – The variable $*, is similar, but does not preserve any whitespace, and quoting, so
"File with spaces" becomes "File" "with" "spaces". – $# is the number of parameters the script was called with. – $?. This contains the exit value of the last run command
47April 21, 2023
Variables - $# $1var3.sh
#!/bin/shecho "I was called with $# parameters"echo "My name is $0" echo "My first parameter is $1"echo "My second parameter is $2"echo "All parameters are $@"
48April 21, 2023
Variables - $# $1• Another interesting variable is IFS. This is the Internal Field Separator. The default value is
SPACE TAB NEWLINE, but if you are changing it, it's easier to take a copy, as shown:
var5.sh
#!/bin/shold_IFS="$IFS"IFS=:echo "Please input three data separated by colons ..."read x y zIFS=$old_IFSecho "x is $x y is $y z is $z“
This script runs like this:$ ./ifs.shPlease input some data separated by colons ...hello:how are you:todayx is hello y is how are you z is today
49April 21, 2023
Variables – Using default values• By using curly braces and the special ":-" usage, you can specify a default value to use if
the variable is unset:
#!/bin/sh
echo -en "What is your name [ `whoami` ] "
read myname
echo "Your name is : ${myname:-`whoami`}"
• There is another syntax, ":=", which sets the variable to the default if it is undefined:
echo "Your name is : ${myname:=John Doe}"
This technique means that any subsequent access to the $myname variable will always get a value, either entered by the user, or "John Doe" otherwise.
50April 21, 2023
External programs• External programs are often used within shell scripts; there are a few builtin commands
(echo, which, and test are commonly builtin), but many useful commands are actually Unix utilities, such as tr, grep, expr and cut.
• The backtick (`) is used to indicate that the enclosed text is to be executed as a command.
$ MYNAME=`grep "^${USER}:" /etc/passwd | cut -d: -f5`$ echo $MYNAME
51April 21, 2023
Functions• A function may return a value in three ways:
– Change the state of a variable or variables – Use the exit command to end the shell script – Use the return command to end the function, and return the supplied value to the
calling section of the shell script
52April 21, 2023
FunctionsFactorial.sh
#!/bin/sh
factorial(){ if [ "$1" -gt "1" ]; then i=`expr $1 - 1` j=`factorial $i` k=`expr $1 \* $j` echo $k else echo 1 fi}
while :do echo "Enter a number:" read x factorial $xdone
53April 21, 2023
Escape Characters • Certain characters are significant to the shell; we have seen, for example, that the use of
double quotes (") characters affect how spaces and TAB characters are treated, for example:
$ echo Hello WorldHello World$ echo "Hello World"Hello World
• So how do we display: Hello "World" ?
$ echo "Hello \"World\""
•", $, `, and \ are still interpreted by the shell, even when they're in double quotes.
54April 21, 2023
Loops – for loops• for loops iterate through a set of values until the list is exhausted:
for.sh
#!/bin/shfor i in 1 2 3 4 5do echo "Looping ... number $i" done
for2.sh
#!/bin/shfor i in hello 1 * 2 goodbye do echo "Looping ... i is set to $i" done
• Other way to make a loop under bash
#!/bin/bash for ((i=1;i<=10;i++)); do echo $i; done
55April 21, 2023
Loops – while loops• while.sh
#!/bin/shINPUT_STRING=hellowhile [ "$INPUT_STRING" != "bye" ]do echo "Please type something in (bye to quit)" read INPUT_STRING echo "You typed: $INPUT_STRING"done
• The colon (:) always evaluates to true; whilst using this can be necessary sometimes, it is often preferrable to use a real exit condition. i.e. use instead of prevoius while while :
See which is the more elegant. Also think of some situations in which each one would be more useful than the other:
56April 21, 2023
Loops – while loops• Two examples on how to do a while loop
i=0;while [ $i -lt 10 ]; do echo $i; i=`expr $i + 1`;done
i=0;while [ $i -lt 10 ]; do echo $i; let i=i+1;done
57April 21, 2023
Test•test is used by virtually every shell script written. It may not seem that way, because test is not often
called directly. test is more frequently called as [. [ is a symbolic link to test, just to make shell programs more readable.
$ type [
[ is a shell builtin
$ which [
/usr/bin/[
$ ls -l /usr/bin/[
lrwxrwxrwx 1 root root 4 Mar 27 2000 /usr/bin/[ -> test
• This means that '[' is actually a program, just like ls and other programs, so it must be surrounded by spaces: put spaces around all your operators; if there isn't a space there, it won't work:
if [ "$foo" == "bar“ ]
• Test is a simple but powerful comparison utility. For full details, run man test on your system, but here are some usages and typical examples.
58April 21, 2023
Test• The syntax for if...then...else... is:if [ ... ]then # if-codeelse # else-codefi
• Be aware of the syntax - the "if [ ... ]" and the "then" commands must be on different lines. Alternatively, the semicolon ";" can seperate them:if [ ... ]; then # do somethingfi
• You can also use the elif, like this:if [ something ]; then echo "Something" elif [ something_else ]; then echo "Something else" else echo "None of the above"fi
59April 21, 2023
Testtest.sh
#!/bin/shif [ "$1" -lt "0" ]then echo "$1 is less than zero"fiif [ "$1" -gt "0" ]; then echo "$1 is more than zero"fi[ "$1" -le "0" ] && \ echo "$1 is less than or equal to zero"[ "$1" -ge "0" ] && \ echo "$1 is more than or equal to zero"[ "$1" = "0" ] && \ echo "$1 is the string or number \"0\""[ "$1" = "hello" ] && \ echo "$1 matches the string \"hello\""[ "$1" != "hello" ] && \ echo "$1 is not the string \"hello\""[ -n "$1" ] && \ echo "$1 is of nonzero length"[ -f "$1" ] && \ echo "$1 is the path of a real file" || \ echo "No such file: $1"[ -x "$1" ] && \ echo "$1 is the path of an executable file"[ "$1" -nt "/etc/passwd" ] && \ echo "$1 is a file which is newer than /etc/passwd"
60April 21, 2023
Case• The case statement saves going through a whole set of if .. then .. else statements. Its syntax is really quite simple:
talk.sh
#!/bin/shecho "Please talk to me ..."while :do read INPUT_STRING case $INPUT_STRING in hello) echo "Hello yourself!" ;; bye) echo "See you again!" break ;; *) echo "Sorry, I don't understand" ;; esacdoneecho echo "That's all folks!"
61April 21, 2023
Using awk and sed• For some things the shell just isn't very good at. Two useful tools are sed and awk.
Consider wc, which counts the number of characters, lines, and words in a text file. Its output is:
$ wc hex2env.c 102 189 2306 hex2env.c
• If we want to get the number of lines into a variable, simply using:NO_LINES=`wc -l file`which would read in the whole line.
• Because the output is space-padded, we can't reliably get the number 102 into the string. Instead, we use the fact that awk strips unwanted whitespace. It puts these into variables $1 $2 $3 etc. So we use this contstruct:
NO_LINES=`wc -l file | awk '{ print $1 }'`
The variable NO_LINES is now 102.
62April 21, 2023
Using awk and sed• Another handy utility is sed - the stream editor. Perl is very good at dealing with regular
expressions, the shell isn't. So we can quickly use the s/from/to/g construct by invoking sed. For example:
sed s/eth0/eth1/g file1 > file2
changes every instance of eth0 in file1 to eth1 in file2.
• If we were only changing a single character, tr would be the tool to use, being smaller and therefore faster to load.
• Another thing that tr can't do, is remove characters from a file:echo ${SOMETHING} | sed s/"bad word"//g
63April 21, 2023
Quick reference Cmd Description Example• & Run the previous
command in the background ls &• && Logical AND if [ "$foo" -ge "0" ] && [ "$foo" -le "9"]• || Logical OR (not in Bourne shell) if [ "$foo" -lt "0" ] || [ "$foo" -gt "9" ]• ^ Start of line grep "^foo"• $ End of line grep "foo$"• = String equality (cf. -eq) if [ "$foo" = "bar" ]• ! Logical NOT if [ "$foo" != "bar" ]• $$ PID of current shell echo "my PID = $$"• $! PID of last background command ls & echo "PID of ls = $!"• $? exit status of last command ls ; echo "ls returned code $?"• $0 Name of current command echo "I am $0"• $1 Name of current command's
first parameter echo "My first argument is $1"• $@ All of current command's
parameters (preserving whitespace and quoting) echo "My arguments are $@"
• $* All of current command's parameters (not preserving whitespace and quoting) echo "My arguments are $*"
64April 21, 2023
Quick reference Cmd Description Example
• -eq Numeric Equality if [ "$foo" -eq "9" ]• -ne Numeric Inquality if [ "$foo" -ne "9" ]• -lt Less Than if [ "$foo" -lt "9" ]• -le Less Than or Equal if [ "$foo" -le "9" ]• -gt Greater Than if [ "$foo" -gt "9" ]• -ge Greater Than or Equal if [ "$foo" -ge "9" ]• -z String is zero length if [ -z "$foo" ]• -n String is not zero length if [ -n "$foo" ]• -nt Newer Than if [ "$file1" -nt "$file2" ]• -d Is a Directory if [ -d /bin ]• -f Is a File if [ -f /bin/ls ]• -r Is a readable file if [ -r /bin/ls ]• -w Is a writable file if [ -w /bin/ls ]• -x Is an executable file if [ -x /bin/ls ]• parenthesis:
( ... ) Function definition function myfunc() { echo hello }
65April 21, 2023
Thank you!Questions … Demos …