system integration in free software

Master on Free Software

Systems Integration on Free Software

Enrique Ocaña González


Index (1/4)

● Administrative tasks– System startup– Users & groups– Task automating– Auditing– Backups


Index (2/4)

● Basic commands– Files & directories– Process control– Documentation– Redirections & pipes


Index (3/4)

● Script programming– Environment variables– Input/output– Command grouping– Wildcards– Quoting– Control structures


Index (4/4)

● Advanced commands– Regular expressions– Grep– Sed– Awk


Administrativetasks


Index

● System startup● Users & groups● Task automating● Auditing● Backups


Introduction

● Administration: security– Two roles

● User: Not concerned about administrative tasks● Administrator (root): Responsible for the proper

operation of the system

– System designed thinking in program and user data separation

– Advice: Only work as root when needed● Avoids unintended damage of the system


Introduction

● Administration: transparency– Config and utilities based on plain text files

and scripts– Looking at the configuration, reading

documentation and following what is done by the scripts, it's possible to understand what the system does

– The root of a problem can be found in a simpler way than in closed systems, where the internal system operation isn't known


Introduction

● Administration: support– User community at internet– Specialized companies


Introduction

● Steps to solve a problem– Read documentation

● /usr/share/doc/package● Project website● FAQs and howtos

– Upgrade the program to the latest version (maybe the bugs have been solved)

– Get in touch with the developers– Return the favour colaborating in projects ;-)


Index



System startup

● System startup sequence:

1.BIOS load, POST tasks, boot sequence selection

2.Boot sector load (MBR)

3.Boot loader: grub, lilo, ntldr

4.Kernel load: /boot/vmlinuz

5. init program execution

6.System tasks: /etc/init.d/rcS

7.Runlevel specific tasks: /etc/rcN.d


System shutdown

● System shutdown– Init process receives a state change signal

that lets stopping (runlevel 0) or restarting (runlevel 6) the system

● shutdown● halt● init

– Executes the actions associated to the new runlevel in a stablished order


System shutdown

● System shutdown stages:

1. Shutdown warning to users and services

2. Execution of actions associated to the system shutdown (runlevel change)

3. Proper filesystem unmounting

4. Computer power off or reset


Init and runlevels

● Init process:– Father or all the processes (PID 1)– Has different states or runlevels with services

and actions associated, to be executed when entering into the runlevel or leaving it

– The most common runlevels are:● 0 System shutdown● 6 System restart● 1 Single user mode● 2-5 Multiuser (2 by default)


Init and runlevels

● Configuration files:– Master init configuration: /etc/event.d

● Used to be at: /etc/inittab

– Real init scripts placed at /etc/init.d – System bootstrap init links: /etc/rcS.d– Runlevel links: /etc/rcN.d (N=0...6)


Init and runlevels

● Symbolic links format:– S|K + order + service name

● Eg. at rc2.d: S89cron > ../init.d/cron● S Start the service when entering the runlevel● K Kill the service when entering the runlevel

● Manual startup:– /etc/init.d/service {start|stop|restart|reload|forcereload}

– /etc/init.d/service shows help


Boot manager

● Common boot managers:– lilo– grub

iAdditional documentation: boot_loader_config.pdf


Index



Users & groups

● root: the user with the highest privileges● Groups allow granting and grouping

privileges of a set of users● UID, GID: Unique numerical identifiers

– root has UID 0

● Base directory or “home”– /home/user

– /root


Users & groups

● Files involved:– /etc/passwd: Information about the user

accounts● Login● Password

– Crypted, “x” when using shadow passwords, “!” when the account is disabled

● UID● Home directory path● Shell● Name, surname, other identification info


Users & groups

● Files involved:– /etc/shadow: Like passwd, but only readable

by root and meant to store the passwords

– /etc/group: Info about groups and which users belong to each one

– /etc/gshadow: Non public info about the groups


User management

● User suplantationsu user

● User task suplantation (“run as”)sudo user command

● Suplantation access control– File /etc/sudoers (don't touch!)

– visudo


User management

● User creation– adduser user

● Creation process

1. Entry in passwd file

2. Entry in shadow file

3. Creation of /home/user

4. Copy of the files in /etc/skel

5. Password setting


User management

● User deletion– deluser user

● Creation process

1. Delete entries in passwd and shadow files

2. Finds all the files outside the home directory and deletes them, making a security copy


Group management

● Adding a group– addgroup group

● Deleting a group– delgroup group

● Process: Adds/deletes the entries in the files /etc/group and /etc/gshadow


Group management

● Adding a user to an existing group– adduser user group

● Deleting a group– adduser user group

● Process: Adds/deletes the entries in the files /etc/group and /etc/gshadow


Password changing

● Change the password of a user– passwd Changes the password of the

current user

– passwd user group Executed as root, changes the password of the user specified

● Executed by a normal user, the command asks for the old password

● Executed by root, it doesn't


User monitoring

● Each time a user logs into the system, at some logs are recorded:– /var/log/syslog Generic log for all the

system events. Maybe also logged on auth.

– /var/log/wtmp Specific log for all the user logins in the system.

● Binary file

– /var/log/btmp Specific log for all the failed user logins in the system.


User monitoring

● Commands– who (or w) Shows who is currently logged

into the system and what are they doing

– last / lastb All the users that logged into the system recently. Lastb shows info of failed attempts

– slay Kicks a user out of the system and kills all her processes


Index



Task automating

● cron System utility that allows launching periodic processes (tasks, actions...) in an unattended way, at a scheduled date

● at System utility that allows to program the execution of a process, but without repetition. It allows to defer the unattended process until a date


Repeated tasks: cron

● The cron service consists of 2 programs:– crond Program (daemon) offering the

service to the system– crontab Program that allows an user to

manage her tasks● crontab e Edit the scheduled tasks of the

current user● crontab l List the tasks of the current user● crontab r Remove all the tasks of the current

user



● Crontab edition:– crontab e Opens an editor predefined by

the variable EDITOR, to edit the user crontab● Tip: export EDITOR=nano

– Each crontab line has the following format, always split by spaces

● Minute: Integer number between 0 and 59● Hour: Integer in the range 0-23● Day: Integer in the range 1-31 (must be a valid

day if the month is also specified)



● Crontab edition:● Month: Integer in the range 1-12● Week_day: Integer in the range 0-7, where 0 and

7 mean Sunday. The short name of the week day can also be used (mon, tue, wed...)

● Command: Command to be executed. The command can be something like “ls /proc >> /tmp/proc” or a command that executes a customized script written by the user



● Moreover...– A “*” in any field indicates all valid values for

that field (respecting the restricions of the rest of the fields)

– A “–“ between numbers means an interval between those numbers

– A list of values split by “,” means a value list– A “/” means step values. Eg: “*/15” would

mean that the task is executed each 15 minutes.



● Moreover...– Lines starting with “#” are comments and

aren't had into account– A task is executed in the moment when all

the requirements of the schedule parameters are met



● Moreover...– Special parameters (instead of the 5 said

fields)● @reboot One time when the computer boots● @yearly Once a year, eq. to 0 0 1 1 *● @daily Each day, eq. to 0 0 * * *● @weekly Each week, eq. to 0 0 * * 0● @monthly Each month, eq. to 0 0 1 * *● @hourly Each hour, eq. to 0 * * * *



● Examples (crontab -e) # min hour day month weekday command

# Run 5 min after midnight, every day5 0 * * * /bin/daily.sh >> /tmp/out 2>&1

# Run at 2:15pm on the first of every month15 14 1 * * $HOME/bin/monthly

# Run at 10pm on weekdays0 22 * * 15 /usr/local/bin/script.sh

# Run 23 min after midnight, 2am, 4am... everyday23 023/2 * * * script.sh


Delayed tasks: at

● at allows to defer the execution of a task. Usage: at datetaskCTRL+d

● Example: $ at 4:45pmat> echo “Time to go”CTRL+d


Delayed tasks: at

● atq Shows pending tasks # atq1 20080101 10:35 b peter2 20080215 00:00 c peter5 20080210 01:00 Z root

● at -c 2 Shows all the execution environment for task 2

● atrm tasknum Deletes the specified task from the task queue


Delayed tasks: at

● Considerations– Admin can control which users cat use at: /etc/at.allow, at.deny. She can also see the tasks of all users

– at has a queue system to set the priority of the tasks

– batch is a command derived form at that allows executing a task only when the system load goes below of 0.8


Index



Log location and format

● At log files are located at /var/log

● Important files– messages System wide messages

– syslog Messages gathered by the log daemon (syslogd)

– debug System logs useful to solve problems

– auth.log Login and logoff attempts



● Log format

● Log continuous viewing

tail f logfile

Oct 26 18:17:01 localhost CRON[3384]: (pam_unix) session opened...

Date and timeComputer

Process name and PID

Log line



● Log files with a non-standard format– dmesg Last system startup info

– faillog Info about the last time users attempted to log into the system (fail number, max allowed, last fail).

● Viewed using faillog u user

– lastlog List of users and the last time they logged into the system.

● Viewed using lastlog



● Log files with a non-standard format– Xfree86.0.log, Xorg.0.log Info from the

X-Window system.● Viewed with standard viewers (tail, etc.)

ihttp://www.debian.org/doc/manuals/users-guide/ch-rsi.en.html


Log analysis and tools

● Log analysis allows to detect intrusions in the computer and hardware/software problems

● logcheck Looks for strange log entries and sends them to the administrator via email

ilogcheck.pdf


Log configuration

● Log file configuration: /etc/syslog.conf

● Format:– Selector: service.priority [[, service...];

service.priority...]● Services: auth, authpriv, cron, daemon, ftp, kern,

lpr, news, syslog, user, uucp, local0...7, *, none● Priorities: debug, info, notice, warning, crit, alert,

emerg, *, none– =priority: only that priority, not higher ones– !priority: only that priority, not lower ones– !=priority: exactly that priority


Log configuration

● Format:– Action

● /.../file: Log to log file (use complete path)● |pipe: Log to named pipe● /dev/console: Log to the console● @hostname: Log to a remote computer● user1, user2, ...: Send to a list of users● *: Send to all the users


Log configuration

● Example:

kern.* /var/adm/kernelkern.crit @earthkern.crit /dev/consolekern.info;kern.!err /var/adm/kernelinfomail.*;mail.!=info /var/adm/mail*.* @mars

iman syslog.conf, man syslogd


Log configuration

● Log rotation: logs are rotated periodically. They are compressed and deleted by age order

● Example:– /var/log/syslog– /var/log/syslog.0– /var/log/syslog.1.gz– /var/log/syslog.2.gz


Log configuration

● Configuración● /etc/logrotate.conf: Main log rotation

configuration● /etc/logrotate.d/*: Specific

configuration for particular services

iman logrotate


Index



Backup policy

● Backup levels– Level 0: Complete backup– Level 1: Incremental, changes from level 0– Levels 2..9: Changes from previous level

● Storage media– Tape– CD, DVD– Hard disk– Network


Backup policy

● Backup calendar– Diary complete

M T W T F S S0 0 0 0 0 0 0

– Weekly complete, diary level 1 M T W T F S S

1 1 1 1 1 1 0

– Weekly complete, diary level 1 M T W T F S S

1 2 3 4 5 6 0

Backup level


Backup policy

● Backup calendar– Monthly hanoi tower

M T W T F S S3 2 5 4 7 6 13 2 5 4 7 6 13 2 5 4 7 6 13 2 5 4 7 6 0

ihttp://www.backupschedule.net/backupSchedules/towerOfHanoi.html


Automating and scripts

● Ad-hoc backup: tar ● Saving

tar zcvf file.tgz files...

● Restoringtar zxvf file.tgz [files...]

gzip create verbose file

extract



● Ad-hoc backup: tar – Other functions

● A: Concatenate some tar files (not tgz!)● d: Find differences between tar/tgz file and files

on disk● delete: Delete files in tar/tgz file● r: Append files at the end of a tar/tgz file● t: List contents● u: Only append files newer than the ones in the

tar/tgz file



● Other useful commands– cpio: Input/output copy to file

find . print depth | cpio ov > tree.cpio

cpio idv < tree.cpio

– rsync: Remote directory synchronizationrsync aP e ssh user@host:rem_path loc_path

– dd: Exact copy from one device to another (or to a file)dd if=/dev/hda1 of=bak_hda1.bin



● Other useful commands– mt: Magnetic tape management

mt f /dev/st0 eject

mt retension

mt status

– Example of an ad-hoc backup on tape#!/bin/sh# SYSTEM CONFIG AND DATA BACKUPtar jcvlf exclude=/var/www /var /etc \ | buffer B s 10k > /dev/st0mt offline



● Simple automating: cron● Advanced backup utilities: Amanda

– Backup automating– Network backup server, multiple client

platforms (even Windows using Samba)– Mail reminder for administrators– Tape dump– Intelligent estimation of backup leven based

on previous statistics



● Amanda configuration– Multiple configs. Eg: /etc/amanda/DailySet1

– Authorized clients: /etc/amandahosts

– Main config: amanda.conf

– List of disks to be dumped: disklist

– List of tapes to be used: tapelist (don't touch!)



● Tape labeling for Amanda useamlabel [f] DailySet1 DMP00

Force, for a previously labeled tape

Configuration tobe used

Tape identifier (syntax defined at amanda.conf)


root@cesar:~ # cd /root@cesar:/ # amrecover C DailySet1 s localhost.localdomainAMRECOVER Version 2.4.4p3. Contacting server on localhost ...220 localhost.localdomain AMANDA index server (2.4.4p3) ready.200 Access OKSetting restore date to today (20041103)200 Working date set to 20041103....amrecover> listdisk200 List of disk for host localhost.localdomain201 /home201 /etc201 /var200 List of disk for host localhost.localdomainamrecover> setdisk /var200 Disk set to /var.


● Recovering: amrecover

The files are going to be extracted in the current directory. It's important to change to the root directory (“/”) of the computer to be restored, in particular if it's a satellite computer (not the one hosting the backup)



● Recovering: amrecoverroot@cesar:~ # cd /root@cesar:/ # amrecover C DailySet1 s localhost.localdomainAMRECOVER Version 2.4.4p3. Contacting server on localhost ...220 localhost.localdomain AMANDA index server (2.4.4p3) ready.200 Access OKSetting restore date to today (20041103)200 Working date set to 20041103....amrecover> listdisk200 List of disk for host localhost.localdomain201 /home201 /etc201 /var200 List of disk for host localhost.localdomainamrecover> setdisk /var200 Disk set to /var.

NOTE: check access permissions at /etc/amandahosts



● Recovering: amrecoveramrecover> setdate 20041020200 Working date set to 20041020.amrecover> ls...20041013 spool/20041013 www/amrecover> add wwwAdded dir /www at date 20041013amrecover> listTAPE DailySet5 LEVEL 0 DATE 20040930 /wwwTAPE /home/amand... LEVEL 1 DATE 20041013 /wwwamrecover> extract...

iman amanda , /usr/share/doc/amanda , http://www.amanda.org


Basic commands


Index



Files basic commands

● I-node: Data structure in a filesystem. Stores basic data about the file, directory or another object in the filesystem– Size and physical location (disk block)– Owner and group– Mode (permissions)– Creation, modification and last access time– Reference counter, accounting how many

hard links are pointing to the inode



● Example:$ ls lisatotal 16766122 4 drwxrxrx 2 root root 4096 .765537 4 drwxrwxrwx 11 root root 4096 ..766125 4 rwrr 2 root root 5 file1766125 4 rwrr 2 root root 5 file2766126 0 lrwxrwxrwx 1 root root 8 file3 > file1

Inode number

Used blocks

Mode

Reference counter

Owner and group

Used bytes



● Permissions (mode)– Permissions

● Read (r): 4 (octal)● Write (w): 2● Execution (x): 1

– Granularity● User (u)● Group (g)● Others (o)● All (a): ugo

750

User: 4(read)+2(write)+1(execution)

Group: 4(read)+1(execution)

Others: (no access)



● Special permissions (4th nibble)– Setuid bit (o+s, 4): Sets the owner of the file

as the effective user when the file is executed– Setgid bit (g+s, 2): Sets the group of the file

as the effective user when executed– Sticky bit (STIcky, Save Text Image) (o+t, 1)

● For files: Keeps the code in swap, just if it's executed later

● For directories: Only the owner of the files inside and the owner of the directory can delete files in the directory

! UNSAFE!!



● Links– Hard links: The file has two entry points at

the same time– Soft links: Reference to the true path of the

file

a b c d...1234...

a b c d...1234...



● Example$ ls lisatotal 16766122 4 drwxrxrx 2 root root 4096 .765537 4 drwxrwxrwx 11 root root 4096 ..766125 4 rwrr 2 root root 5 file1766125 4 rwrr 2 root root 5 file2766126 0 lrwxrwxrwx 1 root root 8 file3 > file1

a b c d...1234...

a b c d...1234...




File the symbolic link is pointing to

Byte size (note that “file1” string takes actually 5 bytes)

Reference counter. file1 and file2 are actually two hard links to the same content, thus its reference counter is 2




The current directory (.) has 2 references: Entry (.) of that directory and entry mydirname in its parent directory.Parent directory (..) has 11 references: the entry tmp in the root directory and the “..” entries in the other 10 directories it has.



● Special files– Devices (high number, low number)

● Blockbrwrw 3, 0 /dev/hda

● Charactercrwrw 9,96 /dev/st0a

– Named pipes (FIFO, socketpairs)srwxrwxrwx /tmp/.X11unix/X0



● Special directories: “.”, “..”

● Hidden files: Start by “.”. Eg: .bashrc

● Brief of the file types:d Directoryl Linkc Char. deviceb Block device

p Pipess Sockets Ordinary



● pwd Shows the current absolute path

● cp Copy filescp [ir] ( file file | dir dir | filelist dir )

● cd Change directorycd path

● mv Move or rename files and directoriesmv file newfile



● mkdir Create directoriesmkdir [p] directory

● ls Shows the content of a directory or info about a file

ls [laR] [dirname]|[filename]

● rmdir Deletes empty directoriesrmdir directory



● rm Deletes filesrm [ir] file1 file2...

● find Searchs files by name, etc.find [searchdir] name “filename”

● cat Shows the content of a file without being able to edit it and without pauses

cat file



● less/more Shows the content of a file without being able to edit it but page by page

less file

● chmod Permission (mode) changechmod [options] mode file

– Examples:● chmod u+srwx,gw,o+rxt prueba● chmod 5755 prueba



● ln Link creation– Hard link: Not allowed among directories

ln sourcefile destinationlink

– Soft linkln s sourcefile destinationlink


Index



Processes

● Linux is multitasked– Each instance of a command or program

being executed is a process

● The kernel controls all the processes– Allocates resources (memory, cpu,...)

– Schedules per priority the processes to be executed

– Controls resource access. Each process has an owner

– Process info at /proc


Processes

● Important properties characterizing a process– Process identification number (PID)– Parent id number (PPID)– State: Running, Sleep (waiting for a resource)– Owner: Usually the user running the process– Priority: importance in respect to other

processes


Process control commands

● ps Lists running processes (snapshot)ps aux

● top Interactively lists running processes. Useful for monitoring. Allows process management

● kill Sends a termination signalkill KILL pid

Shows user, PID, %CPU, %MEM, STAT, time...

Causes the abrupt finishing of the process with PID pid



● nice Starts process with given priority. Range from -20 (higher) to 19 (lower)

nice +10 process● snice, renice

● bg (shell) Sends process to background

– Same as & when launching from shell

● fg (shell) Brings a process to foreground. Opposite to bg



● jobs (shell) Show processes in background for the current user

● CTRL+z (shell) Sends a STOP signal (bg)

● CTRL+c (shell) Aborts the execution

– ls /proc Numbers are currently running processes


Index



Documentation

● program help Shows parameters and a brief summary of the program. Given by the own program.

● man [command|function] Show online help for the program or command

● apropos word Show commands related to the given word


Documentation

● info program Shows extended information about a program– Browseable text format

– Organized in a different way than man

● /usr/doc/package/usr/share/doc/package Specific doc about each package: license, usage examples, etc.


Index



Shell: redirections & pipes

● In Unix, the shell was traditionally sh. In GNU/Linux: bash, ksh, csh, dash,...

● Each process has at least three I/O ways:– stdin (0): Standard input, usually bound to

the current terminal (keyboard)– stdout (1): Standard output, usually bound

to the terminal (screen)– stderr (2): Standard error, bound to the

terminal (screen) together with stdout



● Output redirection– prog > file Overwrites the output file

– prog >> file Appends to output file

● Input redirection– prog < file Reads input from file

● Error redirection– prog 2> file Overwrites the output file

– prog 2>> file Appends



● Examples:

$ echo Hello > out.txt$ echo Hello again >> out.txt$ cat out.txtHelloHello again

$ echo See you > out.txt$ echo See you again > out.txt$ cat out.txtSee you again

$ wc < out.txt1 3 14

$ wc xyz.txtwc: xyz.txt: File not found

$ wc xyz.txt 2> err.txt$ cat err.txtwc: xyz.txt: File not found



● Pipes: One program sends data through a pipe and another one reads it from the opposite side

● prog1|prog2

● Examples:$ echo See you later | wc

1 3 14

$ tar zcvf . | nc 192.168.1.2 10000$ nc l p 10000 | tar zxvf


Scriptprogramming


Index



Defining env. variables

● Variable declarationVARIABLE=”value”

● Variable usage (substitution for its content)$VARIABLE ${VARIABLE}

● Variable deletionunset VARIABLE

Quotes aren't mandatory, but recommended

Don't type spaces



● Examples:$ S=”_s_”; SEA=”_sea_”

$ echo $SEA; echo ${S}EA

_sea_

_s_EA



● Variable substitution● $param, ${param} Simple substitution

(for the value of param)● ${param:value} If param is null, use

value● ${param:=value} If param is null, use

value and assign param=value



● Variable substitution● ${param:?value} If param is null, write

value to stderr. If value isn't specified, writes “param: param null or not set”

● ${param:+value} If param is defined, use value. If not, don't substitute

iman bash (“EXPANSION, Parameter expansion” section)



● Examples:$ echo “Name is ${NAME:unknown}”Name is unknown

$ NAME=Peter; echo “Name is ${NAME:unknown}”Name is Peter

$ echo “Before: ${N}, Now: ${N:='Peter'}, After: ${N}”Before: , Now: 'Peter', After: 'Peter'

$ unset NAME; echo “Name is ${NAME:?}”bash: NAME: parameter null or not set

$ NAME='Peter'; echo “${NAME:+Exists}”Exists



● Data read to be stored in variablesread A B C...

– Input fields are usually split by spaces, line feeds or tabs

– This behavior can be modified changing the IFS environment variable (internal field separator)

– Example:$ export IFS=”:”; read A B C; echo “A/B/C”1:new york;z77c6s1/new york/z77c6s


Global variables, Language

● Variables assigned in the current shell (locals) aren't inherited by other child shells. They must be exportedexport DISPLAY='localhost:0.0'

● Read only variables: readonly VARIABLE

● Examples:$ NAME=Peter$ NAME=John$ readonly NAME$ NAME=Peterbash: NAME: readonly variable



● List of defined variables: set

● Most important global variables– DISPLAY X-Window server which the

current process will send windows to

– COLUMNS Number of console columns

– EDITOR Favourite text editor

– EUID Current effective User ID

– HOME Path to the user home directory



● Most important global variables– HOSTNAME Computer name

– HOSTTYPE Computer architecture

– IFS Internal field separator, input field separator (parameters, etc.)

– LINES Number of console lines

– OSTYPE Operating system

– PAGER Favourite pager (eg: more, less, most)



● Most important global variables– PATH Executable files search path. Split by

“:”– Example:

PATH=/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin

– PPID Parent Process ID

– PS1 Primary prompt string. Supports substitutions

iman bash (“PROMPTING” section)



● Most important global variables– PS2 Secondary prompt string (when some

lines are used to type a command). Eg: cutting with “\”

– PS3 Prompt string for select command

– PS4 Nesting indicator in debug mode

– PWD Current work path

– SHELL Current command interpreter



● Most important global variables– TERM Terminal emulator. Eg: linux, vt100,

xterm

– UID Current User ID

– USER Current user

iman bash (“PARAMETERS, Shell variables” section)



● Language related variables– LANG Locale (lang. localization) to be used.

● Eg: es_ES, es_ES@euro, es_GL, es_ES.UTF-8

– LC_ALL Has preference over LANG and any other LC_ variable to choose a localization

– LC_COLLATE Preference order for path sorting, range expressions, etc.

– LC_TYPE Character interpretation and character class behavior



● Language related variables– LC_MESSAGES Translation for system

messages and any other string like $”message”

– LC_NUMERIC Numeric format for a particular localization

– LC_TIME Date and time format

iman locale



● Shell special variables– $0...$9 Shell executable name and 9 first

arguments that were passed

– $# Number of args. passed to the shell

– $* String with all the args. as one single value

– $@ Like $*, but when wrapped by double quotes, wraps each parameter by double quotes



● Shell special variables– $$ Current script or session PID

– $! PID of the program last sent to background

– $? Exit status of the last program not executed in background

– $ Current options in effect

– ~ Path to the user home directory

iman bash (“PARAMETERS, Special parameters” section)


Index



Input/output

● Input/output– >, < Stdin and stdout redirection

– | Pipes, redirection of a command stdout to the stdin of the next one

– << Read stdin from the script itself until a mark is found (here documents)

$ cat << END> Hello, I'm $USER.> Good afternoon.ENDHello, I'm root.Good afternoon.


Input/output

● Input/output– >> Stdout redirection in append mode

– 0<file, 0<<file Redirection of the input of a descriptor (0) to a file

– 2>file, 2>>file Redirection of the output of a descriptor (2) to a file

– 0<&5 Redirection of the input of a descriptor (0) to the file used by another descriptor (5)


Input/output

● Input/output– 2>&1 Redirection of the output of a

descriptor (2) to the file used by another descriptor

– <& Closes stdin

– >& Closes stdout

– script>>file Redirection to a file of the stdout for all the output of the commands of a script


Input/output

● Input/output– for ... done >> file Redirection to a

file of the stdout for all the operation of a loop

● Interesting files for input/output– /dev/null Discards everything sent to it.

Useful to avoid displaying a command output

– /dev/tty Our own terminal. Anything sent to it, is displayed on the screen


Index



Command grouping

● Command grouping– Semicolon (;) Chains commands in the

same line as if they were a single one

– Parenthesis ( ) Executes the code inside in a subshell. Changes in variables don't stay after returning to the current shell

$ TMP=stays; (TMP=you_cant_see_me; echo $TMP); echo $TMP

Typing spaces and ending in ; isn't mandatory


Command grouping

● Command grouping– Curly braces { } Executes the code inside

in the current shell

– Double ampersand (&&) Executes the next command only if the previous one returned a true result (equal to 0)

$ TMP=stays; { TMP=you_cant_see_me; echo $TMP; }; echo $TMP

Typing spaces and ending in ; is mandatory

$ ls mydir && echo 'That was the content of mydir'


Command grouping

● Command grouping– Double pipe (||) Executes the next

command only if the previous one returned a false result (different from 0)$ ls mydir || echo 'The directory mydir doesn\'t exist'


Index



Wildcards

● File name matching in the shell: wildcards– ? Matches with one single character

– * Matches zero or more characters

– [abc...] Matches with any of the listed characters

– [!abc...] Matches with all but the listed characters


Wildcards

● File name matching in the shell: wildcards– {str1,str2,...} Makes a list of

expressions using one string each timefile.{txt,pdf,rtf}

file.txt file.pdf file.rtf

– \ (escape) Overrides the meaning of any special character, even end of line


Index



Quoting

● Double quotes (“”) Overrides the special meaning of all the characters but $, ` and \

● Single quotes ('') Overrides the special meaning of all the characters

$ echo “The user is $USER”The user is root

$ echo 'The user is $USER'The user is $USER


Quoting

● Back quotes (``) Command substitution. It's executed and output is put in place of the command text

● Backslash (\) Overrides the special meaning of the next character, even inside quotes

$ echo “Today is ###`date`###”Today is ###Thu Jan 10 11:52:26 CET 2008###

$ echo “Double quotes \”are nice\”, aren't they?”Double quotes “are nice”, aren't they?


Index



Control structures

● Condition evaluation– test 5 = 5

– [ 5 = 5 ]

● The result is stored in $? (errorlevel)– True: 0– False: different from 0

These two commands are the same (see /usr/bin). It's important to keep the space. 5 = 5 are in fact command arguments


Control structures

● Possible test conditions– string string isn't null

– n string string isn't null

– string1 = string2 Are equal

– string1 != string2 Are distinct

– eq Numerically equal to (put between the values to be compared)

– ne Not equal


Control structures

● Possible test conditions– gt Greater than

– ge Greater or equal

– lt Lower than

– le Lower or equal

– b File is a special block file

– c File is a special character file

– d Is a directory


Control structures

● Possible test conditions– f Is an ordinary file

– g Has the setgid bit active

– k Has the sticky bit active

– p Is a named pipe

– r Can be read by the current process

– s Size larger than 0

– t File descriptor is open and linked to a terminal


Control structures

● Possible test conditions– u Has setuid bit active

– w Can be written by current process

– x Can be executed by current process

– a AND of previous and next arguments

– o OR of previous and next arguments

iman test


Index



Control structures

● If conditional structureif [ $n lt 5 ]

then

echo 'lower than'

elseif [ $n gt 5 ]

then

echo 'greater than'

else

echo 'are...'

echo '...equal'

fi


Control structures

● Case conditional structurecase $option in

h) echo 'This script has no help'

;;

n) echo 'The computer name is:'

hostname

;;

*) echo 'Default option'

;;

esac


Control structures

● For loop (list)for variable in 1 2 3 /tmp/* `ls` list...

do

echo 'One element:'

echo $variable

done

● If for parameters are skipped, positional parameters $1..$9 are used

List formed by single elements, files in a path (even with wildcards expanded to multiple values), output of command execution, etc.


Control structures

● For loop (counter)for ((i=0;$i<5;i++))

do

echo 'Number:'

echo $i

done

Value change

Condition

Initialization


Control structures

● While loopi=1

while [ $i le 5 ]

do

echo $i

i=`expr $i + 1`

done


Control structures

● Until loopi=1

until [ $i gt 5 ]

do

echo $i

i=`expr $i + 1`

done

● Iteration control in loops– break Exits the loop immediately

– continue Jumps to the next iteration


Control structures

● Functions– Declaration

union () {

echo 'Joining the arguments'

return “$1 union $2”

}

anotherFunction () { echo 'hello'; }

● Usageunion “one” “two”

echo $?

The space between the function and () isn't mandatory, nor even the { appearing in the same line. Parameters are not declared.

Parameters appear as positional parameters

Result is declared by return and used with $?


Control structures

● Functions– Global scope of the variables: variables

declared or changed inside a function stay when the call ends

● Code inclusion: dot (.)#!/bin/sh

. utilities.sh

myUsefulFunction 1 2 3 4


Code examples

● Parameter shifting#!/bin/sh# FILE printargs.shwhile [ $# gt 0 ]do echo “$@” shiftdone

$ printargs.sh fly from “New York” to Moscow fly from New York to Moscow from New York to Moscow New York to Moscow to Moscow Moscow

Shifts the positional parameters deleting the first one ($1)


Code examples

● Command line parameter processing (using shift)#!/bin/sh# setether: set an Ethernet interface's IP configwhile [ $# gt 1 ]do case $1 in a) ARP=”arp”; shift;; b) BROADCAST=$2; shift 2;; i) IPADDRESS=$2; shift 2;; m) NETMASK=$2; shift 2;; n) NETWORK=$2; shift 2;; *) echo “setether: illegal option: $1”; exit 1;; esacdoneINTERFACE=$1ifconfig $INTERFACE $IPADDRESS netmask $NETMASK \ broadcast $BROADCAST $ARP; route add net $NETWORK

Shifts two positional parameters in a row


Code examples

● Command line parameter processing (using getopts)#!/bin/sh# setether: set an Ethernet interface's IP configwhile getopts ab:e:i:m:n: optiondo case $1 in a) ARP=”arp”;; b) BROADCAST=$OPTARG;; i) IPADDRESS=$OPTARG;; m) NETMASK=$OPTARG;; n) NETWORK=$OPTARG;; *) echo “setether: illegal option: $option”; exit 1;; esacdoneINTERFACE=$1ifconfig $INTERFACE $IPADDRESS netmask $NETMASK \ broadcast $BROADCAST $ARP; route add net $NETWORK

The : indicates that this parameter takes an argument


Advancedcommands


Index



Regular expressions

● Regex: Powerful text pattern matching expressions

● Use metacharacters to mean repetition, existence or ranges in character patterns

● Common metacharacters– Dot (.) Matches with any character but line

feed

– Asterisk (*) Matches 0 or more appearings of the char. or regex preceeding it


Regular expressions

● Common metacharacters– Brackets [] Match any of the char. classes

enclosed by them● Circumflex (^) after the opening bracket

Inverse match● Minus () Specifies a range● Metacharacters loose their meaning inside the

brackets. Eg: [ at the begining of the list


Regular expressions

● Common metacharacters– Circumflex (^) Means line begining

– Dollar ($) Means line ending

– Backslash (\) Escape next metacharacter

● Example# grep '^.* 15:[09][09].*kernel.*$' /var/log/messagesSep 17 15:04:13 localhost kernel: eth1: remaining active...Sep 17 15:04:18 localhost kernel: eth1: DSPCFG accepted after...Sep 17 15:04:18 localhost kernel: eth1: link up....


Regular expressions

● Metacharacters exclusive to sed, grep, egrep and awk– Slash-brace (/{m,n/}) Matches a range of

appearings of the single character immediately before the expression

– Variants● /{m/} Exactly m repetitions

● /{m,/} At least m repetitions

● /{m,n/} Betwee m and n repetitions


Regular expressions

● Metacharacters exclusive to sed, grep, egrep and awk– Plus (+) Matches one or more appearings of

the character or regex before it

– Question mark (?) 0 or 1 appearings

– Pipe (|) Matches the previous expression or the next one

– Parenthesis () Groups regular expressions


Regular expressions

● Tip: Remember that some characters are interpreted by the shell (hello|hi) world

● Regex learning & exploring utilities:– kodos– kregexpeditor– visual-regexp

iman -S7 regex

$hello\|hi$\ world'(hello|hi) world'


Index



Grep

● Grep– Outputs the lines from stdin or file(s)

matching a pattern– Only supports basic regexps– Example: Modified files in a local repository

svnstatus | grep v nbproject | grep ^M

Inverse matching


Grep

● Egrep– Supports extended syntax

● Rgrep– Supports recursive search– Use -E for extended syntax


Index



Sed

● Sed: Stream editor for filtering and transforming textsed e script [file]

sed f scriptfile [file]

● Substitutions/source/destination/modifiers

– Example: for SRC in `find type f`do DEST=`echo $SRC | sed e 's/\.txt/\.bak/'` mv $SRC $DESTdone


Sed

● Substitution– Useful modifiers

● g Global (multiple) replacement per linesed e 's/\./,/g'

● w file Write pattern to filesed e 's/kernel/&/w kernel.txt' e 's/gconf/&/w gconf.txt' < /var/log/syslog

● p / !p Print/hide substitued lines (use n to hide default output). ! inverts the restrictionls lisa | sed n e 's/total/total/p'


Sed

● Substitution– Using parts of the result

● Whole result (&)● Substrings: $, $, \1, \2...

sed e 's/$[09]*$\.$[09]*$/Integer: \1, decimal: \2/'3452345.46657Integer part: 3452345, decimal part: 46657


Sed

● Patterns and ranges to apply commands– Line:

sed n e '12 s/src/dest'

– Range of lines:sed n e '12,25 s/src/dest'

sed n e '1,25 p'

sed n e '12,$ s/src/dest/!p'


Sed

● Patterns and ranges to apply commands– Matching a pattern:

sed n e '/pattern/ s/src/dest'

– Matching a range between pattern occurences:sed n e '/begin/,/end/ p'


Sed

● Command grouping { }sed n e '

/begin/,/end/ {

s/1/A/gp

s/2/B/gp

s/3/C/gp

}

p

'

When the pattern is matched, the block is executed.When not matched, the general command is executed.


Sed

● More advanced commands– a Append lines

– i Insert lines

– c Change lines

– = Print line number

iman sed

http://www.grymoire.com/Unix/Sed.htmlhttp://www.panix.com/~elflord/unix/sed.html


Index



Awk

● Awk: pattern scanning and processing languageawk program [file]

awk f programfile [file]

● Most common syntax– Line matching

awk /regex/ file


Awk

● Most common syntax– Matching and processing

awk '

BEGIN {print “Initialization”}

/regex1/ {print $1, $2}

/regex2/ {print $3, $4}

END {print “Final action”}

' file

–

i-th word (like if it was a parameter) in the matched line. $0 is the whole line.


Awk

● Most common syntax– Programming structures inside code blocks

{ if ($1==”1”) print “Winner”;else print “Looser”; }

{ x=1; while (x<100) { x=x*2; } }

{ for (i=0;i<10;i++) { print i; }


Awk

● Most common syntax– Predefined variables (most important)

● NR Current count of number of input lines

● NF Count of number of words in current line

● FILENAME Current processed file name

iman awk

http://www.vectorsite.net/tsawk.html

system integration in free software

Technology