unix

Unix Training Program

What is Unix? A multi-user networked operating system

“Operating System” Handles files, running other programs, input/output Looks like DOS…but more powerful The internet was designed on it, thus networking is an

intrinsic part of the system

“Multi-user” Every user has different settings and permissions Multiple users can be logged in simultaneously

Unix vs. Linux Age

Unix: born in 1970 at AT&T/Bell Labs Linux: born in 1992 in Helsinki, Finland

Sun, IBM, HP are the 3 largest vendors of Unix These Unix flavors all run on custom hardware

Linux is FREE! (Speech vs. Beer) Linux was written for Intel/x86, but runs on many

platforms

• Connecting: - Connect to the Unix boxes via telnet or ssh (ssh preferred) - After connection, you have a shell (command prompt)

• Shells: - Instead of clicking, you type long, cryptic commands - After login, you begin in your home directory - tab completion is nice

Logging in and home directories

The Command Prompt Commands are the way to “do things” in Unix A command consists of a command name and options called

“flags” Commands are typed at the command prompt In Unix, everything (including commands) is case-sensitive

[prompt]$ <command> <flags> <args>

localhost:~$ ls –l -a unix-tutorial

Command PromptCommand (Optional) flags

(Optional) arguments

Note: Many Unix commands will print a message only if something went wrong. Be careful with rm and mv.

Getting help with man man (short for “manual”) documents commands

man <cmd> retrieves detailed information about <cmd> man –k <keyword> searches the man page summaries

(faster, and will probably give better results) man –K <keyword> searches the full text of the man

pages

fiji:~$ man –k passwordpasswd (5) - password filexlock (1) - Locks the local X display

until a password is enteredfiji:~$ passwd

Directories In Unix, files are grouped together in other files

called directories, which are analogous to folders in Windows

Directory paths are separated by a forward slash: / Example: /homes/trainees/troo1

The hierarchical structure of directories (the directory tree) begins at a special directory called the root, or / Absolute paths start at /

Example: /homes/ Relative paths start in the current directory Example: troo1 (if you’re currently in trainees)

Your home directory “~” is where your personal files are located, and where you start when you log in.

Directories (cont’d) Handy directories to know

~ Your home directory

.. The parent directory

. The current directory

ls LiSts the contents of a specified files or directories (or

the current directory if no files are specified) Syntax: ls [<args> … ] Example: ls backups/

pwd Print Working Directory

Directories (cont’d further) cd

Change Directory (or your home directory if unspecified)

Syntax: cd <directory> Examples:

cd backups/unix-tutorial cd ../class-notes

mkdir MaKe DIRectory Syntax: mkdir <directories> Example: mkdir backups class-notes

rmdir ReMove DIRectory, which must be empty first Syntax: rmdir <directories> Example: rmdir backups class-notes

Files Unlike Windows, in Unix file types (e.g. “executable

files, ” “data files,” “text files”) are not determined by file extension (e.g. “foo.exe”, “foo.dat”, “foo.txt”)

Thus, the file-manipulation commands are few and simple …

Many use only 2 letters rm

ReMoves a file, without a possibility of “undelete!” Syntax: rm <file(s)> Example: rm tutorial.txt backups/old.txt

Files (cont’d) cp

CoPies a file, preserving the original Syntax: cp <sources> <destination> Example: cp tutorial.txt tutorial.txt.bak

mv MoVes or renames a file, destroying the original Syntax: mv <sources> <destination> Examples:

mv tutorial.txt tutorial.txt.bak mv tutorial.txt tutorial-slides.ppt backups/

Note: Both of these commands will over-write existing files without warning you!

Shell Shortcuts Tab completion

Type part of a file/directory name, hit <tab>, and the shell will finish as much of the name as it can

Works if you’re running tcsh or bash

Command history Don’t re-type previous commands – use the up-arrow to access

them

Wildcards Special character(s) which can be expanded to match other

file/directory names

* Zero or more characters

? Zero or one character Examples:

ls *.txt rm may-?-notes.txt

• The basics:• ls – lists your files• cp – copies a file• mv – moves (can be used to rename) a file• rm – remove a file• mkdir – makes a directory (synonymous to folder)• rmdir – removes a directory• cd – changes your directory• ps – list the current processes in the shells• kill – kill processes (-9 if you want to kill them violently)

Commands - Basic

Editing Text Which text editor is “the best” is a holy war. Pick one

and get comfortable with it. Three text editors you should be aware of:

pico – Easy! Comes with pine (Dante’s email program)

emacs/xemacs – A heavily-featured editor commonly used in programming

vim/vi – A lighter editor, also used in programming

Your opinion is wrong.

/ – root. The source of everything

/bin – Where executables (programs) are run

/sbin – executables generally of interest only to the super user

/home – where your “home directories” are

/tmp – same as c:\temp in windows. Just for temporary files

/var – system logs and other stuff (probably not intersting)

/usr – layed out like /, but with less system-critical stuff

/dev – has files that represent different devices on the system

/proc – has runtime system info (try: cat /proc/cpuinfo)

Unix File Hierarchy

Commands – Less Basic

•less basic:• ln – creating links (kind of like shortcuts, but they work right)• less, more – keeps your text from scrolling off your screen• man, info – gives you “help” on commands• grep – search in files for a string• find – search for a filename• finger – get some information on other users (like last login)• which – locate where a command is• pwd – print the current directory• diff – find the difference between two (text) files

• Wildcards: ? – matches a single character * – matches any number of characters [xy] – matches either x or y (note: you can use other letters too!)

• Redirection: > – pipes standard output (printf, stdout, cout) to a file < – pipes a file into standard input (scanf, stdin, cin) | – connects the output of one command to the input of another

Wildcards and Redirection

Ownership Files have two owners

Every file has exactly one user owner Every file has exactly one group owner

Everyone is a user Users are in at least one group

Processes have owners, too (known as an “id”) Every process has exactly one user id Every process has at least on group id

Users and groups are really just numbers with names Every username is mapped to a single numeric “uid” Every groupname is mapped to a single numeric “gid”

Who am I? Commands that tell you who you are:

whoami displays your username id displays your username and

groups

Commands that tell you who others are: finger [<name>] displays info for <name> id [<username>] displays info for <username>

Commands that change who you are: su <username> “switch user” to <username> login login as a different user

Types of FilesRegular FilesDirectoriesDevices

Character Devices Block Devices

Sockets and Named PipesSymbolic Links

Files and DirectoriesEverything is referenced via a fileDirectories

List of files & inodes “.” – Reference to the current directory “..” – Reference to parent directory Root (/) – “.” and “..” are the same

Device Files Character Devices

Transfer unit: byte Example: /dev/console

Block Devices Transfer unit: Group of bytes (block) Examples: /dev/hda

Device Numbers Major – Type of device Minor – Device number

Sockets & Named Pipes Enables communication between processes Socket

Processed must have a connection first Example: X Windows

Named Pipe Communication between unrelated processes FIFO Not used very often

Link Files Multiple names for same file Hard Link

Pointer to Inode Can’t cross partitions File removed when all links deleted

Symbolic (Soft) Links Pointer to file path name Dangling symlink – Real file which no longer exists

ln [-s] <real_file> <link_file>

Magic NumbersByte pattern at beginning of filePatterns listed in file called magic

RedHat: /usr/share/magicfile – Tests a file to determine type

Filesystem Test Magic Number Test Language Test

File Attributes

InodeInode

Permissions

Ownership

Time StampsChangeModificationAccess

File Size

Link Count

Pointers to data

Viewing File Attributes

File Type Permissions Link Count Ownership

File Size/Device # Modification Date File Name

ls –l: Long listing (includes attributes)

stat: Lists all attributes

File Type Attribute# ls -ld /home /etc/passwd /dev/console

crw--w--w- 1 root root 5, 1 Sep 27 11:27 /dev/console

-rw-r--r-- 1 root root 559 Sep 22 13:14 /etc/passwd

drwxr-xr-x 3 root root 0 Sep 26 10:42 /home

File TypeFile Type MeaningMeaning

- Regular File

d Directory

l Symbolic Link

b Block Device

c Character Device

p Named Pipe

s Domain Socket

Ownership

User Owner of file User names/UIDs defined in /etc/passwd

Group Organization of users accessing the file Group names/GIDs defined in /etc/group

# ls -ld /home /etc/passwd /dev/console




Permissions

3 levels of access – Owner, Group, Other

# ls -ld /home /etc/passwd /dev/console




OperationOperation FileFile DirectoryDirectory

Read Read file List files

Write Delete/Modify file Create/Delete file

Execute Run program Access file

Changing Access Users & Groups

chown [-R] user file… chgrp [-R] group file

Permissions chmod [-R] <op> file… Numeric: <op> = [#]### Symbolic: <op> = <who op perm>

Who: (u)ser (g)roup (o)ther (a)ll op: (+)add (-)remove (=)set Perm: (r)ead (w)rite e(x)ecute

Numeric/Symbolic Permissions

OctalOctal BinaryBinary SymbolicSymbolic

0 000 ---

1 001 --x

2 010 -w-

3 011 -wx

4 100 r--

5 101 r-x

6 110 rw-

7 111 rwx

Default Permissionsumask Shell Environment VariableDefines permissions to remove

NumericNumeric BinaryBinary Effective PermsEffective Perms

0 000 rwx

1 001 rw-

2 010 r-w

3 011 r--

4 100 -wx

5 101 -w-

6 110 --x

7 111 ---

Special AttributesSetuid (SUID) Bit

Run program with access of owner Symbolic: s Numeric: 4000

Setgid (SGID) Bit Run program with access of owner group Symbolic: s Numeric: 2000

Sticky BitPurpose

File: Force program to stay in RAM (obsolete)

Directory: Cannot remove file unless you own the file or directory

Symbolic: tNumeric: 1000Example: /tmp

Searching the Filesystem

find: Command line search tool Searches through directory hierarchy Search by any combination of file names and

attributes Display files or perform operations on them Examples:

find /var –mtime -1 find / -name core –exec rm –f {} \;

File Permissions Every file has three access levels:

user (the user owner of the file) group (the group owner of the file) other (everyone else)

At each level, there are three access types: read (looking at the contents) write (altering the contents) execute (executing the contents)

Strange Things There are three “strange” permissions:

setuid (run program as user owner) setgid (run program as group owner) text (stay in swap after executing)

Directories act differently “write” (creating/deleting files) “execute” (cd-ing to that directory) “setuid” (ignored) “setgid” (created files have same group

owner) “text” (deletion restricted to user owned files)

Examining Permissions A “long” ls listing shows file permissions:

[zanfur@odin zanfur]$ iduid=8774(zanfur) gid=100(users) groups=100(users),101(mp3)[zanfur@odin zanfur]$ ls -ltotal 524-rwxr-xr-x 1 zanfur users 512668 Jul 31 00:18 bashprw-r--r-- 1 zanfur users 0 Jul 31 00:24 fifo-rw-r--r-- 1 zanfur users 0 Jul 31 00:18 filedrwxr-xr-x 2 zanfur users 4096 Jul 31 00:16 normaldrwxrws--T 2 zanfur mp3 4096 Jul 31 00:14 shareddrwxrwxrwt 2 zanfur users 4096 Jul 31 00:14 tmpdrwxrwxr-x 2 zanfur www 4096 Jul 31 00:15 www[zanfur@odin zanfur]$ _

Permissions Listing Breakdown

Field Description

Valid Values

?---------

file type - (normal file), d (directory), p (fifo), b, c, s

-?--------

user read - (no read permissions), r (read permissions)

--?-------

user write - (no write permissions), w (write permissions)

---?------

user execute

- (no exec), x (exec), S (setuid), s (both)

----?-----

group read - (no read permissions), r (read permissions)

-----?----

group write - (no write permissions), w (write permissions)

------?---

group execute

- (no exec), x (exec), S (setgid), s (both)

-------?--

other read - (no read permissions), r (read permissions)

--------?-

other write - (no write permissions), w (write permissions)

---------?

other execute

- (no exec), x (exec), T (text), t (both)

What You Can Do With Permissions

Permission

File Directory

r (read) Read a file List files in …

w (write) Write a file Create a file in …

Rename a file in …

Delete a file in …

x (execute) Execute a file (eg shell script)

Read a file in …

Write to a file in …

Execute a file/shell script in …

Changing Ownership Changing user ownership:

The command is “change owner”:chown <username> <filename>

but, you can only do it if you are root so just copy it instead

Changing group ownership: The command is “change group”:

chgrp <groupname> <filename> but, you can only do it if you are in group <groupname>

and you must be the user owner of the file

Changing Permissions The “change mode” command:

chmod <level><op><permissions>[,…] <filename>

<level> string of: u, g, o, a (user, group, other, all)

<op> one of +, -, = (gets, loses, equals)<permissions> string of: r, w, x, s, t, u, g, o

(read, write, execute, set-id, text,same as user, same as group, same

as other), Examples:

chmod u+rwx,go-w foobarchmod g=u,+t temp/chmod u=rwx,g=rwxs,o= shared/

• Groups• In Unix, all users belong to at least one group. The idea is that each person in one group has a similar amount of access to the system.

• Permissions• Files and directories all have “permissions” associated with them.• Permissions tell the OS who can do what with your files and

directories. Permissions can be set for three categories, the user, the group and the rest of the world. The permissions are:

read, write, execute

Groups and Permissions - 1

To see the permissions on a file, do a ‘ls –l’ fiji:/u15/awong$ ls –l -r--r--r-- 1 awong ugrad_ce 17375 Apr 26 2000 rgb.txt -rw-r--r-- 1 awong ugrad_ce 17375 Apr 5 02:57 set10.csv drwxr-xr-- 1 awong ugrad_ce 1024 Jan 19 19:39 tests


Changing Permissions chmod [ugo]+[rxw] <filename> e.g. chmod u+w rgb.txt gives me permission to change rgb.txt

Changing Ownership chown <user>.<group> <filename> e.g. chmod awong.iuns rgb.txt makes rgb.txt owned by iuns group

-r--r--r-- 1 awong ugrad_ce 17375 Apr 26 2000 rgb.txt -rw-r--r-- 1 awong ugrad_ce 17375 Apr 5 02:57 set10.csv drwxr-xr-- 1 awong ugrad_ce 1024 Jan 19 19:39 tests


What they do: files: read – Allows you to read the file write – Alows you to modify the file execute – Allows you run the file as a script or binary program

directories: read – lets you get a directory listing of files write – lets you add or remove files from directory execute – lets you access files in the directory

user group other

• .login, .bash_login, .mylogin, .profile, .bash_profile One of these scripts get run at startup…. If you use bash, you should create a .bash_profile so that you know what is getting run.

• .logout, .bash_logout One of these gets run when you logout or quit a shell

• .cshrc, .mycshrc, .bashrc One of these get run you start a new shell (as opposed to logging in. An example would be when you start a new xterm). .cshrc and .mycshrc are run if you use tcsh and .bashrc is run if you use bash.

• .xsession This gets run if you login via xdmcp. This is the only thing that gets run. It must be set as executable (chmod u+x) to work.

Login Scripts (what happens at startup)

What is input/output redirection? Normally, a program’s standard output is

displayed on user’s terminal, and its standard input comes from the keyboard.

Redirecting the output of a program means asking the shell to put the program’s output (stdout [C++’s cout]) into a file.

Redirecting the input of a program means asking the shell to feed a file as the program’s standard input (stdin [C++’s cin]).

Note: redirection works with files.

Why redirect program’s output? You may want to save output and examine it

at your leisure: if the program generates a lot of output if the program takes a long time to run

You may want to present the output to another person

Having the program write to standard output may make the program simpler to write

Standard output vs Standard error By convention, “normal” output of a program

is sent to standard output (stdout [C++’s cout]), while debugging or error output is sent to standard error (stderr [C++’s cerr]).

How to redirect program’s output?

To redirect just the standard output:<program> > <FILE>

To redirect just the standard error:sh/ksh/bash: <program> 2> <FILE>csh/tcsh: ( <program> > STDOUT ) >& STDERR

To redirect both standard output and standard error:csh/tcsh/bash: <program> >& <FILE>sh/ksh/bash: <program> > <FILE> 2>&1

> vs. >>Both > and >> will create the output file, if it

doesn’t already exist If the file does exist, then:

Using > to redirect output will overwrite the output file:

ls > newlisting printenv > my_environment

Using >> to redirect output will append to the output file

cat ch1 ch2 ch3 > book cat ch4 ch5 ch6 >> book

Why redirect program’s input? To run the program repeatedly with the same

(or similar input) Having the program read from standard input

may make the program simpler to write.

How to redirect program’s input?

Simple!<program> < <FILE>

Examplesort < my_grades.txthead < really_long_book.txt

Piping Piping is connecting programs together by using

the output of one program as the input to the next. Syntax:

<program1> | <program2> | … | <programN> A simple example (view a sorted file-listing a page

at a time):ls | sort | less

Note: piping deals with the input/output of programs (that is, stdin, stdout, stderr)

Why piping? Because “the whole is bigger than the sum of its

parts. By combining Unix utilities in a pipeline, you can

build tools “on-the-fly” as you need them.

Piping examples How many .c files are in this directory?ls *.c | wc –l

What files were modified most recently?ls –t | head

What processes am I running?ps auxw | grep <mylogin>

Make an alias for the above, for other Linux boxen too:alias myps=’ ps auxw | grep `id –un`’

The cat utilityEspecially useful: cat

When used with pipes, “copies” stdin to stdout

cat can be used to redirect out of a pipe! Example: make a file containing currently running

processes ps aux | grep evgenyr | cat > my_processes

When used with a file, “copies” the file into stdout cat can be used to place a file’s contents into a pipe! Example: list all the items in a list in alphabetical ordercat my_grocery_list.txt | sort | uniq

Command substitution(aka “what’s up with the backquotes?”)

Command substitution means that the shell substitutes a command’s output for the command itself.

To get the shell to perform command substitution, enclose the command in backquotes (`)

Next stop: utilities (No, not electricity, water, and sewer)diff and patchgrepfind and xargs

diff and patch You have two versions of a file; how do you see

what’s changed between the two versions? diff shows the differences between two files; you’ll

want to use the –u or –c option to diff to produce output in human-friendly format:diff –u my_file my_file.orig | less

patch applies differences to a filediff –u my_file my_file.orig > my_patchpatch my_file < mypatch

my_file is now the same as my_file.orig

grep – search for patterns grep searches for patterns in texts:grep <string> <FILE>

grep uses regular expressions to describe the string(s) to search for

In a regular expression, most characters are treated as-is, but a few are magic Ordinary characters just represent themselves Magic characters do special things

grep’s magic charactersA few different kinds of magic characters:

Some characters “anchor” (parts of) a regular expressions to specific places in the string:^ - The beginning of a line$ - The end of a line

A dot (.) matches “any one character whasoever” (except for newline)

[ ] form a character class: [aeiou] – any single vowel [a-zA-z0-9] – Any single letter or digit [^0-9] – Any single character that isn’t a digit

Here, ‘^’means “not”

Even more of grep’s magic characters

Quantifiers say how many times the preceeding “thing” should be repeated:* means “zero or more times”? Means “zero or one time”

Frequently used grep options

-i : do case-insensitive search

-n : print line numbers

-v : print lines that do not match

-l : only list the files from which output would have been printed

grep examplesPrint all non-blank lines:grep –v ’^$’ my_file.txt

Print all the lines on which my_function is called (or declared):grep –n ’my_function *(’ my_code.c

Show all the xterms I am running:ps auxw | grep ”`id –un` .*xterm”

find

Traverse the tree(s) rooted at <PATHs> , and for each “thing” found, evaluate the <EXPRESSION> until the result of <EXPRESSION> is known.

The evaluation of <EXPRESSION> “short-circuits”, just like expressions in C/C++. false && some_expression can never be true

Options apply to the entire find command, not the individual expression

find <PATHS> <OPTIONS> <EXPRESSION>

Components of a find expression

An expression is zero or more primaries connected by operators

Two kinds of primaries: Tests: just return true or false Actions: do something, in addition to returning true or

false Operators work just as they do in C/C++.

! / -not -a / -and -o / -or , (comma)

Parentheses are used for grouping, just as in C/C++.

find examples Print all the *.c* and *.h* files in and below the current

directoryfind . –name ’*.[ch]*’ –a –print

Show line numbers myfunction calls in the above *.c* filesfind . –name ’*.c*’ | xargs grep –n ’myfunction.*(’

Change permissions on files under your home directory so they’re inaccessible to everyone but you (except for files in the www directory)cd; find . –path ”./www*” –prune –o –exec chmod go-rwx {} \;

How big are my *.c* files?expr `find –name ’*.c*’ –printf ”%k + ”` 0

Read the find manual (info find) for more examples

xargs: combine arguments

Feeds standard input as arguments to <COMMAND> Often used with find

(find … | xargs grep) But it doesn’t have to be used with find:cat filelist | xargs cat {} > concatenated

xargs <OPTION> <COMMAND> <INITIAL_ARGS>

Process ManagementWhat can you do with it?

Start programs in the background Run more than one program per terminal Kill bad and/or crashing programs Suspend programs mid-execution List all jobs running in a shell Move foreground jobs to the background More …

Three States of a Process Foreground

Attached to keyboard Outputs to the screen Shell waits until the process ends

Background, running Not attached to keyboard Might output to the screen Shell immediately gives you another prompt

Background, suspended Paused mid-execution Can be resumed in background or foreground

Background Processes Listing jobs:

jobs lists background “jobs” and job #’s ps lists processes and their process id %<job#> expands to the process id of the job

Stopping foreground jobs Press ^Z (Ctrl-Z) in the terminal window

Starting a process in the background Append a & character to the command line Examples: ls –lR > ls-lR.out &

xemacs my_program.cc & Resuming a stopped job

In the foreground: fg [<pid>] In the background: bg [<pid>]

Killing Processes The “kill” command:

kill [-<signal>] <pid>Send <signal> to process <pid>

The “killall” command:killall [-<signal>] <command>Send <signal> to all processes that start with <command>

Useful signals (kill –l for the complete list):TERM the default, “terminate”, kills things

nicelyKILL will kill anything, but not nicelyHUP “hangup”, used to reload configurationsSTOP stops (suspends) a running process

Programs & ProcessesProgram – file which can be run

Binary executables Shell scripts

Process – A running program Loaded in physical memory Stored in virtual memory (swap)

Virtual MemoryExtends amount of physical memoryUNIX

Size of swap = amount of usable memory Physical memory (RAM) mirrored in swap

Linux: Memory = swap + RAMSwapping

Moving pages to and from memory Page – block (unit) of RAM

Swap AllocationUNIX:

RAM:

Swap:

Linux:

RAM:

Swap:

Which works better when?

Process Components

Sleeping Stopped

Address space mapStatus

OwnerBlocked Signals PriorityResources used

Running Runnable

Zombie

Process OwnershipReal User (User starting the process)

UID: User ID Number GID: Group ID Number

Effective User (Permissions for process) EUID: Effective User ID Number EGID: Effective Group ID Number

Process LifecycleParent process “forks” a childProcess identifiers

PID: Process ID Number PPID: Parent Process ID Number

Child finishes Sends status message to parent Zombie – waits for parent acknowledgment Orphan – init (PID 1) becomes parent

Running a commandType in the command nameShell searched for the command If found, shell “forks” the commandHow is the command found?

PATH environment variablewhich – Displays full path to command Beware of “.” in your PATH

Background JobsForeground process

Returns control to shell after it finished Most common usage

Background process Returns control to shell immediately “&” typed after command name Runs concurrently

DaemonsSpecial programs which performs a

some task without interventionStarted in the backgroud

(often at system startup)Name of most ends in ‘d’

“Daemon” is actually a much older form of “demon”; daemons have no particular bias towards good or evil, but rather serve to help define a person's character or personality.

~Evi Nemeth

fg, bg, & jobsCan switch between modesSend signal to stop a processCommands

fg – Run process in foregroundbg – Run process in backgroundjobs – List shell child processes

SignalsUsed to communicate with processesTerminal signals

<Ctrl><c> - Terminate process<Ctrl><z> - Stop (suspend) process

kill [-signal] pid Sends signal to process PID Default is to terminate the process

Common Signals

## NameName DescriptionDescription

1 HUP Hangup

2 INT Interrupt

3 QUIT Quit

9 KILL KILL

15 TERM Software Termination

STOP Stop

Process PriorityHow much CPU time granted relative to

other processesnice – Sets the priority value at startrenice – Changes priorityRanges

SystemSystem RangeRange

Solaris 0 to 39

RedHat -20 to 20

System Load

Average number of “Runnable” processes

Measure of how busy the system isPerformance deteriorates at loads of 6+Good for creating a baselineuptime – Load average

(5, 10, 15 minutes)

Monitoring Processesps

BSD – Sorts by %CPU Usage SVR4 – Sorts by PID

top Full terminal screen display Sortable Can kill and renice processes

gtop – Gnome top

Information from ps PID USER PRI/NI - Priority/Nice level RSS/SIZE - Resident/Total Memory used STAT - Process State %CPU/%MEM - % of System CPU/Memory TIME - CPU time used Command

/proc Filesystem Pseudo filesystem Interface to memory-related kernel data

structures Each PID has it’s own directory

cwd – Symlink to current working directory exe – Symlink to program cmdline – command line options environ – command line environment stat – process status

Runaway ProcessesCan eat up your resourcesLists users and PIDs accessing a

resourcefuserlsof

Kill the process and clean up

• When you run a command, it starts a new “process” • Each process has a unique number called the PID (Proccess ID)• Unix is a multitasking operating system, so you can run multiple processes simultaneously• After you start a process, it is usually in what is called the “foreground.” That is, it takes over your shell.• You can suspend processes in the foreground with Ctrl-Z• The process is now frozen. You can pull it to the foreground again by typing “fg”.• Alternately, you can make it keep running, but put it in the background so can still use the same shell by typing “bg”.• You can also start a task in the background by putting a & at the end of the command.

Processes 1: suspend, background, foreground

• You can list all the processes that have been run from the current shell with “ps” • To list all the processes on the system, do “ps awux”

root 3723 0.0 0.1 3092 988 pts/20 S 16:21 0:00 –bashawong 3724 0.0 0.1 1406 712 pts/17 R 16.35 0:00 -bashawong 3725 0.0 0.1 2334 716 pts/17 R 16.36 0:00 ps awux

Processes 2: ps, kill, kill -9

If a process of yours freezes, you can login to the same machineagain, do a “ps awux” and find the process number and then kill it.

• If you want to end a process, you can do “kill <pid>” eg. kill 3724

• If that doesn’t kill it, you can do the “super kill,” “kill -9 <pid>” eg. kill –9 3724

• Sometime you want to know what process is taking up the most cycles on a machine. The program that tells you this is “top” • Another measurement utility for system load is “uptime.” uptime tells you how long the system has been running and how many processes are concurrently screaming for attention. The last three numbers tell you how the average number of processes screaming for attention over the last 1 minute, 5 minutes and 10 minutes, respecetively.

4:49pm up 1:15, 19 users, load average: 0.10, 0.11, 0.09

System State: uptime, top, nice

• If you are going to run a long, cpu intensive process, it is often a good idea to “nice” it. This puts it at a slighly lower priority so thatquick, interactive tasks (like checking e-mail) won’t get starved.To do “nice” a process, just do “nice <command as you would>”

What are environment variables? The environment is an array of strings of the formNAME=VALUE

Each process (program) has its own copy of the environment

Processes started by the shell get a (“deep”) copy of the shell’s current environment

Each process can examine and modify its own (and only its own) environment

The “meaning” of environment variables is merely a matter of established conventions

Viewing your shell’s environment To view a single environment variable’s value:

echo $<name> For example:

echo $HOMEwill display

/homes/iws/evgenyr

To view all environment variables at once: tcsh/csh/bash: printenv sh/ksh: set

Setting environment variables tcsh/csh:

setenv <name> <value> Example: setenv PRINTER ps329

bash/ksh: export <name>=<value> There is no space on either side of ‘=’! Example: export PRINTER=ps329

sh: <name>=<value>; export <name>

!

Appending to environment variables

Appending /uns/bin/ to your PATH bash/ksh: export PATH=$PATH:/uns/bin

tcsh/csh: setenv PATH=$PATH:/uns/bin

Prepending is similar: bash/ksh: export INFOPATH=/uns/info:$INFOPATH

tcsh/csh: setenv INFOPATH /uns/info:$INFOPATH

Common environment variables The “meaning” of environment variables is

purely a matter of convention. However, these environment variables are

quite common: PATH, INFOPATH, MANPATH, EDITOR, VISUAL, PAGER, HOME, MAIL, USER

The man page for a program will usually list the environment variables the program pays attention to.

Environment VariablesWhat are environment variables? Think of them as parametersthat get passed to your programs. All operating systems havethem. They give information about the context within which a program runs. The most commonly used environment variableis the PATH variable which tells the shell where to look for programs.

To create an environment variable in bash, type: export VARNAME=value

To create an environment variable in tcsh type: setenv VARNAME value

Don’t forget the export or the setenv, otherwise you create a “Shell variable” instead of an environment variable.

Useful shell features (a preview of upcoming attractions)

AliasesRedirecting input and outputCommand substitutionScripts

Shell as a work-saver: aliases Aliases: textual substitution, similar to

C-preprocessor macros. Syntax:

bash/ksh: alias <text>=’<replacement>’

tcsh/cshalias <text> ’<replacement>’

When you type <text>, the shell “substitutes” <replacement>

Typing alias by itself lists all your current aliases unalias <text> removes the alias Check the csh/tcsh man page for extra features

Alias examples Always print postscript files double-sided

tcsh/csh:alias lpr ’lpr –Zduplex’

bash/ksh: alias lpr=’lpr –Zduplex’

Collect a few tree-friendly options to enscript: tcsh/csh:alias print \ ’enscript –2rhB –SDuplex=DuplexNoTumble’

Line continuation character, just like in C/C++!

Shell as a work-saver: scripts Instead of typing the same series of commands over

and over again, put them in a file and have the shell execute the (commands in the) file!

The file must have execute permission The first line of the file should be:#! <YOURSHELL> (e.g., /bin/bash)

There must be no space (or any other character) between ‘#’ and ‘!’.

Whether to put space after ‘!’ is a matter of style Shell also has control flow, tests, etc. The shell

tutorial on the ACM web page goes into much more detail.

Why Learn Scripting?/bin/sh always thereUnderstand/Modify Existing System

Scripts RC Startup Scripts Daemon Startup Scripts

Automate repetitive tasksGreat for small tools

Writing/Debugging ScriptsKnow how to do job manually firstWrite in small (logical) partsTest each part before continuing

Print modified variables Print command lines to be executed Never work on production data

Comment code!!!

Starting ScriptsPassed as input to a shell

Example: /bin/bash mysript

Run directly from the command line First line of script (i.e: #!/bin/bash) Script must have execute bits set$PATH considerations

Commenting codeAnything following ‘#’ is considered a

comment and ignoredScript Header

Tells what the script does Common Entries:

• Author• Date written• Program name• Command line usage

• Purpose• Files used• Notes/features• Revision History

Argument Variables

VariableVariable PurposePurpose

$0 Name of the program

$1 - $9 Command line arguments

$# Number of Arguments

$* All arguments as 1 word

$@ Each argument is a word

Other Special Variables

VariableVariable PurposePurpose

$$ PID number of script

$!PID number of parent process

$?Exit status of script

(Checked by parent)

Basic Input

read variable [ < filename ]

Examples:read ANSread LINE < file

(only reads first line)

Basic output

echo [ -n ] [ -e ] “string”-n : No new line-e : Allow backslash options

Example: echo –n “Enter your name: “

Backslash Options

CharacterCharacter PurposePurpose

\a Alert (bell)

\b Backspace

\c Don’t display trailing newline

\n Newline

\r Carriage return

\t Horizontal Tab

\v Vertical Tab

\\ Backslash

\### ###: ascii value of character

Expression Evaluationexpr command

Math expressions ( +, -, *, /, % ) String expressions (match, substring, etc.) Logic expressions ( &, |, =, etc.)

Example: COUNT=`expr $COUNT + 1`

Flow ControlNeed to determine which commands to

run based on one or more conditionsUseful to catch errorsProvides flexibility in codeClassic if-then-else model

If-Then condition

Basic decision construct

if condition

then

commands

fi

String Tests

ExpressionExpression True if…True if…

-z string Length of string is 0

-n string Length of string is not 0

string1 = string2 If the 2 strings are identical

string1 != string2 If the 2 strings are different

string If string is not NULL

Numeric Tests


int1 –eq int2 int1 equals int2

int1 –ne int2 int1 and int2 not equal

int1 –gt int2 int1 > int2

int1 –ge int2 int1 >= int2

int1 –lt int2 int1 < int2

int1 –le int2 int1 <= int2

File Tests


-r file File exists and readable

-w file File exists and writable

-x file File exists and executable

-f file File exists and is a regular file

-d file File exists and is a directory

-h file File exists and is a symbolic link

-p file File exists and is a named pipe

File Tests (Con’t.)


-c file File exists and is a character device

-b file File exists and is a block device

-u file File exists and is SUID

-g file File exists and is SGID

-k file File exists and sticky bit set

-s file File exists and size > 0 bytes

Logic Operators

ExpressionExpression PurposePurpose

! Negates the expression

-a AND

-o OR

( expression ) Groups the expression

If-Then-Else condition

if condition

then

commands

else

commands

fi

If-Then-Else If condition

if condition

then

commands

elif condition

then

commands

fi

Case condition

Perform a range of tests on a variable

case variable in

pattern1) commands ;;

pattern2) commands ;;

esac

*) matches all other cases

LoopsNeed to run block of code more than

onceModes

Repeat a set number of times Repeat while a condition is true Repeat until a condition is true

for loop

Repeat for each variable in list

for variable in variable_list

do

commands

done

while loop

Repeat until condition is false

while [ condition ]

do

commands

done

Reading Files

Read a file one line at a time

while read LINE

do

echo $LINE

done < $FILENAME

do-until loop

Repeat until condition is true

until [ condition ]

do

commands

done

Escaping out of loops Sometimes you need to exit loop early

Error conditions Special conditions

break command Example:

while [ 1 ]do

echo –n “Done? “read ANSif [ $ANS = “y” ]thenbreakfi

done

Other scripting languagesPerl

Very popular among sysadmins Many modules available

TCL/TKExpectPython

Intermezzo: quoting Problem: some characters are special to the

shell (*, ?), but you would like to pass those characters to the programs the shell runs as-is.

So you quote the character(s): A single character: by prefixing it with a \ (backslash) A string of characters: by enclosing them in

Single quotes: no characters are special. None Double quotes: some characters (notably $ and `) are still

special. You’ll need to prefix those with a backslash to pass to the program as-is.

Quoting examples List all the files in the current directory:ls *

List the file or directory called ‘*’ (or complain if it’s not there):ls \*

Print $HOME:echo ’$HOME’

Print the path to your home directory:echo ”$HOME”

Print ìd –un`: echo ’ìd –un`’

Print your login:echo ”ìd –un`”

DebuggingThe ideal:

do it right the first time

The reality:

bugs happen

The goal:

exterminate, quickly and efficiently

Debugging Methods The best way: read the code

If you don’t understand it, it’ll happen again

Consistent use of error checking & handling most uncaught errors are not where they appear

to be

Examine the state at key points in the code wrap debug output with #ifdef DEBUG use a debugger, like gdb, to view the state

Efficiency in Debugging

The goal is to isolate the problem and fix it

Don’t try random things, looking for a solution if you don’t understand it…it’ll be back half the time, this takes an insanely long time you don’t learn anything from it

Look for the problem, not the solution figure out two points in code that the problem is

between, and close the gap from there once you know the line the bug is on, it’s easy

Breakpoints What are they?

A “break point”: a point in the code whereexecution pauses, or “breaks”, then continues

What are they for? To examine the state of the running program

How do you make them? modifying code to print information and pause use a debugger to interactively step through code

Debuggers

Advantages over the “old fashioned” way: you can step through code as it runs you don’t have to recompile to set a breakpoint you can examine the entire state of the program

call stack, variable values, scope, etc.

you can modify values in the running program you can view the state of a crash using core files

GDB, the GNU DeBugger Text-based, invoked with:

gdb [<programfile> [<corefile>|<pid>]]

Argument descriptions:<programfile > executable program file<corefile> core dump of program<pid> process id of already running

program

Example:gdb ./hello

Compile <programfile> with –g for debug info

Basic GDB Commands

General Commands:file [<file>] selects <file> as the program to debugrun [<args>]runs selected program with arguments <args>attach <pid>attach gdb to a running process <pid>kill kills the process being debuggedquit quits the gdb programhelp [<topic>] accesses the internal help documentation

Stepping and Continuing:c[ontinue] continue execution (after a stop)s[tep] step one line, entering called functionsn[ext] step one line, without entering functionsfinish finish the function and print the return value

GDB Breakpoints Useful breakpoint commands:

b[reak] [<where>] sets breakpoints. <where> can bea number of things, including a hexaddress, a function name, a linenumber, or a relative line offset

[r]watch <expr> sets a watchpoint, which will breakwhen <expr> is written to [or read]

catch <event> breaks on <event>, which can beused to catch many events, includingthe throwing and catching of C++exceptions

info break[points] prints out a listing of all breakpointsclear [<where>] clears a breakpoint at <where>d[elete] [<nums>] deletes breakpoints by number

Playing with Data in GDB Commands for looking around:

list [<where>] prints out source code at <where>

search <regexp> searches source code for <regexp>backtrace [<n>] prints a backtrace <n> levels deepinfo [<what>] prints out info on <what> (like

local variables or function args)p[rint] [<expr>] prints out the evaluation of <expr>

Commands for altering data and control path:set <name> <expr> sets variables or argumentsreturn [<expr>] returns <expr> from current functionjump <where> jumps execution to <where>

More and More GDB For the most complete and up-to-date

documentation on GDB, use the GDB help subsystem from within gdb. A close second is the GDB info node (info gdb).

There is also an emacs interface to GDB, which allows you to keep the source code open in a separate frame while debugging (documentation for how to use this is available in the GDB info node)

Find more info – on the web

http://www.faqs.org (comp.unix questions FAQ) http://www.google.com http://www.deja.com The support web page:

http://www.cs.washington.edu/lab/ about/ugradcomputing.html

http://www.cs.washington.edu/people/acm/tutorials/

unix

Documents

directory cd

unix commands

current directory example

directory tree

directory synonymous

directory ps

special directory

parent directory