eecs 470 lab 5 - linux shell scripting · unix utilities diff description i...

EECS 470 Lab 5Linux Shell Scripting

Department of Electrical Engineering and Computer ScienceCollege of EngineeringUniversity of Michigan

Friday, 1st February, 2018

(University of Michigan) Lab 5: Scripting Friday, 1st February, 2018 1 / 40

Overview

Administrivia

UNIXFilesUtilitiesConnecting Utilities

Bourne Again ShellVariablesFlow ControlFunctionsGlobbing

Scripting

Assignment


Administrivia

Administrivia

HomeworkI Homework 3 is due Tuesday, 6th February at the beginning of lecture

Projects

I Project 3 is due Sunday, 4th February at 9:00PM

We are available to answer questions on anything here. Office hours can befound in the course google calendar.


https://www.google.com/calendar/embed?src=tcmcrdofi69t5bunjo01onoi1s%40group.calendar.google.com

UNIX

UNIX

What is UNIX?I Mainframe operating systemI Written by Dennis Ritchie, Ken Thompson and Rob Pike (among

others) at Bell LabsI The basis for many modern operating systems, e.g. Linux, BSD, Mac

OSX

History of UNIX

I Written at Bell Labs in 1969I First version of BSD is installed in 1974I Last Bell Labs UNIX (Version 7) is published in 1979I The GNU Project is started by Richard StallmanI Linux Torvalds writes a monolithic kernel operating system in 1991


UNIX

UNIX Philosophy

“This is the Unix philosophy: Write programs that do one thing and do itwell. Write programs to work together. Write programs to handle textstreams, because that is a universal interface.”

– Douglas McIlroy

“Everything is a file descriptor or a process.”– Linus Torvalds


UNIX Files

Files

What is a file?I Anything referenced through a filesystemI Anything with a file descriptor


UNIX Files

Types of Files

I Regular – Anything not in one of the following categoriesI Directory – Can contain other files and directories (read up on inodes

sometime)I Symbolic Link – A pointer to another fileI Pipe – Covered in a few slidesI Socket – Covered in EECS 482/489


UNIX Files

Permissions

r w x - - - - - -User Group Other

Table: UNIX permissions are representedwith one bit for each permission in eachcategory, e.g. 700.

Three PermissionsI ReadI WriteI Execute

Three Categories

I UserI GroupI Other


UNIX Utilities

Utilities

What is a utility?

I A program used to process text streams/filesI Called from some command line/shell

Why do I care?

I Utilities form the basis of “Linux skills”I Useful for automationI Necessary for today’s lab


UNIX Utilities

Navigation Utilities

pwd

I Description: print working directory – where you areI Synopsis: pwd [OPTIONS]

ls

I Description: list directory contentsI Synopsis: ls [OPTIONS]... [FILE] ...

cd

I Description: change directoryI Synopsis: cd [OPTIONS] [PATH]


UNIX Utilities

File Utilities

cp

I Description: copy filesI Synopsis: cp [OPTONS]... SOURCE DEST

mv

I Description: move filesI Synopsis: mv [OPTONS]... SOURCE DEST

rm

I Description: remove filesI Synopsis: rm [OPTONS]... FILE...


UNIX Utilities

diff

Description

I Shows the line-by-line differences between filesI Good for checking if your output is correct

Synopsys

I diff [OPTIONS] FILES

Examples

I diff -uy ../project3_correct/writeback.out writeback.outI vimdiff ../project3_correct/writeback.out writeback.out


UNIX Utilities

Graphical Diff

Alternatives to diffParsing output of diff is hard, so it might be useful to use some kind of“graphical diff” tool, like vimdiff, which opens up two files side-by-side inVim showing their differences. Try it and you’ll see how much easier toparse this.


UNIX Utilities

grep

Description

I Print lines matching a pattern

Synopsys

I grep [OPTIONS] PATTERN [FILE]

Examples

I grep ’@@@’ program.outI ps -axfuw | grep "$USER" | grep "vcs"


UNIX Utilities

Regular Expressions

I Really powerful/usefulI Complicated, and beyond the scope of this presentationI Read up on them

I at WikipediaI in the grep manual


http://en.wikipedia.org/wiki/Regular_expressions

http://www.gnu.org/software/grep/manual/grep.html#Regular-Expressions

UNIX Utilities

man

Description

I An interface to the on-line reference manuals (pager)

Synopsys

I man PAGE

Examples

I man grepI man diff


http://www.gnu.org/software/grep/manual/grep.html

http://www.gnu.org/software/diffutils/manual/diffutils.html

UNIX Utilities

Pager

DefinitionI A program which allows browsing of large text files by breaking them

into screen-sized chunks.

Examples

I lessI man


UNIX Utilities

Other Utilities

These will be useful. . .

I cutI touchI teeI xargs

I tailI columnI findI less

These are harder, but even more useful. . .

I sedI awkI patchI vi(m)

I fmtI tmux


UNIX Connecting Utilities

Program Features

Methods of CommunicationI Standard Text Streams

I stdinI stdoutI stderr

I Return value/code



Return Codes

What is a return code?I The integer value a program returns (e.g. return(0);)I Conventionally, returning zero indicates success, non-zero failureI Specific values other than zero mean different things for different

programs



Standard Text Streams

stdin

I The default input to a programI From a keyboard by default

stdout

I The default output of a programI To a display by default

stderr

I The default error output of a programI To a display by default, requires additional effort to save



Connecting Utilities

Why do we want to connect utilities?

I Combination jobs without intermediate filesI e.g. take the diff of two different grep operations (what you need to

do for today’s lab)

How can we connect utilities?I PipesI Redirection



Pipe

What is a pipe?

I Connects the stdout of one program to the stdin of anotherI Does not connect stderr

How do I use one?I Call a program on one side of the | and then call another on the other

sideI e.g. dmesg | less



Pipe: xargs

ProblemWhat if we want to pipe to utility that uses arguments instead of input?

Solution: xargsxargs splits input into individual items and calls the program that is itsargument once for each input.


http://en.wikipedia.org/wiki/Xargs


Redirection

What is redirection?I Allows for modification of the standard text streams

I stdin is 0I stdout is 1I stderr is 2

I Several types:0< Use a file instead of the keyboard for stdini> Use a file instead of the terminal for the stream i

i>> Like i>, but append to a file instead of overwritingi>&j Put stream i into the same place as stream j

Advanced Bash Scripting Guide: I/O Redirection


http://www.tldp.org/LDP/abs/html/io-redirection.html


Redirection by Example

Example

I ./vs-asm < test_progs/evens.s > program.mem

Example

I make | tee 2>&1 build.out

Example

I ./test 1>&2 2>&3


Bourne Again Shell

Shell

What is a shell?I Before we had graphical environments, we had text shellsI Basically, an interpreter for commands, executing programs and saving

informationI Possibly a Read-Execute-Print-Loop (REPL)


Bourne Again Shell

Bourne Again Shell

What is BASH?I Stands for the Bourne Again ShellI Created in 1989 by Brian FoxI Default shell in most Linux distributions and Mac OSX

Why BASH?

I What I learned firstI Default in CAEN Redhat, finally


Bourne Again Shell

Warning

WarningEverything after this slide will be specific to BASH. Other shells behavesimilarly, but not identically. If you want to use something else (e.g. ZSH,TCSH, etc.), please find other resources.


Bourne Again Shell Variables

Variables

BASH VariablesI Store dataI Contain text, for the most partI No type system

Syntax

I Assignment/DeclarationI variable=value

I ReferencingI $variableI ${variable}



Variable Scope

Scope

I Variables exist inside the shell they’re inI Unless exported

e.g. export EDITOR=vimI This is very important in scripts, particularly the shell startup scripts

(.bashrc, .bash_profile)



Special Variables

$# The number of command line arguments$0 The name of the script/function called$1 The first argument to the script/function$? The return code of the last program run in this shell

$USER The current user$HOME The user’s home directory


Bourne Again Shell Flow Control

Flow Control

if/else

if [ var -eq "string" ]then

commandelif [ var -eq "string2" ]then

command2else

command3fi



Flow Control

case

case "$var" inval)

command;;val2)

command;;( * )

default;;

esac



Flow Control

for

for file in ./*; docommand $filecommand2

done

for (( a=1; a <= LIMIT; a++ )) docommandcommand2

done



Conditionals

Testing

I Testing happens in [ ]I String tests are different than arithmetic testsI Generally use -lt, -gt, -le, -ge, -eq, -neI Tests for files are special, e.g. [ -x simv ] makes sure that the

binary simv has execution permissions


Bourne Again Shell Functions

Functions

FunctionsI Packages of commandsI Arguments are referenced by positionI Useful for packaging up commonly reused bits

Syntax

function (){

commands}


Bourne Again Shell Globbing

File Globbing

What is globbing?

I File name wildcarding in the shellI Expansion is done, and then passed to the command to be executedI e.g. test_progs/*.s

What should I know globbing?

I Superior to parsing ls output in everywayI Can get more complicated, see this page of the BASH manual.


http://www.gnu.org/software/bash/manual/html_node/Pattern-Matching.html#Pattern-Matching

Scripting

Scripting

1. Write a series of shell commands into a text file2. On the first line of the file, specify an interpreter

e.g. #!/bin/bash3. Name it something appropriate

e.g. test.sh4. Add execute permissions

e.g. chmod +x test.sh

5. Run the scripte.g. $ ./test.sh


Assignment

Lab Assignment

I Assignment is posted to the course website as Lab 5 Assignment.I If you get stuck. . .

I Ask a neighbor, quietlyI Put yourself in the help queue

I When you finish the assignment, sign up in the help queue and markthat you would like to be checked off.

I If you are unable to finish today, the assignment needs to be checkedoff by a GSI in office hours by next Thurday.


https://www.eecs.umich.edu/courses/eecs470/Labs/lab5/eecs470lab5assignment.pdf

https://goo.gl/forms/hVQnuF1EUzdXnpbt2

https://goo.gl/forms/hVQnuF1EUzdXnpbt2

eecs 470 lab 5 - linux shell scripting · unix utilities diff description i...

Documents