1

Linux for Researchers

Dr Andrew Richards,Oxford e-Research Centre &Oxford Supercomputing Centre

2

Topics

• What does the operating system do?

• Brief History of UNIX and Linux

• What is Linux ?

• Unix Philosophy

• Comparison to Windows

• Practical Exercises

• Further Information

3

What does the operating system do?

• Manage the computer resources

• Interact with the user

• Run applications

4

Operating Systems?

5

Operating Systems

6

Brief history of UNIX and Linux

• A multi-user, multi-tasking operating system

• 1960s – MIT, AT&T, General Electric develop Multics (Multiplexing Information and Computing Services)

• 1970s – Unics (Uniplexed Information and Computing Service) – renamed to Unix

• 1972 – rewritten in C – resulted in more portable software to different hardware

• 1980 – AT&T licenses UNIX System III – consolidated differences into UNIX System V release 1

• 1983 – AT&T commercialise UNIX System V

– Academia continue to develop BSD Unix as an alternative

– BSD Unix main contributor of TCP/IP network code to the mainstream Unix kernel

• 1990– Open Software Foundation released OSF/1 – standard Unix implementation (Mach and BSD)

• 1991– Linus Torvalds starts Linux (originally Freax). Mixture of BSD and SYS-V

7

What is Linux ?

• Linux is an Operating System– Software loaded into your computer's memory when you switch it on

– Software which manages the hardware

• Linux is “Unix-like”– Unix is an operating system originally developed in the 1970s to run large, central computers

with many users

– Runs everything from phones to supercomputers today

• Features and Benefits– Stability

– Security

– Performance

– Price

– Flexibility & Open Source nature

8

Unix Philosophy

• Unix is based around commands which– Carry out one (or very few) simple tasks

– Carry tasks out very efficiently

– Carry tasks out very quietly

• Commands tend to be typed in a “shell”, the layer between you and the core Linux “kernel”

• More complex tasks are carried out by joining simple commands together:

9

Example

• grep – search for a pattern in a file• sort - sort into order• uniq – only show one copy of identical things

grep OSC *txt | sort | uniq > output.txt

• The above command could search thousands of files for lines containing “OSC” and save one copy of each to a new file (in seconds). What would a Windows user do?

10

More Unix Philosophy

• Treat everything as files and whenever possible, make files plain text and readable by anyone with anything – don't hide or obscure stuff unless it needs to be hidden

– Hierarchical File System

– Top level = /

– /bin

– /sbin

– /dev

– /etc

– /tmp

– …

• Assume that the user knows what they are doing– Comes with it's own perils!

11

Comparison to Windows

• Windows is just another Operating system• Windows is not just Desktop, Server and ‘HPC’ variants• Not Free, Not Open Source• Mostly interacted with through a GUI but there is a ‘shell’ -> ‘CMD’ or PowerShell

• Things to consider:– Installation

– Driver Support

– User Experience

– Development platform? Where will you develop and use your code?

• Read Wikipedia– http://en.wikipedia.org/wiki/Comparison_of_Windows_and_Linux

12

Practical Exercises (Information)

• Logging into a remote system– From a Unix machine (including Mac OS ) – use SSH or GSISSH to access grid resources

– From a Windows machine – use PuTTY (and Xming for X11 support) • http://www.chiark.greenend.org.uk/~sgtatham/putty/

• http://www.straightrunning.com/XmingNotes/• http://sourceforge.net/projects/xming/ (*** Use to Download ***)

• Conventions– Commands are typed in at a “prompt” which is represented by:

– prompt>

 – Parts of a command you should replace (e.g. fill-in-your-name) are in itallics.

13

Remote Login

• ssh

prompt> ssh <username>@<hostname>

prompt> ssh andy@linux.university.ac.uk

prompt> ssh –X andy@linux.university.ac.uk

14

Practical Exercise: Where am I ?

• A Linux system is arranged in a tree like structure with files (leaves) and directories (branches).

• When you do something with files and directories on Linux (making a new file for example), if you do not specify where in the directory “tree” the file should be created, it defaults to your current “location”.

• This will show you your current location

• And with the ‘ls’ command will show the contents of this directory

prompt> pwd

prompt> ls

15

Practical Exercise: Create a Directory

prompt> mkdir training

This will create a new directory called training in your current location

prompt> ls

Now lets move from our home area to the training directory

prompt> pwdprompt> cd trainingprompt> pwdprompt> ls

No content should be listed as this is a new directory with no content!

16

Practical Exercise: Moving around the filesystem

prompt> pwd

To move up one level

prompt> cd ..prompt> pwd

You can also jump to other parts of the filesystemprompt> cd /usr/localprompt> pwdprompt> ls

Now keep trying to go up one directory level until you can't go any higher – you have reached the “root” of the file system – what is it called?

prompt> pwd

17

Practical Exercise: Moving to your home directory

prompt> pwd

To go back to your home directory, there are two short cuts:

prompt> cd

Or

prompt> cd ~

18

Practical Exercise: More moving around

• Pushd and Popd

prompt > pwd

prompt > pushd /tmp

prompt > pwd

prompt > popd

prompt > pwd

19

Practical Exercise: Question

What do you think will happen if you type

prompt> cd ~training

20

Practical Exercise: Permissions (1)

• All users have their own identity on a Linux system and in addition they belong to one or more groups.

• Linux (and Unix) attaches an ownership and set of permissions to every file (including directories) which define what you can and can't do to that file depending on who you are and which groups you are in

• How do we find out what permissions are in place?

prompt> ls –l

The ‘-l’ option is an example of ‘flag’ in Unix

21

Practical Exercise: Permissions (2)

Lets look at the permissions on a specific directory

prompt> ls –l /bin

The permissions of the files are listed in the first column of the output 1st character is the type of file Characters 2-4 are the read/write/execute permissions for the owner of the file Characters 5-7 are the read/write/execute permissions for the group the file belongs to Characters 8-10 are the read/write/execute permissions for everyone else.

-rwxr-xr-x 3 root root 3542 2010-01-20 15:39 zfgrep -rwxr-xr-x 1 root root 1498 2010-01-20 15:39 zforce -rwxr-xr-x 3 root root 3542 2010-01-20 15:39 zgrep -rwxr-xr-x 1 root root 74944 2006-06-16 14:59 zip

22

Practical Exercise: Permissions (3)

First character is “-” so it's a normal file (d means a directory) Characters 2-4 are “rwx” so the user can read and write and execute (run) this file as a

program 5-7 are “r-w” so any other user in the group “root” can read and run the file but not change it 8-10 is also “r-w” so any user in any group can read and run the file but not change it

Permissions can be changed with the commands chmod

prompt> chmod g-w filename

Most common problem on Unix systems is the error message ‘Permissions Denied’

Type User

Group

World Links Owner Owners group

Size Last modification time

Name

- rwx r-x r-x 3 root root 3542 2010-01-20 15:39 zgrep

23

Practical Exercise: HELP !

Help this is getting complicated !!!

All Unix / Linux commands come with their own manual

prompt> man ls

prompt> man chmod

The ‘man’ pages will give further information on ‘flags’ available and often give examples of how to use

Note: - Press ‘q’ to quit a man page

- typing man –k something will search for commands that relate to something

24

Practical Exercise: Wildcards (1)

• As you saw from the /bin directory, some areas contain lots of files.• Many Linux commands support the use of Wildcards

prompt> ls /bin

And then try

prompt> ls /bin/a*

“*” is a substitute for “anything from zero to N characters of any type”. So everything starting with a “a” is listed.

Note: If a directory starts with an “a” then all its files are listed even if they don’t start with an “a”

25

Practical Exercise: Wildcards (2)

Now try

prompt> ls /bin/A*

Is the result the same of different?

Now try

prompt> ls /bin/*a*

Differences ?

26

Practical Exercise: Wildcards (3)

Try the following:

prompt> ls /bin/*aprompt> ls /bin/[a-c]*prompt> ls /bin/[a-c]*f*prompt> ls /bin/*[0-9]*

What is happening in each of the different cases?

27

Practical Exercise: Wildcards (4)

• As well as “*” there is another wild card, “?”. • This means “anything which is is one character long”.

Try the following:

prompt> ls /bin/?a*prompt> ls /bin/*a?prompt> ls /bin/??a*

Can you explain what happens in each case ?

Make sure you understand the difference between “/bin/*a*” and “/bin/?a?”

28

Practical Exercise: More ls

ls and wildcards

prompt > ls

For Directories ?

prompt > ls –d */

prompt > ls –l | grep ^d

29

Practical Exercise: Keyboard shortcuts

Three shortcuts which will speed things up:

1. The up and down arrow keys – these enable you to go back and forth through your history of previous commands

2. The left and right arrow keys, allow you to move along a line so you can change things

3. Tab – this auto-completes command names and file names

30

Practical Exercise: Creating a file with a text editor

• Linux tries to use plain text format files as much as possible• This keeps files simple and readable by many different applications

prompt> cd ~/trainingprompt> lsprompt> nano myfile.txt

Type some text into the window that has appeared.

“The Oxford Supercomputing Centre is a complete High Performance Computing Service”

Now save the file (Ctrl-O) and exit (Ctrl-X).

prompt> lsYou should have a file called myfile.txt

31

Practical Exercise: Looking inside Files (1)

You can use a text editor to look inside a file without changing anything of course but there are quicker (and it turns out, more useful) ways of looking inside text files. To see what's inside a text file:

prompt> cat myfile.txt

(cat is short for concatenate). The problem with “cat” is that big files rapidly scroll off the top of the terminal screen.

prompt> less myfile.txt

A more powerful command is called “more” but that has been supplanted by an even more powerful command named “less”:

Remember!! For more information try ‘man less’

32

Practical Exercise: Looking inside Files (2)

There are even faster ways to look for words in text files.

Use “grep” to look for the word “OSC” on myfile.txt:

prompt> grep super myfile.txt

Note: this will search for ‘super’ exactly as it is written. Case sensitive!

Prompt> grep –i super myfile.txt

This will do the same search but ignore the case of the word

33

Practical Exercise: Copying, Moving and Deleting Files & Directories(1)

Warning: Linux does not warn about deletions

To copy a file, use “cp” and to move or rename it, use “mv”.

prompt> cp myfile.txt myjunk.txtprompt> lsprompt> mv myjunk.txt myold.txtprompt> ls

To delete a file we use the command “rm”.

prompt> rm myold.txt

prompt> ls

Q. What would ‘rm *’ do ?

34

Practical Exercise: Copying, Moving and Deleting Files & Directories(2)

Now make a new directory called “temporary” and then try to remove it using:

1. rm 2. rmdir

rm is for removing files, rmdir for empty directories.

35

Practical Exercise: Making a Shell Script

• Linux is made up from lots of very small commands which perform very specific jobs.

• It is often the case that you need to group lots of commands together into a “recipe” which you use over and over again.

• Such recipes are held in text files and are known as “scripts”.

• There are different flavours of scripts depending on which language they are written in and because we are making a script written with BASH-shell commands, this will be a “shell script”.

36

Practical Exercise: An Example Script

Our script will do two things:

1. List the contents of /etc2. Put the output into a new file called listing.txt

Use “nano” to enter the following script into a file called listing.sh:

# A comment. Other than #!, anything starting with # is ignored

#List /etc and REDIRECT the output to a file called listing.txt

ls /etc > listing.txt

and save the script as “listing.sh”.

There are two ways we can run this script. Method one involves sourcing the script (saying to the shell you're typing at, “please run this text file as a BASH script”.

prompt> . ./listing.sh

The “.” character on its own means “run this” file as a BASH script.

37

Practical Exercise: An Example Script continued…

The second way of doing this is a bit tidier. We place the strange looking line “#!/bin/bash” at the start of the script which forces the script to run as a BASH shell script every time.

Use nano to change the first line of the script so it looks like this

#!/bin/bash

# A comment. Other than #!, anything starting with # is ignored

#List /etc and REDIRECT the output to a file called listing.txt

ls /etc > listing.txt

Now try to run the file directly (so no “.” sign).

prompt> ./listing.sh

You should get ‘Permission Denied’

38

Practical Exercise: Fixing the script to run

• Use the chmod command to add executable permission to listing.sh executable and run it using the line above. Re-read the chmod man page

prompt> chmod a+x listing.shprompt> ./listing.sh

• Now looking inside your new file, “listing.txt” to check the script has worked. Note how “>” redirected the output from a command into a file. It is also possible to redirect files into commands using “<”. This is known as “redirection”

39

Practical Exercise: Other tools, finding tools, adding tools

• which <command>• e.g. which man

• whereis man

Package management (prompt > cat /etc/issue ) (prompt> uname –a )

• Apt (Debian)• apt-get install <package>• apt-cache search <string>

• Yum (Redhat)• yum search <string>• yum install <package>

• Others include yast on SuseLinux, up2date (older Redhat etc. )

40

But its not all Command Line

41

Further Information

• History of UNIX– http://www.levenez.com/unix/

• File system hierarchy standard– http://www.pathname.com/fhs/pub/fhs-2.3.html

• Vi cheat Sheet– http://www.lagmonster.org/docs/vi.html

• X– Xming (use sourceforge version

• FreeNX or NoMachine NX– http://www.nomachine.com/

– http://freenx.berlios.de/