CSE320 Spring 2016 - Homework 4

Due Friday 4/1/2016 @ 11:59pm

Introduction

The goal of this assignment is to become f amiliar with low-level Unix/POSIX system calls related toprocess and job control, f ile access, and IPC (pipes and redirection). You will write a mini-shell with basicoperations (a small subset of Bash’s f unctionality). Expected length of this C program is 1000-2000lines of code (not very long, but the code will be challenging, so start early).

Takeaways

Af ter completing this homework you should:

Understand process execution, pipes, and f orksEnhance your understanding of *nix commands and the command lineWorking with existing C libraries and system callsEnhance your C programming abilities

Working with a partner

You are allowed to have groups of up to two members f or the remaining assignments. If you wish tohave a partner you must email cse320@cs.stonybrook.edu. The email should have the subject:[CSE320] Group Formation . The contents of the email should contain the f ollowing inf ormation:

Each persons f ull name, SBU ID number, and csid.CC your partner on the email.

Page 1 of 18

Group name (this will be used to create a new repository). Group names can not have spaces in them.While we all like to have f un, please don’t pick inappropriate names.

In addition to doing this, the README f ile in your submission should list the same inf ormation aboutboth members who worked on the assignment.

Both group members will be given permissions to access the submission repository.Once you pick a group member you are stuck with this person f or the entirety of theassignment. Make sure you want to work with this person. Once the repository is shared,this is now both of your code, regardless of who does what f or the duration of theassignment.

We will assume this will be your group f or the remainder of the assignments in thisclass. If you no longer want to be a member of a group, please emailcse320@cs.stonybrook.edu and let us know. Leaving a group will close the submissionrepository of the group. T he email should have the title [CSE320] Group Withdrawl .Within the email you should list your name, id#, and the name of the group. When youwithdraw you MUST CC your partner on the email. If you wish to do this you must let usknow WIT HIN T WO DAYS of the assignment posting. If not you are stuck, no exceptionswill be made.

If you don’t have a group member, use this topic on piazza to f ind one.

Helpful References

There are no required readings f or this assignment, but a f ew ref erences explain how shells work insome detail. These ref erences may provide substantial insight into how to complete this assignment.Do NOT copy and paste code f rom these sources into your assignment. Copying code f rom online,any student’s previous project, or the textbook will be considered academic dishonesty.

The GNU C Library. See specif ically Chapter 28.6, “Implementing a Job Control Shell“.Chapter 4 of “The Design and Implementation of FreeBSD second edition”Chapter 8 and 10 in your textbook, “Computer systems of a programmers perspective 3/e”

If you are unsure how something works, you can always ref er back to the bash installed in your Linuxenvironment.

Page 2 of 18

Getting started

First you should f etch and merge the f iles f rom https://gitlab01.cs.stonybrook.edu/cse320/hw4 .git toyour local repo (just like you did in HW0). Place all f iles f or this assignment within the hw4 directory ofyour repository.

The contents of the repo give you the f ile 320sh.c which contains a simple loop which echo’s backwhat you type. It also contains the the f ile launcher.sh which disables signals f or the parent shell(usef ul f or part 2 & 3).

Part I: Core Assignment

Write a C program named “320sh” (f or 320 SHell) that perf orms a subset of commands you’re f amiliarwith f rom other shells like GNU’s Bash. You’re welcome to study the code f or bash, but the code yousubmit must be your own!

When you start your shell, you should be able to type commands such as these and see their output:

$ ./320sh320sh> ls# output from ls320sh> ls -l# verbose output from ls320sh> exit$

Note the # comments represent the output of whatever command your shell is executing. Note that commands like ls are (usually) just programs. There are a f ew built-in commands,discussed below. In general, though, the shell’s job is to launch programs and coordinate their inputand output.

You DO NOT need to re-implement any binaries that already exist, such as ls . You simplyneed to launch these programs appropriately and coordinate their execution.

If you are unsure if something is a built-in or not you can use the command type in your existing Linux

Page 3 of 18

shell.

$ type cdcd is a shell builtin$ type mkdirmkdir is /bin/mkdir$ type exitexit is a shell builtin

Helpful and allowed interfaces

You are welcome to use any standard C version, including C99, C11, gnu99, gnu as well as K&R, ANSI, orISO C.

You will have to parse the command line and then use f ork(2), clone(2), and/or and execve(2) (or f lavorsof exec , such as exece , execle , etc.). Programs you run should take input f rom stdin and outputto stdout and stderr (errors). You will have to study the wait(2) system call and its variants in orderf or your shell to return the proper status codes. Don’t spend time writing a f ull parser in yacc/lex:use plain str* f unctions to do your work, such as strtok(3). You may use any system call (section 2of the man pages) or library call (section 3 of the man pages) f or this assignment, other thansystem(3) .

Note that, by convention, the name of the binary is the f irst argument to a program.Caref ully check in the manual of the exec variant you are using whether you should put thebinary name in the argument list or not.

In general, your selection of libraries is unrestricted, with one important exception: you shouldNEVER use the f unction system , which is really just a wrapper f or another shell. Speaking morebroadly, it is not acceptable to simply write a wrapper f or another shell—you should implement yourown shell f or this assignment. If we f ind that you used the system f unction, you will receive a zero f orthe assignment.

Finding programs

Shells provide a nicer command-line environment by automatically searching common locations f or

Page 4 of 18

commands. For instance, a user may type ls , and the shell will automatically f igure out that thebinary is actually located at /bin/ls . On Linux, the paths to automatically search is stored in theenvironment variable PATH .

You can inspect your environment variables using the printenv command:

$ printenvTERM=xtermSHELL=/bin/bashPATH=/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/home/cse320/bin...

When using PATH , check if the command includes a / character. If so, you may pass this commanddirectly to the exec system call, as the command itself is specif ying a relative or absolute path. If thecommand does not include a / character, then the shell should try each of the values in the PATH list,e.g ,: ls should be checked as /usr/lib/lightdm/lightdm/ls , /usr/local/sbin/ls ,/usr/sbin/ls , /usr/bin/ls , /sbin/ls , /bin/ls , /usr/games/ls , and/home/cse320/bin/ls .

You MUST use the stat system call to check whether a f ile exists, rather than relying onthe more expensive exec system call to f ail.

You may use any exec variant you like f or this assignment, including variants that search the PATHf or you. If you do not implement path searching yourself , be sure to test the case where the userchanges the PATH (as described below), ensuring that the newer PATH value is used.

Each exec variant has pros/cons that you should investigate bef ore choosing.

In general, environment variables are passed f rom the parent through the envp argument to main .Be sure to parse these variables so that you can use them to f ind programs, as well as pass them tochild processes.

The command exit is not a program you’ll execute, but a built-in special program thatshould exit(3) f rom your shell.

To begin, the f ollowing basic f eatures MUST be implemented:

Implement simple command parsing in your shell. Upon reading a line, launch the appropriatebinary, or detect when the command is a special “built-in” command, such as exit . For now,

Page 5 of 18

exit is the only built-in command you need to worry about, but we will add more later in theassignment.

Bef ore waiting f or input, you should write the shell prompt 320sh> to the screen. Af ter eachcommand completes, the shell should print another prompt.

The shell should print output f rom commands as output arrives, rather than buf f ering all outputuntil the command completes. Similarly, if the user is typing input that should go to the runningcommand via stdin, your shell should send these characters as soon as possible, rather thanwaiting until the user types a newline.

We will ref ine the parsing logic in subsequent parts. Hint: you will want to read the inputcharacter by character, as some keystrokes may require action without a newline.

Be sure to use the PATH environment variable to search f or commands. Be sure you handle thecase where a command cannot be f ound.

When you are f inished, your shell should be able to execute simple programs like ls and built-inslike exit .

If a user presses the enter key and there is no command to evaluate then just display the promptagain.

If the user presses the enter key and the shell cannot evaluate the command let the user know

320sh> gshsdksjhsjgshsdksjhsj: command not found320sh>

There’s no such thing as too much white space. Example:

$ ls -l

is equivalent to

$ ls -l

If you build your shell correctly, you should be able to run your assignment f rom HW1 and HW2inside the shell.

Another built-in command you must support is cd to change directory using the chdir(2) systemcall and pwd via getcwd(3) to print the current working directory:

Page 6 of 18

320sh> pwd/home/cse320320sh> ls# shows files in /home/cse320320sh> cd /tmp320sh> pwd/tmp320sh> ls# shows files in /tmp

The working directory can af f ect the interpretation of environment variables, as . , thecurrent working directory, is a valid entry in PATH .

The built-ins that should be implemented are: cd , pwd , echo , set , help , and exit . cd

cd - should change to the last directory the user was in.cd with no argument should go to a user’s home directory (also stored in an

environment variable).handle the targets cd . and cd .. properly.

Every directory includes these f iles if you type ls -a , so this should notrequire special handling.

pwd Print current absolute path of where you are currently located on the f ile system.Use getcwd(3) to assist.

echo Basic echo support; print strings and expand environment variables (mentioned laterin part 2)

set Can modif y existing environment variables and create new ones. Use sentenv(3) toassist.Only need to support set VARIABLE_NAME = VALUE

Example: set PATH = /bin:/usr/bin:.

Use set on Sparky as a baseline (The set program on Sparky is much simpler than theone in bash)

help Print a list of all builtin’s and their basic usage in a single column. (Type help in bashto get an idea)

Page 7 of 18

Now that you can change directories, let’s add some style to the shell. Any self -respecting shell has af ancier command prompt, which includes the working directory.

Add the current working directory to your shell prompt. Rather than simply printing 320sh> ,instead print the current working directory in brackets, like this:

[/tmp] 320sh> ls# shows files in /tmp

Debugging

One f eature which will help with the development of your shell is to add debugging messages, whichcan be enabled when you start your shell.

If you start your shell program 320sh with the -d argument, it should display debugging inf oon stderr .Every command executed should say RUNNING: cmd , where cmd is replaced with the text ofthe command.When a command ends you should say ENDED: cmd (ret=%d) and show it’s exit status.

Part II: Quality of life enhancements

Now that you have a shell which can run some programs, its time to now enhance it so that it can dothings which you would expect f rom a shell. You’ll never notice it until its missing, but being able to usebackspace and the arrow keys is very helpf ul. Also being able to press up and down to see the historyand rerun commands is also a very nice f eature, so you don’t have to keep typing the same things overand over.

Variables and Echo Support

In some sense, a shell actually def ines a simple programming language. Like any self -respectinglanguage, 320sh should have variables. In order to avoid conf usion with commands, our shell will

Page 8 of 18

require all variable names to start with a $ character, and only have either alphanumerical names or asingle “special” character (e.g ., ? , @ , etc.), and are terminated by a space or newline.

For now, we will just add a f ew simple variables, namely the environment variables and a special variableto store the return code ( $? ). You are welcome to add others if you like.

A shell user may use a variable in a command, and the shell will automatically replace the variable withthe value of this variable. Similarly, a user may assign a new value to a variable (including anenvironment variable) using the built-in set commands. A usef ul tool f or debugging variables is theecho command.

[/home/cse320] 320sh> echo $PATH/bin:/usr/bin[/home/cse320] 320sh> set PATH = /bin:/usr/bin:.[/home/cse320] 320sh> echo $PATH/bin:/usr/bin:.[/home/cse320] 320sh> lsfoo.c Makefile[/home/cse320] 320sh> echo $?0[/home/cse320] 320sh> ls /blah/blah: no such file or directory[/home/cse320] 320sh> echo $?1

Add variable support to your 320sh . You should be able to set variables and use them incommands, as illustrated above and echo them back to the terminal.

Test your environment variable support with the printenv binary, which prints all of theenvironment variables and their values. Be sure that, if the user changes an environment variablewithin the shell, the output of printenv ref lects this.

It is OK to treat all variables as environment variables. You may exclude or include $? f rom theenvironment variable list.

Launcher script

Since you will be launching 320sh f rom a parent shell, you need to disable command key capture andjob control on the parent shell. Otherwise, the parent will intercept the Control key, and key sequencessuch as, the arrow keys, backspace, Ctrl+Z or Ctrl+C. We have included a launcher script, invoked as

Page 9 of 18

$ . ./launcher.sh

which will disable job control in the parent shell, so that 320sh will receive control sequences onstandard input appropriately.

You MUST include a single . in f ront of the launcher script. This dot causes the script to be “sourced”,or executed within the parent process, rather than only in the child. By doing this you disable job controlin the parent shell.

Once the launcher script is run, signals are permently disabled in the parent sheel. You mustclose the parent shell and open a new instance to restore them.

Keeping History

A very usef ul f eature of a shell is the ability to keep previously typed commands, and allow a user toeasily re-issue them.

For this f eature, the user should be able to press the up or down arrow and cycle througha buf f er of their previous commands. In other words, if you press an up arrow, you should see aprompt f ollowed by the previous command. If enter is pressed, the command would executeagain as if it was just typed. If you reach the start or end of the history you should just display thef irst or last item.

Add support to clear the command buf f er when the user presses an up or down arrow (similar tohow bash works when you press up and down).

You will also need to run with the launcher script (as described above), as the parent shellbuf f ers input characters until it sees a newline by def ault. The launcher will disable line buf f eringin the parent.

For this part, it is f ine to set a compile-time history length, such as 50 lines. There should be no empty lines saved in history.

In order f or the history to survive af ter the shell exits, most shells will write a f ile in the user’s homedirectory, such as /home/cse320/.320sh_history . For f ull credit, your shell should persistentlystore the user’s command history. You can use environment variables to determine where the user’shome directory is.

Page 10 of 18

Backspace and arrow key support

When the backspace key is pressed, the character should be deleted in-place. When the lef t and rightarrow keys are pressed it should move the cursor lef t or right just how it does in a real shell. You shouldbe able to place characters between existing ones. If you press too f ar lef t or too f ar right, it shouldstop at the appropriate positions. You should not be able to overwrite the prompt. You should notstart typing somewhere in the middle of shell, etc.

Look into using ANSI escape sequences. Another good resource to VT100/ANSI codes is bash hackers.

If you want, you could also redo everything involving output and keys using ncurses. It has abit of a steep learning curve, but you can create some awesome looking shell applications withit.

Command line arguments with spaces

Currently if you were to run a program, and an argument was a string with spaces, your shell wouldconsider that argument as a many dif f erent arguments.

[/home/cse320] 320sh> ./a.out This should be one argument

If you were to split on just spaces this program would have 6 arguments passed to it.

To make it so that the string in the example is treated as one argument you would need to surroundthe string in quotes.

[/home/cse320] 320sh> ./a.out "This should be one argument" thisIsAnother

Support the use of "" around command line arguments, so that the text between "" istreated as one argument.You do not need to implement escaped quotes such as “this is a \” quote character”.

Page 11 of 18

Part III: Output Redirection

One of the most powerf ul f eatures of a Unix-like shell is the ability to compose a series of simpleapplications into a more complex workf low. The key f eature that enables this composition is outputredirection.

Redirection is accomplished by three special characters < , > , and | . You will need to add logic toyour parsing code which identif ies these characters and uses them to identif y shell-level directives,rather than simply passing them to exec .

The f irst two characters can direct input f rom a f ile into a program, and output f rom a program,respectively.

[/home/cse320] 320sh> ls -l > newfile[/home/cse320] 320sh> cat < newfilenewfile320sh...[/home/cse320] 320sh> echo "Hello, World!" > file[/home/cse320] 320sh> cat fileHello, World![/home/cse320] 320sh>

In the example above, the standard output of ls -l is directed to a f ile, named newfile . If this f iledidn’t exist previously, the shell created it.

The ls program does not know it is writing to a f ile, and is not passed the string >newfile as an argument. Instead the shell redirects stdout to the f ile. Similarly, in the secondcommand the contents of newfile are passed to the cat program as its standard input( stdin ).

You are not constrained to just use standard input (handle 0) and output (handle 1) withthese operators. You should be able to put an integer in f ront of the operator to indicateanother handle, such as stderr . You can read more about f ile descriptors here.

[/home/cse320] 320sh> somecommand 2>err.log# This example runs "somecommand" and redirects its stderr to "err.log".

You’ll have to learn how to manipulate f ile descriptors caref ully using system calls such as open, close,

Page 12 of 18

read, write, dup/dup2, and more.

Finally, you can string multiple applications together using the | operator:

[/home/cse320] 320sh> ls | grep '.txt' | wc -l4

In this example, the shell creates three child processes. The f irst reads the contents of the homedirectory and outputs them to the grep program, which searches f or the string '.txt' . The outputof grep , i.e., all f iles with the .txt extension, is then sent to the wc program, which counts howmany lines of input it is g iven (i.e., the number of .txt f iles in the home directory.

Add support f or all three f orms of redirection described above, as well as assigning inputs toarbitrary f ile handles other than stdin and stdout . Be sure to run several test cases f or pipingapplications together, and ensure that termination is handled cleanly.Examples:

ls -l | sort -k2nls -l | grep 2 | wc -l./hw1.out < a.txt > out.txt

Look at the man pages f or grep , sort , wc , head , tail , etc. f or parameters andf lags.

Part IV: Job Control Support

Another usef ul f eature of a shell is the ability to pause and resume execution of a job. In the case of along-running program, it is helpf ul to be able to place it in the “background” — allowing the user toissue more commands interactively while the long-running program continues execution.

Your shell should identif y the special character & , which means that a program should be executed inthe background, returning a shell prompt immediately. The built-in command jobs should list allbackground running jobs, their name, PID, job number, etc. just like bash with their status (running orsuspended). It should also print the exit status code of background jobs that just ended.

Each job can be identif ied by either its process id (PID) or a job ID (JID), which is a positive integerassigned by 320sh . JIDs should be denoted on the command line by the pref ix % . For example, %5denotes JID 5, and 5 denotes PID 5.

Page 13 of 18

In addition to jobs, we will need to add a f ew more built-in commands to make job control usef ul. Thecommand fg (ex: fg 3 ) should make the specif ied job number in your list go to the f oreground (andresume execution if it is not currently running/stopped). The command bg (ex: bg 2 ) should causethe specif ied suspended program to run in the background.

We also need to be able to f orcibly terminate a program. If you press Ctrl+C the f oregroundprogram(s) should be killed. To suspend a running program you pressing Ctrl+Z causes thef oreground program(s) should be suspended and added to the list of jobs (i.e., you send it a SIGTSTPsignal to suspend it; fg sends it a SIGCONT to resume running).

Add support f or job control, including the & character, the built-in commands jobs , fg , bg ,kill , and Ctrl-C and Ctrl+Z . Be sure to run plenty of tests, including handling of piped

applications or scripts.

[/home/cse320] 320sh> ./infiniteloop &[1] (9723) ./infiniteloop &[/home/cse320] 320sh> ./infiniteloop &[2] (9724) ./infiniteloop &[/home/cse320] 320sh> jobs[1] (9723) Running ./infiniteloop &[2] (9724) Running ./infiniteloop &[/home/cse320] 320sh> bg %3%3: No such job[/home/cse320] 320sh> fg %1Job [1] (9723) stopped by signal 20[/home/cse320] 320sh> jobs[1] (9723) Stopped ./infiniteloop &[2] (9724) Running ./infiniteloop &[/home/cse320] 320sh> bg %1[1] (9723) ./infiniteloop &[/home/cse320] 320sh> jobs[1] (9723) Running ./infiniteloop &[2] (9724) Running ./infiniteloop &[/home/cse320] 320sh> quit$

You need to make sure you use the launcher script when doing this part.

Extra Credit

Page 14 of 18

In addition to all the above tasks, you can score lots of extra credit on this assignment. Bef ore youconsider completing any of the extra credit tasks, you should f irst make sure that you have a f ullyworking project. For each extra credit task that you complete, make sure to label it in your README f ileso we know to grade it. All extra credit must work 100%; there is no partial credit.

1. (2 points) Add the built-in command history which dumps the entire saved history to theconsole and the built-in command clear-history which resets the history f ile.

2. (2 points) Support time counting. If you start 320sh with -t , it should count how long eachprogram ran and print stats when the program ends. The output should be something like:

$ 320sh -t320sh> du -sh /usr4.3MB /usrTIMES: real=23.7s user=12.1s sys=7.0s

3. (6 points) Implement the f ollowing additional control sequences

ctrl-a: Moves the cursor to the start of the line.ctrl-k: Deletes all text f rom the position of the cursor in the command buf f er.ctrl-u: Deletes all text bef ore the position of the cursor, not including the current position.ctrl-l: Executes the clear binary.alt-f : Moves the cursor to the start of the next word.alt-b: Moves the cursor to the start of the previous word.

Test them in bash if you don’t understand how they work.

4 . (10 points) Implement recursive lookup of command history

By pressing ctrl-r and then typing it should begin matching text in your history. If there aremultiple matches, you can press ctrl-r to cycle through them. Pressing enter executes thecommand. Pressing the lef t or right arrow keys will f ill the command buf f er with thecurrently matched command, but does not execute it.

5. (10 points) Support f ile g lobbing

File g lobbing should work on both relative and absolute paths. The f ollowing are all valid ways f ileglobbing must work:

Page 15 of 18

The above * operator should work with any command. Using ls above is just asimple example.

Use readdir(2) and getdents(3) as needed to read all f iles f rom the target directory, match themusing strstr(3) and add them to list of args you pass to exec(3). In other words, your shell will beexec-ing a command that’ll be as if you typed the f ull names of all the f iles on the command lineone by one.

6. (10 points) Add support f or tab completion in your shell. If a user types a pref ix of a commandand then hits the Tab key twice, the shell should show all possible commands that match thepref ix. If only one command is possible, the shell should automatically f ill in the rest of thecommand. If all possible commands share subsequent letters, automatically f ill in letters untilthe commands diverge.

Consider using a trie data structure to organize the available commands.

7. (15 points) Add support f or OS X Yosemite 10.10.5. There are Mac servers available f or compilingand testing your shell. Just like Sparky, you will use ssh and scp to access and transf er yourhomework to these machines. You each have a server assigned to you. When you are ready towork on this part of the assignment email cse320@cs.stonybrook.edu with both groupmember’s names and NetIDs f or the domain inf ormation, do not wait until the last minute. Allmachines use your NetID and CS password as the log in inf ormation. Note: you can only accessthese machines while on the Stony Brook intranet.

Hints and Tips

We strongly recommend that you check the return codes of all system calls f rom the verybeginning of your work. This will of ten catch errors in how you are invoking these system calls.

BEAT UP YOUR OWN CODE! Throw lots of junk into your shell and make sure it still works. Goodcode comes through good testing. We wont be nice to it when we test it, you shouldn’t be niceto it when you test it.

Don’t f orget to use version control while you work on the assignment. This will make workingwith your partner easier, and if you mess up something while implementing a new f eature youcan always roll back to a working version.

Page 16 of 18

Hand-in instructions

You are expected to hand in at minimum the f ollowing f iles in your g it submission.

1. All *.c , and *.h f iles you have created f or the assignment2. Makef ile3. README

The README should contain the f ollowing inf ormation: 1. Each partners name and SBU ID#2. Anything relevant to grading your project when errors may occur.3. If you did any Extra Credit tasks label them with [EXTRA CREDIT #] where # is the

number next to the task in the extra credit section.

Your assignment is expected to work on the f ollowing platf orms.

1. Ubuntu 15.10 - 64 bit Desktop Edition2. EXT RA CREDIT : OS X Yosemite 10.10.5

Submitting your assignment

REMEMBER Do not submit at the last minute. We will be using the time stamp of the commitassociated with the tag to determine if your homework assignment is late or not. On top of that wewill NOT GRADE late assignments.

1. Make sure all f iles f or this assignment are in a hw4 directory.2. To tag your submission, make sure f irst that you have pulled f rom the remote and all code

changes are merged. Once everything is merged, push it back to the remote server. Be sure youare done making any and all changes bef ore proceeding. T ags cannot be deleted.

3. Log on to your g itlab account and navigate to the tags page f rom your repository’s main page.4 . Here you can click the green “New Tag” button.5. Enter “hw4 ” f or the tag name, the name of your current branch (typically “master”) and an

optional completion message. Do not put important inf ormation in the message as we will not necessarily see it. Anyrelevant inf ormation to your submission should be in your README

6. To check if you submitted properly, you should now see a “hw4 ” tag in the list of tags f or yourrepository.

Page 17 of 18

When writing your program try to comment as much as possible. Try to stay consistentwith your f ormatting. It is much easier f or your TA and the prof essor to help you if we canf igure out what your code does quickly. If your program doesn’t work correctly the grader maybe able to g ive you partial points, assuming he can read the code that you have written.

Grading appointments

This assignment will be graded by appointment. Appointments will be during the week of April 4 -8. Aschedule f or grading appointments will be posted closer to the deadline.

Page 18 of 18