CPS110: Intro to (Operating) Systems Author: Landon Cox Instructor: Jeff Chase August 24, 2009 Syllabus: prerequisites CPS 100 Basic data structures and memory layout Allocating memory on the stack versus from the heap CPS 104 Basic computer architecture, ISAs Registers: stack pointer, PC, general-purpose Virtual memory translation Page tables TLB, caching Other: C/C++ Syllabus: lectures and textbook Lecture notes on the web (125 pages) Exams based on content of lectures Textbooks Only suggested Modern Operating Systems Easy to find on-line New book: Saltzer and Kaashoek Syllabus: discussion sections One section, starting this week F 2:50 4:05 Teaching Assistant Jie Xiao (Jennifer) Undergraduate Teaching Assistants Matt Jacobson Syllabus: homework problems Posted on web on Monday of each week Should be done before discussion section Not graded, but count toward participation Syllabus: projects Where you will learn the most 4 projects 0: very simple intro to C++ 1: build a user-level concurrency package (thread library) 2: build a virtual memory manager 3: hack into a vulnerable system Projects arent long, but are difficult (and exacting) Only 100-1,000 lines/code, but could be many hours Everything is in C++ Project 0 will be posted in a few days Syllabus: project environment Linux/GNU environment You need a CS account Dont have one? Send me. by August 28 Login to linux.cs.duke.edu to submit Syllabus: project groups All projects done in groups of 2 or 3 groups to By Friday (August 28) {name, NetID, CS login} for each member Group members may be asked to rate each other Procedure for firing, quitting in syllabus Syllabus: project auto-grading All projects are auto-graded Immediate feedback Use submit110 script on cs machines One submission/group/day gets feedback Cant use to debug your project + three bonus submissions/group/project Any group members submission counts More on the auto-grader Very narrow feedback: correct or incorrect Doesnt say what is wrong Follow specifications carefully Still have to write a test suite (except P0) Dont rely on auto-grader feedback alone To get more useful feedback Come talk to us! We will provide many office hours every week (double office hours week before a deadline) Syllabus: project timelines Due at 6pm, accepted until 11:59:59pm Auto-grader clock is the one that counts Last submission to auto-grader is final 3 late days/group/semester Intended for unexpected problems No extensions Start early! Syllabus: project collaboration Ok, among groups C++ syntax, course concepts What does this part of the handout mean? Not ok, among groups Design/writing of anothers program Includes prior class solutions How do I do this part of the handout? We use automated similarity-detection software Just changing the variable names wont save you If in doubt, ask me Syllabus: grades, exams Projects: 35% Midterm: 30% Monday, February 23 Final: 30% Wednesday, April 29, 7-10pm Participation: 5% Projects and exams The two are not independent Familiarity with projects is critical to doing well on exams I like to ask questions about projects on exams Extend Project X to include this functionality Know your project! You can assign roles to different people But each member must understand all aspects Syllabus: getting help Newsgroup Office hours With me: Tu, 1:00 3:00, etc With Jie: Tu, 1:30 2:30, Th 4:00-5:00 UTAs Post to the newsgroup And you may also cc: us Questions about the course? Goals for CPS 110 First part: demystify the operating system How does my computer start running? How does a program load into memory? Second part: demystify the Internet How does myknow where to go? Why is Google so fast? Duke curriculum: one view Hardware Assembly language program gates Hardware Assembly language program gates CPS 104 CPS 1,6,100,108 compiling, reading programs off disk, getting program into memory, reading keyboard, starting the computer, saving files, filenames, networking Applications Ideas high-level programming languages Applications Ideas high-level programming languages Whats missing? CPS 110 Systems: A Bigger Picture Programmable platforms to enable sharing of data and resources Textbook example: operating systems Cloud clusters Etc? Systems as Art Thinking about interfaces Consider the Java language and its key word interface What is a Java object? List of methods and collection of internal state What is a Java interface? Set of methods associated with an object that a programmer can call What do those methods do? Invoke code (let the object do work on the callers behalf) Modify the objects public/private state Why are interfaces useful? They provide an abstraction or simplification Callers dont have to know an object exact type or implementation OS terminology Key terms: interface, resource (cpu, mem, etc), abstraction, virtual What is an interface? An interface is a set of primitives or operations Interfaces provide access to resources and/or data What do we mean by abstraction? How resources are presented to a client Can think of as an illusion that makes resources easier to program What does it mean to virtualize something? Provides an abstraction (simple way to manipulate resources) (mostly) disallow direct access to reality/resources Interface and abstraction Virtualization? Standards, wrappers, adapters What is an operating system? Program that runs on CPU, (mostly) like any other Virtual interface should be easier than physical Hardware OS Applications Physical machine Interface Physical machine Interface Virtual machine Interface Virtual machine Interface What is an operating system? Hardware OS Applications Physical machine Interface Physical machine Interface Virtual machine Interface Virtual machine Interface What interface does the hardware present? What interface does the OS present? Instruction set: Load/store, mem, regs Hardware-software stack Hardware OS Applications OS vs user-level programs OS User program Familiar view How do programs start? Tasks outside program? (net recv) How do prevent CPU hogging? User program Alternate view User program OS OS runs first, calls program Programs run until they return control to OS (by themselves or forced by hardware) Then OS calls another program Who calls whom? Functions of the OS 1.Illusionist Makes hardware seem nicer than it really is Examples? Programs seem to have their own CPU AFS: single, unified file system Name data with human-readable names Directories Packets get lost; OS makes net look reliable Disk is slow; OS makes it look fast via caching Functions of the OS 1.Illusionist Makes hardware seem nicer than it really is 2.Government Divides hardware resources among competing programs What hardware resources does the OS manage? Processor Memory Network Disk Functions of the OS 1.Illusionist Makes hardware seem nicer than it really is 2.Government Divides hardware resources among competing programs Taxes programs (OS needs CPU, memory to run) Taken for granted when it works, cursed when it breaks Why study operating systems? Very few of you will ever write one 1.Illusionist, govn functions appear in many domains Google provides the illusion of a single web server How should basketball seats be allocated? 2.Design principles Proper abstractions, caching, indirection Concurrency, naming, atomicity, authentication Protection, resource multiplexing (fairness) How does OS create the illusions we know/love? Hints for designing systems What is a system? Components, interconnections Interfaces, environment Systems do something for their environs Export this behavior via an interface Cleanly divides the world in two Parts of the system + the environment Systems from 10,000 feet Environment aka the client System Component Why is designing systems hard? 1.Emergent properties Cant predict all component interactions Millennium bridge Synchronized stepping leads to swaying Swaying leads to more forceful synchronized stepping Leads to more swaying 2.Propagation of effects 3.Incommensurate scaling 4.Trade-offs Why is designing systems hard? 1.Emergent properties 2.Propagation of effects Want a better ride so increase the tire size Need a larger trunk for the larger spare Need to move the back seat forward Need to make front seats thinner Leads to worse driver comfort than before 3.Incommensurate scaling 4.Trade-offs Why is designing systems hard? 1.Emergent properties 2.Propagation of effects 3.Incommensurate scaling Consider the giant mouse Weight ~ size 3 (volume) Bone strength ~ size 2 (cross section area) An elephant sized mouse is not sustainable 4.Trade-offs Why is designing systems hard? 1.Emergent properties 2.Propagation of effects 3.Incommensurate scaling 4.Trade-offs Waterbed effect Push on one end, and the other goes up Spam filters and smoke detectors False positives vs false negatives Why is designing systems hard? 1.Emergent properties 2.Propagation of effects 3.Incommensurate scaling 4.Trade-offs In the immortal words of HT Kung Systems hard. Must work harder. History of operating systems History dominated by two trends Hardware: better, cheaper, faster Software: sprawl Microsoft embodies tension between these trends MS gained 90% market share by running on cheap hw Supporting all that hardware complicates the OS (3 rd -party drivers responsible for vast majority of crashes) How is Apples strategy different? Jobs chooses the hardware you will run HW-to-app control reduces complexity, choice, discount First phase: single operator One goal: make it work Interactive (user has entire machine to herself) Users sign up, get room for two hours at a time OS is really just a library linked into your program What is wrong with this timeline? CPU utilization is awful Since CPUs were expensive, this mattered Second phase: batch processing Goal: improve CPU, I/O utilization Machine is no longer interactive Users submit program (stack of cards) to queue One job at a time, CPU idle during I/O, I/O idle during CPU OS is a batch monitor + library of services Loads program, runs program, prints results Loads next program Second phase: batch processing Goal: improve CPU, I/O utilization Machine is no longer interactive Users submit program (stack of cards) to queue One job at a time, CPU idle during I/O, I/O idle during CPU What key OS function starts to matter now? Protection: programs must not corrupt monitor Programs must relinquish CPU to monitor Second phase: batch processing Goal: improve CPU, I/O utilization Machine is no longer interactive Users submit program (stack of cards) to queue One job at a time, CPU idle during I/O, I/O idle during CPU Why wasnt protection an issue before? No batch monitor to corrupt Person in lab coat took CPU back from program Third phase: multi-program batch Goal: overlap CPU, I/O When one job is reading from disk, run another job on CPU Use DMA + interrupts to allow background I/O DMA: devices write to program memory Interrupts: devices can tell CPU the I/O is done Job 1 Job 2 Third phase: multi-program batch Goal: overlap CPU, I/O What are the OSs new responsibilities? Switch between processes Manage multiple I/Os across devices Protect processes from each other Job 1 Job 2 Fourth phase: time-sharing Goal: keep efficiency, restore interactivity Key insight: humans are really just slow I/O devices Switch between programs during think-time Job 1 Job 2 Job 3 Increased complexity: Many jobs Outstanding reqs Many job sources 4a: computing as a social medium JCR Licklider, The Computer as a Communication Device Science and Technology, April 1968. Fifth phase: personal computing What are PC operating systems most like? As PC prices dropped, single-operator became feasible OS was again just a library of services (MS-DOS) With one user, do jobs need to time-share? Early PC OSes could only do one thing at a time Everything waited while printing/loading a program (Mac < X) Need protection if Im the only one using the PC? Protect me from myself (or my buggy software) Early PCs provided no protection (why Windows before XP, Mac before X were awful) PC operating systems are basically time-sharing OSes now Operating system complexity Windows XP > 40 million lines of code Most of this code is device drivers (not written by MS) Windows NT took 7 years to develop Only worked well years after it shipped Windows 2000 Shipped with 63,000 potential known defects Hot research area Simplify, automatically find OS bugs OS Complexity Lines of code Vista: 50 million Linux 2.6: 6 million (mostly driver code) Sources of complexity Multiple instruction streams (processes) Multiple interrupt sources (I/O, timers, faults) How can we keep everything straight? Dealing with complexity Program decomposition Functions OO: Classes, types int main () { cout > input; output = sqrt (input); output = pow (output,3); cout