cs4411 – operating systems practicum december 2, 2011 zhiyuan teo supplementary lecture 5

cs4411 – Operating Systems Practicum

December 2, 2011Zhiyuan Teo

Supplementary lecture 5

•Administrative information

•General implementation hints

Our last lecture…

•A review of things we have done

•Where do we go from here?

•Offline Q&A session

•Deadline for Project 6 has been extended till 11.59pm Sunday, 11 December.

Administrative Information

•The zipfile for Project 6 has been updated on CMS. Please use our new disk.c file.

•CS4410 MP4 is optional for 4411 students.

Administrative Information

•We have been busy catching up on research and final reports.

•One final round of office hours on Monday, 5 December from 3pm – late.

- We promise to honor all regrade requests.- We want to help you!

•These slides generally reveal implementation hints.

Before we begin…

•The only concurrency we will test for is simultaneous writes.

•Focus on correctness first, then performance later.

- You do not have to follow the implementation we describe here!- Consider following the hints only if you are stuck.

- But ideally, we want you to design the file system to be correctly concurrent under any kind of workload.

•inode = “eye node”, not “ee node”.- Don’t embarrass yourself and your instructors!

•mkfs (and fsck) should be minithread programs.

Before we begin…

- You need to compile them as separate programs.- Don’t make mkfs or fsck function calls in your minifile implementation.

•How do the global disk variables work?

Getting started

- They are set inside main(), before minithread_system_initialize() is called.

int main(int argc, char** argv) {

use_existing_disk=0; disk_name = “disk0”; disk_flags = DISK_READWRITE; disk_size = 1000;

minithread_system_initialize(entrypoint, NULL);}

void minithread_system_initialize(proc_t mainproc, arg_t arg) { disk_initialize(&disk);

install_disk_handler(disk_handler);

}

On-disk data structures

•Keep things simple: 1 disk block per item.- One disk block for superblock.- One disk block for each inode.- Don’t share disk blocks.

•Concurrency-related structures should not be on disk.- Reference counters, locks etc.

•Pointers on the disk point refer to disk block number.

The big picture

superblock

magic no.

size of disk

root inode

first freeinode

first freedata block

type

size

direct ptr

direct ptr

indirect ptr

dir inode

name block no.

dir data

name block no.

name block no.

name block no.

direct ptr

direct ptr

indirect ptr

direct ptr

direct ptr

indirect ptr

next free block next free block

file inode

type

size

direct ptr

direct ptr

indirect ptr

data

name block no.

name block no.

file data

free block

The big picture

superblock (blk 0)

magic no.: 4411

size: 1000 blks

root inode: 1

1st free inode: 3

first freedata block

DIR_INODE

size: 3 ents.

dir ptr 1: 100

0

0

dir inode (blk 1)

.. at inode 1

dir data (blk 100)

. at inode 1

abc.txt at inode 2

0

nx free inode: 4 nx free inode: 5

file inode (blk 2)

FILE_INODE

size: 12 bytes

dir ptr 1: 101

0

0

Hello world!

0

0

file data (blk 101)

free block(blk 3)

0…

last free inode block(blk 99)

Superblock

magic number

size of disk

root inode

first free inode

first free data block

superblock

•Use disk block 0 for the superblock.

•Root inode field should contain the value 1.

- Since that inode is located at disk block 1.

Inodes

inode type

size

direct ptr 1

direct ptr 2

direct ptr n

inode

•You can use the same structure for file and directory inodes.

•Size: number of directory entries if inode is a directory inode.

indirect ptr

•Size: number of bytes of a file if inode is a file inode.

Directory data blocks

directory data block

•This is just a table with 2 columns.

•You can’t tell from this table if a certain entry is a file or a directory.

•No indirect pointers in this block.

•Directory data blocks are stored in the region of disk reserved for data blocks.

name inode ptr

name inode ptr

name inode ptr

name inode ptr

name inode ptr

name inode ptr

name inode ptr

name inode ptr

name inode ptr

name inode ptr

- Implement it as two arrays.

- But you can easily access that entry’s inode and look up its type.

Free blocks

ptr to next free block

free block

•Just store an integer that points to the next free block.

•If the next free block says 0, there are no more free blocks after this.

•Use the same data structure for free inodes and data blocks.

•Returning blocks to the free list: prepend and modify superblock.

Data structures for blocks

•You can apply a trick we’ve seen before.

struct superblock {

union {

struct { char magic_number[4]; char disk_size[4];

char root_inode[4];

char first_free_inode[4]; char first_free_data_block[4]; } data;

char padding[DISK_BLOCK_SIZE]; }

}

Data structures for blocks

•You can apply a trick we’ve seen before.

struct inode {

union {

struct { char inode_type; char size[4];

char direct_ptrs[TABLE_SIZE][4]; char indirect_ptr[4]; } data;

char padding[DISK_BLOCK_SIZE]; }

}

Data structures for blocks

•You can apply a trick we’ve seen before.

struct dir_data_block {

union {

struct { char dir_entres[TABLE_SIZE][256]; char inode_ptrs[TABLE_SIZE[4];

} data;

char padding[DISK_BLOCK_SIZE]; }

}

Data structures for blocks

•You can apply a trick we’ve seen before.

struct free_data_block {

union {

char next_free_block[4];

char padding[DISK_BLOCK_SIZE]; }

}

Benefits

•You can cast the struct into a char* and directly use it in disk read and write operations.

- The struct is of size DISK_BLOCK_SIZE, so you will read/write exactly one block.- Use pack and unpack functions to fill/read the fields.

•No need to worry about padding.

•Endianness-independent disk implementation.

•You can use single/double/triple indirect pointers, similar to Linux.

Variations

•You can use a bitmap instead of a free list.

•You may want to use different structures for blocks.

- As long as your file system is reasonable and concurrent.- Describe your implementation in the README file.

•Remember: you don’t have to follow our suggestions.

•Storing names in inodes.

What are not acceptable variations

•Storing directory data or indirect blocks in the inode-reserved section of the disk.

•Constricting free expansion for the number of directory entries or a file’s size.

- But you can assume there will not be more than 232 directory entries.- File sizes will not exceed 232 bytes (4Gb).

•Our suggestion: one ‘big lock’ for metadata accesses that can potentially span multiple inodes.

Concurrency

•One lock per inode for file updates.

•Create some in-memory protection structures.- They have to be dynamically allocated since disk_size is a variable.

- Lock this inode when performing reading/writing, but release it as soon as you can.

•Contains lots of low-level, step-by-step instructions.

Need more implementation hints?

•Credits to Ashik, Christopher and Sean for contributing this gold nugget.

•The Design of the UNIX Operating System, Maurice J. Bach

•You built an OS!

We’ve come to the end.

•Very few students in the world have an opportunity to practice extensive design of real OS components.

•This was not an easy class.

•Hindsight is always perfect.

My parting quote

“Good design comes from experience. Experience comes from bad design.”

-Theodore von Karman

It is our great pleasure to have accompanied you on a part of your academic journey.