computer architecture storage systems prof. jerry breecher

Computer Architecture

Storage Systems

Prof. Jerry Breecher

Chap. 6 - Storage 2

Chapter Overview6.1 Introduction

6.2 Types of Storage Devices

6.3 Busses - Connecting IO Devices to CPU/Memory. Interrupts etc. How is data transferred.

6.5 Reliability, Availability and RAID

Chap. 6 - Storage 3

IntroductionThe Big Picture: Where are

We Now?

We will look at how devices (especially disks) are put together.

We’ll look at how to connect IO devices to the CPU.

And then we’ll look at RAID, the brainchild of Patterson and his buddies.

6.1 Introduction


6.3 Busses - Connecting IO Devices to CPU/Memory.

Interrupts etc. How is data transferred.


Chap. 6 - Storage 4

The Processor

Picture

Chap. 6 - Storage 5

The Processor Picture

Processor/MemoryBus

PCI Bus

I/O Busses

Chap. 6 - Storage 6

Memory

I/O

The Processor Picture

Registers

Cache

Processor

Registers

Cache

Processor

Registers

Cache

Processor

Registers

Cache

Processor

Chap. 6 - Storage 7

Types of Storage Devices

In this section we will:

Take a quick look at how disks work. This is only one example of IO, but we will save networks, tapes, etc. for another course.

6.1 Introduction





8Chap. 6 - Storage

Disk Device TerminologyTypes of Storage Devices

• Purpose:– Long-term, nonvolatile storage– Large, inexpensive, slow level in the storage hierarchy

• Bus Interface:– IDE– SCSI – Small Computer System Interface

– Fibre Channel• Transfer rate

– About 120 Mbyte/second through the interface bus.

– About 5 Mbyte/second off of heads.– Data is moved in Blocks

• Capacity– Approaching 100 Gigabytes– Quadruples every 3 years

(aerodynamics)– Can be grouped together to get terabytes of data.

9Chap. 6 - Storage

Disk Device Terminology

Example: Seagate Cheetah ST336752LC

36 Gigabytes

15,000 RPM

3.6 ms avg seek time.

$699.00


http://www.seagate.com/cda/products/discsales/marketing/detail/0,1121,355,00.html

4 disks, 8 heads (so 8 tracks)

71,000,000 Total Sectors

18,000 cylinders

Average of 4,000 sectors/cylinder or 500 sectors / track (but different amounts on each track.)

MTBF = 1,200,000 hours

10Chap. 6 - Storage

Performance of Magnetic Disks

SectorTrack

Cylinder

HeadPlatter

15,000 RPM = 240 RPS => 4 ms per rev Average rotational latency = 2 ms500 sectors per track => 0.10 ms per sector512 bytes per sector => 5,000,000 MB / s

Response time = Queue + Controller + Seek + Rot + Xfer

Service time


Read Cache

Write Cache

Electronics (controller)

Data Control

Chap. 6 - Storage 11

Busses


Look at various bus mechanisms.In very simple terms, a bus is the

connection between various chips/components in the computer.

The bus is responsible for sending data/control between these various components.

6.1 Introduction


6.3 Busses - Connecting IO Devices to CPU/Memory

6.4 I/O Performance Measures



Interconnect Trends

Network

>1000 m

10 - 1000 Mb/s

high ( 1ms)

lowExtensive CRC

Channel

10 - 100 m

40 - 1000 Mb/s

medium

mediumByte Parity

Backplane

0.1 m

320 - 2000+ Mb/s

low (Nanosecs.)

highByte Parity

Distance

Bandwidth

Latency

Reliability

• Interconnect = glue that interfaces computer system components• High speed hardware interfaces + logical protocols• Networks, channels, backplanes

memory-mappedwide pathways

centralized arbitration

message-basednarrow pathwaysdistributed arbitration

Busses

Connects Machines ChipsDevices


A Computer System with One Bus: Backplane Bus

• A single bus (the backplane bus) is used for:

– Processor to memory communication

– Communication between I/O devices and memory

• Advantages: Simple and low cost

• Disadvantages: slow and the bus can become a major bottleneck

• Example: IBM PC - AT

Processor Memory

I/O Devices

Backplane Bus

Busses


A Two-Bus System

• I/O buses tap into the processor-memory bus via bus adaptors:– Processor-memory bus: mainly for processor-memory traffic– I/O buses: provide expansion slots for I/O devices

• Apple Macintosh-II– NuBus: Processor, memory, and a few selected I/O devices– SCCI Bus: the rest of the I/O devices

Processor Memory

I/OBus

Processor Memory Bus

BusAdaptor

BusAdaptor

BusAdaptor

I/OBus

I/OBus

Busses


A Three-Bus System

• A small number of backplane buses tap into the processor-memory bus

– Processor-memory bus is only used for processor-memory traffic

– I/O buses are connected to the backplane bus

• Advantage: loading on the processor bus is greatly reduced

Processor Memory

Processor Memory Bus

BusAdaptor

BusAdaptor

BusAdaptor

I/O BusBackplane Bus

I/O Bus

Busses


North/South Bridge architectures: separate busses

• Separate sets of pins for different functions– Memory bus – Caches– Graphics bus (for fast frame buffer)– I/O busses are connected to the backplane bus

• Advantage: – Busses can run at different speeds– Much less overall loading!

MemoryProcessor Memory Bus

BusAdaptor

BusAdaptor

I/O BusBackplane Bus

I/O Bus

“backsidecache”

Busses Processor

Director


Bunch of Wires

Physical / Mechanical Characteristics – the connectors

Electrical Specification

Timing and Signaling Specification

Transaction Protocol

What defines a bus?Busses


• Synchronous Bus:

– Includes a clock in the control lines

– A fixed protocol for communication that is relative to the clock

– Advantage: involves very little logic and can run very fast

– Disadvantages:

• Every device on the bus must run at the same clock rate

• To avoid clock skew, busses cannot be long if they are fast

• Asynchronous Bus:

– It is not clocked

– It can accommodate a wide range of devices

– It can be lengthened without worrying about clock skew

– It requires a handshaking protocol

Synchronous and Asynchronous BusBusses


° ° °Master Slave

Control LinesAddress LinesData Lines

Bus Master: has ability to control the bus, initiates transaction

Bus Slave: module activated by the transaction

Bus Communication Protocol: specification of sequence of events and timing requirements in transferring information.

Asynchronous Bus Transfers: control lines (req, ack) serve to orchestrate sequencing.

Synchronous Bus Transfers: sequence relative to common clock.

Busses So FarBusses


• One of the most important issues in bus design:– How is the bus reserved by a device that wishes to use it?

• Chaos is avoided by a master-slave arrangement:– Only the bus master can control access to the bus:

It initiates and controls all bus requests– A slave responds to read and write requests

• The simplest system:– Processor is the only bus master– All bus requests must be controlled by the processor– Major drawback: the processor is involved in every transaction

BusMaster

BusSlave

Control: Master initiates requests

Data can go either way

Arbitration: Obtaining Access to the Bus

Busses


The Daisy Chain Bus Arbitrations Scheme

• Advantage: simple• Disadvantages:

– Cannot assure fairness: A low-priority device may be locked out indefinitely

– The use of the daisy chain grant signal also limits the bus speed

BusArbiter

Device 1HighestPriority

Device NLowestPriority

Device 2

Grant Grant Grant

Release

Request

wired-OR

Busses

Order is:1. Request2. Grant3. Release.


• Even memory busses are more complex than this

– memory (slave) may take time to respond

– it may need to control data rate

Bus Request

Bus Grant

Cmd+AddrR/WAddress

Data1 Data2Data

Simple Synchronous ProtocolBusses

Clock


Address

Data

Read

Request

Acknowledge

Master Asserts Address

Master Asserts Data

Next Address

Write Transaction

t0 t1 t2 t3 t4 t5

t0 : Master has obtained control and asserts address, direction (not read), data. Waits a specified amount of time for slaves to decode target

t1: Master asserts request line

t2: Slave asserts ack, indicating data received

t3: Master releases req

t4: Slave releases ack

Asynchronous Handshake (4-phase)Busses

This is Fig. 6.11


Address

Data

Read

Req

Ack

Master Asserts Address Next Address

t0 t1 t2 t3 t4 t5

t0 : Master has obtained control and asserts address, direction, data

Waits a specified amount of time for slaves to decode target\

t1: Master asserts request line

t2: Slave asserts ack, indicating ready to transmit data

t3: Master releases req, data received

t4: Slave releases ack

Read Transaction

Slave Data

Busses


• All signals sampled on rising edge• Centralized Parallel Arbitration

– overlapped with previous transaction• All transfers are (unlimited) bursts• Address phase starts by asserting FRAME#• Next cycle “initiator” asserts cmd and address• Data transfers happen on when

– IRDY# asserted by master when ready to transfer data– TRDY# asserted by target when ready to transfer data– transfer when both asserted on rising edge

• FRAME# de-asserted when master intends to complete only one more data transfer

EXAMPLE: PCI Read/Write TransactionsBusses


– Turn-around cycle on any signal driven by more than one agent

EXAMPLE: PCI Read TransactionBusses

27Chap. 6 - Storage

How The CPU Talks To The IO

The interface consists of setting up the device with what operation is to be performed-

• Read or Write• Size of transfer• Location on device• Location in memory

Then triggering the device to start the operation

When operation complete, the device will interrupt.

Interfacing I/O To The Processor

I/O instructions (in,out) unique from memory access instructions.LDD R0,D,P <-- Load R0 with the contents found at device D, port P.

Device registers are mapped to look like regular memory:LD R0,Mem1 <-- Load R0 with the contents found at device D, port P.

This works because an initialization has correlated the device characteristics with location Mem1.


CPU

IOC

(1) Issuesinstructionto IOC

memory

(2)

(3)

Device to/from memorytransfers are controlledby the IOC directly.

OP Device Address

target devicewhere commands are

IOP looks in memory for commands

OP Addr Cnt Other

whatto do

whereto putdata

howmuch

specialrequests

(4) IOC interrupts CPU when done


How The CPU Talks To The IO

ROM

RAM

I/O

Virtual Memory Pointing at IO space.

29Chap. 6 - Storage

Memory Mapped I/O

ROM

RAM

I/O


Some physical addresses are set aside.There is no REAL memory at these addresses.Instead when the processor sees these addresses, it knows to aim the instruction at the IO processor.

30Chap. 6 - Storage

Transfer Method 1:Programmed I/O (Polling)

CPU

IOC

device

Memory

Is thedata

ready?

readdata

storedata

yesno

done? no

yes

busy wait loopnot an efficient

way to use the CPUunless the device

is very fast!

but checks for I/O completion can bedispersed amongcomputationallyintensive code



Device Interrupts

• An I/O interrupt is just like the exception handlers except:

– An I/O interrupt is asynchronous

– Further information needs to be conveyed

• An I/O interrupt is asynchronous with respect to instruction execution:

– I/O interrupt is not associated with any instruction

– I/O interrupt does not prevent any instruction from completion

• You can pick your own convenient point to take an interrupt

• I/O interrupt is more complicated than exception:

– Needs to convey the identity of the device generating the interrupt

– Interrupt requests can have different urgencies:

• Interrupt request needs to be prioritized



add $r1,$r2,$r3subi $r4,$r1,#4slli $r4,$r4,#2

Hiccup(!)

lw $r2,0($r4)lw $r3,4($r4)add $r2,$r2,$r3sw 8($r4),$r2

Raise priorityReenable All IntsSave registers

lw $r1,20($r0)lw $r2,0($r1)addi $r3,$r0,#5sw $r3,0($r1)

Restore registersClear current IntDisable All IntsRestore priorityRTI

Ext

ern

al I

nte

rru

pt

PC saved

Disable

All Ints

Superviso

r Mode

Restore PC

User Mode

“In

terr

up

t H

and

ler”

Device Interrupts

• Advantage:– User program progress is only halted during actual transfer

• Disadvantage, special hardware is needed to:– Cause an interrupt (I/O device)– Detect an interrupt (processor)– Save the proper states to resume after the interrupt (processor)


33Chap. 6 - Storage

Transfer Method 2:Interrupt Driven Data Transfer

CPU

IOC

device

Memory

addsubandornop

readstore...rti

memory

userprogram(1) I/O

interrupt

(2) save PC

(3) interruptservice addr

interruptserviceroutine(4)

Device xfer rate = 10 MBytes/sec => 0 .1 x 10-6 sec/byte => 0.1 µsec/byte => 1000 bytes = 100 µsec 1000 transfers x 100 µsecs = 100 ms = 0.1 CPU seconds

User program progress only halted during actual transfer. Interrupt handler code does the transfer.

1000 transfers at 1000 bytes each: 1000 interrupts @ 2 µsec per interrupt 1000 interrupt service @ 98 µsec each = 0.1 CPU seconds

Still far from device transfer rate! 1/2 in interrupt overhead



Delegating I/O Responsibility from the CPU: DMA

• Direct Memory Access (DMA):

– External to the CPU

– Act as a master on the bus

– Transfers blocks of data to or from memory without CPU intervention

CPU

IOC

device

Memory

CPU sends a starting address, direction, and length count to IOC. Then issues "start".

IOC provides handshakesignals for PeripheralController, and MemoryAddresses and handshakesignals for Memory.


35Chap. 6 - Storage

Transfer Method 3:Direct Memory Access

CPU

IOC

device

Memory

Time to do 1000 xfers at 1000 bytes each:

1 DMA set-up sequence @ 50 µsec1 interrupt @ 2 µsec1 interrupt service sequence @ 48 µsec

.0001 second of CPU time

CPU sends a starting address, direction, and length count to DMAC. Then issues "start".

IOC provides handshake signals for PeripheralController, and Memory Addresses and handshakesignals for Memory.

0ROM

RAM

Peripherals

IO Buffersn

Memory Mapped I/O



RAID

Redundant Array of Independent Disks


• Motivate a need to have greater reliability and availability for disk data.

• Look at ways to get this greater reliability.

6.1 Introduction





37Chap. 6 - Storage

Array Reliability

• Reliability of N disks = Reliability of 1 Disk ÷ N

1,200,000 Hours ÷ 100 disks = 12,000 hours 1 year = 365 * 24 = 8700 hours

Disk system MTTF: Drops from 140 years to about 1.5 years!

• Arrays (without redundancy) too unreliable to be useful!

Hot spares support reconstruction in parallel with access: very high media availability can be achieved

Hot spares support reconstruction in parallel with access: very high media availability can be achieved

RAID

38Chap. 6 - Storage

Redundant Arrays of Disks

• Files are "striped" across multiple spindles• Redundancy yields high data availability

Disks will fail

Contents reconstructed from data redundantly stored in the array

Capacity penalty to store it

Bandwidth penalty to update

Mirroring/Shadowing (high capacity cost)

ParityTechniques:

RAID

39Chap. 6 - Storage

Redundant Arrays of DisksRAID 1: Disk Mirroring/Shadowing

• Each disk is fully duplicated onto its "shadow" Very high availability can be achieved

• Bandwidth sacrifice on write: Logical write = two physical writes

• Reads may be optimized

• Most expensive solution: 100% capacity overhead

Targeted for high I/O rate , high availability environments

recoverygroup

RAID

Probabliity of failure (assuming 24 hours MTTR) = 24 / ( 1.2 X 106 X 1.2 X 106 ) = 6.9 x 10-13 = 170,000,000 years

40Chap. 6 - Storage

Redundant Arrays of Disks RAID 3: Parity Disk

P100100111100110110010011

. . .

logical record 10010011

11001101

10010011

00110000

Striped physicalrecords

• Parity computed across recovery group to protect against hard disk failures 33% capacity cost for parity in this configuration wider arrays reduce capacity costs, decrease expected availability, increase reconstruction time• Arms logically synchronized, spindles rotationally synchronized logically a single high capacity, high transfer rate disk

Targeted for high bandwidth applications: Scientific, Image Processing

RAID

41Chap. 6 - Storage

Redundant Arrays of Disks RAID 5+: High I/O Rate Parity

A logical writebecomes fourphysical I/Os

Independent writespossible because ofinterleaved parity

Reed-SolomonCodes ("Q") forprotection duringreconstruction

A logical writebecomes fourphysical I/Os

Independent writespossible because ofinterleaved parity

Reed-SolomonCodes ("Q") forprotection duringreconstruction

D0 D1 D2 D3 P

D4 D5 D6 P D7

D8 D9 P D10 D11

D12 P D13 D14 D15

P D16 D17 D18 D19

D20 D21 D22 D23 P

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Disk Columns

IncreasingLogical

Disk Addresses

Stripe

StripeUnit

Targeted for mixedapplications

RAID

42Chap. 6 - Storage

Problems of Disk Arrays: Small Writes

D0 D1 D2 D3 PD0'

+

+

D0' D1 D2 D3 P'

newdata

olddata

old parity

XOR

XOR

(1. Read) (2. Read)

(3. Write) (4. Write)

RAID-5: Small Write Algorithm

1 Logical Write = 2 Physical Reads + 2 Physical Writes

RAID

43Chap. 6 - Storage

Subsystem Organization

hostarray

controller

single boarddisk

controller

single boarddisk

controller

single boarddisk

controller

single boarddisk

controller

hostadapter

manages interfaceto host, DMA

control, buffering,parity logic

physical devicecontrol

striping software off-loaded from host to array controller

no applications modifications

no reduction of host performance

RAID

Cache

44Chap. 6 - Storage

System Availability: Orthogonal RAIDs

ArrayController

StringController

StringController

StringController

StringController

StringController

StringController

. . .

. . .

. . .

. . .

. . .

. . .

Data Recovery Group: unit of data redundancy

Redundant Support Components: fans, power supplies, controller, cables

End to End Data Integrity: internal parity protected data paths

RAID

45Chap. 6 - Storage

System-Level Availability

Fully dual redundantI/O Controller I/O Controller

Cache &Array Controller

Cache & Array Controller

. . .

. . .

. . .

. . . . . .

.

.

.RecoveryGroup

Goal: No SinglePoints ofFailure

Goal: No SinglePoints ofFailure

host host

with duplicated paths, higher performance can beobtained when there are no failures

RAID


Summary6.1 Introduction






Course SummaryDuring this course, we’ve started to learn about the details of

computer architecture. Items included:

Instruction Sets - especially a glimpse at the Intel instruction set.

Pipelines - the gyrations necessary to speed up the processor.

Memory - the various elements in the hierarchy designed to speed up the effective access to data.

IO - a brief look at disks, busses, and how they are put together.

computer architecture storage systems prof. jerry breecher

Documents