computer architecture lecture 6: pipelining · issues in pipelining: ... t he c ont r ol st r uc t...

Computer Architecture

Lecture 6: Pipelining

Dr. Ahmed Sallam

Based on original slides by Prof. Onur Mutlu

Agenda for Today & Next Few Lectures

Single-cycle Microarchitectures

Multi-cycle and Microprogrammed Microarchitectures

Pipelining

Issues in Pipelining: Control & Data Dependence Handling, State Maintenance and Recovery, …

Out-of-Order Execution

Issues in OoO Execution: Load-Store Handling, …

2

Recap of Last Lecture

Multi-cycle and Microprogrammed Microarchitectures

Benefits vs. Design Principles

When to Generate Control Signals

Microprogrammed Control: uInstruction, uSequencer, Control Store

LC-3b State Machine, Datapath, Control Structure

An Exercise in Microprogramming

Variable Latency Memory, Alignment, Memory Mapped I/O, …

Microprogramming

Power of abstraction (for the HW designer)

Advantages of uProgrammed Control

Update of Machine Behavior

3

Review: A Simple LC-3b Control and Datapath

4

C.4. THE CONTROL STRUCTURE 9

Microinstruction

R

Microsequencer

BEN

x2

Control Store

6

IR[15:11]

6

(J, COND, IRD)

269

35

35

Figure C.4: The control structure of a microprogrammed implementation, overall block

diagram

on the LC-3b instruction being executed during the current instruction cycle. This state

carries out the DECODE phase of the instruction cycle. If the IRD control signal in the

microinstruction corresponding to state 32 is 1, the output MUX of the microsequencer

(Figure C.5) will take its source from the six bits formed by 00 concatenated with the

four opcode bits IR[15:12]. Since IR[15:12] specifies the opcode of the current LC-

3b instruction being processed, the next address of the control store will be one of 16

addresses, corresponding to the 14 opcodes plus the two unused opcodes, IR[15:12] =

1010 and 1011. That is, each of the 16 next states is the first state to be carried out

after the instruction has been decoded in state 32. For example, if the instruction being

processed is ADD, the address of the next state is state 1, whose microinstruction is

stored at location 000001. Recall that IR[15:12] for ADD is 0001.

If, somehow, the instruction inadvertently contained IR[15:12] = 1010 or 1011, the

Simple Design

of the Control Structure

A Simple Datapath

Can Become

Very Powerful

C.2. THE STATE MACHINE 5

R

PC<! BaseR

To 18

12

To 18

To 18

RR

To 18

To 18

To 18

MDR<! SR[7:0]

MDR <! M

IR <! MDR

R

DR<! SR1+OP2*set CC

DR<! SR1&OP2*set CC

[BEN]

PC<! MDR

32

1

5

0

0

1

To 18

To 18To 18

R R

[IR[15:12]]

28

30

R7<! PCMDR<! M[MAR]

set CC

BEN<! IR[11] & N + IR[10] & Z + IR[9] & P

9

DR<! SR1 XOR OP2*

4

22

To 11

1011

JSR

JMP

BR

1010

To 10

21

20

0 1

LDB

MAR<! B+off6

set CC

To 18

MAR<! B+off6

DR<! MDRset CC

To 18

MDR<! M[MAR]

25

27

3762

STW STBLEASHF

TRAP

XOR

AND

ADD

RTI

To 8

set CC

set CCDR<! PC+LSHF(off9, 1)

14

LDW

MAR<! B+LSHF(off6,1) MAR<! B+LSHF(off6,1)

PC<! PC+LSHF(off9,1)

33

35

DR<! SHF(SR,A,D,amt4)

NOTESB+off6 : Base + SEXT[offset6]

R

MDR<! M[MAR[15:1]’0]

DR<! SEXT[BYTE.DATA]

R

29

31

18, 19

MDR<! SR

To 18

R R

M[MAR]<! MDR

16

23

R R

17

To 19

24

M[MAR]<! MDR**

MAR<! LSHF(ZEXT[IR[7:0]],1)

15To 18

PC+off9 : PC + SEXT[offset9]

MAR <! PCPC <! PC + 2

*OP2 may be SR2 or SEXT[imm5]

** [15:8] or [7:0] depending on

MAR[0]

[IR[11]]

PC<! BaseR

PC<! PC+LSHF(off11,1)

R7<! PC

R7<! PC

13

Figure C.2: A state machine for the LC-3b

Review: The Power of Abstraction

The concept of a control store of microinstructions enables the hardware designer with a new abstraction: microprogramming

The designer can translate any desired operation to a sequence of microinstructions

All the designer needs to provide is

The sequence of microinstructions needed to implement the desired operation

The ability for the control logic to correctly sequence through the microinstructions

Any additional datapath elements and control signals needed (no need if the operation can be “translated” into existing control signals)

8

Review: Advantages of Microprogrammed Control

Allows a very simple design to do powerful computation by controlling the datapath (using a sequencer)

High-level ISA translated into microcode (sequence of u-instructions)

Microcode (u-code) enables a minimal datapath to emulate an ISA

Microinstructions can be thought of as a user-invisible ISA (u-ISA)

Enables easy extensibility of the ISA

Can support a new instruction by changing the microcode

Can support complex instructions as a sequence of simple microinstructions

Enables update of machine behavior

A buggy implementation of an instruction can be fixed by changing the microcode in the field

9

Multi-Cycle vs. Single-Cycle uArch

Advantages

Disadvantages

You should be very familiar with this right now

10

Microprogrammed vs. Hardwired Control

Advantages

Disadvantages

You should be very familiar with this right now

11

Can We Do Better?

What limitations do you see with the multi-cycle design?

Limited concurrency

Some hardware resources are idle during different phases of instruction processing cycle

“Fetch” logic is idle when an instruction is being “decoded” or “executed”

Most of the datapath is idle when a memory access is happening

12

Can We Use the Idle Hardware to Improve Concurrency?

Goal: More concurrency Higher instruction throughput

(i.e., more “work” completed in one cycle)

Idea: When an instruction is using some resources in its processing phase, process other instructions on idle resources not needed by that instruction

E.g., when an instruction is being decoded, fetch the next instruction

E.g., when an instruction is being executed, decode another instruction

E.g., when an instruction is accessing data memory (ld/st), execute the next instruction

E.g., when an instruction is writing its result into the register file, access data memory for the next instruction

13

Pipelining

14

Pipelining: Basic Idea

More systematically:

Pipeline the execution of multiple instructions

Analogy: “Assembly line processing” of instructions

Idea:

Divide the instruction processing cycle into distinct “stages” of processing

Ensure there are enough hardware resources to process one instruction in each stage

Process a different instruction in each stage

Instructions consecutive in program order are processed in consecutive stages

Benefit: Increases instruction processing throughput (1/CPI)

Downside: Start thinking about this… 15

Example: Execution of Four Independent ADDs

Multi-cycle: 4 cycles per instruction

Pipelined: 4 cycles per 4 instructions (steady state)

16

Time

F D E W

F D E W

F D E W

F D E W

F D E W

F D E W

F D E W

F D E W

Time

Is life always this beautiful?

UNDERSTANDING PIPELINE

17

The Laundry Analogy

“place one dirty load of clothes in the washer”

“when the washer is finished, place the wet load in the dryer”

“when the dryer is finished, take out the dry load and fold”

“when folding is finished, ask your roommate (??) to put the clothes away”

18

- steps to do a load are sequentially dependent

- no dependence between different loads - different steps do not share resources

Time76 PM 8 9 10 11 12 1 2 AM

A

B

C

D

Time76 PM 8 9 10 11 12 1 2 AM

A

B

C

D

Task

order

Task

order

Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.]

Pipelining Multiple Loads of Laundry

19

Time76 PM 8 9 10 11 12 1 2 AM

A

B

C

D

Time76 PM 8 9 10 11 12 1 2 AM

A

B

C

D

Task

order

Task

order

Time76 PM 8 9 10 11 12 1 2 AM

A

B

C

D

Time76 PM 8 9 10 11 12 1 2 AM

A

B

C

D

Task

order

Task

order

- latency per load is the same

- throughput increased by 4

- 4 loads of laundry in parallel

- no additional resources


Pipelining Multiple Loads of Laundry: In Practice

20

Time76 PM 8 9 10 11 12 1 2 AM

A

B

C

D

Time76 PM 8 9 10 11 12 1 2 AM

A

B

C

D

Task

order

Task

order

Time76 PM 8 9 10 11 12 1 2 AM

A

B

C

D

Time76 PM 8 9 10 11 12 1 2 AM

A

B

C

D

Task

order

Task

order

the slowest step decides throughput


Pipelining Multiple Loads of Laundry: In Practice

21

Time76 PM 8 9 10 11 12 1 2 AM

A

B

C

D

Time76 PM 8 9 10 11 12 1 2 AM

A

B

C

D

Task

order

Task

order

A

B

A

B

throughput restored (2 loads per hour) using 2 dryers

Time76 PM 8 9 10 11 12 1 2 AM

A

B

C

D

Time76 PM 8 9 10 11 12 1 2 AM

A

B

C

D

Task

order

Task

order


PERFORMING PIPELINE

22

An Ideal Pipeline

Goal: Increase throughput with little increase in cost (hardware cost, in case of instruction processing)

Repetition of identical operations

The same operation is repeated on a large number of different inputs (e.g., all laundry loads go through the same steps)

Repetition of independent operations

No dependencies between repeated operations

Uniformly partitionable suboperations

Processing can be evenly divided into uniform-latency suboperations (that do not share resources)

Fitting examples: automobile assembly line, doing laundry

What about the instruction processing “cycle”?

23

Ideal Pipelining

24

combinational logic (F,D,E,M,W) T psec

BW=~(1/T)

BW=~(2/T) T/2 ps (F,D,E) T/2 ps (M,W)

BW=~(3/T) T/3 ps (F,D)

T/3 ps (E,M)

T/3 ps (M,W)

More Realistic Pipeline: Throughput

Nonpipelined version with delay T

BW = 1/(T+S) where S = latch delay

k-stage pipelined version

BWk-stage = 1 / (T/k +S )

BWmax = 1 / (1 gate delay + S )

25

T ps

T/k ps

T/k ps

Latch delay reduces throughput

(switching overhead b/w stages)

More Realistic Pipeline: Cost

Nonpipelined version with combinational cost G

Cost = G+L where L = latch cost

k-stage pipelined version

Costk-stage = G + Lk

26

G gates

G/k G/k

Latches increase hardware cost

Pipelining Instruction Processing

27

Remember: The Instruction Processing Cycle

Fetch

Decode

Evaluate Address

Fetch Operands

Execute

Store Result

28

1. Instruction fetch (IF) 2. Instruction decode and register operand fetch (ID/RF) 3. Execute/Evaluate memory address (EX/AG) 4. Memory operand fetch (MEM) 5. Store/writeback result (WB)

Remember the Single-Cycle Uarch

29

Shift left 2

PC

Instruction memory

Read address

Instruction [31– 0]

Data memory

Read data

Write data

RegistersWrite register

Write data

Read data 1

Read data 2

Read register 1

Read register 2

Instruction [15–11]



Add

ALU result

Zero


MemtoReg

ALUOp

MemWrite

RegWrite

MemRead

Branch

JumpRegDst

ALUSrc


4

M u x

Instruction [25–0] Jump address [31– 0]

PC+4 [31–28]

Sign extend

16 32Instruction [15–0]

1

M u x

1

0

M u x

0

1

M u x

0

1

ALU control

Control

AddALU

result

M u x

0

1 0

ALU

Shift left 2

26 28

Address

PCSrc2=Br Taken

PCSrc1=Jump

ALU operation

bcond


T BW=~(1/T)

PIPELINE DATA PATH

30

Dividing Into Stages

31

200ps

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instruction

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

Address

Data

memory1

ALU result

M u x

ALU

Zero

IF: Instruction fetch ID: Instruction decode/

register file read

EX: Execute/

address calculation

MEM: Memory access WB: Write back

100ps 200ps 200ps 100ps

RF write

ignore for now


Instruction Pipeline Throughput

32

Instruction

fetchReg ALU

Data

accessReg

8 nsInstruction

fetchReg ALU

Data

accessReg

8 nsInstruction

fetch

8 ns

Time

lw $1, 100($0)

lw $2, 200($0)

lw $3, 300($0)

2 4 6 8 10 12 14 16 18

2 4 6 8 10 12 14

...

Program

execution

order

(in instructions)

Instruction

fetchReg ALU

Data

accessReg

Time

lw $1, 100($0)

lw $2, 200($0)

lw $3, 300($0)

2 nsInstruction

fetchReg ALU

Data

accessReg

2 nsInstruction

fetchReg ALU

Data

accessReg

2 ns 2 ns 2 ns 2 ns 2 ns

Program

execution

order

(in instructions)

200 400 600 800 1000 1200 1400 1600 1800

200 400 600 800 1000 1200 1400

800ps

800ps

800ps

200ps 200ps 200ps 200ps 200ps

200ps

200ps

5-stage speedup is 4, not 5 as predicted by the ideal model. Why?

Enabling Pipelined Processing: Pipeline Registers

33 T

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instruction

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

Address

Data

memory1

ALU result

M u x

ALU

Zero

IF: Instruction fetch ID: Instruction decode/

register file read

EX: Execute/

address calculation

MEM: Memory access WB: Write back

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uctio

n

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Data

memory

Address

No resource is used by more than 1 stage!

IRD

PC

F

PC

D+4

PC

E+4

nP

CM

AE

BE

Imm

E

Ao

ut M

B

M

MD

RW

A

ou

t W


T/k ps

T/k ps

Pipeline performance impact

34

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uctio

n

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Instruction fetch

lw

Address

Data

memory

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uctio

n

IF/ID EX/MEM

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX MEM/WB

Instruction decode

lw

Address

Data

memory

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uctio

n

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Instruction fetch

lw

Address

Data

memory

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uctio

n

IF/ID EX/MEM

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX MEM/WB

Instruction decode

lw

Address

Data

memory

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uctio

n

IF/ID EX/MEM

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX MEM/WB

Execution

lw

Address

Data

memory

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uction

IF/ID EX/MEM

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

Data

memory

1

ALU result

M u x

ALU

Zero

ID/EX MEM/WB

Memory

lw

Address

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uctio

n

IF/ID EX/MEM

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write data

Read dataData

memory

1

ALU result

M u x

ALU

Zero

ID/EX MEM/WB

Write back

lw

Write register

Address

97108/Patterson

Figure 06.15

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uction

IF/ID EX/MEM

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

Data

memory

1

ALU result

M u x

ALU

Zero

ID/EX MEM/WB

Memory

lw

Address

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uctio

n

IF/ID EX/MEM

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write data

Read dataData

memory

1

ALU result

M u x

ALU

Zero

ID/EX MEM/WB

Write back

lw

Write register

Address

97108/Patterson

Figure 06.15

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uctio

n

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0

Address

Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

Data

memory

1

ALU result

M u x

ALU

Zero

ID/EX


All instruction classes must follow the same path and timing through the pipeline stages. (compare lw, add)

Any performance impact?

Pipelined Operation Example

35

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uctio

n

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Instruction decode

lw $10, 20($1)

Instruction fetch

sub $11, $2, $3

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uction

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Instruction fetch

lw $10, 20($1)

Address

Data

memory

Address

Data

memory

Clock 1

Clock 2

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uctio

n

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Instruction decode

lw $10, 20($1)

Instruction fetch

sub $11, $2, $3

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uction

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Instruction fetch

lw $10, 20($1)

Address

Data

memory

Address

Data

memory

Clock 1

Clock 2

Instruction

memory

Address

4

0

AddAdd

result

Shift

left 2

Instr

uctio

n

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

3216Sign

extend

Write register

Write data

Memory

lw $10, 20($1)

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Execution

sub $11, $2, $3

Instruction

memory

Address

4

0

AddAdd

result

Shift

left 2

Instr

uction

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Execution

lw $10, 20($1)

Instruction decode

sub $11, $2, $3

3216Sign

extend

Address

Data

memory

Data

memory

Address

Clock 3

Clock 4

Instruction

memory

Address

4

0

AddAdd

result

Shift

left 2

Instr

uctio

n

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

3216Sign

extend

Write register

Write data

Memory

lw $10, 20($1)

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Execution

sub $11, $2, $3

Instruction

memory

Address

4

0

AddAdd

result

Shift

left 2

Instr

uction

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Execution

lw $10, 20($1)

Instruction decode

sub $11, $2, $3

3216Sign

extend

Address

Data

memory

Data

memory

Address

Clock 3

Clock 4

Instruction

memory

Address

4

32

0

AddAdd

result

1

ALU result

Zero

Shift

left 2

Instr

uction

IF/ID EX/MEMID/EX MEM/WB

Write backM u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

M u x

ALURead data

Write register

Write data

lw $10, 20($1)

Instruction

memory

Address

4

32

0

AddAdd

result

1

ALU result

Zero

Shift

left 2

Instr

uctio

n


Write backM u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

M u x

ALURead data

Write register

Write data

sub $11, $2, $3

Memory

sub $11, $2, $3

Address

Data memory

Address

Data

memory

Clock 6

Clock 5

Instruction

memory

Address

4

32

0

AddAdd

result

1

ALU result

Zero

Shift

left 2

Instr

uction


Write backM u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

M u x

ALURead data

Write register

Write data

lw $10, 20($1)

Instruction

memory

Address

4

32

0

AddAdd

result

1

ALU result

Zero

Shift

left 2

Instr

uctio

n


Write backM u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

M u x

ALURead data

Write register

Write data

sub $11, $2, $3

Memory

sub $11, $2, $3

Address

Data memory

Address

Data

memory

Clock 6

Clock 5


Is life always this beautiful?

Illustrating Pipeline Operation: Operation View

36

MEM

EX

ID

IF Inst4

WB

IF

MEM

IF

MEM

EX

t0 t1 t2 t3 t4 t5

ID

EX IF ID

IF ID

Inst0 ID

IF Inst1

EX

ID

IF Inst2

MEM

EX

ID

IF Inst3

WB

WB MEM

EX

WB

steady state

(full pipeline)

Control Points in a Pipeline

37

PC

Instruction memory

Address

Instr

uction


MemtoReg

ALUOp

Branch

RegDst

ALUSrc

4

16 32


0

0Registers

Write register

Write data

Read data 1

Read data 2

Read register 1

Read register 2

Sign extend

M u x

1Write

data

Read

data M u x

1

ALU

control

RegWrite

MemRead


6

IF/ID ID/EX EX/MEM MEM/WB

MemWrite

Address

Data memory

PCSrc

Zero

AddAdd

result

Shift

left 2

ALU

result

ALU

Zero

Add

0

1

M u x

0

1

M u x

Identical set of control points as the single-cycle datapath!!


PIPELINE CONTROL SIGNALS

38

Control Signals in a Pipeline

For a given instruction

same control signals as single-cycle, but

control signals required at different cycles, depending on stage

Option 1: decode once using the same logic as single-cycle and buffer signals until consumed

Option 2: carry relevant “instruction word/field” down the pipeline and decode locally within each or in a previous stage

Which one is better?

39

Control

EX

M

WB

M

WB

WB

IF/ID ID/EX EX/MEM MEM/WB

Instruction

Pipelined Control Signals

40

PC

Instruction memory

Instr

uctio

n

Add


Me

mto

Re

g

ALUOp

Branch

RegDst

ALUSrc

4

16 32Instruction [15–0]

0

0

M u x

0

1

AddAdd

result

RegistersWrite register

Write data

Read data 1

Read data 2

Read register 1

Read register 2

Sign extend

M u x

1

ALU result

Zero

Write data

Read data

M u x

1

ALU control

Shift left 2

Re

gW

rite

MemRead

Control

ALU


6

EX

M

WB

M

WB

WBIF/ID

PCSrc

ID/EX

EX/MEM

MEM/WB

M u x

0

1

Me

mW

rite

Address

Data memory

Address


PIPELINE ISSUES

41

Remember: An Ideal Pipeline

Goal: Increase throughput with little increase in cost (hardware cost, in case of instruction processing)

Repetition of identical operations

The same operation is repeated on a large number of different inputs (e.g., all laundry loads go through the same steps)

Repetition of independent operations

No dependencies between repeated operations

Uniformly partitionable suboperations

Processing an be evenly divided into uniform-latency suboperations (that do not share resources)

Fitting examples: automobile assembly line, doing laundry

What about the instruction processing “cycle”?

42

Instruction Pipeline: Not An Ideal Pipeline

Identical operations ... NOT!

different instructions not all need the same stages Forcing different instructions to go through the same pipe stages external fragmentation (some pipe stages idle for some instructions)

Uniform suboperations ... NOT!

different pipeline stages not the same latency Need to force each stage to be controlled by the same clock

internal fragmentation (some pipe stages are too fast but all take

the same clock cycle time)

Independent operations ... NOT! instructions are not independent of each other

Need to detect and resolve inter-instruction dependencies to ensure the pipeline provides correct results pipeline stalls (pipeline is not always moving)

43

Issues in Pipeline Design

Balancing work in pipeline stages

How many stages and what is done in each stage

Keeping the pipeline correct, moving, and full in the presence of events that disrupt pipeline flow

Handling dependences

Data

Control

Handling resource contention

Handling long-latency (multi-cycle) operations

Handling exceptions, interrupts

Advanced: Improving pipeline throughput

Minimizing stalls

44

Pipeline Stalls

Stall: A condition when the pipeline stops moving

Resource contention

Dependences (between instructions)

Data

Control

Long-latency (multi-cycle) operations

45

DEPENDENCES

46

A computer program

Following the Von Neumann model, the program is a sequence of instructions

L1: Mov AX, 3

Add ax, bx

….

jmp l1

47

Dependences and Their Types

Also called “dependency” or less desirably “hazard”

Dependences dictate ordering requirements between instructions

Two types

Data dependence

Control dependence

Resource contention is sometimes called resource dependence

48

Handling Resource Contention

Happens when instructions in two pipeline stages need the same resource

Solution 1: Eliminate the cause of contention

Duplicate the resource or increase its throughput

E.g., use separate instruction and data memories (caches)

E.g., use multiple ports for memory structures

Solution 2: Detect the resource contention and stall one of the contending stages

Which stage do you stall?

Example: What if you had a single read and write port for the register file?

49

UNDERSTANDING

DEPENDENCES

50

Data Dependences

Types of data dependences

Flow dependence (true data dependence – read after write)

Comes from the running program semantic

Anti dependence (write after read)

Output dependence (write after write)

Which ones cause stalls in a pipelined machine?

For all of them, we need to ensure semantics of the program is correct

Flow dependences always need to be obeyed because they constitute true dependence on a value

Anti and output dependences exist due to limited number of architectural registers

They are dependence on a name, not a value

51

Data Dependence Types

52

Flow dependence r3 r1 op r2 Read-after-Write r5 r3 op r4 (RAW)

Anti dependence r3 r1 op r2 Write-after-Read r1 r4 op r5 (WAR) Output-dependence r3 r1 op r2 Write-after-Write r5 r3 op r4 (WAW) r3 r6 op r7

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uctio

n

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Instruction decode

lw $10, 20($1)

Instruction fetch

sub $11, $2, $3

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uction

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Instruction fetch

lw $10, 20($1)

Address

Data

memory

Address

Data

memory

Clock 1

Clock 2

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uctio

n

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Instruction decode

lw $10, 20($1)

Instruction fetch

sub $11, $2, $3

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uction

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Instruction fetch

lw $10, 20($1)

Address

Data

memory

Address

Data

memory

Clock 1

Clock 2

Sub $10,$2,$3

Instruction

memory

Address

4

0

AddAdd

result

Shift

left 2

Instr

uctio

n

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

3216Sign

extend

Write register

Write data

Memory

lw $10, 20($1)

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Execution

sub $11, $2, $3

Instruction

memory

Address

4

0

AddAdd

result

Shift

left 2

Instr

uction

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Execution

lw $10, 20($1)

Instruction decode

sub $11, $2, $3

3216Sign

extend

Address

Data

memory

Data

memory

Address

Clock 3

Clock 4

Sub $10,$2,$3

Instruction

memory

Address

4

0

AddAdd

result

Shift

left 2

Instr

uctio

n

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

3216Sign

extend

Write register

Write data

Memory

lw $10, 20($1)

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Execution

sub $11, $2, $3

Instruction

memory

Address

4

0

AddAdd

result

Shift

left 2

Instr

uction

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Execution

lw $10, 20($1)

Instruction decode

sub $11, $2, $3

3216Sign

extend

Address

Data

memory

Data

memory

Address

Clock 3

Clock 4

Sub $10,$2,$3

Instruction

memory

Address

4

32

0

AddAdd

result

1

ALU result

Zero

Shift

left 2

Instr

uction


Write backM u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

M u x

ALURead data

Write register

Write data

lw $10, 20($1)

Instruction

memory

Address

4

32

0

AddAdd

result

1

ALU result

Zero

Shift

left 2

Instr

uctio

n


Write backM u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

M u x

ALURead data

Write register

Write data

sub $11, $2, $3

Memory

sub $11, $2, $3

Address

Data memory

Address

Data

memory

Clock 6

Clock 5

Sub $10,$2,$3

Instruction

memory

Address

4

32

0

AddAdd

result

1

ALU result

Zero

Shift

left 2

Instr

uction


Write backM u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

M u x

ALURead data

Write register

Write data

lw $10, 20($1)

Instruction

memory

Address

4

32

0

AddAdd

result

1

ALU result

Zero

Shift

left 2

Instr

uctio

n


Write backM u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

M u x

ALURead data

Write register

Write data

sub $11, $2, $3

Memory

sub $11, $2, $3

Address

Data memory

Address

Data

memory

Clock 6

Clock 5

Sub $10,$2,$3

Example output dependence

53

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uctio

n

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Instruction decode

lw $10, 20($1)

Instruction fetch

sub $11, $2, $3

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uction

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Instruction fetch

lw $10, 20($1)

Address

Data

memory

Address

Data

memory

Clock 1

Clock 2

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uctio

n

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Instruction decode

lw $10, 20($1)

Instruction fetch

sub $11, $2, $3

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uction

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Instruction fetch

lw $10, 20($1)

Address

Data

memory

Address

Data

memory

Clock 1

Clock 2

Sub $11,$2,$10

Instruction

memory

Address

4

0

AddAdd

result

Shift

left 2

Instr

uctio

n

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

3216Sign

extend

Write register

Write data

Memory

lw $10, 20($1)

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Execution

sub $11, $2, $3

Instruction

memory

Address

4

0

AddAdd

result

Shift

left 2

Instr

uction

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Execution

lw $10, 20($1)

Instruction decode

sub $11, $2, $3

3216Sign

extend

Address

Data

memory

Data

memory

Address

Clock 3

Clock 4

Sub $11,$2,$10

Instruction

memory

Address

4

0

AddAdd

result

Shift

left 2

Instr

uctio

n

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

3216Sign

extend

Write register

Write data

Memory

lw $10, 20($1)

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Execution

sub $11, $2, $3

Instruction

memory

Address

4

0

AddAdd

result

Shift

left 2

Instr

uction

IF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

Write register

Write data

Read data

1

ALU result

M u x

ALU

Zero

ID/EX

Execution

lw $10, 20($1)

Instruction decode

sub $11, $2, $3

3216Sign

extend

Address

Data

memory

Data

memory

Address

Clock 3

Clock 4

Sub $11,$2,$10

Instruction

memory

Address

4

32

0

AddAdd

result

1

ALU result

Zero

Shift

left 2

Instr

uction


Write backM u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

M u x

ALURead data

Write register

Write data

lw $10, 20($1)

Instruction

memory

Address

4

32

0

AddAdd

result

1

ALU result

Zero

Shift

left 2

Instr

uctio

n


Write backM u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

M u x

ALURead data

Write register

Write data

sub $11, $2, $3

Memory

sub $11, $2, $3

Address

Data memory

Address

Data

memory

Clock 6

Clock 5

Sub $11,$2,$10

Instruction

memory

Address

4

32

0

AddAdd

result

1

ALU result

Zero

Shift

left 2

Instr

uction


Write backM u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

M u x

ALURead data

Write register

Write data

lw $10, 20($1)

Instruction

memory

Address

4

32

0

AddAdd

result

1

ALU result

Zero

Shift

left 2

Instr

uctio

n


Write backM u x

0

1

Add

PC

0Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

M u x

ALURead data

Write register

Write data

sub $11, $2, $3

Memory

sub $11, $2, $3

Address

Data memory

Address

Data

memory

Clock 6

Clock 5

Sub $11,$2,$10

Example of flow dependence

54

Control Dependence

Question: What should the fetch PC be in the next cycle?

Answer: The address of the next instruction

All instructions are control dependent on previous ones. Why?

If the fetched instruction is a non-control-flow instruction:

Next Fetch PC is the address of the next-sequential instruction

Easy to determine if we know the size of the fetched instruction

If the instruction that is fetched is a control-flow instruction:

How do we determine the next Fetch PC?

In fact, how do we know whether or not the fetched instruction is a control-flow instruction?

55

DEPENDENCES DETECTION

56

Interlocking

Detection of dependence between instructions in a pipelined processor to guarantee correct execution

Software based interlocking

vs.

Hardware based interlocking

MIPS acronym?

Microprocessor without Interlocked Pipeline Stages

57

Approaches to Dependence Detection (I)

Scoreboarding

Each register in register file has a Valid bit associated with it

An instruction that is writing to the register resets the Valid bit

An instruction in Decode stage checks if all its source and destination registers are Valid

Yes: No need to stall… No dependence

No: Stall the instruction

Advantage:

Simple. 1 bit per register

Disadvantage:

Need to stall for all types of dependences, not only flow dep.

58

Scoreboarding

59

Instruction

memory

Address

4

32

0

AddAdd

result

Shift

left 2

Instr

uctio

nIF/ID EX/MEM MEM/WB

M u x

0

1

Add

PC

0

Address

Write data

M u x

1

Registers

Read data 1

Read data 2

Read register 1

Read register 2

16Sign

extend

Write register

Write data

Read data

Data

memory

1

ALU result

M u x

ALU

Zero

ID/EX

Not Stalling on Anti and Output Dependences

What changes would you make to the scoreboard to enable this?

counter for writing operation, not just 1 and 0

60

Approaches to Dependence Detection (II)

Combinational dependence check logic

Special logic that checks if any instruction in later stages is supposed to write to any source register of the instruction that is being decoded

Yes: stall the instruction/pipeline

No: no need to stall… no flow dependence

Advantage:

No need to stall on anti and output dependences

Disadvantage:

Logic is more complex than a scoreboard

Logic becomes more complex as we make the pipeline deeper and wider (flash-forward: think superscalar execution)

61

DATA DEPENDENCE

HANDLING

62

Once You Detect the Dependence in Hardware

What do you do afterwards?

Observation: Dependence between two instructions is detected before the communicated data value becomes available

63

How to Handle Data Dependences

Anti and output dependences are easier to handle

write to the destination in one stage and in program order

Flow dependences are more interesting

Five fundamental ways of handling flow dependences

1. Detect and wait until value is available in register file

2. Detect and forward/bypass data to dependent instruction

Detect and eliminate the dependence at the software level

No need for the hardware to detect dependence (MIPS NOP)

Do something else (same program “reorder”), (different program “fine-grained multithreading”) and No need to detect.

Predict the needed value(s), execute “speculatively”, and verify

64

Right place to eliminate dependency

Which one of the following flow dependences lead to conflicts in the 5-stage pipeline?

65

MEM

WB IF ID

IF

EX

ID

MEM

EX WB

addi r1 - -

addi - r1 -

MEM IF ID EX

IF ID EX

IF ID

IF

addi - r1 -

addi - r1 -

addi - r1 -

addi - r1-

?

Safe and Unsafe Movement of Pipeline

66

i:rk_

j:_rk Reg Read

Reg Write

iFj

stage X

stage Y

dist(i,j) dist(X,Y) ??

dist(i,j) > dist(X,Y) ??

RAW Dependence

i:_rk

j:rk_ Reg Write

Reg Read

iAj

WAR Dependence

i:rk_

j:rk_ Reg Write

Reg Write

iOj

WAW Dependence

dist(i,j) dist(X,Y) Unsafe to keep j moving

dist(i,j) > dist(X,Y) Safe

RAW Dependence Analysis Example

Instructions IA and IB (where IA comes before IB) have RAW dependence iff

IB (R/I, LW, SW, Br or JR) reads a register written by IA (R/I or LW)

dist(IA, IB) dist(ID, WB) = 3

What about WAW and WAR dependence?

What about memory data dependence?

67

R/I-Type LW SW Br J Jr

IF

ID read RF read RF read RF read RF read RF

EX

MEM

WB write RF write RF

Pipeline Stall: Resolving Data Dependence

68

IF

WB

IF ID ALU MEM

IF ID ALU MEM

IF ID ALU MEM

IF ID ALU

t0 t1 t2 t3 t4 t5

IF ID MEM

IF ID ALU

IF ID

Insti

Instj

Instk

Instl

WB

WB

i: rx _ j: _ rx dist(i,j)=1

i

j

Insth

WB

MEM

ALU

i: rx _ bubble j: _ rx dist(i,j)=2

WB

IF ID ALU MEM

IF ID ALU MEM

IF ID ALU MEM

IF ID ALU

t0 t1 t2 t3 t4 t5

MEM

Insti

Instj

Instk

Instl

WB

WB i

j

Insth

ID

IF

IF

IF ID ALU

IF ID i: rx _ bubble bubble j: _ rx dist(i,j)=3

IF

IF ID ALU MEM

IF ID ALU MEM

IF ID ALU

IF ID

t0 t1 t2 t3 t4 t5

IF

MEM

ALU

ID

Insti

Instj

Instk

Instl

WB

WB i

j

Insth

ID

IF

ID

IF

i: rx _ bubble bubble bubble j: _ rx dist(i,j)=4

IF

IF ID ALU MEM

IF ID ALU MEM

IF ID

IF

t0 t1 t2 t3 t4 t5

ALU

ID

Insti

Instj

Instk

Instl

WB

WB i

j

Insth

ID

IF

ID

IF

ID

IF

Stall = make the dependent instruction wait until its source data value is available 1. stop all up-stream stages 2. drain all down-stream stages

Sample Assembly (P&H)

for (j=i-1; j>=0 && v[j] > v[j+1]; j-=1) { ...... }

69

addi $s1, $s0, -1 for2tst: slti $t0, $s1, 0 bne $t0, $zero, exit2 sll $t1, $s1, 2 add $t2, $a0, $t1 lw $t3, 0($t2) lw $t4, 4($t2) slt $t0, $t4, $t3 beq $t0, $zero, exit2 ......... addi $s1, $s1, -1 j for2tst exit2:

3 stalls

3 stalls

3 stalls

3 stalls

3 stalls 3 stalls

Readings

P&H Chapter 4.9-4.11

Smith and Sohi, “The Microarchitecture of Superscalar Processors,” Proceedings of the IEEE, 1995

More advanced pipelining

Interrupt and exception handling

Out-of-order and superscalar execution concepts

70

computer architecture lecture 6: pipelining · issues in pipelining: ... t he c ont r ol st r uc t...

Documents