slide02 parallel computers

8/6/2019 Slide02 Parallel Computers

http://slidepdf.com/reader/full/slide02-parallel-computers 1/44

Parallel ComputerArchitecture



The End of the Road



Advantages of

Multi rocessors• Able to create powerful computers by

simply connecting multiple processors

• More cost-effective than building a high-performance single processor

• Obtain fault-tolerance to carry on thetasks, albeit with degraded performance



4 Decades of Computing



Batch Era (1960s)

• IBM System/360 mainframedominated the corporate

computer centers (10 MBdisk, 1 MB magnetic corememory)

•Typical batch processing

machine

• No connection beyond thecomputer room



Time-Sharing Era (1970s)

• Advancing in ss-memory &ICs spawned the

minicomputer era

• Small, fast, and inexpensiveenough to be spreadthroughout the company at

the divisional level

• Still too expensive anddifficult to use to hand overto end-users

• Time-sharing computing

•Existing 2 kinds:

• centralized data processingmainframes

• time-sharing minicomputers



Desktop Era (1980s)

• PCs were introduced in1977

• Many players (Altairs, Tandy,

Commondore, Apple, IBM,and etc)

• Became pervasive and

change the face of computing

• Along came networkedcomputers (LAN & WAN)



Network Era (1990s)

• Advance network technologiesled to network computingparadigm

• Transition from a processor-centric view of computing to anetwork-centric view

•A number of commercialparallel computers withmultiple processors:

• Shared memory systems

• Distributed memory systems



Four Decades of ComputingFeature Batch Time-Sharing Desktop Network

Decade 1960s 1970s 1980s 1990s

Location Computer Room Terminal Room Desktop Mobile

Users Experts Specialists Individuals Groups

Data Alphanumeric Text, numbers Fonts, graphs Multimedia

Objective Calculate Access Present Communicate

Interface Punched card Kbd & CRT See & point Ask & tell

Operation Process Edit Layout Orchestrate

Connectivity None Peripheral cable LAN Internet

OwnersCorporate

computer centersDivisional IS

shopsDepartmental

end-usersEveryone



Current Trends

• The substitution of expensive and specialized parallelmachines by the more cost-effective clusters of workstations

• A cluster is a collection of stand-alone computersconnected using some interconnection network

• A pervasiveness of the Internet created interest innetwork computing and more recently in grid

computing

• Grids are geographically distributed platforms of computation - dependable, consistent, pervasive, andless expensive access to HPC facilities



Flynn’s Taxonomy of

Com uter Architecture• Based on the notion of a stream of

information

• instruction

•data

CPU

Memory

fetch execute(manipulate data as

programmed)



SingleInstruction

MultipleInstruction

SingleData

MultipleData



SIMD Architecture



Single Instruction,

Multi le Data SIMD

t i m e

P1 P2 Pn



MIMD Architecture



Multiple Instruction,

Multi le Data MIMD

t i m e

P1 P2 Pn



SIMD Architecture Model

• Consists of two parts:

• a front-end

computer

• a processor array

• each element in the processor array is identical toone another and performs operation on different datain sync

•front-end can access PE’s memory via the bus



SIMD Architecture Model

• lock-step

synchronization

• Processors either donothing or exactly thesame ops simultaneously

• In SIMD, parallelism is exploited by applyingsimultaneous operations across large sets of data



SIMD Configurations

Each PE has its own localmemory

PEs and memory modulescommunicate via the IN



MIMD Architecture

Interconnection Network

P

MM M M

P P P

Shared Memory MIMD Architecture


P P P P

MM M M

Message Passing MIMD Architecture

information exchangethrough central shared

memory

information exchangethrough network in

message passing systems



MIMD Architecture


P

MM M M

P P P


• using bus/cachearchitecture

• called SMP (symmetricmultiprocessor) since

• equal chance to read/

write memory

• equal access speed



MIMD Architecture


P P P P

MM M M


• also known asdistributed memory

• no global memory• using message passing to

move data from one toanother (Send/Recieve

pair of commands)

• this architecture giveway to Internet

connected systems



MIMD Architecture


P

MM M M

P P P



P P P P

MM M M


programming is easier provided scalability

DSM (distributed-shared memory) is

the hybrid between the two



DSM

• memory is physically distributed [messagepassing]

• memory can be addressed as one (logicallyshared) address space [shared memory]

• programming-wise, the architecture looks

and behaves like a shared memorymachine, but a message passing architecturelives underneath the software



SGI Origin2000



SIMD

• access control - which process accesses arepossible to which resources

• synchronization - constraints limit the timeof accesses from sharing processes toshared resources



SIMD

• protection - a system feature that preventsprocesses from making arbitrary access toresources belonging to other processes



MIMD

• nodes are typically able to simultaneously

• store messages in buffers

• perform send/receive operations

• scalable - the number of processors can beincreased without significant decrease in

efficiency of operation



InterconnectionNetworks



Interconnection

Networks INs• Can be classified based on

• mode of operation

• control strategy

• switching techniques

• topology



Mode of Operation

• Accordingly, INs are classified as:

• Synchronous

•a single global clock used by all

• operating in a lock-step manner

• Asynchronous

•does not require a global clock

• handshaking signals are used

• Sync tends to be slower than async, sync is raceand hazard-free, however.



Control Strategy

• Accordingly, INs are classified as

• Centralized

• a single central CU is used to overseeand control the operation

•Decentralized

• the control function is distributedamong different components



Control Strategy

• The function and reliability of the central

control unit can become the bottleneck ina centralized control system

• While the crossbar is a centralized system,

the multistage interconnection networksare decentralized



Switching Techniques

• INs can be classified as:

• circuit switching

• a complete path has to be established and remain

existence during the whole communication

• packet switching

• communication takes place via messages that are dividedinto smaller entities (packets)

• packets travel in a store-and-forward manner

• While packet s/w tends to use resources more efficiently, itsuffers from variable packet delays



Topology

• Topology describes how to connectprocessors and memories to other

processors and memories



Shared Memory INs

bus-based switch-based



Message Passing INs

• Static interconnection network

• Dynamic interconnection network



Static INs



Dynamic INs

• Establish a connection between two ormore nodes on the fly as messages are

routed along the links

• The number of hops in a path from sourceto destination node is equal to the number

of point-to-point links a message musttraverse to reach its destination



Single-stage



Crossbar switch

slide02 parallel computers

Documents