parallel processing steve terpe cs 147. overview what is parallel processing what is parallel...

of 18 /18
Parallel Parallel Processing Processing Steve Terpe Steve Terpe CS 147 CS 147

Author: chastity-rodgers

Post on 03-Jan-2016




1 download

Embed Size (px)


  • Parallel ProcessingSteve TerpeCS 147

  • OverviewWhat is Parallel ProcessingParallel Processing in NatureParallel Processing vs. MultitaskingAmdahls LawChallenges in Parallel Processing

  • What is Parallel Processing?How to make machines solve problems better and faster?Physical barriers limit the extent to which single processor performance can be improvedClock Speed & Heat DissipationThe next most obvious solution is to distribute the computing load among several processors

  • What is Parallel Processing?Parallel Processing encompasses a wide variety of different things:Intel Core Duo, Quad, Cell multiprocessors, networked and distributed computer systems, [email protected], [email protected], neural nets are all examples

  • Parallel Processing in NatureThe worlds most powerful parallel processor comes standard

    For everything else theres American Express

  • Parallel Processing in NatureFor humans parallel processing comes easy

    Human VisionColor, Motion,Depth, Shape

  • Parallel Processing in NatureMachines that are more like us, and hence more useful, will need to be able to process information more like usin parallel.

    Parallel Processing is key element in pattern recognition which distinguishes human & machine intelligence

    This is not an easy hurdle

  • Parallel Processing vs. MultitaskingToday, computers are great at multi-tasking, but

    Multi-tasking only creates the illusion of parallel processing

    The processor must switch between activitiesit is only the speed with which it does so that creates the illusion of simultaneous execution

    The illusion is most easily shattered when running virus scan while attempting to do anything else

  • Parallel Processing vs. MultitaskingThink about it this way:


    Parallel Processing

  • Amdahls LawConsider a single processorOr twoWe tend to think that 2x as much work will be done in the same timeOr that the same amount of work will be done in half the time

  • Amdahls LawDo n processors imply that a computational job should complete in 1/n time?

    Sadly, no

  • Amdahls Law1967 Gene Amdahl recognizes the interrelationship of all componentsOverall speedup of a system depends on the speedup of a particular component & how much that component is used

    S = 1/([1- f] + f / k)

    S = overall system speedf = fraction of the work performed by the faster componentk = speedup of new component

  • Amdahls LawAdditionally, no matter how well (or much) you parallelize an application there will always be a small portion of work that must be done serially.Other processors must simply sit and wait in this intervalEvery algorithm has a sequential part that limits potential speedup

  • Challenges in Parallel ProcessingNot always obvious where to split workload or even possible.If you dont use it, you lose itprograms not specifically written for parallel architecture run no more efficiently on parallel systems

  • Challenges in Parallel ProcessingConnecting your CPUsDynamic vs Staticconnections can change from one communication to nextBlocking vs Nonblockingcan simultaneous connections be present?Connections can be complete, linear, star, grid, tree, hypercube, etc.Bus-based routingCrossbar switchingimpractical for all but the most expensive super-computers2X2 switchcan route inputs to different destinations

  • Challenges in Parallel ProcessingDealing with memoryVarious options:Global Shared MemoryDistributed Shared MemoryGlobal shared memory with separate cache for processors

    Potential Hazards:Individual CPU caches or memories can become out of synch with each other. Cache Coherence

    Solutions:UMA/NUMA machinesSnoopy cache controllersWrite-through protocols

  • ConclusionWhat is Parallel ProcessingParallel Processing in NatureParallel Processing vs. MultitaskingAmdahls LawChallenges in Parallel Processing

  • Questions?