Grid Computing in the Industry

Download Grid Computing in the Industry

Post on 10-Feb-2016




0 download

Embed Size (px)


Grid Computing in the Industry. North Carolina Statewide Grid. Duke University, Nov 2, 2004. Wolfgang Gentzsch Managing Director MCNC Grid Computing and Networking Services - PowerPoint PPT Presentation


<ul><li><p>Grid Computing in the IndustryNorth Carolina Statewide GridDuke University, Nov 2, 2004 Wolfgang Gentzsch Managing Director MCNC Grid Computing and Networking Services </p><p>This Lecture aims at complementing your background in distributed computing technologies and developments, providing an actual overview of Grid Computing, and its applications and benefits for Education, Industry and Economy.</p></li><li><p>NC BioGridMCNC Enterprise GridNC Statewide Grid InitiativeMy current Background:</p><p>MCNC &amp; The Grid Infrastructure for North CarolinaNCREN North Carolina Research &amp; Education NetworkMCNCMicroelectronics Center of North Carolina Founded in 1980 Independent, private, non-profit organization Operates NCREN since 1984 Past: Supercomputer Center for universities Now: Grid Service Provider, offering Video, Network, Grid, and Datacenter Services 50+ employees</p></li><li><p> The Industrys View of a Grid :</p><p>An IT Utility on a . . . </p><p>Grid Middleware (the glue)Managing . . . </p><p>Networked DistributedResources</p></li><li><p>Why Should We Care about Grids ?</p><p> Its the next big thing ! Grid technologies advance Science and Education in that we can do things which havent been possible before. </p><p>Grid infrastructure attracts and enables new businesses and creates new jobs, especially in today's rural areas. </p><p>Grids make us more competitive by better utilizing resources, bringing results to market faster, and delivering with higher quality.</p></li><li><p>Access: transparent, remote, secure, wirelessSharing: enable collaboration over the networkFailover: migrate/restart applications automaticallyOn Demand: get resources, when you need themProductivity: more work done in shorter timeVirtualization: access compute services, not serversHeterogeneity: platforms, OSs, devices, softwareResource Utilization: increase from 20% to 80+%Virtual Organizations: build &amp; dismantle on the fly</p><p> Grid BenefitsDepartment, Enterprise, and Global Grids</p></li><li><p>However, There is Still a Long Way to Go ! Grids are over-hyped: currently, they promise much more than they can really offer. Grid technology is far from mature and complete. Grid standards are (mostly) still missing. Grids are very complex IT infrastructures. Grids bring new challenges: sharing resources, loosing direct control, security, intellectual property, legal, social, political issues . . .</p></li><li><p>Grid Architecture, Technology, Standards</p></li><li><p>Architectural RequirementsAvailabilityDowntime impactIndividual jobsMaintenance windowsScalabilityGrowth 1-3-5 yearsScaling strategy and Response to peak loadsTechnology refresh, evolutionManageabilitySkill set / workload of SAExpected stabilityCode management, software distributionSecurityUser authenticationInternet accessData Security requirementsData DistributionLocation, volume, refreshSecurity of dataUsabilityAdministrative Skill set and Client environmentPsychological factors importantOperations Management100s CPUs / SAResources added in large blocksChange control is critical</p></li><li><p>The Globus Toolkit and the Open Grid Services Architecture (OGSA)Integration of Grid technologies and Web ServicesOGSA defines a Grid ServiceIn terms of WSDL interfaces, defines mechanisms:creating distributed systemslifetime managementchange managementcredential managementnotificationNotificationAuthorizationService CreationService RegistryManageabilityConcurrency</p></li><li><p>Global Grid Forum (GGF)Community-driven set of working groups that are developing standards and best practices for distributed computing ("Grids" and "Metacomputing") effortsFormed in 2001 (merger of US, APAC, Euro efforts)Open Grid Services Architecture (OGSA)</p></li><li><p>Enterprise Grid Alliance (EGA)Industry-driven consortium to implement standards in industry products and make them interoperableFounding members: Oracle, Sun, Fujitsu, </p><p>Enforce complementary position in pushing GGF into the academic corner</p></li><li><p>The Changing Grid LandscapeHPTC GridWeb SvcsPublic SOA</p><p>TechnologiesMarket making / Productization Enabled Business ModelsOnDemandUtility ComputingCompute Powerfrom the wall socketOutsourcing/HostingSoftware as a Svc Developer View: Write here Run anywhere Service Composition IBMEnterpriseIT MgmtProductsSunHPIBM. . .Details see backup slidesStandards</p></li><li><p>Cluster Grid Implementation, Example </p></li><li><p>Department GridsSolaris ServersWorkstations(Linux or Solaris)</p><p>Linux ServersBrowser to CGM(Remote Server Setup &amp; Configuration)</p><p>Auto Download of Modules Auto OS DeploymentGrid Installation/MgmtCentralized Server Mgmt</p><p>Compute Grid Manager</p></li><li><p> Enterprise Grid Reference ArchitectureServers,Blades,&amp; VIZ WorkstationsGrid ManagerLinux RacksOptional Control Network (Gbit-E)Data Network (Gbit-E)HA NFSScalable QFS/NFSNAS/NFSMyrinetMyrinetMyrinetSun Fire LinkSimple NFSSunRay Access Browser Accessvia GEPWorkstation Access</p></li><li><p>Grid Enterprise Policy ManagementClear definition and documentation of prioritiesand policies is key to Grid resource management</p></li><li><p>Globus &amp; Avaki over multiple local GridsGrid Engine cluster mgmnt within an admin domain &amp; file system areaGlobus/Avaki knits together resources, handles files, binary management, and high level resource selectionOrganization AOrganization BOrganization CAvaki Data Grid</p></li><li><p> Applications for The GridSingle-CPU Jobs: jobmix, many users, many serial applications, suitable for grid (e.g in universities and research centers) </p><p>Array Jobs: 100s/1000s of jobs, one user, one serial application, varying input parameters, suitable for grid (e.g. parameter studies in Optimization, CAE, Genomics, Finance)</p><p>Massively Parallel Jobs: one job, one user, one parallel application, no/low communication, scalable, fine-tune for grid (time-explicit algorithms, film rendering, pattern recognition)</p><p>Parallel Jobs: one job, one user, one parallel application, high interprocs communication, not suitable for distribution over the grid, but for parallel system in the grid (time-implicit algorithms, direct solvers, large linear algebra equation systems) </p></li><li><p>Grid Computing Today in Industry</p></li><li><p> Computing as a Utility</p><p> =&gt;&gt; Enhancing The Grid with a Business Model On Demand: Get a service at your finger tipFrom the Wall Socket: Don't care about the infrastructure Metering &amp; Billing: Pay as you go, for what you use What's a Utility?Like electricity, water, gas, heat, telephony</p></li><li><p>Customer Challenges</p></li><li><p>Critical Customer Requirements</p></li><li><p>Who Uses Department and Enterprise Grids?Life SciencesStartup and cost efficientCustom research or limited use applicationsMulti-day application runs (BLAST)Exponential CombinationsLimited administrative staffComplementary techniquesElectronic DesignTime to MarketFastest platforms, largest GridsLicense ManagementWell established application suiteLarge legacy investmentPlatform Ownership issuesFinancial ServicesMarket simulationsTime IS MoneyProprietary applicationsMultiple PlatformsMultiple scenario executionNeed instant results &amp; analysis toolsHigh Performance ComputingParallel Reservoir SimulationsGeophysical Ray TracingCustom in-house codesLarge scale, multi-platform execution</p></li><li><p>How Real is all this Stuff?department enterprise globalvendor interestresearch activitydirection of technology adaptation</p></li><li><p>Company Example: </p><p> Sun Microsystems</p></li><li><p>Sun Microsystems &amp; Grid Computing</p><p>Slogan: The network is the computerStrategy: N1, manage N computers as 1 AcquisitionsGridware =&gt; Sun Grid Engine, Distributed Resource ManagerPirus =&gt; Dynamic Storage Provisioning Terraspring =&gt; Automatic System Provisioning Centerrun =&gt; Automatic Application Provisioning </p></li><li><p>Managing Grids What we did for the big SMPs...</p><p>...we are now doing for the NetworkResource Virtualization Domains, ZonesInterdomain resource mgmtRoutingSoft cabling within the box Solaris N1</p></li><li><p>Virtualization of ResourcesStorageVerticalHorizontalStorage NetworkComputeService 1Service 2Service 3Compute &amp; Data Grid</p></li><li><p>123ServicesServicesServicesVirtual StoreVirtual ComputeVirtual NetworkServices sharing resourcesN1: managing services, not servers</p></li><li><p>N1 Provisions Grid ComputingWeb ServerWeb ServerWeb ServerWeb ServerAppSvrAppSvrAppSvrDBDB</p></li><li><p>The N1 Effect on EfficiencyRadical improvement in costs &amp; uptimeSys Utilization6-15%80+ %Time to DeployWeeksHoursPorts/Admin50-100500+Terabytes/DBA1TB100TBAvailability99.9 %99.999+ %Server/Admin15-30500+CostsUptimeN1</p></li><li><p>Grid Computing</p><p> Use Cases, Deployments, Best Practices</p></li><li><p> Leeds, York + Sheffield Universities Deliver stable, well-managed HPC resources supporting multi-disiplinary research Deliver a Metropolitan Grid across the UniversitiesExample: White Rose Grid in England</p></li><li><p>WRG HardwareMaximaTitaniaMaximaSnowdonPascali</p></li><li><p>WRG Architecture OverviewMaximaGT2.0GEEESnowdonGT2.0GEEEPascaliGT2.0GEEETitaniaGT2.0GEEEportalWhite Rose GridGT2.0SolarisLinuxSolarisSolaris</p></li><li><p>WRG Key Components Globus Toolkit 2.4Provides a secure means for inter-campus actionsTransferring jobsMoving data Gathering information about resources Grid Engine Enterprise EditionManages the campus grid compute resourcesDelivers a single interface for a heterogeneous gridGuarantees a share of campus resource for grid and local users</p></li><li><p>WRG Key Components Grid Portal Development KitProvides a portal interface into Globus ToolkitTransferring jobsMoving data Gathering information about resources MyProxy MyProxy provides a server with client-side utilities to store and retrieve delegated X.509 credentials via the Grid Security Infrastructure (GSI). </p></li><li><p>Grid Engine Enterprise Edition, Share PoliciesGT2.0GEEEGT2.0GEEESolarisSolarisGT2.0GEEEGT2.0GEEESolarisLinuxWhite Rose Grid</p></li><li><p>NC BioGridMCNC Enterprise GridNC Statewide Grid InitiativeExample:Grid Service Provider MCNC &amp; The Grid Infrastructure for North CarolinaNCREN North Carolina Research &amp; Education NetworkMCNCMicroelectronics Center of North Carolina Founded in 1980 Independent, private, non-profit organization Operates NCREN since 1984 Past: Supercomputer Center for universities Now: Grid Service Provider, offering Video, Network, Grid, and Datacenter Services 50+ employees</p></li><li><p>GCNS Grid Computing and Networking Services Mission Advance education, innovation and economic development </p><p> throughout North Carolina by delivering next generation information technology that enables the academic, research, government and business communities </p><p> to discover, create, share and apply knowledge.</p></li><li><p>The GridThe next IT Infrastructure of North Carolina2002200320042005NC BioGrid Proving ground for Grid Successful prototype apps Catalyst for collaboration International recognition</p></li><li><p>2002200320042005NC BioGridMCNC Enterprise Grid Cluster and SMP resources Research platform for GTEC Core component in NCGrid Revenue generation The GridThe next IT Infrastructure of North Carolina</p></li><li><p>2002200320042005NC BioGridMCNC Enterprise GridNC Grid Initiative State-wide partnership Leverage lessons learned Grid education &amp; training resource Enable first mover applicationsThe Grid The next IT Infrastructure of North Carolina</p></li><li><p>CY03CY04CY05CY06MCNC: 3-Year GCNS Grid RoadmapEasy Access: Training, Web Courses Access Grid Node Grid Appliance QuadA Grid Service Provider:Network, Computing, Data, Video Partner with SC Sites Build GTEC Service Portfolio Start Grid Consulting Annual GSP WorkshopAwareness Creation:Conferences, Workshops, PRMCNC Enterprise GridNC BioGridNC Statewide Grid</p></li><li><p> North Carolinas Foundation for Grid: NCREN CiscoEPA4-7 MCNC-owned Clusters distributed throughout the stateLocations still under evaluationExisting: Blend of owned and leased fiber and circuits moving toward resilient rings powered by Cisco routers</p><p>Planned: Strong focus on owned and leased fiber, Lambda, and few circuits, in resilient rings powered by Cisco routers and Wave Division Multiplexers</p></li><li><p>32-CPU SGI AltixLinux SMP Server128-CPU IBM LinuxCluster (64 nodes)8-TB StorageLSF Master Job SchedulerGrid Gatekeeper / Interactive Nodes</p><p>Global Grid Resource DB (GIIS)</p><p>UsersCampus GridsGCNS Enterprise Grid</p><p>AvakiDataGrid</p><p>Data Grid Access Servers(8 total, i.e. 1 per 8 nodes)</p></li><li><p>Network, Grid and Data Center ServicesNCRENState-wide Grid ServicesEnterprise Grid ServicesValue-add Information Systems ServicesSelf-serve Data Center ServicesDATA CENTERHosting &amp; InfrastructureGrid ComputingGTEC, NLR, ANR and other Innovation InitiativesInformation Security ServicesData Archival ServicesInformation AssuranceDEPLOYMENT</p></li><li><p>The NC State-Wide Grid Roadmap06/04: Do-Grid-Yourself Workshop07/04: Phase 1, Awareness Creation10/04 06/05: Deliver/Connect Grid Appliance Clusters to University Partners in NC10/04: Develop Do-Grid-Yourself Training Course10/04: Start QuadA Project: Access, Accounting, Authentication, Authorization12/04: Start Deliver Grid Training to Partner Univs01/05 06/05: Work with Grid Users to Port Apps03/05: Build Access Grid Node 06/05: 1st NC Statewide Grid Workshop</p></li><li><p>Grid Projects Driving Grid Adoption2000 20004 2008 MCNC Grid SP Grid ColLab Kids Grids Gaming Grid School Grids Grid-Info Grid Download-The-Grid AAAA Project Grid Training Courses Startup Grid Grid Appliance Grid Portal GGF &amp; EGA NC Statewide Grid MCNC Enterprise Grid NC BioGridMCNC Supercomputing</p></li><li><p>Grid Vision</p></li><li><p>Our Vision :</p><p>The Three Waves of Grid Computing The Research Wave The Industry Wave The Consumer Wave</p><p> Technology, Prototypes Grid-Enabled Products Commodity Virtual Organizations Enterprise Solutions IT Utility Standards Interoperability Integration GGF, IETF, OASIS...</p></li></ul>


View more >