future grid futuregrid overview dr. speaker. future grid future gridfuturegridfuturegrid the goal of...

Download Future Grid FutureGrid Overview Dr. Speaker. Future Grid Future GridFutureGridFutureGrid The goal of FutureGrid is to support the research on the future

Post on 31-Dec-2015




0 download

Embed Size (px)


  • FutureGrid OverviewDr. Speaker

  • FutureGridThe goal of FutureGrid is to support the research on the future of distributed, grid, and cloud computing. FutureGrid will build a robustly managed simulation environment or testbed to support the development and early use in science of new technologies at all levels of the software stack: from networking to middleware to scientific applications. The environment will mimic TeraGrid and/or general parallel and distributed systems FutureGrid is part of TeraGrid and one of two experimental TeraGrid systems (other is GPU)This test-bed will succeed if it enables major advances in science and engineering through collaborative development of science applications and related software.FutureGrid is a (small 5600 core) Science/Computer Science Cloud but it is more accurately a virtual machine based simulation environment

  • FutureGrid Hardware

  • Compute Hardware

    System type# CPUs# CoresTFLOPSTotal RAM (GB)Secondary Storage (TB)Site Status Dynamically configurable systems IBM iDataPlex2561024113072339*IU New System Dell PowerEdge19211528115215TACC New System IBM iDataPlex16867272016120UC New System IBM iDataPlex1686727268872SDSC Existing System Subtotal7843520338928546 Systems not dynamically configurable Cray XT5m16867261344339*IU New System Shared memory system TBD404804640339*IU New System 4Q2010 Cell BE Cluster480164IU Existing System IBM iDataPlex6425627681UF New System High Throughput Cluster1923844192PU Existing System Subtotal46818721730081 Total125253925011936547

  • Storage HardwareFutureGrid has dedicated network (except to TACC) and a network fault and delay generatorCan isolate experiments on request; IU runs Network for NLR/Internet2Additional partner machines could run FutureGrid software and be supported (but allocated in specialized ways)

    System TypeCapacity (TB)File SystemSiteStatusDDN 9550(Data Capacitor)339 LustreIUExisting SystemDDN 6620120GPFSUCNew SystemSunFire x417072Lustre/PVFSSDSCNew SystemDell MD300030NFSTACCNew System

  • Network Impairments DeviceSpirent XGEM Network Impairments Simulator for jitter, errors, delay, etcFull Bidirectional 10G w/64 byte packetsup to 15 seconds introduced delay (in 16ns increments)0-100% introduced packet loss in .0001% incrementsPacket manipulation in first 2000 bytesup to 16k frame sizeTCL for scripting, HTML for human configuration

  • System MilestonesNew IBM SystemsDelivery: December 2009Acceptance: March 2010Available for Use: April 2010Dell SystemDelivery: January 2010Acceptance: March 2010Available for Use: April 2010Existing IU iDataPlexMove to SDSC: January 2010Available for Use: March 2010Storage Systems (Sun & DDN)Delivery: October 2009Available for Use: December 2009

  • FutureGrid PartnersIndiana University (Architecture, core software, Support)Purdue University (HTC Hardware)San Diego Supercomputer Center at University of California San Diego (INCA, Monitoring)University of Chicago/Argonne National Labs (Nimbus)University of Florida (ViNE, Education and Outreach)University of Southern California Information Sciences Institute (Pegasus to manage experiments) University of Tennessee Knoxville (Benchmarking)University of Texas at Austin/Texas Advanced Computing Center (Portal)University of Virginia (OGF, Advisory Board and allocation)Center for Information Services and GWT-TUD from Technische Universtitt Dresden. (VAMPIR) Blue institutions have FutureGrid hardware

  • Other Important CollaboratorsNSFEarly users from an application and computer science perspective and from both research and educationGrid5000/Aladin and D-Grid in EuropeCommercial partners such asEucalyptus .Microsoft (Dryad + Azure) Note current Azure external to FutureGrid as are GPU systemsApplication partnersTeraGrid Open Grid ForumPossibly Open Nebula, Open Cirrus Testbed, Open Cloud Consortium, Cloud Computing Interoperability Forum. IBM-Google-NSF Cloud, and other DoE/NSF/ clouds

  • Future Grid UsersApplication/Scientific usersSystem administratorsSoftware developersTestbed usersPerformance modelersEducatorsStudents

    Supported by FutureGrid Infrastructure & Software offerings


  • FutureGrid Usage ScenariosDevelopers of end-user applications who want to develop new applications in cloud or grid environments, including analogs of commercial cloud environments such as Amazon or Google.Is a Science Cloud for me? Is my application secure?Developers of end-user applications who want to experiment with multiple hardware environments. Grid/Cloud middleware developers who want to evaluate new versions of middleware or new systems. Networking researchers who want to test and compare different networking solutions in support of grid and cloud applications and middleware. (Some types of networking research will likely best be done via through the GENI program.)Education as well as researchInterest in performance requires that bare metal important

  • Selected FutureGrid TimelineOctober 1 2009 Project StartsNovember 16-19 SC09 Demo/F2F Committee Meetings/Chat up collaboratorsJanuary 2010 Significant Hardware availableMarch 2010 FutureGrid network completeMarch 2010 FutureGrid Annual MeetingSeptember 2010 All hardware (except Track IIC lookalike) accepted October 1 2011 FutureGrid allocatable via TeraGrid process first two years by user/science board led by Andrew Grimshaw

  • FutureGrid Architecture

  • FutureGrid ArchitectureOpen Architecture allows to configure resources based on imagesManaged images allows to create similar experiment environmentsExperiment management allows reproducible activitiesThrough our modular design we allow different clouds and images to be rained upon hardware.Note will be supported 24x7 at TeraGrid Production Quality Will support deployment of important middleware including TeraGrid stack, Condor, BOINC, gLite, Unicore, Genesis II

  • Software GoalsOpen-source, integrated suite of software to instantiate and execute grid and cloud experiments. perform an experimentcollect the resultstools for instantiating a test environment, Torque, MOAB, xCAT, bcfg, and Pegasus, Inca, ViNE, a number of other tools from our partners and the open source communityPortal to interactBenchmarking*http://futuregrid.org*

  • Draft GUI for FutureGrid Dynamic Provisioning

  • Command linefg-deploy-imagehost nameimage namestart timeend timelabel namefg-addlabel nameframework hadoopversion 1.0

    Deploys an image on a host

    Adds a feature to a deployed image


  • FG StratosphereObjectiveHigher than a particular cloudProvides all mechanisms to provision a cloud on a given FG hardwareAllows the management of reproducible experimentsAllows monitoring of the environment and the resultsRisksLots of softwarePossible multiple path to do the same thing

    Good newsWe worked in a team, know about different solutions and have identified a very good planWe can componentize Stratosphere*http://futuregrid.org*

  • RAIN: Dynamic Provisioning Change underlying system to support current user demandsLinux, Windows, Xen, Nimbus, EucalyptusStateless imagesShorter boot timesEasier to maintainStateful installsWindowsUse moab to trigger changes and xCAT to manage installs**http://futuregrid.org

  • xCAT and MoabxCAT uses installation infrastructure to perform installscreates stateless Linux imageschanges the boot configuration of the nodesremote power control and console (IPMI)Moab meta-schedules over resource managers TORQUE and Windows HPCcontrol nodes through xCATchanging the OSremote power control**http://futuregrid.org

  • Experiment ManagerObjectiveManage the provisioning for reproducible experimentsCoordinate workflow of experimentsShare workflow and experiment imagesMinimize space through reuseRiskImages are largeUsers have different requirements and need different images