aws re:invent - accelerating research

Download AWS re:Invent - Accelerating Research

Post on 12-Jan-2015




3 download

Embed Size (px)


20 slides for a 10 minute talk! Short presentation for the 2012 Amazon Web Services re:Invent conference. This briefly covers some Computer Aided Engineering (CAE) simulation work done on Amazon using CST Studio software. Interesting Linux/Windows/Cloudbursting use case.


  • 1. Accelerating Research10-minute Vignette for 2012 Amazon re:Invent Conference chris@bioteam.net11

2. Im Chris.Im an infrastructure/bio geek.I work for the BioTeam.Twitstuff: @chris_dag 22 3. BioTeam - Not your standard AWS userWho, what & why Independent consulting shop Run by scientists forced to learn IT, SW & HPC to get our own genetics research done 12+ years bridging the gap between science, IT & high performance computing 3 3 4. Meta-IssuesWhat you dont know WILL hurt you. 4 4 5. Lets Be BluntA common theme across all our biotech, pharma, EDU andresearch clients ... Its almost 2013 AWS is comprehendible in minute:hours; pilot projects and proof-of-concepts can be done in hours:days. Really. The TECHNICAL bits are easy or at least manageable The POLICY & PROCEDURE bits will destroy your soul If I could only give 1 piece of advice to researchers ... 5 5 6. One Piece of Advice ...For anyone doing research HPC or science on the cloud The technical issues surrounding cloud-based scientic research are easy or at least surmountable The hard part is sorting out the internal policies and procedures that govern cloud use Ignore this at your peril66 7. Five Minute Case StudyComputer Aided Engineering (CAE) on AWS77 8. NMR Probehead Simulation on AWSNext Generation Nuclear Magnetic Resonance CAE Simulation Project via Software: CST Studio 2012 My role: Volunteer HPC Mentor 8 8 9. Why this was an interesting projectSimulating next-generation NMR probeheads Frontend interface is graphicsheavy and requires Windows Studio solvers run Linux orWindows; support GPUs and MPItask distribution Simultaneous use of local andcloud-based solvers actually works exLM license server involved Non-trivial security and geo-location requirements99 10. Initial Architecture 10 10 11. At experiment end it looked like this ... 1111 12. Amazon FTWAWS makes the easy stuff fun and the hard stuff possible 12 12 13. Nightmare on any other cloudWhy this work was easy on Amazon AWS ... Tick Tock ... Dont have the time to get into the CAE simulation details Lets discuss why this simulation workload would be much, much harder to do on some other cloud platform ...1313 14. Nightmare on any other cloudWhy this work was easy on Amazon AWS ...Brand X Cloud AWS EC2, S3, EBS, RDS, SNS,1. Virtual ServersSQS, SWS, GPUs, SSDs,2. Block StorageCloudFormation, VPC, ENIs,SecurityGroups, 10GbE3. Object Storage DirectConnect, Reserved4. ... and maybe some other Instances, ImportExport, stuff if Im lucky Spot Market And ~25 other products andservice features with moreadded monthly14 14 15. One very specic exampleEasy on AWS; much harder elsewhere The widely used FLEXlmlicense server uses NICMAC addresses whengenerating license keys Different MAC? Sciencestops. Screwed. VPC ENIs allow separationof MAC address fromNetwork Interface.Badass. 15 15 16. A few other examples ...Why this work was easy on Amazon AWS ... Incredibly powerful. Actually useful. VPC Approachable even if you are not an IPSEC or BGP routing god. Compelling economics. Once you start youll likely never run anywhere else.Spot Market The competition cant compete. cc* & cg* Fat nodes with bidirectional 10GbE bandwidth.ec2 instance And dont get me started on SSD or Provisioned- types performance EBS volumes.1616 17. Science on AWSWrapping up ... Its not rocket science; yeah we can do this just about anywhere Its just easier and faster on AWS because they have more building blocks; I can spend more time on actual science problems You mean I get paid for this?1717 18. Science on AWS, cont.Wrapping up ... There is no one-size-ts-all research design pattern ... Very few of us have a single pipeline or workow that we can devote endless engineering effort to We are not going to toss out hundreds of legacy codes and rewrite everything for GPUs or MapReduce For research HPC its all about theI love my job building blocks { and how we can effectively use/deploy them }1818 19. One last plug Want more info or details? Watch for upcoming posts that document in detail the CST Studio on AWS effort And by the way ... round #2 is starting soon; They need volunteers, resource providers and new use cass1919 20. Thanks!Slides online at: Twitter: @chris_dag2020