A Tour of Research Computing at Genentech

Download A Tour of Research Computing at Genentech

Post on 11-May-2015




3 download

Embed Size (px)


  • 1.A Tour of Research Computingat GenentechReece Hart, Ph.D.Scientific ManagerResearch Computing & InformaticsGenentech, Inc.April 27, 2009Bio-IT World ExpoBoston, MASlides available at http://harts.net/reece/pubs/ 1

2. Organization 2 3. Research needs and Corporate needsmust be balanced.Research NeedsCorporate Needs Modern infrastructure Operational efficiencythrough rapid evolution and through standardization andagile processes.consolidation.3 4. Genentech IT is centralized. Centralized and Manufacturingstandardized IT architecture. Development CommercialResearch Centralized IT operations. Centralized IT accounting. Centralized support forproject management, legalreview, security. Corporate IT4 5. Divisions have unique needs and dedicated teams.CIT support forResearch~6 (> time)Bioinformatics~50 (4%)Research1228DevelopmentProduct Operations Commercial 33614523 2492COMM-IT 64 (3%)DEV-IT 157 (5%)PROP-IT 123 (3%)CIT 1210 (10%)Dynamic needs Stable, predictable needsAgile development Large, long-term projectsLots of custom dev. Mostly purchased applications with in-house integrationNon-validated systemsValidated systems Technical challenges Reliability challengesScaling challenges5 6. How is Research different from other groups? Projects are smaller, cheaper, shorter. Project cycle shorter than budget cycle! Needs and solutions are dynamic. Needs and options evolve quickly. Good soon is better than perfect later. Projects are iteratively refined. Validation and reliability secondary tofunctionality. Lots of custom development and gluing. 6 7. RCI ensures alignment between Research & CIT. Steering Committee Strategic needs Biomedical ImagingBudgeting Small Molecule DDStructural BiologyBioinformaticsStaffing Pharmacology Operations Communication Project oversight Staffing 4 100% (2 open)Research Computing & Informatics 6 > time Shared legal, budget Architecture & EngineeringSystems Operationsadmin, CRM, otherDatabase Administration staff.Storage... 7 8. Clear roles make for good partnership. Research owns theUser applications. Command line and webApplications & Web Services Database instances and content OS Distribution & Libraries CIT is reponsible for infrastructure. Facilities and core services Database Storage, database, computeStorage We partner on the rest. Disaster recovery planningCore Services ELN/e-signatures(Kerberos, LDAP, DNS) High-throughput screening and sequencingFacilities(Plant, Power, Network) Animal facility infrastructure8 9. Research computing is driven by needs. Diagnostics Computational discovery of markers that identifydisease and enable personalized medicine. Small Molecule Development Computational chemistry design and compoundscreening; electronic lab notebooks. Biomedical Imaging and Microscopy Image acquisition and reconstruction to understandbiological mechanism. Structural Biology Structural basis of antibody and small molecule action. Bioinformatics LIMS and and scientific analysis/support.9 10. A few examples of needs.Expectations of CIT2009 needsBenefit UsersDesktop & workstation support (Mac,Desktop application deployment Use corporate mechanisms andWin, Linux);unburden Research scientific personnel Applications Systems architecture; service installation Electronic lab notebook (selection,Improve integration and searching ofsupport; operations support on major architecture, installation support)Research datasystemsData integration and decision supportIncrease value of existing data, improve toolsdecisions through availability of completeand timely data.OS Distribution System administrationUpgrade to modern Linux distribution Enable access to additional tools with & Librariesgreatly reduced effort. ComputingProvision and operate server hardware; Enable rare, large-scale computing needsHardwareidentify new compute hardwareefficiently (cloud?)opportunitiesDatabaseAdminister, support, and backup allCentralized user authn/zImprove security policy and decrease (cloud)databases.unreliable manual efforts Credential-based accessEnable secure, decentralized computing MySQL supportProvide for multiple existing andunsupported MySQL instances.Storage Provide reliable, high-performanceImproved IO performance(cloud) storage; monitor and plan for growth.Secure user accessData triage & information life cycleDecrease storage costsmanagementCore Services Provide highly available DNS, LDAP, AD, LDAP upgradeetc.Credential-based loginsGroup cleanup (AD-LDAP sync;consolidation; unification)FacilitiesMaintain modern data center andphysical infrastructure. High-throughput sequencing architectureSpecializedand support ProjectsDisaster recovery planning / Tomcat AppService planningSystem PlanningWebAuth External collaboration & web hosting 10 11. Computing Environment11 12. Research Computing EnvironmentUserApplications & Web Services Load Balancers OS Distribution & Libraries Web Cloud(Static, CGI, Servlet) Compute Cluster DatabaseStorageCore Services Databases(Kerberos, LDAP, DNS) StorageFacilities(Plant, Power, Network) 12 13. CIT infrastructure is top-notch.Network Operations Center (NOC)Backup power2 independent battery strings3 diesel locomotives 14. Files and databases are the foundation. Primary NAS Two active-active NetApp 6080, w/SATA & FC Remote mirror for disaster recovery & tiered data No tape Exceptions used as necessary Primary SAN HP EVA 8K for databases and exceptional needs Primary Database Oracle 10g on Linux (SLES10) Gaps NAS Performance ILM / data triage Virtualized storage for CIFS Alternative storage and database options14 15. Compute Infrastructure Shell & batch computing 30 x 8-core, 64GB HP DL685 blades (Opteron) Altix 3700, 96 Itanium2 cores, 512GB RAM PBS Pro cluster scheduler Novell SLES10 (SLES11 coming) Legacy: Going: Tru64/alpha Gone: Solaris/SPARC, IRIX/mips Gaps Cluster scheduler tuning Missing nails for the Tru64/alpha coffin15 16. Web Tier Static and CGI VMWare virtual machines running Linux, Apache Java Application Services WebLogic 9 on Solaris F5 Load Balancer Gaps Tomcat support Holistic monitoring16 17. We got a lot right. Security Kerberos provides reliable authentication for manyservices, across multiple hosts, and on manyplatforms. Enables us to push user identity close to the data. File system layout A single logical filesystem and NAS enable datasharing in a hetergeneous environment. Goldilocks Migrations Aim for not too soon and not too late. Cluster strategy Great adoption, few complaints, no barriers. Awesome talent Weve hired exceptionally talented systems architects,engineers, admins, and support staff. 17 18. From here, where? Technology & Services Organize ELN efforts. Implement disaster recovery hardware. Explore and expand hosted cloud computing Write documentation. Organization & Attitude Actively prioritize needs using available staff. Strive to use the right amount of process in the rightplaces. Embrace continual change. 18 19. Planning for disasters is hard.19 20. Research needs and Corporate needsmust be balanced. Modern infrastructure Operational efficiencythrough rapid evolution and through standardization andagile processes.consolidation. 20 21. Acknowledgments Steering Committee CIT Lynne Ahn Storage: Steve Cachia, Jeff Blaney Chris Chu, Phil Seto Nick van Bruggen A&E: Munther Megdadi, Chris Jones Marc Lambert Cris Lewis DBA: Ben Nguyen, Melissa Starovasnik Jignesh Joshi Chris Wiesmann Sysops: Kathy Rinaldi, Zemin Zhang Boman Abadan, Paul RCI Bulanadi, Simran Albion Baucom, JimHansrai, MahwishFitzgerald, David Hamid, MichaelKonerding Kennedy +2 openings! Lots more! Elsewhere Kevin Clark, Borlan Pan,Nick Skelton 21