cern: accelerating science with puppet - puppetconf '12

39
Accelerating Science with Puppet Tim Bell [email protected] @noggin143 PuppetConf San Francisco 28 th September 2012 1 PuppetConf 2012 Tim Bell, CERN

Upload: puppet-labs

Post on 10-May-2015

7.021 views

Category:

Technology


0 download

DESCRIPTION

Watch the video here: http://www.youtube.com/watch?v=-Ykb2j2ojYU View the rest of the PuppetConf 2012 videos here: www.puppetlabs.com/community/videos/puppetconf "Accelerating Science with Puppet" by Tim Bell, Operating System and Infrastructure Manager at CERN. Talk given at PuppetConf 2012. Abstract: This talk will review CERN's objectives and how the computing infrastructure is evolving to address the challenges at scale using community supported software such as Puppet and OpenStack. Speaker Bio: Tim Bell is responsible for the CERN IT Operating System and Infrastructure Group which supports Windows, Mac and Linux across the site along with virtualisation, E-mail and web services. These systems are used by over 10,000 scientists researching fundamental physics, finding out what the Universe is made of and how it works. Prior to working at CERN, Tim worked for Deutsche Bank managing private banking infrastructure in Europe and for IBM as a Unix kernel developer and deploying large scale technical computing solutions. Learn more about Puppet: http://bit.ly/QQoAP1

TRANSCRIPT

Page 1: CERN: Accelerating Science with Puppet - PuppetConf '12

Accelerating Sciencewith Puppet

Tim [email protected]

@noggin143

PuppetConf San Francisco28th September 2012

1PuppetConf 2012 Tim Bell, CERN

Page 2: CERN: Accelerating Science with Puppet - PuppetConf '12

What is CERN ?

PuppetConf 2012 Tim Bell, CERN 2

• Conseil Européen pour la Recherche Nucléaire – aka European Laboratory for Particle Physics

• Between Geneva and the Jura mountains, straddling the Swiss-French border

• Founded in 1954 with an international treaty

• Our business is fundamental physics , what is the universe made of and how does it work

Page 3: CERN: Accelerating Science with Puppet - PuppetConf '12

PuppetConf 2012 Tim Bell, CERN 3

Answering fundamental questions…• How to explain particles have mass?

We have theories and accumulating experimental evidence.. Getting close…

• What is 96% of the universe made of ?We can only see 4% of its estimated mass!

• Why isn’t there anti-matterin the universe?

Nature should be symmetric…

• What was the state of matter justafter the « Big Bang » ?

Travelling back to the earliest instants ofthe universe would help…

Page 4: CERN: Accelerating Science with Puppet - PuppetConf '12

Community collaboration on an international scale

Tim Bell, CERN 4PuppetConf 2012

Page 5: CERN: Accelerating Science with Puppet - PuppetConf '12

Tim Bell, CERN 5

The Large Hadron Collider

PuppetConf 2012

Page 6: CERN: Accelerating Science with Puppet - PuppetConf '12

PuppetConf 2012 Tim Bell, CERN 6

Page 7: CERN: Accelerating Science with Puppet - PuppetConf '12

LHC construction

PuppetConf 2012 Tim Bell, CERN 7

Page 8: CERN: Accelerating Science with Puppet - PuppetConf '12

8

The Large Hadron Collider (LHC) tunnel

PuppetConf 2012 Tim Bell, CERN

Page 9: CERN: Accelerating Science with Puppet - PuppetConf '12

PuppetConf 2012 Tim Bell, CERN 9

Page 10: CERN: Accelerating Science with Puppet - PuppetConf '12

Superconducting magnets – October 2008

PuppetConf 2012 Tim Bell, CERN 10

A faulty connection between two superconducting magnets led to the release of a large amount of helium into the LHC tunnel and forced the machine to shut down for repairs for one year

Page 11: CERN: Accelerating Science with Puppet - PuppetConf '12

Accumulating events in 2009-2011

PuppetConf 2012 Tim Bell, CERN 11

Page 12: CERN: Accelerating Science with Puppet - PuppetConf '12

PuppetConf 2012 Tim Bell, CERN 12

Page 13: CERN: Accelerating Science with Puppet - PuppetConf '12

Heavy Ion Collisions

PuppetConf 2012 Tim Bell, CERN 13

Page 14: CERN: Accelerating Science with Puppet - PuppetConf '12

PuppetConf 2012 Tim Bell, CERN 14

Page 15: CERN: Accelerating Science with Puppet - PuppetConf '12

PuppetConf 2012 Tim Bell, CERN 15

Tier-1 (11 centres):•Permanent storage•Re-processing•Analysis

Tier-0 (CERN):•Data recording•Initial data reconstruction•Data distribution

Tier-2 (~200 centres):• Simulation• End-user analysis

• Data is recorded at CERN and Tier-1s and analysed in the Worldwide LHC Computing Grid• In a normal day, the grid provides 100,000 CPU days executing 1 million jobs

Page 16: CERN: Accelerating Science with Puppet - PuppetConf '12

PuppetConf 2012 Tim Bell, CERN 16

• Data Centre by Numbers– Hardware installation & retirement

• ~7,000 hardware movements/year; ~1,800 disk failures/year

Xeon 51502%

Xeon 516010%

Xeon E5335

7%Xeon

E534514%

Xeon E5405

6%

Xeon E541016%

Xeon L5420

8%

Xeon L552033%

Xeon 3GHz4%

Fujitsu3%

Hitachi23% HP

0% Maxtor

0% Seagate15%

Western Digital

59%

Other0%

High Speed Routers(640 Mbps → 2.4 Tbps) 24

Ethernet Switches 350

10 Gbps ports 2,000

Switching Capacity 4.8 Tbps

1 Gbps ports 16,939

10 Gbps ports 558

Racks 828

Servers 11,728

Processors 15,694

Cores 64,238

HEPSpec06 482,507

Disks 64,109

Raw disk capacity (TiB) 63,289

Memory modules 56,014

Memory capacity (TiB) 158

RAID controllers 3,749

Tape Drives 160

Tape Cartridges 45,000

Tape slots 56,000

Tape Capacity (TiB) 73,000

IT Power Consumption 2,456 KW

Total Power Consumption 3,890 KW

Page 17: CERN: Accelerating Science with Puppet - PuppetConf '12

Our Challenges - Data storage

PuppetConf 2012 Tim Bell, CERN 17

• 25PB/year to record• >20 years retention• 6GB/s average• 25GB/s peaks

Page 18: CERN: Accelerating Science with Puppet - PuppetConf '12

PuppetConf 2012 Tim Bell, CERN 18

Page 19: CERN: Accelerating Science with Puppet - PuppetConf '12

PuppetConf 2012 Tim Bell, CERN 19

45,000 tapes holding 73PB of physics data

Page 20: CERN: Accelerating Science with Puppet - PuppetConf '12

New data centre to expand capacity

PuppetConf 2012 Tim Bell, CERN 20

• Data centre in Geneva reaches limit of electrical capacity at 3.5MW

• New centre chosen in Budapest, Hungary

• Additional 2.7MW of usable power

• Hands off facility• Deploying from 2013

Page 21: CERN: Accelerating Science with Puppet - PuppetConf '12

Time to change strategy

• Rationale– Need to manage twice the servers as today– No increase in staff numbers– Tools becoming increasingly brittle and will not scale as-is

• Approach– We are no longer a special case for compute– Adopt an open source tool chain model– Strong engineering skills allows rapid adoption of new technologies

• Evaluate solutions in the problem domain• Identify functional gaps and challenge them

– Contribute new function back to the community

PuppetConf 2012 Tim Bell, CERN 21

Page 22: CERN: Accelerating Science with Puppet - PuppetConf '12

Building Blocks

PuppetConf 2012 Tim Bell, CERN 22

Bamboo

Koji, Mock

AIMS/PXEForeman

Yum repoPulp

Puppet-DB

mcollective, yum

JIRA

Lemon /Hadoop

git

OpenStack Nova

Hardware database

Puppet

Active Directory /LDAP

Page 23: CERN: Accelerating Science with Puppet - PuppetConf '12

Training and Support

• Buy the book rather than guru mentoring• Newcomers are rapidly productive (and often know more than us)• Community and Enterprise support means we’re not on our own

PuppetConf 2012 Tim Bell, CERN 23

Page 24: CERN: Accelerating Science with Puppet - PuppetConf '12

Staff Motivation

• Skills valuable outside of CERN when an engineer’s contracts end

PuppetConf 2012 Tim Bell, CERN 24

Page 25: CERN: Accelerating Science with Puppet - PuppetConf '12

Prepare the move to the clouds

• Improve operational efficiency– Machine reception and testing– Hardware interventions with long running programs– Multiple operating system demand

• Improve resource efficiency– Exploit idle resources, especially waiting for tape I/O– Highly variable load such as interactive or build machines

• Improve responsiveness– Self-Service– Coffee break response time

PuppetConf 2012 Tim Bell, CERN 25

Page 26: CERN: Accelerating Science with Puppet - PuppetConf '12

Service Model

PuppetConf 2012 Tim Bell, CERN 26

• Pets are given names like pussinboots.cern.ch

• They are unique, lovingly hand raised and cared for

• When they get ill, you nurse them back to health

• Cattle are given numbers like vm0042.cern.ch

• They are almost identical to other cattle• When they get ill, you get another one

• Future application architectures tend towards Cattle but Pets with configuration management are also viable

Page 27: CERN: Accelerating Science with Puppet - PuppetConf '12

OpenStack

PuppetConf 2012 Tim Bell, CERN 27

• Open source cloud run by an independent foundation with over 6,000 members from 850 organisations

• Started in 2010 but maturing rapidly with public cloud services from Rackspace, HP and Ubuntu

Platinum Members

Page 28: CERN: Accelerating Science with Puppet - PuppetConf '12

Many OpenStack Components to Configure

PuppetConf 2012 Tim Bell, CERN 28

Compute Scheduler

NetworkVolume

Registry Image

KEYSTONE HORIZON

NOVA

GLANCE

Page 29: CERN: Accelerating Science with Puppet - PuppetConf '12

When communities combine…

• OpenStack’s many components and options make configuration complex out of the box

• Puppet forge module from PuppetLabs (Thanks, Dan Bode)• The Foreman adds OpenStack provisioning for user kiosk

PuppetConf 2012 Tim Bell, CERN 29

Page 30: CERN: Accelerating Science with Puppet - PuppetConf '12

Scaling up with Puppet and OpenStack

• Use LHC@Home based on BOINC for simulating magnetics guiding particles around the LHC

• Naturally, there is a puppet module puppet-boinc• 1000 VMs spun up to stress test the hypervisors with Puppet,

Foreman and OpenStack

PuppetConf 2012 Tim Bell, CERN 30

Page 31: CERN: Accelerating Science with Puppet - PuppetConf '12

Next Steps• Expand tool chain

– Mcollective– Puppet-DB

• Deploy at scale in production– Move towards 15,000 hypervisors over next two years– Extimate 100-300,000 virtual machines

• Work with labs on common solutions for scientific computing– Batch system configurations– Grids– Publishing to http://github.com/cernops

• Investigate desktop and device management– Linux desktops– Macs– KVMs, PDUs

PuppetConf 2012 Tim Bell, CERN 31

Page 32: CERN: Accelerating Science with Puppet - PuppetConf '12

Final Thoughts

PuppetConf 2012 Tim Bell, CERN 32

• A small project to share documents at CERN in the ‘90s created the massive phenomenon that is today’s world wide web

• Open Source• Vibrant community and eco-system

• Working with the Puppet and OpenStack communities has shown the power of collaboration

• We have built a toolchain in one year with part time resources

• Running 15,000 servers and up to 300,000 VMs is scary but achievable

• Looking forward to further contributions as we move to large scale deployment

Page 33: CERN: Accelerating Science with Puppet - PuppetConf '12

For more details, see Ben Jones’ talk at 15:50 todayConfiguration Management at CERN – From Homegrown to Industry Standard

Tim Bell

Page 34: CERN: Accelerating Science with Puppet - PuppetConf '12

References

PuppetConf 2012 Tim Bell, CERN 34

CERN http://public.web.cern.ch/public/Scientific Linux http://www.scientificlinux.org/Worldwide LHC Computing Grid http://lcg.web.cern.ch/lcg/

http://rtm.hep.ph.ic.ac.uk/Jobs http://cern.ch/jobsDetailed Report on Agile Infrastructure http://cern.ch/go/N8wp

Page 35: CERN: Accelerating Science with Puppet - PuppetConf '12

Backup Slides

PuppetConf 2012 Tim Bell, CERN 35

Page 36: CERN: Accelerating Science with Puppet - PuppetConf '12

CERN’s tools

• The world’s most powerful accelerator: LHC– A 27 km long tunnel filled with high-tech instruments– Equipped with thousands of superconducting magnets– Accelerates particles to energies never before obtained– Produces particle collisions creating microscopic “big bangs”

• Very large sophisticated detectors– Four experiments each the size of a cathedral– Hundred million measurement channels each– Data acquisition systems treating Petabytes per second

• Top level computing to distribute and analyse the data– A Computing Grid linking ~200 computer centres around the globe– Sufficient computing power and storage to handle 25 Petabytes per

year, making them available to thousands of physicists for analysis

PuppetConf 2012 Tim Bell, CERN 36

Page 37: CERN: Accelerating Science with Puppet - PuppetConf '12

Our Infrastructure

• Hardware is generally based on commodity, white-box servers– Open tendering process based on SpecInt/CHF, CHF/Watt and GB/CHF– Compute nodes typically dual processor, 2GB per core– Bulk storage on 24x2TB disk storage-in-a-box with a RAID card

• Vast majority of servers run Scientific Linux, developed by Fermilab and CERN, based on Redhat Enterprise

– Focus is on stability in view of the number of centres on the WLCG

PuppetConf 2012 Tim Bell, CERN 37

Page 38: CERN: Accelerating Science with Puppet - PuppetConf '12

New architecture data flows

PuppetConf 2012 Tim Bell, CERN 38

Page 39: CERN: Accelerating Science with Puppet - PuppetConf '12

OpenStack

PuppetConf 2012 Tim Bell, CERN 39

Gold Members