cern computer centre consolidation project
DESCRIPTION
CERN Computer Centre Consolidation Project. Vincent Doré IT Technical Forum - 10 th June 2011. Overview. Current configuration of the CC (excluding B613) Current issues Consolidation project Goals Technical choices Plans and views New IT room Main dates - PowerPoint PPT PresentationTRANSCRIPT
Computing Facilities
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF
CERN Computer Centre Consolidation Project
Vincent Doré
IT Technical Forum - 10th June 2011
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF Overview
• Current configuration of the CC (excluding B613)• Current issues• Consolidation project
– Goals– Technical choices– Plans and views– New IT room– Main dates
• On-going work outside the project• Questions
Computer Center Consolidation – VD - 100611 - 2
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF Current configuration - 1
• Designed and built in early 1970s• Fully refurbished around 2000 • 2 types of equipment both on UPS:
– Physics (single power feed, no diesel backup) – Critical (dual power feed and diesel backup)
• Power capacity:– Nominal current capacity: 2.5MW (including 240kW of critical
power)– Extended (usable) capacity: 2.9MW (including 340kW of
critical power) forfeiting redundancy for all UPS systems• 2 main IT rooms (R-050 -1400m² & Vault - 1200m²).
Several dedicated labs (telecom)
Computer Center Consolidation – VD - 100611 - 3
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF Current configuration - 2
• 3 physics UPS, each with 4*400kvA modules• 1 critical UPS with 2*300kVA modules• Air cooling installations: for IT equipment including telecoms
(capacity up to ~2.5MW), technical areas... and offices– (3+1) Chillers not secured by UPS nor diesel (water at ~5C)– Pumps and ventilation units secured by diesel – Water-tank for stored cooling capacity
• Water cooling installations: for IT physics equipment (capacity up to 800kW)– (1+1) Chillers not secured by UPS nor diesel (water at ~14C)– Pumps secured by (physics) UPS– 80 water-cooled racks secured by physics UPS– Water-tank for stored cooling capacity
Computer Center Consolidation – VD - 100611 - 4
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF Current issues - 1
• Cooling:– Insufficient cooling for critical UPS room – Impossible to improve the cooling in critical UPS
room– CC not cooled when running on UPS without diesel– Insufficient cooling when running on diesel
• IT critical equipment running• Pumps & Ventilation Units running• No chiller and insufficient stored cold water
– Offices in B31 cooled by the CC
Computer Center Consolidation – VD - 100611 - 5
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF Current issues - 2
• Power– Insufficient critical power available – No redundancy for critical UPS (> 240 kW)– No redundancy for physics UPS (> 2.7 MW)
• Other– Limited fire protection in B513– Critical areas and physics areas strongly coupled
• Share the same locations, cooling infrastructure and fire risks)
• Improving critical infrastructure implies improving physics one... at an expensive price
Computer Center Consolidation – VD - 100611 - 6
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF Consolidation Project
• Main goals– Secure the cooling and fire-protection of Critical IT
equipment... Which should be able to run as long as power is available (for ever on diesel)
– Increase the available critical power for IT equipment– Restore N+1 redundancy for both critical and physics
UPS– Decouple the cooling of B31 from the CC
• Main points on the wish list– Increase the physics power (if cost-effective)– Secure the cooling of physics equipment
Computer Center Consolidation – VD - 100611 - 7
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF Technical choices - 1
• Upgrade of the electrical infrastructure– Total capacity 3.5 MW of IT equipment including 600kW of
critical equipment– N+1 redundancy for critical and physics UPS
• Separation of the critical and physics infrastructure– 3 critical areas :
• new IT room (450kW)• extended telecom labs (50kW)• legacy network area in R-050 (100kW)
– New dedicated cooling installation for the critical areas (dual power feed, diesel)
Computer Center Consolidation – VD - 100611 - 8
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF Technical choices - 2
• Construction of a new partially sunken building dedicated to the cooling of the critical areas– (2+1) new chillers– 1 new storage tank– Pumps – New ventilation system for the critical UPS room
• Conversion of the Barn area to house– 2 Electrical rooms (batteries, UPS and switchboards)– An IT room with water cooled racks for IT critical equipment
(up to 480kW)– New ventilation system for the legacy critical network area
in R-050 and for new electrical rooms
Computer Center Consolidation – VD - 100611 - 9
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF Technical choices - 3
• Modification of the telecom area– Extension into adjacent labs– Fire protected walls– New ventilation systems
• Modification of the technical rooms (basement)– Move of the critical UPS (from S-435 to S-437)– New cooling control (in S-400, S-401 and S-402)
• Addition of 2 new electrical transformers (adjacent to the existing ones)
• Addition of 1 new water tank for physics equipment (adjacent to the existing one)
Computer Center Consolidation – VD - 100611 - 10
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF 3D view from the car park
Computer Center Consolidation – VD - 100611 - 11
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF 3D view of the ground floor
Computer Center Consolidation – VD - 100611 - 12
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF Ventilation in the Barn
Computer Center Consolidation – VD - 100611 - 13
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF Telecom critical area
Computer Center Consolidation – VD - 100611 - 14
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF 3D view of the basement
Computer Center Consolidation – VD - 100611 - 15
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF The new IT room
• Critical IT equipment only (450kW, room designed for 480kW)
• Cooling of IT equipment still being discussed, most likely water-cooled racks with passive heat exchangers– No power consumption– Cheap (compared to other cooling solutions)– No fan problems– No cold/hot corridors, cold room– Cooling issues for heterogeneous equipment???
• Very likely 90 racks (70cm*110cm*47u), average power density 5kW/rack
Computer Center Consolidation – VD - 100611 - 16
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF Views of a passive rack
Computer Center Consolidation – VD - 100611 - 17
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF Plan of the new IT room (draft)
Computer Center Consolidation – VD - 100611 - 18
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF Important dates
• Autumn 2009 : serious discussions with EN• May 2010 : approval of the project• October 2010 : barn ready for dismantling• January 2011 : preliminary electrical works• May 2011 : start of the main civil engineering works • Summer 2011 : MS and IT for the water-cooled racks• November 2011 : end of the main civil engineering works • November 2011 : start of the main cooling works• November 2011 : start of the main electrical works• May 2012 : start of the works for the additional water tank• July 2012 : end of the main cooling works• July 2012 : end of the works for the additional water tank• August 2012 : end of the main electrical works• October 2012 : end of the works
Computer Center Consolidation – VD - 100611 - 19
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF Work outside the project - 1
• Problematic cooling of the Physics equipment– When running on UPS without diesel power : IT
physics servers up... without cooling– No easy solution. We need power (500kW) for the
pumps and ventilation in R-050 and the Vault (>500kW)
– No agreement reached when the consolidation project was approved
– However... on-going discussions/investigations and a good chance to power the ventilation units from the UPS
Computer Center Consolidation – VD - 100611 - 20
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF Work outside the project - 2
• Improvement of the cooling efficiency– To be able to cool more IT equipment and to save on
cooling power– Main steps already achieved
• Cold corridors (from 2008)• Temperature increase of blown air in R-050 from 14C to 16C
(September 2010)• (More) use of outside fresh air (February 2011, net
immediate gain 200kW)• Blown air in R-050 from 16C to 18C (2 May 2011)• Blown air in R-050 from 18C to 19C (11 May 2011)
Computer Center Consolidation – VD - 100611 - 21
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF Work outside the project - 3
• Improvement of the cooling efficiency– Future steps
• Continue to increase the temperature of the air blown in R-050 (up to 22? While ensuring air < or = 27C for all IT equipment)
• Increase the temperature of the Vault and possibly of some UPS rooms. Discussion starting with EN/CV
• Better understand and improve the air flow from the main ventilation units to the servers in R-050
• Continue to improve cold air confinement where relevant• Increase the temperature of the chilled water?
Computer Center Consolidation – VD - 100611 - 22
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF Work outside the project - 4
• Improvement of the monitoring of the infrastructure of the Computer Centre– Important work already done to better understand
the power flows in the CC and store/display them in Lemon. .. Still some work to do, in particular on the alarms
– Work to be done to include temperatures and related alarms in Lemon
– Progressive implementation of intelligent power bars in IT racks (end-2011 installations & new IT room)
Computer Center Consolidation – VD - 100611 - 23
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF Video on the outside work
• To upload
Computer Center Consolidation – VD - 100611 - 24
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
CF
Questions?
See also http://cern.ch/it/cc2012
Computer Center Consolidation – VD - 100611 - 25