Modeling Must Do’s – Part I

• Proper Cooling Path Modeling for Centralized Distribution Systems• What aspects must be modeled when analyzing central distribution systems

modeled?• Rooftop delivery cooling units• Vertical plenum• Ducted delivery

• How is this different from a underfloor only cooling path?• What type of simulation results must be looked at for proper analysis?

• Proper Use and Analysis of Containment systems• Does containment always work?

• Does it need to be modeled?• Does containment affect the airflow delivery under the raised floor?

• What simulation results matter?• How can we improve a room using containment?

Modeling Must Do’s – Part IIMinimum CFD Inputs

• Accurate Representation of IT Devices• How does representation of air jets from high density servers, such as blades,

affect results?• How to model the affects of blanking plates and leakage through rails

• Buoyancy modeling• How much does buoyancy affect the flow field?

Modeling of Centralized Air Handling Schemes

What is a Centralized Cooling Distribution System?

Why are these being Used?

• Centralized cooling provides better redundancy – the air is mixed prior and after entering cooling infrastructure

• Cost • Less floor space required• Maintenance is away from computing space

• These scheme is different from raised floor perimeter cooling design• Plenum orientation is different• Exhaust path is different• Mechanical cooling is different

• For all these reasons – this must be modeled and analyzed properly!

What must be considered?

• The entire cooling path must be considered – from cooling air supply to heat load to exhaust path back to the cooling device• Penthouse or gallery design

• How are the units characterized?• How is the space characterized?

• Mixing chamber or plenum• Does the cooling air get mixed? This will distribute the cooling load

amongst the cooling units• How does the air enter the data center space? This can be with the use of

a raised floor or direct from the vertical plenum• How is this different from raised floor design?

• All these details must be taken into account! How?

Sample Central Air Distribution System

Penthouse Air Handlers

Vertical Plenum

Internal Wall Air Distribution

Equipment Racks on a Non-Raised

Floor

Complete Flow Path

Airflow Distribution Concerns

• How is the pressure/airflow distributed in the vertical plenum?

• How is the air distributed coming out of the vertical plenum?

• What consequences (symptoms) are a result of certain design choices

Design Variations

24 in

32 in

• Vertical plenum acts similarly to a raised floor plenum

• Delivers air from cooling units above to equipment on the ground

• 2 Variations are being modeled

• 24 inches versus 32 inches

• Typically, air distributes better in a higher raised floor – does this apply to vertical plenums?

Pressure Distribution – PLENUM WIDTH CALLOUT

• Different pressure profiles exist going from the plenum into the computing space

• The wider plenum distributed pressure more evenly

• Similar to a higher raised floor distributes perforated tile air more evenly

• Simulation must be used to determine an adequate width of the vertical plenum

• How wide is wide enough?

Vent Velocity Plot

• Different discharge profiles can be seen between the two different designs

• Discharge profiles are dependent on the plenum design and NOT uniform

• The wider plenum has a more even distribution from top to bottom of the vent

• Simulation must be used to determine an adequate width of the vertical plenum and also provides a platform to design the distribution vent

• Similar to raised floor depth and perforated tile / grate type deign

Vertical Velocity Plot – MAKE A MORE EASY TO DIFFERENTIATE IMAGE

• Vertical slice through the center of the cold aisle shows the velocity profile

• The two different plenums create a different velocity profile in the cold aisle

• Simulation shows the affect of the gallery width to the airflow in the cold aisle

Velocity – SCALE

• High Velocity air (in excess of 1200 fpm) penetrates into the cold aisle more in the case with the more narrow vertical plenum

• Cross flow in excess of 800 fpm is NOT RECOMMENDED for computer room design

• Simulation is necessary to evaluate the affect of the vertical plenum on cross flow velocity

Temperature

• Hot spots develop in both cases

• The case with the wider vertical plenum develops a smaller hot spot

• These are symptoms of the overall issues with the delivery system – the plenum design must be examined more

Centralized Distribution System Must Do

• The delivery path must be analyzed from the air handler down to the equipment and back

• The entire cooling path must be considered and modeled properly• Proper penthouse layout• Proper vertical plenum dimensions• Entry of air into underfloor/datacenter space• Air passing by server inlets• Return of exhaust air

• Critical variables are all important• Pressure distribution in the vertical plenum• Velocity of cross flow in front of the computing equipment• Temperature of computing space and server inlet air

Modeling of Containment Solutions

Various Applications of Containment

• Containment is being implemented for various reasons• Energy efficiency• Mitigating risk in critical facilities• Creating high density computing areas• Reducing server temperatures

• How has containment been implemented?• Retrofits for hot rows or high density rows• Overall upfront data center design

Uncontained Hot Aisle / Cold Aisle Distribution

ASHRAE Temperature Compliance

Two Areas of Concern

Containment Modeling

ADD PRESSURE SLIDE PRIOR TO THIS - Containment Modeling

• Containment worked in the local area it was intended for

• Containment took racks out of ASHRAE compliance at uncontained locations

• Modeling must be done and the entire room must be looked at for proper localized containment installations

Full Containment Design

• What happens if a design utilizes full containment of cold aisles?• Will containment always work?• What must be looked at to ensure a good design?

Case Specifications

• The following is a case of the same room with more challenging design envelopes• High flow servers

• 180 cfm/kW• High efficiency room – not a lot of excess cooling

Full Cold Aisle Containment

Uncontained Contained

Temperature Differences

Uncontained Contained

Where is the Hot Air Coming From?

• Can we just contain everything and seal the leakages?• Fundamental issues in the room WON’T go away with sealing and

containment

Full Containment

• Why is this room having problems?

Pressure Analysis

• Full cold aisle containment is similar to extending the raised floor plenum to the level of the floor• Pressure MUST be looked at!• Post-processing/Plotting MUST be looked at above the raised floor

Lower Pressure in Cold Aisle

versus Hot Aisle

Local Fix to Increase Pressure in Problem Aisle

• Since the fundamental issue is lack of pressurization, fixed can be done to increase pressurization in problem aisle• Floor tiles can be swapped out to grates in low pressure aisle

• In the following analysis, only this first remedy was tried. Other fixes can be used as well:

• Increasing flow in local area• Sealing leakages• Moving CRAC’s to more evenly distribute flow and pressure

Pressure Differences

• Changing out some of the perforated tiles creates a much better pressure situation in the contained cold aisle

Contained Contained with Local Fixes

ASHRAE Compliance

• Fixing the fundamental pressure issue also fixes the temperature issues

Contained Contained with Local Fixes

Proper Containment Modeling

• Not all containment is the same

• Freezer curtains, rigid partitions, various assembly schemes affect the effectiveness of containment

Proper Containment Modeling

Idealized (no leakage)

Containment

Containment Must Do’s

• Effect of containment on the entire datacenter MUST be analyzed• Local containment can negatively affect other areas of data center• Both the contained and uncontained areas must be looked at

• When using containment, pressure MUST be analyzed• Room level pressure is an indication of how effective a containment system can

be• Identifying low room level pressure allows the designer to look at local design

solutions and fixes

• Containment type MUST be characterized and modeled like they would exist in a real installation

• Appropriate leakage must be looked characterized and applied• Realistic gaps must be modeled

Minimum CFD Inputs for ANY Data Center Model

Capturing Exhaust Jets

“Simple CFD” – • Little or no capability to model multiple

outflows – lumped outflow is best way of modeling…

• The outflow for this model is same height and width of the device

RoomLite - Outflows are modeled as per specification i.e. width and height

Thermal Footprint – ADD TABLE TO SHOW DIFFERENCE IN AREA AND VELOCITY

Both servers are the same power density and same exhaust flow volume but have a different thermal footprint

Sample Room

• 7 Racks with IBM Blade Center Chassis

• 2’ Raised Floor• Single ACU• 30% Perforated Grills

Result of Modeling Details

With Accurate Thermal Footprint of Blade Center Fans

With a Lumped Exhaust from Blade Center

Demonstration of Ducted Delivery and Buoyancy

• Overhead ducted Air delivery system

• Return Vent in hot aisle

• 3.5 kW per cabinet

Demonstration of Ducted Delivery and Buoyancy

Gravity Modeled Gravity Ignored

Buoyancy driven flow not captured when Gravity Ignored. Hot air seems to sink!

Demonstration of Ducted Delivery and Buoyancy

Gravity Modeled Gravity Ignored

Major Variation in Airflow Paths

Demonstration of Ducted Delivery and Buoyancy

Gravity Modeled Gravity Ignored

Results in Major Variation in Cabinet Inlet Temperature