Electronic Cooling Solutions Inc.

• Thermal management consulting company

• Located in the heart of Silicon Valley

• Provide solutions for thermal design problems

• Use of experience, modeling & experimental methods in the design process

• Clients include over 60 companies

2
Outline
• Introduction
• Need for Innovative Cooling Solutions
• Objective
• Scope of this Presentation
• Cooling Solutions – Air-based Cooling Systems
– Water-based Cooling Systems
– Refrigerant-based Cooling Systems
• Comparative Analysis
• Conclusion
• Vendor Contacts

3
 Introduction - Power Density and Heat Load Trends
• Electrical Power = Waste Heat
• Increase in power densities from the CPU-level to the System-level

Component Board-level System-level Rack-level Room-level

level

• Current power density at the rack level = 1 to 3 kilowatts

(up to 30 kilowatts per rack in two to four years - Hannemann and Chu ‘07)
4
Need for Innovative Cooling Solutions at Rack-Level

• Datacenter TCO is characterized on a per rack basis

• Addition of newer and higher powered equipment in existing
datacenters
• Hotspots in datacenter resulting from high-density servers
• Hotspots resulting from unavailability of cooling air from CRAC units
• Design shortcomings within the rack result in inefficient cooling

5
Objective
• Cooling of equipment in rack using airflow and impact of minor design
changes for better cooling
• Cooling of high density equipment using water-based cooling techniques
• Cooling of high density equipment using refrigeration-based cooling
techniques
• Provide vendor data for the above mentioned products

6
Scope of this Presentation
• Limited to discussion of cooling solutions only at the rack level
• Closed racks and commercially available products
• Discussion on Component-level, Board-level and Room-level cooling can
be personally consulted

7
Airflow-based Cooling Techniques

8
Cooling of Racks with Conditioned Air
• Most easy to implement and maintain
• Limitation based on cooling capacity, acoustics and power consumption
• Detailed analysis could improve efficiency of air cooling

Application of Computational Fluid Dynamics (CFD)

• Pressure drops and airflow patterns
• Determine by-pass air
• Determine areas of re-circulation
• Determine failure modes of cabinet fans
• Requires testing to develop confidence in models

9
Cooling of Racks with Conditioned Air

Airflow Enhancement in Racks

• Based on CFD Analysis
• Sample case study 1
• Sample case study 2
• Cabinet powered fans – airflow layouts
• Best practices
• Airflow enhancing products

10
Sample Case Study 1

• Study done by Electronic Cooling Solutions Inc.,

• 42U Cabinet fitted with twenty one 2U units
• 15.75 kilowatts per rack
• Inlet temperature of 40 deg C
• Airflow – 1932 CFM
• 10,000 ft Altitude Conditions
• Simplified cabinet shown here
• Objective was to optimize the cabinet for better
cooling/use of higher powered equipment

11
 Sample Case Study 1 (Contd …)

Higher
temperatures
at the inlet
sides

Inlet Temperature Re-circulation of Flow 12

Sample Case Study 1 (Contd …)
• Added vertical blockages

Open space Vertical

between the blockages
rack rails and between the
cabinet sides rack rails and
cabinet sides

13
 Sample Case Study 1 (Contd …)
• Added blockage above the topmost unit of the rack

Top of 2U server

Top of rack cover meant Area above the

for passing cables from topmost 2U rack is
front to rear blocked

14
Sample Case Study 1 (Contd …)

15
Sample Case Study 1 (Contd …)

Comparison of
Inlet Temperatures

No blockages Blocked passages

16
Sample Case Study 1 (Contd …)
Fixed Exhaust Temperature (deg C) 55 60
No Blockages With Blockages No Blockages With Blockages
Mean Inlet Temperature (deg C) 43.9 41.2 43.9 41.2

Fixed Airflow 92 CFM

Power (W) per unit 397 493 575 675
Total power of Rack (kW) 8.3 10.3 12.1 14.1

• Current study shows 17 to 25 % increase in power dissipation based

on exhaust temperature.
• By blocking re-circulating flow, it is possible to use higher powered
equipment in the rack.
• Blocking can be done by employing Brush Strips.
• Avoid using larger racks with rails set to lower rack width settings
(Using 23” rack with rails set to hold 19” equipment)

17
 Sample Case Study 2 Container wall

• Study done by Electronic Cooling

Solutions Inc.,
Heat Exchanger
• Racks placed in containers 2U Servers
2U Servers
• Create airflow model of blowers
• Evaluate alternate designs for
blower module
Blower Module

Symmetry Walls Rack Model

18
Sample Case Study 2
Testing

Module 1 – 860 CFM Module 2 – 1024 CFM

• Apprx 20% increase in airflow with the re-designed

baffles and perforated casing

Testing to collect
data for modeling 19
Cooling of Racks with Conditioned Air

Airflow Enhancement in Racks

• Based on CFD Analysis
• Sample case study 1
• Sample case study 2
• Cabinet powered fans – airflow layouts
• Best practices
• Airflow enhancing products

20
 Airflow Enhancers – Cabinet Powered Fans Ceiling

Front or
Footprint Inlet

Raised Floor Supply Air Raised Floor Supply Air

Front In – Rear Out Front In – Top Out

21
 Airflow Enhancers (Cabinet Best Practices)

Raised Floor Raised Floor

Supply Air Supply Air

22
Airflow Enhancers (Cabinet Best Practices (Contd…))
Dropped Ceiling Dropped Ceiling

Raised Floor Raised Floor

Supply Air Supply Air
23
Airflow Enhancing Products
•APC 2U Rack Air Distribution
•Delivers air directly from the raised
floor into the rack inlet
•Minimizes top-bottom inlet
temperature distribution
•Allows rack loads up to 3.5
Bottom-Top Rack Air Distribution Unit
kilowatts per rack

Side Rack Air Roof Air Removal

Distribution Unit Unit
Application View

Images: APC (www.apc.com)

24
Airflow Enhancing Products (Contd…)

Rittal Side Breathing Air

Rittal Enclosure Blower Baffle System
Images: Rittal (www.rittal-corp.com)

•Higher density rear door rack air removal unit

•Allows rack loads up to 16.5 kilowatts /14
kilowatts per rack
•Challenges in obtaining flow through tiles in
the datacenter APC Liebert
XDA
25
Images: www.apc.com and www.liebert.com
Water-based Cooling Techniques

26
Water-based Cooling
• Basis: QLOAD = mCp DT = rVCp DT
(Water has 3000 times higher heat carrying capacity than air)
• Chilled water from building supply
• Cooling high density servers up to 70 kilowatts per rack
• Lower energy cost as some of the CRAC units can be removed
• Avoid hotspots due to high power-density equipment
• Possible to have redundant systems (Chillers, pumps, piping, and power supply)
to avoid downtime
• Importance of CDU
• Electrically conductive, corrosiveness and high flow rates

27
Water-based Cooling
Combination of Air and Water Cooling Pure Water-based Cooling
Heat Exchanger

Raised Floor Raised Floor Heat Exchanger

28
Water-based Cooling
• Cooling system design by Naissus Thermal
Management Solutions
• Heat removal of 20+ kilowatts
• Closed liquid loop with bottom mounted fin
and tube heat exchanger
•Thermal test done with 5 blade servers Heated Air
Performance Table - North American units SI units
Inputs Heat Load 24.0 kW 24.0 kW
Water Temp 12.8 degC 12.8 degC
Water Flow 0.76 l/sec 0.76 l/sec

Thermal Core
Results Cold Air Supply 22.8 degC 22.8 degC
Hot Air Disch 49.8 degC 49.8 degC
Air DT 27.0 degC 27.0 degC
Water Rise 7.1 degC 7.1 degC
Water Disch 19.9 degC 19.9 degC
Water P.D. 48.3 kPa 48.3 kPa
Background Server Fans 1680 CFM Varies by manufacturer Water from Chiller Water sent to Chiller
Fan Power 700 watts

29
Water-based Cooling

Temperature Distribution inside the Rack

30
Water-based Cooling
•Cooling system design by Vette Corp.
•Heat removal of up to 30 kilowatts
•Rear door closed loop liquid heat exchanger
designed by IBM
•Currently available only for IBM Enterprise Rack
•Available from Rittal for retro-fit designs

IBM Rear Door Heat

Exchanger
Images: www.vette-corp.com

Pressure drop across the heat exchanger for a typical 1U fan setup 31
 Water-based Cooling
•Cooling system design by
APC.
•Heat removal of up to 70
kilowatts
•Controlled in-row cooling
•Row air containment
•Modularity
•Similar designs from HP (35
kilowatts)
•Similar concepts available
from Rittal (30 kilowatts)
•Similar concepts available
from Liebert (8 kilowatts
and 17 kilowatts)
Heat Exchanger and Front View Rear View
Fan Assembly 32
Images: www.apc.com
Refrigerant-based Cooling Techniques

33
Refrigerant-based Cooling

•Phase change (latent heat transfer)

•Electronics-safe
•Low flow rates and non-corrosive
•Some systems are stand-alone and hence flexible
•CRAC units are the most common ones
•Chilled water from building supply may be used for supplemental cooling
•Expensive ( comparable to water+ additives)

34
Refrigerant-based Cooling
•Cooling system design by Liebert
•XDF- Cooling capacity of 14 kilowatts
•Stand-alone unit

Liebert XDF Self Contained Unit

Images: www.liebert.com

35
Refrigerant-based Cooling
•Cooling system design by APC.
•Heat removal of up to 43 kilowatts
•Modularity
•Rack air containment

APC In-Row Cooling

36
Refrigerant-based Cooling
•Cooling system design by Liebert
•XDV- Rack mount air conditioners (10 kilowatts) - Almost no floor space required
•XDH-Rack Cooling capacity up to 30 kilowatts
•Also available from Rittal

Liebert Roof Mount Cooling Liebert In-Row Cooling Rittal Rear Door Hx

Images: www.liebert.com 37
Liquid Based Touch Cooling

38
Water/Refrigerant-based Touch Cooling

•Direct contact cooling

combined with chip
cooling
•Remove heat at the
source
Cold plate with Liquid Cooling Liquid Cooling of Boards
Images: www.rittal-corp.com Images: www.ibm.com

Available from:
Clustered Systems
Rittal (Power electronics)
SprayCool (20 to 30KW)

Spray Cooling
Images: www.spraycool.com
39
Refrigerant-based Touch Cooling
•Cooling system design by Thermal
Form and Function
•Pumped liquid multiphase cooling
•Heat removal of up to 10 kilowatts
per evaporator (Modular)
•Designed for retro-fit applications
•Air/Water cooled condenser unit can
be used

Thermal Form and Function

Refrigeration Unit

Images: http://www.thermalformandfunction.com/

Two Phase Flow 40

Comparison of Cooling Techniques

41
Comparative Analysis

• Study by Hannemann and Chu –

Interpack ’07
• Comparative study of cooling
technologies with a model
datacenter

Capital Expenditure of Cooling Equipment

Area required for Cooling Equipment Power Consumption of Cooling Equipment 42

Conclusion

• Reviewed innovative and commercially available technologies for cooling

racks
• Discussed design approaches with use of CFD to maximize performance of air
cooling
• Reviewed products and techniques for enhancing airflow within a rack
• Reviewed cooling of high density equipment using chilled water and refrigerant
• Selection of cooling strategy will depend on the specific requirements of the
client

43
Vendor Contacts

• APC – Morrison, Harold Wells Associates (925-355-9900)

• Rittal – Sales (800-477-4000)
• Liebert – Frank Stone (925-734-8660)
• Spray Cool – Sales (866-993-2665)
• Clustered Systems – Phil Hughes (415-613-9264)
• Trox AITCS - Thomas Hudgens (347-325-4347)
• Thermal Form and Function – Joe Marsala (978-526-9672)
• Vette Corp - Skye Emerson (508-203-4694)
• Naissus Thermal Management Solutions - Mirko Stevanovic (416-892-4071)

44
References
• Product websites and communication with vendors
• Hannemann, R and Chu, H., (2007), “Analysis of Alternative Data Center
Cooling Approaches”, ASME Interpack 1176, Vancouver, BC.

45
Acknowledgement
• Speck Design
• Our colleagues at Electronic Cooling Solutions:
Khyati Varma
Ceferino Sanchez
Adriana Romero
Sridevi Iyengar

46
47