Dat:a Cent:er

Best:

Version
Student: Cuide
• •

DC110
Data Center
Design
Best Practices
o Copyrlgh l 201 0 by BICSI

Printed in the United States of America

Bicsi"
Version 1.4, First Prin ting

All righlS re\!'"rwd. No pan of this workbook may be used, reproduced, or uan'>min ed in any form or
by any means, electronic or mechanical. including photocopymg, recording. or by any information
storage and retrieval system, without prior agreement and "'fltten per mission from t he pu blisher,
except for the inclusion of brief quotat ions i n a review.

Thecoments of th is wo rkbook are subject to revision without nol ice due to continued progress in
met hodology, design, and manufacturing.
THE GUIDE IS PART OF TH E BleSI DESIGN COURSE CURRICULUM AND IS
PRESENTED H 1\ WITHOUT WARRANTY OF ANY KIND, RESPECTING THE
CONTENTS OF THIS GUIDE INCLUDING BUT NOT UMITED TO IMPUED
WARRANTIES FOR THE GUIDE'S QUALITY, PERFORMANCE, MERCHANTABILITY,
OR FITNESS FOR ANY PARnCULAR PURPOSE. BI CSI SHAll NOT BE LIABLE TO THE
PURCHASER OR ANY OTHER ENTITY WITH RESPECT TO ANY LIABILITY, lOSS, OR
DAMAGE CAUSED DIRECTlY OR INDIRECTLY BY TH IS GUIDE.
All brand names, trademarks, and registered trademarkS are th e property of their respective holders.
BleSI lX'odd He.ld<juarlcr ~
8610 Hidden River Parkway
Tampa, Fl33637· 1000 USA
800.242:.7405;+ 1813.979. 1 991 ; ~x "' 1813 .97 1.4311
\Neb si ll': www.blcsi.org
S1"dom Guido o..t.t Centw& & Tho Datolgn Pmco..

CQU~: DC 110 Data Center Design & BlI&t Practices

This Is;, c:ourse em Oat.l Center Design and Best Practices, the plimery
source for the matenal presented here Is:

BICSI-002 Data Center Design and Implementation Best Practices

S..-Ouldo

Module 2: Data Centers and the Oeslgn ProC:e$$

Module Goals 2.1
lNs introductory module is meant to introduce !he basic cor'ICepr:s
assoClated with data centers and data center design; after finishing
this introductory module you should be able to:
• Identify and distinguish the 3 primary types of data centers
• Identify !he typical structures and systems thai make up a data
center
• Identify !he 4 bilsic stages of data center design

...
llIat Pl""' .... y Pro"ft)'-OG NootCop,
Stud ..... GukIo Data C ........ & Tn. Doslgn Proe ...

Data Canhtr Ovarvlew 2.2

Data Centers contain mission critical computer systems. In addition to
computer hardware, they typically Indude:
o Sped.,.1 environmental controls and cooling systems
o Specia I electrical systems and backu p power supplies
o Redundant data communication connections
o Special high-security systems
o Build Ing automation Ii 'control systems

Data Center Types 2.3

There are three basic types of data centers:

o Enterprise Data Centers
• Internet Data Centers
• Collocation Data Centers

Enterprise Data CenleB 2.4

Enterprise data centers serve a single cllent·s data proc.....ng and IT
eq uiproent storage needs. Enterprise data centers typlca Ily possess:
o Placement on the same physical site as a dient's other business
operations
o less redundancy
• Cost management as a primary design driver

Internet Data Centers 2.5

An Internet Data Center (or 'managed hosting facility') manages IT
systems and applications fot off-site customers. Hos~ng fadllties
typically encompass:
• Higher hardware densities to maximize potential revenue
o Greater redLlndancy
o Availability/uptime as tile primary desJ9n driver

<4OI.aiC$/'
Slude", G.. de

Collocation Data C.."ler 1.6

A ooIlocatlon facility provides space, power, and cooling for different
customers' IT ~quillme"t. Collocation faci Hties typic~lIy posS</SS:
• Uirqer floor are". tha" Enterprise alld Intemet data centers
• Lower equipm,,"t densities
• More centralized patch fields and distribut ed networks
• Seculity and spacing requirements as primary design drrvers

Data Center Deslgn-$trutlures \IS. Systems 2.7

Data Center desiqn involves not only building design and layout, but
also \t1e design and Jayout of Infrastructu~ systems, such as the
design power, rooling, voice. & <lata cabling, and control systems

VOK:~&
Power Data
C.bling

Autom.~on &.
Cooling Control
~ Syslenu

Component Systems CDmponent Structures

81udontGuldo Data Cantlrs & The DHlgn Procan

Component Structures 2.8

The physical components of data centers Include:

• The computer room
• The buildi ng support spaces
• The supportJng off-sire services and structures

~~ v ll~ 'PI I'

l..uk ..II'I'"'"f.iII .... \ ~"".,II'"' .....
. ..,
0UCIfflS .Jtn!1IG s~ IlllUlIIe:
~,,.*

............ ,
I

............
D
Et'u.J1K1 roo!'I()l DltAQtIll:efd~ .l1li
_tI\Tif'rC.Jl ~ m .

,
"\ / "

0
~room,allll
Tw"""m~~1

~"'" .... ...... II I

~'I~nll
~
lMtJ...,ooclcl

;
I
can,llwbl. Raon
The Compuler Room 2.11
The heart of a data center is a computer mom housing IT equipment
racks, a main cabling distribution "",,,, and often a special area for
gigital information storage (or Storage Area Network (SAN»

,- - - ;i~c;.,;;,,;-~ - -- I
I I
- '-
"~-·a
I M ""'. ...-
~""r=Q '''''A. t--~"'l{It'U
::lo" .... """M I

-
OfIIbot . c.-~
c - t .• ~
.i
,.1-hI<\.I~'
: e:. .."i!
I
- --
.......
( AI .. ,..

MaiI1 Oi$(ribullon
- ___ I
,---- .... ---,
I
I -, """-
""'" ~i~ I
I
Te l..ooro RMm
n.r..:.tII~'" A,... ~~ . I
I1r.ot11Ont.1ll Cro!;1li'COOI'1llU
,oa,:, e eq.rp!T'el1 ~
""'"'" /l,t.:lln elU!!l!f-Cllm-..'1,i
~1f1. 1 Hdril'QI'IUoI
C:jIm4If £qu~ 0I-
o.JJWn:CI~DI'l
I
I
QI~,-r.-nt'\I ""..t I
. I ~~Ir' l~"'~R~_1

,,/
"
/ ~laI
.,..DIroIll'ilMltivn

ijMi:....,1,fI 1 Ct~1WI9d
~
.....
OI .... lmutioII

HlilrIllll!Ml c~·c.:..~
." ""A():,... f._"".
.,~' ! "

,,1

,,':
,J
student Guide Data Coni". & Tho Doo;go P"' .....

The Building 2.10

The computer room's location within a particular build ing needs to be

designed w~ power, security, ilnd cooling recuirements foremost in . . ';1;-; i,,>(»">
mind ,"" ~ - ' t .··~ - (' ~~ 'A .
f
--I
;>'
- - - - - - o5tlrr.1l1t M~u'~

"1.1-0", I
~ l' ''' EI ",,~
~~ ....'''. lon:~" olI,. NC1'.I.oro I t ".crf~~ ~ ·

""" IV,'".. 1'I c:.c.o: I Aor",,~.

... ; ...
- - - - - - -- I ~

" ,.. Ccn> PLr.:lf ~

,.....,
I
- - --- ---- - -I ~~~ '"

I H"",,~vr, I !orI~' IIr.d: "~I I .. »

I r)~ , .
L - - -- --- - - - J 5.~~,.:t!

JiIl.!'''''~ 1 wcu"'" <~~:

~~~.

~~~r~L=I.l.;lln=

""',
MQ~ ~.",
ep,i l ~ r
!
';'OE;<r
I. l;<=Im
~ I .r .....,.,., ....' mt~' ""~ ~

DC 11.: DMI c~ DwiIP1" "f"~

H
81CS1 Propftet8ryPropert)'--llo Not Cop.,
Student Guide

Off-sfte 2.11
Off'slte components of the d. ", centerolten Include:
• Power and Telecom Access Providers and Feeds
• External 'Generators
• Emmal Thertnil l St oroge I Water Tanks
• Offslte - Data Storagel Dis. ster RecOIIery F"dlities

I _ eu.o..n~ IIUIOI'« Wi_1OI , . ~

Q - H.I.......,.
11ft ~_

,,- "le ICl'

~

! · ...1"")4'11 .. _ to...." ._"" __ .. _

' _~ «o': _
.... . .... _ 1..
""" . I'II,\4 ..... ~

Data Center Infraslructunl 2.12

In addition to the physical components, data center design
encompasses inft-astructure systems and their inter' relations. These
systems include:
• The Electncal Systems
• The Heating and Cooling Systems
• The Telecommunications Systems
• The Automation ContnJl Systems

...
IICSI Prapri.t;;uy ~ frlot tOM!
Student ~Ide

Data CenlN Electrical Systems 2.13

Data Centers require b~cku~ electrical systems and generators to

ensure continuous operations and must SErve increasingly high
electrica I loads to hill her density blade se ..........

[hote Centar Coonng Systems 2.14

Mnst of the electrical load delivered to data center IT e<julpment will be
converted to heat which must be removed to prevent the e<julpment
from malfunctioning

Data Center Telecommunications & Cabling 2.15

The IT equipment in the data center will need to be connected witn
both internal and external computer networks, creating uniquely dense
cabling req uirements

Data Center Control Systems 2.16

Data Centers are Increasing ly designed with a 'lights out' philosophy,
minimizing access to equipment and creating the need for unique
monitoring and control functions

Data C.... ter V$. Standard Facliity Design 2.17

As a mlsslon-crttlc.1 facility , Data Center design differs from standard

facility design in many ways, induding:
- 10 to 30 times as much power as a standard office space
• Equipment densities 3 to 5 times denser than standard offIce s~ace

• Operating CC's t 2 to 3 lime higher than stan~rd office space

operating CQ.sls
&luaent Guidt

Integrated Design 2.18

The data ~ef1ter design process needs to refiect the different n~ture of
data ce'nter requirements
• Data center design should be integrated rlIther than linear to
ensure effldent use of IT, power, and cooling assets
• Maximum coordination is required during 111e earliest plMnlng
stages for efficient data center power usage. cooling. and cabling
arcn~res

Data Cenlar De8ign 2.19

There are 4 kev considerations in data center design:
• Avail. bllity
• Scalability
• Security
• Efficiency/Cost

AvaHablllty 2.20

The primary purpose of the data center is to ensure cuslOmer and

employee access to data and computer applications whenever It Is
needed

.Sample Downtime Costs Per Industry. per hour:

Stock brokerage: $7.000.000.

Finane<!. Credit Card: $3.000.000.
Bankins : $3.000.000
Media COmpa ny: $1.000.000
Cross·lndustry Individual Company average: $47.000

oDl'O'IO Bidil'
:>-10
BICSI p ...... taty pr,~ ~otCo~'1
StudonlGuldo 0 ... c.nhM'& & Tho Dooign PrOCH.

5<:alability 2.21

The potent i~1 for growth must also be consi~ere~ in ttle data center's
design
• Large businesses typically experience 50% growth In data per year
• Power reqUirements for a data center can dou ble over the course of
2 years
• IT equipment will often completely turnover over the course of 3
yea... IL . ,f
. ,.. r\rk .... ~:~~,.o.'\ ~ -:. ,J(t H.~~ S(....1.:... ''() (,.tr"'h_~ · )

Security

In addition to insuring reliable "ccess, the ~"ta center designer must

also protect against unauthon,.,d access "nd 10 •• of phy'i,.1 _ssets.
Secu rlty question. datil center·de5ign.,... need to add ress:
• How is the data to be protecte~ from criminal threat?
• HOW is the data to be protected from natural disaster~?

Efficiency/Cosl 2.23

Data center design must consider not ",,'y initial costs, but also the
total cost of a fadllty over Its IIfetlme. Considerations r<!garoing
effiCiency that designers should address Inelude:
• How much redundilncy is it worth p"ying for to ensure availabil~y?

• Are the power requIrements belnlJ met efflc",ntly?

• Is the IncINsed cost of newer equipment justifiable, given Its
nigher power ~nd cooliny requirements?
• Is !he building space being utilized effidently?

Client CommunlcalJon and tbe Design Process 2.24

The data center design process Is driven mOr<! by unique client IT

equipment needs rather than standard person nel spaci og and now
requirements
• Because of this, the data center deSigner must gather as much
inFormation from the dlent as possible about their Information
teclmology needs, plans, and goals
student Guido Data Cent*" &lh& Dallgn PI'OCOIIiS

Data Centers and the Mission' Critical Design Process 2.25

There are four essential steps In deslgnlng mission critical lacilltle~
such as data centers :
• Risk Ana lysis
• Problem Definition
• SOlutkln Development
• Implementation

The Design Process, Phase f: Risk Analyels 2.26

Risk analysis i. the process of balancing fulllre costs <>f downtime with
til e present costs <>1 p~ntion

• Data Center risk.analysis addresses three key design fealllres:

Operational reqUirements-the opportunity to suspend
operations lor maintenance
Availability reqUirements-the targeted uptime of the system
during operations
Impact of downtime-the Impact of unplanned dis,upHons

RIsl< Analysis & the High Nines of Reliability 2.21

Wh~e customers and clients may seek "6 nines" In terms of reliability.
risk analysis I~ the process of balancing that desire against the high
costs assodated with Insuring limited downtime

Reliability Visruption TIme per Year

99% as hours
99.9% 8.8 hoors
99.99% S3 Minutes
99 .999% 5.3 Minutes
99.9999% 32 seconds

O2IItO .1CSi'
 student Guide Data ~ & The D.,.lgn Proceso

Phase II: Problem DefInition 2.28

rusk-analysis estabflshes a target I'l!4labllity level for a facility while
problem definition concerns tne task of meeting this ta'llet In terms of:
• Facility space
• rr assets
• Project locatk>n
• Proj ect budget
The end result of the problem definition phase of design i. the creation
of the fBdflty program

Phase III: Solution Development 2.29

The third stage of Oata Center design Is the development of one

(occasionally two) design solution(.) to be submitted to the dient for
approval

The design solution phase usually involves:

• Planning
• Space Programming
• Cost Estimation
• Const",Clion Documentation

SoluUon Development: Planning 2.30

• Planning typically Involves coordination between technologlcol

consultants and architects regarding:
- Space
- Power
- ·Cooling
- Security
- Fioer leading
- Grounding
- Eledrical protection "J."". 'Ql \ .. '1.'\
- Telecommunication pathways

."
a!csIP~~NotcQFIY
StucMnt Guld. 0$ C."tar& & Tho DesIgn Pro:esl

Solution Development: Spac:e Programming 2.31

Space Progrclfnming requires estimating data center requirements at

rull capacity f1Jr:
• IT hardware &. ",cks
• Electrical Equipment
• HVAC equipment
• Space for personnel amI airllow
• TelecQmmunications equipment and cabling
• Furu re growth

Solution Development: Cost Estimation 2.32-33

Cost estimation involves COflsideration of one-time, recurrent, and

Intangible costs:
• One time costs:
- Real estate costs
- Local tax Incentlve~

- Cost to bring utilities to Site

- Demolition and site prepitrittion costs
- Cost of temporary 5e/Vlces to support IT migration
- Cost of equipment relocation
- ConSillrant costs
• Recurrent 'Costs
- Utility usage costs (power, water, sewer, gas)
- Telecommunication services
- Wages
- Lease costs
- Taxes
• Intangible Costs
- Availability of alternate or mu ltiple telecommunications access
providers
- Availability of utility se/Vices
- Availability ot transportation

H.
811:11 f'rupri.w.y p~ Mot Copy
Student <kIld. ON Centers &Thll o.sign P'tocMI

"haae IV: Implementation 2.34

Alter ~ clesi9" solution has been chosen, the final stage of data center
design takes place which Includes;
• Construction
, Commjssioning (· . . . 'c(Ii---"~~'d' +""" ;" O'~» [).'l~J"i)
Conflnlction 2.35
The construction phase of design typically lndudes the coordination of
• Structural proje<ts
• Mechanical projects
• Electrical prOjects
• cabling projects

Commission Ing 2.36

Commissioning Includes;
• In~lI.tion of vendor equipment
• Fi<!ld testirlll equipment
• Oevekll>ment and lmplementatlon of oper-..tio",,1 and maintenance
procedures
 Dot> C."".r& & Tho Dool911 Proan

Review Questions
1. What are the 3 types of Oata Centers?

,
- j
'::f~~tr r ,
I , .,~

" \: I'- ~v "'" \

.. \Uhl(f"L~ 'i ~."

2. What are the 4 key design criteria or data centers]

_ {I",,\ h !e" .l .. ~
- '7(,,\,,~ I,~
• S<~rl~
- (\\';L'C •• ;,{\ f[c'J\

J. What are the 4 st~ges of mission rrltlcal facllltv design?

- Ri4<- f\""\~ ~ ..
- ,(robi"",- \)i.\' I" \.'\)1'.
_ .:,,! ""~\'tl" \)<I..,;t\ ~1Y\~\-
r. . . ~\t IV\V\.\<I-\\~"
 Risk. RMllability. .. Class Ranklngl

Module 3; Risk, Reliability. & Redundancy

Module Goals 3.1

A first step in data center design is determining a target reliability
dass; ofter completing this module you should be able to :
• Identify tt1e components of a reliability class ranking determination
Determine a target reliability class ~iven customer reQuirements
• Use a cost benefit analysis to communicate the benefits of a higher
Initial investment to adient

'0'
. . . , p,..~PI""~oc, Not Copy
 Reliability & tile Risk of Downllm& 3.2
,. target reliab ility class results from balancing the f uture cost of
downtime vs. the present cost of prevention
• Reduced downtime ~ increased redundancy
• Greater redu ndancy = increased cost

Data Centers and Target Reliability ClasBes 3.3

Data centers are typically classified according to iI 4-01 .... ranking

.,..-: i..1J~ \? system

\""'e~~
,h \7-J .... • OilSS F1 is the lowest In terms of reliability (avalla~ lity - 99%).
and Class F4 is the hl9hest (availability g 99.999%)
"'f I-i""~· ( • As dasses correspond {in part) to fulu", availability forecasts, their
I /"'n ;, "'>" classlfk:!tkm tends to be both art and science
-h-V cft... .... ,1+IY?i ~d.Jj~' E~en SOl it i$ Oil usefuJ met1ic lor balandng the cost of downtime
with the cost of prevention

Components of a Reliability Clan Ranking 3A

Setting a target reliability level for the data center requires:

• Determination 01 operallonal requirements
• Determination 01 availability requirements
• Delermi" atlcn of Impact of downtime

"~"
· .......... 1 .
...
' .!"":':::;.,.''"
--
~~~

" . .. -. , ,.
' _ft4iInd~
I r ' ,
I _"

O2IItO aics,"
Student Guide R.... Rollobilly. & C..... Ranklngs

Determining Operational Raqui",menta 3.5

The operational requirements account for the time the facility's
components must be operating. plus time allowable for maintenance:

Ann ....1 Allowable

Deoc:r1.,- D_allon.1 LAlv.1
...'nten.nce Hours
Where fUflCborlS are operational ~$

.....
!hall 24 ,",ou'! ~ d.y, hiss tha~ 7 dt"'~"

Wher.e fum.tions ar.e Clpercrtional up 10

>400 Level 1

24 hours. iii day, 7 days..a wiNk. up to 50

>200 ~el ~
w~k$ ~ ~r- and l1I.\11tntenOlnce time Is
av;a.ilallie durin~ wort: in~ hClUl'S
WMt'C' Nnaloni "r. olM!r~~.Il4
b(lllJr,s a ~V, 7 dalf' fI \lIftk, for SO
we!!ki Of" more-and no t ime Gan b e >0 I.1!vel 3
made available -durins wort:ing hours tOf"
mai~nance.

1NtIe1'1' JundiOtl:ii Ir!!" opt(ttloNf.4

oo,,~ l:I .cW, 1 d.iIys II we~ 52 Y/ft:Ic$ 0 Level 4
each year

DC t til: I»tIIl ce .... .,..gt'l &. Bt$t Pndk'*1!

Student Gu5de lIi.k. R"Uabmty, & Class Ranklngs

Detennlnlng Operational Requirements 3.6

Ca lculaH n9 the~me In hours that the data center systems ca n be
brought offline for ma I"tena nee allow$ one to detefTTline an operational
level
• For ""ample, Jf a facility could allow for 120 hours of maintenance
per year that would put them In Operationa l Lf!'Ie/ 3

Annual Allowable
Maintenance Operational
DeIIcrlpilon
HOUfS level

Where functions are opermional I... th,,"

>400 Levell
24 hours a day. Ies.; than 7 days a week
Where . ,,~are v>"',, up_to 24
hou rs a day, 7 day. a week. up to 50
weeks a year- and maintenance time Is > 200 Level 2
available hours
Where tunctions are operational 24 "vu,~
a day. 7 days a week, for 5<1 weeks or
more-a nd no time ca n be made .vailable >0 Level 3
during working hours for maintenance

Where f" are operational 24 hours .

a day. 7 days a week, 52 weeks each 0 Level 4
year

H
I leSi P'rOpti""'Y~ Na1 c ....Y'
Student Guide RJ~. Reliability, a CIUG RAnking,

Determining Avanabllfty Re'l"lrements 3.7

Availability Is the total uptime a fadllty must support. It can be
expressed as an A"ailabillty Ranking, lying at the Interse<:tlon of an
intended maximum annual downtlme and the Ol)erational level,

AIIorwllble UBiIlfmurn Ann.... Twgotad AvaJaWRy

Dow"' .... III'n ....) "'-90
500 - 5000 9910 99,9
50 - 500 ",,-9 10 99,00
5 - 50 99.99 10 00,_
0_5- 5.0 00,99910 00,9009

Opel'Btiol1ii1
Lo ..1
__
Alow.bie Mauntwn An"ual OtJlMltme ( .. in ....)

50-SUO 5-50 0,5 - 5.0

le,..' 1 1 2
"3
L~'tt!I :2 1 2 2
Le'fEI1 3 2 2 3 4
L.:i'fi!l14 3 3 4 4

D8lllrml nlng a Sampl" Availability Ranking 3.8

We saw .... rlier tltat a fadlity that could allow fot 120 hours of
maintenance was in Operational Level 3
• If the target availability for this facility was to have no more tha n a
hours (480 minutes) of unscheduled downtime a year, then the
target avalhlM/lty rtlnkfng of the faci lity would be Level Z
AII ....b.. Mu'mum - . . , D _.... (lI'n_)
OperdoQeI
Lo..,
!!OO ....,..., B - 5DD .-50 0,5 - 5J)
LeYoEll1 1 'II 2 3
L'I'<r 2 1 2/\ 2 3
~ 3 4
~1 4 3 3 • 4
stoliont Guido R1.k. RoII.blllly. & CI... Ronking.

Determining the Impact of Downtime 3.9

Not all downtime has the ~me Impact. There are fou, downtime
1m pact dasslHcations :

Description Classlflcatl""
lmp,(t of downtime Is toca l ln scope r and iIIf'Yects on ly a sft'lgle site or
QperBticn, or results in a minor disruption or delay In 8chle\l1ng keo; La ...1
orqanizatiooill Qbjective.s
0'
Impact downtime i$ regional In s cope, affeaing a portion of tt,le
enteo'pn •• (although not in Its entirety) or ,esvlling In a mod...,.te Regional
dlsruptjon Of'" delay in achieving key crganlutfonal Objed:tve5

Iml'act of downtime is m ulti~reg lonal In scope, arfecttng a majOr

pO<t:lon or th. ent.rpM •• (although not in its ent i ~ty) or ....s ulting In
Multl-~Io""t
p mC!ljor dlsruptfon or delay in iKhieving key organtzatronal
objectives
Impact ¢ downtime I. global In !i<Dpe and affects the quality of
service delivery across the enttre enterprise, Qr results fn 21 Enterpri&e -
siQnUlcant disruption or delay'n achieving ke y ~Bnization a l W'ode
objectives

Pulling It all Toget .... , 3.10

With the operational requirement, availability r~nklng , and downtime
Impact assessment in hand, a rellabllltv class can be assigned:

I_<tof Awilltblfty A..IIabMlly .o.voIJoblU." A,",,"""'ity

DownI_
R_1 RBnk2 Ronk3 Rank.
E<nerprio!' W"," Cl8s.s;;Fl CI... F3 CISS5F4 e lll56 1="4
t.lJ 1 ' I ~Qion. 1 0...F2 a...F3 Clason CI... F4
RogoanaI C/a5s F1 a ... F2 ClaosF3 Clossf3
L.ocal aa.&F1 CI.>ss F1 Oooof2 C I ~F3
Sample Reliability Clas8 Ranking C_lcu'-IJon 3.11
We saw eartler that a faci lity that could schedu le up to 120 hours for
maintenance a yesr while allowing for up to 8 hours of unscheduled
downtime a year would have an availability rank of Z
• If we detennine th"t t he.impact of downtime for such a facility Is
multi-regional in scope. affecting a major portJon of business
operations, but not stopping business opera~ons entirEly. then this
Ir\dicates ar\ apPr\Opr1ate target reliability c:lass ",nklng for the
fac:lllry of Class F3

Avall.bUIly ......""'U.y ..... ,""'111y ..... ilabOilty

ufD_me Rank t R.nI< 2 RoM 3 RonI< 4

Enterprise Wk:le ct•• 1i F3 CI ... Ciao. F4 Ciao. F4

Multi-Rag., .....
,." .~. ( CIM_ F3) CI... F3 CIaO$ F4

~on.1 Clasa F1 Cia.;. F2 CIa . . F3 Cia. F3

1.oc.1 CI_Fl Class f1 ctU:I F2 CI_F3

Classes & Facility Profiles 3.U

Each dass is assodated with a specific mission -critical facility pnofi:\e

defined by:
• Component redundancy
• System redundar\cy
• Quality of components
_.ntGuldo RIok, RwlI.bIKly. & Cia.. Ranking.

The Cla88 F1 Data Center ~I<." ~ ~ 3.13

A dass Fl data center Is a basic data center where the impact of

downtime is deemed not to be overly critical
• Down~me Im~act = minimal
• Component redundancy requ ired- none
• System redundancy required; nooe
• Quality control = standard
• Survlv:' billty = none
• Annual scheduled malnt""ilnCe hours = 200 Or more
• Targeted ava ilabil ity; 99.0%
• 1m pact of downtime - local or Regional

The Class F2 Data Center \ ,..:.,,,,,,,\ " •.-0"'- \...;,...~") 3.'4

A Class F2 facility has higher ,ssoclated downtime costs. It guards
against failure of components most likely to fail (such a. UPS, controls,
generators, etc.)
• Component redundancy; for critical components only
• Sy.tem red unda ney = none
• Quality control ; premium for ctitlcal components only
• Survivability ~ moderate hardening for sea;rity and strueto",'
Integrity b\. ,. ..... ' \
• Ann uill scheduled maintenance hours = 50 - 200
• Targeted avail.bility = 99.9%
• Impact of DownUme = local, regional, or multi-regional
 SIudontGuido

Class F3 ~~". f !,>,.\.. (<.,~.,.') 3.15

Class F3 facilities must provide for reliable, continuous power when

major components are OLlt of service
• Component «<Iunclaney ; redundancy is provided for all
components not part of a redundant system
- System Redundancy ~ system redundancy may be provkled
w~otll. component redundancy I .f...: \ ,.,.,. .. \
• Quality Control ;premium quality for all oompone<1ts
- Survivability ~ s~niflcant hardening for security and structural
Integrity (<'"'1'1,'-"')
• Annua l maintenance hpur< ~ 5 - 50
• Targeted Availability; 99.99%
• Scope of Imp&et ~ local ti> Enterprise-wide

Class F4 3.16

aaS. F4 facilities provide the highest levels of redundancy

• Component redundancy ~ redundancy Is provided fer all
components
• System redund'mcv ~ redunda ncy is provided for all systems
• Quality control - premium quality for all romponents
• Survlvablltr.y ~ aU building systems are self-supporting and , )
protected against tile highest levels of natural' forces (~".,'- .. ?I'''''''
• Annual planned maintenance hours ~ 0
- Targeted availability; 99,999%
• Multi-regional or Enterprise wide

'i201o Illcar'
_".Guldo Rllk. RlU.biltty... CIas& Rankings

Class Summary 3.17

CI ..... Fl Class F2 Class F3 class F4

Cgmponent None For Critical For all For all
~.dundancy 'Components ·compo·n,mts IXlmponents
not part or.a
redundant
System
System No No Where For ali
Redundancy ~on'lponent systems
red undancy Is
lacking
$unrivll bilily ~.on" Moderate· Slgnlflcilnt H ighestl~1
Hardeoil'l9 ·hardening of protectiOn
Quality Stand.rd Premium For Premium Premfu m
Control critical
components
Targeted ··99% 99;9'11, 99.99% 99.999%
Availabilig:

B1C811 flroprt • ....,.

.....
"""1It'ty--Oo ~ CQPY
8tudAnt Gukle

Seri.1 va. Parall8~Klnds Of Redundancy 3.18

It Is Impo.unt to understand that target reli~bility cless typically refer.;
to systems as a whole
• The difference between individual and system reliability can be
illustrated by considering the difference between two systems will1
equa lIy rei iable components run nlng serially VS. running in parallel

- 5Ysc-C-';--
.1\1
) R~SO%
~ystE1fl.n ~Y<"'"
Bl
A~ 50'> hlO,.
I
Syst~m 1\2
R~SO%

I
Paral"'l Sys .. m

Calculating Rella bilili" 3.19

• Serial systems' reliab ility Is calculated in terms of the produd of the

component systems
• Parallel systems' reli"bility Is Cillculated In terms of the complement
of the component systems

Serial System Reliability Ca leu/alion. 3.20

If lIle two 'serial component systems have rel iabilities ~.5. then the
total system reliability equals:
• R;.,. R;., ~ .5 * .5 - .25 ; .2S
I
System
Al
R= 50","",,-.1

I
I
Serial System
Stllllont Guide Rllk. R41loblitty. & Ci8s. Renklng"

Parallel System Reliability 3.21

If t wo parallel component systems have rel;abillUes ~ .50, the~ the

t otal sYstem reliab ility -
1 - [(1- .5) • (1 - .5)] ~ I. - (.5 '· .5) ; 1 - .25; .75 ~ .75%

Reliability of the Pal1ll vs. Reliability 01 the Whole 3.22

The foregoing calculations show that it isn't soleiV the reliabil ity of
individual comporients that is crucial t o el1Suring a reliab le S'/stem
• Rather, the reliability of a system depend. upon how its
compo!1ents are put together

CIa- & the W<lakeat link 3.23

People sometimes spe.1<. of the class rating of a particu lar sub-system

(such as a Class IV power/electrical system)
• The dass ratings of the Individual sub·systems of the data center
can difflor from the class ratings of the system as a whole
- The ove",11 reliability of a system as whole is determined by the
rel iability of its ·well kest l ink"- its least reliable component

The Lo_.tCommon Denominator 3.24

A d"ta center may have different class ratings for different portions of
tts In frastructure
• For example, a data center mav be rated Class F4 for UPS power,
but C"'ss Fi. for the mechanical system's power train
- He", tile eJectr~ I S'/stem class rating is equal to tile lowest
rating of its components, yield ing a aass f2 electrical rating

l .n
BlCSI Propri.tlry ProtMlrCY-Oo Moe CQ,y
Btua.ntGuldl

Reliability & Slngl" Pornls of FaUu", 3.25

BecauSE a system's reliability Is defined by its weakest link, the

presence of s;IIgle poillts of failure Is crucia I to class ratings
• Single points of failure should be eliminated to Improve reliability
• Eliminating single points of failure means In<:reasin.! ! system
rt!dundancy
• C,"55 F3 and Class F4 ratings require Ihe elimination of all single
points of failure

Risk and CO$tJBenafit Analysis 3.26

The last part of risk "n"lySiS involv,," an appropriat" weighting of rosts

and benefits, and communicating this ttl the dlent In terms of a target
class
• The following table shows one way of demons_ing the relatWe
upfront costs In terms of initial investment and e><ped;ed downtime
of a Oass Fl, f2, and F3 data center

Availability Total First Expected

Costs Downtime

99% 10,000,000 B3 Hours of

Downtime
99.9% 15,000,000 BMour

99.99% 20000 000 1 hour

5tudtnt Gukie Risk. RwlI.I>IIty. " Cia•• ~","ng.

Calcu latlng Annual CoSts of Downtime 3.27

TI>e prior tabde illustrates the difference between 83 hou~ of expected

downtime for a Cia,..; Fl fadllty vs. 8 hours of clowntime for a Class F2
• One way to lIIust",te ttlis to a client is to simply multiply these
hourn by the $47,000 cross-industry average mr the cost of
down~me to compute an annual cost of downtime

~Iillbility Total First Expected Estim .. _

Costs Downtime Anl'luul
Cost of
Downtime
aass FI 99% 10,000,000 83 Hours of $3.901,000
Downtime
Class F2 99.9% 15000000 8 hour $376000
Class F3 99 .99% 20000000 1 hour $47000

Showing Ihe Benefits of Highet' I"Hlallnvestmenis 3.28

The prior table demonstrates tnat a Class F2 faCility can save an

average cl ient over l.5 m lilion dollars annuiJ/ly Over a Class F1 facility
• These an n·ual savings can r>Ow be used to demonstl"llte·the value to
clients of a h igher initial expend iture re lative to long-term costs of
a lower class

Reliability Total First Expected Annual cost :I yea,

Costa Downtime of dowoilime savlnlls
over Class
F1
99% 10,000,000 83 Hours of 3,901,000 Nta
Downtime

99.9% 15,000,000 8 hour 376 ,000 10, 575.000

99.99% 20,000,000 1 hour 47,000 11,592,000

S~Guide Riot, RellobiU\)', & as •• RanlUng.

Review Questions

1. What are the 3 preliminary detenninations that must be made

befol'i! determining a target reliability class?

. () POJ" tv:,. I P't''' N/W·--JS

- Av/..}(to.'>.(,·';; Rt~""\'<.«G-.h
- r."'rd c1 <.L:..v',--" ~.\...£
2. What are the targeted availability ranges asso<:lated with each of
the 4 class ran kings?
S" c:::c _S uP,.) <,"r to "''<. "I

>" _$"0<> 't ". "r I" '" "<. 'l."

'S" .. ')0 " 'i. ~1" f' -"'i,« 'i <"I
c . ., _'5. 0 '( 'I ."l" "i ~'-' <"("(, "I 'i f '/

J. If their comllOnents were equally "'liab.... wouk! " ...ri,,1 or parallel

sy.tem be more reliable?
r"··'(( L (
SCudlnt Guide Locadon

Module 4: Location

Module Goals 4.1

This module examines factor'S relE!llant to choosin9 a stJ~able location

for a dat.'> center; after completing the module you should be able to:

• Advise clients regarding data center site suitability

• Identify best pr~ctices reganding power and telecom acce;; feeds
• DlstUss the recommendations regandlng app<opriate d<lta center
property adJ.""ncies
stud.nt Guido

Clreul! " utility F•• de 4.7011

The ta,.get reliabilitY' class will determine the type of utility feed
reoommended for a data center
• 5i "!lIe FMd from 51 ngle utility $11 .... Statlon - the least
desirable method for power delivery to a site, suitable only for
Class F1

'-- iL . ~~:;- __--II

LI_....... € '"<~- LI_'_~I_~_"'---,
• Redundant Fee""
fro ... Dillerse Prollld....-the preferred
method for power delivery to a Class F2 or F3 facility

• R.. dund.nt!Pive...... F...,ds from Diverse Utility Sclb-StatlDns

- preferred method for power delivery to a Site, 5LJita ble for Classes
F3 and F4

1__ I~O?i _oo_oJ ~&'1 o.•~ I

.-~.:.':""- ~ '2ut n tll tloro·2 ~

• Redundant Faeds from Dillerse Providers-the preferred

method for _ delivery to a Oass F4 facility

I
student Guide

4.9

All electrical service et1tri1nce feeds should be underground wiltl •

minimum separation of 4 ft (l'. 2m) from other utilities along the entire
route
• If ovemead utility lines to the site cllnnot be avoided, provKle
multiple power source paths
o Separate electrical servi<:e entrances snoJ,Jld have a minimum
separation of 55 It (20m) from One a/'lother

M lnlrnuO'1 G., Jt Cz.o 01)

_Guide Loclllon

Location and Telecom Acce&$ 4.10

The primary concem regarding location and telecom access is

redundancy in terms of both access reeds and service providers
• Data centers should be located in an area· w~h easily sustainable
connectivity prov ided by two or more access provider central offices
- Diverse service feeds Me recommended to have a minimum
separation distance of 20 m (66 ttl along the entire route

IWfIolmurn 06 f l / 20 rn )

location and Telecom Access Types 4.11

Telecom access feed recommendations are determ ined by target

reliability dass rankings

Class F1 acee •• recommendations:

• One service feed from one access provider cenlral office
Class F1 Access Provider Layout

Long Dj,tance ...,. . Access Providers A •

Central Office • . Central Office
..
r
Data Cente,

...
ateSI fI....". .....,. P....Cy-OO NcIt COpy
cr_ F2 Acceu R&comrnendatlons 4.12

Oass F2-Mooerate Redundancy:

• Diverse servke reeds r,om one centra I office
• ead! service teed using separate and diverse paths

I Long Distance ~:_=:::::~ Centlal Office ~::==:>~ Data center

Redundant Telecom Access 4.13

Red unda nt Oass F3 recommendations:

• At least one service feed from two separate and d istirICI: ao:ess
provider central offices
o Ead1 access provider central offICe conn..cted to ..me long-distance
carrier
• Each service reed to the site using separate a nd diverse paths

Centra! Offlc" ~ 1

Data Cenr~r
long Dirtance

C~tral Office'2
 ~d."Gc.oldo

The site should be at least 1 mile (1.6 km) away from :

• Gas stations
• Auto body or paint shops
• Self-storage fa~ifities "" ... ,.;,_:>.,.,: '.J~
- High voltage power d!stribatillri lines
• Public utility SoJ bstatlons

• Water storage towe",

• Commercia I nlilways
• Highways and other major traffic arteries
• Coasta I or Inla nd waterways
The data center shouk! be at least 100 yards (91 m) away from :
• 100-year flood ha2ard areas

NMghbors You Do Want 4.23

The data center should be within at least 5 miles (8 km) of:

... A police 5wtion
• Afire department station

The data center should be at wttl1ln at least 10 miles (16 km) of;
• A hospftal ,,'£ ~
\) ... tt~\·, h-\.. ;:'" \ L IN'~
• A major metropolitan area <'" , 1"' Lt..<- u\... /.,....,..,
" Q ~ .

C0l1111 aal'
_ .. Ould. LoeoIIon

Wind 4,26

An area wiltl less than 2% annual probability of Wind over 80 mph Is

preferred
• If the site Is in an area with 8 higher probability than this, specific
detail in the "hardening" of ltIe building must be incorporated Into
the design

M.p tiho'J¥t". fo,n.do RI.k Ar._

In TM C'ontl!rminoue Unni!'d .Statea

-_... ...

,..., Showlna HIII"I1eaI'l~.ktlyit~

In Tile. ~irlD1D u ... ~ S1MH
"

,- '.,.
Sludont Guido Lotallon

Air QualHv 4 .27

In arws witll exisMg air qua lity problems, regulations re!1ardi09

nit,ogen dioxide (NO,) emissions (produced by backup and eme'!}ency
generators) need to be consulted
• Er.osure that generator nun-time permit requests are issued in a
!Imely manner to federal and state environmental authorities
• In most cases, annual operation hours will be restricted and
compli~nce must be vertfied
 Stu.,.,. Guido L.o<:alion

Review Questions
1. What type of power utllity feed Js preferred for a Class' F4 data eenter7

R~'<~.f Ft..)", +',<.~ D;.·t/"s, Pre,.,'.! t ·· '>

2. What type of telecom "ccess feed is preferred for a Class F2 facility?

p oJ..'~''''' ( P.:..t:(<j {no",," I'I'v".--$,- fro, . .tv;.

3. How f~r away should a data center be from the nearest Jake~

<] ,.....:( ( '7

021)11) BleSi"
•. t6
afest ~pritlwfY ,,~~ !'tot i;QPJ
 Buildings

Module 5: Building Considerations

Module Goal. 5.1

In tl1ls module we wllJ look at what makes a building a good candidate
for housing a data center; after comjJ!eting this module, you should be
able to :
• Identify the tvpical support spaces of a da\ .. center
• Ie/entity the "",h;tectural parameters for a data center
• Give recommendations regarding a bUilding's suitability for a data
center Installation
 Bulldlngo

Structural In9reGlenl& of Data Cent..... 5.2

A data center Is typically composed of iI computer room for housing

mission cr1tleal data storage and application hardware, as well as a
collection of specific support spaces that typically include,
• Telecom/Entrance Room
• Electrical Room
• Storage/Receiving Room
• Control Room
• SUlff Su pport Space
• Entry POint/lobby
• Battery Reom
• Print Room
• Mechanical/Chiller Room

Building SIIeII Considerations 5.3

Data Centers should be i nstalled in a steel Or concret~ framed building

• Building e~!eriors should be made of noo·flammable durable
material
• Building sections shall allow for a minimum r.ais~ floor to ceiling
height of 10 It (3.1 m)
• The slab to structure above should be a minimum of 14.5 It (4.5 m)

19 ft (3.. t "I) !Hod Ibor

Ie CI!iriG 145 ~(4.Sm) slab \0
floor above

:1 I
...
IICIf ,",p"'" p,,,,,,Jtro-Oo .. "" COPJ
student Quk:le llulldlnll"

Data Centet Floor Lev....

The ground noor is often the most advant.l9eou~ for data center
placement due to equipment access and Hoot loading considerations
• Frequent changes In IT and Oata Center equipment are common
• The average IT sel'ller has a life span of approximately 3 years
• The average data (:(!nter will eIlange out its entire equipment
inventory over the cou~ of 3·5 years

Data Canters· Equipment Aceen 5.5

The data center should be d",lgned to ensure ease of eq uipment

del ivery and removal
• The delivery paltlway sha ll allow for equipment 35 la .... e as 10 It (3
m) long, 39 in (t m) deep, 6. 75 It (2. 1 m) wide, and weighing as
much as 7500 pounds (3400 kg)

Upper Floor Data Cen...... 5.6

Although less convenient for equipment acress, upper floor data center
placement can provide added security against unauthorized Intrusion
and wMer damage
, In areas subject to major lateral force issues (.uell ..s hurricane,
Wind, earthquakes, etc.) M upper floor will contribute to structural
illliecw:itv • , .I j
1 ... ) \ l liJ ;' ' ; 6'
Dete Canter Neighbors 5.7

If a data center is in .. multi-tenant bUilding, ,"""ices to the data

center should be separate f,om ollle, tenants
• All wate, lines, sprinkler lines, duc\Werle, gas lines, etc. serving
areas outside of the dat a C<!nter .hall nO! pass ltlough the data
Genter area
• ~o haUlrdQu$ systems shall be located in or a,ound the data
~nter
 Bulldln9>

Walls 5.8
The data center'S perimeter walls s~all be full height (slab to deck
above), l·~our rated construction, fire sealed, and sealed ro prevent
chemical fire suppnesslon leaks
• Interior walls sholl be constructed of a minimum 22 gauge (0.65
mm) metal studs lor walls up ro 11 feet (3.5 meters), 8 gauge (1
mm) lor walls exceeding 11 feet (3.5 meters)
• Studs 5h,,11 be " minimum 01 5.5 In (140 mm) depth to
oo:ommodate boxes and piping requited to be Installed In the wall
• Walls shall be sheathed 10 ftre rated wall board such as .6 in<h (16
mm) type ")to 9)IPSum board
• Non·,ated walls should be braced no more than every ten feet (3
meters)
• Gypsum boa rd for non-rated wall Is to be .6 inch (16 mm) type "X"

Fire Rating of Data c.n .... ' Spaces 5.9

• Perimete" I-hour rating, full rn.;ght, slab to slab

• Computer Room: No rctting 4 (1001"" to c.elling I between
equip"",nt I-hour to non-computer mom
spaces
• Control Room, No ",tlng between computer room;
1-hou, rating to non-computer room spaces
• Printer Room: No rating between computet and control
rooms; l·hour rating to other support spaces
• Media Stora~e : 2-hour, lull height
• Electrica I Room: 1-hour, full "ei9ht
• Enlrilnce Room : No ratlng between computer room;
I -hour to other support spaces
• Battery Room: I-hour, lull height
• Receiving Room: No rating hetwe.en ~ompute, and control
room; t-hour between other support spaces
 lluldlngs

Wan finishing $,10

computer room and related walls should be finished with a non-

p"rticulating smooth water-based epoxy paint
• Prior to painting, gypsum board is to be sea led with primer
• All penetrations In the perimeter wa lis a re to be completely sealed

5.11
The data center should be motsture/v" POT sealed around the
perimeter, floors, and ceiling.
• If data center Is a 'ground up' building, slab and all below grade
components should be continuously sealed with a rubberi.ed
mojsture barrier
- All exterior opening/penetrations are to be sealed prior to wori<
on intenor walls In the data center

Floora 5.12

A data cent..,- Hoor shal l be a minimum slab of 5,5 inches (14 em) thick
• The Hoor slab shall be designed for" minimum of 150 Ibs/It' (7.2
kPA)
- for data centers with hiqh-denslty racks, floor slab is to be a
minimum of 8 inches (20 em)
- Centers with high-density racks shall have" slab designed for
2.50 IbS/ft' (12 kPA)

...
BlCS. PftI~J)' Jtrro,.lty-C~ HQt c~
 Studlnt Guldo Bulldln!!"

Access FIooI"8 5.13·14

A raised access floor system should be considered for all electro~lc
processing eq uipmen! ~re .s, telecom areas, control rooms, and other
spaces that requ ire precise t emperature and humlditv control
• Ra ised floors require higher init ial investment,;, but often alloW tor
more economical long tenm cooling solutions
• The access floor shall be a minimum of 18 in (.46 m) above .s1ab
• For da ta centers with high density racks, t he acce-ss noor shall be
24 in (.6 m) above the slab

Access Floo", 5.15

The raised floor system shall be a eontinuous bolte~ smnger system to

tra nsfer latera I load
• Raised floor system shall allow for static t ransfer to the grounding
system

atl10SICSi"
54
I IClI Prvpr'iIo'*"Y Pi~HDI. Cop,.
 Bulldl"ll'

Ceiling_ 5.16

Ceilings shall be composed of a 2 It (600 mm) x 4 II: (1.2 m) lay-in

grid with a clean room vinyl faced gypsum lay-in tile Il:l prevent
contamination t-~,~ ....... 1"01(,.' . ....... t.t '-.
• The lay-in tiles shall be smoolh, non-particulating, faced t He, with
sufficient weig ht to prevent the IIrt-under pressure of a gas fire
suppresSion agent
• The T-bar shall have neoprene seal
• All penetrations In the ceiling shall be sealed
• Ceiling systems shall be suspended with an independent 4-wav
win n9 system

Doors 5.17

The primary entrance door to the data center shall minimally be it P<lir
of 3 ft (0.9 m) wide by 7 It (2.1 m) hig h doors
• The primary entrance doors sh""ld have no center post a nd no
doorsills
• Data ~nter doors shall be solid core, minimum 1 3/4 in (4.5 em)
th ick, either wood or steel, mounted in steel frames
• The rest of the data center doors should be a minimum of 3.67 It
(1.1 m) wide by 7 it (2 .1 m) high for it single door, or 6 It (1.8 m)
wide by 7 ft .(2. 1 m) high for it pair

Data Cenier Floor Planning S.18

The in itia I layout of a data center site, the computer room + support
places, shou ld ideally allow for future computer room expan;lon.
Avoid placing the computer room next to :
• Outside wa lis
• Elevator shal\s
• Other Int~maHlxed buildin9 wa II!;
8Ndent Guide elllidings

The Computer Room va. Support Spaces 5.19

Data centers should be designed with a "li9hts out" approach that
attempts to minimize needed access to the computer room
• ll1~design and layout of personnel and support. sp~ce sIlould be
such that minimal acress to the computer is needed . \~",,< <1111\ ,
.- - - - - - - - I
rPTI"~1bo1"'l
r.. ~""" !
fbtlfOry E~' rl l,,1 UI'trJl~ i I ~lnrN~1" AI~ .. l\rIeIWrJIh I I .1I00'''l''I'',

""""' RoI~n
R""m
I
I
- - - - - - - -I ...
" " , _'
~,

rt:.i'{~ I~;u1"· IC. 'O"1

(~ 'r1lm1

,- - - - - - - - .- - -
I
1\>,,"

I HI'ltl lllH!..rt~ • ~o(>t u(!r 1iI~'" H:)vo'1

I T.d,,1"
M<dI,
I - - - - - - - - - - J
Stor.lgt

ft¥<~rvi"') J !oI!(Vn! ~on!~r

Stt.l ta 91

~d ...ltlt. 1 I 'elecorn
'5oi*-l, Uri-! thl.dIAg
""', (M1t'1' R.oom RiSOItI I If ~:IOfIone l Sp.ll~
W"'" I'
 Student O4.Ikte Bolldl"1I"

M~in Entry Point 5.20

The entry to the da~ center should be pOsitioned awe y from the entry
point of lt1e building
• Entry to t he da~ center from non- data center spaces should lead
into a controlled space, prior to providing access til the equipment
areaS
• Entry tor equipment should be controlled by the data cent@r
p~nelonl y

• Equipment entry sIlould be located near a staging/storage area for

preparation of equipment prior to entry Into comput@r room
d.>J."' "'" r""-i<:,. i ~ ,( '" L, ('., f C/
Control Room 5.21
A control room monllOrs the function ing of the computer room
electronically as well as typically being troe center of building
monitoring and automaMn systems
• It should be near the main entrance
• A console is rernmnlen<ted to house all mon itors
• It should provide direct access to the computer room

Telecom Entrance Room 5.22-23

A data center should pos5eSS a telecom entra nee room - either

adjacent to or within" secu red section of the compute r room - to
house access provider demCtrcatign l tennina~on , ~nd provisioning
equipment
• The design should accommodate a minimum of two cabinets for
each canier
• A minimum of OIl<' owner-supplied patching rack should be supplied
for every 10 carrier racks
The ent",nce room shoukl be posit ioned with orcuit distance
limitations In mind and should be ildjacert to the computer room
• The entrance room should allow for sepa ration between customer-
owned and access-<WIned provider equipment
• Customer-owned equipment should be secured against access-
prcvider technici. ns

&611) BICst'
S1udonl Ouill.

Electrielll Room 5.24

A data center should have a dedicated electrical equipment room for

housing switchgear, the uninterruptlbUl power systems, distributiofl
panels, and sometimes generators and power distribution units
• Two eKits are required for the electlical room
• If there are two electrical service entronces I pathways into tile
data center, then there should be two dedicated eleetrical
rooms/spaces

Generators & Fuel Storage 5.25

Indoor backup generarors are' recommended

" Indoor generators provide ease of access and a re more secure
• Indoor generators "Iso require spedal ventilation
- Regardless of whetlJer indoor or outdoor, ol'\S~e generater fuel
storage space must be planned
- Ttoe amOll nt of space set aside for fuel storage should be
erlough for between 2 and 7 days worth of fuel

Battery Room 5.28

A battery room should be adjacent to the electrical room. The size will
depend on the type and number of batteries required
• The batte ry reom may also need to Include the Installation of
disconnect switches
• The battery room will require two exits
• Battery rooms sha ll include halardous containment, either a
contain ment system or hazardous mats
Check with AHJ regartling local cocle and dedicated exh8ust systems

BICSI
....
""""1' fI'r<IP4'I1Y-Oo'" Q)py
Student Gulet. Bulldl",,"

MechanicaliChiller Rootn 5.27

Space needs to be provided outside of the computer mom for cooling
system equipment such as pumps, fuel tanks, ch iller tanks, and
associated controls
• Tl1e data center deSigner ~s to coordinate with the mechanical
systems .d eslgner to determine the equipment spadng needs

Media Storage Room 5.28

Data centers that produce In·house storage med ia (such as backup

storage tapes) shall provide a separ1!Jte room for media storage prior
to its transfer to a permanent stol'llge facility
• The media storage room should have a 2-hour fire rollting

Print Room 5.29

Printers should be located In a dedicated print room separate from the

computer room

• The print mom should have its own dedicated air handling sy<tem
• A separate puper storage area near the print mom should be
provided ]4.'''f Fr ... .t-.-, -\;- ,.1i" .... /.• r.,.''"'' /<> ... ,~;M ,-~-{
Fire Control Room (,·~ ~~,... .. ,..,-h"" . ~4 LIZ . 5.30

For the highest tier data centers, a separate room should be provided
for the pre-action control valve system

• Space should be providecl for the placement of chemical lire

su ppression tanks
• Tanks need to be Iocsted to assist easy serviceability
• Tanks should not be lo~ated In the ceiling area above equipment
• Tl1e preaetlon sprinkler system is simi lar to the wet sprinkler
system eXCEpt the piping In mliral areas does not contain water
untH there is a fire
• Tl1erefore, this system provlcles protection against waler damage to
the data processing equipment due to an accidental di.chatge

"11
.lCSl ~""t8IV""rft-OO Hoi Con
lkudontGulclo Buildings

Receilling and SIont\l8 Room 5.31

Aniving equipment should be stored, uncrated and prepared In a I"Oom

away from the computer room, with filtnnion on the ne!IJm air le3vlng
the room
• The storage room should be adjacent to the equipme<>t entrance of
the dIlta ce<lter
• The storage room can he a component of the s~lng room , or a
separate reom near ltIe staging area
o The staging area should be monitored by CCTV
• A sep<lrate secured storage room shOUld be considered for high-
value equ ipment and for vendors' equipment

Sacum loading Dock 5.32

A secure loading dock should be provided for the delivery of high value
equipment
• Tne secure loading dod< should protect equ ipment from .evere
weather
• The secure loading dock should be monitored by CCTV

"11
tRCII ,.,...~~-OO NotCopt
Review Questions

1. Why is the ground n()()r considered advantageous for a data center?

k ~c...; ;I'"'t~{ A<. (.,;' j.» -€-

r- (w' .
(c ,-,A k.-/f, '
....
./

2. What 3 things shOllld one avoid placing the computer room ne~t to?

V '" ~<,i .f ( "'-~ (r.,

f 1,.,,,1<, s"~f{?
L'
u . J ....J
fl'-t-r I,...r . . ~-
(;I',J 1:;4,;ld·'~1
J
fA. '1/ I ')

3. What minimum number of days worth of generator fuel should be on

hand?
Studonl Guido Tho Comp,*' Room

Module 6: The Computer Room

Module Goal, 6.1

In this module we will examine the crucial elements of ~ data center
computer room; afte!" completing this module you should be able to :

• Identlfv the components of st~ndanl computer room cabling

topologies
• Explain the hot/cold aisle oonrept of computer room eQuipment
layout
• Describe the basic .paCing recommendations regarding standanl
computer room layout

~IIIIIJICSI"
_ntGuldo

Computer Room Setup & Design 6.2-3

Once the locatio", building, and lloor layout are In place, one can tum
to the desi9n ·c onsiderallons for the computer room, the heart 01 the
data center

The computer room serves as a secure enviromnent for the IT

equipment. 1n addition to computer hardware, it will often contain:
• Switch In9 a nd patching racks (the Main Distribution Are. I>
Horizontal Distributions Areas)
• Server and equipment racks/cabinets (lf1e Equipment Distribution
Area)
• Digital storage I SAN (Storage Area Network) frames
• Special ceoling equipment (Computer Room Alr Conditioning units)
• Cabling pathways
• Fire prnrection equlpment
It will sometimes c:ontaln:
,
• Power distribution units (POU) .- "",
• Remote power panels (RPP)

Physical Network Topologies

The"" are two basic possibilities for a physica' IT networ1<-a ring or a

star

, ,
J
-.
....1

- -
I I
Or
~

-..
oj ',

. -.-... .-.
dO
. ~'

-
...I~.
...
r .

--- -
~
i "

i i ...;;;;..
.....
I
~ '-
'. I
.-I
............
M

Ring Su. StAr

h , IICSt'
."
atcSl P~tIt1t'~"'" Copy
studlnt Goldo The CGmpt.lter Room

Advantages of Star Topologies

Star topologies are more likely to limit failures in the physical

infrastructure of a network to particular sy.terns
• Because of this, star topologies are star>dard for data cetlter
networ1cs

Star TOplllogie& & Data Center cabling Di&tributlon 6.6

Tl1e star topology of the data center indude> a Main Distri bution Area
(MDA) as a central "backbone" hub, and horizontal distribCltion o..,os
(HDAs) as the end points 01 the networ1c 'bad<bone'

;;s;
...,
,,00.....
.' ... ,. ~
d='." •""""'" .""""" m
~

•.
._ _ J _ ",. , .

IiIiiil;U:;'
•
. U \ •I

" ',.1 .

,.
 Student Guide The Compu,*r ROOf't'I

Data c.nter Design vs. Standard Facility Design 6.7~

Many wmputer room spaces are comparable "to standard o;>mmercial

building spaces :
• The ma in distMbutioo area (MDA) In a data center Is the equivalent
of a commercial building equipment room (ER)
• The horizont.11 distribution area (HDA) in a dilt.1 ceoter is the
e"uivillent of a ~ommercla l building TR or TE
• The zone distribution area (ZOA) In a dat.1 center Is the equ ivalent
of a commercial building consolidation point (CP)
• The equ ipment distribution area (EDA) in a datil center Is the
equivalent of a commercia l buildin9 work area (WA)

Standard Building Topology

. ,,
~G'AJ! TEHJ- TR
~ f.tWA1-' [kp ---
\ "-
~" j~ I II
,
fWA ~
BO . ~ n.

I TR_I _ c. ,

BockOOM
EF co, .. ER

CI2O'I'OBICSI"
...
lIeSi ~y~p~N~Copy
Siudoni Ouido The ecmputer Room

Data Center Topology

Actee .. IiIAyf_,.. - ~nl ... ftCA ~fII -~. pn:tvlcl ....

C Cn"';~ .Q ~u'"rne " l
~ Ikt'Ilerc .lj~)
O'rk1<N~
.....,......... C'IIII"'-.
_ppvrtr. __

._----------- -----.--_ ....... . .............. . ~,

,,
I ,,

-,
.... 'U:vf'!t.1
1IIJn:r1~"Io.,. ......
(1.At4( 6#t.N / KV1"
S"'~'~l
....... I_n._Ic.."'"
ltqWlllftHlnt
......... tlen .. _
(,'acI(/CIlItl lnet)

---------------

.10.CSi"
Stud~nt Guide T.... CGmpulor Room

Main Dlslrl_on Anla (MDA) 8.9

• The rnaiO distribution area Is the primary interface between the data
re"terand the outside voice and data feed • • It supports the main
cress-connect for the data center 8 nd houses core routers, LAN
switches. and SAN switdJes, PBX., and IP-PBXs.

T_
I
el'1tlilnC;:1!I
Cclrnputer Room
H~2 0 f'\UMI
Olwlbvtlon
"""'" Main Distribution
A...
Ar ••

=m
SuikllKlg
f-"
StQl"0911
Ar.iI
NdwOII!
t Platti!)twIIl Di1illjbu(Joo A,eOl
Offic!!'
Support
Space

. ."'" ... •

ElJCSf
C2Gl1

InCsa ,~
.... DC '10;

Property Go Net Copy

~ c.tt.t D......... Bell Pnd.IcM
student G ... kle The COmputer Room

HDA 6.10
Horizontal distribution areas support horizo nta I ca bllng to the server
cabinets, the LAN, SAN, consoles, and KVM switches

6.11

The equipment distribution arell is typically made up of parallel rows of

adjacent server racks

PaTe" Co-t"d
into Server

Pe1chCord
into Swik.h

IlBA J~DA

..,
SIC&f ',¥PI""",,yPJof*il-Ol) Hoi tIPPY
Student GWde lh. Camp0t6r Room

Zone DIstribution Area (ZDA) 6.12

The zone distribution area Is a consolidation point In the horizontal

cabling system. Zone cabling allows quicker and easier manges to the
equipment and cabin¢s

.....

Equipment Layout & Hot/Ci)ld Aisles 6.13

The general layout of the equipment in the computer room should be
done with equipment cooling requ irements foremost In mind

• HOf/cold alsFe Is " preferred melf10d of laying out equipment in the

data center to maximize cooling effiCiency
- Equip ment a nd server racks are arranged in parallel rows
- The cold aisle has the fronts of two rows of racks (adng one
another
- The hot aisle has the backs of two rowS of radks facing one
another \,. ,-'X
. .--
~----------------------~?~~~-----.,------------~
u,;.1f¥d!
~
".~
..
.....
'-
:""1
;-

-
-;
'
·-1
,-,
- ,.
I.--.--~
 The Computw Room

ComlM'l8r Room Thermodynamic. 6.14

Air Is brought into the cool aisles from undem .... th and exhausted from
the hot aisles overhead
• This produces constant air circulalion through the rack5, pro~lded
there are no "holes' through which cool "ir can leak

. .
...F"'-·-.- . - - - -......

Server Racks & Rows 6.15

• Computer room servers are typICally storeel in racks/cabinets that are
up to 24 inches (60 -cm) wide and 48 inches eleep. To conserve space
and direct airflow, racks/cabinets should be placed side by side In
rows.

• Row length sooul<l not exceed 20 racks, 0< 50 It (1 S m)

• Rows that are closed off at one end should not exceed 10 racks

• Local jurisdiction. may require shorter row lengths to meet

egress requirements
 StudO" GoIdo Tho> Comp ...... Room

Cold Aisle Spacing 6.18

The cold aisle should be 4 It (1.2 m) wide at a m inimum

• This allows for two rows of 21t (.6 m) ti les throu9h the cold aisle
that can be perforated for a ir now manogement
- 6 It (1 .8 m) Qf spacing is pr~enred , as th is allows a center aisle
for traffic in addition to two rows ·for air f low management

,-
.....

·
·
.
...
- :

.. .
• •

IfIiriUI Bksr DC 110: D*c.naer DHIQII'I &B68tFnIik.'n

i . i1
lal Prct.,rt.cary PrClPl'l1Y Do ~ eopy
 The Computer Room

HGtAJsle 6.17

The Hot aisle should allow for 11 minimum of.3 It (.9 m) of eiearnnce

• A uniform Tack standard depth of 42 in (1.1 m) allows for rows to

be aligned flush wittl the tile of enid aisle while allowing for 3 ft (.9
m) of clearance for the hot aisle

·• · ·•
;--:-
·
'------'--
·· ·
~
· ·
·
"----'
• •
;--;
·
:..--'-
·
'---'
• •
:-;-
·
· ·
• ·•
• I~'V ... .
•
M .."".... •• H •• ' 4 ·..4 .. "'f' .... "':;

r-'~ {"t"'" :'" h,~ ",:5',- ',t "t"""'" O·

Rack Spacing and the Perimeter Wan 6.18

Equipment cabinets and racks should have a minimum dearance of 4

ft (1.2 m) from the walls and any special electrical equipment along
the wall

Racks/cabinet RBWS and Perforated Floo, Tiles 6.19

Ensure !hat racks and cablnelS are positioned and aligned so that
adjacent cleat Hoor tile can be removed without Interference from the
rack/cablnet
 Studmt G~jd. The Computer Roo",

Equlpmtlnt Rows and Cooling Unit Placement 6.20

The cold aisle should be a minImum of 8 ft (2.4 m) from the Computer

Room Air Conditioners to allow hot .. ir a cle..rer return path to lt1e
CRAes

....
"'C"'.~=

Air Handling Un il8 6.21

The preferred location for the air handling unit< is aloo9 the penmeter
of the computer room

Remote Power Panels 6.22

The preferred location for remote power panels supplying power to the
racks is at one or both ends of an equipment row

CI2ItD BldiI"
...,
B rCS! PraprlIItII)' "'o~ Mgt copt
 The Compuw Room

POW$r Distribution equipment & th<I Computar Room 6.23

The preferred location for the Power Oistribu~on Units feeding the
Remote Power Panels is outside of the computer room, In adjac""'t
space
• locating the POUs in a<!iacent space removes a ma int enanee item
and heat Source from the computer room

a u, d G PO ......... F~ .,.CII

E I ~trilall Roo Con'IPU", f ftOOtI'\

--'-

I u"'" . PP

Pou
I \:" rl.~ ..,...,
'--
$e"~ ~6o(f\1.

SAN Frames 6.24

SAN frilmes are network In terface racks d1at typically conSist of large
quantities of high-density fi ber termination panels
• Where SAN frame. consist of core and access switches, they snould
be Installed In separote racks
• SAN equipment often does not fit standard se/Ve.- rock dimensions
and requl"" special placement

Power and Telecommunicalion5 Cables 6.25

Power and telea>mmunicat.lon cables may be distributed either

overhead or from beneath the access floor
• They may both be distributed from abo~e . or both from below, or
one from above and one from below
 The Camp ...r Room

Cable TrayS 6.26

Cable trays shall be no more than 6 Inches deep (150 mm) and shall
be at least 2 inches (SO mm) above the slab and below the access
floor.

• cable trays should be si~d with a 50% additional capacity

allowance to account for future growth
• ""wer and communication cables should b.. positioned at
different
hei~hts '0 tIlat they can cross each other without interference
- Power is typically the lowest pathway with the communic<ltion
pathway mounted 6 incnes (150 mm) above the power
- Where power ar>d communIcation cabling cross one another,
they shou lei do 50 at right i1ngles to one another

cable Pathways 6.27

If both power and telecommunications cabling are distributed from

below a raised fioor then :
• Power cablin~ should be routed either adjacent to or within the cold
aisle
• Telecommunications cabling should be routed adjacent to or within
tile hot aisle

Fiber 0Jjtic cabling &. Ac:c:ess Floors 6.28

If fiber optic cabling Is Installed beneath a raised ncor it should be

protected from damage by plaCing it into it closed lid -containment
system

Fir. ConlrOl Systams &. Acceu Floors 6.29

Fire deteCtlon/"",,ventlon systems must be insta llecl at the highest

eievatioo under the raised Hoor

0-1.
IIlIte.t PI'Opr'ital1')' PW~ Ho1 CoJ2'1
Stud.nt Guido

Lighting 6.30

When occupied, the computer room shQuld h.we a minimum of 500 lux
(SO It .:and les) maintained in the horizontal plane and a minimum of
200 lux (20 It cand les) maintained In the v~rtH:a1 plane, both
measured at 1 m (3 tt) above the fini!hed Hoor
• PortatNe, battery-powered lanterns are suggeste(i in all computer
rooms

Compute< Room Drainage &.31

computer rooms should always contain a drainage system
• Computer room noor drains s~oukf always be Iltted with back·fIow
valves
• The computer room ceiling should be constructed to drain Or 9 uIde
water away from the computer room
• The computer room should have a supply of waterproof sheets or
covers available to cover all handware and equipment
• An industrial wet/dry vacuum cleaner should be kept clo .. to the
computer room in the event of a woter leak·

Ramps 6.32

The computer room may need to be equipped with eRher a ramp or

elevator to Msure wheelchair "nd equipment access
• The maximum $loJ:te for ramps is B de:grees for movement of
cabinets with equipment
• ADA specifies ~ maximum ri!ie of 1: 12, Or about 4.8 degrees for
wheelchair access
• I'm wh .... lchalr access, the ramp must be at least 36 inches I 9<)(}
mm wide, have hand ralls on both Sides, and have a 60 inch /1 .5
m clear landing at the top and bottom

e201a BIleif'
... ,
. tcsrp.,.,letary ..... rty-tJro .01 capy
 TM Computer Room

Review Questions

1. What type of equipment is typically found in the main dlstnl>utlon area?

lOrLr.J .... Ir-\r~ .~ ..... ; k"fl7

IrftlJ -" :- r.W: ':7
.:, It '" ~",:w..... ~
~w..
~ I' - ~'9y::

2. What ~ the hot/cold aisle met hod of laying out rack mounted equipment?

J. .lJ- ,""sk l, ~,. ,~ ,~ c"t, 4-I''''~c V~ ~

r (CJj.L, . J.)·' ChL ,.,
k.\- <\.\S' ~ v" ~('L,," "

3. What i$ the recommended maximum number of rackS to place In " row?

Ciiii. BiCii"
 Studa.-c Guilde

Module 1: IT Equil!ment & Racks

Module Goals 7.1

In thiS module we will examine IT equipment layout in the computer
room, after completing this modu Ie VOOJ should be able to :
• Identify the typical kinds of equ ipment housed in a Data ~enter
computer room
• Explain now equipment requirements can be translated into spa~e
requirements
• DiSCIJss the best practices for t/le storage or IT equipment In racks
and ca bi nets

!:ID10 BIles"
Equipment ~yout & Inte!Jrated Deelgn 1.2
The prjm"fY purpose of a data center is the storage and rna intena nee
of IT equipment ; here we will loolc at some of the mai n types of IT
equipment to be stored and best practices fer storing it
• It is important to understand that proper eq ulpment layout also
jJlvolves consideration of:
- Power Distribution Planning
- Cooling Planning
- Data Cobling Planning
- Future Growth and Equipment Planning
• Power, Cooling , and oat. Cabling planning wJII be examined In later
modules.

Future Growth Planning 7.3

Future growth planning is complicate<l both bv different clients' needs

and by the pace of technological dlange
• Data Center des~ners 500uld malce allowances for IT equipment
!",nover every 3 yea rs

,OB
1984 ~
I.~
student Guido

IT Equipment Trends 7.4

The general trend for data centers over the past 5 Ye<!rs has been
lewa rd shrinking space requirements wil7\ IT eqUipment growing
denser
• Server equipment has been getting smaller, but its power
requirements have remained constant
- Higher densities of equlpMel1t require greater cooling and noor
loading capadties

IT Equipment 7.5
The IT equipment typle.lty Iloused In a data center includes:
• ,.,,,twork Servers
• Digital storage Equipment (Disk Anavs I Tape storage)
• Network Route.-. and Swill:hes

Serve .... 7.6

Network server computers house the process; ng hardwa re and

software applications accessed by corPorate and internet dient
computers. They can come jn either a rack·moufltab!e Of" "'bl.Qde n
fonn.

Blade Servers 7.7

Blade servers are Shrunken versions of.stilndard servers Ihat have

been stripped of their external casings and ca n be .tored collectively In
• smoller space
• They are increasingly common, take up less space, but have power
requirements slmilar to standard rack·mounted servers

N
elC .. _ _ .... c ...
lIIudont Guide JT Equipment & RacMs

storage Area Nelworll. (SAN) Equipment 7.8-i

In addition to applic:<ltion serve rs, many data centers will employ
hardware ded icated to digital storage; these may be disk arrays or
tape systems
• Unlll<!! application servers, dedicated storage devices are less
uniform in terms of size, weight, power, and cooling requirements
- Tape storage takes up lots space and Is h".vy, but is more
efficient in terms of pOwer cMsumption
Digital disk array storage devices can requ lr" huge amounts of
power and require special clearance areas around the cabinet to
ensure adequate cooling

Networl< ROOI8I'11 & Switches 1.10

Network routers and swltthes connect the data center hardwd re and
external client users; they c.;n come In either a rac k- mounta~e form,
or a .til nd-alo ne form

IT Equipment stor"9" & Racks 7.11

The bulk of a data center's IT equipment will be rack·mountable

servers . !tacks are the prima ry storage u nit for server equipment and
are typically 20- 24 Inches wide and 32·4 6 inches deep.

Olltf!rWldth Gan Vary

~ lJ

g .
g

• IIwningIinsui:Ili
• l'il:hesflll~ mml

~
i
Student Guidlt IT Eqillpmont & Roots

7.12

Vertica I spacing ill rad<s is clefioed by the rac~ unit or RU

• 1 RU ~ 1.75 In ~ 44.5 mm

,
.•"'
I _..W
1 t· -t.. ."
c..~:l~W

•
_
r
lllu .n ")
I
Jn.II'-';~
. .j ... , .
g.... --
_
I I
, JQ' ~ . ~"

-, iI};;:-~- . -
J·U- 1·
-:--t_- '.

j
O,IllS' i -
.j ~,
v. s--
r . -~
, J -i.--
I O.6"1 ~"
..
1..25' -,
I au;a-
_L-., _-4_- ;
u _!"
! .. ..,
- ' . -;
-

1 J,.:--'. 't:"':
T-;;:;',..
. f •
l .,-t><l~·
f ,
-' -
-.
"'

Storing IT Equipment: Racks/Cabins... 7.13

All Data Center IT equipment that is not in a "stand·alone" folTO foctor

shou ld be stored in either, 2 or 4 post racks, or cabinet.
_Guido If Equipment & 1Iac_.

Number of Racks 7.14

The nu mber of racks required for the computer room should be
determined in conJunctJon with an assessment of the current and
future IT eq ulpment needs of the data center
• Power and cooling are t ypica lly the limiting l actor on the maximum
number of servers per rack
• Adequate space should be allocated for patch panels, power strips,
and ",bling for" rack/cabinet when It Is at its maximum number of
servers ~ rack
.2U
~~rsb'lp
'U
Switch 2U

UPSI~r
l5U rot;l!y .U

""""'"
N_ 4U

.....
Switct'l

$0< . . ,
.u

1.15
Data Centers shall emplov 2-post racks that utilize itandard .i>lng
cr;mventions where channel dimensions, spadng, thread system, and
usable aperture conform to CEA'31()-E
• Maximum height should be eight reet
• Each rack should have vertical ca ble managers attached on both
sides
rack
• A l'aci!>grouncl bar should be installed in the Dr a ground wire
should be mou nted on the bold< of one of the uprights
 IT Equipment & Racks

7.16

Employ +post racks that adhere to stilndard sizing conventions where

rail dimensions, panel opening, usable aperture, and spacing conform
to CEA-310-E
• Maximum height of 81t (2.4 m) and depth 43 in (1.1 m)
• Each rack should have vertical cable managers attached on both
sides
• A rack ground bar SIlould be installed in the rack or a ground wire
should tie mourned 011 the tx.ck of olle of ttle upright.;

Cabinets 7.17

Data centers shall employ cabinets made of non-combustibte material

that adhere to standard spadng conventions where rail dimensions,
panel opening, usable aperture, spacing conform to aA-310-E
• ~quipment mounting rails should be adJustabje front-to-rear and
should have RMU number Indications
• Door hinge orientation should be reversible or du al hinged
• C~ble sh~1Ibe able to enter the cabinet from both the top and
bottom; top access ports should be closable
• Maximum height should be 8 ft (2.4 m) and depth 43 in (1.1 m)

Basi Practices for Racks alld Cabinet& 7.18-20

Best pract>ces fur both racks and cabinet.; indude:

- Select cabinets, racks, and vertical cable managers whose design
minimizes obstruction of air circulation '~""' \ I0'1 A
• Qr!e rack unit of horizontal table management should be provlded""'" 1;" ""J. S.
for each rack unit of;.;p:;;a;;;t.;:;<h.;..::;~~nel 4.. ~"
• Gable management should be sized with a 50% additional capacity \!., - c"- \ I
Jr
aliowarlCe to acoount for hlture growth ~v"'"I
1<.0
• Shorter keyboard-vldeo-mouse (KVM) cables should be use<J to ~\.J'. \~
reduce amount of cable management necessary I
• Cable openings in floor tiles should be dlnec~y under vertical cable
managers
• Floor tile openings under racks and tablnets should be no larger
ttlan required for entry of cables
student Guide IT Equlpmont & ~""ko

• The use of gaskets or brush grommets on CIl ble floor openings is

recommended to minimize cool air leakage from under-Hoor air
plenums
• Power .c ords should not be run under equipment, mats, or other
covE1'ing other than access floor tiles
• Ensure that cabinets, rileks, and cable trays are properly grounded
and Iilat bonding cond uctors "re secure

Best practices for racka 7-21-2

Best practices for racks include:

• Racks supported by ~ n iiccess floor shall be bolted directly 10 the

tement slab, secured to the slab using threaded rods, or secured
using th readed rod to a steel eha nnel bolted to the slab
• Sharp edges at the top of the threaded rods should be SiIllJI!d; the

•
... posed threads under the access fioor should be covered
. ~ -
When installed on an acress Roor, racks should be located so that
the rods securing the rael< to the slab lOliIl opt .... AelMte It>&lIQor
tile stringer
-
• Racks shou Id be Ieyeled throughout a row

-
• ShlmmlOg of anchor points shOUld not exceed '1.0 inch (12.5 mm)

~ - \J ........J -..
-~:'
• Adjacent racks should be ganged together belon! anchors are

'1 ~ ~~
-
<X ~ t v.....
Best practices for cabinets 7.234

Best practices for cabinets include:

• )!Lank panels shou Id be instil lied in unused rack posltions to
maintain air sepa ration between hot a nd colds aisles
• Cabinets should have adequate width and depth to a~id_routin!l. ot
callies beh!n.<1.elllllprn~
• Doors should be removable without tools
• Side panels should be removable and lockable
• Both trent and rear doors should be a minimum of 63% open to
aicHow

EaOia liCit'
,.
IICSI ~plietary PlopMfj 00 MIrl c~
 studem Guide IT Equipmenl & Racks

Power Stripe 7.25

Power Strips should be IP capable to allow for SNMP monitoring and
reactive reporting of power consumption, spikes, dips, ane! pre-action
alerts
o Eq vipment that has dual power supplies should be plugge.!!. intq
sepa~ power strips supported by redu ndant UPS and generator

Equipment Labeling
-
systems

7.26

Documefltation and labeling "re crttiC<l1 to proper operaMn and

maintenance of a Data Center
• All computer room spaces should have a unique identifier
• floor space .should track the data center grid. Most data centers will
to
require at least ~ letters and two numeric di~1Is Identify every
500 mm x 600 mm- (or 2 It x 2 tt) Hoor tile.

Room Grid CoordlnaIA S)'BlAm 727

In general, lloor sp"ce identifiers should be formatted as,

o nnXXYY, where

- nn~ lloor number (In multi Hoor data centers)

- XX~ floor tile grId column
- YY~ Hoor tile grid row
_________ ~ __________ U ___ M ___ ~
-.II'~~"'"

..
---
h

--.
~

•
~

"
"t
'~ · ~I_'"'

Q20H' IICiSi"'
StudOlIt GulcM IT Equ/pm... & Rooko

Rack, Cabinet and Frame .... bellng 7.2lI

All racks/cabinets shall be labeled ,!UlIIt and WI'

• La bels shou k:I be affixed at both the top and bottom of the front
and rea r surfaces -
• A filcl</cabinet summary report should be available l isting all racks,
cabinets, and frames, including their type, location, size, capacity
and current usage status

Rack Labeling and tIM> Computer Room Grid 7.29

111e preferred.cack labeling practice is to inCOlrutte thJ fade's grid

location in the computer room , The grid shouUse 2' or 600 mm rx
x 600 mrn square correspond ing to the access ftoor grid (if one exists) .

, " A .. .. - -

:- ..... ·1 . .. ;., .... .. , 1 , ~, . ,' .l r ..·• ·\ ,1. t~

7·10
B I (09 1 "~~Not~
student Guide

Active equipment & Labeling 7.30

All active equipment. including servers, routers/switc~es, hubs.

multiplexers, storage hardware, etc. should have a unique identifier
keyed to its rack
• A two dOgit oounter should delineate the actl~e equipment In each
rack/cabinet
• The counter should start at the top and proceed downward. Shelves
with more thiln one piece of equipmen t, tHe counter sHoUld
increase from left to right
• In .general. active equipment IDs s~ou ld ~ forma!Y!d; nnXXY'Y-
ZZZpp, where
- nn;;;;;: floor number
- XXY'Y~ r?lck/cabinet/frame ID
- ZZZ~ a 3 character eqUipment type JD
- pp~ a 2 digit active equipment counter
• An active eqUipment summary report should be available

Equipment labeling 7.31

For e~ample, the t~ird server from the top In cabinet BA 14 on the 2nd
floor of a data C~~t should be called: 02BA14-SVR03 where SVR I.
Sn abbreviation for the server or equipment type
 IT EClUlpmont'" be","

TermlnaUon Hardware ldentifiens 7.32

All termination hardware shall have a unique identifier keyed to the

data center floor tile grid and rack/cabinet/frame in which it resides
• It should have a single ~Ip~eti~al dasi9".,lJon within the rack.
startjng from the 0 and then pro~ln.!1 downwards
• Pa;rl strand/po~ IdentJfiers shou Qne to three d~ts In length
arK! should be "I the e~jfjer
- For example, the 8th port of the third patx:h panel on a rack
10Ulted in the 8'" column (AJ) and fofth row (05) should be called
patch panel J>Ort AJOS-C-OS

Sample Tarminallon tiardware labeling 7.33

"

Tannlnallon tiantw ..... llIbellng 7.34

All termination hardware shall be labeled :

• With manufacturer-recommended labeling systems
• With only m"chine generated, self·adheSive labels
- On both sides (where applicable)
A termination hardware summary report should be available listing ali
termination hardware, their types, locations, cap~ltv, current usage,
assignments, etc

omilicsr
51udent Guide IT Equipment & Rad<s

Revi_ Questions

1. What has been the general trend In data center spadng

requ irements for the past five years?

<; .....,,1h- ·.W'" <..

;

2. What are the s~cial wncems associated with SAN/dedicated

storage equipment layout in the computer room?

3. What Is the maximum recommended rack/cabinet height?

 s.... ri~. Dlo.Islor Plonnlng. & F~. Stopping

Module-S: Security, Disasler Planning. & Fire Stopping

Module Goals 1.1

Th is module will look at 'systems for sewn"9 data center physical

assets again.t theft, naWl'lI1 disaster, and flre; alter completing ttl is
module you should be able to,
• Identify the operational, architectural, and electronic components
of site security
• Draft a data center serunty plan &. a diSilster re<:overy plan
• Explain the .".,oal requirements of data center and computer room
fire protection
student Guide So.umy, _lOt PIo."'n9. & Fir. Stopping

Security Components 8.2

Oat<> center security has 3 key components:

• Architedu re
• E~ronics
• Operational Proc;,(Iures &. Planning

Atdlitectural Components of Security '.3

There are two ways that a site's architecture can contrfbute to site
security
• En vlronmental Desig n
• Building Design

Security by Environmental Design

Security should be enhanced durinll the des".ln phase by applying

these -environmental design principles:

• Natu ra I Access Control-tile control of the flow of visitors and

personnel should be directed away from the high security areas
• Natural Surveillance-entrances to high security areas sh<>uld be
visually, as well as electronically, monitored
• Territorial E'nrorcement-hlgh security areas shou ld be eaSily
recognizable -
 Shulonl Golde Securtt)'t Dlsater F*tanNng, & flrw. Stopping

Natural Ace,," Control 8.5

Data center spaces should be designed according to one 01 four
distinct types of security eategories :
• Public-space available for all pedestrian traffic, including lobb~,
dining areas, pal'ldng 9arage~, and hallwavs
• Sem i-Public-space thaI is freely accessible, but not offidaHv
availa ble to everyone, such as clas. roOI1\$ and conle,..",ce rooms
• Sem l-Prlvate- space where access must be granted to pedestria n
traffic, such as a utility room
• Private_space restricted to both pedestrians and many
emplo""es, such i'IS the computer room

Natural Su rvell"'n"" 8.6

Naruml surveillance relies on the use and placement of environmental

features such as walkways, open spaces, and pedestrian traW",
patterns to maximize visibility
• Data center secur1ty designers should design spaces so that there
are rna ny ways for observers to see unauthorized actil/ity

Territorial Rei nfOl'cement 8.7

Territorial reinforcement is the creiltiOn of a .en5e of community such
that if an unauthorized person stray. into an area, they fre! out of
place and are eaSily identified
• Ensure that the atmosphere contributes to a sense of territoriality;
secure space should produce iI sense of ownership .. nd discourage
entrance
• Clear borders should be visually recognizable between oontroUed
and public spaces

i!iOOi; lli¢sl"
student Guide Security. DlNlto< PIo""ing . .. Fii& 8(oPp4ng

~rlty & Building 08&ign 8.1

Building design components of security Include:
• Walls
• Barriers
• Openings
• Doors
.. Windows

Security & Building Exteriors '.9

EVjlluiltion of ardlltectural drawings and onsite inspection of an
existing bUilding for sectJrity should ev~lu"te the following:
• Amount of SPilt'. exi<ling between walls, ceiling and f k>ors
• Rlsk Introduced bV HVAC air-handling spaces
• Modifrcation of t~ original w..lIs, ceiling, anod nco..
• Rooftop accessibility
• Under·noer accessibility through tunnellng/subtemmean
passageways

Wall~ 8.10
Al l concrete walls should indude steel reinforcing bIlrS (or'rebar"), as
concrete offers little security by Itself
• Additional rebar will Increase wall durability and reinforcement

Building Openings & Barriers 8.11

Any building opening less than 18 It (5.5 m) above gl'Qund and larger
than ~ in' (620 em') should be"protected by a barrier, alarm and/or
t!)onitcr
~

• B.. rr"'" are not impenetrable, but are used to delay entry
-
entrance
• Barriers may be m.de out of:
-
So rrlers sheu Id foro! the intruder to leave evidence of their

- Bric.k, concrete, steel r wood , glass

StudMt Guide Socurily, DI .....' Planning, & Fir< Stopping

Non-door Open;nsp 8.12

Roof hatches must be manufactured from a minimum of 16-gauge (1.3

mm) steel and should locJ<atole from the inside Q,(!!y
• When ladders for the rooftop are mounted on the build ing exteriDr,
a secu,ity ladder cover must protect \t1e first 10 feet (3 mj Of the
ladder
• All permanent roof access must be from the I ntelior of the bui kling
• Doors leading t.o the reef must meet the security requirements fDr
exterior doors including double·CYlinder deildbolt locks

Doors 8.13

The ·lollowing a,e best practices concerning data center doors:

• If wooden doors are used, ensure that nD gap exists between the
doorstop and \t1e doorjamb
• Hinge pins and mounting screws should always be mounted
tcwards the protected side of the doo'
• Hinge pins should always be welded or Hanged tD prevent
unauthorized removal
• When possible, hinges should be fastened through the doorframe,
Into the wall st ud
• Doorfr"mes should be securely fastened to the wan studs
WlndC)W$ 8.14

Windows mOU nted on the exterior of a dati> center building shell are
not advisable for high security c.ata centers; best practices for data
center windows Include :
• Only pictun! windows shouKl be installed in any ""tenor wall of the
data center shell
• If a oomputer room wall is an ex teriOr wal l, then that wa II should
not ha lie • window
• Exterior windows should never use putty or molding to secure the
panes of glass or plastiC In the frame. OnlV rr"me mounted
(grooved) wi ndow mountings should be used.
• Window frames should be constructed with rigid sash material that
Is anchored on the inside and Is resistant to being pried open
• If windows are used in the computer room, the 5111 should be at
least VI m (l! feet) "bove fini.hed floor

Window Glazing 8.15

'G~zjn9j is the inSitanatlon of glass, plastic, or-laminates to Increase
su rt..ce resista nee to Impact, hre or other threats
• Tempered glass should always be used In dala center windows and
doors
• Lami nated glass, consisting of alternat1ng layers of glas> "nc
piastic, should be ·consldered where impact ..,.;5tance Is .,;sentlal,
such a s exterior windows and doors

EhH:tronic Com ponents of Security B.16

The electronic components of secu rity indude:

• Electronic Access Control System
• EJectronic locks
• Alarms
• Su rveiliance .& Monitoring Systems
• Llghti nQ Systems

...
BlCSI froprfoaa.y Pf~ Ncn CsJP1
Stullon' GullIo S-tilY. O.....r Planning. & fire stopping

Access Control System. 8.17

All Dat~ centers ,liould restrict entr.nce to authorized employees,

torltractors. and visitors either through S<!curity guards or electronic
access control (EAt)

EAC & the Computer Room 8.18

Electronic access control (EAC) sySt ems shall be in place at a minimum

to control entry into the computer room; !hese systems shoukl tie able
to track personnel movement through access points
• Eadl employee should have" unique idi!flMer
• Automated EAC systems should recond:
- Entry a nd exit t ime
- IdentifICation of the ent rant
- Authoriz"tlon mechanism
StUGenl: GUide Security, Disaster ",.,nips, & Fn StOf)ping

Electronic Aeee8. Contl'()l Typ.. 8.'9

Keypads: When keypads are used, they should utilize high security
approach when: ea<;h numeral Is randomly scrambled to a new
position each time the start button Is pl1!S5ecr= -
• It Is rec:ommended that keypads be used in conjunction with a card
reader or other EAC method
Accns Cards: Access card technology should be integrated into the
1D badge when possible. Access card systems should Incorporate an
active anti'passbad< feature, permitting only one entry without an
acrom >'inylr19 ex it
Biometrics: If Fingerprinl/Eye/Face recognition hardware Is used, It
snould be used in conjunction with card access to decrease databese
seard> times

Doors & Electl'<Jnic Security 8.20

All pedestrian doo~ I~ted within the data center an d computer room
should possess:
._-
• A full-sare lock
• PaniC/fire-exit hardware that can unlod< the door
-~

• A "req...est-to-e,,~P (REX) manual releltse device that unlocks the

dooi'TOTatleast 30 .emnd.
- Doors to high .ealrity moms should also be equipped with door
r;>O.sitiOn ..,nsors that ~er an alarm If the door is propped
open for more than 3 ml!lY~s

Intrusion Alenns 8.21

Intrusion alanns a re designed to detect unauthorized person nel; d eta

center intrusion alarms should utilize. one or more of the following
a fa rm trigg~:
• A break In an electrical cira/lt
• Light beam Interruption
• Sou nd Detection
• Vibration Detection
• Olpaclt;lnce Q\ange Detection

atriIIUCSi"
.~
aeel p"''''''''y~ NiI!It Copy
Studoott Goklo s..ct."Y. 01 •• _ Plann'"". & Fire Stopping

Ala(t1llnlegrallon 8.22

All "Iarm i:lufputs should be terminated in one of t~ese klcations;

• Central Command Station
• Dinlct COm>ection to EMS, Fire. or Law Enforcement
• Proprietary COnnection (Secure COmmand Center)

Local AIann Termination 8.23

Data center alarms should not be terminated solely locally
• Loca lly tenninated alarms may be a psychologica I deterrent and
temporarily interrupt aimina) behi;lvio..-, but there is no guarantee
the alarm will be detected "rod acted upon by outsi.d e aU1;i1orlties

mtUIBIC3"
Studont Guido Saeurity, 01_ Pllnnlng, & FI,. SlopjIlng

Surveilta nee Cameras 8.24

Data .Centers contain mission cHtlcal computer "I'stems. In a~di~on to

computer hardware, thev·typkally include:
• Computer rooms
• Cammor Cross-connect Spaces
• Common Riser ClOsets
• Elevators a nd Stair Wells
• Plen urn a nd Ceiling Spaces
• Telecommunications Spaces
• Util ity Rooms
• Vehicular and Pedestri<ln Gates

..
~

.
~. t!"
I I

Vi",", rT'I!Ineltttrletl ~
Sl/9te h"l l)luS ~ Of
J
lstr~r
"~ ..-I(I I!O ""'1lI'W

......
IICSJ Pr4lpriealyPrIIP8rtY Dg NotCop'l
Student GUilM

Sec:urlly & Intelligent Surveilianc. Ca ....r.s 8.25

cameras utiliz ing artificial intelligence snould be used to monitor high-

value areas of the data ce<lter where a speclllc asset Is at risk, or
where constant surveillance is Impractical
• Artifidal intelligence (AI) allows 'triggering' movements detected by
the cameras to nespond with alarms, recording, or I~ entries

Surveillance Camara s.curlly 8.26

CCTV Cilmeras should be protected against lt1eft and vandalism.

Protection for cameras should indude:
• Mounting the ca mera out 'Of reach of pedestrian traffic
t A secure endosure or dome
• Environmental protection for any extreme heat or cold
• A secure mounting to a stationary object

lighting 8.27

Basic security lighting should be provided at the following a",as:

• Perimeter fencing
• The bui Iding perimeter
• Entrance gates - pedestrian and vehicular
• Build ing enm.nces and exitS
• Vestl buieS/lobbies
• G<>tehouseS/Gu,mdhouses
• Windows and exterior openings
• All public area 5
• Pedestria n walkw"ys
• Stairwells
Student Guide Security. Dioo_ Plonoing. & Fi,.. S\op"",,U

Minimum lighting Lev.ls 8.28

MINIMUM LIGHTING (FOOTCANPI-ES)

Outer Perimeter 0.15

Inner Perknet8r C.•
B~ or P~rWnetet FenCl! 1.0
Vtll'Plcular Entr.inces ' .0
Pedl:Strisrl EnlrlinCM Z.O
Reslric:ted StruaUI"$ 2 .0

Crear ZoI'llM 0.2

Parking AnI!!ato 1.0

Operallon,,' Components of Security 8.29

The op ..Iiltional component of site security Includes the development

and implementation of,
• "The Data Center Security Plan
• The Data Center Disaster Recovery Plan

Inilial Security Aagagsment 8.30

Every data center should have an initial security survey completed
du ring site preplanning to assess a nd provide written
recommendations regandlng'
• Natural, technologica l, and human threats
• Risk/probability of breached seaJlity
• The potential impact of breached security
• Counterme~sures for b~ched security
• Cost/benefit. analysiS of possible c.ounteomeasur-es
so.dent Goli. Sooorily, D._tat" Planning. & Flo. Stopping

Security Pilln 8.31

The Initial security assessment should lead to 'the formulation of the

data center's security plan, a written physical 5eOJrity plan detailing
pollc~s concerning :

• AcceS5 Control Polides (for both visitors and staff)

• Alanm Pol icies
• Surveillance Policies
• Equipment Control PoliCies
• Personnel POlicies

Secu rlly Plana & Policies 8.32

The data center security plan shO<Jld have separate component polie~
regarding tt1e protection of:
• Personnel
• Physical assets
• Data

On.Golng Security A8 .......menls 8.33

The data center security plan shou ld be revi<lw.ed and updated:

• Once annually
• Following any major remO<ieling project
• Following any major breach of security or disaster
• When a change In a local security event wamlnts

@»H'BLe'"
Slueont Guldl s.curtty, Dbl.ne.r PI.nnlnG. & Rre stoppJng

The Securtty Plan & AceG.. ComroJ Polic), 8.34

The· Data Center Security Pion needs to detennine the following access
control pollc~ '
• Who should h~ve access to the data center?
• During what hours should the dal<l center be «cessible?
• What are the entry points of the data center?
• How Is access typically granted?
• How is access granted to visitors and contractors?
• Is sole occupancy prohibited or allowed?
• How are \llsltors to be monitored?
• Are access control proced ures different during non-business hours?
- What types of doors/b<lmers and locks are to be used for each
entry-point?
• How are keys mIlnaged?
• How are access breQches to be handled?

Access Control: Best Practices 8.35

Best practices for Data Center access rnntrol include :

• The computer room doors should rema in closed an d locked at a II
times
• All after hour entry/exit into the computer room should be tra-cked
either manually andlor electronically
• All visilors and contractors should be r<!quired to Sign ..'" entry/e~lt
109 prior to entering the computer ",om, even wilen they are
escorted
• A badging policy should be in place for the computer room

~BS Control & Authenlica1lon 8.36

Access control should Include one or more of the follow ing methods of
authorization ~uthentication ,
• What"a perSOn has, e.g., keys and cards
What a perSOn knows, e.g ., passwords and codes
• What a person Is, e.g., guard recognition, biometric data
Stude'" Guido Soout1ty.III...tar Planning, & Flro Stopping

B8d~ Polk:y-Be5t Practices 8.37

Data Center personnel should be re<iui~ to display b.adges

authorizlng entry
• The security plan should designate whether and how visitors
acquire temporary identification badges for entry Into secure areas
• Meosures to Identify expired badges shoold be in place. These may
Include :
- A seasonally rotating color scheme
- A printed expiration date
- Self-adhesive colored "'kens with set expiration deres
- Visually di.• t inctive styles that "xplre at a pre-detenn ined date

Slgllage 8.38

The sewrity plan should identify the wording and location of .ignage
and displays intended 1:0 control pedestrian traffic from straying into
unaL1l:horized areaS

Alarm Policies and PtOI:edur.. 8.39

The Data Center security plan should have written guidelines detailing
the appropriete responses to all bui lding alanns
• The security plan should specify what types of build ing alamis are
in place, as well a. the appropriate personnel response to each
alarm
• The security plan should specify special ~ttess control proced .. res
In the event an a larm is activated

Survaillance Procedures 8.40

The security pli!n should derail any high-risk aneas in need of special
sLlrveillance .
• In the eYfllt, that closed-circuit television surveillance is to be used
for such aneas, the security plan should detail the appropriate
placement for surveillance cameras
o The sewrity plan should derail:
- types of event s t hat should trigger any surveillance alarms
- appropriate responses to those events and alarms

DC 110': DI&I &n_ DlMI ...... het P~.

BICSJ """""'-Y
"',$
~ Mal ~W'
 S1udttrt Guide

Equipment Conlrol 8.41

All data center backup power supplies and oltler m ission-Oitieal
utilities .hould be located In a secure area, with periodic inspectio ....
for sabotage
• Computer room doors should rem. in dO$ed and locked at all times.
When posSible, the .door( 5) should be alarmed to prevent propping
• Authorlred data center personnel should remain w1th all vendors
while Installation, maintenance, or upgrades are performed
• The security plan should coordinate with the eyber/IT plan on how
live data and phone jacks In public Of" semi-publiC areas are
m<l<1 ~red a nd secured

Theft Protection 8.42

The .data center should have a detailed asset log or database In which
all IT and telecommun icotions equipment i$ traCked
• Unique Identification numbers should be permanently affixed,
etched or stamped on all computer, network and
telecom mun Icatlons eq uipment
• In high-riSk areas, computer and network equipment should be
pennanently fastened to some part of the building or pi<!ce of
furniture
• The us<! of an electronic asset program (EAP) is encouraged. EAI'
tags should be affixed to all IT and telecommun ications equipment

O2t1liCSf
Personnel PoIiciM 8.43

The security plan shoUl" detail "II hiring and termination procedures
and should detail mandatory information from applicants, including:
• Name and vanatlons (venf..,d)
• Address
• Otlzenshlp
• Military record
• References
• Security dearances
• Education
• Employment history
• Spousal/Family infonnatlon
• Criminal convictions

Personnel Exit 'nklrviews 8.U

The security plan should require that "II data center employees have
an exit interview to;
• Reduce the loss of wnfidential information through disrussion of
non-disdcsur-e agreeme:nts
• Discover problems that existing em ployee. won't revea I
• Reduce lOSS of dat.. center property though the use of checklists
• Give the employee a chance to express grievances, reducln-g the
possibility of sabotage

DiBaBter ReCOVilry Planning 8,45

All data centers should have a del'alled disaster recovery plan in

addition to, or as part of, their overall sewrlty plan
• The disaster re<;Overy plan should integrate physica I a nd IT d isast ...
recovery with an orlJanization's business continuity plan (BCP)

mn,'JiSi'
 Security, O.....r Planning. & Fn StoppinG

8.46

Disaster recovery planning s~ould tak~ into account the toHowlng

types of major disasters:
• Earthquakes
• Epidemics
• Falling objects
• Fi~

• Landslides
• Loss of utility services
• Severe weather

Criminal Disasters 8.41

The disaster recovery plan should take into account the following typeS
of crimin,,; actMty:

• Arson • Kidnapping
• Blackmail • Looting
• Burglary • Rape
• Bribery • Riot
• Conspiracy/Racketeering • Terrorisn1
• Disorderly conduct • Theft
• Emt>ez.zlement • Trespassing
• Extortion • VandaliSm
• Fraud • White-collar crime

The Disaster Recovery Plan 8.4a

The disaster ~covery pia n sf10uld indude sections detailing:
• Pre' dlsaster actlvltle5 I testing
• Emergency activities
• Response activities
• Recovery activities

~.BICSt'
 Disaster Planning: Tellti"g 8.49
Data centers should regularly and routinely cooouct disaster recovery
tests
• Dlsaster .plannlng should Indude the creation of mutual aid
Bgreem!!l1ts with other businesses and erga nizatlons
• Disaster planning should indud. coordinatiM with federal, state,
and k>l:al agencies

Disaster Planning: Enuorgencin 8.50

The plan should detail measures to protect the following during a

dlsi!ster:
t Occup~nts

• Disaster aid responders

• ConlJnLJed operations
, Site USii3 bility ilod telecommunir;(ltkms infrastructure
• Regulatory and contractual obligations
• Company reputation

Emergency Operations 8.51

Disaster recovery plans should indude detailed plans for all aspects of
emergency oper..tion of the data center InCluding:
... Access control plans for emergency cmd non-emerg~ncy personnel
• Operatlonal checklists, with detailed documentation and
InstrucIJons on operation of equipment
• Facility shut-down procedures
• Emergency security procedures
• The k>l:ation of all emergency eqUipment
• The use of fire and eme'1lency equipment

@JCI108U:::Si"
 S<ocurlly, DI ...... ' PIII""i"!/. a ron> 81Oj1p1ng

Fire Dete<;llon Component 8_61

Every Data Center shall be equipped with an addressable,

electronically supervised, Rre alarm sy.tem
• Ear1y Detection--ln areas where maximum fire protediOfl ;s
required, e"rly-warn lngjlnclpient systems should be Installed at
selected r<l isec floor areas
Incipient Systems typically consist of an air S<l mpling pipe
network of ccpper or PVC pipes lnot.lled below the floor with
strategically ploced air sampling potts

Incipient System Fire Detection

amI

L
T"'~ ~I.ooPIlDrt "[]I~
hl r d~~thllfl
O.'!H m (1 tI) ""I~~
_01'11 Guide _ t y, D I _ Planning, & A .. SlOPlNng

Fire Detection, cont. 8.62

Cross-zoned photoelectric and ionization type detectors should be

provided in computer, electrical, and battery rooms
• ionlzatiort type detectors are. designed for areas of low air velocity
• Photo-electric type detectors are designed for areas of high air
velocity

lonization-type Detector

PIloto.,ectrie·Type Detector
$Iud.", ","do Security. 01....' Plonnlng. & Fire 51""ping

Fino Detection. Cont. 8.83

Generator rooms shou ld be equipped with thermal fire detection
systems

FIr.. Detection System8 Sum....ry 8.64

All!/; CareCTION SYSTEM

Coml><lle'Room Indpient or EarlyW.mil9 InIItIligont
Nl!lrtlOrk Op8OltiQrK c.rrter Incip;"n! or e.,ly W.milg Inllellig"nt
E:DttU1~ Room In,"pionl or eorlyWamilg Intelligent
Office loniutionlPholotoloctric
E10ctJical S'>i!chge., lonizationlPho~ l ectric

Sal!ery & UPS Room. loniutionlPncloeloc1nc

G"""rator Room. Th."".lor Flame Oolocun
C~iij ... Room l""i<lItionrPholotoloctric
Back-<>l Kou.. loniz.tionlPhotooloctric
 Stud.nc Golde SeC:UrHyT DI .. ster Pbnnlng, & Fir. Stopping

Fire Systems: ProtecClon C<lmponent 8.65

The protection component of the lire system includes those steps to

prellent fore spread; these should include:

• Walls separatlnq data center electiical rooms, b.ttery rooms,

mechanical rooms, and telecommunIcations rooms shall have a
minimum of I-hour separatlon rating
- -
- Doc rS and frames within a rated waU sh all match the ratl ng of the
wan con Slruction
• Glazing within" noted w,,11 should match the ralinQ of the wan
• Electrical rooms and battery rooms should have gla2ing within the
doers only
• FloorS above and below the data center shan be Z hour rated
• Air fj~rs should have a dass 1 noting
.-
- Dutt covetings and insulation should hiwe fTinne spread ratings less
!han 2S and smoke developed ratings less than 50

Are System: Suppression Component 8.116-7

There are two basic types of sprinkler systems:
• Wet Systems-these are the more common type of systems, their
pipes alway. contains water
• Dry Systems-these are best R@CW;:e for mission cr~lcal areas;
a
their pipes are dry until lire is detected and pose less risk of
aoddental water damage

AREA SP~INK1.ER SYSTEM

Ccml)l,Jler Room Preactlon SDJlnkler System
Nat'war1r: Operationii' Cenler Prescticn S~rjnklet" Sl(S1ern
~ Provide, Enlr'ance Pread:ion Sprir.t.:~ System,
Office We+. SPl'inkls( Sy&tar'n
f:le~ic.,1 5wltcllgeer Wet Sp~'I.r System
BattElry & UPS Rooms Prsaclion SQlin~r System
Ger1etator Roomz;. Wet SorinkJer Sy&lem
C"IIM Room Wet $pri",1e< Syslom
eac>.-ol ~o ... Wet Sprinkle< System

~UI.ICSi"
....
• Ieta "",~p,~ D-o N«"Copy
 Stud_'" Oula. 8.curtty, Di"'lter PhuuUrtg, & fIN Stopping

Fire Protection: GerMral Conslder.tlons 8.18

General fire stopping policies shoo Id indude;

• All air duets in the computer room should have.fire dampers

• The computer room sh.oulc! not have any trash recepta(;lcs
• Tapes and other back-up media, as well IS critical dOOJments,
should be stored either In a separate room with cp>t<-hov, fire rated
walls or in a certified-fireproof c.binet/s;Jre --Iv"
• Water to the data center sprinkler system should be controlled hya

-
different valve than t hose used by the ,est of the building

'PlQIHCI Blcsr
....
IlIIt8rPreJN..,"~M04copy
Studani Guida _"rtiy. 0 ....1or P......ing. & Fin> ~ping

Fino Stopping System: General Requl ...... enls 8.69·71

Every Data Center shall have a fill! protection system which should
include :
• Dual- intE 'lock pre-action sprinklers
• An early smoke dEtection system
• A water leak detection system
• An under floor fire p,otectJon system
• H and-held fire extinguishe,,;
• The system shall provide manual pull stalion. at all e.it doors
• Firefighters' control ponel, graphic smoke control panel, pnnter and
annunciator shan be IOC<Ited at the main security office
• The system shall provide connections (rom flow/t<lmper swi\d1es to
the fire alann system
• The system shall interface with security system; upon activation,
the fire alann system shall release all se<:u lily doors
• The system shall provide monitoring of fire pump and generators, jf
provided
.
• Handheld, dean agent extinguishers shall be provided as required
by local codes and jurisdictions
• ElCtinguishers that use dry·chemlcal agents shall not be used
• The system shall provide smoke detectors In all unoccupjed spaces
• Ionization detectors shall be provided In conjunction with magnet"
door holders, where applicable
• Fixed temperature heot detectors Or Hame detectors shall be
provided in generator rooms and the temperatu re set point
coordinated with the sprinkler system

Fire Stopping S)'&tem: Comput.... Room Cansldem10ns 8.72

Cross-zoned photoelectriC and ionilatlon-type smoke detectors snail be
provided in computer rooms, and In UPS eq ulpment and battery rooms
- Cross-zoned smoke detectors are to work In conjunction with
the dry pipe sprinkler system and the dean agent Hre
su ppression system
Student Cklkte Security. DlsasMr PI_nnf"g, & Fire $topping

Finl Stopping: Operational CompDnent 8.73

As part of the data center's lire protection system, there should be

formal pOlides in place that detail :

• The type and location of fire suppression equipment

• The type of protective containers to be used for sensitive
material

• The policy 0<1 the stotage of combusti ble chemic<lls and

materials

labeling and Signage 8.74

Emerge<lCV procedures should be posted on all fire alarm control

panels

• FIre alarm buttons should be clearly la beled

• All devices ,mould be labeled with their fire alarm address

• lunctJon boxes are to be painted red
Paper Storage and Waste ~ptaclRS 8.75

Paper products and supplies should always be stored o..!!..tslde of tile

computer room -

• If paper products are reqUired to be stored in the computer

reom, then the 5upply sllould be limited to no more th,m a 'one
daysuWiy

Waste containers In data centers should have ~ta! lids to assist in

Quickly eJCting ulsh lng fires

• Receplades should be emptied frequently to reduce the

accumulation of IIamma ble mat erials
Student Guida s..c..rity, DI.....' P'-n ..... g, & FI,. SioPl'lng

Review Questions

1. What are the 3 key components In data cent er security design?

- .", (../.-. ~JVt

_ '"\ { L''''':h t':..

- <:> t '<I'' ' fi ~""" \

2 . What information should electron ic access control systems record In

regard tiJ computer- room entry ?

3. What are the five policy a rea. a data center's secu rity pl.n sI10uld
addNlss7
 Power &Eloecrick~

Module 9: Power.()ala Center Electrical Systems

Module Goals 9.1

There are 4 major data center infrastructure systems: electrical.

cooling, teletommunicatlon, and automation/control systems.

ThiS. module will 1001\ at ttle first of these s\IS1ems-the electrical

system , after f in ishing this mo"ule you should be able to:
• Identify the typical components of a data center's electrical system
• Esbmate computer room power requirements
• Describe how electrical system architecture rel<ltes to target class
reliability rankings

mcI10 BiCSi"
..,
sal ,~.rtICBry Pr~ I'Iol co~
Studont Gul. ""wa' a. EIoCll1cil:y

Components of the Elktrical SY$tema 9.2

The design of a data center's ~lectri"ll system includes ~Qn.ideratlon

of the follow ing components :
• Power Requ irements
• The Utility Provider
• Backup Power Supply / Generators
• Utility Entr~n~e " Switch Gear
• Unl nt""uptlble Power Supplies (UPS)
• Power Distribcrtion Swltchooards
• Power Olstribcrtion Units (PDUl
• Grounding System

O~a Center Pow@r systems

'""10, .. :.= J
i

L
~ I:,h~ '_'~IL4~------1L__~__"_U_P____~
_____ __ ~ (jene~a[OI'~

1
\ JPS

I_S. .
dk hbo.olnk

1
_IIIOulde P..... , & El.ClriCily

The Electrical System & Clan Ratings 9.3

A data center's e'ec~rlcal system design 15 d riven by Its ta rget
reliability class
• The reliability class of the electrical .ystem i. primarily determined
by the deg.ree of redundancy It exhibits
- Class ranking, are dete,mined by the 'weakest link'; if a high
class rating Is required for the data center as a whole. a high
dass rating is also required for the electrlca I system and ~
electrical sub-system

Redundancy & Singl. Points of Failuno 9.4

Class F3 and Class F4 ratlngs require the elimina~on of single points of
failure from t~!I.electrlcal SYMms
~-".

• Redundancy Increases fault tolel'ance and maintainability. but

Increases system complexlty--a leading cause of human error
.'>0
- Complex ltv must be weighed against ease of operatIOn. cost,
and overall system capadty

Eluctrical System Redundancy 9.5-7

Electrical system redundancy Is usually d1aracterl,ed on the basis of
the -N" rating system; according to this rating system:
• N - meets bilse.requiremems for minimum load kW with no
redundancy

UPS#'
I PDU *1

• N+l - one additioDalllnit/CTlQdLl~h. mo", than base

~~.jremeilf;' the stoppage of a $tRgle .uoi t will nct..dl$rupt

-
oper~tjQn~ ' .

~~,_U_P_S*_1-,~~_____1
PDU.1
__
4 UPS *2 ~ PDU i2

W
BICS. ",pn.wy Pr~~ ,.,r Cop~
 st._Guide

• N+2 ~ tw<> additional units/modules/paths il'\ addition to the base

requ irement; the stoppage of two units/modules/paths will not
.disrupt opera~on~ • .. - .. _. - - - . -

..,:., \, rl lJPS 111 rl 000" f-

,"
f'''' ~
",. lJrs u POlIo>

-{' UPs;J~ -1 ",un f-

• 2" ~ wo complete unitS, modules, or paths, for every one

o:!'qu,red for . e. . ase .)<$tem; failure of one enbre system will not
-Ilisrupt operations . ....

UPSNI PDU'l

PDU t2

• 2("'+ 1) ~ two complete (N+ r) units/modules/paths

PDU.I

UPS" rou .2

r Uf'S')

---1y UPSl4 8f-----C........,.,:,....,:-:---.,JI---

student Guide Power & EJ.ctt~

Classes & Electrical System Topologies 9.8

A Data Center must meet 3 requIrements to secure an electrica I class
rating : -
• The site must have a back-up supplv/ generatoc..
" The site must have an on-line UPS syste,!!'
" The site must be properly grounded
~

Upon meeting !he.e requirements, the overall design of a data center's

e~rical system typically falls within one of t~e foHow ing four
categories :
" Class Fl - The SIngle Power Path Oat.. Center
( " Oass F2 - The Single Path Data Cente.- With Red undant
components
• Oass F3 - The Conc"lTI!ntly Maintainable and Operable Data Center
( " Class F4 - The Fault Toler"n! Fully Redundant pata Center

-molIIUCS"
. ...
8/CSI ~, Pt'l:lp.rty-DQ Mot ~
Student Gukte P....r & Electrlolly

Class F1 Elactllcal System .,.

A ClaSS F1 electrical system serves a basic data center w~hout
redundancy
• Industry DesaiptKm: Single Path
• CompOnent Redund~ncy : No
• System Redundancy: 1110
• UPS : One
• Maintenance Under load: No
• Failure Recovery: No

t Utility

l~ ·.,· .
Electrical Distribution

S1<IIk UPS , Malntename

Bypa,s , Bypa<S

"

.DU
I
~

CRITICAL
-TTl Loads
SllldonI Gul ....

Class F2 Et.ctrical Sysam 9.10

A Oass F2 system may have fedundant components in UPS or
generator systems, but not in distribution. Large-scale maintenance
requires interruption.
• Industry DeSCMptlon : Sin~!~L~!!~itj, Redundant Components
• Component Redundan.(.. '" + 1 '
• System Redundancy: No
• Maintenano. Under load; At the system level only
• Failure Recovery: At system )evel onlV

,
- - -- .

_ UPS UPS Mom""",

~Mi~ Bypil5S

,
'.
'DU .,

•
CIII1ICAL
TIl Load.
 _ & EI.<1riclly

Class Fl Eleclrical System 9.11

A Cia 5S F3 electrIcal system has redundant power pilths to the critIcal

load
• Industry Description: Concurrently MaintaInable and Operable
• Component Redundanq(: N + 1 (or greater)
• System Redundancy : No
• UPS : One system with multiple power paths t o tritk:a I load
• Maintenance While Under Load : Yes
• Failure Recovery : At the plant and distribution level

~~
_..,......_" .
. c . ." " - . . . . , .. _ - - - '
I ...... .
-1>11.......
. . " - ; .,

-. .... --
.... -~
-.~
•
-:-.;. I

-.-_
_
~-
..
..... _~. ; •. ,;.....,...:.:.......-~\ll ~
III!:
" _4 '
. ..
.com<Il
11« •
'< .
I :

•
JI*l- ~ tl..Pc....1
CRflt~
. PIlU
. . ~ "- I'~
- "~
. . ,' ... .............
.-'
", ..... ...

CU1C1 BiCSi"
 Shuiettl Guide Powo< & Electricity

Class F4 Electrical System 9.12

A Class F4 eled:lical system Is fully redundant with no single points of

failure
• Industry Description; Fault Toler.ant
• COmponent Redundanc(:N+1 or hi9heJ:'
• System RedUndan~~~~r't;i;her . :>
o UPS: Two or more active
• Maintenar>ee Under Load: Yes
o F. ilure Rewvery : Yes

,-- .-
,J ,.;",;,;-
, • ·1

....
'-
-- - - • ..
....,.... -..
~Dirln~~.. t·---..····-······-······--..········· .CIUIj:IJt~~.J

PW ~ .. - .. :~ J .' i:~J p~
L. CIIITI~ I ...
FIIB / """ j
. ~

omo1G IICSi* DC 11.: Data CetlIfr OHIgn & BaIIIt Pnctices

ttlJdem Guide

Class F4 Electrical System 9.13

iii
It..., '. -, . ,~......
.. _ ,. ,
_ . ...... ;""'<3
t - - , '

__ - ...- .', .1

-. .- UP5
, ..:
r· -
t
,
,. .
,
.' .~

.:
- .....--- ....
CtIIk '
UPS
P. ,'
......
--~"tr""'''''t
, ...
-----....> ~
.;1 --~ , ~,","' ...,. ' . ,

PbU PIIU PIIU

L___

1·10
lICit ,~ .. ,<lp,""""'OO H,ot COH
Studonl Guido Power & Eloc1riclly

Electrical Syswm Design: Cakulating Capacity Raqul_nl$ 9.14

While setting a target reliability class will dl,tate a broad outlirn; for a
data center's electrical system, many other factors will depend
cruclalJy urx>n the size of the power load that needs to be served
• Underestimating lt1e required capacity ~n result In power
disruptions when fon:ed to later Increase capacity
• OVerestimating will lead to nigher costs

Power Componenl$ of capacity Requlremenl$ 9.15

Whi~ systems <:<In vary, the following represents a typl<:<ll breakdown

of the components the electrical sYSt.:!m wiII need to serve, along with
their respective shar-e of powe.- consumption ~

• MechanlcaVCooling 50%
• Li<}hting 3%
• UPS I Batteries n%
• Critl~1 Load 36%

Sclmple PoWef" Consumption Breakdown

Power Consumption

Lighting
stu_tGuldo

Consumption V8. Capacity 9 .16

Power eonS4.Jmptiorl isn't the only ttlirig that must be considered when
designing a data center's power system as gtowth and safety factors
must be considered also
• Capadty is the target power requirement for which the data center
is designed. To determine the capacity for a given data center one
rn L!St estimate!
Critoe" I Load
Growth Factor
Safety Factor
UPS Allowa nee
Ughllog Allow"nce
- Cooling Allowan""

Crttlcal Load 9.17

Th e O'iti",1 load is made up of aII of the IT hardware components

(servers, routers, storage devices) along with their essential support
(security, fire, and control) systems
• To begin ",Iculating t he data center's power requirements, list all of
this eq uipment along with its n"rneplate power rating a nd voltage
requ irements

-
. run ning loa cis

".2
BICSI Pra,....ry Pre!,...,. Do MIll: Copy
student Guide Po ...... & Elednl);ty

Estimatln9 ACtual Powet' Consumption 9.18

Data SlJpplied by IT hardware manufacturers ean yary. Some offer

online CIlp"City planners and detailed equipment speclncatlon • . Others
offer less. 41)0 W per RU of equipment has been a reasonably accYfS!te
historical guide. -

sample pow... Specs

Power Supply Power Power

"
' .~Max·: '} ,. Jd!e'- • Ayerag~,

Mulbpie Blade ChassIS, Fully Loaded

Watt. per 4200 2186 2700
Chassis
Watts per 520 318 406
Blade

Sample the Critical Load 9.19

If • actual' power consumption fig ures are unavailable, tile oameplate

(max power) ratings should be summllfl and multiplied by .67 to get
an awrage power rating -

Sample Power cakylalion by Nameplate rating

100 SerYers rated at 600W ~ 60000W

10 modula r dlsl< alTays at 480W- 4800W
64800W

Average Power -64800 • .67 - 43,416

Sample Critical load calculation - 43,4kW

Student Guidi _ , & a.corlcil)'

Equipment Growth facto( 9.20

Allowances for equ'pment growth wililiary per dlent; assessments of
eqUipment growth must be m~de in consultation with client IT
per-,;onnel··3 yea r hOliron is typical

Growth factor ~ % of equipment ~rowttJ over next 3 years

.-
Sample Growth Allowan"" Calculation 9.2f

Assuming an eq uipment growttJ factor of 20% oVe<" 3 years, tIlen we

can take the previOUS e~tlmate of 43.4kW, multiply that by .2 for a
f.!jlm" power allgrta !'ICe
future Power ~ Initial Estlmate ~ Growth factor: 43.4 •• 2
{o, Future Power allowance = 8. ?kW

~'F~~l 9.22

Allowances also need to be made for periodic fluctuation. in total

eq uipment power draws
• It 15 unlikely that a II of a center's servers would be operating at
their peak power draws (i.e. 33% higher than their norm) at the
same time; 'as such it is standard for the peak power allowance to
be between :;- and 10% of the InltJal critical load calculation

Sample Power Safety Allowance Calculation 9.23

To calculate the safety allow~nce, we can multiply a safety factor of

-
. OS by the init,a I load ~timate + the growth factor

•. ~ Safety factol"'" (Critical loa d + Future Allowa "ce) .. .05.

S"fety factol"'" (43.4 + 8.7) •. 05
Safety factol"'" 52.1 ... 05
Safety Factor= 2.6kW
Stud.1II Cluldo Poww & Eleotri<>1y

UPS Power Allowance 9.24

Both UPS o!>"ration and power $torage (i.e. batteries) wnsume power
that need. to be accounted for In estimating the dat<o center's capacity
requirements
• Batteries: fully charged new batteries will consume neQliglble
pewer, but batt"", charging can consume 20% of UPS Ioed
• UPS Ineffldency: The amount of power the UPS consumes while
operating 15 lIccounted for in its 'iDeffideo<l'. ",,(tag:"
- For example, a UPS system w~h a 90% effiCiency rating will
consume 10% of the critical load it 'serves

UPS Power Calculation 9.25

To calculate UPS Power requirements, multiply the initial + future

load. by the UPS Inefficiency and by a battery cI1arglng factor of 20%
UPS Power Requirement r ,,/ ( C, Joe..
+:- ,?f"'-" ~_' f')_
UPS IneffiCiency (Initi3lload + Grnwtt/t • .•10· •
Battery Charging + (Initial Load t ~wttJ) ,. .20
4' sc. . . .s :'?, /, ,1_ (~~ {LQ
Using OU"- example: ;~ ...-
UPS IneffiCiency 52.1 •. 10 - 5.2
UPS Charging 52.1 •. 20 -.llIA
UPS POwer AllowalM:<! 15.6 kW

Lighting - Lighting Power Density 9.26

Ughtlng costs for the data center are" fUnction of the area of the data
center. This is known as a Ughtlng Power Density (LPD) calculation.
LPD avera9es lighting usage as .0021<W per It' (,021SkW ~r m').
'-
• For example, if the data center was 1000 It' thenI w/ff"-
lightIng costs can
be estimated as
Lighting Allowance •• 002*1000 = ;;tkW
Cooling & Initial Thermal Load Calculations 9.27

To get a rough estimate of the power for t he cooling load, fi rst sum
the ",quirements for the Initial Crltkalload , Growth Factor, Peak
Power Allowi,"ce, UPS and Battery requirements

• Using the previous estimate we would h..ve

I

• initial Critical LOad ~ 43.4 kW

\..--, • Growtl'l Factor ~
B.7 kW
!

(,1 • Safety Factor - 2.6 kW

,>

• UPS - 15.6 kW
• Ughtioq ~
2 IsW
.
I

Power Load Requirement ~ 72.3 kW

Sample Cooling Power Requirement Calculation 9.28

The cool ing load is typically the biggest power draw for the data center
because \lirj]lally aU pf tbi' pgwer suoplied is..sonwertsc1 into heat t~
must be removed
• Cooling systems will vary in efficiency; the most efficient will
typically, require about.?5% o~ the po!;:er of the thermal load they
are .servlfl9
- Thus to get a rough estimate of the power required for the
cooling system, We can simply multiply OUr previously calculated
power load, 72.3 kw, by .75

Cooling Power Requirement - 72.3 * . 75 = 54.2 kW

' -18
SICS. Pnlprr.c.ry Property-DG Holt c.,..y
Student Gukkl

Calculating total Power Requirement 9.29

w~ can now sum the previous requirements- for the total power
reqUirement for our data ~nter:
/

( •
•
Inillal Critical Load;
GrowU1 Factor =
43.4
8.7
kW
kW
, • Safety Factor = 2.6 kW
• UPS ~ 15.6 kW
11 • Lighting ~ 2 kW
I • QJoli[lg 54.2 I5W
• Total Power Requirement - 126.5 kW

-
Power Requirement and Sizing Electlical SeMe.

The electrical service required for the data center can be calculoted
'rom the previously detennined power req u I~ent as follows,
9.30

• First, multiplv the total electrical capacity required in Kilowatts

by a building p.eal5 toad safety'faclor of 1.25 _ . . . .

• Then detennine the t hree phase AC volt<>ge of the servl~e to be

supplied

• This Is typically 480 Volts AC in the Un itee States but will

vary In other parts of the worid

• Then use the following formu I" to detennine the electrical

seniice $;2e to supply the dat<l center, ,In Amps:

• Amps ~ (kW xl000) I (Volts x 1.73*).... ·

• 'Note: 1.73 is the oonstant forthl'l!ii phase voltage

Sample Power Requirement Sizing Calculation 9.31

Using our previously detennined power requirement numbers, this

woukj yield

• 126.SkW (power t'e<:iulremerlt) • 1.25" (building rating fioctor) ~

• (158.1 * 10(0) I {480 * 1.73)
» 158,100 wattage I 830.4 va Itage

Electrical Service Si2il\g =190 Am~

 SWdont Guido

Utility Provider Plannl n9 9.32-3

With a Ulnjet class rating and service requirement in hand, one shou Id
then tum to power. provk:ter planning. Key Questions to address when
consider power service providers Include,
• Will you be on a shared service? If ~, who else Will be on the
service?
• Are there any high Impulse loads (foundries, paper, plants.
smelters) that will impact on power quality?
• 15 the 5el"\llce to the site primarily overh""d or underground 7
• What are the required capacities of the service initially and
u IIJ mate IY? How can services be expanded?
• Are diverse services av.llable? How are they routed to the site?
• What are ttle setvice voltages in the .re.7
• What are' the reqUirements for dosed transition operation, if
employed?
• What a re the drcuit proteCl!on requirements that the site must
provide?
.. What are the service construction requiremef\ts~
• What has been t he reliability and availability of the service to the
site?
• What Is the available fa ult duty at the main s\V~c h?

Redundant Entrance F eede 9.34

If redundant utility service entrances will be employed, separate

entrance feeds on oppOSite sldes of t he building are preferred

.,.,.~

- """"'"'"'-. r-

- ..
~ - '--
;-v

e:teUI- BlesI'
.
,.,
• rc. I P~ry~"ot~
 Pow... & Elo<Iriolly

utility Service & Clas....s 9.35

Utll ity Service CI_Fl Class F2 a~S5 F3 Cl8ss F4

Entnonce Siogle Feed . Single 511\9'" or Dual Dual
Feed Fee~ Feed/Different
Substations .
~ IIltJple Service. No No Not required Yes

Underg ......ndl Elttler clttler Under9roun~- Unal!i"ground-

Overhead COncrete ConCrete
Encased Encased

POWlOr into lIIe Data Center 9.36

lleyond t he power provider, the electrical entrance room (s) and

equipment need to be planned. Th is eq ui pment typically involves:
- Transformers
• Switchgear
• Generators
• Switchboards
• UPS
• PDU

..
1" lflii u1~~1
~

&tUlluIAv.. Com~ ' D

I "'"
it
!>It,t_wNI
u. 1:
h '" It.w,,l trrm ! I!Mt1 IUl ftf
UPS

GGncrrt'lOr ~ P- p'
_~Ro om
SCUdent Guide

Electrical Transformens 9.37

EJectncal t ... ns/onnen; are typically required on the site
• Transforme~ convert medium voltage to low voltage
• Transformers may either be external or Internal to the bIJildin9
• .I"«,mal tr"nsfonners are recommended for grea«,r see\Jrlty

Swlldlgear 9.38
Service entrance switchgear provides several functions :
• The interface !>etween the Utility and the site
• The distribution of Uti lity power to downstream systems
• Transfer cootrol between the Util ity t<.ed and the generator
- The switchgear should be designed for growth and the IIltlmate
loai:I expected
for the facility ~

] II: ....

+
Input Souna Trane~' 9.39

Transfers can occur within" self-contained system or vi. a ciroJit

breaker transfer pair. There are 3 famil ies of controls :
• Open Transition
• Closed TranSition/QuiCk Transfer
• Closed Transltlon/Loa<l Walk-in

BlC$'~~NGtCopy
...,
 Powor & Elootrt.1IjI

0""" T.... n&ilion T.... nsfer 9.40

Opet'l trans/lJon OCOJrs when there is gap between t~e main power
feed's failure and the second~ry feedCOtning online
• Upside:
- Requires the least amount II. ~1...tri<;~1 "'''tr!l.ls and relaying
- Typically d(;es not require the Utility's approval to deplOy
• Downside:
- Transfer results in a brief disronnect that 01te<1 causes a
mechanical system restart and reduces the U~ 's battery life

Closed Transit/on I Quick Transfer 9.41

aosed transition/Quick Transfer occurs when t11e utJllty and Qern!rator

are p'!!rallelec fur less than one second, and the paralleling tim~ is
typically thfl operating time of tli'e Automatic Transfer Switch 0IID
or
thfl b"", ker transfer
• Upside:
- Tl!ere Is no load interrulllion
, Downside;
- It ...,quire. more contmls and relaying
- The electoeal system's short drcuit duty must account for the
Utility's and the site generater'. rontributlon
- The transfer can hang up if the source. do not remain within
to!eranee

Closed Tran5ilion I load Walk:ln 9,42

--~
Th", occurs when the generator clOd Utility share the load for a period
of severa l seconds up to 1-2 minutes
• Upside :
- Less stress put on tI1e site's electrical system
- Ellmlnotes load In rush
• Downside:
- Utilities do not always allow it
- Substantial relaying and complex operations are reQuired

t ·.,
Bal ~P~IYP~ Not cap,
S.udonl Gu.de

-_.
Switchgear and CIa8MS

T,.ntfec' Conttall ClaM F1 Cia... f:3 C_Fl

9..43

e..,. F4

DodI_,.
""'*".-
.. 81,....,.,.& 1 tor .....~C'1IIcaI
IooId
DeiiIGOIod 10 UPS Oed.".t....'!
UPS poI!t UIIS_

lMechani'cal
TIWl5"r& Control
Po_

CorltIota
1 For atnett tOld

5Ingilatslaild"'GM"
DlClIelc.d 10 anti
,"d~ MechBnical
po",
81.« _ _
-.....
DHlcaIad to

TflKhltlic.1

a.d.-del,.
.....
Dedicated tD

II'IItCh1rIlc:1I

""'"
~dlim

RoI .. _ O)mmtr<a.. gr.cle Com'"tIf'd.. or

lnduatd~ , ...
CDmmtm:W. q,. .............
.rade
lttdounol9l'l"

Generators 9.44

The d~ta center ge nerator plant Is a site-controlled power system th~t

offers:
- A power supply during utility fuilu re
- A power supply during critical maintenance operation. on the main
power system
• A power supply for the entjre load of the data "",ter ·

Genen0t0f8-5izl no Conslderallons 9.45

The following are the crucial a.>nsiderotions in generMJ>r slzlng:

• What is the · ~r scheme - dosed or open transition7
- What is the harmonic ceolllo! of the load?
- What Is the i\IlQWable voltage sag or droop for the mechanical and
IIghtjng systemsr--
• How many generators are requ ired ror the load l maintenance
notation, and for redundancy?

,,,,
SICS! p . . . reuryfl~ NotCo,'1
Generatonl - Gen_' Considerations 9.46
The following are Important general"considerlltion. For data center
generi31tors :
• How will pollution abatement be handled?
• Wf1at Is the ru n time for the system?
• liow is coord inatlon of hl9 h reslsta nee grounding with the eJectri",1
system to be handled?
• liow are UPS battery ~chargin9 loads to be handled?

Generator· Starting Sys_ 9A7

The m~ comnnon genenltor problem is iI failure to start
• larger generator. tend to halle multiple starters based on the size
of the engine, but the multiple starters offer nothing as far as
redundancy - they simply get the engine turned o\ler more quickly

Generator Fuel System 9.48

Poor fuel quality Is a leading cause of system Interruptlol1 during

extended generatcr runs
• Fuel quality can be managed in three separate ways:
- fuel additives
- fuel treatments
- fuel filters on the generators

Genemtor Cooling System 9.49

There must be a dedicated generator cooling system in place to ensure

that the gene",tor windings and engine bloclc. water jacket remain
with in the manufacturer-speCified temperature tanges

Generator Mounting 9.50

Generators offer the compounded issue or large live loads that vibrate
while operating
- Substantial foundations are nequired for any generator system,
and th is should f)e coupled with some form of vibration Isolation
 Studlnt Guidi P"",.r & elodrl.i1y

Gen.."tor Conlrols 9.51

111e generator Cllntrol module should possess all controls and metering
for the individ ual machine, and not require any outside connections for
its o~ratiOn
• Unless multiple generators an: being operated In panillel, generator
controls for 'individuaJ machines should be completely contained on
the genenltor skid or within the generator control module
.--
--- ~ -- -. -
." General,,", & Cia __ ) 9.52
I
"

BlldI;lIp J GtIf'Ulriltvr c.... ,., Cf. . F2 CI. . '4

-...... .
"'"" ItUn 11mB 6 hours mirlNnum ~ I"IOl.Irs mirtfnla.jm "12h~
"" h.....
minrmwn rri~lmuni

Ii"llne Type st:andby, C ontinU(.lU5 0' Prime.

Continuous., Or CQrIUnuo-,.s, or Pr1me
P'rimlll
11'IIItII11ftII:Wt Qt,J1door 01 Indoor OuIllClOt"..bf Itdour ,,,tlOOt In<!oo<

Bearlng~ No ... """...., ",

........... ", 10. 'Of ,0.

~.numpt;on ,m 2m 1m
.Imo ""
FU.eI FlItw . ,.... ~illrd SbDndard
...""" p,tarimt

Co ....... Ontxlerd gert~tor Onboillrd~n~ Onbolrd \Mbor!td

01 oentralrzed II genenrtor Of ~encralQr 01
r>:Ic'II te t~ urlfr-MIzed ;( cllf1tr~i:ed Ir
psralleted p!lraliNd
Stud.nII Guld. Power & Eleobio;ty

Unlnterruptlble PowerSupp/y (UPS) 9.S3

The LIPS system malnhllns a continuous SlJpply of power to equipment

when uWity power 15 not available. The main design and
conside ... tions for UPS systems:
• UPS power maps to Ihe pow..r distJibution systems
• Provides automatk, single-step, response to a failure
• Umits failures to the system that broke
• Can carry the critical load during all input power failure

"""

•
UPB: Responses 10 Failure 9.54-15
UPS failures should automatically heal without riskin9 the load
• failu,;, response should allow Ihe UPS $Ystem to qul<;kly settle into
• steady state
• The response should accomplish one of the following with a slnr!e
.....,.,feT _ation:
- fall the UPS system to ~other UPS system, or
- fall to .n unconditioned bypass source
• failures should !Ie limited to the portion of the power chain that
experienced the failure
- In Ihe case of a .d lstrlbuted power system, the I nlerconnection of.
the systems should not allow failure in one system to C<luse a
failure in the supporting systems
- A U~lIty provider power failure should result in the generator
starting • nd the facility being transferred to the generator
- The ability to carry vltalloads during the power tr""sfer"
necessary for any data center design

DC 'i1o: Dwte c • .,....- ~I'l & ..... Pracecel

student Gajde

UPS Types 9.56

There ~re 3 main types of UPS systems; Alltomatlc, Rotary, Hybrid

• Automatic: ups •. conV~M:.POllorer to DC power, storos DC pow..-
(by way of a battery 0 flywh~nd then converts DC power oock
toAe ~J
- (,ower ef!lj;Jew;y than Rotary or Hybrid
- Lower 'nltiaS ceet
,

---
~,'I\'\oool·.'"

....'-=.....=---'-----1~-

0-,

Rotary UPS 9.57

Rotary UPS- -uses a battery powered motor·generator set to tide

through brief power interruptions, power goes to critiC<l l loads by
means of a generator drillen by a motor. Uke automatic systems, DC
power may be stored In flywheels and batteries.
• His.h Effic"""",
• High Fault Clearing
~

• ~!!der
_.... and -,...
Heavier
...

-..".-
III.... '
~r ~,.

A~~ ~=~~'-L-----i~r-~=f--r-~~:J

-.
N'
lIleSI Pr9.p~f~ HCltCoP1
student Gukle

Hybrid UPS 9.58

Provides two conditional pat/1s to power I()l!d, stored energy is

provided via flywheel/batteries coupled to a gener.>tor
• Can provide for two independent power soun:es·the batteries are
·only utilizec if both paths of power tall -
• High Fault Clea,;~g
• High Efficiency
.• . .
• Higher Cost

......,"" _ .".......

UPS Power StoI"ll9": Flywheel va. Battery 9.59

Batteries can typieally supply the crmealload for up to 15 minutes

wh~e flywheels are often closer to. f ifteen seconds .-

( f/ywhe:r ·';105: [., .... ,,'- s.,1 ~I--i'_ •

._ ._ -
• Require less maintenance than battery systems
• Do not require the controlled temperatu!!' e~vironm<!nts that
batteries do - --
• A flywheel will take up less space than a UPS battery system
• A battery systems shou ld have spill controVneutraUzatlon while.
flywheels need ~
Flywheel COns:
- If the generator does not start on the first time, 15 seconds may
not be ehOug h time to power a second start

..zr
~ PrcprilWy ProP."lP.t:t'lll"l"·-<00
.. "",I Copr
 Student Gulae Po...,.. & Electrioity

UPS dlslrtbution system

UPS dl'lribution system addresses four areas :

• Inputs
• BVp·sses
• Interconnects
• Load synthroniz<IIJon

... UPS Di~tribU(lon

So""....,

UPS Inputs 9.61

Paralleled module Inputs are to be fed from t he same substation/
transmnner/ distribution point
Distributed modules may be fed from different upstream points "5 long
as there is some form of load svnchronlzatlon downstream
P.1o r ~tId

~tx.htd

".'• •A•
.."
"•
-..,
,•0
,
•
•0 0

M
0
0

•• -I . " ••• • 1
I-I POU , .
"

"••
,\
•"
N
•
,
0
-,
,•
0

•
•
N
•
"•
M
O.
•
••
0,

N - I ,, -~ ~

J-l .. Olin

<aUI BlCSr
Studont Gukl.

UPS Bypasses 9.62

Automatic Bypasses:
• Dedican.d automatic swItch Inputs are mandatory for all Class F3
aod F4 systems and for paralleled ~ystem contml cabinets;
automalJc Inputs may be combined with rectifier Inputs for Class Fl
and Class F2
Maintenance Bypasses:
• Maintenance bypas.ses are opbanal for dass F1 and Class F2 and
.mandatPl'Y for all.. f3 and Class F4; when used they shall have ..
Q:edlgl'fd keEl-

Mainlenance Bypass
u
T
1
R
S
W
- - - - - + PDU
I A I

iu~
L N 1
I
T
Y
s
F

.."•
C
Ii

"•
A
It ~

l R
~ Siotit 8yl>O"
- - - - J

UPS Sizing 9.63

UPS salng is linked to power requirements, dass ratin9, and system
desIgn
• lnitial sIZIng Is based on fully meeting the critical power load with
the fewest modules or pathways available during maintenance or
fal lure modes of operolion
Othe r sizing factors to consider
- Load Growth
- Safety Factor

UPS Under-slzlng 9.64

Whi le scalability is a primary goal of Data Center design, under loading

can lead to system inst ability for la~e scale data centers
- A UPS should be operated at no less than 20% of its capadty

CllI10 Bleil"
student Guide

UPS Sizing & Safety Factors '.65

It is inadvisable to load any electrical system to its full capacity; best
practi~ always allows for a safety factjlT with the UPS

• A ".fety factor of at least .250/0 of capacity should be employed;

-
90% of capacity is reccmmended
- for example, if a 200kW load is being served, the UPS should be
sized to serve at least a 210kW load, and preferably a 220kW
load
 Poww & EI..mcily

\ UPS SY$- & Classes'> 9.86

UPS 5ya..,. CI.u '1 C. . . . . 2 CI . . . F3 a .......

- ......... ...,;
MiIIlllUnlnc. eypu. Byptcs pOIll'et takeR ByPSSS"powe:r E5y~ power- T.... frorn 0
fr1Im IJtiIIt\' r.ecs te~.. fhJ';' utI/!V
. _ ""'" ""'LtV
"teet! pOwen.l by
alternate SD~
VOIt.te~ 12Q120BV ~ to 141O/108V ~p to t20/ 208V up to 12aJ208\1 ~ m
ar 1440 WA
....
lOads 0( H'IO WA • IOIld5 I~ Of 1440 tvA loads of 1440

POu.t...r .. U
.....
oISOV

No
lor ~rgB" , .OOVfor_
""do;
He
, 4S<J'WI rot .....-ger
,
y••
kvA • ..aOV for
larger leadS"
Yes
ceJIIIpIb!r' eqIIIprna ..
",ad _
'10 flO y.. y"

,...__
S","cru"onlmtlan

..........-
Il..tu. .nt UPS .Al./tomiltlc UPS /'dom~or
_ryUf>S,.
~e ..
R.DtzI'1J', AutoI11.IC
...-..-.
R(l1;Bry, i!!" .
""
UPS I)q ~M'.~
tI~1041 .,.~
tram t;ompd\W
~o
A.otaltDg MoG set

...
(:onwrt.1tr1i Hvtwld IJIIS.
V.,
-.uipn.n\:

UPS a_"es & ClasH8 9.67

1IPS . . u.i . . ca. .. !F1 Cia" Fi o_n (:1. .sf4

NDnltDrlnt ~t1Dn r ... 6t COpadtv only .. Ca_' .t Cllptll:ttv at 1[1d111Jdllll caRs,

UPS UPS tndlvk:lUill DiIII&:

'os
....
Mantt9,. .
~rt.ilC8

OrMI~
with
No No

~d ves
Yo'

Yes
. " , . pM'

MIftiImumlull
rttt:MNM ""
5 mlJM!:!i ~O mlftVte. 10 minutes, 10f1IIin~e:~, (:OOI'(!lnated ..nth

.....
Joed~~bV roordlnat1:!d with
UPS topoqv
UPS wlXllogy

II:IDUntJ,.

---tv
W~p~d"'"

"frOm .... ,
..w.or
ubi,*,
No

No
"'-
"""" '"
'es
y~
~p.n~

Yet

'IS
a- ..<b
Ye,

,.
SWIitCl~r room

"OI'I~g uPS seH- UPS ...• UPS $elf· CentralfzD!;S autottlMed

SV...... ftlOftltlJr lng monitoring mOf\it»ring .sv stem
 _Guido

.- - -
,- Distribution Switchboards & C188Be8 9.68

DlI'lMlilutton c,-_n c.... F2 0 ... "

Cklnf.&
SWltdllKlrards

lA<OtkIn Indoor Qr IndQDt Indoor ~r

:~ lJtCk'orfI f
CClftJCNetij~ S\l,litd'lb OE!lr~ Of' S\llllftd)~QI SWttchbaard or Swtuh5JllJlllr
Swi(dlic:!r s.. itetlgeot """"''Ie,,,

-
_100,
cant ....
..iUl~m"~c: or
0",..,.,
SI"'Ule
"t>ft)mari~ ~r
~UI

51ngteor
redumlant
or.rwoot only

Rtd\I,..C~
D r;nr,l~ "(Jrfy

Redunda nt

Pow.r Distribution Unib! 9.69

Power distribution units (I'OU) ,Ire where the IT equipment u~lizat""

voltoges are deve loped ~nd the IT equipment loads are directly served

They typical ly consist of output panels protecllng b",nch circuits and

may also possess'

• An autllmatic switch for management of poly-corded IT po~r loads

• Dual power inputs depending upon the type of UPS system being
served
• Larger than stondard wiring gutters
 Siudent GlAd.

Power Distribution System 9.711

The term. 'PDU·. can a Iso be used to pick out tlle remot" power pa nels
(RPP) tt1at are typically at the end of equipment rows, as well as the
cabinet power strips. While these latter two are part of the critical
load serving power dlstri butlon system, they a re distinct from the PDU.

PDUa & Automatic Transfer Switch... 9.71

The reliability a n he deli"ered power ca n be improved by using two

PDUs each protected by a separate UPS and L!Sing an automatic
transfer switch
• In the event of the failure of one UPS system, the automatic
monsfer switch can qu ickly switch power without affecting
downstrea m eq uipment .

"DU POU

PDUs & Classes 9.72

PDUs CI"ss F1 Cis" f2 Ch.lIS F3 OusF4

lnpvta SIngle Single or Single or Sing'" or

Dual Dual Dual
Redundancy Type I Type 2 Type:Z or 3 Type 3

.... alter Automatic Automatic Aummatic Automatic

tyP"

m:I10 BICSi"
....
IIlesI Ptopliltlltyf',..." hMot Co.py
_.tGutde P _ & Eloclriclly

P....... Cabin & Dlstrlbulion to tha Computllr Room 9.73

Distribution to the loads may be overhead or under-floor

• PVC-coate<! flexible cable assemblies are common, although some
Jurisd ictions requ ire a hard conduit
• 30Amp/Single Phase feeds to cablnet'base<! plug strips are
common. To ac.:ommodate future pow.... requirements, installation
of three-phase / 60 amp capable power cables are recommended.
 P.,.,.r & EIOdJ1clty

Power Cabling Bnt Practices 9,14·76

The following are power cabling best practices ,

• TwI5t-~k receptades ~nd plug>: for all under noor or overhead

receptacles are require"
• Loding receptacles are requlre<l f()l' all plug strips and larger rr
equipment
- For In-mck 10ilds using straight- b~de cords, employ anti-pullout
!\ll.downs on Itle cord plug strip (onnecbons -
• Power cables to racks should onginate from @2 diffe .... nt PDUs Or
RPPs; power cables from dllferent PDUs and RPPs should have
different colors or markings
• Branch circu~ breakers should be SM 250/0 higher than connected
IT load. ~

• FacUitJes with dual P<lWer feeds shou ld employ equipment wit" dual
power cords: -----
•
--
Single-corded equipment, and equipment wi\l1 multJple CCrdS that
are not fault tolerant, should be plugged into a cabinet power
distribution unit or fed by an automatic transfer switch (STS)
• l·corded eq.uipment should have each cord fed by a different-----
source, with no more than one of the cords fed by an ~utomatic
transfer switch - -
• All recepta~s and gower stnes shall be labeled with the PDU/RPP
and'Circuit breaker number -
• All POU/RPP circuit breakers shall be I~beled with the name of the
cabinet/rack ~r grtd CO<!rdinates of the equipment that it supports
.. Every compute.- room, entrance room, access provider room and j

service provider room circuit should be labeled at the receptacle

with the POU and circuit breaker number
• Abandoned power cables not terminated or marked for future use
shall be removed
• Power cords should be secured to the bot:!:Qm of the rack
- ,- -
• Power cable management is recommended fon::!asses F1 'lOd F2
and r~:,ire<l fo~ Classes F3 and F4
5todem Guide P _ & ElaC1ritlly

Power Strips! Plug Strips 9.77

The following are requirements for power strips:

• Power strips (plug Strips) shall be installed where thermogr21phic
tes~n9 ,ecuipmenl can observe all critiCal connections

• The plug strip' shall never be placed undemeath a raised floor

-
• Ea<:h ou~et should be wired Independently, should not have a
power switch or Inter nal surge suppressor, and shall be r21ted for
60% of
- -- . -thUl~m breaker and wiring system
• Power Strips shall be mech,mically attached to the cabinet interior
• Power Strips shall be labeled and organlzed to not interfere with
""twork ca blfng •

Surge Prot8Ction Devices 9.78

Surge protection Devices offer protection "9Bins!:

• Lightning
• Transient Voltage

For increasJng dass levels, surge protection becomes more prevalent

throughout the power system; for lower class levels, surge
suppression Is often located on the Utility entrance only
• For higher classes, surge suppression may also be found at:
- lJIjllty service entrances
- Generator busses
- UPS in puts
- UPS outputs
- UPS power d;stribution switchboards
- POUs and critical power distribution panels ,

...
e lCl1 PWlprietaty flropetty--Oo Not CoP'J'
8t~GLlld. Pow... & eleelri.iIy

Surge Suppre8sion & ClaslIH 9.79

-~

"MiS CIiIKFl elM' PZ ct.aU a .....

"'1tNI1t ~ulre d V.. Ves

..
R.squh~lI:j

Utlity I!f:Itrllnce onlY' Ir reqUired

f<Ir lightning
OnIV ff required for
jlghtnlng protection
, V..
IlroCOct:'ion
G.n4I .....
UPS Input ...
No N.
No
No
V"
Veo

'"
... ,,,
UPS_

......
"'IIS/Powef No too
--
Yes Yes

""'~""

E..-gancy Power Off (EPO) Systems 9.80

Emergency Power-Off systEms are typically required by local
jurisdictions for use by safety personnel a nd firefighters
• These systems pose one of the greatest rlsk5 to electrical system
uptime In most data centers
- Accidental EPO·activations onur via employee sabota~e,
employee mistake, and during routine maintenance
L, -'\ +....,,)-1;
EPO Best Practices ---\ 9.8t

EPO system best pJ<lctices include :

• EPO systems for higher classes should be three-stage systems,
w~ Off, Test, and Armed modes of operations

• EPO activation stations should require th""" steps to operate or

should reqUire the simultaneous 'operation of two or more buttons
• EPO activation stations should offer a pre-alarm function, such as
an audible and visible ala rm when a button cover Is lifted
• EPO activations shovld be on a Ijme delay, if allowed by local
jurisdiction
• Security cameras should be installed at EPO stations so that the
face of the operator of the EPO can be clearly seen
• EPO stations should not be located near light switches, phones or
any other device that Is routinely touched
• EPO systEms should be isolated from the lire alarm systEm to
prevent accidental activation

DC 110: 0... e.m" Design I. ..... Pr~

SlUM.1 Guide p.,...... & EleGiriclty

EPO SyStems & Classes 9.12

EPO Systems Class Fl Class F2 Class F3 Class F4

. Sing Ie Sta~e Yes No No No
3 st~~e No Yes Yes Yes
CCTVon ·EPO
.'
No Y~s Yes' Yes
Stations
2"" Zone fine No No Yes Yes
ala rm system
activation

Power to the Mecl>anlcal Syshlma 9.83

There are two main consJd.".a~ons for t ~e electTical system support of

the mechanical system:
• Capacity for quick restart of the cooling system (from the cooling
plant to the ventilalion systems)
• The redlJ ndaney and diversity of ttle eJectri"", paths matches the
recundancy of the system being served

Cooling System Restart V8. IT Equipment Restart 9.84

After an electri",,1 system failure, cooling system restart often requires
more time tIlan IT system restart
• The poten~al for the IT load coming online before the cooling
system needs to be kept ill mind as heat levels can Increase rapidly
• Mec:hanical system controls vital to t he operation of the cooling

-
. -
system shall be pl"ted on lJPS...power
OJ iIIe" w"ter will often stay cool for long enough to bridge the
moling system restart time
As such, the condenser water pump and ventilation systems are
more vital to keep on bac kup/con~nuous I!0wer.
SIudont Guido

Temperature Controls 9.85

Temperature controls need to be maintained during normal,

maintenance, .arld IIOwer failure mode. of OIlerotions
• Where redundant temperature and cooling controls are provK!e<l,
redundant drcults should be provided commerlSurate with the
diversity of the (Ontrol system

Mechanical Power & Classes 9.86

CIa •• f2 aoos .. CBI5F4

FeeM .Slngle .S.ngle Possibfy Du,,~ LIFcfJfy' ON]

_In
NDne ManualOf'" Maooalor

-- --,
AUl1)m~ti( Auromat:ic

"'.pped ro MiIIPPllc·1],
Medlanici!ll Syl!itam
51""'" s,.tEm Red_""",
Iterl\l~!]ty Redundi!lrg

Grounding 9.61

Grounding and bonding are essential for the .. fety and perform"",e of
electrical power systems, IT equipment, and surge protection devices
• Data center grounding add.",;.",,:
- Building Grounding
- Power System Grounding
- The Mesh Bonding Networ1</Computer Room Grou nding System
- personal grounding and automatic discharge

. ....
I ICIA P'rvpriltw-y PI1:lPf"'tY-gq "DI C(lW
Stuunt GukN. " .....," Electricity

Building Grounding-The Ground Ring 9.88

A building ground electrllde ring shall be installed for all facilities. The
ground ring shoulo-tre a m ltiomum 'of '4 A~ ~:ZS-m-m~ifie Copper

--
wire buried 1m (3 ft) deep' and !.!!1Jn£)frolll tf1e ~I.
- .....
• Grllund rods, typically 19mm x 3 III ('!Io In x 10 It) cop;>erdad steel,
should be connected to the ring and spaced every 6 to 12 III (20 to
40 It) along-the ""ri_e< gmu",Hoop
• The ground ring should be bonded to building steel at least at every
-
other..building colu Illn -- " - - -- -

Building GroundingoConcrated Encaaed Elactrod.... 9.119

Concrete encased electrodes shall be used to supplement the

9rounding system
• Concrete encased electrodes shall be encased by at !east 51 mm (2
in) of concrete, located within and near the bottom of a concrete
foundation or fooUng that is in dinect
......
~
contact
_. with the earth ~

• Concrete encased el~es shall be at least 6.1 m (20 tt) or bare

copper conductor no smaller than 2Smm' ( 4 AWG), or
Olliei'
ccnductive meta' rods at le,,<;to 12.7mm (.5 in) thick
• Concrete encased elE>ctrodes shall be bonded III any other
9 rounding electrode system at the site

The Building Grounding Syatam 9.90

• The building grounding sv<;toem ~e directjy bonded to .11 major

power distribution eqUipment, Including switchgear, generators, UPS,
etc., as well es to the telecommunications systems .nd surge
protection system

• The fadllty shall possess a Data Center Master Gi"01J nd Bus

(DCMG6), Wlitm! all of the '"rye-load feeder grounds terminate
$1ud*rrt Guide Power & EkKltrieity

Data Cenler Grounding System 9.91

The Equipment Grounding SY.jtem (EGS) renders lt1e ground reference

for the-tllfrent carrying portions of
the power system
• The EGS receives exp~ grounds from the derived system of
!,~otheanjc \1onds in the power dlstributlOi1 e<1ulprnent and has a
large, sing le connection to the Data Center Master Ground Bus
(DCMG5)
TIle aea~_ ~!ounding Systgrn (CGS) renders U1e ground for the non·
current ca rrying portion of the data center (cabinets, etc.)
• Ii. major portion of CGS is the Mesh 50nding Networi<, wh leh creates
an equiPotential ground the fcrculhpot¢1 i QOlfi

Equipment Ground BU8 9.92

TIle key grounding connection for the EqUipment ground bu. is the
exothermic bond on the PDU directly upstream of the loads in the data
center
• All lo.<ls in the critical environment will be provided with dedicated
and e.press ground wires from the load to the derived ground- point
at the PDU

Equipment Ground 8lK

100 I'D<! Uj"

b:::;
~
::::

tsolilted Ground 8~

The Clean Grounding SYl'1IIm and Meah Bonding Nelwortt 9.93-94

The Mesh flooding Network is the heart of the Clean Grounding

System (CGS) and COi1sists of .. copper conductor g~1d tMat covers the
en Hre computer room
• W~en a raised floor is used, tile Mesh Bonding Network requires
that the flOOring system also be electrically continuous
- This may be IICcomplished via the ground wire grid system, •
H3t copper strip system, or Via the ral$ed floor system itself
•

b¢ l~O : D.at8 C4IIl'It8I"llUign II Be. ~

 SCud.nI Guide ........ & ElK1rIcI.y

The Mesh Bonding Network 9.95

The Mesh Bonding Network has three !"''Poses:

• Reduce electrical noises and enhance the reliability of signal
6'iinsfer acms inre.-connected ~ems of IT equipment
• Prevent circuit damage by providing a low imped'D~""'oUi#ective
ground reference for comP<Jter room ell!ftiical equipment
• Minimize damage to .clrcults and equipment-when a power system
Q(Dl..IOd- fatlLt event occurs

Mash Bonding Natwort< Best Practl""" 9.96-97

Best practices for the Mesh Bonding Network include:

• The Mesh Bonding NetworJ< should be prefabricated out of #26
gauge (.4 mm) x 2 Inch (Slmm) wide copper strip with all crossing
inter-connections welded, not punched
• If round conductors are used, they should be at lea.t #6 AWG;
bare copper is preferred
• The grid spacing of the Mesh Bonding Network shou ld be .6 m a
tt). Adjacent rolls of the Mesh Bonding Network·are exothermically
welded in the field to fonn a continuous grid.
• The copper stripS ca n be welded into a grid pattern prior to
installation and tl1en rolled out onto the floor In sections
• When a wire Mesh Bonding Network Is constructed using the r.al&e<1
floor pedestals thE! "oorlng system must be ~ctricalty continuouL
- Every fourth pedestal should be. bonded
. -
- The bonding jum~er from the pedestal to the Mesh Bonging
-
Network should be no greater t han 2 tt (.6 m) In length

€l20,. Blear
 student G'-do

Recommended Groundin" Connections 9.98

CDftJll!Cdon Minimum I

- ...--- CanGudg.r Si.

OCMG& BuIldIng: MaIn Groul"ld 8uss .250 MCM bare
· 0

EGS ~ DCMGS 0250 MCM~'"

~. - .

CGS - OCMGB IM'2SO MCM bate

POu - DCMG& Pel" N[( 250 - 121
HVA' Equipmrent Mesk Bondi!!l Network ._ --_...116 AWG bilre
Stell!!!l Co..!~~~: Mesh Bonding Ne~wortt .4AwGba~
Reb"r M.illlFQurldonon . Mt:$h 6011<1:11"1, J4AWGbim!
Network
Cilble M~l1qeme"t traY. conduit #6 AWG bare
Raise<! floor PedE!'stctl- Mesh 8ondilli: 'f6 AWG ~re
N@rtWotk
Sprinkler Piping rvh~.$h &oncfiog Netwon IIf'6AWGaarto
t-lVAC OuC1Work Mpsh Bonding Netwotk /16 AWG bOlire
cabjF'l~b - Mesh Bondillll! Netw.£~t .. . ~_~~Gbilre
Equipment .End~u ...e:; - M~h 8cndlns ,tI6 AWG bale
r--~ehllort .
" ..mes • M~sh Bondi",! Netwot~ N6 AWG bare
Othe" Ml!'ta)]jc; Syst f!rt1 EndD~Llres - Mesh #6 A.WG b"re
Sct'ldlna Network

Data Cenler Grounding Schematic 9.99

~ IH: iAllid$ CIt "oL1IOe

I~f!o oJeJoIl (~~ (TI~ I'
051~~~

,- - - ,'- -_. - , - ,- - - .- - - .- -- -- .- ._ ._ -- -_ .-
- ,

f .'lttBI b:drd other ClQUlpmr.m wtll" the nllIIIIlhJuld 1*

boIoa1l1:t II"C. .tiQutllJllent G~IU .~ .

ezo-tIBtCSf'
.~)
ete_ "l"OfMlII*'1P~.....o& McotClJpf
Studont Oukte P..... & EI ••irfc!ty

Rack/Cabinet Grounding kl tha Mesh Bonding Network 9.100-1

Best practjces for rack/cabinet ground ing Include :

-
• All non·current·canylng metallic endosures shall be grounded
• Ea", equ ipment cabinet/rack reQuires Its own ground ing connecUon
to the Me." Bonding Network
• A minimum of a #6 AWG insulated ~tranded copper conductor
mechanically tenmlnated to the Mesh EIondlng Network and cabilu!t
via a machine .screw, bolt connection or spot weld, Is recommendtld
• Grounds.t opposite ends of a series of racks shou ld be attached
with the framework grounding conductor gOing directly to.cnest>

-
bonding network
• Bare metal·to·meta l contact i. mandatory for all bonded
connectkms1 with antj-oxidaot applied at the connection pOint

RackICabinet Grounding Connec;1ione 9.102

Each cabinet or raele should have a suitable connection pOint to wh ich

the rad< framework grounding conductor can be booded
• Options fur this connection point are:
- Rack ground bus
- A dedicated copper ground bar or copper stl1p to the rack (the
mounUng "rew<: shall be th ...... d·forming not .elf-tapplng or
sheet metal screws)
- Direct connection to the rack
- Bond ing to the rack (two·hole lugs are recommended)

IT Equipment Bonding to Ihe

-
Cabinet or Mesh Bonding Network 9.103

The IT equipment ch...sls ~ou l d be bonded to the rac~ using one of

the follow ing methods:
, Manufactu .... r provided groundinQlocat!£!l

-
• Grounding Via tna mounting system
• Grounding via the power cord_
allldontGUd. _ & Eloc:frIcWy

Personal Grounding and Automatic Discharge 9.104

Automa~c discharge \Yr.ist stra~s are typical In most IT manufactu""rs'

Insta lIa~on 9 uidelines
• Wrist strap ports sI10uld be attacned to the rock to ensure electrical
continuity to ground
• Pedestrian automatic discharge mat. may be required for certain
raised Hoor envimnmen'"

Grounding & Cbos","" 9,105

c._. . .
"_ ....
CiMl6ftdlna
RQiSWr.oe
S oiVll lohm lohm loflm

EleCCl'k:aI Labeling and Signage 9.1ot1

ECJuj~ment labeling should possess al l Critical infonn"tion concerning

the sy.tem to which It Is affixed. This Information should indude:
• Equipment nomenclature and designation (e.g., "Generator AH54')
• System capaCity rating in kVA and kW (e.g., 750 kVA (675 kW)
• Input voltage, phasing and connectJon {e.g., 480V. 3-pi1ase. 3-
wi",,}
• Output voltage, phasing and connection (e.g ., 480V, 3·phase, 3-
wire)
• Power factor (e.9 .• 0 .9 la991 ny)
• System or switchboard serving this piece of equipment
• The system, sw~chboan:! oiload that is being served by this
ecu1pment
5t.don! GUll»

Electrical labeling Signage ElUIRlpie 9.107

Un interruptible Power Supply

System 1A
750 kVA/575 oW
-480 V , .3 Pha&e. 3 W tre InJ)ut
480 V . 3 Phase. 3- Wire Output
50 Hz • .9 pf

Served By Substation 1A, t::ifcu it BtelIIikM 5

located in R~m 8-121

SeNlng Critical Power DistributiOn Switchboard 1A

loeptad In Room 8-141

~tO 11tS,.
.... DC ,10: ~ c ...... DMlGII , BMI PI"BI:Ikle&
1I1eSI A'Op."tttlty p~ Hot e..,
 Stodotrt Guide __ & Eloctric~y

Review Questions

1. What are the 3 essential requirements to secure an electrical

System class rating?

;< , What are the four components of power usage that can be uoed to
detennine electric,at service sizing for the computer room?

-'v'rs
_ ~" 1- , '-"' \
I
\t"v\
_ \t·LI--.'\
~ . I)
_ \-";,, Il()'

3. What are the 3 main types of UPS systems and d1eir relative
merit.?

e:20tO DiCit"
...,
. ltSIP, .....y,../'tIIMI~ . . C""
stucMnt Gukie HVAC & Coolil1g

Mod uIe 10: Data Center Cooling

Module Objectives 1D.1

This module will ~xamine {lata cent~ cooling systems; after
completing thiS module you should be able to:
• Estimate computer room cooling requirements
• Id~ntify the components of a typical computer room cooling system
• Oescribe the relative merits of overhead, fioor, and liquid rack-
oriented cooling architectures
S1udent Guida ttY AC & Cooling

Date Centar Cooling 10.2

Though some clata center support spaces (such as the battery room)
have special air condltJoning requirements, the milin forus of cooling
planning in data center design Is the computer room
, Modern computer rooms are typically not designed to provide
human comfort
- High densityrr equipmef.1t produces extreme heat loads that put
the equipment at risk tOt'" fi>ilure
- Tempet"atures, air velodties, and noise levels often exceed
human c:omfort levels

Cooli n9 & Integrated Design 10.3

Proper dE!$ign of data center cooling systems requires coordination of
power, cabling, ..nd eqUipment layout planning
• Power requirements largely determine the mollng reqUirement.
• Gabll ng, whether above Or below the floor, needs to be don e "" as
not to restrict cooling delivery or heat removal
• Equipment layout, both within the noom (I.e. hot/cold aisle
arrangement) aod within a cabinet. n<!ed. to be done with cool air
d<:lJver:v foremost in mind

Cooling System Go_I Conslderallons 10.4

Severa I factors Impact the cooling needs of the computer room,

indud ing:
• Size of the com puler room
• DenSity of Equipment
• Number and capacity of HVAC units
• Rellablilty/clas. requirements
• Reom location relative to the exterior of build ing
• Ceiling heig hI
• Absence or presence of raised floor
• LQC<!I d imate

tOo,
BICSI PropMIary p~ _COpy
 HYAC & Cooling

Design Ing a Data Center Cooling System 10.5

The steps for designing. a clata center cooling system Include:

• Understi!nding tile computer room thermal taf1Jets
• Estimating the thermal load
• Choosing" cooling system arch Itectu re

Computer Room Thermal TBrgets to.6

Precise control of temperature in the computer room is best ach """,d
by maintaining conditions at the equipment air Inlet within the
following gUidelines; -

- Room Temperature: 59·90 o F (15-32°C); ,.11,- ·,'< (

~ ~,(20 -2S°C) ~ommendect
r. ~''il b .",
- Max. Temp. RiIte of Change : gOF (SoC) per hour
- Relative Humidity: ¢%-80% "' fI •• , ..bi ~
40 - 55% recommended
- Maximum dew-point: ~Ff17'C
o These conditions are stated at sea level
• NOTE: Operation in the "allowable" (vs. "recommertded") range ~
only acceptable for short periods oftime. Prolonged periods of
time outside of the recommended ranges will shorten equipment
life. A S -H Nl'- .E:
Thermal Guld.llnes and HoUCoId Air Mixing 10.7

Note that for data centers, both equipment exit air, as well as the cold
.ir supply are often outside the therma l recommenda~on5

- Equipment exit air conditions are very often dose to 100°F (38'C)
and 20'1'0 RH. and the CRAC supplied air Is often between SSc6SOF
(13·l8°C) and close to 90'1'0 RH
• To achieve 6S • 77°F (ZO-2S°C) at the Inlet, altstream mixing must
OCCur in the proper proportion. and the resultant ~ir distributed
uniformly along the server ''lei<

GatO BlCSI"
 mAC & C<>allng

Tem"... ....... Differentials 10.8

Thi! air temperature r ise

through servers is not constant, sinoo most
servers use variable speed 'ans to balance temperature, but 30' F is
common
• Computer Room AC units typically will operate with a varying
temperature rise based on load· ·a typical CRAC unit temperature
difference Is 15 • 20' F

, ".
; """
'"
"' "
,","-
I

---1 n
J&6 l"
, , , ,,

Special Considerations of Data Center Cooling DesIgn 10,9

Computer room ventHatlon and cooling design differs from general

HVAC design in that conditions at t .... eq ..ip..... nt air I n let (i. e. the
front of the eqUipment racks and cabinets) are the e!!,mary focus·
• Temperature sensors that a're furnished with eRAC units usually are -
Ioea red at the AC air inlet a nd are no! representative of the
tem~rature polnt of interest -
.

~
1?-....
p j.

"';:.-
:-
- ....
_Guido HVAC & Cooling

EstlmaUng Ih& C4mput... Room ~nnalload 10.10

QnCfl the ~rget l11ermal conditions for the computer room are
understood, the ne><t ste~ in designing the cooling system Is the
calculabon of the thermal load for the computer room
• The heat output of the computer room, assuming there are no
'outsicle' faCing walls or windows, depends on the thermal output of
its components, including:
- The IT equipment
Ughting Equipment
- Power Distribution Equipment

ComputerRcom lh!fmal Brtakdown

• PlJU,~
• Uililil\! 1%
· nEquipmeot 90%

Air Conditioning Equipment's Thennalload 10.11

• While AC units will produce excess heat, thiS is typically accounted for
In their sizing and does not create an extra thermal load that needs to
be conslclered In calculating data cent'lr cooling needs

1'1<1
etesl 'MpfJatMy Property-Do Nol CDpy
 Stullfent Guide HVAC & COC>Ilng

Calcu 18IIng IT Equipment Cooling Requirements 10.12

computer room IT equipment creates ltle biggest ltlenm.lload, as

nearly all of the power It consumes is converted to heat that need. tD
be removed
• One ex""ptiQQto this rule Is Voice .OVer In\emet Protocol (VOIP)
Routers which only convert app",xim~tely 70% of consumed power
into heat
• For the rest of the IT eq ulpmen!;
-
- If
- _-
IT Pewer..Consumption
the
~
__
IT Therma
- . l Output
..
IT po_r load of the computer room is 100kW, then all
100 kW needs to be counted as the IT contribution to the
thennal load

lighting Heat Output 10.13

CIIk:ulating the thermal output Qf the lighting system requires knowing

the floor area in sq ua re feet or meters'
• Ughtlng thermal output in watts - 2 • Floor area It') or (21.53 •
m') 1. ~.:..><~f ~ /f.-f .....
• If our data center was 2000 sCI It,. then the thennalload of the
I ighting system would be
- 2· 2000 square feet ~ 4000W - 4 kW

PDU Thermal Output 10.14

The thenma l output of the power diStribution system elemen~ (PDU.

RPP. power strip) requires knowi ng the total IT load they are selYlng
and the rateel power of the data center
__--1\7' PDU Thermal output; ( .01 • Rated Power) + (.02 * IT Load)
- ASsuming a facility was rated 200 I<W, and the current IT load
was 1 OOkW. the PDU therma I contribution woulel be calculated
as
- (.01 * 200 kW) + (.02 * 100 kW) ~ 4 kW

DC 110: bIil c:.rur Dn$g~ " IIett ~

_ntGuide INAC & CooOing

Cooling Requlremenl$ Sum ....tlon 16.15

To determine the total computer coolin9 ~uirements for the

(omputer room, you now can simply sum the total IT, PDU, and
Lighting caIOJlation.:

• IT Load 100 kW
PDU 4kW

t Ug~tlDg
Tota l

Thennal Unit Conversions

HW
108 kW

10.16

Outside of the U.S., the cooling capadty of equipment is typically

measured In kW. In the U.S., British Thermal Units and Cooling Tons
al"!! more oommon. Kilowatt estlmat6 can be converted to these
measures by way of the following oonversion factors:
• 1 wmt - 3.41 tHUs per hour
• 1 watt - .000283 cooling tons
• 108 kW - 108,000" 3.41 - 362,280 BTUs
• 108 kW - 108,000 * .000283 - 30 . ~ CQ9ljng ~

Choosing a Cooling Archllecture 10.17

Once the COOling targets and requirements for the computer room are
determined, decisions then need to be made "boot how the cooling will
be delivered to the equipment
• The baSiC types of computer room ·cooling anchi\ecrures:
Qyeme.,,;!, air distribution
UnderHoor air distribution
Rack-oriented liquid cooling systems
- ;;0"

,..,
8ICSI PnI~1". Pt(:l~ rt« CoJl'
Studlnt Guidi ItVAC & CooJlri9

OWrhead /IJr Distribution 10.18

Overhead dlstribu~on Is sometimes dlosen when no raised Hoor Is

~-~ ~
• OVemead distribution typically limits the choice of HVAC
equipment, but allows cool air to be delivered more directly to the
cool aisle
• OVemead air ducts should deliver air 10 tile cold aisle Oflly

----- Lt
·"S..i~ l"dAJI
,

, I
\1 •• J
,
•
•·•.
-- -
- I

· j

1'"
BICSI P'I1IIprlMatyP~O Mot Cow
 tll/AC & Cooling

ov.rtIead Air: Spadal Considerallons 10.19-20

Spedal considerations fOl' overhead air Indude:

• Ovemead air systems should always be used in conjunction with
hot/cold aisle raCK configuration
• Particular attention must be given to retum air path \'111M the use of
overhead air dIStribution
• Enough space betWeEn the tops of racKs and ce; ling must be left to
ensure an area through which warm air can rerum to the retum-air
inlets, which should be as high as practical
• Overhead systems create mOl'e difficulties for redirecting air flow
when equipment is ,..,Iocated
• Overhead ductwork must be closely coordinated with lighting,
sprinklers, and power or network cabling where these are overhead
• Overhead ducts wider tIlan 48 inches (1.2 m) will require sprinkler
heads to be located below the ductwork
• Diffuser outlet air velodty should be 800-1000 CFM
• Vertical displacemEnt should be considered in rooms with multiple
rows of racks, allowing the heat plume to <i.e in the ..me direction
~5 the. ventilation

• Horlwntal displacement systems will be limited by the rna x imu m

ccaling densny and the physical CCnfl9UriOtion of the room

Computer Room Air Handle", & Overhead Air 10.21

Central air handlers (CRAHLare p""fe~ over CRAC units when a

d.,!!.d;ptI "'[>#31)' ur ""e"'5d air is.'",~. CRIIC units do not offer the
designer the flexibility with fan selectio D tu ... 8 td, tl,e -dtict

-
performance.
• Central air handlers also allo... the use of air- side economizers for
improved effiCiency when the phYSical conflguratlon of the building
and location are ravorable

ave .......d Air & Ratum AJr Paths 10.22

II ceiling height of ~t least 10 It pm) above the raised acceS5 Hoar will
typica lly allow for an effective 110t air area abOve racks and cabinet<
and optimize the return air path

• Ceilil'l!ls higher than 10 It (3m) are recommended lor data

centers w"illil1igh clensity server areas

1$.0
BQI Ptofrilta" ~ Not Cc:I~
Underfloor Air DIstribution 10.23

The mo:;t commo!l..ilir cooling design for data centers involves cool air
delivery under a raised floor through perforate< tile with self conta Ined
Computer Room Air ConditiOning (CRAQ units
-
• This sy.tem allows for maximum flexibility of cool air delivery
th rough modification of perforated tile placement

Raised Floor Cooling Solutions 10.24

Underl100r air distribution can provide many advantages, including;

• Allows flexibility in 1000000tion of load to CRAC unit
• Imposes fewer limits on locating CRAe units In the space
• Piping services may be co"c:ealed below the raised floor
• More compatible with gravity conderlSate drainage frem cooling
coi Is and h urn Idlf\ers
• No overhead supply ductwork to obstruct the return air path. or to
interfere with lighting, sprinkler heads, or overhead power/cable
distribution sy5tems
• Permits the use of nearly any cooling technology, regardl..ss 01 air
supply/return configuration
• The use of Individual self-contained mom coolers makes under-floor
cooling design more easily compatible with redundant (i.e. fault
tolerant) mechanical system design

Und&rfloor Cooling; Retum Air Path 10.2$

The retum air p~th is typically 'through the room' i" tflis type of
system with many CRACs h avin g retum air duct extensiOns built over
tflem

,.
, ~ ~ ~ .
, f-l- fL ' ':' - ~ ..J... .'
iJ;,
I'; ~ :. •
I'
rr
~

c- . r :..; c-
H- ,
~

,,
- C-
I ..
I: '
. I-

r- T_
c-'- I- I- ' - ,

eaGtllal"
1"1.
~C:l 1 PropMta'J PrOlNl~Do tfo4; COPt
Studa., Gu,d<> HVAC & COOling

The ~piM<l Floor 10.26

Floor height should be selected based on the combined needs for

airflow, power distribution, networ1< / communications cabling, and
chilled water distribution
• Raised Hoor heigi'rts greater than ~6 Inches (914 mm) Introduce
additional cost to the 1100rlng system and do not sjgfJl.fkantly
enhance the uniformity of air dlstribu~cn

Underfloor CoolirllJ Special Concems 10.27

Special considerations when undenHoor cooling is employed i ndude:

• Underfloor air plenums need to be ~ept clean and relatJvely free of
obstruction In the cool aisles.
• Unifunn air pressure "",n be difficult to obtain undemeath a raised
floor .md hence cool air distribution may not be unifonn across the
room
• Ceol air distribution within a partie\Jl.r rack may be uneven as the
higher equipment In the rack can be subject to recjrculaijpn of hot
air from above if the ceiling height is insufficient

..I. ......------- Hot Alr Retum

...

,.

10·n
. IC$I ............. i PrtJp.1ty-Do Hot COp,'
_Guld. HYAC & Cooling

Floor Cooling Distribution Coneistooncy 10.28

The type of air delivery mettlod should be consJstent throughout the

- -
• Cable penetrations into the botttlm of racks should be n lled
-- --
com puter room - do not mi~ perforated til ... with al r supply gri lies

• Chi lied air should always be delivered into the cold aisle in front of
the cabinets. rather than from beneath the racle since:
- Some of the air will pass out throug h t he back of the rack
- Air supplied directly Into the bottom of a cabin·e.t may be cooler
tho n manufactu...... recommendations

CRACICRAH General Guldeli_ 10.29

Computer room air conditioning/handling unit best practices include :

• Computer Room Air Conditioners (CRACs) and Computer Room Air

Handlers (CRAHs) should support dual power sources to ensute
continuous operation if one power l:i5lJiCE IS lost -
• Each computer room should h"ve at least !li1L
redund!.nt
CRAC/CRAH
• Do not set CRAC/CRAH set points lower than necessary to keep c,J.
temperature a t the equ ipment injake In the 20 to 25 degrees C (!itS
to ~egrees fJ range
• Do ri/,'t
set CRAC/CAAH higher (or lower) than requ ired to keep
rejative humidity between ~O% and 55%; set all CRACS/CRAH5 to
the same humidity settings and use a 5% dead-band on humidity
set points
• In data centers with raised f loors. CRAC units should be supported
~endE!l1tJy so they do not tran,mit vibrations to the raised hoor
system

CRAC Placement 10.30

CRAC units shou ld be located in the hot aisle path when the return air
path is the free .pace in the room

• If CAACs/CRAHs are not fu lIy ducted for both air iota ke and
dgharge, they shOUld be arranged p~ndiOJta·i Itl equipment
rows
• Return air Intake ducts fur CRACs/CRAHs should be placed as high
up in the ceiling as possible and be ,\!Igned with hot aisles

10.12
ltell PRllft'liNr'/ P,.ftY-{)o NDIt Copy
Studont Guido HVAC & Cooling

Chilled Water Systems 10.~1

Systems using chillers as the primary cooling source are generally

more energy effic~nt than self· contained CRAC units
• The water piping systems should have two paths so that all
CRACs/CRAHs will continue functioninglHlierels a break. in one
pipe
• The most effective way to accomplish 'dual path" supply to any air
handling unit Is to use a looped piping systw

~1~1I(k.
~t!rw'I"~d;

C<tn1""'~1 ~ AC unit

Rack.()rlented Uquid Cooling Architectures 10.32

The maximum rack thermal load that can be air cooled is likely in the

• -
vicinity 0115· 25 k~ per
rack
A 42U rack Stilt"'' ' WIth
. 42 1U servers would oper.te at
approximately 16.8 kW (.4*42) average power and 24 kW at peak
power
• A 42U rack can currently hold 60 blade servers and operate 24 kW
average power and 32 kW peak power
• GiVEn that cunrent technologies are already pushi og the I imits of ai r
<coUng, rack oriented liquid cooling systems may be conSidered for
data centers geared toward especially high and dense thermal
loads

"-1'
BICSI ~Pn:lP'WtY'-1)Q ~",C4py
Sludalll Golde HVAC & CoOling

Rack Oriented Liquid Cooling 10.33

-.5'1
-
The heat carrying capacity of water is ~...!Imes greate.- than ai r
The heat transfer capability of water Is 2 to·) orderS of magnltlJde
greater than air

Supplemenlalliq uid Rack Cooling 10.34

R"" k~ with self-contained liquid cooling ~ystems can be used In

conjunc~on with row/room air cooling technologies

....,"'
F_
EqulprTlI!In!

GRAC

••
R.etvmAif
---.:.: ""'"

Liquid Cooling Racks to.35

Altematlvely, a fully liquid-cooled racK-oriented HVAC ard1ltecture can
be employed

,...
 HVAC '& Cooling

Uquld Cooling a nd Piping 10.36

Even If a data center is nol currently planning to employ rack-orlMted

I~uid ~oolln9, under floor d.illed water piping capacrty should be
ccnsidened for most contemporary data centers as future heat
densities may require rack orientlld solutions

_W,W".... '
... , ..... t ... .,.

... - ...
... ...
... ...

Rack...,rlentad Liquid Cooling: Special Considerations 10,37

Special con.id....alions when employiflg rack-orieflted liquid cooling

at<;hitectu"," indude ;
• Rack cooling architectures tend to beJ!'ss effident as one rack's
excess cooling capacity cannot be used by Its neig hbors
• Both the Initial and total life cost of rack orfented cooling solutiOns
tend to be higher than standard cooling solutions

---
.. 'Given that rack ortent ed solutions involve more coolin9 equipment
maintenance and repairs tend to be coSlller and more freQuent
• Individual rack cooling .",its can constitute a single point of failure,
and supplemental row oriented CRAC units may be required fo'f-
-
j

reliability/redundancy

1,,"U
sal f"rc~ P..,...,t)'-Oo Nt.! Ccpy
 HVAC & Cooling

Ventilation 10.38
Ventilation is air su pplied to Or removed from a S9<'CE! for the purpose
of CO/'ltrolling air contaminant levels, humidity, and temperature
• Ventilated air is typically interpreted as t he portion of the SlJpply air
that Is "fresh" outdoor air lt1at has not been re-drl:ulated Or
translerred
• Human 0= pa ncy In data centers i. typically low; however,
remova l of internally generated pollutants needs to be oonsidered
• GR-3029-CORE recommends 0.25 air manges per hour for removal
of Internally generated contaiiiTlllrnts
- ThiS equals standard code reqU irements when tile total ceiling
height + raised floor height ~ 12 feet (3.7 m)
- It exceeds code requirements when this dimension is greater
than 12 f1: (3.7m)

Computer Room Pressurization 10.39

Press<J '" differential with respect to surrounding areas
A po-.ltlve
should be maintained
• The typical recommended range of computer room pressure
differential to adJacencle.s Is O.OIl to 0.05 indles (3 - 12. Pal

Almarne Contam Inants 10.40

Particulates in the air degrade computer operations
• The most common ~rces of particulate conlamination In the
computer room - cardboard and storage of paJ?!',!' should be limited
or eliminated
• Maintaining pOSitive pressure In the data center with ~pect to
adJ;ocent spaces will aid In redud n; parti,ulates and humid/dry air

Air F1ltr.Jllon 10.41

The air handling unit supplying outdoor air .hould be equ ipped with
filters to at least MERV 13 (ASHRAE 80 • 90%)
• !f th is Isn't pOSSible, the air supply should be filterec to this leve l
before It Is supplied to the computer room

1~tI
BICst ,~Pf'hbn't P~"01 CtI .. ,.
Student Gold& HV AC & Cooling

OUtside Air & Air FlltnUlon 10.42

Air handling units supplying outdoor air should be equipped with
filtered low-leakage frames to limit bY\lilSS air to less t!1an 1%
• lncreasing the filtration level at the .Ir handlers has only a
rna rginal benefit compared to tile additional erergy expended by
the fan systems
• The standard Hlters fumlshed with packaged compUter room air
condltl<>ning equipment are eilher 20% or .30% effic~ncy. Higher
efficiency filters at CRAC units will not Significantly improve air
quality, and will result in higher energy costs

Balt9ry Room ..: Special VenUlatlon Requln!tnents 10.43

Battery rooms are tvplcally required by local code to have a dedi<:ated

ex /Jaust system to limit hydrogen g.s concenlnltlon to 1 %
• Conservative HVAC enginee.... olten design the ixlttery room

-
exhaust system for 2 cfm/sf (0.61 m' per mlnutelm')
• Redundallt exhaust fans should be providoq, along with an
iiftennate soorq of mal<eup air in the event of venti lotion system
r.ulure
• Hydrogen detectl<>n systems should be provided, and monitored by
the bulldln§"""aGIOffldtion/man.gement system

cable Man~gement & Cooling 10.44

cable management needs to be planned with cooling requirements In
mind ;
• The cold aisle plenum space should rema in as llnobstructed by
cable"lrays as possible
• When overhead callie systems are used placement and grouping of
cable should be planned to fIllnimlze !he alfeEts """ ,em" 1ft'.

--
• D..ta cables under the raised access floor should run in the bot aislE
• Cab'l es shall not be lett lIbandoned.<mder raised noor, In overhead
cable trays, or 'aboye suspended ceilings. Inactive cables should be
removed or termloated on at least QFIo6..e.o1J and marked "'for future
use" ,

10'"
BICS! Pl'Clprillbl')'PYOf*'tY-lla ~~ Cqpy
 S1vcIent Gukle HIIAC & CoolifIg

HVAC Reliabitity & Redundancy 10,45

The rndundancy of a dat<l center's cool ing system needs to match the
target dass requirements
• Electl'lCal power for critical HVACequipmenl should be on generator
backup
• Data <lOnters with high density loads should Include provisions tD
keep equipment fmm overheating during generator start-up, sudh
as "ilving the fans powered by the UPS
• The lil/AC.centrlll system should have backup powerfeeds
• There shou ld be two Independent sourceS of w"ter for HVAC
systems that employ liquid coo ling
• Every facility should have a I!2rtab!eHVAC unit and an adequate
number of.)!!me fans..in storage to provide temporary cooling to
ec
critical uopmen!

Supplemental Cooling & Redundancy

Supplemental cooling systems may be req uired to "chieve IS rget class

redundancy and otten include: .
• Spot CDoling systems
• Generator/UPS feeds for supplemental cooling systems

Aoo....iId<
SUpploe:met1ti;ll
COOling Fan

ilmIU'I u:St'
10-11
SICSllI'tep,,",pY '«:IP'I~Ilj) Hot eo",
 tfVAC 8. Cooing

Water Chillers & Redu ndancy 10.47

Chilled water pumps, chillers, cooling towers, and condenser pumps

-
should support dual, power sources to ensure continuous operaUon
.-
• There should be one redundant chilled water pump, chiller,
cooling tower, and condenser per system

• Dual power to these items must be accomplished in the design

of the electrical power distribution svstem as none of this
equipment is ,"vililable with the equivalent of 'dual cords"

0:2." BICit tiMe

OC 111: 1»1:1 c.ntet Deligl'l 6. BHt Pt.flCliCe&
.
a.c. Pr......., p~ No;rt I;opy
ShJ_Guldo HVAG & Cooling

Review Questions

1. Wh~t are ine advantages and disadvantages to o.erilead CDoling

delivery?

2. What ere some of the main concerns wilh underffoor cool ing
architectures?

3. How can ciass requir>!d redundancy be achieved for the cooling

system?

@2lnOSlcsr
$N<»m Guld.

Module 11: Energy Efficlency-Green Dala Center Design

Module Goals 11.1

In this module we will look at ways to Incre~se data center energy

efficiency ; alter completing this module you should be able to:

• Identify the benefits associated green data center design

• Calculate lon9 term savings asscoated w ith energy efficient data
center design
• IdentlfV steps to increase data ce<1ter energy efficiency

U-1
IJCIf PnlPJWaryP~Do ~ot CaPf
Energy SfflclQncy & Green Data Cantllr DesIgn 11.2

Data center desigrl has traditiorlally focused on uptime and

redundancy with little concern for energy efficiency; however,
increasing energy costs and environmental concerns are beginnin9 to
demand more energy efficient desilln solutions
• Green data center design Involves focusing on ways to reduce
energy consumption without sacrificing system reliability

Gree" EnvironmentlGnen Money 11.3

D~;gning a mo", energy effiCient data center can!

• Reduce total cost of ownership over the life of the data center
• Protect aga Inst the potential future costs of government mandatee
energY-Efficiency requirements

Electrical Power Usage & Total Cost of Own81"8hip 11.4

Even thoug h pow .... cosIs olten exceed equipment cost over the life of
a data. center, such costs have rarely been <t des:;gn c:nterion~ F~ctors
I nft uencing ttl is include :
• Electrical usage costs occur later in the life of the data center and
are rarely linked to inlHal design decisions
• Electrical usage costs are often viewed as fixed arid inevitable
• Electrical costs a.... often not part of the responSlbility of the data
center operating grou p
• Data center electrical costs are often part of a "'rg~r electrical bill
that simply .Iurnps the all of a company's electrical ccsts together
• Decision makers are rarely provided sufficient uptront infonnation
regarding long-tenn electrica! cost consoquences

D21110 1BiCSi"
Sludo .... Gulclo Green Olta c.t'Qr D.s~n

Designing Gras .. 11.S

Green tI~ta center design attempts to denne the ways th~t in ~ial
design decisions affect power consumption over the life of the data
center-

Estimating Total P _ Con .. umptlon Costs 11.6

While electrical power costs will vary by region, rurrently $.12 per kW
hr provides a reasonable estimate in the U.S.
• This trcmsl~tes to approximately $1000 per vear for each I<W of
load
• $400 a year to power a server
• $400 a year to cool a server
• Cun-ently, energy prices ~re t rend ing upwand

Estimating Total Power com from IT Load 11.7

We saw ear1ler that .4 k.W is a reasonable esUmate for a single server

and that for each kW of IT load there Is typically an equal amount of
powe r required for cooling
• So for a data center with 200 servers-we can conservatively
estimate the is·year l ifetime power usage cost by:
first .multiplying the number of servers by ,4 and then by $1000
(200 ".4) * 1000 - Yearlv IT pow« consumption c....t 480,000
Then doubling this to account for cooling system power
80,000 * 2 - Yeany TotallT Power co.ots - $160,000
Fi nally multiplying by the expected lifetime of the d~ center
160,000 * 15 - Lifetime Total IT Power Costs - $<1,400,000

" . DC ItO: DMII ca" ....

B leel P'IDIII'ftCa,,~ Not C:QPY

DHliiI'lI ..... Ptadk::w
Sleps lor Improving Data tenter Efficiency 11.B

There are many steps one can lake to reduce the lifetime ~wer costs
associiltl:Cl with data centers. These Include:
• Equipment retirement planning
• Implementing server vlrtualization
• Designing ~dent lighting systems
• Right-sizing UPS systems
• Employing server power management
• Employing outside air and water economizers
• Improving under-flOOr air delivery efficiency
• USing supplemental enollng

Power Savings & IT EqUipment Retlrement 11.9

Many data centers employ older servers for archival or resear<;h

purposes; however, newer IT equipment is typically signir",ant(y more
energy efficient
• Older IT equipment (prior to 2004) typically consumed power at a
coo,tant level independent of computationa l load
• Newer IT equipment can rI:CIuce power coosumption when
computa~on81 load is low

- Migrating to newer equipment can ofb!n result in power

consumption savings of 20-25%
o A physical inventory and retirement plan shou Id be In place
to enSure equipment power consumptIon savings

" ..
I4C:M ,.. . . . rJ~~Copy
Student Qulde

Se ....er VIr1uallz;rtlon 11.10

Tod~y's soltware platforms allow for multiple applications (and even

opera~ng systems) to be operated simultaneously from the sa",e
physjcal device
• The process of com~ni ng. software operations from two or more
servers onto a Sing le server Is known as 'server virW~lization'
• The IT load savings from elim inating a Just single server running at
.4 kW woukl be approximately $800 per year
• In 2002, 80% of servers ran at 5-15% of thel r max Imum
computatioMlload
• Some U. S. states. such as california offer tax breaks and incentive
payments for companies to employ server virtuali~tion projects

Lighttng & Ene'1lY Efficiency 11.11

There are 3 ways to increase energy efficiency through 119 htl n9

<:onsideratlons:
• The use of windows fcr nafUl'lIllIghtinll and heating in data center
support spaces
• The use of ftum5Cent lighting and lighting dimmers
• The use of occupancy sensors to tvm off nghts in unoo<:upied rooms

Improving Efficiency through lIIe Raised Floor 11.12

There are 4 ways to improve power efIi<:iency through ra lsed floor

1m pl1lvements:
• The use 01 raised flooring heating and cooling throughout the data
center. not just in the computer room
• Comoct sizing and placement of vented noor tlles
• Closing off wiring vents In the ",ised f loor
• Removing underfloor obstructions

"..
IUCSI PtofWletarypr~ Hol¢QPY
_010.01 .. G.- DaIa Conlet Design

Floor Tiles a nd Cooling Efflciency 11.13

In many data cenrers ventec1 tUes are orren misplaced ~nd too many
a re Installed
• If there are too many venred tiles In me data center, then cool air
pressore will be too low for cool air to reach the tops of racks

Compuboliolllli Fluid Dynamics 11.14

The uS<! of Compuu.t1onal Fluid (}ynamics (CfD) in the datacenter can
hel p optim i ze d ala center cool air flow by forecasting optimum vented
tile placement and by negulalJng the number of vents that are open at
i;I given t,me"
I Some vendors offer cooling optimization services that have
demonstrated" 25% neducljon In cooling system power usage

Rai""'" Floor Seal. 11.15

Raised floor effICiency can also be improved by:

• l'mploving rubber gaskets under each tile to minimize air leakage
• Blocking cable openings til pnellent cold air waste Into Ihe hot aisle

Clearing the Raised Floor Plenum 11.16

A last consideration fllr improving underfloor air delivery efficiency Is

moving e><C5sive power 'lnd data cabling from beneath the raised
ncor ~nea to above me racks to provide" clearer path for cool air
c1elivery

ClO10 BICSf
....
Brest ~ry'f'CIPIIIC)'--DO tb: Copy
S"dontGulde

Righl-5lz1ng UPS Power S)'8~m8 11.17

.Right-sizing the UPS power system to the load will often have one of
the. largest Impacts on increasing the electncal system's elfrdency
• There are fixed 10S5eS In UPS power systems that are present
whether the rr load Is being served or not
• These losses are proportional to the over.11 power "'ling of the
system
• A UPS system runs at highest eff"'''ncy when it Is near Its
maximum rated capacity-as load level drops, so does etr",iency

Modular UPS S)'8tems 11.18

One path to ' right-sizing' while preservio9 scalability Is the

employment of modular UPS architectures
• The use of multlpJe small UPS modules (10 kVA to 50 kVA) to make
up larger UPS systems (from 40 INA to 1,000 kVA) has become
increasing Iy common
- A primary advantage of the modu lar approach is the ability to
grow capacity
- Also, a. the modules are hot swappable, one module c;on be
repla<;e<! while the system is under load
- Modular systems also usually accept one mone module than is
reqUired for their rated ca~dty, ",ok.ing ttlem Inhe.... ntly 'N+l"
at " lower cost than a non-modular UPS system

Modlliar UPS Syatems-5peclal Concerns 11.19

One conslderation against modular UPS systems Is their typical

employment of multiple valve-regulated lead add (VRl.A) batteries
• These batteries carry certain failure risks and lifetime limitations
that can add up in replacement and m.intenance costs over time

11·7
lal Prv~1')' ProJ*tY Do- Hili Co","
1IIud.", BlOIdo

3.f'hase Voltage Power Inputs 11 .20

In most U.S. 'd ata centers, another opportunity for power efficiency
lies downstream from the uPS-!f1e UPS system typically provides
power at 4S0V, 'which is then 'stepped' down to 208V In the power
distribution system
• These step downs incur power losses that can be el lminated by
converting UPS output pawer directly to 41SV
- The 41SV three-phase output can proVide 240V sinllie-phase
input direc~y to a server eliminating step·down losses
- Most servers and IT equipment can handle 240V AC Input
witMout problem

......'
lAIWP'\,HH6

Greening the Cooling System 11.21

One Wi> Y to Increase cooling system efficiency ,md often create la rge
energy savings Is to raise the target temperature of!f1e computer
room air conditioning «(RAC) units

• In 2007, EPA studies indic:ated most data centers target the

computer room's temperature too low-by merely raising tne
computer mom temperature a few degrees, electricity usage can be
red uced by as much as 20% annually
• Computer intake air needs t o be in the 68 to 77· (20-25 0 Cr range
- there is no!f1ing to be gained In t.enms of equipment
performancelprotection by keeping tI1e cold aisle a Ir cooler than
this tempef<1ture range

mtl10BlCS"
st:ucIent Guida

Inefficiency and Air Cooling 11.22

Another opport on~y for coo ling system savings comes from
coordinating ""mputer room air rond itioners
- Air conditioners runfling corronuOlJ sly with low output temperatures
can over-dry air which other AC units will then try to re- humidify by
hea~ng it

- This creates a condit ion for a room with multiple air conditioners
to have those air con ditioners simultaneously (and inefficiently )
heating and cooling a room at the same time
• Computer room air conditioners are rurrently available w ith
intelligent communication systems that allow for multiple units 10
coordinate room Illoling

Supplemental Cooing Units 11.23

When combined with room/ row or1ented COOling, supplemental cooling

units can redu~ cooling sy,;tem power usage costs in high density
compu te, rooms by up 10 25%
• These savings are achieved by bringing the COOliflg source closer to
the heat source, reducing the fan powe, required to del iver cooling
• A typical supplemental cooling system delivers relMgerant to the
modules t hrough an overhead piping system

Ovemead Supplemental Cooling Unl.,. 11.24

Overhead supplemental cooling units can also he1p to imprcve row-

oriented coolln9 ard1itecture efficiency by blocking .the ",turn of hot air
to the cold a l~le ov", t he tops of racks

CRAC Unl... with Variable Speed DriV85 11.25

An add itional measure that heips te> improve cool ing system effidency
is the employment of CRAC with variable speed drives
• Using va riable speed blower systems to m"lch cooling requirement.
can save signiftcant enel'1ly--a 10% reduction In air now
can result
in a 25% n:ductlon in the energy consumption

11 ...
INCSI proptttt.) ProJ*fY-UO Not CODY
Air Condltlonlnll Economizers 11.26

One last measure to improve cooll"9 effidency In colder climates is the

use of coOl extemal air to help power the data centers cooling
architecture-mechanical systems that enable environmental cooling
are known as··economlzers' and come In two types :
- Ak-slde-lt1ese us directly imported cool air from ti1e external
environ ment
• water-side-these use water chilled In an external storage tank
instead of (or In addition to) mechanically chilled water

POW<I. Management Software 11.Z7

Mlsskln critical facilities are sized for peak usage condition; ihat rarely
exist; server ma nagement software c.on help power down IT
equipment during periods when usage is low
• Pata center IT equipment usage typlClllly falls off from ~ pm to S
am, but its power consumpllon remains rela~velv high eVeil after
computation load decreases
• In Idle mode, most servers still consume around 75 percent of full
capacity power
• Most modem servers have built-In power management feature. to
reduce this idle power consumption; how~ver, these f~ab..Hes a~
often di.abled because of concerns over response time during pea k
usage hours
• Server management software can oversee and regulate a ~ervers
built-In power management features so that these feature. are only
employed during off-peale hours

1'·10
lUeSI P'rap~P1'o~~ Hot Co",
Energy Efficiency SUmmary 11.28

savings Mode ~nij.1 Energy

Y~n~Si pel' YNr

Air ConditlC>n Coordination 0 -10%

Raised Floor /J,jr Management 4-16%

EQu 1pment Retirement Planning 0 -2a%

Ene'!IY EfIldent ughting System. 1-3'%

Rignt-sizing f'OW<!r System/UPS .10-25%

Air &. W&t@r Srde t-IVAC Economi2ers 0-15%

Server Virtual lZiltl.an 10-40%

Supplemental COol ing 0 -25%

1h1t
l a l Ptut-.... , '"'~ 00 ttol CtIpJ"
StuMm Guide ~ ~ Can.... O•• lgn

Review Questions

1. Why ti~s,,'t
energy efllclen.:y tr;Idit"",~lIy been part of the Initial
~esjgn criteria?

2. WhY 'is having too mi>"Y ven~d Jiles a bed thing?

3. Wh~t do you thlnl< Is likely the best w~y to adl leve energy
effICiency In the data center?

DC ;10: 0. .. c...n.r DNign a Bett PrItCticM

student Guide

Module 12: Telecommunications & Cabling

Module Goal .. 12,1

This module will examine tlle telecommunications & cabling

components of Data Center design; after completing this module you
should be a ble to :
• Idenllfy the elements. of a typical data center's information
transport system
• Communicate tlle necessary Infor-matlon to access providers to
setup telecommunications services
• Design" cabling system topology in accordance with BleSI t>.st
practices

12.1
BICSt PrcfIrIaiMy hofN'rfY--Oo "'CIt ~
_on! Guido Telooom""mlorion. '" CaillIng

Designing Vo~ata Communications Systems 12.2

A data center's telecommunications system runs from the access
provider to the netll/orlc equipment and typically includes a service
entrance room/are<!, a main distJibution pOint, horizontal distributOon
points, equipment racks. and the networked ecuipment

--_. -
(rIIl"roU
fdlfT
iMllloo!~
";roI~mr
1,
J
"""
no,..""......
.....
..... . ""... I
f)I.~., ..I...'1

I.<p.llpmljllt
1lI\1'~TllOtI
N", '- llllO:n

~~2"1
Coordinating with Aece5s Provide .... 12.3
One of th~ earlie·s t ancl most important step. in the design ~nd
impJementatlon of a data center's telecommunications system i.
coordination with the provld~«5) . Information the design,,, will
typically need to give ta service providers includes;
• Add reS/; ot the building
• PI~ns of the entrance conduits from the property II~ to the
entrance area
• Assignment of conduits and inner ducts to the provider
• Routing ot cables in the entrance area (acceS/; floor vS. overhead)
• Quantity and type of circuits to be provided
• Locat"'" and interface for demarcation of each type of circuit
provided
• Camer office diversity required for target r~lIablllty class

,W
OIell .. -.yP~ J.loI COPt
" ............
S1udent Guidlr TCtltoommunlcatlons & CI~lng

Information Needed from Acceea ProvIders 12.4

lnfo'ma~on typically needed (rom ~ccess provider(s) Indudes:
• S~ce ~nd mounting requirements for protectors on copper'~lr
""bles
• Dimensions of provider rodeS/cabinets or s~ce requirements if
they are to use client racks/cabinets
• Power requirements for equipment, Including receptacle types
• Location of provider central offICes
• Route of cabling to the data center and the sepa ration between
routes
• DescriptiOn of pathways used to the building
• Installatloo and service schedule

From the Pnovider to the Building 12,5

Once co-ordination with the access provider has begun, deslg ning the
pathway from the provider. Into the data ce nter.",," occur
• Underground cabling is recommended
• COstomer owned satellite dish Fanns or communication towers
should be located as close as possible to the perimeter of the
fatility in a secure a rea

Entrance Palhway Types and Quanlllies 12.6

The data center should include multiple duct banks from the property
line to the center with tvstomer owned maintenance holes
• Maintenance holes sh~ II be lockable
• Duct bank should consjst of at least four 4-inch (100 mm) conduits
If more than three access providers are used, provide an
additional conduit lor each addl~onal provider

,»
SICS' • . . . . , . , . . , _ .... Co"
_Guide

Entrance Condul~ 12.7

The ~umber of entrance conduits required depends on the number of
.ccess providers and the number/type of circuits.
• The entrance pathways should have adequate capacity to handle
growth and "ddltlonal access providers
• Each access provider should have at least one 4 In (100 mm)
cood uit at each en!","ce point
• C<lndu lts used for optical fiber entrance cables should have th .....
In r>er ducts :
- Two 38 mrn (1.5 in) and ooe 25 mm (1.0 In) or
- Th....., 33 mm (1.25 in)

Redundant Entrance Path ....ys 12.8

Muttip!e "n("moe pathways from the property line to the entra nce
mom(.) eliminate a Single point of failure and are required for Class F2
fac1J itie:s and above
• The ""'Intenance holes and entrance pathways should be on
oppos;te sides of the building and at le"st 20 m (66 It) apart
- Data centers with two I!ntrnnce rooms should have at least two
entrance abl,," through diverse maintenance holes

Redundant Providers 12.9

Telecom access reliability Js best ensured by usIng diverse providers,

central offices, and mutes
• The customer should ensure that serllices are proVisIoned from
different offICes and routes
• Diversely routed pathways should be separnted by at Ieiost 20 rn
(66 It) at aU points along their routes if possible

6i610 SiCsI'
 The Entra IKe Room 12.10

A dBbl center should lIossess a service provider entrance area for

demarcation, t ermlnaUon, and provisioning eQuipment
• Diverse entrance facilities are recommended for ClaS6 F2 data
c~ters and required for Classes F3 and F4
• Diverse entrance facilities shou Id have comp~te route diversity 10
different central offices ilnd diverse service providers

11114 1 ' il l '~,I":; !t ,~ " , ~ I ;'·I;"I .. I ~ I HldII11ll I IW')~i

B ,

J
=

12.11
The entronce room/area should be outside the computer room to
improve security
• Addition ill entr. nce moms may be ""luired to support distance
limited Circuits such as T ·lIE- l and T·3/E-3

caDl0 iiiCif
 SI\Idooi Guide rolecGm>unlcotk>n& & Cabling

Redundant Entrance Rooms 12,'1:~

Multiple entrance rooms may also be installed for redundancy rather

than circuit distance reslrictions
• Access provlders shOUld install circuit provisionin9 equipment in
both entrance .rooms so that circuits of all required types can ~
provisioned from either room
• Redundant entrance areas should be in separate fire protection
zones and should net share power distribution units or etir
condl~onlng equipment

Entrance Room De,,"•...:.otion 12.13

The entrance room will have up to tour separate areas for aCC<!SS
provider demarcation Including:
• Low-speed copper-pair circuits, (05-0, ISDN BRI, and telephone
line')
• H/gh-speed.copper-pair circuits (DS-i, T-l, ISDN PR!, CEPT-l/E-
1)
• Coaxial.;,,_ circuits (DS-3/T-3 and CEPT-3fE-3)
• Optk:alllber circuits (SONET OC-x, SOH S'rn-" FDDI, F"st
Ethemet)
Access providers should provide demarcation toelheir circuits in
one of these area s (delineated by circuit type1

l.ntr.lllrt! ArC'~

L.4W-:!:pf!ecj Ill"IIgn ~peed Il(o:iIIOdI II ~ICAI

t:r20tll IIIclI,.
, ~..,

BICSI ' ..... Ii_i~r--Do~oI CcpJ

Studont Guido reilcanwnunluttons & cabling

Demarcation of Law..."eed Circuits 12.14

Access providers should be asked to provide dem~rt:ation of low·speed

circuits On IDe conned:lng hardware
• Cabling from the low· speed drcult demarcation "rea to the ma in
distribution area shoukl be terminated on 10C connecting hardware
• Each 4· pair CB ble from the entTa nee room should be lenninated in
an .I DC connect9r in t he entrance room and an elght position
modular jack outside of the ent rance room
• Dual-sided frames should be used for mounting large amounts of
IDC ","neeting hardware (3000 + pairs)

Demarclltlon of High-speed (T-l) Copper Circuits 12.15

Access provldel"S should be askec t o hand-olr T -\ circuits on RJ48X

Jacks (Individual S-posltion modular jacks with loop back)
• A OS)(-1 patch panel mounted On a <uslomer-ownec rack Installed
in the 05-1 demarcation area is preferred
• The OSX·I pat<:h panels may require power for indicator lights
- Access providers may require flick space for rectifiers to power
DSX-I patch panels

T-l Circuits and IDC C<>nnecllng Hardwa ... 12.16

Access provide~ can also hand off DS-l circuits on J DC connecting

h~rdw~re

• The 10C connecting hardware can be pl.ced on the same ,""ck for
low-speed drw its

T-l Cables 12.17

A single 4- palr cable can accommodate one T1 Ir.nsm lt and receive

pair
• When multiple T1 Signals are placed over multi-pair unohlelded
twisted -pair cable, the transmit si~n~1s should be placed in one
cable and the receive signals placec in a separate cable

. tH
'JC"~~Pr,,~IfOtC~
studonl Guido T_unlcatlon. & Cabling

Demarcation of E-3 & T -3 Clrcllltll (Coaxial Cable) 12.18

Access providers should hand-off E-) orT-3 d rcurts on pairs of female

BNC conn...:tors on a DSX-) patch "anel on a cusromer-ownec rack
installec in the E-3/T-3 demarcation area
• The DSX·] .patctJ panels may require power for Indicator lights
• Access providers may require rack space for rectifiers to p"wer
DSX·3 patch panels

E-lIT-3 Cabling 12.19

cabling from the E-3/T-] dema=tion are" ro the main distribution

area should be 734·type coaxial cuble
• Cobles in the E-3/T-3 demarcation area can be tenminated on a
customer patch panel with 75·chm BNC connect<>rs, or directly on
an access provider DSX-3 patch panel
- BNC patch panels should be oriented with the front of the patch
panels on the .. me side of the rack a. the rear of DSX-3 panels

Demarcation ,,' Optical Fiber Circuits 12..20

Ae,,"ss providers should hand-off optical fiber d,cults on fiber patrh
panels installed on tacks in the fiber demarcation area
• fiber patch panels from multiple access providers and the customer
may occupy the same mck
• The racks in tile fjber demarcation area do not require power

Optical Fiber Cabling 12.21

Cabling from the fiber demarcation area to the main distribution area
snould be sing Ie-mode optical fiber cable
• If the .entrance services are terminated in multi mod" optical fiber
cabJe, 111.. data center cabling can also include multi mod" optical
Hber cabie

DC ,11: ~ o-tan & . . . Pra.nl"""

student Guide

Cabling Pathways from the Entrance 10 the Computer Ruom 12.22

Telecommuni"'tions cabling for data centers shall not be routed

through spaces accessible to the public or t enants of the building
unless t/1e ca bles a re In endosed oontluit or other se<:u re pathways
• All pull boxes and splice bo. es shall lIave " lock

Into the Computer Room 12.23

• There are two telecommunications cabling Subsystems In the computer

room:

• The Backbone System

• The H orizonta I System

_.

DC til: o.aa c.ttar DMIgn & ..... Pmcec:.111

Backbono Cabling & the MDA '12.24

The heart of the backbone cabling system is the Main Dlstr1butjo n Area
(MDA), the centrol acteSS distribution point for the data center
• Backbone cabilng also connects tile Main DistTibution Ar"'a to the
rest of Ihe Oata Center
- The Ma in Distribution Area is typically in the computer room,
though It may be placed in Its own room for i mpreved se<:urit;'

f3a<koo~e CatJUn9 ~yst tm c..ompUl~' Rt>"fJt

H.>rizornhll
Dln,l"",nolfl. ~
H c> r Ib:I J)'IoJo

lntr.m=fbam
...
Dn.! IlootlDflo
~,

~"

....
C~m.IIIlClIItlon ~In
/\"J4
Dbmhutlon
I

."""~
1,,,,,,,,,,
RboM

The Main DIstribution Area 12.25

The main dlslribullon area typically houses:

• The Main Cross-connect
• Core roLlters
• Core LAN swltc~

• CDre SAN swilehes

• PBXjJ PBX "'1 ulpment
• M13 multiplexers

rnl:t'UllIIC3r
12'·10
lreSI ~'Yp,~ Not Copy
 T".commUftJeat~n. & elbing

Backbone Cabling Topology & the MDA 12.26

The backbone cabling shall use the hienJrchir:alflt1llr topology. where
earn horizontal cross-connect (He) in a hori:<ontai distrlbutkln area
(HDA) Is cabled directly to a main cross-connect in the main
distribution area (MOA)
• If cable spacin9 limitations require It, a horizontal cross-connect
may be connected d I'ectly to M ent,a noe area feed

Ring
I Il n.o.

"DO.

Backbone Cabllr>g Types 12.27

• Backbone cabling shall conslsl of one or more ofthe following media

types :
- l00-ohm CIltegory 3fclass C minimum (category 6/dass E or high«
recommended)
- OM 2. 5Of125 ~m multlmode optical fiber minimum (OM 3, laser
optimized, recom mended)
• 051. single mode optical ftber cable
• 7S-ohm coaxial cable (Telcordia GR- 139-COR.E, ?34-type, and 735·
type)
- If specifIC hardware requires ClIbllnQ other than these types,. it
may be installed in addition to these

'1-1,
.C~I~ryP'""ty OG"'~
student Guida T._muNe .. lons & Cabllnog

Backbone Cab1i ng Length Limitations 12.2B

Assuming an uninterrupted ",bling run, the following distance

IImlta~ons for backbone C<lbHng s/lovld be ob~rved:

• Category 3 multi-pair balanced 100 ohm backbone cabling should

be limited to a total of 100 m (328 ftl
• Category 6 balanced 100 ohm bockbone cabling should be limited
to a total of 100 m (328 ttl
- These dlstane<>s Include equipment patch cord.
• For b.l~nced twisted-pair cabling, to reduce the effect of mvltiple
connections In dose proximity, the HDA terrnimotion point should be
located at least 15 m (SO ft) away from the MDA termination

Centralized Optlc;of FIber Cabling 12.29

Many single tenant users of optical fiber implement a cMtralized

network, rather than using horizontal cross-connects
opt~1
• Centrall.ed an alternative to
fiber cabling is designed iIS
locatlng an opt/ca I cross-connection in a horizontal distribution .=
- Centralized ""bling provides connections from RacksfEDAs
dinoctiy to a ,entraliled cross-connect by the use of pull-through
cable-So, I nterconnectlonsl or splices

I;;;~;-'l 2-;i' u I
Mr..'

DistJibuled \IS.

,.... ". '

02Q10 BiCSf
1MZ
1110 ............ "'.....,.....0. wore ...
 studern Gukle TelKomrnunlatians & Cab ....

Centralized Fiber_tic Baclbone Implernflttation Guldelln" 12.30-1

Best practices for centralized fiber-optic b~cldJone ""tworts indude:

- Centralized fiber optic ca bllng Implementations shall be located in

the same building as .Racks/cabinets served
• Administration of Moves, Adds, and Changes shall be performed at
the centralized cross-connect
- Centralized cabling design shall allow for migration (in part or in
total) of the pull-through, Inten:onnect, or splice implementation to
a cross-ocl"mectlor! implementation
• Centralized fiber cabling sy>terns shall leave suffldent space to
allow for the addition of palm panels needed for any migration
• Sufficient cable SlaCK shall exist In the HDA to .Ik>w movement of
the cable~ when migrating to a cross-connection
- Optical fiber slack sha ll be stored in protective endosures
- Optical fiber ch~nnel~ shall be limited to 300 m (984 ftl, including
all patch cord.

Backbone Cabling & Redundancy 12.32

Redundant backbone cabling protects against an outage due to
da mage to one of the backbone cables, and can be provided In several
ways :
- One way to provide backbone cabling redundan,y is simply to have
two cables running fnlm the MDA, prefera~y along different routes,
to a single horizontal cross-connect
• A second way would be to have a backup data cable nunning
between two horizontal cross-connects

M,,. .DO

"".

€l2110 IiCit'
I IC81p ..
___ '2·"
, P/'6f*1)-Cc ~ot: Copy
.....
 S...-.cGuldo T.~mmunfc~n. & CalMIng

Fully Redundant Bac:ltbon& Cabling 12.33

• Th<! highest level or redundancy in terms of the backbone ca bling

system would be to have two "ccess feeds 10 a horizontal cross-
conn-ect from diverse MDAs fed by different service providers

ro!..,r'Q" Att 4lv ,~

I·.., .. ,"~ I P

F.tll'~i'I. 41
E""~""'''
"""" , "-,
Mo.o, OlD>

H l)l loWnl.'
C~Soi '~ Cln~CI
i -"""
t.J=-~-CCJn lM'!:t

To the Equipment~!I$tIIUatlon Pathways to the Racks 12.34

Data and po .. er cables sha lll:>e Installed In cable pathways-

conduits/raceways:, trays, and ladders, are recommended
• The pathways shall have 1>0 sharp edges, bum;, Dr projections that
can damage the cable Insulation
• Pathways may be installed eitt1er overhead"o, beneath a raised
floor

ibii1' Ilear
11-14
81CS. "'q~ Pt9pe..,....oo Hen COpit
 Te4ec:ommUnjcoti.,... II. Cablk>g

O\tet"Mad Cabling Pathways 12.35

Advantages of overhead cabling include:

• Cable lengtl1s are typically kept to a minimum
• Adc itiOna I cables can usually be placed with minimal difficulty
• Dee. not interfere with under-floor cool air delivery to racks
Disadvantages include;
• May pick up interference f rom Ilg htlng fixtures
• Are visible and poten bally more prone to sa botage
• May Inte<fere with sprinkler systems
• Expose cabling to higher temperatures

O.".,head Cabling Guidelines: 12.3601

Best pracbteS fur overhead cabling pathways include :

• In data centers th.t use overhead cable trays, • minimum of 300
mm (12 in) between the top of one tray to the bottom of another
tray abQve shall be provJded and maintained
• Unsh ielded Twisted Pair (UTP) cables should be kept at least
127mm (5 In) from fluorescent lights
• Overhead trays should either (a) have a solid bottom. (b) be at.
least 2.7m (9 It) above the Hoor, or (el have an altemate means of
protection from sabotage
• There should either be separate raceways or a divide.. ina racew~v
to separate copper "nd fiber optiC cable
• If copper and fiber optical cables are run together, optk:al cables
should lie on top of copper ones
• l ntra-[ad llty trunk cabling snail n ot be placed within a dedicated
optical nb... pathway with optical f,ber patch cords
• Optic,,1 fiber ca bles should never touch the fioor (or slab)
Student Guide ToIoccmmun4clll... & Cabl/nil

U nder-floor Cabling Pathways 12.38

Advantages of accesS floor cabling pathways Include :

• More easily accessed by technicians, do not require ladders for
access
• Allow for easier changes to denvery pathways
• cables are hidden from view
Disadvantages indude :
• Will interfere with under-floor cool air delivery
• Raised Hoors have a high..... i nitial cost
• Allow for haphazard placement or cable

Undar-floor Cabling Pathway Guidelines 12.39-40

All telecommun k:. t lons cabling under the rilised floor should be run in
a pathway that Is classified by a nationally-r<'Olgnized testing
laboratory
• Se""rate aisles for power and telecommunications cabling are
prefenned
- Telecommunications cabling should be In the hot ai~les

- Power cabling should be In the cold aisles

Data center<; that distribute cable below a raised floor and do not
separate data and power cables by . Isle should provide beth hori:<ontal
and vertlca I sepa ration of power and telecommunk:atioM cables
• provide horizontal <eparatlon by allocating dlffer""t rows of tilEs In
the aisles mr power and telecommunicBoons cabling
• Provide verbcal separation Ily plaCing the telecommunications
cabling in trilys as far above the power cables as poSSible, with the
top of the cable tray SO mm (2 Inches) below the bottom of the
access floor tile
• Cable tr1!ys underneath 3 rilised fioor should have a maximum
depth of 150 mm (6 in)

1M8
BlC31 'io,rre'I:WI"~Hot eo",.
 Student Guide ToI._nlo..vono & Cabling

Power & Data Cabling Separatlon Dialanc$8 12.41

The following distances should be maintained between power cables
and twisted-pair data cables (local electrical codes may require greater
separation dlmnces) :

# of Cin;uits ClKult TVpe separation Distance

l - 1S ZOA 1l0/24OV I-phase 25 mm /1 In

.shlll!lded or unshielded

16-30 lOA ll0!24OV I-phase 50mm/:2 in

shielded

31-60 20A ll0!24OV I-phase 100 mm / 4 in

shielded

61-~O lOA llO/240V I-ph.... 150mm/6in

shielded

91+ lOA ll0!240V I·ph ... 300 mm 112 in

shielded

H LOo.o 41SV 3 -phase shielded 300 mm / 12 in

Cabling Conduits & Separation Dis~nc"" 12.42

No separation distance is required when either the data cables or the

power ea bles are endosed in metallic raceways/conduitS that meet the
following requirements :
• The conduit completely encloses the cables
• The conduit is properly bonded and grounded
• The'condult's wall thickness is at least 1 mm (0.04 in) if ma<le of
steel or 1_5 mm (0.06 in) if ma<le of aluminum

D2f1tO IiIICSr
SWdontOukio Teleeommunlcattons &cabling

The Horizontal Cabling SY$ief1l 12.43

The horizontal cabling system Involves the cabling distribution to the
server l'acks and Includes ttle horizontal distribution areas (HDAs), the
~ulpment distribution areas (EDAs), and any zone distribution Sress
(ZDAs) ; the horizontal cabling system should also be designed In a
star topology

The Horizonlal Cabling System Componenta 12.44

The IiDA typically includes a horizontal cross connect, LA~, SA~, and
KVM switd1es

ll'1e Zone Distribu~on Area is an optional interconnection point

between the HDA a nd fDA to allow for frequent reconflgu ratJon and
Hexlbillty

• Hori,cntal cabling shall contain no more than one consolidation

point (CP) in the ZOA between the HDA Bnd termination in tile EDA
ll'1e EDA is the s~ce allocated for end equipment, e.g. racks & servers

Horizontal Cabling Types 12.45

Horizontal cabling should consist of one or more or the following media

types:
• 4-pai' 100-ohm bIl lanced cat"9ory 6/dass E minimum (category
6A!class EA or higher recommended)
• OM 2, SQ/125 ~m multlmode optical fiber m inimum, (OM 3, laser
optimized recommended)
• 05 1, single mode opbcal fibeT cable
• 734-type and 73S-type 75·ohm coaxial cable for E·l, E·3, and T- 3
circuits
If reQuired, other types of cabling (e.g., Infinlband copper cables, EIA-
232, etc.) may be Installed to 5upplementthe types listed

1~·1.
IIM:SI ,,~w ~I*"t-DD Nel Cop,.
Student Guidi

HoriZonllll Cable Management Guidelines 12.46-48

o Best practices for cable management in the horizontal ca bling system

indude:

o Use appropriate cabl!! roulln!! and dressing "xtures in cable

pathways to organize and manage the different cable types
o ~Iiminate cable <tress caused by tension In suspen ded ca bl!!
runs by limiting supported spans to 1.S m (5 It) or less

o Eliminate c,1ble stress caused by tlght1y Cinched ca ble bundles

by keeping jacket deformation to • minimum
o Patch cord/jumper slack is a major rontriblltOr to pOOt cable
management, CClbte stress, ~nd loose COllnec::tKlns at the .cross--
connect

• Eliminate or minimize patm cord/jumper slack after each

cross"connectlon Is completed

• Horizontal cables mu<t be terminated On connecting hardware

that meets the applicable reqUirements for the type of cabling
used

o Use patch cords or jumpers to make connections between

horizontal cables and bat;kbone cables

o Remove only as much cable jacket as is ceq uired for termination

and trimming

• For unMlsting cabling, FTl8lntaln pair twists as dose 8S possible

tei the termination point. The amount of untwisting must not
exceed 13 mm (0.5 In) ror cate~ory 5e and higher cables.

oNtll Bar
12:~19
oc "110: ow. e.,.,. ,*-.." &m PnIoJ:IIo_
ElICa. """'p,.....,~ ,..Copy
HDf1zontal Cabling System ~dundan"y 12.<19
Horizontal cabling redundancy should also be employed to preV<!nt
outages d~e to cable failure
• ~edundant horizontal cabling can be achieved either by di""~e
noutes within a single HOA, or by dual noutes from different HOAs
- The dE!llree of redundancy from diverse HOAs typically will not
offer much mo~e In terms of increased reliability if the HDAs are
in the same fire zone
- Ensure that redundant cabling does not exceed its length limits

--
HDA l HI>A

, /

- -'
--
En>
'''''

Abandoned Cable Raquirements 12.50

iIb<lndoned cabJe refe~ to installed telecommunications cable the! is
not termin&ed at both ends at a connector
• cables not terminated on at lea.t one end in ttle main distribution
area or a horizontal distribution area should be removed

Cabling Identfficatlon and Labeling 12.51

All cabling .hould have a unique Identlfl"r. Backbone cabUng should

reference the sou",e Qnd destination. Hortzontal cabling names should
reference tihe horizontal faceplate/jacks.

• AU cablin9 sh<tll ba labeled:

- With manufacturer-recommended labeling systems
- With only machine generated, self-adhesive labels
- At each end
• A cabling summary report should be available
StUdent Gukl.

Cross.connecl Labeling and Identification 12.52

Cross-connects should have a unique jden~~er, Considera~on should

t>e given to colOr code cross-connects by their Intended use or
a pplieaoon.
• For example, a patch cord between port~, patdt panel 4 on rack
ABOl and port 11, patch panel 201'1 rack AC02 shall be called:
- AB01-O-005/ AC02-S-011
• All cross-connects shall be labeled:
With fTliInufucturer-recommended labeling systems
With only machine generated, self-adhesive labels
At each end with tennination h.rdware/cabling sm.nd/pairjport
and/or equfpment port information from Ixlth side.
• A cross-connect summary repart should be available

Pathway labeling 12.53

All pathw.ys shall have it unique idenMer.

• All pathways shall be labeled at ;oil endpoints. Additional labeling
should be provided at:
- Intennedi<lte pOints such as pull boxes and Joined cable tray
segments
- Regular1y spaced intervals in dosed-loop pathways such as
cable tray rings
- Partitioned pathway~ such as duct b,,,,d or innerduct; unique
Identifiers should be provided for each segm~nt
• A pathways summary report should be available listing all
pathways, Including their type., origins, destinations, total
capacities and present fill conditions

"'"
.nj......"'f PAJPfit7-00 Not ¢op,.
SlCSI f'T'......
student Guide T,"communlu'lions & Cabling

Classes and TeII>communlcaUons 12.54

dass F2 data centers provide a redundant ao:ess providerfeed and
corn ponent ~vel redundao"l for netwonc ~qu ipment such as core
routers and sw itches

"Ft<d &uancdoom
-;

Mllol
.., • ~,..
/ "
fc.AE ""~,;­ "-'/
F.:' ''+1',':1::' .~

--
b~ll'UI::I' ~

£OAt Itadr!.

ci2fiut BiCsr DC 1ta: Dab C"II,.....- DnIgrJ ~ hit ~

&Ie .. ~~
....__ ,p..,...,......,. .... eo.,
il .1t
 TelecommunKoation, & Cabling

Class F3 12.55

Oass F3 facilities must provide redundancy at the system level,

redundant access prOViders, redundant entrance rooms, redundant
backbone cabling; the netwon. should be configured such that no
interruption in sentice will result f rom the failure of a single netwoli<
device or component

.
Primary Au.f?5
'ZFeed SC'I;Qn~r)'
(nh'~I'c. ROOfl"l t:nrrMce ROcIm
,

-
MilA

Fu II y Redundil nt
HOA Networi< oevl <es

EOA J Racks:
SludontO.-

Class F4 12.56

Class F4 data centers add a secondary main distribution area and

require that the equipment ~",a5 be served by two horizontal
distribution areas; al l single points of failure should be eliminated.

"n~
frim2lry A
En!..,nc. Room Entnn~ Room
/
~
..,
........
- .--

MOA MOA -

~........
./

HDA HDA •
- I

EDA I I\4ciu
i

~tll.ICSi"
Studonl Guide reteeommunle.ttons & Cabling

Review Questions

1. How far apart should redundant access provider feeds be?

2. What are the cost</beneHts of rIllsed noor vs. overhead rnble

di5tribution?

3. What are the benefits of a star rnbllng topology?

&015 Blei"
"'"
Bleil PrvtIIfW;IJY "~"Y-Do MDt CcP1
Module 13: Building Automation, Monitoring, & Control Systems

Module Goals 13.1

In this module we will be examining buildlnll automation, monitoring
and control systems; alter finishing this module you should be able to:

• Identify the typical systerns mon itored by data center control

operations
• Know the basic oomponents of typical data center building
automatjon, monitOring, and control systems
• Describe how autometian, monitOring, & control systems relate to
Oass reliab ility " ,nkings

u ·,
.ncII Prop~ PrOfMl'tY-Do WcIot CoPW'
Student Gujd. II<JIldlng Automation & Control 811._.
Building Automation Systems 13.2

The Building Automation System (BAS) is a collection of products

designed to optimize control and monitoring operations

• Monitored systems In data centers typically include;

- POl'/er
- Heating/ventilation/air conditioning (HVAC)
- Security and access control
- Fire alarms

The Control Room 13.3

For many dQta centers, monitoring and control functJor'ls are routed to
~ cenU(llized space or 'control room'

• The control room shall have a.::<"so to the romputer room

• The control room should possess system <:ansale. ~ri<ed over
PrIvate IP address spaLe
• Center should be equipped with large screen displays for
system/netwoli< monitoring .
• Monitoring of BAS alarm. shall be available in the control room

Control Rooms & BAS Cabling 13.4

Build Ing automation systems often require dedicated interwnnect

wjrlng between the mOl1itored system and the monitoring/control
devices

• BAS cabling can often be proprietary or non-standard; bec"use

of limitations typical with this type of cabling, the control room
sI10uld be centrally located to limit cabling distances as much as
possible

om~tOBICiii"
,.. OC1tCl'; om e.m.ro.lIIn .... Pr,IIW,","

BlCSI ,~f:ttoperty--Co NCII. C.,.".

StudontG""'"

AutomaUon and Cont...,1 51$"'" Components 13.5

• Building automation syStems (BAS) typical ly Indude the following

components:
• Input/OUtput Devices-sensor> and controls for. single
function, such 2.S a pressure meter
• Unitary Processors-Reid-level Cl>ntrois for a single su b-systern,
such as an air handling unit
• Distributed ProceHors-system-level controls for a diverse
system, such as the CO<llinl/ system
• H<Mlld End-management-level controls for mu Itiple d istri bute<!
processors, sud"! as a wark·station int!!ffa.c~
- The head end typically provides real-time control

BAS Hamwa... 13.6

All monitoring ond control hardware should be fault tolerant; it should

f.ail in the loperl' 01'" 'on' (.ondltlon, wittlout affecting the monitored
system

Due to potenttallimitatlons In BAS cabling capadt!es, sensors may

need to t>e directly wired to system-level [ontroners rather to a
hortzontal connection point or dlstrlbution .rea

I(l 1oeofI:Ii1Yj. OI!' iletd I~I

~ Clnlon:.l..,.

hI , .. ,..Iotol~ "'I r,~tJ· bwol

toi'TttAlM'$

UlO11 BlCSF
studont Guide

BAS Network Protoeol5 '13.. 7

• Most med1anical and electrical system manufacturers are now u.ing

' open" protocols for BAS rommunlcatlons, such as BACI1et, lonTalk.
5N MP, Kill K/ElB, a nd X ML

• The BAS/Conb't>1 system should confonn to an open protocol

archltectun! to insure comp<ltibility with future equipment uPQnldes
• The BAS system should support the Simple Network MMagement
Protocol (SNMP) for reporting to softwan! monitoring applications
• Conb't>1 systems usually c.an rommunlcate wlltl power and
med1an ic:a1 systems using any oflhe open protocols; it is the
owner/ operat or'. dedsion as 10 which prntocol best suits their
needs .

BAS Cabling-()ut of the Control Room 13.8

The BAS cabling system will be deployed throughout the data center
and should :
• Integ rate common services
• Accommodate d iverse. BAS appl ications
• Provide the capacity for rapid deployment of new equipment and
services
• PrOvide redundancy to satisfy safety and reliability d ess
requirements

BAS Cabling & Inbograted Design 13.9

While BAS cabj ing has often been Installed prior to voice/data cabling,
combining t he instalia~o n of the two will allow for ltle .foUowing
benefits:
• Reduction of initial construction and ongo ing maintenance co5I:s by
allowing one project team to Install and manage .all of the cabl ing
• Reduction of cabling installation hours
• Reduction of damage to finished surfaces by having all cabling
installed during in itia I cons\nlction
• Provision of future Hexl billty and easier response to ~uests for
changes
studant Guida Bulldlng AIlIomlllon & Conlrol System.

BAS Cabling Syatem Guidelines 13.10·11

BAS C~bling System best prar:tlces Include:

• BAS cabling $'(Stems fur data centers should be capable of
operation without Interruption in case of power outage. fire; and
broken cables
• BAS cab~ shall meet local FJre autnority codes to ensure the·
system can survive in case of fire
• Building automation cabling shall not share the same ·sheath with
communkattons 'cabl,ng
• D.,,;igners shall """sider ensuring BAS system survivability by
eltner
- CAl Dual cabling that takes .diverse pathways
- (6) Redundant BAS controller hardware store<1ln separate
lo~tions

• POE midspan devices shan be instal~ where til. link end (such as
the MDA or HDA) ~ides

- Power over Ethernet (PoE) Midspan devices connect and power

dQwnstre~m devices (such as a surveillance camera) to a non-
PoE enabled networl< switch
 Student G\Jide Building _00 & Control ~.

8AS Cabling Topology 13.12

Unless specltle<J otherwise by the equipment manufacturer, the BAS
cabling shall employ a hierarchical star topology

EI.~~I ROOf"! C:hll'-dMe t: h&nI COll ' Ro o m

C IF" l nA~rm

TD Indi",;dLNI e .....
S~:Iodor.l. __ =- "-. •
""A

r;c.t'I"~1 ROlOn'!

• In larger data centers, multip Ie zone"distribution areas may be

employed

P",ful'l1ld B.t.S cabling Media 13.13

Where control devices do not require specifIC proprieta ry media, use,

• Copper twisted-pair (L/TP. ScTP, STP) cat"9ory 5. or above
(cate~ry 5 or 11 igher is recommended)

• Fiber-optic Single-mode, multimode 6~ . S/l2StJm, Or multimode

SO/12StJm f iber optic (SO/12StJm is preferred)

Q2IItO .iCii" DC ttl: 0-. c.tI .... 0.-'0" & Beat Pnll:ticH
 Student: GuIde 8ulldlrlllwloma."" & ConlrOI S)<$tam.

BAS Cabling Pathways 13.1'

The BAS ca bling system may use sej:>arate pathways lor improved
safety and security
• BAS and communications cabling should always use separate
pathways whenever there is likely to be electromagnetic
Interference
• Pathways and spaces for BAS cabling should be clearly markee
• Data center BAS cabling should terminate In separate dedicated
enclosures and patch panels
• BAS endosu",", and p..tch Pilnel. shall be cI""rly marked

BAS Equipment Enclosures 13.15

BAS equipment enclosures shall be lockable

• Enclosures With electrical deVices shall allow lot proper grounding
• Wall mountabl .. enclosures shall provide for secure attachment to
the w.. lI, and should pr<>tect equipment from any harsh
environmental conditions

Monitoring & Control of Iha Power System 13.16

When addressing poW<!r system monitoring, there are three

reC<lmmended properties for observation :
• Power levels noting Itoltage, current,. and frequency
• Hannonic content
• Waveform imaging and capture

Power System Integration 13.17

The electrical system should integrate to a Single "electrical"

supervising management system; th is system may be Integrated Into
a I~~er overall monitoring and control system
• The electrical system's monitOring integrator should mask and
manage duplicate alarms ror subordinate systems through
consolidated control points, such as paralleled UPS modUles or
generators

e2:C110 Dlesl'
 Studont GuI<W

Power Syshlm COfltrois 13.18

Power system controls should follow these guidelines;

• LC>Ca1 (at the equipment) control only
• Remote monitoring always
• Control interfaces on each piece of eq ulpment should be easily
understood
• Color code equipment to denote the pa rlJcular system
• Stand~rd operating procedures should be posted on each p;ece of
equipment

IfVAC Systtm Mon Raring & Control 13.19

• GuIdelines for room temperature moo itoting are:

• Temperature measurement sensors should be regularly calibrated
• Sensors typically shoold be mounted on a nxed surface, ideally 1.5
m (Sit) aboV<! the II"",
- Temperature measuring ~ints should mimIc the equipment Inlet
cond Itlons .
• Temperature and humidity sensors fumlshed with eRA<: units are
Installed at their inlet, and do not indicate the conditions at rack
Inlets

Battery Room Monitoring 13.20

• A hydrogen detection system should be provided for the battery
room and mon~red by the building automation system

021'1 Blei"
...-
.."................
_ ...",. rrw.~ NatcbpV
Student Guide Building Au'e .... '1on & Contlol Systemo

Waler Detec;lion Alal1Tt5 13.21

Leak detection should be provided at any location where water is most

II kely to exist; the most common SOU rces of lea kage are water pipes,
II! Ives, and cooling colis In HVAC equipment
• Water detection sensors should be located under raised floor areas
to detect water leakage from piping sYstems
• Intemal water d<'lettion sensors. should be provided in computer
room air conditioning units and alanms should be reported to the
monitoring/control system

Security S}'IItem Monitoring 13.22

Clooe<l OrcuitTelevision (CCTV) should be provided to:

• To centrally "few many locations simultaneously
• Provide it visual record of monitored area during alanms and access
control evems
• Record crJmes, dvU, and operattonal offenses. for use as evidence jn
prOsecu~on and/or human resources processes

• Record monitoreel areas and employee actiVity for use a5 a defense

In civil and criminal proserution against the data center

Are Alarms & Conlrol Systems 13.23

An addressable. muttlplexed. microprocessor-based. electronically

supervised fire alarm shOuld be provided
• The smoke detection system shall be designed to automat"" IIV
conlrOl air supply and exhaust systems and shall be interfaced with
tl1e building automation system

Monitoring & Clasaes 13.24

Kaving monitor,ng systems in place is mandatory f<>r all class levels,

With inereas,og levels of observation, scope, and granularity, with the
higher class levels
Student Guido Bpildlng ~Uon & Controf Sywt:ems

Automation and Control System& ... Cia,," 13.25-6

Automati"" • Oas.Fi Oassf2 Clas. F.l CiassF4

control Sys...... s
SUmm"'Y Ala"" add y.. y"" Ves Y..
TrOooblo AI_

-
Ulility 1kI. N. No ~"" Yes
Genl!lfBtor 8.,./
... loG V"" ". Yeo
Generilt4;tn Veo Yeo V.. Yeo
Mechlinlc.r Power No · Na Yeo .y.,;.
&r-
UPS Bus No Ve. Yo. Yes

_ .. ,_ed
UPSJ4Odut_ Ye,
Ot!$i5"8ble
Ye~
Yes
Yes
V••
Ye,
Y••
h~Svstem

PDlIs No Yes Ve. Y.,

a>o_
_lie Switch ...
lltanc'h Qrcult
Dli!iI::rilxltiD ..

FlreAI.",
No
No

NO
vos
Yes
No
~es
Yes
YO!;
·V.,

Ves
Yes
Yes
y.,

Yos
Yo<
sy .... m
........rQlNoIity No UPS OLltpoJt only utillt'f, -rhroughDUt
Genem:Qf5 alld
·UPS· output
blitab ... tor No No Yes y""

"..
alannl!ii
_..,.·Q...itty No ·Yes yO<.
-.1",,1"11

........-
Utility

UPS CkI~t Bu.

No
No·
No
No
Ne>
Ve.
V..
IiQ.
YH
Y..
Veo
Yes
OC:_tH>n No No No V..

''"'.
IIIICB. PftIptieUty ~ 00...,. Copy
Student Guide Building Automation & c""""" Syatomo

Revi_ Questions

1. Why are a utomated control systems important in data <:enter

design?

2. Why are open protocols important for ""ntrQl systems?

3 . Should BAS cabling use separate cabling pathways? Why or Why

not?

130-11
bICSf PNprllOf)'h..."".....Oo WrJt CI;IP'f
student Guide Commuionlng & Malmelllnc.

Module 14: CommissIoning & Maintenance

Module Goals 14.1

In this module we will look at the commlsslon ln9 proeess; after

completln9 this module you should be able to:
• Identify the components or a data .center com missioning process
• Realll and create commissioning reports
• Describe how to formulate a maintenance plan

oc '11: DaIt Cef'IIIr tJnlllI & """ p~

Commissioning & Mallllllnance 14,2

Commission ing is the process of ensuring systems are deslg ned,

Insta lied, tested, and maintained according to the owner's deslg n
Intent and needs
• CommisSioning should occur throughout the desigf1/development
process ar'ld culminates In the formlilization of a maintenance plan
that reflects tM design's intended reliability

Commissioni"9 Phases 14,3

• The commissioning process includes the fcllowing five phase<;;
- Program
- ~19n

- Construction
- Acce ptance
- Post accepta nee

Commissioning' Program Phase 14.4

The first commiSsioniilg phase is the 'program phase"-thls

determines the 50lpe of work and systems to be ·commissioned
• Objectives In the program phil"" include;
- Final decisions mad~approved
- Project "Design Intent" Is est.blislled
- Budgets are established
- A commissi<>ning team is designated
- Th~ systems to be commissioned "nd tested are deterrr ined

~UI8iCSi'
 CommISlk)nlng & Malm:enant::11!1

Commissioning: Deeign Pha. . 14.5

The second phase ofthe commissioning process is the 'design phase'

where contracts, spedfieatlons, and system documents are completed
and the commissioning agent reviews all documents to ensure
compliance
• Objectives In the design phase Include:
- Architectural reviews are completed
- IT asset requirements are determined
- Design intent documenliiticn Is submitted/approved
- Scope of work for all partJdpants, Including contractorS, and
vendors ~s rev~ed

Commissioning' Conslroctlon Phase 14.6

During the 'construction' phase the commlssiorilng authority monitors
prog"'s. to en~ure the -design intent is being followed

• Objectives In the constructJon phase Include:

- Performance of milestone monltoling
- Performance of Pre-functional tes~ng

- Field Inspections and progress repOrting

- Monitcring of chanlle oroe, process and approval authority
- Documentatlon of any modlncallon of the desfgn Intent
SIUdo", Guido Com_.lenlog & MoInte_

Commissioning: Acceptance Ph.... 14.7

Dunng the "acceptance" phase, performance test ing takes place and
the following occurs:
• Systl!ms are calibrated
• Performance gUIdelines established during the design phase are
checked
• Nonperforming systems are fixed
Objectives in the acceptance phase Inelude :
FUnctional performance beSting
ComplelJon of site audit
ComplelJon of warranty audit
Final dorumentatlcn submittals and all test reports

Commissioning: Post.Acceptance Ph",_ 14.8

Ounng the po.t-acceptance phase :

• Operations and maintenance procedures are defined
• A prncess for documenting new eq ui pment a nd verifying the desig n
inten! should be established
Objectives in the p05l:-ac"'ptance phase inclUde:
- O&'M (Oper<loons &. Maintenance) procedures established
Document storage and modlflcatlon procedures are defined
- Perwnnel are lrai ned
- MAC (Moves, Adds &. Changes) procedu"". established
- Change control procedu res and polices implemented

Commissioning Components 14.9

The five phases of the commiSSion Ing process can be broken down
further Into II1ree cruda l components :
t Documentation
• Testing
• Creation of maintenance procedures

,....
I81CSI PruP"'rfPt~ Mol. COp';
 Commissioning & Maintenance

Documentation 14.10
Documentation is a CEntral fec>ture of the
commissioning prlXess .md
involves having foor types of records on hand:

• D~i9n DOOJment<
• PrO<;ess Documents
• VerilJcation Documents
• Ope",tion '" Maintenance Documents

Design Documents 14.11

~Ign .documents Include:

, Design fntent spedfic~tion$

• Building Drawings
• General equipment spedficatlons
• System subm~tals
• An:hl~ural requirements
• Change order(s)
• Seq uence of operation

Process Documents 1<1.12

Prnces$ documentation Includes:
t Commissioning P'im
• Reporting forms
• Settee ules and tlmellnes
• Me..ung minutes and progress ",!'orts
• Conflict and resolution records
• Start-up and testing plans and schedules
• Training program documents

....
II(" ProprlltrfPropwt)' 00 N;otCopy
51_Guido Comml..klnfng &Mafntenlnce

Verification OQcurnents 14.13

VerifICation documentation includes:

• Construction observation ",ports
• Deficiency logs and resolutions
• Test reports
• Manufacturer speclflcations
• Cootnlctor CTEdentials
• Com missloning agent reports
• Final commiSsioning report

Operation & Maintenance Documents 14.14

Operation &. maintenance documentatJon Indudes:

• Warranties
• Equipment Operation &. Maintenance (O&'M) manuals
• A~·bullts

• Control drawing.
S.udtnt Gukla CommiMl."ng & Molin_nee

On Site Doc:umentallon 14.1~

Once the data center has been commissioned the following document
sheu Id be stored ensite :
• Contract Documents
• Specffic;:ations
- Floor l'1a n Drawings
- CltJtside Plant Drawings
- Connectivity Riser Oiilgrams
- EqUipment Layout Drawing & Details
• Rack &. Cabinet Elevations
• Cilble Pathway Details
I .... As- BuiltH Drawings
• Con.tructlon Administration Reports
• Test Reports
• Punch -list Report
• Opera~en5 and Maintenance (0&",) MilnUals
• Close-out, Sign-<>ff and Atteptance CemHcates
- Certificate of Oc<:upancy
• Ch~ nge Milnilgement documentation Ind "ding system inventories
• Relocation planning documentiltioo such as eqUipment inventories
Fln,,1 Commissioning Roport
'''.17
The following should be included in the Anal Commissioning Report:
o Project name
• Name, address, firm t and phone number of Commiss4oning
Authority
o Description of the building (5i<e. Location, Cllnstructlon)
• HVAC and other il'lStalied systems
• Ust and description of commissioning !aSks
• Commissioning Plan
• Completed design intent document
• Completed pre-functiona l test cheddists
• Completed functional pruionn. ne<! tests
o Any non-rompli"nce forms
o Summary of f,nd lngs
o Recommendations for monitoring the ongoing performance
o Ree<>mmendations for system improvements

Functional T....ting 14.18

Testing Is the sewnd crudal iI.pert of the commissioning pmcess and
should follow this basic sequence:

1. Validation of the equipment specifications via. fadl)ry

demonstration/factory testing

2 . SiI.ie electri"" I construction quality control

3. Fundional f ield -testing of indiv idual com ponents

4. Functional fie ld-testing of a multi-component. individuill system

S. Complete testing of the electrica l system

6. Testing of all utility systems to examine their inte",ct;on

.U
DK:IlfI~Pro~MGt Copt
Funcllonal Testing Components 14.19
Functional testing of individual data c..nt er components should Involve
the following ,
• Equipment descMptlon
• Description of purpose of t he test
• Detail of requ ined personnel and equ ipment
• Design information pertlllent to the system under test
• Detailed sequence of operatiOns
• Special Instruc:tions or warnings
• Dennltlon of ex~ed results
• Sampling strategies

FunctiOnal Tasting Procedu",s 14.20

The proced ure for carrying out fu nctional tests should be as follows,
• Inspect equipment for manufacturing and installation defects
• Document test <DndltiOns
• Perform Integrated sy.terns test
• Verify test ~on$e
• Compare actual response to acceptance criteria

. ~
1UCS1.....,....iaryPrl)peIt)'-Do MIJt CIIpY
Studont Guido

Sample PDU Commissioning T98t Document 14.21

,. UI"PC> s.
1'0 est.!lblbh '9U!p$ nlPCe!SSiilry ~(I fini&h QOmml:n~o'f'linO tnt J:lOIJ" N'I<I periotm tI'I~ 11'1tegrl!ltN Systerrr\S
Tat in cu~ 1'001'10 "1 ..... hite .,...,lrnll:lI\g p~u. ifJIpe,ct to a1llcel cu9tOmer ~u"m~1
Ol'ervitllll
• COI'ltli!d temporal)' polll'l!lr jw.p." from M188 tit~"" 10 1MClPSOP2
• Shift crilitlilil bad <0 UPS 28 jl tM ASlS POIJ f.e'¥ei .
I Triill'lSftlt UP'S 11$"""A. to Jntemtll bws" flom UPS modI!
, Eniffijl~ A side POV!!; .. CU5CDmBr room #,
• TllIn~ UPS ~tam 1A from intftnaJ b)'Das:.ti to UPS mOCl:t

• Pow!Jr UI3' h B *I~ ~ ~tto",., room., POO/A$TS''!Io UU'IQ tempol"m1 power.

• PI!I'fonTI ASTS CcmnNs'!lkxllnl} Procedt.-e 011 STS 38

• Li!la...a'" :sir;» POUt in c.uI4oc'Mr ~ ' " on the 1A UPS

4 Shirt rntK:a1 lead In CU:5tom ... ftgom #3 and t~,"work to UPS 1A 811M AST5 POU 1e-vE!L
t Trans.ftt' UPS "ystem 2S 40 Internal bypa~ from UPS mode
• t;:1'1~ B:!!Ide :FIOl..r:! in t:usooml!f room:/11
• lraflSier UPS :!I.)llIilMn 2B fr'gm 1n6l1"lM1 b\IPI1t 10 UPS mo<Ie
• fi:Htor-ll power to normal petWtr ~
S)'s~tl~d
TIJ,e klllowJn!:l ~rns rna:.' be impac.Ud by thi~ proc:8d~tI: UPS $ySt.m 1/1... ufJ'$ ,~ :28 lOCI "II
STSPDU!i:
Elsclc out pliln
It a probll!l'Tl oeCi:JfS ~ OIl PDU or SIS durinll tMt.1~. it will t. 150IiIttd fl'Qfn Ihlr" ~~m until It i:;
1l!!Ipaae.d.

,,,ottO
elC$l ""opo~ryPI~ NolCop't
Student Guida Comfr'M ..lonlng & M.ime,.fllillOe

Data Center M~jntenanc. & Maintenance Planning 14.22

The last truei!,1step in commiSSioning a data center Involves the

form.tion of a maintenance plan, as insvffident maintenance will
result In extended downtime; a cnuclal part of delivering a data center
w~h a particular class rating is ensuring that regularly scheduled
maintenance Is performed
• Some evidence indicates data c",nt",r Int",rruptk>J'1s are more IIk",ly
to oo:u r during maintenanc", thM at any other tl me
• As such, aU maintenance activities need to be car"fully planned ,ond
Implemented

Malntananca Testing 14.23

All electrical and mechanical equipment should be initially tested after

installation <in<! periodiCally throughout the life of U,e equipment
• Testing should be performed on the equipment individu"Uy and as
an integrated system
• Fl"<'quency of testing should typically conform to manufacturer's
recommendation
• Backu p system testing should be scheduled -during off' pea k hours,
ev~n II the electrical systems are designed to keep critical loads
running during outages

Maintenance Planning 14.24

Each da~ center sub-syst<!m will have unique maintenance needs and
reqUire special maintenance planning

/ """)
...,,~

, /. •
~
I f:Oalll'lJ.
! "
~ule
 Commlulonlnu & Mointonano.

Cabling System Maintenance 14.25-6

cabling system. normally require little maintenance, provided the

system is designed and installed appropriiltely

ca ble Maintenance Best practices include:

• Annua I Inspection performed by cabling leclmicians
• Inspect/repair/replace wom, cut, pinched cabling entering cabinets
and al stress poi nts in me cable management systems
• Inspect/repair mountings of all cable management systems
o Inspect/repair congestion points within telecom switches and
serverS
• Inspect/repair floor, wall, and ceiling cable penetrations for correct
fire stop andlor cable damage
• Inspect/repa ir any loose cable that coul~ be tripped over
.. Remove unused cable
• Ensure any unused terminated fiber i. capped to prevent dust
contamination or scratches on fiber end.
o Ensure that cabinets, moo, and cable !mys are properly grounded
and that bonding conductors ate secure. Grounding connecUons
should be Inspected ann"ally at minimum
• EnSlJre mat all cabinets, racks, paw. panel., cables, cnd ""tch
ca bles are properly labeled

Elactrl<:al SyStem Maintenance 14.21..a

The malntenal'lCe requirements of electncal subsystems Will vary.
Generato'"', batt~r les and UPS systems will require perlodk:
maintenance at manufacturer-speclNed or industry standarn Intervals.

Best Prac~ces Include:

• Perform inspections as required by local jurisdiction
• All power connections secure
o 1R ThemlO9raphy scanning is recommended once per yea r to
Identify loose and / or poor connections, unbalanced electrical
loads, all characterized by Increased Mstance or temperature rise
• All receptacles and power strips properiy labeled
• All safety features in ptace and operational
 Commissioning & Maintenance

• Doors and cover pla~s on breaker panels and power distribution

units In place and operating properly
• UPS and power generation systems periodic mainreroa nee
inspections perfonned in accordance with rna nufacturer's
specifications and/or AHJ
• Consider battery monitoring In UPS systems
• Lighting systems checked and bulb. replaced as required
• Emergency ligl>tin9 operational check as required by AHJ
• BAS and fin! alann systems opera!lonal check as required by local
jurisdiction
• Insure complete compliance with all safety regulatiol'lS a. mandated
by OSHA, NEe, local code and AHJ

. 1.1)
IiICSI Ptoplk.ibd, PNopIIrty-Oo NatCop'f
_Goide

HVAC 5)'$*"8 14.29-30

HVAC systems in data centers are a critical component iD the

·ope ... tion of the center. It Is very Important that HVAC systems
operate at optimal performance.

Best Practices include:

Monthly:

• Verify each module is m<lintaining temperature & humidity set

points
• Recon:t room temperature 8< humidity near the return vent
• Inspect for "~ible exterior damage
• Inspect for environmental damage (dirt, dust, debriS, liquid .!<lins)
• Inspect bellS (If applicable) for wear and tightness
• Check return <lir filters; replace It necessary
• Check drain pan; remove any aCOJmul"tion
• Ensure the blower/motor are moving freely
• Inspect set·screws on fan blades &. bushings
• Venfy condensate line 15 fiowl ng freely
• Venfy that the humid meation system does not have lea ks
• Ensure there are no kinks in the steam delivery sy<tem
• Check fur build-Up In deanable cylinders
• Replace dispooah'" humidifier cylinders
• Venfy chilled water supply temperature
• Inspect electrical panel to ensu re tig ht con nections
• Confirm that the Incoming ma in power Is within toHoranw listed on
the module's nameplate
• Verify control voltage' il within manufacturer's recommendation

...,.
BICSI ... opiCIry PmpIII'ty-Oo HlIt Copy
5tudont Guklo Commtsslon~1lI & Yaintenanctt

HVAC System Maintenance, Cant. 14.31

Quarteriy:
• Check watEr lines for leaks
, Vefify tightness of blower and its motor hardwa re
• Record amperages for critical components (such as cooling coli fan
motors, reheater, humidifier, etc)
Semi-Allnually:

• Verify that contacts 8t retays are op!!ra~ng correctly

... o,ec;:k the main power wiring for Criticat components such as reheat
element:s evapOf'"iltor mctorS.1 and humidifiers
j

• Verify oper<ltion of the .willluid (water regulating) valves 8t record

pressure
• Check the openltion of alf.system alarms

IT Maintenance 14.32

Some IT systems must be maintained by factory authorized service

techni<;i."s because of their proprietary nature "ndlor complexity
• care must be taken during design to allow for sufficl<!nt equipment
access by techn ldans

IT Maintenance checks shoul" be performed accord ing to manufacturer

specifications. Regular checks typic.ally indude:
- Cleaning of cooling fans (If equipment fans are in use)
- Cleaning air filters/tlltratlon pathways
- CI1ecking for failing/failed power supplies
- CI1ecking exposed c.abling (power or data) for possible d~mage

- Conftnning all computer and telecommunications equipment is

properly labele"
Stuchilnl: Guida

Acc.ss Floor Mainl8nance 14.33

Regular maintenance inspections'and cleaning of the a~ces~ Hoor area

are reQ ulred
• BesrPracti~es for a~~ess Roor maintenance lnclude:
- Develop and implement good dally! weekly housekeeping
pro~edures

- U.e care when handling Hoar panels to avoid da maging panels

- Do no! exceed floor loading capacity
- Implement monthly, quarterly and annual Hoor Inspections
- Implement quarterly or annual .ubRoer clean ing

FI ... Protection System Maintenance 14.34

The fire protection & su pp.-ession system shall be malnta Ined In

accordance with local Ii... code. and manuladurers' recommendations

Security System Maintenance 14.35

Maintenance on a data center's security ,ystem shall be perfonmed by

certified/authorized secUlity system personnel
• Maintenance ac~vlt les can Include but are no! limited to,
- Camera adjustments
- Motion detector sensitivity and locatJon adjustments
- Automatic door locl</dosure adjustments
- Video reccrdlng system server updates
- Video tap" rotation and storage

Maintenanc. Recor<ItMplng 14.36

Maimenan,e record keeping is recommended as a valuable tool to
establish maintenance histories. baselines, and data trending
• A maintenance software data base is recnmmended as an effiCient
way of tracking maintenance informatiOn and trends

.4-1'
BleSI "'p........_
......1J!~ Ntli Cop,
student Oukle Comml..Jonlng & Mllnt«nan.-

Commi..ioning & em... 14.37

Commlssionl". & (!;mes daIS Fl Oass F2 CLass F3 aass F4

All electrical 'YSlems tested Yes Ves V.. Yes
prior to operallon
All electrical ,y.tem Yes Yes Yes Ye,
equipment lab@led wit"
«rtifi",tioo from 3rt! party
toS! laboratory
foctory testing Not Typical NOITypical Optio,..1 y",
I";e·Functiooal Start·Up Yes Y.. Yes y",
Eq uipment Functional Optional Ye, Y., Yo:!
Testing
S'lsWn Functional Testing Optional Ves Ye, Yes
E"ctrk<1 System T.Sltng Optional Optional Ye, Yes
Whole Building T..~ng Nor Typical Not Typic.1 Optional Yes
Studlnt Guide

Review QuestIons

1. What are the S phases of the ccmmissiooing process?

2. What are the 3 crucial components of Cflmmissioning?

J. What do yoo think is the most important aspect of the

commissioning process?

'<-1'
IICSlflnoJiMbtyA'Operty-Oo Nat Copy