Commissioning:

Early and Often

Ryan Orr
Senior Consultant, Uptime Institute

Webinar: 16 December 2014

2014 Uptime Institute, LLC

 Observations from the Field
Commissioning resources are being marginalized
(schedule, financial, human resources)
TCCFs are illuminating issues that should be caught by
basic commissioning processes
Research and experience reveals the need for enhanced
focus on rigorous commissioning activities
catch problems in commissioning before an outage occurs

2 2014 Uptime Institute, LLC

 The Issues
Commissioning has become a wasted opportunity in the
industry due to
a lack of planning
assumptions that proof of concept is adequate
a lack of understanding of the benefits of commissioning
Operations personnel are minimally involved, or not at all

3 2014 Uptime Institute, LLC

 Why is Commissioning Important?
Unique period of time to verify system performance as
designed without risk to mission critical IT loads
Can reduce critical infrastructure early equipment failure
rates
Opportunity for maintenance and operations teams to get
hands-on equipment experience
Opportunity to verify detailed written procedures that will
govern live facility maintenance and operations
Only opportunity to test the facility limitations

4 2014 Uptime Institute, LLC

Commissioning Defined

2014 Uptime Institute, LLC

 Level 1 Factory Witness Testing (FWT)
Allows for basic verification of operation and capacity for critical
infrastructure
Performed in the original equipment manufacturers (OEM)
factory or in a third-party testing facility
Verifications are performed with conditions and procedures in
accordance with manufacturer, national, or international
standards
Helps prevent delivery of components with unrepairable defects
Reports generated by OEM, provided to client, and reviewed by
the Commissioning team
Performed on engine generators, UPS systems, chillers, air
conditioners, and switchgear

6 2014 Uptime Institute, LLC

 Level 2 Receipt, Installation, &
Post-Installation Checks
Inspection, verification, and tests upon delivery
equipment matches that procured and tested during Level 1
no damage
no alterations
Post-installation checks
each and every component and auxiliary appurtenance installed
in accordance with drawings, plans, and specifications,
accessibility, maintainability, health and safety requirements, local
code compliance, and manufacturers installation requirements
and directives

7 2014 Uptime Institute, LLC

 Level 3 Functional Component Testing
Verification that the installed component is operable at a
basic level
Maintaining checklists for mechanical and electrical start-
up
Initial performance verification by OEM
Mechanical systems should go through a Pre-Test and
Balancing effort to ensure accuracy before Level 4

8 2014 Uptime Institute, LLC

 Level 4 Functional System Testing
Verification of system readiness for integration with other
systems supporting the data center
Test, Adjust, & Balance (TAB) of the mechanical systems
to ensure design airflow and water-flow rates
Demonstrations to ensure that related components,
equipment, and ancillaries of a defined system operate
and function to acceptance criteria
normal, maintenance, and emergency modes of operation to
verify settings, alarms, capacities, and performance of associated
monitoring and control functions

9 2014 Uptime Institute, LLC

 Level 5 Integrated System Testing (IST)
Verification that all data center systems work together
under a variety of load conditions as designed
Verification that systems respond to various actions,
maintenance activities, or faults as designed per
Sequences of Operation
Verification that each component and system as a whole
respond as intended to expected and unexpected events

10 2014 Uptime Institute, LLC

11 2014 Uptime Institute, LLC
12 2014 Uptime Institute, LLC
Commissioning Execution By Phase

2014 Uptime Institute, LLC

 Commissioning Stakeholders
Owner or Owners Representative
Contractor (inclusive of OEM equipment vendors)
Architects & Engineers
Operations Personnel
Commissioning Agent (CxA)

14 2014 Uptime Institute, LLC

 Pre-Design Benchmarks
Owner or Architects
Contracto Operations Commissioning
Owners &
r Personnel Agent
Representative Engineers

Develop Request for Proposal

(RFP) for Commissioning Agent

Select Commissioning Agent

Establish a Commissioning Team

with key stakeholders

Include Commissioning in the

Overall Project Schedule

Identify the budget for

Commissioning

Develop Owners Project

Requirements (OPR)
Documentation

Develop Commissioning Plan

Review Commissioning Plan

15 2014 Uptime Institute, LLC

 Pre-Design Requirements
Account for completion of all commissioning activities
Schedule has significant flexibility
Sufficient time allotted for correcting installation and
performance deficiencies
Assess the requirement and/or capability for post-
occupancy commissioning activities
include provisions for seasonal commissioning to assess the
performance of critical components in a variety of ambient
conditions

16 2014 Uptime Institute, LLC

 Design & Pre-Construction Benchmarks
Architects
Owner or Owners Operations Commissioning
Contractor &
Representative Personnel Agent
Engineers

Review Design for OPR

Concurrence

Review Design for Operations

Concurrence

Review Design for Ease of

Commissioning

Develop Commissioning Plans,

Checklists, and Reports for
Level 1, 2, and 3

Review Commissioning Plans,

Checklists, and Reports for
Level 1, 2, and 3

17 2014 Uptime Institute, LLC

 Design & Pre-Construction Requirements
Review project schedule and budget
ensure adequate resources and time remain
Verify adherence to OPR and BOD
amend as necessary to keep up to date
Scalable and Phased: include enhancements to allow for
future commissioning with reduced risk

18 2014 Uptime Institute, LLC

 Design & Pre-Construction Requirements
Equipment specifications
specified capacity is net of any deductions or tolerances allowed
by national or international manufacturing standards for Level 1
Determine if the systems can be commissioned per OPR
add additional design elements as required to allow for the
commissioning program to meet the minimum OPR

19 2014 Uptime Institute, LLC

 Design & Pre-Construction Requirements
Ensure that RFPs include commissioning requirements
include OEM on-site technician support for Level 4 and Level 5
assess adherence throughout equipment delivery and installation

20 2014 Uptime Institute, LLC

 Construction Benchmarks
Architects
Owner or Owners Operations Commissioning
Contractor &
Representative Personnel Agent
Engineers

Execution of Level 1
Commissioning: Factory
Witness Testing

Execution of Level 2
Commissioning: Receipt,

Installation, and
Post-Installation Checks

Execution of Level 3
Commissioning: Component
Functional Testing

Develop Commissioning Plans

& Scripts for Level 4 and 5

Review Commissioning Plans

& Scripts for Level 4 and 5

21 2014 Uptime Institute, LLC

 Construction Requirements
Review project schedule and budget
ensure schedule continues to have adequate time and budget
Throughout construction protect equipment stored
pending installation from hazards
Engineers should provide a finalized Sequence of
Operations document to the Commissioning Agent
CxA creates the Level 4 and Level 5 commissioning scripts
Verification of circuit breaker settings per short circuit
and breaker coordination study

22 2014 Uptime Institute, LLC

 Construction Requirements
Log critical asset information (e.g., make, model, serial
number) as equipment is delivered
Repeat factory testing activities in actual data center
environment
Verification that building management system (BMS) is
functional and ready to support critical Level 4 and 5

23 2014 Uptime Institute, LLC

 Construction Requirements
Submit formal reports to Owner detailing all items
tested, steps taken to test, and the results

24 2014 Uptime Institute, LLC

 Commissioning Benchmarks
Architects
Owner or Owners Operations Commissioning
Contractor &
Representative Personnel Agent
Engineers

Execution of Level 4
Commissioning: Functional
System Testing

Execution of Level 5: Integrated

Systems Testing

Final Site Clean-Up

Ensure all equipment is back in

normal position

25 2014 Uptime Institute, LLC

 Commissioning Requirements
Do not tolerate representative testing
each and every critical component must be tested thoroughly
Electronic systems involving programming and configuration
if a testing step does not complete successfully and programming or
control wiring is altered to correct, repeat the entire testing procedure as
it is possible to have an unexpected impact
Unsuccessful first tests should be considered for multiple
retests to ensure subsequent successful test was not an
anomaly
Level 4 load bank testing
engine generators, UPS, and UPS battery systems at design and rated
capacities
recommended: minimum continuous runtime durations of 8 hours
best practice: continuous runtimes of 24 hours

26 2014 Uptime Institute, LLC

 Commissioning Requirements
As possible without causing damage, test emergency
conditionssuch as N-1 and no cooling with design
Install aisle containment as part of Level 4 and Level 5
Identify, document, and validate normal operating set
points, alarms, and component settings during Level 4
and Level 5
Changes made during Level 5 to fix deficiencies must
include evaluation to determine which, if any, tests must
be repeated

27 2014 Uptime Institute, LLC

 Commissioning Requirements
Complete testing on both utility power and on engine-
generator power
Simulate multiple fault types across separate tests on
each piece of equipment
on highly automated data centers that rely heavily upon field
sensors, include sensor failures in testing scope
Load banks should be as small as reasonably possible to
best simulate the actual IT environment for Level 5

28 2014 Uptime Institute, LLC

 Commissioning Requirements
Test a variety of load conditions (e.g., 25%, 50%, 75%,
100% step loads)
Distribute load banks within the critical areas to best
simulate the actual IT environment distribution
physically located within racks
forced cooling on horizontal path
Locate Commissioning team members strategically
throughout the data center to monitor all systems during
Level 5

29 2014 Uptime Institute, LLC

 Commissioning Requirements
Before Level 5, complete the building management and
control system (BMCS) graphics
Monitor alarms generated in BMCS and electrical power
monitoring system (EPMS) to ensure accuracy and
usefulness
Take electrical load readings and critical area
temperature readings constantly during Level 5
Ideally use calibrated automatic data loggers

30 2014 Uptime Institute, LLC

 Commissioning Requirements
Isolate equipment to simulate maintenance activities at
the upstream circuit breaker
not the units local disconnect
Perform Level 5 with active fire detection and
suppression systems to ensure there are no adverse
impacts

31 2014 Uptime Institute, LLC

 Commissioning Requirements
After Level 5, replace air filters for the electrical systems
and heating, ventilation, and air conditioning (HVAC)
systems
Flush and clean piping and ductwork to ensure
construction debris does not impact future mechanical
plant performance

32 2014 Uptime Institute, LLC

 Turnover-to-Operations Benchmarks
Expeditiously complete all formal activities turning the facility
over to the owner and the operations team
final commissioning documentation and reports
final operating procedures
Critical juncture for operations team to apply lessons
learned
Make all possible support resources available to operations
team
The longer it takes, the longer the facility will be at risk!

33 2014 Uptime Institute, LLC

 Re-Commissioning & Future Installation
Increasingly designs are made scalable for incremental
buildouts to more efficiently deploy capital
leading to rigorous initial commissioning, but no Level 4 or 5 on
the subsequent phases
increased risk to the facility
Owners are not examining the risk of only light
commissioning for future phases
Rigorous commissioning in the beginning will make
follow-up commissioning activities far easier
Consider altering the deployment schedule to
accommodate less risky commissioning

34 2014 Uptime Institute, LLC

 Re-Commissioning & Future Installation
Elements and Benchmarks

Architects
Owner or Owners Operations Commissioning
Contractor &
Representative Personnel Agent
Engineers

Develop detailed
commissioning scripts for
all levels

Review Commissioning Scripts

Ensure IT hardware is dual

corded (for those facilities that
require it)

Execute Commissioning
Update SOPs, MOPs,
and EOPs

35 2014 Uptime Institute, LLC

 Re-Commissioning & Future Installation
Elements and Requirements
Adequate notice to service owners to gain concurrence
from IT end users
schedule, duration, risk, and countermeasures in place
Verification that existing critical load is appropriately dual
corded (as required)
Consider load bank placement carefully to not impact
critical IT equipment

36 2014 Uptime Institute, LLC

 Re-Commissioning & Future Installation
Elements and Requirements
Prepare and follow detailed commissioning scripts to
ensure minimal risk to existing IT equipment
priority to the live production IT environment
back-out procedures in place to ensure an optimal mean time to
recovery (MTTR) in case of a power down event
Perform seasonal testing to verify performance in a
variety of climatic conditions including extreme ambient
conditions
ensures that economizers, where used, will be tested properly

37 2014 Uptime Institute, LLC

 Conclusions
Not an opportunity to be wastedthe benefits should not
be undervalued
We, as an industry need to be better about identifying and
addressing potential roadblocks for a rigorous
commissioning program earlier in the project
Start With The End In Mindadditional emphasis needs to
be placed on involving the operations personnel early and
often
Sets up your Operations team for success
Education regarding the benefits of commissioning is the
key!!

38 2014 Uptime Institute, LLC

 Questions?
Ryan Orr
Senior Consultant

rorr@uptimeinstitute.com

2014 Uptime Institute, LLC