1

© 2014 Uptime Institute

Start With The End In Mind

R. Lee Kirby,Chief Technology Officer

Uptime Institute

Eric M. Carter, PESenior Technical Manager

Balfour Beatty ConstructionMission Critical

2

� Uptime Institute research and field experience confirms• Data center operations start before the conceptual design

phase of construction• Data center operations continue to serve as the source of

continuity for knowledge management and transition to production

� Why an Operations Focus?• Increase return on investment• Increase uptime• Increase data center efficiency• Reduce costs• Reduce risks

Start With The End In Mind

© 2014 Uptime Institute2

3

� Bring Operations in from the beginning� Design ���� build ���� operate is misleading� Design-build (expand, retrofit) are all point-in-time, nested

functions of change management� Change is constant—but a planning continuum that

includes operations will:• Mitigate risk of disruption• Ensure reliability standards are not compromised by changes

� Change management lets you avoid rather than respond� Value engineering in isolation from operations can leave

ongoing legacy of increased costs

Start With The End In Mind

© 2014 Uptime Institute3

4

� Develop and test EOPs when commissioning new systems

= decreased time to restore normal operations after an incident

� Preparation (training and drills) for catastrophic events = the difference between an incident and an outage

� Operations is unlike other aspects of project development• Not a one-time set up• Regular assessments must be ongoing

� Certifications are a benchmark, not the destination

It’s a Journey—Not a Destination

© 2014 Uptime Institute4

5

6

7

� Data center strategy • Define measures of success, SLAs, KPIs• Concept of operations• Concept of maintenance• Vendor requirements

� Organizational alignment • Roles and responsibilities• Organizational chart (operations, facilities, IT, security)

Roadmap to Success: PRECON

© 2014 Uptime Institute7

8

� New Technology (NBAD – Abu Dhabi)

• UPS technology• New technology vs. staff experience• Vendor support• Parts availability

� Major Maintenance Efforts (Confidential – RTP, NC)

• Battery replacement• Opened vendor options• Reduced critical shutdowns and rentals to a single MOP• Saved approx. $120K in CAPEX by moving MOP forward

In Practice: PRECON

© 2014 Uptime Institute8

9

10

� Maintenance and Operations Design Review • Support and specialty space analysis• Security, access, and setbacks analysis• Flexibility for incremental capacity increases analysis• Ease of maintenance analysis• Design concurrence

Roadmap to Success: DESIGN

© 2014 Uptime Institute10

11

� Maintenance and Operations Planning • Establish vendor/contractor SLAs and support contracts• Define staffing levels and shift strategy• Define staff qualifications and assess capabilities • Establish equipment maintenance plans• Establish operations standards consistent with site mission,

reliability and availability requirements, and industry best practices

• Identify and acquire MMS and other key operating systems• Establish asset life-cycle analysis program

Roadmap to Success: DESIGN

© 2014 Uptime Institute11

12

� Maintenance Access (Confidential Client – UT)

• Battery maintenance• Specific equipment• Access around racks• Saved 2 weeks and 4 hours / maintenance effort

� Vendor and CX Spec Development (SAS – NC)

• Chiller selection (vendor support feedback)• Quick start testing and timing• Identified MOP tests for CX• Identified staff roles and responsibilities

In Practice: DESIGN

© 2014 Uptime Institute12

13

14

� Management and Operations Program Development• Develop operations procedures (SOPs, MOPs, and EOPs)• Implement systems and processes (MMS and other key

operating systems, document repository)• Develop training program• Establish minimum shift and daily inspection protocols• Develop weekly/monthly walk-through equipment checklists• Establish monitoring and controls systems reports• Develop escalation policies and protocols including contact

lists (addressing increasing levels of severity including alerts, events, and incidents)

Roadmap to Success: CONSTRUCTION

© 2014 Uptime Institute14

15

� Management and Operations Program Development• Establish inventory of critical spare parts and consumables• Develop housekeeping policy and Critical Environment work

rules• Develop Critical Environment work approval and change

management processes (normal, expedited, and emergency)• Develop Critical Environment work approval procedures and

forms• Establish risk windows and allowable activities• Develop predictive maintenance program

Roadmap to Success: CONSTRUCTION

© 2014 Uptime Institute15

16

� Testing and Script Development (BB&T - NC)

• BMS Sequence tweaks (pumps)• Critical MOP development > EOP development• SOP and EOP Training• Maintenance planning• Organize documentation• Avoided 1 shutdown ($$)

In Practice: CONSTRUCTION

Method of Procedure (MOP) Form

© 2014 Uptime Institute16

17

18

� Maintenance and Operations Readiness• Commission operations procedures (SOPs, MOPs, and

EOPs)• Critical infrastructure systems operations training• Emergency operations drills and systems recovery training• Safety training and on-site safety planning• Key vendor/contractor on-site training• Electrical safety training• Populate MMS and other key operating systems

Roadmap to Success: COMMISSIONING

© 2014 Uptime Institute18

19

� Testing / Training (SAS – NC)

• Local utility issues• Modified commissioning scripts to include fault testing from

other facility issues on campus• Training vendors and 2nd shift staff

� Timing (BB&T – NC)

• Thermal storage discharge• Thermal storage recovery• “A” to “B” plant change over process (manually)• Made major modification to storage recovery based on

manual options timing

In Practice: COMMISSIONING

© 2014 Uptime Institute19

20

21

� Turnover and Transition • Review commissioning results and prioritize punch list• Conduct operational readiness assessment• Implement operations management program• Refine operations procedures (SOPs, MOPs, and EOPs)• Exercise all procedures to ensure optimal effectiveness

� Tier Certification of Operational Sustainability (TCOS)

Roadmap to Success: TURNOVER

© 2014 Uptime Institute21

22

� Transition to Operations – MOST OVERLOOKED• Good & Bad handoffs:

– SAS was GREAT – heavily involved throughout entire process– Confidential Healthcare Client – not involved until L5 CX = lot of

“warranty” calls & general system operations questions.

• Training starts during design & procurement – reduced learning curve at turnover

• Ops & security coordination training– Alarms– Response testing & coordination

• SOP, MOP, & EOP documentation organization refinement– (SAS) BIM model was abandoned for data base system

In Practice: TURNOVER

© 2014 Uptime Institute22

23

23

A Bridge Between Business Objectives and Sustained Operations

© 2014 Uptime Institute

24

� The End is Just the Beginning• Regular reviews lead to discovery• Discovery leads to learning• Learning leads to change• Change leads to improvement

� Start with the end in mind and you will• Increase return on investment• Increase uptime• Increase data center efficiency• Reduce costs• Reduce risks

Summary – Start With The End In Mind

© 2014 Uptime Institute24

25

For More Information Contact:R. Lee Kirby

lkirby@uptimeinstitute.com+1 425.241.0088

Questions?

© 2014 Uptime Institute25