Planning Douses Fire’s Impact

®

DRJ’s Spring WorldDRJ’s Spring World

March 2012March 2012Rick FabrizioRick Fabrizio

VP & CIOVP & CIO

Agenda

Business profile and background

“The Fire”e e

Planning leading up to the fire

Lessons learned from fire

Next steps

®

2

Business Profile & Background

®

3

Business Profile

Nation’s largest retail propane distributor 1.3 million customers

Approximately 1 0 billion gallons sold annually Approximately 1.0 billion gallons sold annually Approximately 1,200 locations serving all 50 states 6,000+ dedicated employees

®

4

December 7, 2010December 7, 2010

Diversified Customer Base

5%5%AgricultureAgriculture

TransportTransport

13%

37%

40% ResidentialResidential

Motor FuelMotor Fuel

®

5

37%

CommercialCommercial

December 7, 2010December 7, 2010

Counter-seasonal to heating season

11 million cylinders per year

AmeriGas Cylinder ExchangeAmeriGas Cylinder Exchange

30,000 locations at home centers, supermarkets, convenience stores

Vending machine (24 hour access)

®

6

December 7, 2010December 7, 2010

National Footprint - Key Competitive Advantage

®

7

December 7, 2010December 7, 2010

Approx 600 district office and 600 satellite locationsApprox 600 district office and 600 satellite locations

Key Corporate Locations

Call Center

Call Center HQ

®

8

December 7, 2010December 7, 2010

•• Valley Forge HeadquartersValley Forge Headquarters

•• Cleveland Call CenterCleveland Call Center

•• Rocklin Call CenterRocklin Call Center

•• Houston Supply OfficeHouston Supply Office

Supply

2009 Headquarters for UGI and AmeriGas

•• King of Prussia, PA King of Prussia, PA

•• Approximately 350 Approximately 350 employeesemployees

300 AmeriGas and 50 UGI300 AmeriGas and 50 UGI•• 5 story 100,0005 story 100,000--sqsq--foot buildingfoot building

•• Built in Built in 1973, a “1973, a “ttiredired” building” building

•• Improvements started in 2009Improvements started in 2009

300 AmeriGas and 50 UGI300 AmeriGas and 50 UGI

•• Houses corporate Houses corporate ddata center ata center

•• Satellite office down the street forSatellite office down the street for

ERP i l t ti j tERP i l t ti j t

®

9

ERP implementation projectERP implementation project

“The Fire”

®

10

•• New lighting installed in overhang New lighting installed in overhang around the exterior of the building around the exterior of the building

•• 6:15 PM 6:15 PM electrical fire electrical fire broke broke outout

December 16, 2009

•• Approximately 50 employees in buildingApproximately 50 employees in building

•• Smoke alarms sounded Smoke alarms sounded

•• Everyone safely evacuatedEveryone safely evacuated

®

11

December 16, 2009

•• Two alarm fire Two alarm fire

•• Smoke and water damage Smoke and water damage throughout the throughout the structurestructure

•• “What do we do now?”“What do we do now?”

•• Initial Initial assessment was way assessment was way off baseoff base

®

12

•• Forced to vacate the building and Forced to vacate the building and relocate employees to temporary relocate employees to temporary space for approx 10 monthsspace for approx 10 months

Immediately Establish Recovery PMO

®

13

IT resources could not manage overall recovery programIT resources could not manage overall recovery program

Business operations immediately post fire event

Cost accounting

Workspace/recovery for over 90 Workspace/recovery for over 90 transactional employees 24 hours transactional employees 24 hours after fire occurredafter fire occurred

Asset management

Accounts payable

Credit, Lockbox, Collections

Financial Reporting

Treasury

160 employees fully functional

Purchasing

Safety

UGI AccountingQuarter close completed on time

6 PM 6 PM Dec 16Dec 16

Dec18Dec18 Dec Dec 2121 Jan11Jan11

Payroll completed successfullyCash collected and applied to customer accountsRecovery of Recovery of

Midnight Midnight Dec 16Dec 16

Dec17Dec17

®

14

Vendor payments resumedto customer accounts

Daily Volume Margin report (critical management tool)

All daily activities for core

departments now at full strength

Remote access for everyone else:

- Sales, Marketing, HR, Audit, etc.

critical services critical services in data centerin data center

Declared Declared DisasterDisaster

Ten Month Recovery Timeline

IT, furniture contractors, electricians and wiring contractors worked through the New Year’s holiday weekend

JanJan MayMayMarMar JulyJuly SeptemberSeptember

The first wave of employees moved in

holiday weekend

New office space selected for temporary headquarters, within a week of the fire.

Data Center remediation

Vendor testing and insurance claim

Complete final wave of Establish new Credit &

Move back to renovated HQ

FebFeb

Building renovations

®

15

week of Jan 5th , two weeks after property selection.

functionality for phone system at temporary HQ

Collections facility

Planning

®

16

D/R & Business Continuity Implementation Timeline

2006 2007 2008 2009Identified the clear need for a Identified the clear need for a Disaster Recovery planDisaster Recovery plan

Contracted with 3Contracted with 3rdrd party to party to serve as the DR site and coserve as the DR site and co--location site for backlocation site for back--upup

Successful “test” of a Valley Successful “test” of a Valley Forge event with remaining Forge event with remaining of core functionsof core functions

2nd successful test at DR 2nd successful test at DR site, several logistics and site, several logistics and system issues improvedsystem issues improvedlocation site for backlocation site for back up up

hardwarehardwareof core functionsof core functions system issues improved system issues improved

over the first testover the first test

Planned the relocation of one Planned the relocation of one of two STARS host computers of two STARS host computers from Sacramento, CA to Valley from Sacramento, CA to Valley Forge, PA with redundant Forge, PA with redundant polling capabilitiespolling capabilities

Developed “playbooks” for all Developed “playbooks” for all core departments that were core departments that were needed onneeded on--line ASAP after an line ASAP after an incidentincident

Houston plan developed by the Houston plan developed by the Houston Supply TeamHouston Supply Team

Developed similar plans for Developed similar plans for the Westlake and Rocklin the Westlake and Rocklin Call CentersCall Centers

Houston successfully Houston successfully implements its DR program implements its DR program during Hurricane Ike when during Hurricane Ike when the local office could not be the local office could not be used used

Westlake’s 1st test using Westlake’s 1st test using the workspace recovery the workspace recovery sitesite

Rocklin contract being Rocklin contract being finalized with workspace finalized with workspace site, test planned for site, test planned for Spring 2010Spring 2010

®

17

Actual Valley Forge event Actual Valley Forge event December 16, 2009December 16, 2009

Goal: Minimize Impact to Customers, Employees and Cash FlowGoal: Minimize Impact to Customers, Employees and Cash Flow

In Scope Systems for Recovery

Sacramento Cleveland Houston Valley Forge

Systems in scope for 1Systems in scope for 1stst round of recovery listed belowround of recovery listed below•• During a disaster, target recovery time for 1During a disaster, target recovery time for 1stst round of recovery is 24round of recovery is 24--96 hrs96 hrs

•• Remaining systems (not listed below) will be recovered after 1Remaining systems (not listed below) will be recovered after 1stst round based on priorityround based on priority

2007

Sacramento Cleveland Houston Valley ForgeCIS - Host

Supply System

National Accounts

Centralized CIS

Customer Portal

Truck Computer App

Routing & Dispatch

Qualcomm

PC-Miler

Wide Area Network

District Connectivity

Connectivity to CIS Host and Supply

Financials

Consolidation & Treasury System

Payroll

E-mail and Blackberry

Critical Reports – “Volume Margin Report”

®

18

Recovered at Valley Forge Data CenterRecovered at Valley Forge Data Center Recovered at 3Recovered at 3rdrd party D/R siteparty D/R site

Project Update Project Update -- Valley Forge RecoveryValley Forge Recovery

33rdrd Party D/R and Workspace recovery facility Party D/R and Workspace recovery facility

“Co“Co--location” System Recoverylocation” System Recovery

–– Secure facilities in Valley Forge & Bethlehem with 10 Secure facilities in Valley Forge & Bethlehem with 10 mWmW, , fully redundant generatorsfully redundant generators

20072007

y gy g

–– Existing Test/QA servers moved to D/R siteExisting Test/QA servers moved to D/R site

Workspace Recovery Workspace Recovery

–– 30 seats for functional staff (non IT)30 seats for functional staff (non IT)

–– Phones, AmeriGas imaged PC’s, WorkstationsPhones, AmeriGas imaged PC’s, Workstations

–– Access to Conference RoomsAccess to Conference Rooms

®

19

Alternative Workspace site, BethlehemAlternative Workspace site, Bethlehem, PA, PA D/R Site & Workspace site D/R Site & Workspace site Valley Forge, PAValley Forge, PA

CoCo--location location Data CenterData Center

June 19June 19thth Test ResultsTest Results

Successfully performed test of logistical readiness

Disaster Recovery procedures tested and verified for all “tier 1” systems

Successfully performed “desktop” test of critical business applications

20082008

y p p pp

Successfully accessed and gained business sign-off for:

– Disaster Recovery Environments for remote data centers

Identified several connectivity issue/bandwidth issues at recovery site “slow”Identified several connectivity issue/bandwidth issues at recovery site “slow”

®

20

Business Continuity – VF Finance Test conducted in June 2009

BC Workspace Objectives and Results

June Test ResultsJune Test Results20092009

BC Workspace Objectives and Results

Objective Result Within a timely fashion, successfully travel and gain access to the DBSI facility. Successful

Log-on and acclimate yourself to the desk-top environment Successful

Send and receive e-mail Successful

Access and Map the network drives housing your data Successful

Successfully access your Business Unit’s recovery scripts Successful Successfully access all supporting documents by following the hyperlinks embedded in the recovery scripts. Successful

From DBSi, each team member will put a new personal greeting on their Valley Forge phone Successful

Each team member will ask someone to call their Valley Forge phone, listen to the new personal greeting, and then have that someone call the team member at DBS

Successful

Test the band-with capabilities between DBSi and Valley Forge Successful

S f l

®

21

7

144 of 148 business objectives met.

Connectivity gap from 2008 test remediated

Verify that any gaps from the 2008 test have been closed SuccessfulSomeone other that the author of the scripts successfully conduct business by following one or more recovery scripts.

Successful

Lessons Learned

®

22

Lessons LearnedLessons Learned

Adequate seats available

Connectivity critical

Key peripherals not included in past tests

Relationship with key vendors important

®

23

Certain areas recognized lack of participation in testing

Lessons LearnedLessons Learned

Need to formalize call tree methodology

Establish business led BCP Steering Team to engage other areas of the business and justify further investments

All locations should define site management plans similar to the ones developed “on the fly” post fire event

®

24

Define viable work from home strategies

Next Steps

®

25

Next StepsNext Steps

Phase-in connectivity diversity

Provide warm fail-over for critical systems

Address remaining gaps

Continue annual testing for all locations

Evolve crisis management plan

®

26

Partner with sister companies

Explore cloud options for data center