Joel A. Manfredo

June 14, 2012

1

Getting to a Service Catalog

Background

Timeline

Phases 1 - 5

Results

Next Steps

Conclusions and Lessons Learned

Agenda

County of Orange

$5.6 Billion Budget 7 Programs 32 Agencies / Departments

CEO IT Services

Services are delivered through an Internal Service Fund

“Running a Service Delivery Business”

Pricing a Service

What does internet access cost? Hardware: •Redundant Firewalls •Redundant Routers •Redundant Core Switches •IPS / IDS Hardware

•Monitoring (Orion) Servers •Content Filtering Appliances •Content Filtering Report Servers •DNS Servers

Software: •Firewall Security Software licensing

•Content Filtering Software licensing & Report Services

•Orion Monitoring Software licensing

•DNS Software (Unix and Wintel)

Data Center Facilities Contracts: •Redundant Internet Service Providers:

1.TW Telecom

2.VPLS (this is a fail-over service) •IPS / IDS monitoring Service

•Redundant WAN Circuit Contracts

•Redundant Firewall Maintenance Contract (HW and SW) •Redundant Routers Maintenance Contract HW and SW)

•Redundant Core Switches Maintenance Contract HW and SW) •Content Filtering Maintenance Contract •County ARIN IP and BGP AS Services

Depreciation: •Firewalls - Hardware and software Depreciation associated •Routers - Hardware and software Depreciation associated •Core Switches - Hardware and software Depreciation associated •Monitoring - Hardware and software Depreciation associated •Content Filtering - Hardware and software Depreciation associated

Other CEOIT Services: •Contracts & Procurement Management •Resource Management •Financial and Cost, Budget Analysis •Billing Processing

Labor: •Network Engineering Labor •Security Engineering Labor •Availability Management Analysis •Capacity Management Analysis •Incident Management (Service Desk) •Change Management •Management

Pricing a Service

What does internet access cost? Hardware: •Redundant Firewalls •Redundant Routers •Redundant Core Switches •IPS / IDS Hardware

•Monitoring (Orion) Servers •Content Filtering Appliances •Content Filtering Report Servers •DNS Servers

Software: •Firewall Security Software licensing

•Content Filtering Software licensing & Report Services

•Orion Monitoring Software licensing

•DNS Software (Unix and Wintel)

Data Center Facilities Contracts: •Redundant Internet Service Providers:

1.TW Telecom

2.VPLS (this is a fail-over service) •IPS / IDS monitoring Service

•Redundant WAN Circuit Contracts

•Redundant Firewall Maintenance Contract (HW and SW) •Redundant Routers Maintenance Contract HW and SW)

•Redundant Core Switches Maintenance Contract HW and SW) •Content Filtering Maintenance Contract •County ARIN IP and BGP AS Services

Depreciation: •Firewalls - Hardware and software Depreciation associated •Routers - Hardware and software Depreciation associated •Core Switches - Hardware and software Depreciation associated •Monitoring - Hardware and software Depreciation associated •Content Filtering - Hardware and software Depreciation associated

Other CEOIT Services: •Contracts & Procurement Management •Resource Management •Financial and Cost, Budget Analysis •Billing Processing

Labor: •Network Engineering Labor •Security Engineering Labor •Availability Management Analysis •Capacity Management Analysis •Incident Management (Service Desk) •Change Management •Management

The Story…

Operational Transformation in the Face of Financial Distress

Environment and Results:

Organization: 1. A culture built on Heroics 2. Bad relationship w/ vendor (expiring K) 3. Siloed organization 3. Assigning blame 4. Resistance to change 5. No priorities

Results:

1. Moved to top 10% against benchmark

2. Reduced Budget 27% 3. Increased customer satisfaction 16% 4. Reduced SLA exceptions by 77% 5. Reduced incidents by 38% 6. Improved incident resolution from 5 days to 2 days

How:

Combination: 1. Leadership 2. Soft skills introduction 3. Sustained “drumbeat” 4. The persuader 5. ITSM principles

Method: 1. Plan 2. Implement 3. Absorb 4. Measure

“Be transparent, or be gone”

Phases of the Journey

Method - Plan, Implement, Absorb, Measure

Phase 1: ITSM Training April 2009 DISC (Soft Skills) Training May 2009 Capacity Management May 2009 Availability Management June 2009 Resource Planning June 2009 Project Prioritization June 2009

Phase 2: Network & Platform Services Reorganization October 2009 Service Design Package November 2009 Monthly MAP meetings March 2010

Phase 4: Network Operations Center January 2011

ITSM Full Suite (BMC – “AMP”) March 2011

Phase 3: Marketing Brochure (Mini Service Catalog - BRAND) October 2010

Service Desk re-implementation December 2010

Phase 5: Re-benchmark AIMM July 2011

Service Catalog – Version 1 October 2011 Customer Ranking of Services January 2012 Service Catalog – Version 2 February 2012

ITSM Reorganization Future

Service Level Management Future

Further Automate Capacity & Availability Future

2009 2010

Chaos

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

NPS Reorganized: -Plan & Design -Build & Transition -Operate & Maintain

ACS Program Executive hired

Availability Reporting

Capacity Reporting

(DISC)

Resource Planning

Project Prioritization

AIMM (HP) Report 46 -Technology & Architecture -Mgt Tools & Process Maturity -Culture & IT Staff Maturity -Demand, Supply & IT Governance

“One Team”

Training

Ticket Aging Process

Service request turnaround improvements (100%)

Incidents – from over 5 days to 1

Not a single failed non-standard RFC

Not a single Service Level Breach

CTO Portal

Incident Mgt Process

Customer Survey (Jul – Dec 09)

Customer Survey (Jan – Jun 10)

Communication Team Formed

ITSM Integrated Suite Service Desk

County Connection: OCDC

Data Center Tour Tech Fest

Marketing Brochure “Agency Success is

Our Mission”

IT Service Management Path to Maturity

AIMM (HP) Report 34

SDP Process

Certifications: 95

Driving Continuous Improvements

Performance Audit Report

“A journey of a thousand miles begins with a single step”

Phase 1

Phase 2

Phase 3

2011 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

Network Operations Center Launched

ITSM Integrated Suite Launch – FULL SUITE (CMDB, Change, Release…)

MSB Planning Success

Virtual Server Farm Replacement

Method

Application Monitoring

Driving Continuous Improvements

Service Level Management

ITSM based Reorganization

Customer Survey (Jul – Dec 10)

AIMM (HP) Report 10

2012 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

IT Service Management Path to Maturity

Service Catalog v 1 Service Catalog v 2

Customer Grading by

Service

Phase 4 Phase 5

Phase 1

DISC Assessment

Phase 1: Dominance, Influence, Steadiness, Compliance

0

50

100

150

200

250

300

350

400

Nov Mar April May June Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jul Aug Sep

Cumulative Monthly Accomplishments

Management Action Plan

Results through Present

2010 2011

Table of Contents Executive Summary............................................................................................................ 4

Dashboard Summary........................................................................................................... 4

Risks.................................................................................................................................... 4

Warnings ............................................................................................................................. 4

Scope................................................................................................................................... 5

Capacity Measurements ...................................................................................................... 5

Network .......................................................................................................................... 5

County Wide Area Network ....................................................................................... 5

Time Warner Internet.................................................................................................. 6

Security ........................................................................................................................... 7

Storage ............................................................................................................................ 8

Tier-1........................................................................................................................... 8

Tier-2........................................................................................................................... 8

E-Mail ........................................................................................................................... 10

Virtual Environment ..................................................................................................... 13

Analysis Additional Capacity ................................................................................... 14

Conclusion ................................................................................................................ 14

Recommendation ...................................................................................................... 14

Power5 Systems............................................................................................................ 15

Mainframe..................................................................................................................... 18

Power ............................................................................................................................ 19

Data Center Space......................................................................................................... 22

Floor Space ............................................................................................................... 22

Office Space.............................................................................................................. 24

Telephone Services ....................................................................................................... 25

APPENDIX – A – Report Data Source and Contributors ................................................ 26

APPENDIX – B – Virtual Environment Cluster Information .......................................... 27

Capacity Report Availability Report

Capacity & Availability Reporting

Phase 1

Capacity Report

Capacity & Availability Reporting

Phase 1

Values

Row Labels ETC Hours 3 Week FTE

Enterprise Infrastructure Srvcs 13.92 0.12

Networks and Platform Services 523.87 4.37

Security & Bus Continuity Oper 15.50 0.13

Grand Total 553.28 4.61

Resource Planning

Key Metrics

3 Week Summary (by organization unit):

Resource Planning

Key Metrics

3 Week Summary (by project):

Projects with late tasks:

Row Labels Sum of ETC

Disaster Recovery Implementation (FY11-12) 336.00

Enterprise Secure File Transfer Migration 33.20

ePages Replacement 22.00

Exchange 2010 Production Build out 112.00

HR & FS Access Definition Project Schedule 1.50

Secure FTP Infrastructure Upgrade 48.58

Grand Total 553.28

Sum of ETC

Row Labels Grand Total

ePages Replacement 10.00

Grand Total 10.00

Phase 2

2009 2010

Chaos

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

NPS Reorganized: -Plan & Design -Build & Transition -Operate & Maintain

ACS Program Executive hired

Availability Reporting

Capacity Reporting

(DISC)

Resource Planning

Project Prioritization

AIMM (HP) Report 46 -Technology & Architecture -Mgt Tools & Process Maturity -Culture & IT Staff Maturity -Demand, Supply & IT Governance

“One Team”

Training

Ticket Aging Process

Service request turnaround improvements (100%)

Incidents – from over 5 days to 1

Not a single failed non-standard RFC

Not a single Service Level Breach

CTO Portal

Incident Mgt Process

Customer Survey (Jul – Dec 09)

Customer Survey (Jan – Jun 10)

Communication Team Formed

ITSM Integrated Suite Service Desk

County Connection: OCDC

Data Center Tour Tech Fest

Marketing Brochure “Agency Success is

Our Mission”

IT Service Management Path to Maturity

AIMM (HP) Report 34

SDP Process

Certifications: 95

Driving Continuous Improvements

Performance Audit Report

“A journey of a thousand miles begins with a single step”

Phase 2

ASG

IRM

Process

Security

NPS

OPS

TSD

PMO

BA

Plan & Design

Build & Transition

Operate & Maintain

ITSM Organizational Design

Phase 2: Reorganization of NPS

Pla

n a

nd D

esig

n

Overall Architecture and Technologies / Standards*

Capacity Requirements

Availability Requirements

Training Requirements

POC requirements

Design Diagrams and Requirements

MOUs/SLRs

Success Criteria

TCO Calculations

Alternatives

Backup requirements

Process / Procedure requirements

Monitoring requirements

Access Requirements

Standard Requests

Engineering and Ops review and acceptance

Continuous improvement or scalability plans

Approval of the Design from Peer Groups

Approval of the design from Chief Architect

Approval of the design from Customer

Bu

ild a

nd T

ransitio

n

Design Validation / Review and Acceptance

Procurements

Configuration and Installation

Update and Audit of Asset Lists/CMDB

RFC Execution and Coordination/Communication

Rollout Plan

Backout Plan

Risk Assessment for RFC

Ops Training and Handoff

QA and Validation of Systems

Feedback to Design Team

Approval of Rollout from Ops

Approval of Rollout from customer

Approval of Rollout from Design group

Support and monitoring during cool down/warranty period

Op

era

te a

nd

Ma

inta

in

System acceptance

Registering application in ops (contact information, service levels, monitoring and backup information, new or changed procedures, Access

levels)

Update procedures

System lockdown

QA and Validation of Monitoring, Backups and Access

Assist in Negotiating SLRs/SLAs

Freeze and Warranty Period Management

ITSM Organizational Design

SDR SDP – Issues identified through the process

Phase 3

2009 2010

Chaos

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

NPS Reorganized: -Plan & Design -Build & Transition -Operate & Maintain

ACS Program Executive hired

Availability Reporting

Capacity Reporting

(DISC)

Resource Planning

Project Prioritization

AIMM (HP) Report 46 -Technology & Architecture -Mgt Tools & Process Maturity -Culture & IT Staff Maturity -Demand, Supply & IT Governance

“One Team”

Training

Ticket Aging Process

Service request turnaround improvements (100%)

Incidents – from over 5 days to 1

Not a single failed non-standard RFC

Not a single Service Level Breach

CTO Portal

Incident Mgt Process

Customer Survey (Jul – Dec 09)

Customer Survey (Jan – Jun 10)

Communication Team Formed

ITSM Integrated Suite – Service Desk

County Connection: OCDC

Data Center Tour Tech Fest

Marketing Brochure “Agency Success is

Our Mission”

IT Service Management Path to Maturity

AIMM (HP) Report 34

SDP Process

Certifications: 95

Driving Continuous Improvements

Performance Audit Report

“A journey of a thousand miles begins with a single step”

Phase 3

It’s working...

Phase 3: “Mojo”

November 2009

Not a single Service Level Breech

Incidents – from over 5 days to 1

Not a single failed non-standard RFC

Service request turnaround improvements (100%)

- Marshall Goldsmith

Marketing Brochure

Phase 3: “Mini Service Catalog”

Marketing Brochure

Phase 3: “Mini Service Catalog”

Phase 4

Phase 4

Network Operations Center

Network Monitoring

Handoff/Review Logs

Orion Network Dashboard

Daily Walkthrough AM/PM

IP Monitor Int/Ext/DR

INMON

Server Monitoring

Handoff/Review Logs

Orion Server Dashboard

VM Console

IP Monitor Probation/Int

HP SIM

Results Custom Written

Phase 4: BMC Suite through the Cloud

Integrated ITSM Suite

More Results

Results

Evidence of Improvements

Incident Tickets

Percent Resolved

0

100

200

300

400

500

600

700

800

50%

60%

70%

80%

90%

100%

Jan

-09

Feb

-09

Mar

-09

Ap

r-0

9

May

-09

Jun

-09

Jul-

09

Au

g-0

9

Sep

-09

Oct

-09

No

v-0

9

De

c-0

9

Jan

-10

Feb

-10

Mar

-10

Ap

r-1

0

May

-10

Jun

-10

Jul-

10

Au

g-1

0

Sep

-10

Oct

-10

No

v-1

0

De

c-1

0

Jan

-11

Feb

-11

Mar

-11

Ap

r-1

1

May

-11

Jun

-11

Jul-

11

Au

g-1

1

Sep

-11

Results

Evidence of Improvements

0

100

200

300

400

500

CY2009 CY2010

SLA Exceptions Reduced by 77%

Results

Evidence of Improvements

0

2

4

6

8

10

12

14

CY2009 CY2010

Average Days to Resolve Tickets Reduced 75%

Results: HP’s Converged Infrastructure Maturity Model

Evidence of Improvements

Results: HP’s Converged Infrastructure Maturity Model

Evidence of Improvements

Evidence of Improvements

1.00

1.50

2.00

2.50

3.00

3.50

4.00

Technology &Architecture

Tools & Processes Culture & IT StaffMaturity

Demand, Supply & ITGovernance

Orange County

US Public Sector

Orange County versus US Public Sector Benchmark July 2011

Results: HP’s Converged Infrastructure Maturity Model

Evidence of Improvements

Results: HP’s Converged Infrastructure Maturity Model

0

10

20

30

40

50

60

70

80

90

100

Dec-09 Jul-10 Jul-11

Better

Equal

Orange County Percent of Characteristics "At or Better" July 2011

Head

count

- 27%

Top 10%

Customer Survey Scale: 1 = worst; 4 = best

Evidence of Improvements

Phase 5

2011 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

Network Operations Center Launched

ITSM Integrated Suite Launch – FULL SUITE (CMDB, Change, Release…)

MSB Planning Success

Virtual Server Farm Replacement

Method

Application Monitoring

Driving Continuous Improvements

Service Level Management

ITSM based Reorganization

Customer Survey (Jul – Dec 10)

AIMM (HP) Report 10

2012 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

IT Service Management Path to Maturity

Service Catalog v 1 Service Catalog v 2

Customer Grading by

Service

Phase 5

Sorted Top to Bottom (and vice-versa)

Sorted by: Customer

Ave

1 Enterprise Time Synchronization 96.50

2 Midrange Platform (AIX Platform) 96.50

3 Mainframe Platform 96.00

4 Enterprise Network Connectivity (WAN) 94.71

5 Enterprise Secure File Transfer Services 94.20

6 Enterprise RightFax Services 94.00

7 SLA1 – Unmanaged Server Hosting 94.00

8 Database Mgt – Information Resource Mgt 93.67

9 Enterprise Domain Name Resolution 93.20

10 Blackberry Services 92.67

11 OnBase 92.60

12 VTL Replication Service 91.67

13 Standard Backup Service 91.50

14 Enterprise Email Services 91.33

15 Finance Services 91.33

16 Enterprise Storage Services (SAN) 91.25

17 Enterprise Internet Service Provider (ISP) 90.86

18 Technical Consultation 90.80

19 Applications Services Group 90.50

20 Desktop Support 90.00

21 Enterprise Global User Directory Services 88.67

22 Publishing Services 88.67

23 Enterprise Network Security 88.50

24 DeltaCopy Data Protection Servce (DCDP) 88.33

25 Business Recovery Center 88.33

26 SLA3 Service 87.80

27 Enterprise Remote Access Infrastructure 86.86

28 OCTNET 86.39

29 Project Management 85.00

30 Service Desk 84.78

31 eGov Services 83.33

32 Enterprise Secure Web Filtering 82.50

33 Disaster Recovery 66.50

Sorted by: Customer

Ave

1 Disaster Recovery 66.50

2 Enterprise Secure Web Filtering 82.50

3 eGov Services 83.33

4 Service Desk 84.78

5 Project Management 85.00

6 OCTNET 86.39

7 Enterprise Remote Access Infrastructure 86.86

8 SLA3 Service 87.80

9 DeltaCopy Data Protection Servce (DCDP) 88.33

10 Business Recovery Center 88.33

11 Enterprise Network Security 88.50

12 Enterprise Global User Directory Services 88.67

13 Publishing Services 88.67

14 Desktop Support 90.00

15 Applications Services Group 90.50

16 Technical Consultation 90.80

17 Enterprise Internet Service Provider (ISP) 90.86

18 Enterprise Storage Services (SAN) 91.25

19 Enterprise Email Services 91.33

20 Finance Services 91.33

21 Standard Backup Service 91.50

22 VTL Replication Service 91.67

23 OnBase 92.60

24 Blackberry Services 92.67

25 Enterprise Domain Name Resolution 93.20

26 Database Mgt – Information Resource Mgt 93.67

27 Enterprise RightFax Services 94.00

28 SLA1 – Unmanaged Server Hosting 94.00

29 Enterprise Secure File Transfer Services 94.20

30 Enterprise Network Connectivity (WAN) 94.71

31 Mainframe Platform 96.00

32 Enterprise Time Synchronization 96.50

33 Midrange Platform (AIX Platform) 96.50

A

B

Service Catalog

ISP

Service Catalog

Storage Area Network

Extend ITSM Based Organization

Move the Plan & Design Group and the Build & Transition Group “up”

Conclusions and Lessons Learned

Conclusions & Lessons Learned

1. Financial distress clouds have their silver linings in IT environments. a. Working together in the face of a single galvanizing event

b. Speak with “one voice”

2. It Takes an IT mojo-driven strategy to implement sustained continued improvements in

financial distress. a. Courage and consistency; buy-in

b. Incremental changes (process, organizational), planning and soft skills provided results

3. New leadership can ignite IT Mojo. a. Common goals

b. Common messages

4. Establish a drumbeat to pace the changes and measure visibly and consistently. a. Messaging consistency

b. Allow the organization to digest changes

5. Establish key measurements of the categories shown in the customer surveys. a. Report on them

b. Make sure everyone knows what’s being measured

6. Establish a consistent program of accountability . a. Report on them

b. Make sure everyone knows what’s being measured

7. Benchmark the organization before commencing the journey.

Questions and Answers