st. louis cmg show & tell - wells fargo server capacity & performance management february...
TRANSCRIPT
St. Louis CMG
Show & Tell - Wells FargoServer Capacity & Performance Management
February 23, 2010
About Wells Fargo Wells Fargo & Company (NYSE: WFC) is a diversified financial services
company providing banking, insurance, investments, mortgage and consumer finance. One in three households in America does business with WFC. WFC has $1.2 trillion in assets and more than 279,000 team members across 80+ business. WFC is ranked 4th in assets, and 3rd in market value of our stock among US peers (9/30/09). WF Advisors is the 2nd largest retail brokerage in the US.
WFC has: More than 10,000 stores
More than 12,000 ATMs
More than 70 million customers
Around 100K MIPS in production
Over 40,000 servers
Around 200 data center locations
Technology and operations budget of around $5 billion
3
Agenda Our risk based approach to server Capacity Management
How do we deliver on this approach?
ECMP
Integrating with ECMP
The Tool Set
How do we measure ourselves?
Plans for 2010
Our Approach To Capacity PlanningOur risk managed model has a bias for Capacity.
Our Capacity Planning approach seeks to provide sufficient resources in the right places.
We seek to limit/reduce costs while not increasing Wells Fargo’s risk.
We seek to right-size our existing capacity by reducing/removing under-utilized resources.
Capacity Planning is the discipline of managing risks and priorities.Capacity Planning seeks to maintain an appropriate balance among differing priorities:
Cost vs. Capacity – ensure that existing processing Capacity, and Capacity to be purchased, is not only cost justifiable in terms of business need, but also makes efficient use of purchased resources.
Supply vs. Demand – ensure that the available supply of processing power matches the demands made on it by the business, both now and in the future.
The value and risk associated with each drives our approach to Capacity Planning.
We focus on delivering an optimal, cost effective, and risk balanced computing environment for our customers and team members.
5
How do we deliver this?
We provide a comprehensive set of services: Operational Capacity Management (tactical)
Resource Consumption Tracking & Reporting
Current Health Checks
Future Health Projections
Notification of Risks
Comprehensive Capacity Planning (strategic) Capacity Study (aka Determine Headroom)
Sensitivity Studies, three types:1. Scalability Study – Analyze the addition of physical resources to a fixed workload
2. Speedup Study – For an increasing workload, analyze the addition of physical resources
3. Release Study – Quantify the change in resource consumption per application release
Empirical Study (aka Benchmarking, Load Testing Analysis)
Sizing
Performance & Capacity Bottleneck Analysis (tactical)
Optimization Analysis
Manage External Engagements for C/PM Services
Service Readiness
ServiceIn
Service
Cost &
Recovery Mode
l
Engagement Model
Comm. Plan
Metrics
Defined &
Documented
Maturity
Operational Capacity Management
Capacity Study
Scalability Study
Speedup Study
Release Study
Empirical Study
Sizing TBD
Bottleneck Analysis
Optimization
External Engagements
ECMP Overview
3Q09
7
The Enterprise Capacity Management Practice
The Enterprise Capacity Management Practice: ECAP provides oversight of capacity planning and functions to increase Capacity
Management Capability across the enterprise. Provides: Oversight, Process, and Tools Supports: The Enterprise Capacity Vision
The ECMP Process: Standard enterprise procedures for planning and adjusting current and future capacity
needs to meet end user availability with acceptable levels of risk and investment Enables an enterprise-wide view of:
Capacity planning; Risks; and Investments Drives predictive planning practices Facilitates reviews and reporting of high priority risks
The Tool: A common repository for critical application/system capacity planning information. Documentation trail to show Wells Fargo has performed due diligence in the event of
an audit. Training and Education material
• Goals in Product Terms
• Business Drivers
Business Partners (LOB) & Product Manager
Resource utilization thresholds
SLM thresholds
Tuning
Analysis
Monitoring
Implementation
Tuning
Analysis
Monitoring
Implementation
Tuning
Analysis
Monitoring
Implementation
CAP PLAN
BCM
SCM
Capacity Management Database (CDB)
ResDatasvc
Data
RCM
Risks
• ECM Stewardship• Planning Tools
Process Oversight (ECAP)
Business data
dem
an
d
su
pp
ly
• IT Resource Management
• Resource Provisioning
TIS & Open Systems
Enterprise Communication &
Reporting
View in Slide Show mode (4 clicks)
BizData
M/Ware - ECAP10.2009
• Liaison to CIO organizations
• Bridge CIO/ACL to IT Infrastructure
Service Delivery (SDT)
Integrating with ECMP – Future Process
9
Integrating with ECMP - Overview
10
Integrating with ECMP – TIS Role
The Tool Set Data Collection: HP OVPA
Diagnostic Tool: HP OVPM
Data Visualization & Reporting: HP OVPI
Automation & Analytics Tool: Performance Surveyor
Modeling Tool Set: Hyperformix Capacity Manager and Hyperformix Performance Optimizer
Benefits:
A diagnostic tool that provides comprehensive real-time metrics without impacting or logging into the monitored server
An automated analysis capability providing the ability to scale to over 40,000 servers without proportionately scaling staff
Full and complete reporting/analytics capability with scheduling and automated report distribution/notification
ITIL and ITSM/BSM compliant
Tool set extensible to other ITIL service management processes and databases (CMDB, incidents, change control, etc)
Tool set extensible to other data sources containing business volume measures, growth forecasts, application usage statistics, load test results, etc.
11
12
How do we measure ourselves? With KPIs & Metrics
Tool Coverage & Licensing License usage
Installations & compliance
Service Usage & Penetration Monthly health check usage by Community
Issues & risks by Community
Productivity, Practicality, & Effectiveness Issues & risks volumes – open, closed, in progress
Recommendations implemented
Tool Coverage
13
-
2,000
4,000
6,000
8,000
10,000
12,000
14,000
16,000
18,000
Total Channel Unassigned
Corp IS Corp TGS Corp TIS EFO Lending Retail Wholesale
Wells Fargo - OVPA CoverageDecember 2009
Total Server Count
Exemptions
Servers Collecting - OVPA
Servers Collecting - OVO
Outstanding
Tool Licensing
14
-
5,000
10,000
15,000
20,000
25,000
30,000
Jul-
09
Se
p-0
9
No
v-0
9
Jan
-10
Ma
r-1
0
Ma
y-1
0
Jul-
10
Se
p-1
0
No
v-1
0
Co
un
t
HP Open View Performance Agent License Counts & Forecasts
Servers w/Agent
Servers w/Data
Licenses
Forecast Sep-10 Retained LWB
Forecasting starts Feb '10
Service Usage & Penetration
15
Productivity, Practicality, & Effectiveness
16
Out of Scope Servers - LWBAuto Exemptions
1) All VMWare guests (rooms) are exempted. Their data is collected at the VMWare ESX host (hotel) level.
2) By the following tables:
Community Locations Oses Roles Server TypesEvergreen EVERGREEN DR Apple Critical Workstation Blade Chassis
EVERGREEN INVESTMENTS Blade Chassis Lab Hardware Domain Controller ServerSEAGRAMS BUILDING Fujitsu Chassis Lab Hardware- Isolated Network Domain Name Server
Fujitsu SMC LOB Config File and Print Services ServerHP Superdome Chassis Powered Off DR File ServerIBM Chassis Pre-Implementation Finance Center ServersIBM HMC SA Config FTP/NDM/File TransferMP-RAS Spare (Decommissioned) Grid ServerNetApp Standard (unallocated) HMC ServerNetware Workstation HP Ignite ServerVIO HP Physical Server (No OS)VxWorks HP System Management System
IBM Physical Server (No OS)Jump ServerKeon MasterLog ServerPrint ServerSAN Manager ServerSun Physical Server (No OS)Telephony ServersTeradata Administrative WorkstationTeradata ChassisTerminal ServerTSM Backup ServerVIO Server
18
Plans for 2010 Engagement model & communication plan
Reporting & analysis expansion Virtualization analysis – VMware & AIX
Proactively identify virtualization candidates
Utilization & performance dashboards (DC, LOB, …)
New service – future health projections
More work in KPIs & metrics Service quality & usage
Team productivity & efficiency
Marketing deck
Analyst handbook