presentation racsig 090730

Managing Performance and Availability for 25,000 Siebel Contact Center Users with Oracle Real Application Clusters

Roland Höhl, Perot Systems / Deutsche Telekom AG 30rd of July, 2009RAC SIG Webseminar

[email protected]@telekom.de

2

Introductions Project Overview & Business Drivers

Scope and Usage of the system Technical Footprint - Handling 25,000 Users

Architecture & Infrastructure Session Management RAC Performance System Monitoring Reporting

System Interfaces Lessons Learned Future plans

Agenda

3

Who am I / Biography Roland Höhl

46 years old

Studied at the university in Darmstadt / Germany

Perot Systems employee Manager Application Development

Working for Deutsche Telekom as external consultant in the CRM-T project since 2006

In the Deutsche Telekom CRM project responsible for technical architecture infrastructure sizing and planning release management

4

Company overview – Deutsche Telekom

Internationalization: Deutsche Telekom is represented in ~ 50 countries worldwide..

Employees: Approx. 242,000 employees worldwide (December 31, 2007)

Revenue: 62.5 billion euros (2007 financial year)

Figures T-Home 37 Million Fixed Line User (PSTN) No.1 in German DSL-Market

(total amount12, 5 Mio Customer) 150.000 new Customer for IPTV (Entertain) in 2007 1.800 Sale points (Biggest integrated sales- and

Service organization for Fixed and Mobile products)

5





Agenda

6

Facts and figures Project scope

Siebel upgrade from 7.0 to 7.8.2.5 Change from DB2 Host (OS390) to Oracle 10g RAC Unix Reimplementation of customized Siebel business processes Data migration from DB2 to Oracle 10g (based on EIM load) Building a ESB around Siebel CRM (based on MQSeries)

Project staffing: Oracle – Software vendor (Siebel / Database) IBM Global Services – Development Siebel / EAI T-Systems - Operations and hosting

Duration: Project start: 11/2005 First release: 04/2008 Involved employees: ~800

7

Project GoalsStrategic Goals

Improve customersatisfaction and raise call center efficiency

Time reduction to develop new

products and services

Cost reduction for operations,

maintenance and enhancements

Standardization, risk mitigation and

protection of investment

Functional Scale identical with CRM-C V2.130(KM, KKM, OM)

Upgrade to version:Siebel Standard SW 7.8

Align reduction of the application toSiebel Standard

Reduction of interface complexity(Service oriented integration platform)

Hardware change – Change of database platform if applicable

Master function of CRM-T for customer andcontract data (End of parallel operation, KoSi)

Improved application operation, incident and problem management

Operative Goals

8

Project Business Drivers Advantages for the customers:

Improved customer service Increased customer satisfaction

Advantages for the employees: Enhanced graphical user interface Simplification of IT systems Single and completed view of customer information Transparency of processes Decreased administrative complexity

Advantages for the Deutsche Telekom: Simplification of IT processes Improvement of IT stability Increased flexibility through open standards Retirement of legacy systems Achieved saving potentials in operations, support and development 1. Milestone in redesign of the IT architecture

9





Agenda

10

Deutsche Telekom – T-Home - CRM Landscape

2 Data Centers Data Warehouse

Legacy Systems

T-ShopT-Shop

Call CentersCall Centers

PartnerPartner

T-E

SB

11

Scope and usage

89 call center within Germany 20000 named call center users 7000 named T-Shop and partner users

80 million accounts up to 40.000 customer contacts per day

(Inbound items, i.e. Calls, Mails, Docs) ~ up to 10.000 orders / hour ~ 19 TB Data in the Database

~ 40.000.000 user requests / hour ~ 50.000.000 disk reads / hour Up to 7.000.000.000 buffer gets / hour

42 Legacy systems connected several million messages per day transferred

12





Agenda

13

Infrastructure Architecture

14

Web Server Infrastructure

Software Configuration• AIX 5.3 ML8• IBM IHS 2.0• Siebel Web Server Extension 7.8.2.5

Hardware Configuration• 12 x IBM P5 9133-55A

•4 x 1.65 GHz P5 CPUs•24 GB Memory

• CISCO CSS 1150•Frontend and Backend Load balancer

Front End Load

Balancer

Back End Load

Balancer

15

Software Configuration• AIX 5.3 ML8• Siebel 7.8.2.5• WebSphere MQ V6 (on EAI Servers only)• IBM HACMP (Cluster) on EAI Servers• Genesys CTI Driver

Siebel Application InfrastructureHardware Configuration

Users related Servers

• 6 x IBM P595 (eCommunications)•48 x 5 GHz P5 CPUs•336 GB Memory

• 3 x IBM P595 (eCommunications)•32 x 2.1 GHz P5 CPUs•128 GB Memory

• 2 x IBM P590 (Siebel eConfigurator)

EAI Server• 8 x IBM P590 (Active)

•8 x 2.1 GHz CPUs•24 GB Memory

• 8 x IBM P590 (passive – Virtual Servers)•0.1 x 2.1 GHz CPUs•24 GB Memory

Batch/Read Audit Servers• 4 x IBM P590 (active + passive)

•8 x 2.1 GHz CPUs•32 GB Memory

16

Oracle RAC Infrastructure

Software Configuration• AIX 5.3 ML8• ORACLE 10.2.0.3• ASM, Streams

SSDSANSSD

SSDSANSSD

Fibre Channel Switches

Hardware Configuration

Oracle RAC Nodes

• 4 x IBM P595•36 x 2.1 GHz P5 CPUs•312 GB Memory

SAN Storage 160 TB• 10 x IBM DS8300

•RAID 10•16 x 146 GB Disks•2 x 2 GB/sec FC

Solid State Disks• 2 x Texas Memory Systems RamSan-500

•2 TB Flash RAID•2 GB/sec Bandwidth

17





Agenda

18

Session Management for Performance, Availability, and ScalabilitySessions Handling

•Up to 20.000 user session + several hundred sessions from EAI Components will be handled on the Siebel System

•2 Options to handle them on the Database

•Oracle MTS

•Siebel Connection Pooling

EAI

DatabaseService

19

Session Management for Performance, Availability, and ScalabilitySessions Management

Siebel Connection Pooling of 5:1 for User Sessions•Session are connected via DB Services

•Up to now each session will connect via the same service•Multiply Services allow to

•Separate – Distribute – Prioritize - … load over the different RAC Nodes.

Siebel Connection

Pooling

EAI

DatabaseService

20

Session Management for Performance, Availability, and Scalability

Siebel Connection

Pooling

Resource Management

•Siebel implements Transparent Application Failover TAF internal

•Bad SQL statements have been hard to stop

•Solution was Resource Profiles

•CPU limitation

EAI

DatabaseService

DatabaseService

21





Agenda

22

RAC Performance & Availability Aspects

General memory considerations Session size up to 15 MB Keep enough memory for the operating

system 10-20% more memory on RAC environments

Amount of cluster nodes Expected load and hardware limitations

defines the minimum # of nodes More nodes provide better availability / less

failure impact (MTBF, upgrades, …) More nodes allows dedicated resource

assignment Less nodes reduce interconnect traffic,

management overhead

•Interconnect•Dedicated infrastructure•Failover Interfaces / Channel bundling

23

RAC Performance & Availability Aspects II

Disaster Scenarios and Stretch Cluster Limitations 20 – 50 km Max Disaster Recovery

Consider Rolling Upgrades Increase the availability – reduce planned down time

Define database services as soon as possible Group Servers Separate load Control performance and availability

Watch out for Architectural bottlenecks Centralized Tables, Sequences Prevent, Tune, or Invest

24

RAC Performance Configuration

Consider to use Partitioning Try to spread out load but keep the data local Consider that i.e. DB Triggers will be fired on each node

Watch out for High frequently changing DB objects Consider less records per block Less records decreases block concurrency

Less nodes request the same block at a single point in time Use caching

Increase cache size for sequences Pin small tables into the cache

Oracle features ASSM, ASM, Reverse Indexes, Function Based Indexes,

Partitioning Follow best practices

25



Architecture & Infrastructure Session Management RAC Performance System Monitoring


Agenda

26

System Monitoring

Joint Team of Admins IBM Tivoli Oracle GRID Control Quest Foglight Experience Monitor Interface monitoring

27

Joint Team of Admins

Joint team of Siebel , EAI and Database Admins Total overview of the system

Database monitoring Siebel view (functional SQL process monitoring) Database view (technical monitoring)

Queue Monitoring (Siebel inbound and outbound)

Technical Monitoring Logs, CPU, Memory, Storage etc.

Propose technical improvements

28

System Monitoring


29

IBM Tivoli

Operational Management System Automation Monitoring (CPU, Memory…)

Workload Scheduling Service Level Advisor

Storage Management: Tivoli Storage Manager SAN Volume Controller

Alerting

30

System Monitoring


31

Oracle GRID Control

RAC monitoring

Long running transactions

Wait States

32

System Monitoring

Joint Team of Admins IBM Tivoli Oracle GRID Control Quest Foglight Experience Monitoring Interface monitoring

33

Quest Foglight Experience MonitorCoverage:

Round about 60 business processes are being monitored (90% coverage)

Various report sets are available to users

Applicable Fields: Monitoring of software quality after

deployment Alarm based control of the

production environment Information base for sizing

recommendations and server consolidation

Acceptance: FxM is accepted as an objective

measuring instrument which reflects “end user reality” and is used to support SLA monitoring and reporting

34

System Monitoring

Joint Team of Admins IBM Tivoli Oracle GRID Control Quest Foglight Experience Monitor Interface Monitoring

35

Interface Monitoring The Telekom Enterprise

Service Bus

36

Interface Monitoring

Evaluation of auditing messages

Monitor traffic and response times on interfaces

37





Agenda

38

Reporting

OA Mirror (Operative analysis)

RO Mirror (Read only)

Logshipment

(arch and redo)

CRM RAC Database

Stage DB

Capture processPropagation process

RO Mirror DBPhysical Standby

read only

Logshipment(arch and

redo)

OA Mirror DB

Apply processes (16)CDC-capture

CDC-apply (8)

Data warehouse

CRM OAReports

39

CRM technical achievements

Reduced Response times Increased Automation Faster time to market and increased flexibility for new products Less development more configuration Service Oriented Architecture is implemented Host platform retired OLAP database with realtime mirroring based on Streams

technology

40





Agenda

41

System Interfaces - ESB

42





Agenda

43

Lessons Learned

Get a 360° view on the system Test with full blown database as early as possible Test the infrastructure with worst case scenarios Test the system infrastructure carefully before roll out Test your migration with real data Always have a look at system and application performance Avoid load mixing of batch and OLTP Keep commit sizes low (Batch Jobs) Keep work on a single node when possible Be prepared for long running queries Slim line tables with highly changing content

44





Agenda

45

Future Plans

Additional customer groups will be integrated and served through the new platform Small business customers Medium business customers

Additional product types will be integrated to process the complete portfolio Internet access and value added services Next generation network services

Service Architecture will be enhanced Additional services will be externalized Further automation improvements Continuous systems consolidation and renewal of IT architecture Upgrades to Oracle 11g and Siebel 8.x

46




System Interfaces Lessons Learned Future plans CRM-T Infrastructure details

Infrastructure in detail

47

CRM-T Infrastructure detailsWeb Server and Reporting

48

CRM-T Infrastructure detailsQuest Monitoring

49

CRM-T Infrastructure detailsObject manager and Batch Server

50

CRM-T Infrastructure detailsEAI Server and ESB

51

CRM-T Infrastructure detailsDatabase

52

CRM-T Infrastructure detailsStorage

presentation racsig 090730

Technology

siebel crm

crmt project

project start

new customer

mio customer

project goals strategic

siebel standard sw

mqseries project staffing