beyond 2 node rac

8/6/2019 Beyond 2 Node RAC

1/49

Beyond

2 Node RACWorkload Management

in the GridErik Peterson

Real Application Clusters Development


2/49

Agenda

A Gradual Approach to the Grid

An Architectural Mindset Change

Automatic Workload Management

More Customer Examples


3/49

Operates over3,700 retail stores

Presence in over910 cities across 18 countries/regions

13 retail brands

Manufactures a full range of leading beverages from

bottled water, fruit juices, soft drinks and tea products

Employs over64,000 staff

2003 turnoverHK$60bn


4/49

Phase 1 EBIS Single Node DB and Application server.

Database Node

Application server Node

Users

RAC being designed.

DONE

!


5/49

Phase 2 EBIS Dual Node DB and Application server.

Database Nodes with Veritas VCS

in failover mode. Failover node is

usually idle.


Users

RAC being released.

DONE

!


6/49

Phase 3 EBIS Dual Node RAC DB and Application server.

Database Nodes with Veritas DBAC cluster

and RAC. Load is balanced between bothnodes.


Users

RAC being adopted.

DONE

!


7/49

Phase 4 EBIS + PNS Dual Node RAC DB and Application server.

Database Nodes with

Veritas DBAC cluster

and RAC. Load is

balanced between both

nodes.

Application

server Node

RAC being proved.

10G being released.

EBIS

Application

server Node

Park n Shop

Retek RMS


8/49

Phase 5 EBIS + PNS Four Node RAC DB and Application server + PNS D/W.

Application

server Node

RAC being adopted.10G being proved.

EBIS

Application

server Node

Park n Shop

Retek RMS

Park n Shop + Fortress RDWBusiness benefit is that we can use spare capacity on EBIS for PnS


9/49

Phase 6 Multi-Node RAC DB and Application servers + RAC Development.

EBIS, PnS Retek RMS, PnS RDW, Fortress RDW Production Cluster.

Application

server Nodes

Development

cluster.


10/49

Phase 7 Full development and Production Cluster.

Production Cluster

Development Cluster

Intel/Linux based blade

server for applications

and web front end.

LOAD

BALANCER

LOAD

BALANCER

LOADBALANCER

Nodes can be moved from Development to

Production to cope with seasonal demands.

User Population


11/49

An Architectural Mindset Change


12/49

Monday Morning

Email Payroll OE DW

100% 20% 50% 40%


13/49

Payroll Run (night)

Email Payroll OE DW

10% 100% 10% 80%


14/49

Xmas Orders

Email Payroll OE DW

40% 20% 100% 40%


15/49

Xmas Orders Desire

Email

PayrollOE

DW


16/49

Xmas Orders:

Before Failure

EmailPayroll

OEDW


17/49

Xmas Orders:

After Failure

Email

Payroll

OE

DW


18/49

Paradigm Shift

0

500

1000

1500

2000

2500

3000

3500

4000

60~70% Grid

ComputingTraditional

Computing


19/49

Requirements

Location Transparency

Workload Balancing

Resource Management

Manageable as a Single Environment


20/49

Automatic Workload Management

Application workloads can be defined as

Services Individually managed and controlled

Assigned to instances during normal startup

On instance failure, automatic re-assignment Service performance individually tracked

Finer grained control with Resource Manager

Integrated with other Oracle tools / facilities


21/49

Services are a unit for

performance A new level dimension for performance tuning

workloads are visible and measurable tuning by service and SQL replaces session and

SQL in most systems where sessions are shared.

Performance measures for real transactions

Alerts and actions when performance goals areviolated


22/49

Automatic Workload ManagementProvides Visibility


23/49

Useful Service Views

Service status in V$ACTIVE_SERVICES,

DBA_SERVICES, V$SESSION,V$ACTIVE_SESSION_HISTORY

Service performance in V$SERVICE_STATS,V$SERVICE_EVENTS,V$SERVICE_WAIT_CLASSES,V$SERVICEMETRIC,V$SERVICEMETRIC_HISTORY

Service, MODULE, and ACTION performance inV$SERV_MOD_ACT_STATS.


24/49

Example Service Metrics

Service time - current NAME ELA(s)/CALL CPU(s)/CALL

--------------------------- ------------ ------------

ERP 0.1940 0.0082

Service time - history every 60 seconds

NAME ELA(s)/CALL CPU (s)/CALL

--------------------------- ------------ ------------ERP 0.1940 0.0082

0.2046 0.0085

0.2154 0.0093

0.2248 0.0105

0.2160 0.00970.2185 0.0104

0.2211 0.0104


25/49

Service Thresholds and Alerts

DBMS_SERVER_ALERT.SET_THRESHOLD

SERVICE_ELAPSED_TIME SERVICE_CPU_TIME

Warning and critical levels for observed periods

Import from EM baselines

Comparison of response time againstaccepted minimum levels

a desire for the wall clock time to be, at most, acertain value.


26/49

30 service, module, action statistics

user calls DB time response time

DB CPU CPU/service

parse count (total)

parse time elapsed parse time cpu

execute count

sql execute elapsed time

sql execute cpu time

opened cursors cumulative

session logical reads

physical reads

physical writes

redo size

user commits

workarea executions - optimal workarea executions - onepass

workarea executions - multipass

session cursor cache hits

user rollbacks

db block changes

gc cr blocks received

gc cr block receive time

gc current blocks received

gc current block receive time

cluster wait time concurrency wait time

application wait time

user I/O wait time


27/49

Automatic Workload ManagementProvides Visibility

0

5

10

15

20

25

30

9 10 11 12 13 14 15 16 17 18

D

C

B

A


28/49

AWR Automatically Measures

Service AWR measures response time, resource used

Automatically for work done in every service

AWR monitors thresholds, sends AWR alerts

response time, cpu used

maintains runtime history every 30 minutes

Statistics collection and tracing are HA

persistent for service location / instance restart. enabled and disabled globally for RAC.


29/49

Service High Availability using

RAC 10g Focus is on protecting the application services

more flexible/more cost effective than other HAthat focus on availability of single physical systems

Services are available continuously with load

shared across one or more instances. Any server in the RAC can offer services

in response to failures

in response to planned maintenance.

in response to runtime demands


30/49

Automatic Workload ManagementIntegration with Other Tools

Job Scheduler

Job classes mapped to services

Parallel Query / DML

Query coordinator connects to a service like any

other client

Parallel slaves inherit service from coordinator

Oracle Streams Advanced Queuing

Queues accessed by service

Achieves location transparency


31/49

Failure Notification (FaNTM) with Oracle

Application Server 10g

Fast, coordinated

recovery without humanintervention

Oracle Database 10gRAC signals JDBC

Fast Connect Failoverwhen instances failuresoccur

Immediate recovery formid-tiers

< 4 seconds from15 minutes

self correcting

Oracle Clusterware

AppServer

10g

RAC


32/49

Failure Notification (FaN)JDBC Fast Connection Failover Processing

Supports multipleconnection caches

Datasource for eachcache mapped to aservice

Keeps track of serviceand instance for eachconnection

Distributes new workrequests acrossavailable instances

SERVICE 1

SERVICE 2

SERVICE 3

JDBC / Mid-Tier Database Tier

INSTANCE X

INSTANCE Z

INSTANCE Y

CACHES


33/49

Initial State

C2

C3

C4

C5C6

C1

Instance 1

Instance 2

Service across RAC

Connection Pool


34/49

Instance Join

C2

C3

C4

C5C6

C1

Instance 1

Instance 2

Instance 3

Service across RAC

Connection Pool


35/49

Instance Join Undesired CasesOption 1: Nothing Happens

C2

C3

C4

C5C6

C1

Instance 1

Instance 2

Instance 3

Service across RAC

Connection Pool

Instance Join Undesired Cases


36/49

Instance Join Undesired CasesOption 2: Add New Connections

Randomly

C2

C3

C4

C5C6

C7

C8

C9

C1

Instance 1

Instance 2

Instance 3

Service across RAC

Connection Pool


37/49

Instance Join Desired Result

C2

C3

C4

C5C6

C7

C8

C9

C1

Instance 1

Instance 2

Instance 3

Service across RAC

Connection Pool


38/49

Rebalancing upon Instance

Join Desire: Create new connections to new

instances & potentially disable some old ones. Method:

10gR1: Fast Application Notification (FAN)

Using the HA up and down events for services inRAC

9i: DBMS session disconnect (Manual steps)


39/49

Node Leaves

C2

C3

C4

C5C6

C7

C8

C9

C1

Instance 1

Instance 2

Service across RAC

Connection Pool


40/49

Work

request

C2

C3

C4

C5C6

C7

C8

C9

C1

Instance 1

Instance 2

Instance 2

Service across RAC

Connection Pool

Bad

OK

Good

1 2

10gR2 Connection Pool Load Balancing


41/49

Customer Examples


42/49

Critical 4

Node-4

New Distribution Possible

Chicago Stock Exchange

Batch

Low

Critical 1

Node-3Node-2Node-1

Critical 2 Critical 3

Medium 1 Medium 2



43/49


Resource Management Plan -

Daytime

Critical1

50%

Remainder

25%

Remainder

30%

Low

80%

Medium 1

30%

P1

P2

P3

Critical 2

25%

Medium 2

40%

Need flexibility to change plan to meet changing demands

Batch

20%

D ll


44/49

Gains: Manages Priorities, Visibility of Use,

Can turn off reporting during aggregation

Node-1 Node-2 Node-3 Node-4

Adhoc Reports P3

Loads/Aggregation P2 0-100%0-100%0-100%0-100%

Dell

European Operations DW

0-100%0-100%

0-100%0-100%0-100%0-100%Critical Reports P1

0-100%0-100%


45/49

Talk America

DW

OLTP 1

OLTP 2

OLTP 3

OLTP 4

Node-4Node-3Node-2Node-1 Node-6Node-5

BatchRepor

ting


46/49

Consolidated eCommerce Project

Retailer

eCommerce

Backups

Order Management

Real Time

Reporting

Spare Capacity

DW


47/49

Large Node Proofpoints

Amazon.com 16 node Clickstream DW (17 TB), 8node ETL DW on Linux

a US Telecom 16 node Grid on Solaris

Telstra 8 Node - 18 TB DW on Linux

Mercadolibre 8 Node OLTP on Linux/Itanium Gas Natural 8 Node DW, 7 Node SAP BI system

AC3 - 63 Node POC on Linux)

OpenWorld Japan w/ Sun & Cisco - 128 NodeCluster


48/49

Closing Remarks

Use 4+ Nodes Greater HA & Scalability

Connect by Service: not SID, not Instance, notHost

Instrument your code to use Module & Action

Implement Services weather your DBconsists of one application or many


49/49

Q U E S T I O N SQ U E S T I O N S

A N S W E R SA N S W E R S

beyond 2 node rac

Documents