tuning oracle rac wednesday, june 04, 2014 guy peleg president maklee engineering...

Tuning Oracle RAC

Monday, April 10, 2023

Guy PelegPresident

Maklee Engineeringguy.peleg@maklee.com

Agenda

RAC overview & Performance ExpectationsPerformance Tips SQL Tuning

Typical RAC configuration

Public Network

Private Network(Interconnect)

Storage Network

AppServer1

AppServer2

Client

Node 1

DatabaseInstance 1

Node 2

DatabaseInstance 2

Local Storage Local StorageShared

Database

Oracle RAC

Oracle RAC provides two main features:Availability

Scalability

May operate in two modes:All nodes are active (load distributed between nodes)

Active/Passive

RAC scaling/performance considerations are similar to OpenVMS clustering scaling/performance considerations

Interconnect

Locks

Sharing

RAC Scaling – Maklee’s Golden Rules

Application that does not scale on a standalone node – will not scale on RAC

Start with single instance tuningshutdown all nodes measure scalingtest scaling by adding CPUs

Add one node at a time to measure scalability

Scalability Benchmark

2 nodes cluster1.3 Ghz rx2600, running OpenVMS V8.3-1H1

Oracle 10gR2 RACLatest set of patches

Test database contains information about 50,000 customers

200,000 customer orders

200,000 ordered items

Scalability Benchmark

PL/SQL procedure to fetch data about 2000 random customers

Read only test

All data in SGA

No I/O

CPU Bound

Scalability Benchmark

0

200

400

600

800

1000

1200

1400

1 Job

6 Parallel jobs -one instance

3 Parallel jobsinstance 1

3 Parallel jobsinstance 2

Elapsed time (seconds per job) to complete the testLess is better

Elapsed time (seconds per job) to complete the testLess is better

RAC Proof Of Concept

MAKLEE Engineering recently performed a RAC proof of concept installation at a large chain of department stores in Switzerland.

Benchmarked a single Alpha GS1280 (production node) vs. a RAC cluster running 2 Integrity servers rx6600.

The goals were:Install RAC Get hands on experience with RACPerform RAC scaling testsMake a go/no go decision on implementing RAC in production

Hardware & Software Configuration

Oracle RAC Configuration:2 nodes OpenVMS ClusterEach node is rx6600 with 8 coresOpenVMS V8.3-1H1EVA8000 storage

Products installed:Oracle CRS (Cluster Ready Services)Oracle 10g R2DBCA executed for configuring RAC enabled databaseDatabase patches

27 Parallel Database Import Jobs

Minutes to complete database import

less is better

0

20

40

60

80

100

120

140

160Standalone GS1280

rx6600RAC/standaloneItanium

Database Import

Itanium outperformed Alpha

Operating in RAC environment does not increase the throughput of the import operation

Spreading the jobs across two nodes or running all jobs on one node yields identical performance/throughput

No performance degradation witnessed

Batch Processing Benchmark

0

5

10

15

20

25

30

35

40

4554 jobs - SingleAlpha GS1280

54 jobs - Singlerx6600

54 jobs - Spreadacross the RAC

Minutes to complete batch processing cycle

Less is better

Minutes to complete batch processing cycle

Less is better

Batch Processing Benchmark

Itanium outperformed Alpha

RAC allows scaling outside of the box

Second RAC node adds 40% more throughput

Another Example – European Bank

European Bank migrating from Alpha to Itanium

2 nodes AlphaServer ES47 -> 2 nodes rx7640

Migrating to Oracle 10gR2 RAC

Availability is main concernInteractive users will be distributed between nodesNo plans to distributed batch load between nodesNeeded to verify that RAC does not degrade performance

Another Example – European Bank

Benchmarked various batch jobs – focusing on one specific batch job.

Initial results did not favor Itanium.

Batch Processing Benchmark

0

5

10

15

20

25

30

35Alpha ES47

rx7640 out of thebox

rx7640 after tuning

Minutes to complete selected batch job

Less is better

Minutes to complete selected batch job

Less is better

European Bank - Summary

Tuning is critical for achieving optimal performance

Don’t run “out of the box”.66% improvement after (minimal) tuning

The specific benchmark is running 52% faster on Itanium comparing to Alpha.

European Bank - Summary

All other batch jobs/applications witnessed similar improvement.

RAC increases availability and does not degrade performance.

RAC will go into production in few weeks

Performance Tips

CRS Base Priority

CRS is running in batch

Usually, runs in a dedicated batch queue

By default, base priority of a batch queue is 4

On a system with thousands of processes, CRS may need to compete (and sometimes lose) for CPU resources

CRS should be given high priority

Set base priority of CRS queue to 12

RAC Cluster Interconnect

The performance of the cluster interconnect is critical to the performance of the RAC.

Interconnect used forCluster managementLocksCache Fusion

Oracle requires (at least one) dedicated cluster interconnect

Gigabit Ethernet is highly recommendedEnable Jumbo FramesTransfer rate of ~ 25MB per second (faster than some disks ;-)

Cluster interconnect Performance

Latency is CRITICAL for RAC performance

Measure the latency of the interconnect:

set numwidth 20

column "AVG CR BLOCK RECEIVE TIME (ms)" format 9999999.9

select

b1.inst_id,

b2.value "GCS CR BLOCKS RECEIVED",

b1.value "GCS CR BLOCK RECEIVE TIME",

((b1.value/b2.value) * 10) "AVG CR BLOCK RECEIVE TIME (ms)"

from gv$sysstat b1,

gv$sysstat b2

where b1.name='gc cr block receive time'

and b2.name='gc cr blocks received'

and b1.inst_id=b2.inst_id;

Cluster interconnect Performance

Latency should be lower than 15ms

OpenVMS achieved 0.5ms onblades RAC (BL860)V8.3-1H1Gigabit EthernetJumbo Frames enabled

Load distribution between instances

set pagesize 60 space 2 numwidth 8 linesize 132 verify off feedback offcolumn service_name format a20 truncated heading 'Service'column instance_name heading 'Instance' format a10column service_time heading 'Service Time|mSec/Call' format 999999999select service_name,  instance_name,  elapsedpercall service_time,  cpupercall cpu_time,  dbtimepercall db_time,  callspersec throughput  from gv$instance gvi,   gv$active_services gvas,   gv$servicemetric gvsm  where gvas.inst_id=gvsm.inst_id  and gvas.name_hash=gvsm.service_name_hash   and gvi.inst_id=gvsm.inst_id   and gvsm.group_id=10  order by   service_name,   gvi.inst_id;

Standalone Database Import

Minutes to complete database import

less is better

0

1

2

3

4

5

6

7

8

rx6600 beforetuning

rx6600 after tuning

37% Improvement

Database import

Install imp.exe as resident image with shared address space$ install add imp.exe/resident/share=addr

Increase default quotas for BEQ’s mailboxes$ define/sys ORA_BEQ_MBXSIZ 64000$ define/sys ORA_BEQ_MBXSBFQ 64000

Set DEFMBXBUFQUO to 64000Set DEFMBXMXMSG to 64000

DBMS_STATS.GATHER_SCHEMA_STATS

0

10

20

30

40

50

60

70

80

90

100

rx6600 beforetuning

rx6600 after tuning

Minutes to gather database statistics (350GB database)

Less is better

Minutes to gather database statistics (350GB database)

Less is better

DBMS_STATS.GATHER_SCHEMA_STATS

Calling gather_schema_stats results in a database server process being created

The server process in not multithreaded

Typically consumes 100% of one CPU

Performance improvement achieved by affinitizing the server process to one CPU and increasing QUANTUM to 20.

SORT

Analyze the efficiency of sort operations

Determine the number of optimal, one pass and multipass operationsSELECT optimal_count, round(optimal_count*100/total, 2) optimal_perc, onepass_count, round(onepass_count*100/total, 2) onepass_perc, multipass_count, round(multipass_count*100/total, 2) multipass_percFROM (SELECT decode(sum(total_executions), 0, 1, sum(total_executions)) total, sum(OPTIMAL_EXECUTIONS) optimal_count, sum(ONEPASS_EXECUTIONS) onepass_count, sum(MULTIPASSES_EXECUTIONS) multipass_count FROM v$sql_workarea_histogramWHERE low_optimal_size > 64*1024);

Sizing the SGA

Reserve memory for the SGA (SYSMAN)

Avoid automatic memory management in the SGA whenever possible.

The following query will help properly size the SGA

select sga_size, sga_size_factor as size_factor,estd_physical_reads as estimated_physical_reads from v$sga_target_advice order by sga_size_factor;

Sizing the SGA

SQL> select sga_size, sga_size_factor as size_factor,

2 estd_physical_reads as estimated_physical_reads

3 from v$sga_target_advice order by sga_size_factor;

SGA_SIZE SIZE_FACTOR ESTIMATED_PHYSICAL_READS

---------- ----------- ------------------------

4356 ,75 44485808

5808 1 24659539

7260 1,25 24659539

8712 1,5 24659539

10164 1,75 24659539

SQL>

What’s wrong in this picture?

$ show memory

System Memory Resources on 1-APR-2008 15:32:35.62

Physical Memory Usage (bytes): Total Free In Use Modified

Main Memory (GB) 64.00 58.27 5.69 0.02

Extended File Cache (Time of last reset: 31-MAR-2008 15:14:46.99)

Allocated (MBytes) 397.03 Maximum size (MBytes) 32768.00

Free (MBytes) 17.82 Minimum size (MBytes) 3.12

In use (MBytes) 379.20 Percentage Read I/Os 77%

Read hit rate 99% Write hit rate 0%

Read I/O count 5368075 Write I/O count 1578011

Read hit count 5315683 Write hit count 0

Reads bypassing cache 79 Writes bypassing cache 241954

Files cached open 739 Files cached closed 2255

Vols in Full XFC mode 0 Vols in VIOC Compatible mode 52

Vols in No Caching mode 0 Vols in Perm. No Caching mode 0

....

Of the physical memory in use, 8.52 GB are permanently allocated to OpenVMS.

$

SQL Tuning

The next step in improving performance

SQL Tuning !

With previous “Alpha Vs. Itanium” benchmarks we had to play it fare

Not a single SQL statement was changed.

SQL tuning may improve performance by magnitudes

SQL Tuning

All the tools that are required for SQL tuning are shipping with the database:

Automatic Workload Repository (AWR)Endless amount of performance related informationEnhanced version of statpak

Active Session History (ASH)Automatic Database Diagnostic Monitor (ADDM)SQL Access AdvisorSQL Tuning Advisor

Statspack analyzer (not part of the DB but available for free)

The power of SQL tuning

AWR was used to analyze the “scalability benchmark”97% of the time was spent executing single SQL statementAfter SQL tuning – elapsed time of the benchmark was reduced from 411 seconds to 3.18 seconds !

0

50

100

150

200

250

300

350

400

450

Untunedversion

Tuned version

130 times

faster!!!!

The power of SQL tuning

“Real life” examplerx6600, Oracle 10g, DWH DBSingle SQL statement required 140 minutes to completeBy biasing the optimizer, elapsed time reduced to 10 minutes

0

20

40

60

80

100

120

140

Untunedversion

Tuned version

See us at www.maklee.com for:

• Performance improvements

• Oracle Tuning

• Platform Migration

• Custom Engineering solutions

• Custom Training

Questions?