extreme performance tuning game plan for success chris lawson

EXTREME PERFORMANCE TUNING

GAME PLAN FOR SUCCESS

Chris Lawson

Agenda

What is Extreme Performance Tuning? Seven attributes of Extreme Perf Tuning Chris’ favorite tuning techniques Some pitfalls to avoid

When it doesn’t work—the missing piece of

the puzzle

A Key Question

After you attend a seminar on performance tuning, do you

discover that you have become a performance

expert?

Performance Tuning is Not:

Having really tricky scriptsRunning colorful programsDelivering long reports full of

generalitiesBeing smarter, faster, better-

looking than other DBAs

The Paradox of Performance Tuning

Being extremely good means being adept in multiple areas

A single strength will not work.Example: New DBA uses tool (e.g.,

OEM) to analyze performanceQ: What will be the likely outcome?

The 7 Facets of Extreme Performance

Scripts

Tools

Techniques

Hands-on Experience

Good Judgment

Focus

Logical Reasoning

Q: Which Ones are Taught in Seminars?

Scripts

Tools

Techniques

Hands-on Experience

Good Judgment

Focus

Logical Reasoning

Only These are Taught in Class

Scripts

Tools

Techniques

Hands-on Experience

Good Judgment

Focus

Logical Reasoning

Let’s Examine the 7

1. Good Judgment

2. Logical Reasoning

3. Good Techniques

4. Scripts

5. Tools (maybe)

6. Focus

7. Hands-on Experience

Strategy for Success

ScriptsTools

Techniques

Hands-on Experience

Good Judgment

(1)

Focus

Logical Reasoning

Good Judgment: Thinking Clearly

Good Judgment

Difficult to explain—it’s a sound approach to gray areas

Q: How do you spot this in candidates?

I don’t mean I.Q.

“The man who was too smart to talk to”

Good Judgment: Thinking Clearly

Good Judgment

Easier to spot bad judgment than good

Bad judgment stands out

Tale of the “No-good Nested Loops”

Good Judgment May be Hidden;Bad Judgment Stands Out

Good Judgment ?

“The Thrill of Thrill Hill”

The result . . .

Good Judgment

Firms want people with good judgment, but how do you measure?

Common sense reasoning maybe not so common?

Some examples of smart people using not-so-smart judgment

Common Sense Reasoning not so Common . . .

Common Sense Reasoning not so Common . . .

Sybase DBA violated SOX

Common Sense Reasoning not so Common . . .

Sqlserver DBAQA ?

A Class for Good Judgment?

Realistically, I probably can’t teach you good judgment.

Is there some class to attend, to teach you to not do certain things, like these:

Common Sense Reasoning not so Common . . .

Common Sense Reasoning not so Common . . .

NOT PG&E employees

Note parallelism

Degree = 3

Common Sense Reasoning not so Common . . .

DBA of the day

On-Call DBA

DBA in training

Strategy for Success

ScriptsTools

Techniques

Hands-on Experience

Good Judgment

Focus

2

Logical Reasoning

Focus: Keep Your Eye on the Ball

Focus

Focus

Doggedly pursue the root cause

Losing Focus: Some Distractions

Pet ideas—maybe RAC or Streams? To redesign the application—let’s make

it more “elegant” Trying to solve problem in general My favorite: Avoid solving root problem

by blaming others

“The burnt-up power box”

An Unhelpful Tactic

When my application runs slowly, likely root cause is found in:

1) My app,

2) Database issue, or

3) Somebody else's’ app

“Those other guys are slowing the disk!”

Staying on Focus is Hard Work

It’s tempting to switch to generalities. Benefit:

1. You don’t need to know much.

2. Avoids hard analysis.3. Everybody knows what

“Add CPUs” means.

Cary Milsap calls this using the “aggregate”

Avoid the Dark Side“Aggregate” Method

Avoiding hard work of detailed analysis is immensely appealing.

It’s widespread because anyone can become an expert

You “fix” lots of problems at once!Solution usually ”Add more hardware.”

But it only works for a while . . .

Focus: Stay on Target!

Home-in on the exact problem & fix “Micro” not “Macro.” Solutions are directly applicable to

the problem at hand, so…

1. You can estimate improvement

2. It will often be a huge pickup

Strategy for Success

ScriptsTools

Techniques

Hands-on Experience

Good Judgment

Focus

Logical Reasoning

(3)

This is a Huge Deal …

Logical Reasoning

Similar to staying focused

My definition:

“An inductive process of working

from facts to conclusion.”

Barriers to Logical Reasoning

Logical Reasoning

•Presupposing form of the

answer

• Pet solutions

“Must be the db writer..”

A Logical Process:“Physician to Magician”

Logical Reasoning

(1) State problem

(2) Gather facts

(3) Identify bottleneck

(4) Create solution

(5) Document & Declare Victory

Strategy for Success

ScriptsTools

TechniquesGood

Judgment

Focus

Logical Reasoning

Hands-on Experience

(4)

There Really Isn’t a Substitute

Hands-on Experience

“We don’t want DBAs with OCP!”

(esp. if in bold red on the resume)

An Important Question

•Why is experience so important?

•What does it achieve?

Why is Experience So Critical?

•Save time by skirting crazy ideas.

•Reduces scope of possible solutions

•Know how Oracle works in practice

•Pattern recognition: “Aha—I’ve seen that one before!”

•Gain credibility with your customers.

•Inspired by other ideas you’ve seen.

You Can’t Transfer Experience

Hands-on Experience

“Spend 5 minutes & tell me what you

do and …”

A Big Turn-off: the Hard Work

Hands-on Experience

Expert: 10,000 hours of

deliberative practice

Talent Is Overrated By Geoff Colvin

Strategy for Success

ScriptsTools

Techniques

Hands-on Experience

Good Judgment

Focus

Logical Reasoning

Helpful in Some CasesTools

A matter of personal preference--could be useful.

Caution: Tool may direct your effort: “OEM says …”

Pitfall: Never gain competence ?

A useful analogy >

Doctor versus DBA:What do YOU want in a Doctor?

Pleasant?Nice Office?Cheap?

My Thoughts

Takes the job seriouslyListens to what you sayUnderstands the problemHas expertise in the fieldKnows practical solutionsHow often is he sued?*

* Courtesy Dan Grey, PG&E

Do You Ask the Doctor This?

“What brand of stethoscope do you

use?

Strategy for Success

Scripts(6)Tools

Techniques

Hands-on Experience

Good Judgment

Focus

Logical Reasoning

Scripts Are Important

Scripts

Ten DBAs > 10 different scripts. There usually isn’t just “1 way.”Your process should drive your

scripts, not the other way around.“Roll your own” scripts or borrow

ideas from others

Be Open to Different Methods

There’s more than 1 way to solve a problem. There’s tons of smart people here. Learning

from others is better than formal classes—it’s supremely practical.

It’s hard for anyone to be an expert at more than 1 thing.

Watch for people who have a better way at doing something and copy them!

Script ideas

Most of my scripts are from others Other ideas came from user groups and

forums, etc. One idea (following) came from a

beginner with almost no experience!

What I Learned from a NewbieObject Waited-on

SeleCT DISTINCT v$session.module, Sid, username, Substr(program,1,19) PROG , sql_text,object_nameFrom V$Session, V$Sql, dba_objects oWhere v$session.status = 'ACTIVE'And username is not nulland o.object_id = row_wait_obj#And v$session.sql_hash_value = hash_valueand v$session.sql_address = v$sql.addressand username <> 'SYS'

The Reality of Scripts

“Good scripts, by themselves

do not a performance expert make.”

Strategy for Success

ScriptsTools

Techniques(7)Hands-on

Experience

Good Judgment

Focus

Logical Reasoning

Aha! Good Techniques

Techniques

Absolutely critical This is where we can

learn from others White papers, seminars Pretty close to experience

Ten DBAs will use 10 different techniques

Techniques

Techniques

But—not all techniques are created equal.

Here are some sub-optimum Techniques

Bad Techniques

When Driving forklift, do not drive near edge of loading dock.

Common Sense Reasoning not so Common . . .

Bombsafety violation ?

Bad Techniques

Do not drive car into power lines

Common Sense Reasoning not so Common . . .

Bad Techniques

Think carefully when putting name of company on van.

Common Sense Reasoning not so Common . . .

Not the greatest choice of names

Develop Your Own Techniques

Which method should I use?

“That’s not the way Tom does it!”

Your Way May Be the Best!

Aha! Chris’ Ten Techniques

TechniquesHere are 10

techniques I use

You will have others of course

Technique #1: Use Simple Rules of Thumb

Q: What are some simple metrics that you like to use?

Develop Rules of Thumb

My favorite: Single-block read rate

select EVENT, TOTAL_WAITS, TIME_WAITED ,round(100*total_waits/time_waited) ratefrom v$system_eventwhere event like '%db file sequential read%'order by 1

Samples from unrelated databases:200, 320, 125, 307, 472, 154

Some Rules of Thumb I Use

On RAC, I expect about 15% overhead due to cluster.

You can see this in AWR reports: SQL ordered by Cluster Wait Time

SQL ordered by Cluster Wait Time

Cluster Wait

Time (s)

CWT % of

Elapsd Time

Elapsed Time(s)

CPU Time(s) Execs

4,820.79 15.29 31,525.18 2,429.77 300,054

4,649.80 19.62 23,695.91 1,867.30 189,178

1,509.99 18.13 8,327.73 502.36 209,420

Other RAC Rules of Thumb

On RAC, I expect to see a fast interconnect between nodes.

Script “node_times.sql” calculates how fast nodes communicate.

Uses very simple internode event called 'gc cr grant 2-way‘

I found that 1 ms internode rate is typical.

Sample RAC Node Rate

SNAP_ID NODE BEG #WAITS RATE-------- ---------- --------------- ---------- ---------- 30131 5 14-oct-10-23:00 24935961 1.5 30147 5 15-oct-10-15:00 1276505 1.1 30148 5 15-oct-10-16:00 2415870 1.3 30216 5 18-oct-10-12:00 1370563 1.5 30240 5 19-oct-10-12:00 615880 1.4 30241 5 19-oct-10-13:00 584385 1.1 30242 5 19-oct-10-14:00 1184510 1.3 30248 5 19-oct-10-20:00 2043124 1.6 30249 5 19-oct-10-21:00 6142631 .9

Reference: Node_Times.sql

WITH BASE AS (SELECT instance_number, SNAP_ID, TOTAL_WAITS, time_waited_micro/1000 timemsec, LAG(time_waited_micro/1000, 1, 0) OVER (ORDER BY snap_id) AS PREV_TIME_MSEC, LAG(total_waits, 1, 0) OVER (ORDER BY snap_id) AS PREV_waitsFROM dba_hist_system_eventWHERE event_name ='gc cr grant 2-way'and instance_number = 5and snap_id > 30130)SELECT b.SNAP_ID, b.instance_number NODE,to_char(begin_interval_time, 'dd-mon-yy-hh24:mi') BEG, (TOTAL_WAITS-PREV_WAITS) "#WAITS",ROUND((TIMEMSEC-PREV_TIME_MSEC)/(.001+TOTAL_WAITS-PREV_WAITS), 1) "RATE" FROM BASE b,dba_hist_snapshot Swhere b.instance_number = s.instance_numberand b.snap_id = s.snap_idand (total_waits-prev_waits) > 99900ORDER BY 1

Technique # 2: Map Complex Joins

A VERY powerful technique

Graph Complex Joins: How?

List the tables in the joinShow join conditionNote size of tableNote how selective the filters are

Thanks to Daniel Tow, of “Singing Sql” who introduced me to this method 13 years ago.

Map Complex Joins

Table 1

Table 4

Table 2

Table 3 Table 5

Table 6

Join conditions

Sample 1

Sample 2

Sample 3

I cover this in detail in

The Art & Science of Oracle Performance

Tuning

Technique # 3 Verify Parallel Processing

Please Don’t “bolt-on” parallelism

“But the cost is less if I use “default” parallelism.”

Consider WHY You are Using Parallelism

Is this what I intended?

Are slaves doing multi-block or single-block reads?

Is This What I Intended?

column child_wait format a30column parent_wait format a30column server_name format a4 heading 'Name'column x_status format a10 heading 'Status'column schemaname format a10 heading 'Schema'column x_sid format 9990 heading 'Sid'column x_pid format 9990 heading 'Pid'column p_sid format 9990 heading 'Parent'column program format a12break on p_sid skip 1

set linesize 200

For Reference:The Parallel Slave Script (part 1)

select x.server_name , x.pid as x_pid , x.sid as x_sid, w2.sid as p_sid , v.osuser , v.schemaname , program , w1.event as child_wait, w2.event as parent_wait

from v$px_process x , v$lock l, v$session v, v$session_wait w1 , v$session_wait w2where x.sid <> l.sid(+)and to_number (substr(x.server_name,2)) = l.id2(+)and x.sid = w1.sid(+) and l.sid = w2.sid(+)and x.sid = v.sid(+) and nvl(l.type,'PS') = 'PS'

and x.status not like 'AVAIL%'and w2.event not like 'SQL*Net%‘ order by 1,2

For Reference:The Parallel Slave Script (part 2)

Technique # 4 Get Familiar with Wait Events

Sequential versus Scattered readsLog sync SQL*Net message from client [RAC] gc current request Esp. the top summary in AWR

Use ASH to Find Wait Events for a Specific Time

With P1 As (Select /*+Parallel(a 6) */ Distinct Sample_time, Session_id, Sql_text, Event, Instance_number,

Blocking_sessionFrom Dba_hist_active_sess_history A, V$sqltext B Where A.Sql_id = B.Sql_idAND Sample_time Like '30-SEP-09 10.52%AM'And Piece = 0) Select Instance_number Ins,session_id, SAMPLE_TIME, Sql_text, Event,

Blocking_sessionFrom P1 Order By 3

AWR is an aggregate and gives summary. ASH is great for nailing specific, brief issue. Drawback: 10 minute runtime on big db.

Technique #5Find Bind Variables

You don’t need to guess.

Find Bind Variables

Select INST_ID, c.name||'/'||c.value_string bind_var

from GV$sqlarea a, dba_users b,

v$sql_bind_capture c

where b.user_id=a.parsing_user_id

and b.username != 'SYS‘ and c.address=a.address

and a.sql_id = [enter]

We often need to know typical values the user sets for a bind variable.

Useful for testing a performance solution For historical, use DBA_HIST_SQLBIND

Find Bind Variables

Actual case:Sql looks good, but

always scans all partitions.

Technique # 6 Understand Stored Outlines

Another extremely powerful technique

Example: Critical Sql has changed exec plan in production. It’s vendor

code.

What are you going to do?

Stored Outlines: How They Work

A stored outline preserves an execution plan:1. Oracle captures the hints that ensure a

particular plan.2. When that exact sql is run in the future,

Oracle applies those sql hints to keep same exec plan.

3. Outlines are stored in 3 tables.

A Powerful Trick with Stored Outlines

What if you don’t want to preserve a plan—you want a different plan?

The scheme:

1. Create outline of exact sql;

2. Use sql hint to create another outline with plan you want;

3. Trick Oracle to use the new exec plan even when we don’t supply the sql hint.

For reference: Stored Outline detail*

Turn on stored outline gathering. Run Sql. Then run 2nd sql with hint added. We now have 2 stored outlines:

No hint >> Oracle uses Outline 1 (bad plan)With hint >> Oracle uses Outline 2 (good plan)

Reverse hints so that Oracle will apply Outline 2 when it sees the sql without the hint.

Update Outln.Ol$hints << this table has the hints Set Ol_name = Decode(ol_name, One, Two, Two, One)

*Thanks to Simon Leung for this tip!

Technique #7 Become Adept w/Sql Hints

Q: Why do we ever need to use sql hints?

Some Hints I Use a Lot

FULL

LEADING

ORDERED

USE_NL, USE_HASH

PARALLEL, PARALLEL_INDEX,

INDEX_FFS

INDEX

CARDINALITY

Technique # 8Don’t Blindly Apply OLTP Ideas

Indexing strategies for OLTP are not appropriate for large batch jobs.

“Always use index” is not a reliable maxim for batch jobs.

“Row by row” processing is often a disaster. Consequence of wrong exec plan with batch

job is catastrophic. Large batch jobs often need a sql hint.

Technique #9 Don’t Presuppose Form of Solution

This is very common, & impedes entire process. Similar error: Assuming problem even exists. Adding to confusion: Problem descriptions are

often inaccurate, or really a conclusion, not an observation:

“It is a capital mistake to theorize in advance of the facts.” Sherlock Holmes, A Scandal in Bohemia

“You need more indexes”“We are having contention.”“We’re out of memory.”

Case: Presupposed Solution Wastes Several Days at a Publisher

Scenario: Unknown performance problem on critical database.

DBA advised: “Don’t spend any time checking out the network. We already know it’s fine.”

The real bottleneck was . . .

THE NETWORK!

Case 2: Presupposed Solution: More Threads are Better!

Scenario: Payment processing job uses 4 threads, but still slow.

Without identifying actual bottleneck, threads set to 10.

New run time Same as Before! Reason: This solution addressed transactions

that only accounted for 10% of runtime. Real bottleneck was a single Insert statement

that didn’t use the multiple threads.

Technique # 10 Insist on Root Cause Before Solution

When a solution is proposed, this suggests that the root cause has been identified.

So ask (nicely), “What problem does this fix?”

Wait—I’ve Forgotten Something!

I’m missing a critical component in performance tuning

What is Missing!

Scripts

Tools

Techniques

Hands-on Experience

Good Judgment

Focus

Logical Reasoning

??

A Clue: What Saved 3 Lives?

April 14, 1970, Apollo 13 explosion. Problem: No more air scrubbers. Result: Crew will soon die a slow and painful

death--poisoned by carbon dioxide. Your Mission: Devise an air scrubber using

simple materials. Timeline: A day or so.

The Pep Talk

Gene Kranz: “Failure is not an option.

This crew is coming home.

Now get going!”

Can We Use our “Extreme” Strategy?

Tools: No way. Logic: Helpful. Judgment: Helpful, but inadequate. Techniques: None Scripts: None Focus: Helpful, but inadequate. Hands-on Experience: Never done before.

A “Bizarre Contraption” Saves Lives

Bill of Materials:

A bunch of cardboard

(1) plastic bag,

(1) air hose,

(1) sock,

+ some duct tape.

How Did the Bizarre Contraption get Built?

Missing Ingredient: Creativity

Extreme Performance Tuning

Scripts

Tools

Techniques

Hands-on Experience

Good Judgment

Focus

Logical Reasoning

Creativity

Creativity??

Without creativity, we are “dull people in a dull profession.”

A “thought experiment” to foster creativity: Pretend people will die

Ask, “Isn’t there anything you can do?”

The un-Innovative Innovation magazine

Don’t Be a Dullard!

DBA taking class at OpenWorld

[“before” picture”]

This DBA didn’t have any creativity

After Following Chris’ Performance Tuning Program

“After” picture

Look what happened!

Focus Case Study: Getting Some “Breathing Room”

Daily inventory report: 48 hours. Root cause: Running 5 million SqlDirector: “Fix it today!”Short-term answer: Change report-

server parameter to run 20 threadsResult: 2 hour runtime.Everyone kept their jobs.

Let’s Test Your Creativity

Scenario: Critical batch job to company needs to finish faster.

Runs 1 million sql via one session. Bottleneck is mostly disk. No time to redesign program. Pretend people will die unless you fix it. What can you do?

JUST RUN “SUPER BOOST”

Just warm up the cache Used 25 threads to

touch the blocks that will be needed shortly.

Got about 50% pickup in performance.

Summary: The Art & Science of Oracle Performance Tuning

Extreme Performance tuning demands good analytical skills—“left brain” thinking.

It also requires excellent “right-brain thinking”

“It’s almost all art!”-- Brian Keating, independent consultant

Thank You for

Attending!

Chris@OracleMagician.com

Extreme Performance Tuning:

The Greatest Job on Planet Earth