EXTREME PERFORMANCE TUNING
GAME PLAN FOR SUCCESS
Chris Lawson
Agenda
What is Extreme Performance Tuning? Seven attributes of Extreme Perf Tuning Chris’ favorite tuning techniques Some pitfalls to avoid
When it doesn’t work—the missing piece of
the puzzle
A Key Question
After you attend a seminar on performance tuning, do you
discover that you have become a performance
expert?
Performance Tuning is Not:
Having really tricky scriptsRunning colorful programsDelivering long reports full of
generalitiesBeing smarter, faster, better-
looking than other DBAs
The Paradox of Performance Tuning
Being extremely good means being adept in multiple areas
A single strength will not work.Example: New DBA uses tool (e.g.,
OEM) to analyze performanceQ: What will be the likely outcome?
The 7 Facets of Extreme Performance
Scripts
Tools
Techniques
Hands-on Experience
Good Judgment
Focus
Logical Reasoning
Q: Which Ones are Taught in Seminars?
Scripts
Tools
Techniques
Hands-on Experience
Good Judgment
Focus
Logical Reasoning
Only These are Taught in Class
Scripts
Tools
Techniques
Hands-on Experience
Good Judgment
Focus
Logical Reasoning
Let’s Examine the 7
1. Good Judgment
2. Logical Reasoning
3. Good Techniques
4. Scripts
5. Tools (maybe)
6. Focus
7. Hands-on Experience
Strategy for Success
ScriptsTools
Techniques
Hands-on Experience
Good Judgment
(1)
Focus
Logical Reasoning
Good Judgment: Thinking Clearly
Good Judgment
Difficult to explain—it’s a sound approach to gray areas
Q: How do you spot this in candidates?
I don’t mean I.Q.
“The man who was too smart to talk to”
Good Judgment: Thinking Clearly
Good Judgment
Easier to spot bad judgment than good
Bad judgment stands out
Tale of the “No-good Nested Loops”
Good Judgment May be Hidden;Bad Judgment Stands Out
Good Judgment ?
“The Thrill of Thrill Hill”
The result . . .
Good Judgment
Firms want people with good judgment, but how do you measure?
Common sense reasoning maybe not so common?
Some examples of smart people using not-so-smart judgment
Common Sense Reasoning not so Common . . .
Common Sense Reasoning not so Common . . .
Sybase DBA violated SOX
Common Sense Reasoning not so Common . . .
Sqlserver DBAQA ?
A Class for Good Judgment?
Realistically, I probably can’t teach you good judgment.
Is there some class to attend, to teach you to not do certain things, like these:
Common Sense Reasoning not so Common . . .
Common Sense Reasoning not so Common . . .
NOT PG&E employees
Note parallelism
Degree = 3
Common Sense Reasoning not so Common . . .
DBA of the day
On-Call DBA
DBA in training
Strategy for Success
ScriptsTools
Techniques
Hands-on Experience
Good Judgment
Focus
2
Logical Reasoning
Focus: Keep Your Eye on the Ball
Focus
Focus
Doggedly pursue the root cause
Losing Focus: Some Distractions
Pet ideas—maybe RAC or Streams? To redesign the application—let’s make
it more “elegant” Trying to solve problem in general My favorite: Avoid solving root problem
by blaming others
“The burnt-up power box”
An Unhelpful Tactic
When my application runs slowly, likely root cause is found in:
1) My app,
2) Database issue, or
3) Somebody else's’ app
“Those other guys are slowing the disk!”
Staying on Focus is Hard Work
It’s tempting to switch to generalities. Benefit:
1. You don’t need to know much.
2. Avoids hard analysis.3. Everybody knows what
“Add CPUs” means.
Cary Milsap calls this using the “aggregate”
Avoid the Dark Side“Aggregate” Method
Avoiding hard work of detailed analysis is immensely appealing.
It’s widespread because anyone can become an expert
You “fix” lots of problems at once!Solution usually ”Add more hardware.”
But it only works for a while . . .
Focus: Stay on Target!
Home-in on the exact problem & fix “Micro” not “Macro.” Solutions are directly applicable to
the problem at hand, so…
1. You can estimate improvement
2. It will often be a huge pickup
Strategy for Success
ScriptsTools
Techniques
Hands-on Experience
Good Judgment
Focus
Logical Reasoning
(3)
This is a Huge Deal …
Logical Reasoning
Similar to staying focused
My definition:
“An inductive process of working
from facts to conclusion.”
Barriers to Logical Reasoning
Logical Reasoning
•Presupposing form of the
answer
• Pet solutions
“Must be the db writer..”
A Logical Process:“Physician to Magician”
Logical Reasoning
(1) State problem
(2) Gather facts
(3) Identify bottleneck
(4) Create solution
(5) Document & Declare Victory
Strategy for Success
ScriptsTools
TechniquesGood
Judgment
Focus
Logical Reasoning
Hands-on Experience
(4)
There Really Isn’t a Substitute
Hands-on Experience
“We don’t want DBAs with OCP!”
(esp. if in bold red on the resume)
An Important Question
•Why is experience so important?
•What does it achieve?
Why is Experience So Critical?
•Save time by skirting crazy ideas.
•Reduces scope of possible solutions
•Know how Oracle works in practice
•Pattern recognition: “Aha—I’ve seen that one before!”
•Gain credibility with your customers.
•Inspired by other ideas you’ve seen.
You Can’t Transfer Experience
Hands-on Experience
“Spend 5 minutes & tell me what you
do and …”
A Big Turn-off: the Hard Work
Hands-on Experience
Expert: 10,000 hours of
deliberative practice
Talent Is Overrated By Geoff Colvin
Strategy for Success
ScriptsTools
Techniques
Hands-on Experience
Good Judgment
Focus
Logical Reasoning
Helpful in Some CasesTools
A matter of personal preference--could be useful.
Caution: Tool may direct your effort: “OEM says …”
Pitfall: Never gain competence ?
A useful analogy >
Doctor versus DBA:What do YOU want in a Doctor?
Pleasant?Nice Office?Cheap?
My Thoughts
Takes the job seriouslyListens to what you sayUnderstands the problemHas expertise in the fieldKnows practical solutionsHow often is he sued?*
* Courtesy Dan Grey, PG&E
Do You Ask the Doctor This?
“What brand of stethoscope do you
use?
Strategy for Success
Scripts(6)Tools
Techniques
Hands-on Experience
Good Judgment
Focus
Logical Reasoning
Scripts Are Important
Scripts
Ten DBAs > 10 different scripts. There usually isn’t just “1 way.”Your process should drive your
scripts, not the other way around.“Roll your own” scripts or borrow
ideas from others
Be Open to Different Methods
There’s more than 1 way to solve a problem. There’s tons of smart people here. Learning
from others is better than formal classes—it’s supremely practical.
It’s hard for anyone to be an expert at more than 1 thing.
Watch for people who have a better way at doing something and copy them!
Script ideas
Most of my scripts are from others Other ideas came from user groups and
forums, etc. One idea (following) came from a
beginner with almost no experience!
What I Learned from a NewbieObject Waited-on
SeleCT DISTINCT v$session.module, Sid, username, Substr(program,1,19) PROG , sql_text,object_nameFrom V$Session, V$Sql, dba_objects oWhere v$session.status = 'ACTIVE'And username is not nulland o.object_id = row_wait_obj#And v$session.sql_hash_value = hash_valueand v$session.sql_address = v$sql.addressand username <> 'SYS'
The Reality of Scripts
“Good scripts, by themselves
do not a performance expert make.”
Strategy for Success
ScriptsTools
Techniques(7)Hands-on
Experience
Good Judgment
Focus
Logical Reasoning
Aha! Good Techniques
Techniques
Absolutely critical This is where we can
learn from others White papers, seminars Pretty close to experience
Ten DBAs will use 10 different techniques
Techniques
Techniques
But—not all techniques are created equal.
Here are some sub-optimum Techniques
Bad Techniques
When Driving forklift, do not drive near edge of loading dock.
Common Sense Reasoning not so Common . . .
Bombsafety violation ?
Bad Techniques
Do not drive car into power lines
Common Sense Reasoning not so Common . . .
Bad Techniques
Think carefully when putting name of company on van.
Common Sense Reasoning not so Common . . .
Not the greatest choice of names
Develop Your Own Techniques
Which method should I use?
“That’s not the way Tom does it!”
Your Way May Be the Best!
Aha! Chris’ Ten Techniques
TechniquesHere are 10
techniques I use
You will have others of course
Technique #1: Use Simple Rules of Thumb
Q: What are some simple metrics that you like to use?
Develop Rules of Thumb
My favorite: Single-block read rate
select EVENT, TOTAL_WAITS, TIME_WAITED ,round(100*total_waits/time_waited) ratefrom v$system_eventwhere event like '%db file sequential read%'order by 1
Samples from unrelated databases:200, 320, 125, 307, 472, 154
Some Rules of Thumb I Use
On RAC, I expect about 15% overhead due to cluster.
You can see this in AWR reports: SQL ordered by Cluster Wait Time
SQL ordered by Cluster Wait Time
Cluster Wait
Time (s)
CWT % of
Elapsd Time
Elapsed Time(s)
CPU Time(s) Execs
4,820.79 15.29 31,525.18 2,429.77 300,054
4,649.80 19.62 23,695.91 1,867.30 189,178
1,509.99 18.13 8,327.73 502.36 209,420
Other RAC Rules of Thumb
On RAC, I expect to see a fast interconnect between nodes.
Script “node_times.sql” calculates how fast nodes communicate.
Uses very simple internode event called 'gc cr grant 2-way‘
I found that 1 ms internode rate is typical.
Sample RAC Node Rate
SNAP_ID NODE BEG #WAITS RATE-------- ---------- --------------- ---------- ---------- 30131 5 14-oct-10-23:00 24935961 1.5 30147 5 15-oct-10-15:00 1276505 1.1 30148 5 15-oct-10-16:00 2415870 1.3 30216 5 18-oct-10-12:00 1370563 1.5 30240 5 19-oct-10-12:00 615880 1.4 30241 5 19-oct-10-13:00 584385 1.1 30242 5 19-oct-10-14:00 1184510 1.3 30248 5 19-oct-10-20:00 2043124 1.6 30249 5 19-oct-10-21:00 6142631 .9
Reference: Node_Times.sql
WITH BASE AS (SELECT instance_number, SNAP_ID, TOTAL_WAITS, time_waited_micro/1000 timemsec, LAG(time_waited_micro/1000, 1, 0) OVER (ORDER BY snap_id) AS PREV_TIME_MSEC, LAG(total_waits, 1, 0) OVER (ORDER BY snap_id) AS PREV_waitsFROM dba_hist_system_eventWHERE event_name ='gc cr grant 2-way'and instance_number = 5and snap_id > 30130)SELECT b.SNAP_ID, b.instance_number NODE,to_char(begin_interval_time, 'dd-mon-yy-hh24:mi') BEG, (TOTAL_WAITS-PREV_WAITS) "#WAITS",ROUND((TIMEMSEC-PREV_TIME_MSEC)/(.001+TOTAL_WAITS-PREV_WAITS), 1) "RATE" FROM BASE b,dba_hist_snapshot Swhere b.instance_number = s.instance_numberand b.snap_id = s.snap_idand (total_waits-prev_waits) > 99900ORDER BY 1
Technique # 2: Map Complex Joins
A VERY powerful technique
Graph Complex Joins: How?
List the tables in the joinShow join conditionNote size of tableNote how selective the filters are
Thanks to Daniel Tow, of “Singing Sql” who introduced me to this method 13 years ago.
Map Complex Joins
Table 1
Table 4
Table 2
Table 3 Table 5
Table 6
Join conditions
Sample 1
Sample 2
Sample 3
I cover this in detail in
The Art & Science of Oracle Performance
Tuning
Technique # 3 Verify Parallel Processing
Please Don’t “bolt-on” parallelism
“But the cost is less if I use “default” parallelism.”
Consider WHY You are Using Parallelism
Is this what I intended?
Are slaves doing multi-block or single-block reads?
Is This What I Intended?
column child_wait format a30column parent_wait format a30column server_name format a4 heading 'Name'column x_status format a10 heading 'Status'column schemaname format a10 heading 'Schema'column x_sid format 9990 heading 'Sid'column x_pid format 9990 heading 'Pid'column p_sid format 9990 heading 'Parent'column program format a12break on p_sid skip 1
set linesize 200
For Reference:The Parallel Slave Script (part 1)
select x.server_name , x.pid as x_pid , x.sid as x_sid, w2.sid as p_sid , v.osuser , v.schemaname , program , w1.event as child_wait, w2.event as parent_wait
from v$px_process x , v$lock l, v$session v, v$session_wait w1 , v$session_wait w2where x.sid <> l.sid(+)and to_number (substr(x.server_name,2)) = l.id2(+)and x.sid = w1.sid(+) and l.sid = w2.sid(+)and x.sid = v.sid(+) and nvl(l.type,'PS') = 'PS'
and x.status not like 'AVAIL%'and w2.event not like 'SQL*Net%‘ order by 1,2
For Reference:The Parallel Slave Script (part 2)
Technique # 4 Get Familiar with Wait Events
Sequential versus Scattered readsLog sync SQL*Net message from client [RAC] gc current request Esp. the top summary in AWR
Use ASH to Find Wait Events for a Specific Time
With P1 As (Select /*+Parallel(a 6) */ Distinct Sample_time, Session_id, Sql_text, Event, Instance_number,
Blocking_sessionFrom Dba_hist_active_sess_history A, V$sqltext B Where A.Sql_id = B.Sql_idAND Sample_time Like '30-SEP-09 10.52%AM'And Piece = 0) Select Instance_number Ins,session_id, SAMPLE_TIME, Sql_text, Event,
Blocking_sessionFrom P1 Order By 3
AWR is an aggregate and gives summary. ASH is great for nailing specific, brief issue. Drawback: 10 minute runtime on big db.
Technique #5Find Bind Variables
You don’t need to guess.
Find Bind Variables
Select INST_ID, c.name||'/'||c.value_string bind_var
from GV$sqlarea a, dba_users b,
v$sql_bind_capture c
where b.user_id=a.parsing_user_id
and b.username != 'SYS‘ and c.address=a.address
and a.sql_id = [enter]
We often need to know typical values the user sets for a bind variable.
Useful for testing a performance solution For historical, use DBA_HIST_SQLBIND
Find Bind Variables
Actual case:Sql looks good, but
always scans all partitions.
Technique # 6 Understand Stored Outlines
Another extremely powerful technique
Example: Critical Sql has changed exec plan in production. It’s vendor
code.
What are you going to do?
Stored Outlines: How They Work
A stored outline preserves an execution plan:1. Oracle captures the hints that ensure a
particular plan.2. When that exact sql is run in the future,
Oracle applies those sql hints to keep same exec plan.
3. Outlines are stored in 3 tables.
A Powerful Trick with Stored Outlines
What if you don’t want to preserve a plan—you want a different plan?
The scheme:
1. Create outline of exact sql;
2. Use sql hint to create another outline with plan you want;
3. Trick Oracle to use the new exec plan even when we don’t supply the sql hint.
For reference: Stored Outline detail*
Turn on stored outline gathering. Run Sql. Then run 2nd sql with hint added. We now have 2 stored outlines:
No hint >> Oracle uses Outline 1 (bad plan)With hint >> Oracle uses Outline 2 (good plan)
Reverse hints so that Oracle will apply Outline 2 when it sees the sql without the hint.
Update Outln.Ol$hints << this table has the hints Set Ol_name = Decode(ol_name, One, Two, Two, One)
*Thanks to Simon Leung for this tip!
Technique #7 Become Adept w/Sql Hints
Q: Why do we ever need to use sql hints?
Some Hints I Use a Lot
FULL
LEADING
ORDERED
USE_NL, USE_HASH
PARALLEL, PARALLEL_INDEX,
INDEX_FFS
INDEX
CARDINALITY
Technique # 8Don’t Blindly Apply OLTP Ideas
Indexing strategies for OLTP are not appropriate for large batch jobs.
“Always use index” is not a reliable maxim for batch jobs.
“Row by row” processing is often a disaster. Consequence of wrong exec plan with batch
job is catastrophic. Large batch jobs often need a sql hint.
Technique #9 Don’t Presuppose Form of Solution
This is very common, & impedes entire process. Similar error: Assuming problem even exists. Adding to confusion: Problem descriptions are
often inaccurate, or really a conclusion, not an observation:
“It is a capital mistake to theorize in advance of the facts.” Sherlock Holmes, A Scandal in Bohemia
“You need more indexes”“We are having contention.”“We’re out of memory.”
Case: Presupposed Solution Wastes Several Days at a Publisher
Scenario: Unknown performance problem on critical database.
DBA advised: “Don’t spend any time checking out the network. We already know it’s fine.”
The real bottleneck was . . .
THE NETWORK!
Case 2: Presupposed Solution: More Threads are Better!
Scenario: Payment processing job uses 4 threads, but still slow.
Without identifying actual bottleneck, threads set to 10.
New run time Same as Before! Reason: This solution addressed transactions
that only accounted for 10% of runtime. Real bottleneck was a single Insert statement
that didn’t use the multiple threads.
Technique # 10 Insist on Root Cause Before Solution
When a solution is proposed, this suggests that the root cause has been identified.
So ask (nicely), “What problem does this fix?”
Wait—I’ve Forgotten Something!
I’m missing a critical component in performance tuning
What is Missing!
Scripts
Tools
Techniques
Hands-on Experience
Good Judgment
Focus
Logical Reasoning
??
A Clue: What Saved 3 Lives?
April 14, 1970, Apollo 13 explosion. Problem: No more air scrubbers. Result: Crew will soon die a slow and painful
death--poisoned by carbon dioxide. Your Mission: Devise an air scrubber using
simple materials. Timeline: A day or so.
The Pep Talk
Gene Kranz: “Failure is not an option.
This crew is coming home.
Now get going!”
Can We Use our “Extreme” Strategy?
Tools: No way. Logic: Helpful. Judgment: Helpful, but inadequate. Techniques: None Scripts: None Focus: Helpful, but inadequate. Hands-on Experience: Never done before.
A “Bizarre Contraption” Saves Lives
Bill of Materials:
A bunch of cardboard
(1) plastic bag,
(1) air hose,
(1) sock,
+ some duct tape.
How Did the Bizarre Contraption get Built?
Missing Ingredient: Creativity
Extreme Performance Tuning
Scripts
Tools
Techniques
Hands-on Experience
Good Judgment
Focus
Logical Reasoning
Creativity
Creativity??
Without creativity, we are “dull people in a dull profession.”
A “thought experiment” to foster creativity: Pretend people will die
Ask, “Isn’t there anything you can do?”
The un-Innovative Innovation magazine
Don’t Be a Dullard!
DBA taking class at OpenWorld
[“before” picture”]
This DBA didn’t have any creativity
After Following Chris’ Performance Tuning Program
“After” picture
Look what happened!
Focus Case Study: Getting Some “Breathing Room”
Daily inventory report: 48 hours. Root cause: Running 5 million SqlDirector: “Fix it today!”Short-term answer: Change report-
server parameter to run 20 threadsResult: 2 hour runtime.Everyone kept their jobs.
Let’s Test Your Creativity
Scenario: Critical batch job to company needs to finish faster.
Runs 1 million sql via one session. Bottleneck is mostly disk. No time to redesign program. Pretend people will die unless you fix it. What can you do?
JUST RUN “SUPER BOOST”
Just warm up the cache Used 25 threads to
touch the blocks that will be needed shortly.
Got about 50% pickup in performance.
Summary: The Art & Science of Oracle Performance Tuning
Extreme Performance tuning demands good analytical skills—“left brain” thinking.
It also requires excellent “right-brain thinking”
“It’s almost all art!”-- Brian Keating, independent consultant
Thank You for
Attending!
Extreme Performance Tuning:
The Greatest Job on Planet Earth