how to find and fix your oracle-based application performance problem

HowtofindandfixyourJavaAPEXADFOBIEE.NETSQLPL/SQL

applicationperformanceproblem

CaryMillsapMethodRCorporationandAccentureEnkitecGroup

@CaryMillsap

DOUGJanuaryMeeting·Richardson,Texas5:00p–7:15pThursday28January2016

©2006,2016CaryMillsap

1

TMMeTHOD RTM

http://twitter.com/!CaryMillsap

http://twitter.com/!CaryMillsap

@CaryMillsap 2

CaryMillsap

http://twitter.com/CaryMillsap


@CaryMillsap

Q WhatisthemostcommonOracleperformanceproblemyousee?”

3

“



@CaryMillsap

WhatisthemostcommonOracleperformanceproblemyousee?”

4

“

Assumingthatotherpeople’scommonproblemsmustbeyourproblem.

...

QA



@CaryMillsap 5

JavaAPEXADFOBIEE.NETSQLPL/SQL



@CaryMillsap

Whatisaperformance problem?

6



@CaryMillsap 7@CaryMillsap



@CaryMillsap

Performance is notan attribute of a system.

8



@CaryMillsap 9

ID USERNAME OPERATION R SLR-- -------- --------- ------ --- 1 FCHANG OE BOOK 12.019 2.0 2 RSMITH OE SHIP 3.528 5.0 3 DJOHNSON OE PICK 1.211 5.0 4 FFORBES OE BOOK 0.716 2.0 5 FCHANG OE BOOK 1.917 2.0 6 LBUMONT PA MTCH 1.305 2.0

#define FASTid ((Rid ≤ SLRid)?”FAST”:”SLOW”)



@CaryMillsap 10

ID USERNAME OPERATION R SLR SPEED-- -------- --------- ------ --- ----- 1 FCHANG OE BOOK 12.019 2.0 SLOW 2 RSMITH OE SHIP 3.528 5.0 FAST 3 DJOHNSON OE PICK 1.211 5.0 FAST 4 FFORBES OE BOOK 0.716 2.0 FAST 5 FCHANG OE BOOK 1.917 2.0 FAST 6 LBUMONT PA MTCH 1.305 2.0 FAST

#define FASTid ((Rid ≤ SLRid)?”FAST”:”SLOW”)



@CaryMillsap

Performance is an attribute ofeach individual experience

with a system.

11



@CaryMillsap 12

TASK• id• name• ...

EXPERIENCE• id• task-id• user-id• ip-address• start-time• end-time• ERROR-code• WORK-done

SQL• ID• Task-id• ...

N

1

N

1



@CaryMillsap 13

<experience id="b3196c98-906d-4394-bc55-0339518a63b2" task-id="7" uid="238" ip="142.128.130.186" t0="2014-04-10T08:32:14.137886" t1="2014-04-10T08:32:17.891173" err="" work="3"/>



@CaryMillsap

has to finish quickly.”

clickbuttonlinkrow

queryreport

job

}{“My

14

Thisiswhatperformanceis.



@CaryMillsap

has to finish quickly.”

clickbuttonlinkrow

queryreport

job

}{“My

15

Aperformanceproblemiswhenitdoesn’t.



@CaryMillsap 16

“How long does it take?”

Responsetime(R)Durationfromservicerequesttoservicefulfillment.

Sanjay Nancy Ken Jorge

R

t0

t1

R=t1–t0

Twobigquestions...1. Howlongdidittake?2. Why?



@CaryMillsap 17

Twobigquestions...1. Howlongdidittake?2. Why?

“How long does it take?”

Responsetime(R)Durationfromservicerequesttoservicefulfillment.

Sanjay Nancy Ken Jorge

R

t0

t1

R=t1–t0



@CaryMillsap

MethodR

18



@CaryMillsap

1. Selecttheexperienceyouneedtoimprove.2. Measureitsresponsetime(R)indetail.3. Executethebestnet-payoffremedy.4. Repeatuntileconomicallyoptimal.

19

MethodR



@CaryMillsap


20

MethodR



@CaryMillsap


21

MethodR



@CaryMillsap 22

MethodR

OP

TIMIZE ANYTHING

MeT H O D R



@CaryMillsap


23

MethodR



@CaryMillsap


24

MethodR

Howdoyoudothis,

whentheitisyourcode?



@CaryMillsap

EXADATAD ATA B A S EENTERPRISE EDITION

D ATA B A S ESTANDARD EDITION

D ATA B A S EEXPRESS EDITION

OracleextendedSQLtracingisafeatureofeveryOracleDatabase.

26

Oracle71992Oracle81997Oracle8i2000Oracle9i2001Oracle10g2004Oracle11g2007Oracle12c2013



@CaryMillsap

MeasuringOracleresponsetimes

27



@CaryMillsap 28

❶Activatetracing

❷Getthetracefile

❸Understanditsstory



@CaryMillsap

❶Activatetracing

❷Getthetracefile


29



@CaryMillsap 30

Thisisthehardestpart....Butonlythefirsttime.

Afterthat,youjustlather,rinse,repeat.



@CaryMillsap

https://app.com/apex/f?p=150:1:5547991082303::NO:::&P_TRACE=YES

31

Well,it’seasyinOracleAPEX.Todecideatruntimewhethertotraceyourcode...



@CaryMillsap

Othertechnologiesrequirealittlemorework.

First,thebasics.

32



@CaryMillsap 33

Todecideatcompiletimetotraceallyourcode...

dbms_session.session_trace_enable( waits =>true, binds =>true, plan_stat =>'ALL_EXECUTIONS' );

--YourOEBOOKcode

dbms_session.session_trace_disable();



@CaryMillsap

if(should_trace('OEBOOK',dbms_random.value(0,1)){ dbms_session.session_trace_enable( waits =>true, binds =>true, plan_stat =>'ALL_EXECUTIONS' );}

--YourOEBOOKcode

dbms_session.session_trace_disable();

34

Todecideatruntimewhethertotraceyourcode...



@CaryMillsap

subshould_trace(t,r){ selecttrace_proportionfromtrace_controlwheretask_name=:t; return(r<=trace_proportion);}

35

...whereshould_tracelookslikethis.

task_name trace_proportion

OEBOOK 0.05

OEPICK 0.02

OESHIP 1.00

OEINVOICE 0.01

should_trace("OEBOOK",0.00)→trueshould_trace("OEBOOK",0.01)→trueshould_trace("OEBOOK",0.02)→true...should_trace("OEBOOK",0.05)→trueshould_trace("OEBOOK",0.06)→falseshould_trace("OEBOOK",0.07)→falseshould_trace("OEBOOK",0.08)→false...should_trace("OEBOOK",1.00)→false

5%

95%

trace_control



@CaryMillsap

OracleDatabasehelpsyouimplement

runtimetracingdecisions...

...withouthavingtomakeyourdevelopersdotheifblockstuff.

36



@CaryMillsap

dbms_monitor.serv_mod_act_trace_enable( service_name =>'SYS$USERS', module_name =>'OEBOOK', action_name =>dbms_monitor.all_actions, waits =>true, binds =>true, plan_stat =>'ALL_EXECUTIONS');

37

TheDBAdoesthis,atruntime.

Butthisworksonlyifyourcodesetsitsmodulenameto“OEBOOK”.



@CaryMillsap

Howyousetyourmodulenamevariesbytechnology.

SQLPL/SQLJavaADF.NETOBIEE

38



@CaryMillsap

--PL/SQLexample

dbms_application_info.set_module( module_name =>'OEBOOK', action_name =>sys_guid());

--YourOEBOOKcode

dbms_application_info.set_module( module_name =>null, action_name =>null);

39

SQLPL/SQLTosetyourcode’smoduleandactionnames...



@CaryMillsap

// JDBC 11g example

Stringmetrics[]=newString[OraCxn.END_TO_END_STATE_INDEX_MAX];metrics[END_TO_END_MODULE_INDEX]="OEBOOK";metrics[END_TO_END_ACTION_INDEX]=UUID.randomUUID().toString();conn.setEndToEndMetrics(metrics,(short)0);

//YourOEBOOKcode

metrics[END_TO_END_MODULE_INDEX]="";metrics[END_TO_END_ACTION_INDEX]="";conn.setEndToEndMetrics(metrics,(short)0);

40

JavaADFTosetyourcode’smoduleandactionnames...



@CaryMillsap

// JDBC 12c example

Properties p = new Properties();p.put("OCSID.MODULE", "OEBOOK");p.put("OCSID.ACTION", UUID.randomUUID().toString());connection.setClientInfo(p);

// Your OE BOOK code

p.put("OCSID.MODULE", "");p.put("OCSID.ACTION", "");connection.setClientInfo(p);

41

JavaADFTosetyourcode’smoduleandactionnames...



@CaryMillsap

//C#example

conn.ModuleName="OEBOOK";conn.ActionName=Guid.NewGuid().toString();

//YourOEBOOKcode

conn.ModuleName="";conn.ActionName="";

42

ODP.NETTosetyourcode’smoduleandactionnames...



@CaryMillsap 43

OBIEETosetyourcode’smoduleandactionnames...



@CaryMillsap

Here’sthegoal.

44



@CaryMillsap

User’sRexperience

Oracletracefile

45

AppUser Oracle DB

time

Youwantthistobesmall

Youwantthistobesmall



@CaryMillsap

Anotherexperience

Anexperience

Notthetracefileyouwant

46

AppUser Oracle DB

time



@CaryMillsap

Anotherexperience

Anexperience

Youwantonetracefileperexperience

47

AppUser Oracle DB

time



@CaryMillsap

Thegoal:

Traceexactlyeachuserexperienceyoucareabout.

...Sothatyoucanseehowyourcodeconsumestimewhenitbehavesproperly,andwhenitmisbehaves.

48



@CaryMillsap

Thisiswhatyou’re

lookingatwhenyouusesystemwideaggregations.

50

AppUser Oracle DB

time



@CaryMillsap 51

❶Activatetracing

❷Getthetracefile




@CaryMillsap 52

Thisistheboringpart....Butit’saninexpensiveproblemtosolve.



@CaryMillsap

Somethingstoknow...

YourtracefileisontheOracleDatabaseserver(s),called:select tracefile from v$process where addr=( select paddr from v$session where sid=( select sid from v$mystat where rownum=1 ))

SessionswithDOP=kcancreatemorethan2k+1tracefiles.

53



@CaryMillsap 54

Please, will you help me find my trace file?

@CaryMillsap

Therearelotsofwaystofetchthetracedata.FTP

SambaNFSmountportabledisk

USBthumbdriveOracleDatabasedirectoryobjects

MethodRTraceextensionforOracleSQLDeveloper

55



@CaryMillsap

Fetchingtracefilescanbeeasy.Youcanbuildtools,oryoucanbuythem.

56

Fn’m [ mifp_^ jli\f_g.



@CaryMillsap 57

❶Activatetracing

❷Getthetracefile




@CaryMillsap 58

ThisistheFUNpart.



@CaryMillsap 59

What’sinthere?!



@CaryMillsap 60

AnOracletracefileisalogthatshows

whatyourcodedidinsidetheOracleDatabase.



@CaryMillsap

Somethingstoknow...

Oraclewritesatracelinewhenacall(db|os)finishes.

Therearetwoprimarylineformats:onefordbcalls,oneforoscalls.

EachcallisassociatedwithaSQLorPL/SQLstatementthroughacursorid.

Eachlinecontainsatimestamp(tim)andaduration(e|ela).

R≠∑(e|ela)becauseparentcalldurationsincludechildcalldurations.

61



@CaryMillsap 62

method-r.com

Formoredetails...



http://method-r.com/papers

http://method-r.com/papers

@CaryMillsap

Let’slookatsometracelines...

63



@CaryMillsap 64

begin prepare CPU latch-related syscall CPU end preparebegin exec CPU write(SQLNET_OUT, result_to_client);end execread(SQLNET_IN, next_request_from_client);begin fetch CPU latch-related syscall CPU write(SQLNET_OUT, result_to_client);end fetchread(SQLNET_IN, next_request_from_client);begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results);end fetchread(SQLNET_IN, next_request_from_client);begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results);end fetchread(SQLNET_IN, next_request_from_client);begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results);end fetchread(SQLNET_IN, next_request_from_client);

Oraclekernelcodepath

Thisisthekindofstuffyourcodecausesthe

Oraclekerneltodo.



@CaryMillsap 65

WAIT #42: nam='latch: library cache'…

PARSE #42:c=10000,…

WAIT #42: nam='SQL*Net message to client'…EXEC #42:c=10000,…WAIT #42: nam='SQL*Net message from client'…

WAIT #42: nam='latch: cache buffers chains'…

WAIT #42: nam='SQL*Net message to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…

WAIT #42: nam='SQL*Net message to client'…WAIT #42: nam='SQL*Net more data to client'…WAIT #42: nam='SQL*Net more data to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…



OracleextendedSQLtracedatabegin prepare CPU latch-related syscall CPU end preparebegin exec CPU write(SQLNET_OUT, result_to_client);end execread(SQLNET_IN, next_request_from_client);begin fetch CPU latch-related syscall CPU write(SQLNET_OUT, result_to_client);end fetchread(SQLNET_IN, next_request_from_client);begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results);end fetchread(SQLNET_IN, next_request_from_client);begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results);end fetchread(SQLNET_IN, next_request_from_client);begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results);end fetchread(SQLNET_IN, next_request_from_client);

Oraclekernelcodepath

Thisisthekindof

tracedatathekernelproduces.



@CaryMillsap 66


PARSE #42:c=10000,…







OracleextendedSQLtracedata

Ofcourse,youdon’tdirectlygettoseethekernelcode

path.



@CaryMillsap 67


PARSE #42:c=10000,…








...OrthathelpfulgridthatIdrewforyou.



@CaryMillsap 68

WAIT #42: nam='latch: library cache'…PARSE #42:c=10000,…WAIT #42: nam='SQL*Net message to client'…EXEC #42:c=10000,…WAIT #42: nam='SQL*Net message from client'…WAIT #42: nam='latch: cache buffers chains'…WAIT #42: nam='SQL*Net message to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…WAIT #42: nam='SQL*Net message to client'…WAIT #42: nam='SQL*Net more data to client'…WAIT #42: nam='SQL*Net more data to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…WAIT #42: nam='SQL*Net message to client'…WAIT #42: nam='SQL*Net more data to client'…WAIT #42: nam='SQL*Net more data to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…WAIT #42: nam='SQL*Net message to client'…WAIT #42: nam='SQL*Net more data to client'…WAIT #42: nam='SQL*Net more data to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…


Allyougettoseeisthis.



@CaryMillsap

WAIT #42: nam='latch: library cache'…PARSE #42:c=10000,…WAIT #42: nam='SQL*Net message to client'…EXEC #42:c=10000,…WAIT #42: nam='SQL*Net message from client'…WAIT #42: nam='latch: cache buffers chains'…WAIT #42: nam='SQL*Net message to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…WAIT #42: nam='SQL*Net message to client'…WAIT #42: nam='SQL*Net more data to client'…WAIT #42: nam='SQL*Net more data to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…WAIT #42: nam='SQL*Net message to client'…WAIT #42: nam='SQL*Net more data to client'…WAIT #42: nam='SQL*Net more data to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…WAIT #42: nam='SQL*Net message to client'…WAIT #42: nam='SQL*Net more data to client'…WAIT #42: nam='SQL*Net more data to client'…FETCH #42:c=20000,…WAIT #42: nam='SQL*Net message from client'…

69

OracleextendedSQLtracedataOraclekernelcodepathbegin prepare CPU latch-related syscall CPU end preparebegin exec CPU write(SQLNET_OUT, result_to_client);end execread(SQLNET_IN, next_request_from_client);begin fetch CPU latch-related syscall CPU write(SQLNET_OUT, result_to_client);end fetchread(SQLNET_IN, next_request_from_client);begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results);end fetchread(SQLNET_IN, next_request_from_client);begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results);end fetchread(SQLNET_IN, next_request_from_client);begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results);end fetchread(SQLNET_IN, next_request_from_client);

Youcanlearntoenvisionthekernel’scodepaththatmotivated

yourtracefile.



@CaryMillsap

Therearelotsofwaystosummarizeatracefile.tkprof

SQLDeveloper[Trace]ViewerTraceAnalyzer

tvdxstatxtraceOraSRP

MethodRProfiler

70



@CaryMillsap

Profilingtracefilescanbeeasy.Youcanbuildtools,oryoucanbuythem.

71

Fn’m [ mifp_^ jli\f_g.



@CaryMillsap

Whatyoucandowithtracefiles

72



@CaryMillsap

Example1

73



@CaryMillsap 74

mrskew"r1-fixed.trc"

CALL-NAMEDURATION%CALLSMEANMINMAX--------------------------------------------------------------------------SQL*Netmessagefromclient1,403.92794299.7%2,1610.6496660.0000000.927028FETCH3.0135490.2%2,1610.0013950.0000000.005000directpathreadtemp1.2590220.1%830.0151690.0032870.046968SQL*Netmoredatatoclient0.1412130.0%2,4600.0000570.0000050.001269SQL*Netmessagetoclient0.0079640.0%2,1610.0000040.0000010.000376--------------------------------------------------------------------------TOTAL(5)1,408.349690100.0%9,0260.1560330.0000000.927028

99.7%ofthetimeis2,161networkround-trips.

WhatSQLstatementscausetheround-trips?



@CaryMillsap 75

mrskew--group='($sqlid=~/^#/?"":"[".$sqlid."]")'--gl=SQLID--name='messagefromclient'"r1-fixed.trc"

SQLIDDURATION%CALLSMEANMINMAX--------------------------------------------------------------[7d0bv6ds85q1f]1,403.927942100.0%2,1610.6496660.0000000.927028--------------------------------------------------------------TOTAL(1)1,403.927942100.0%2,1610.6496660.0000000.927028

Justone.All2,161round-tripsareexecutedonbehalfofjustoneSQLstatement.



@CaryMillsap 76

mrskew--rc=p10.rc--name='SQL\*Netmessagefromclient'"r1-fixed.trc"

RANGE{min≤e<max}DURATION%CALLSMEANMINMAX----------------------------------------------------------------------------1.0.0000000.0000010.0000000.0%10.0000000.0000000.0000002.0.0000010.0000103.0.0000100.0001004.0.0001000.0010005.0.0010000.0100006.0.0100000.1000007.0.1000001.0000001,403.927942100.0%2,1600.6499670.5471100.9270288.1.00000010.0000009.10.000000100.00000010.100.0000001,000.00000011.1,000.000000+∞----------------------------------------------------------------------------TOTAL(11)1,403.927942100.0%2,1610.6496660.0000000.927028

Eachround-tripconsumesanaverageof.649967≈.650s.

Why?



@CaryMillsap

App Oracle DB

time

77

~.001s

~.001s

~.650s~.648s

EachSQL*Netmessagefromclientcall(~.650s)lookslikethis.

Ifround-tripnetworklatencyis~.002s,thenthisexperienceisspending~.648sintheJavacodeexecutedbetweendatabasecalls.



@CaryMillsap 78

mrskew--name=':dbcall'--select='$row'--slabel=ROWS--precision=0"r1-fixed.trc"

CALL-NAMEROWS%CALLSMEANMINMAX-------------------------------------FETCH216,017100.0%2,16110017100-------------------------------------TOTAL(1)216,017100.0%2,16110017100

Onefinalcheck...

Thetracefileshowsthattheapplication,atleast,isfetchinganaverageof100rowsperfetchcall(perround-trip).

ThishelpsexplaintheJava-sidelatency,butstill,.648stoprocessjust100rowsneedssomeexplaining.



@CaryMillsap 79

mrskew"r1-fixed.trc"

CALL-NAMEDURATION%CALLSMEANMINMAX--------------------------------------------------------------------------SQL*Netmessagefromclient1,403.92794299.7%2,1610.6496660.0000000.927028FETCH3.0135490.2%2,1610.0013950.0000000.005000directpathreadtemp1.2590220.1%830.0151690.0032870.046968SQL*Netmoredatatoclient0.1412130.0%2,4600.0000570.0000050.001269SQL*Netmessagetoclient0.0079640.0%2,1610.0000040.0000010.000376--------------------------------------------------------------------------TOTAL(5)1,408.349690100.0%9,0260.1560330.0000000.927028

Nomatterhowlongyoutryto“fixthedatabase”here,you’regoingtoseeatmostonlya.3%differenceinresponsetime.

TheproblemhereisintheJava.



@CaryMillsap

Example2

80



@CaryMillsap 81

mrskew"prd1_ora_9031.trc"

CALL-NAMEDURATION%CALLSMEANMINMAX--------------------------------------------------------------------------PARSE735.42619778.9%6981.0536190.0000004.498316SQL*Netmessagefromclient104.76222911.2%1,3780.0760250.0003913.554818FETCH91.8000289.8%6800.1350000.0000000.506923dbfilesequentialread0.1046700.0%140.0074760.0010670.016408EXEC0.0839880.0%3490.0002410.0000000.002000gccrblock2-way0.0732330.0%960.0007630.0002800.001968gccurrentblock2-way0.0312980.0%470.0006660.0003610.001640gccurrentgrantbusy0.0280370.0%470.0005970.0001560.001508SQL*Netmoredatafromclient0.0258190.0%8370.0000310.0000000.002564CLOSE0.0189990.0%6980.0000270.0000000.00100012others0.0615760.0%1,6330.0000380.0000000.001687--------------------------------------------------------------------------TOTAL(22)932.416074100.0%6,4770.1439580.0000004.498316

PARSEcallsaccountfor78.9%oftheexperienceduration.

Thatisneverappropriate.



@CaryMillsap 82

mrskew--rc=p10.rc--name=parse"prd1_ora_9031.trc"

RANGE{min≤e<max}DURATION%CALLSMEANMINMAX--------------------------------------------------------------------------1.0.0000000.0000010.0000000.0%3070.0000000.0000000.0000002.0.0000010.0000103.0.0000100.0001004.0.0001000.0010000.0079920.0%80.0009990.0009990.0009995.0.0010000.0100000.0330000.0%330.0010000.0010000.0010006.0.0100000.1000007.0.1000001.0000008.1.00000010.000000735.385205100.0%3502.1011011.3337974.4983169.10.000000100.00000010.100.0000001,000.00000011.1,000.000000+∞--------------------------------------------------------------------------TOTAL(11)735.426197100.0%6981.0536190.0000004.498316

That’salotoftimespentparsing,andthesePARSEcallsarereallyexpensive.



@CaryMillsap 83

mrskew--name=parse--group='$sqlid'--gl=SQLID--sort=4nd"prd1_ora_9031.trc"

SQLIDDURATION%CALLSMEANMINMAX----------------------------------------------------------gkbss8w49204k4.1763630.6%3490.0119670.0000004.13537166kf30526wrgy3.1535210.4%13.1535213.1535213.1535213r3dhkb0z824v2.9115580.4%12.9115582.9115582.9115583tzra8a2a7pny1.6057570.2%11.6057571.6057571.6057572hycpfzdzsu983.1555200.4%13.1555203.1555203.1555206ppu3s1jszy3a2.2086650.3%12.2086652.2086652.20866566vkb784j9rcu1.9017110.3%11.9017111.9017111.9017115wamvs45j6nh41.4927730.2%11.4927731.4927731.492773dj1buvhxg7h191.4997720.2%11.4997721.4997721.49977241yrts4g94ghn1.6287530.2%11.6287531.6287531.628753340others711.69180496.8%3402.0932111.3337974.498316----------------------------------------------------------TOTAL(350)735.426197100.0%6981.0536190.0000004.498316

Onestatementwasparsed349times;atleast348ofthoseareunnecessary.*

Thereare350distinctSQLstatementsexecutedbythisreport....Whichisfunny,becauseyouknowthisreport,andyoudon’tremembertherebeingthatmany.*Actuallyall349areunnecessary,becausethetracefileshowsthatthere’sneveranEXECcallassociatedwithanyofthesePARSEcalls,butthat’sastoryforanotherday.



@CaryMillsap 84

mrskew--rc=distinct-texts.rc"prd1_ora_9031.trc"

SSQLIDDISTINCT-TEXTS%CALLSMEANMINMAX---------------------------------------------41518124977020.0%7011136423202577020.0%7011120477701237020.0%7011119285472397020.0%7011111389170666919.7%69111395741418510.3%349001---------------------------------------------TOTAL(6)350100.0%698101

Forthefirst5“sharedSQLid”valuesshownhere,thereare~70distinctstatementsthatcouldhavebeensharable.

Youshouldbeabletoreducetheparsecallcountfrom698to6,bywritingsharableSQLstatements,andpullingPARSEcallsoutofloops.



@CaryMillsap 85

mrskew"prd1_ora_9031.trc"


Beforeyourbosswillletyou“fix”thiscode,youhavetopredictthebenefit.

Reducingtheparsecountfrom698to6shouldreduceparsingdurationfrom~735to~7,asavingsofabout730s.Responsetimeshouldimprovefrom~932sto~200s,justfromeliminatingthePARSEcallsonly.



@CaryMillsap

OPTI

MIZE ANYTHING

M eT H O D R

Youmighthaveknownthatyoushould“usebindvariables,”butyoucouldn’thavequantifiedtheRimpactontheexperiencewithoutthistracefile.

86



@CaryMillsap

BASELINE:foreachinvoicenumber{ cursor=parse(“select...whereinvoice_number=”+number); exec(cursor); loopovertheresultsettofetchalltherows;}

87

BAD

Thisishorrific:

• UsestoomuchCPUforPARSEcalls

• Serializationonlibrarycacheandsharedpoollatches

• Consumestoomuchmemoryinthelibrarycache

• Mayexecutetoomanynetworkround-trips



@CaryMillsap

BASELINE:BADforeachinvoicenumber{ cursor=parse(“select...whereinvoice_number=”+number); exec(cursor); loopovertheresultsettofetchalltherows;}

FIX1“Hey,let’susebindvariables”:foreachinvoicenumber{ cursor=parse(“select...whereinvoice_number=:a1”); exec(cursor,number); loopovertheresultsettofetchalltherows;}

88

STILLBAD

Alittlebetter,butstillreallyawful:

• UsestoomuchCPUforPARSEcalls

• Serializationonlibrarycachelatches

• Maybe,toomanynetworkround-trips



@CaryMillsap

FIX1“Hey,let’susebindvariables”:STILLBADforeachinvoicenumber{ cursor=parse(“select...whereinvoice_number=:a1”); exec(cursor,number); loopovertheresultsettofetchalltherows;}

FIX2:cursor=parse(“select...whereinvoice_number=:a1”);foreachinvoicenumber{ exec(cursor,number); loopovertheresultsettofetchalltherows;}

89

BETTER

Better(only1PARSEcallnow!),butstilllotsofnetworkround-trips.



@CaryMillsap

FIX2:BETTERcursor=parse(“select...whereinvoice_number=:a1”);foreachinvoicenumber{ exec(cursor,number); loopovertheresultsettofetchalltherows;}

FIX3:cursor=parse(“ select...whereinvoice_number in(selectinvoicenumberfromwhereveryourforeachwasgettingthem)”);exec(cursor);loopovertheresultsettofetchalltherows;

90

Now,only1PARSEcall,only1EXECcall,andtheminimumpossiblenumberofnetworkround-trips.**Unlessthere’sawaytoreturnfewerrows.

BETTERYET



@CaryMillsap

Andsoon...

91



@CaryMillsap

BadSQLBadPL/SQL

SlownetworkMissingindexesParsinginaloop

HotblockproblemsNotenoughmemoryDisklatencyproblemsRowlockingproblems

Row-at-a-timeprocessingBaddatastructurechoice

HardwaremisconfigurationsToomuchloadonthesystem

OSparameterssetinadequatelyOracleparameterssetinadequatelySQLreturnsmorerowsthanitshould

Databasebuffercachehot/coldproblemsOraclequeryoptimizerchoosingbadplans

ReportsrunwithpoorlylimitingparametervaluesInefficientcodebetweendatabasecallsintheapplication 92

Atracefileshowsyouwhereyourtimehasgone.Performanceproblemscannothidefromthat.

@CaryMillsap

Thereareonlytwopossiblerootcauses

foranyresponsetimeproblem:

❶Callcountistoobig.

❷Latencyistoobig.*

*Probablybecausesomeoneelse’scallcountsaretoobig.

93

#ProTip



@CaryMillsap 94

mrskew--top=10"prd1_ora_9031.trc"


SeehowthereareonlytwowaystoreduceaDURATION?YouhavetheCALLScolumn,andtheMEANcolumn.

ProfileslikethismakeiteasytoseehowanythingyoudotomakesomethinggofastermusttranslatetoamanipulationofeitherCALLSorMEAN.



@CaryMillsap

Withagoodtracefile,youcanpredictthe

responsetimeimpactofaproposedchange.*

*Thisisnearlyimpossibletodowithsystemwideaggregatedstatistics.

95

#ProTip



@CaryMillsap 96

Itjusttakespractice.



@CaryMillsap

Conclusion

97



@CaryMillsap

Yourcodedoesstuff.

IncludingsomestuffinsideOracle.

Thetimethisstufftakesisyouruser’sresponsetime.

Youcanseeexactlywhatitis.

It’snotthathard.

98



@CaryMillsap

References

99



@CaryMillsap

RobynSands,etal.2010.ExpertOraclePractices.Apress

DetailedinformationaboutinstrumentingyourOracleapplicationcode.

CaryMillsap.2011.MasteringOracleTraceData.MethodRCorporation

Textbookfor1-daycoursethatteachesyouhowtomasterOracletracedata.

RonCrisco,etal.2011.ExpertPL/SQLPractices.Apress

DetailedinformationaboutinstrumentingyourOracleapplicationcode.

CaryMillsap,JeffHolt.2003.OptimizingOraclePerformance.O’Reilly

DetailedinformationaboutOracletracedataandwhattodowithit.

100



http://www.amazon.com/gp/product/1430226684/ref=as_li_ss_tl?ie=UTF8&tag=methodrcom-20&linkCode=as2&camp=1789&creative=390957&creativeASIN=1430226684

http://www.amazon.com/gp/product/1430226684/ref=as_li_ss_tl?ie=UTF8&tag=methodrcom-20&linkCode=as2&camp=1789&creative=390957&creativeASIN=1430226684

http://method-r.com/courses/trace-data-masterclass

http://method-r.com/courses/trace-data-masterclass

http://www.amazon.com/gp/product/1430234857/ref=as_li_tf_tl?ie=UTF8&tag=methodrcom-20&linkCode=as2&camp=217145&creative=399373&creativeASIN=1430234857

http://www.amazon.com/gp/product/1430234857/ref=as_li_tf_tl?ie=UTF8&tag=methodrcom-20&linkCode=as2&camp=217145&creative=399373&creativeASIN=1430234857

http://www.amazon.com/gp/product/059600527X/ref=as_li_tf_tl?ie=UTF8&tag=methodrcom-20&linkCode=as2&camp=217145&creative=399369&creativeASIN=059600527X

http://www.amazon.com/gp/product/059600527X/ref=as_li_tf_tl?ie=UTF8&tag=methodrcom-20&linkCode=as2&camp=217145&creative=399369&creativeASIN=059600527X

@CaryMillsap 101

method-r.com www.enkitec.commethod-r.com/facebook facebook.com/enkitec@MethodR @[email protected] [email protected]

Q&A



http://method-r.com

http://method-r.com

http://www.enkitec.com

http://www.enkitec.com

http://method-r.com/facebook

http://method-r.com/facebook

http://facebook.com/enkitec

http://facebook.com/enkitec

http://twitter.com/@MethodR

http://twitter.com/@MethodR

http://twitter.com/@Enkitec

http://twitter.com/@Enkitec

mailto:[email protected]?subject=Presentation%20followup

mailto:[email protected]?subject=Presentation%20followup

mailto:[email protected]?subject=Presentation%20follow-up

mailto:[email protected]?subject=Presentation%20follow-up

how to find and fix your oracle-based application performance problem

Data & Analytics