dtrace talk at oracle open world

1

Thursday, September 23, 2010

<Insert Picture Here>

Oracle Performance Measurement and Tuning with Solaris DTraceJeff Savit Principal Sales Consultant

Angelo RajaduraiPrincipal Software Engineer


3

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


4

<Insert Picture Here>

Agenda

• Basic Concepts of Performance Tuning• Performance Tuning Features in Solaris• DTrace - Dynamically Observing the Entire

Application Stack• Dynamic Tracing Oracle WebLogic Server with

DTrace• Further Reading


5

Basic Concepts for Performance Management

Define Goals

Measure Performance

Analyze Model &Deduce

Take Action

PerformanceModel


Start with the Best PlatformSolaris - Designed for Performance & Scale

6

Dynamic, multithreaded, preemptive kernel and process modelTightly integrated file system and virtual memory64-bit kernel, supporting both 32-bit and 64-bit applicationsResource managementOptimized, SMP-hot network stackCPU and memory scalability from low to high. Today, up to:

256 CPUs on T5440, 512 on M9000; 4TB RAM on M9000Optimized for modern multicore, multithread, NUMA systemsPremier x86 support optimized for Intel Nehalem and AMD OpteronSolaris Containers (Zones): negligible-overhead, built in virtualization


• High system time is not a problem• You may be asking the system to do stuff

• Slow I/O is not a problem• You may be streaming bulk I/O

• 100% utilization is not a problem• You are using all the box you paid for :-)• Modern CPUs, NICs, disks, can do more than 1 thing at a

time without adding delay• Fine even if saturated unless some work unable to get

enough cycles to meet SLA

• None of these are problems unless they relate to business metrics

Define the Problem

7


8

Native Apps

ruby

C C++

Database Oracle 11g

Operating System Solaris/OEL/Windows

Container WebLogic Server

Frameworks

Monitor: Right Location & Tools

PresentationJSP/Javascript/Ajax/JavaFX

python

php

perl

shell

Apps run in VM


Solaris Tool chest

9

Proc tools

cputrack - per-processor hw counters pargs – process arguments pflags – process flags pcred – process credentialspldd – process's library dependencies plockstat – process lock statistics psig – process signal disposition pstack – process stack dumppmap – process memory map pfiles – open files and names prstat – process statistics ptree – process treeptime – process microstate times pwdx – process working directorypgrep – grep for processes pkill – kill processes list pstop – stop processes prun – start processes prctl – view/set process resources pwait – wait for processpreap – reap a zombie process

Tracing & Debugging

abitrace - trace ABI interfaces mdb – debug process/kernel

truss – trace function and system calls

System Statistics

acctcom – process accounting busstat – Bus hardware counters

cpustat – CPU hardware counters iostat – IO & NFS statistics

kstat – display kernel statistics mpstat – processor statistics

netstat – network statistics nfsstat – nfs server stats

sar – kitchen sink utility vmstat – virtual memory stats

Toolkits

DTraceToolkit – DTrace Tools K9Toolkit – perl perf tools

nicstat – network stats


10

perfbarsee the big picture from across the room

Don't be shy about starting with a simple graphical tool!

http://blogs.sun.com/partnertech/resource/tools/perfbar.{sparc,i386}

See:LDOM Booth for use of perfbar


http://livepage.apple.com/




11

FenxiReading a 1000 lines of text does not show you the big picture

Open source tool written at Sun for aggregating and displaying vmstat, mpstat, iostat data

https://fenxi.dev.java.net/


https://fenxi.dev.java.net

https://fenxi.dev.java.net

12

So why do we need another tool?


13

Native Apps

ruby

C C++

Database Oracle 11g

Operating System Solaris/OEL/Windows

Container WebLogic Server

Frameworks

Observing the Stack is hard

PresentationJSP/Javascript/Ajax/JavaFX

python

php

perl

shell

Apps run in VM


The Blame Game

14

Devel

oper:

“I’d as

k the

datab

ase ad

min” Database Admin:

“May be the middleware guys

will know”

Sysadmin

“I don’t know! Ask the

Developer” Middlew

are Ex

pert:

“Lets

ask th

e Sysa

dmin”


Static instrumentation is time consuming

15

Instrument Validate

Run in production

Collect infoProcess

Add Code Compile

Custom Instruments

Stop AppRestart w/ flags

Prebuilt Instruments


Debugging Transient Problems In production. Impossible!

16

Please let me observe a live system.


• turn instrumentation ON and OFF dynamically

• probe any arbitrary location• not just pre-defined probe points

• collect arbitrary data • have low probing overhead• have no overhead for disabled probes

If Only We Can ...

17


Dynamic Tracing

18

Native

cc+ph

perl pyth

oru

b

shelDatabase

Operating System Solaris/Linux/OS X/

Container Glassfish/

Frameworks

Application Logic

PresentationJSP/Javascript/Ajax

12::SELECT id, name, symbol, conversion_rate FROM currencies WHEREstatus = 'Active' and deleted = 012::SELECT category, name, value FROM config12::SELECT id FROM outbound_email WHERE type = 'system'

-> foo()-> bar()<- bar()

<- foo()


19

/ /predicate when

{

}

actions do what

D - Scripts

: : :provider module probe name whereprobe


DTrace examples

20

nthreads{ printf("new thread created by %s(%d)\n", execname,pid);}

pid$1::malloc:entry{

@=quantize(arg0);}

malloc size observer

Thread monitor

pid$1:::entry{

@[probefunc]=count();}

hot methodfinder


More DTrace examples

21

profile-101/execname==”java”/{

@[tid]=count();}

pid$1::sleep:entry{

@[ustack()]=count();}

Who went to sleep?

How many Java threads are active?

io:::done{

@=quantize(args[0]->b_bcount);}

Size distribution of IO


DTrace and Java

• DTrace can observe the following in Java• When VM starts and ends• When thread starts and ends• When class gets loaded and unloaded• When object allocated and freed• When GC starts and ends• JNI calls• When a method is called and method returns

22


DTrace and Java - Example

23

hotspot$1:::class-loaded{ printf("%s loaded\n", copyinstr(arg0,arg1));}

hotspot$1:::method-entry{

@[copyinstr(arg1,arg2), copyinstr(arg3,arg4)]=count();}

JVM needs to be started with -XX:+ExtendedDTraceProbes

Class Loading

Java Methods


DTrace and Java - More Example

24

GC

#!/usr/sbin/dtrace -qshotspot$1:::gc-begin{ printf("GC started at %Y\n",walltimestamp); printf("%20s | %-10s\n", "pool", "time(ms)"); printf(" ------------------------------------\n"); self->ts = timestamp;}

hotspot$1:::mem-pool-gc-begin{ self->pool_ts[copyinstr(arg2,arg3)] = timestamp;}

hotspot$1:::mem-pool-gc-end{ self->pool = copyinstr(arg2,arg3); printf("%20s | %-10d\n", self->pool, (timestamp - self->pool_ts[self->pool])/1000);}

hotspot$1:::gc-end/self->ts/{ printf(" ------------------------------------\n"); printf("%20s | %-10d\n\n","Total GC time", (timestamp - self->ts)/1000);}


Other Providers

25

DTrace providers

javaperlruby

javascript mysqlpostgres

cpciptcpudp

plockstatnfsv3

nfsv4

kerberos

sh

iSCSI

USDT

FC

DTracecore provider

DTracecore

IOsyscall

profilesysinfo

vminfo

PID

mib

fbtproc

lockstat

python


DTrace Eco-Systems

26

• DTraceToolkit• Collection of useful D-scripts

• Chime• DTrace Data Visualization

• D-Light• Sun Studio based tool to observe application using DTrace

• DTrace analytics• DTrace based front end for Sun ZFS storage


DTrace & Oracle WebLogic Server

27


Instrumenting JVM vs Byte Code

28

JVM

Garbage Collector

Class Loader

JIT Compiler

Execution Engine

Byte Code

gc-begingc-end

class-loadedclass-unloaded

method-compile-beginmethod-compile-end

method-entrymethod-return


Instrumenting JVM vs Byte Code

29

JVM

Garbage Collector

Class Loader

JIT Compiler

Execution Engine

Byte Code @OnMethod{ clazz=”myclass”, method=”mymeth”}

BTrace Supports other annotations@OnTimer, @OnError, @OnEvent & @OnLowMemory


30

{

}

action do what

BTrace Code Structure

@OnMethod ( clazz = “java.lang.thread”,method = “start” where

import com.sun.btrace.annotations.*;import statuc com.sun.btrace.BTraceUtil.*;@BTracepublic class BTraceSample {

)

public static void func()


Sample B-script

31

import com.sun.btrace.annotations.*;import static com.sun.btrace.BTraceUtils.*;

@BTrace public class ThreadStart { @OnMethod( clazz="java.lang.Thread", method="start" ) public static void onnewThread(@Self Thread t) { println(strcat("starting ", name(t))); }}

New Java Thread

Tracer


More B-scripts

32

@BTrace public class AllMethods { private static Aggregation count = newAggregation(AggregationFunction.COUNT);

@OnMethod( clazz="/javax\\.swing\\..*/", method="/.*/" ) public static void m(@ProbeClassName String probeClass, @ProbeMethodName String probeMethod) { AggregationKey key = newAggregationKey(strcat(probeClass,strcat(".",probeMethod))); addToAggregation(count,key,1); }

@OnEvent public static void onEvent() { truncateAggregation(count,10); printAggregation("Count", count); }}

Count all javax.swing methods


DTrace Vs BTrace

33

DTrace for JVM BTrace

Dynamic tracing entire system Only within a Java Process

Instrument JVM Instrument Java Byte Code

D-script Java like instrumentation Lang

Link Java to entire stack Just within Java

High overhead for method tracing

Low overhead for method tracing


Combining DTrace & BTrace

34

DTrace BTraceBest

ofBoth

Worlds

DFr

ame


Dynamic Tracing Framework for Oracle WebLogic Server

35


The DFrame config file• The Framework is controlled using the config file• Two types of probes

• mapped• config file format

class:method:entry/exit:logical name• probe format

weblogic*:::event(logical name, class.method)• unmapped

• config file formatclass:method

• probe formatweblogic*:::method-entry(class, method)weblogic*:::method-return(class,method)

36


How to use the Framework• Only one command to remember

embedProbes• Usage

embedProbes <pid>embedProbes -f configFile <pid>embedProbes -d [debug on]embedProbes -B [save B-script]

• The command reads config file, creates the b-script and embedded dprobes into weblogic

37


38

Demonstration


Reducing the Observer Effect

39

• Use low overhead tools✓dframe overhead for most probes very low (<3%)

• Ability to dynamically enable and disable probes✓dframe can be turned on and off dynamically

• Measure the probe effect before running in production✓ex: for medrec application on a Sun T5140

for processing 2016 transactions,this is 487,240 events the overhead us just %3.8


Learn more

• White paper on DFrame– http://bit.ly/DFrame

• Need pre-release access?– email [email protected]

40


http://bit.ly/dframe-paper

http://bit.ly/dframe-paper

mailto:[email protected]

mailto:[email protected]

DTrace Resources

41

• DTrace Community Page• http://hub.opensolaris.org/bin/view/Community+Group+dtrace/

• DTrace Documentation• http://wikis.sun.com/display/DTrace/Documentation

• DTraceToolkit• http://hub.opensolaris.org/bin/view/Community+Group+dtrace/dtracetoolkit

• DTrace Chime• http://hub.opensolaris.org/bin/view/Project+dtrace-chime/

• DTrace Hands on Lab• http://dtracehol.com/ - AMI ami-22fd164b (dtrace-hol)


http://hub.opensolaris.org/bin/view/Community+Group+dtrace/

http://hub.opensolaris.org/bin/view/Community+Group+dtrace/

http://wikis.sun.com/display/DTrace/Documentation

http://wikis.sun.com/display/DTrace/Documentation

http://hub.opensolaris.org/bin/view/Community+Group+dtrace/dtracetoolkit

http://hub.opensolaris.org/bin/view/Community+Group+dtrace/dtracetoolkit

http://hub.opensolaris.org/bin/view/Project+dtrace-chime/

http://hub.opensolaris.org/bin/view/Project+dtrace-chime/

http://dtracehol.com

http://dtracehol.com

42


dtrace talk at oracle open world

Documents