![Page 1: Cloud Observation and Performance Analysis using Solaris 11 DTrace](https://reader033.vdocuments.mx/reader033/viewer/2022050906/553b97754a7959eb2d8b46c7/html5/thumbnails/1.jpg)
Amit Hurvitz, ISV Engineering, Oracle
DTrace, for Solaris (zones inside) and Java
![Page 2: Cloud Observation and Performance Analysis using Solaris 11 DTrace](https://reader033.vdocuments.mx/reader033/viewer/2022050906/553b97754a7959eb2d8b46c7/html5/thumbnails/2.jpg)
Program Agenda
Introductory Demo What's Dtrace? Enabling Dtrace from Inside the Zone Java Statically Defined Tracing TCP Client and Server Flow Tracing - Demo Future Thoughts
Introductory Demo What's Dtrace? Enabling Dtrace from Inside the Zone Java Statically Defined Tracing TCP Client and Server Flow Tracing - Demo Future Thoughts
![Page 3: Cloud Observation and Performance Analysis using Solaris 11 DTrace](https://reader033.vdocuments.mx/reader033/viewer/2022050906/553b97754a7959eb2d8b46c7/html5/thumbnails/3.jpg)
Introductory DemoIntroductory Demo
![Page 4: Cloud Observation and Performance Analysis using Solaris 11 DTrace](https://reader033.vdocuments.mx/reader033/viewer/2022050906/553b97754a7959eb2d8b46c7/html5/thumbnails/4.jpg)
What's DTrace? - cont.
Zero performance impact when not in use Completely safe; no way to cause panics, crashes, data corruption or
pathological performance degradation Powerful data management primitives eliminate need for most post-
processing
![Page 5: Cloud Observation and Performance Analysis using Solaris 11 DTrace](https://reader033.vdocuments.mx/reader033/viewer/2022050906/553b97754a7959eb2d8b46c7/html5/thumbnails/5.jpg)
DTrace – D Language
probe descriptionprobe description
/predicate//predicate/ {{ actionsactions }}
Probes Probes Which events we are interested in monitoring Predicates (optional) Predicates (optional) When do we want to monitor the events Actions (optional) Actions (optional) What do we want to do when the above happens
One liner # dtrace -n 'probeprobe/predicate//predicate/{actions}{actions}'
![Page 6: Cloud Observation and Performance Analysis using Solaris 11 DTrace](https://reader033.vdocuments.mx/reader033/viewer/2022050906/553b97754a7959eb2d8b46c7/html5/thumbnails/6.jpg)
DTrace – Probes
Programmable sensors (points of instrumentation) made available by providers placed all over the Solaris system
provider:module:function:name tcp:ip:tcp_send:entry Syscall:::
Providers: syscall,io,pid,profile, hotspot, tcp, udp, ip, iscsi,...
Modules: nfs, zfs, cpc, …
Names: entry,return
Listing Probes # dtrace -l [-P provider | -m module | -f function name | -n name]
![Page 7: Cloud Observation and Performance Analysis using Solaris 11 DTrace](https://reader033.vdocuments.mx/reader033/viewer/2022050906/553b97754a7959eb2d8b46c7/html5/thumbnails/7.jpg)
DTrace – Predicates and Actions
Predicates /cpu == 0/ /execname == “date”/ /ppid != 0 && arg0 != 0/
Actions Commands separated by “;” trace(execname) printf(“%s %s %s”, execname, probefunc, copyinstr(arg0));
Predefined Variables execname, probefunc, pid, ppid, cpu, timestamp,
arg0, arg1, ...
![Page 8: Cloud Observation and Performance Analysis using Solaris 11 DTrace](https://reader033.vdocuments.mx/reader033/viewer/2022050906/553b97754a7959eb2d8b46c7/html5/thumbnails/8.jpg)
DTrace – Aggregations
Used to aggregate data and look for trends Has the general form:
@name[keys] = aggfunc(args)@name[keys] = aggfunc(args)
Aggregating functions count(), sum(), avg(), min(), max(), quantize(), lquantize()
Examples: % dtrace -n 'syscall::read*:entry{@[execname]=count();}' % dtrace -n 'syscall::read*:entry{@[execname,arg0]=count();}'
![Page 9: Cloud Observation and Performance Analysis using Solaris 11 DTrace](https://reader033.vdocuments.mx/reader033/viewer/2022050906/553b97754a7959eb2d8b46c7/html5/thumbnails/9.jpg)
An Example – Off-CPU Tracing#!/usr/sbin/dtrace -sBEGIN{ start_timestamp = timestamp;}
sched:::off-cpu/pid == $1/{ self->ts = timestamp;}
sched:::on-cpu/self->ts/{ @[stack(), jstack(), "ns"] = sum(timestamp - self->ts); self->ts = 0;}
END{ printf("elapsed time: %d\n", timestamp - start_timestamp);}
![Page 10: Cloud Observation and Performance Analysis using Solaris 11 DTrace](https://reader033.vdocuments.mx/reader033/viewer/2022050906/553b97754a7959eb2d8b46c7/html5/thumbnails/10.jpg)
Enabling DTrace in a Zone
![Page 11: Cloud Observation and Performance Analysis using Solaris 11 DTrace](https://reader033.vdocuments.mx/reader033/viewer/2022050906/553b97754a7959eb2d8b46c7/html5/thumbnails/11.jpg)
Java Statically Defined Tracing (JSDT)
Insert your own DTrace probes in desired locations inside your methods
– Use them in conjunction with any other probes
Make points of interest in your application easily monitored
#!/usr/sbin/dtrace -s
MyProvider:::start{ self->start_time = timestamp;}
syscall:::entry/self->start_time/{ @[probefunc] = quantize();}
![Page 12: Cloud Observation and Performance Analysis using Solaris 11 DTrace](https://reader033.vdocuments.mx/reader033/viewer/2022050906/553b97754a7959eb2d8b46c7/html5/thumbnails/12.jpg)
JSDT – define a Provider
Provider Interfacepublic interface MyProvider extends com.sun.tracing.Provider {
void methodEntry();void methodReturn();void start();void dataAdded(int x, int y);void myProbe(int intData, String stringData);
}
![Page 13: Cloud Observation and Performance Analysis using Solaris 11 DTrace](https://reader033.vdocuments.mx/reader033/viewer/2022050906/553b97754a7959eb2d8b46c7/html5/thumbnails/13.jpg)
JSDT – insert probes to your Java code
Provider Interface
Call probes actions
Create the provider
import com.sun.tracing.*;
MyProvider provider;
public static void main(String args) { ProviderFactory factory = ProviderFactory.getDefaultFactory(); provider = factory.createProvider(MyProvider.class);}
public void method() { provider.methodEntry(); ... provider.myProbe(i, str); ... provider.methodReturn();}
![Page 14: Cloud Observation and Performance Analysis using Solaris 11 DTrace](https://reader033.vdocuments.mx/reader033/viewer/2022050906/553b97754a7959eb2d8b46c7/html5/thumbnails/14.jpg)
Flow Trace- DemoFlow Trace- Demo
![Page 15: Cloud Observation and Performance Analysis using Solaris 11 DTrace](https://reader033.vdocuments.mx/reader033/viewer/2022050906/553b97754a7959eb2d8b46c7/html5/thumbnails/15.jpg)
Next Thoughts
A special ‘Java-DTrace’ utility to do implicit instrumentation Probes look like native DTrace PID provider:
JDDT$target:class-name:method-name:entry JDDT$target:class-name:method-name:return
# jdtrace java-dtrace-script.d -p <process-id> jdtrace will take care of all required dynamic instrumentation
Clean instrumented code on script end Any suggestions?