secrets of building a debuggable runtime: learn how language implementors solve your runtime issues

J9, OMR and beyond

Serviceable Runtimes

IBM Runtime Technologies 2

• Bjørn Vårdal

• IBM Runtimes Developer

• J9 JVM

• @bvaardal

• MethodHandles

• Lambdas

• VarHandles

• Project Valhalla

Bjørn Vårdal

Introduction

IBM Runtime Technologies 3Bjørn Vårdal

Most die young_______runtimes

unless you service them.


The information contained in this presentation is provided for informational purposes only.

Whilst efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided “as is”, without warranty of any kind, express or implied.

All performance data included in this presentation have been gathered in a controlled environment. Your own test results may vary based on hardware, software or infrastructure differences.

All data included in this presentation are meant to be used only as a guide.

In addition, the information contained in this presentation is based on IBM’s current product plans and strategy, which are subject to change by IBM, without notice.

IBM and its affiliated companies shall not be responsible for any damages arising out of the use of, or otherwise related to, this presentation or any other documentation.

Nothing contained in this presentation is intended to, or shall have the effect of:

- creating any warrant or representation from IBM, its affiliated companies or its or their suppliers and/or licensors

Bjørn Vårdal

Disclaimer


• Advanced core dump analysis for J9

• Used by the J9 service team

• Used by J9 developers

• Front-end: DDR Interactive

Bjørn Vårdal

Direct Dump Reader


$ java -cp j9ddr.jar com.ibm.j9ddr.tools.ddrinteractive.DDRInteractive core.12345

Bjørn Vårdal

DDR Interactive

Ships with J9



Bjørn Vårdal

DDR Interactive



*0 : PID: 2824; !j9javavm 0x7f27480eef0Run !j9help to see a list of commands>

Bjørn Vårdal

DDR Interactive



*0 : PID: 2824; !j9javavm 0x7f27480eef0Run !j9help to see a list of commands> !j9help

Bjørn Vårdal

DDR Interactive



*0 : PID: 2824; !j9javavm 0x7f27480eef0Run !j9help to see a list of commands> !j9help

threads Lists VM threadsstack Walks the Java stack for <thread>stackslots Walks the Java stack (including objects) for <thread>...classforname Find the class corresponding to name (with wildcards)dumpromclass Dump the specified J9ROMClass. Wild cards are allowed in class name.findpattern Search memory for a specific patternwhatis Determine the type of structure at a given address

Bjørn Vårdal

DDR - !j9help


$ java -cp j9ddr.jar com.ibm.j9ddr.tools.ddrinteractive.DDRInteractive core.20170315.010552.2824.01.dmp

*0 : PID: 2824; !j9javavm 0x7f27480eef0Run !j9help to see a list of commands> !threads

Bjørn Vårdal

DDR - !threads




!stack 0x0228d70 !j9vmthread 0x0228d70 !j9thread 0x07f274807d70 tid 0x4b (75) // (main)!stack 0x02294f0 !j9vmthread 0x02294f0 !j9thread 0x7f27480b0bf8 tid 0x4e (78) // (JIT Compilation Thread-1)!stack 0x022af20 !j9vmthread 0x022af20 !j9thread 0x7f27480bfcb0 tid 0x55 (85) // (JIT-SamplerThread)!stack 0x022b2e0 !j9vmthread 0x022b2e0 !j9thread 0x7f27480c0228 tid 0x56 (86) // (IProfiler)!stack 0x0238620 !j9vmthread 0x0238620 !j9thread 0x7f2748238170 tid 0x57 (87) // (Common-Cleaner)!stack 0x023f480 !j9vmthread 0x023f480 !j9thread 0x7f27482386e8 tid 0x58 (88) // (Signal Dispatcher)!stack 0x023f840 !j9vmthread 0x023f840 !j9thread 0x7f27483d01f8 tid 0x5a (90) // (Concurrent Mark Helper)!stack 0x0023fc0 !j9vmthread 0x0023fc0 !j9thread 0x7f27483d0c90 tid 0x5b (91) // (GC Slave)!stack 0x0002420 !j9vmthread 0x0002420 !j9thread 0x007f26e084d0 tid 0x5f (95) // (Attach API wait loop)!stack 0x0240730 !j9vmthread 0x0240730 !j9thread 0x7f27483d2218 tid 0x60 (96) // (Finalizer thread)

>

Bjørn Vårdal

DDR - !threads





>

Bjørn Vårdal

DDR - !threads


> !stack 0x0228d70<1a2970> Generic special frame<1a2970> !j9method 0x01B9A718 X.print(Ljava/lang/String;)V<1a2970> !j9method 0x01B9A6F8 X.m(I)V<1a2970> !j9method 0x01B9A6D8 X.main([Ljava/lang/String;)V<1a2970> JNI call-in frame<1a2970> Native method frame

>

Bjørn Vårdal

DDR - !stack


> !stackslots 0x0228d70 <1a2970> *** BEGIN STACK WALK, flags = 0401 walkThread = 0x01A2970 ***<1a2970> ITERATE_O_SLOTS<1a2970> RECORD_BYTECODE_PC_OFFSET<1a2970> Initial values: walkSP = 0x01AE6D60, PC = 0x01, literals = 0x0, A0 = 0x01AE6D78, j2iFrame = 0x0, ...<1a2970> Generic special frame: bp = 0x01AE6D78, sp = 0x01AE6D60, pc = 0x01, cp = 0x0, arg0EA = 0x01AE ...<1a2970> Bytecode frame: bp = 0x01AE6D90, sp = 0x01AE6D80, pc = 0x07FDCB43A6344, cp = 0x01B9A580, arg0EA = 0x01AE6D98, ...<1a2970> Method: X.print(Ljava/lang/String;)V !j9method 0x01B9A718<1a2970> Bytecode index = 16<1a2970> Using local mapper<1a2970> Locals starting at 0x01AE6D98 for 0x01 slots<1a2970> I-Slot: a0[0x01AE6D98] = 0x0FFEF46D0<1a2970> Bytecode frame: bp = 0x01AE6DB0, sp = 0x01AE6DA0, pc = 0x07FDCB43A630A, cp = 0x01B9A580, arg0EA = 0x01AE6DB8, ...<1a2970> Method: X.m(I)V !j9method 0x01B9A6F8<1a2970> Bytecode index = 6<1a2970> Using local mapper<1a2970> Locals starting at 0x01AE6DB8 for 0x01 slots<1a2970> I-Slot: a0[0x01AE6DB8] = 0x01234<1a2970> Bytecode frame: bp = 0x01AE6DD0, sp = 0x01AE6DC0, pc = 0x07FDCB43A62DB, cp = 0x1B9A580, arg0EA = 0x1AE6DD8, ...<1a2970> Method: X.main([Ljava/lang/String;)V !j9method 0x01B9A6D8<1a2970> Bytecode index = 3<1a2970> Using local mapper<1a2970> Locals starting at 0x01AE6DD8 for 0x01 slots<1a2970> I-Slot: a0[0x01AE6DD8] = 0x0FFEE5668<1a2970> JNI call-in frame: bp = 0x01AE6E0, sp = 0x01AE6DE0, pc = 0x07FDCB96DE314, cp = 0x0, arg0EA = 0x01AE6E0, ...<1a2970> New ELS = 0x0<1a2970> JNI native method frame: bp = 0x01AE6E88, sp = 0x01AE6E08, pc = 0x07, cp = 0x0, arg0EA = 0x01AE6E88, ...<1a2970> Object pushes starting at 0x01AE6E08 for 12 slots

Bjørn Vårdal

DDR - !stackslots



Bjørn Vårdal

DDR - !stackslots


> !j9object 0x0FFEF46D0

J9VMJavaLangString at 0x00000000FFEF46D0 {

struct J9Class* clazz = !j9class 0x1A5B500 // java/lang/StringObject flags = 0x00000000;[B value = !fj9object 0x86f36828 (offset=0) (java/lang/String)I count = 0x00000007 (offset=4) (java/lang/String)I hashCode = 0x00000000 (offset=8) (java/lang/String)"foo4660"

}

Bjørn Vårdal

DDR - !j9object


GDB• Requires intimate knowledge

about the JVM internal structures

• Requires debug symbols• Source must match• Platform specific

DDR• Has intimate knowledge about

the JVM internal structures

• Doesn’t require debug symbols• Cross JVM version• Cross platform

Bjørn Vårdal

GDB vs DDR


> !stackslots 0x0228d70 <1a2970> *** BEGIN STACK WALK, flags = 0401 walkThread = 0x01A2970 ***<1a2970> ITERATE_O_SLOTS<1a2970> RECORD_BYTECODE_PC_OFFSET<1a2970> Initial values: walkSP = 0x01AE6D60, A0 = 0x01AE6D78, <1a2970> Generic special frame: bp = 0x01AE6D78, sp = 0x01AE6D60,<1a2970> Bytecode frame: bp = 0x01AE6D90, sp = 0x01AE6D80, arg0EA = <1a2970> Method: X.print(Ljava/lang/String;)V !j9method 0x01B9A718<1a2970> Bytecode index = 16<1a2970> Using local mapper<1a2970> Locals starting at 0x01AE6D98 for 0x01 slots<1a2970> I-Slot: a0[0x01AE6D98] = 0x0FFEF46D0<1a2970> Bytecode frame: bp = 0x01AE6DB0, sp = 0x01AE6DA0, pc = <1a2970> Method: X.m(I)V !j9method 0x01B9A6F8<1a2970> Bytecode index = 6<1a2970> Using local mapper<1a2970> Locals starting at 0x01AE6DB8 for 0x01 slots<1a2970> I-Slot: a0[0x01AE6DB8] = 0x01234<1a2970> Bytecode frame: bp = 0x01AE6DD0, arg0EA = 0x1AE6DD8<1a2970> Method: X.main([Ljava/lang/String;)V !j9method 0x01B9A6D8<1a2970> Bytecode index = 3<1a2970> Using local mapper<1a2970> Locals starting at 0x01AE6DD8 for 0x01 slots<1a2970> I-Slot: a0[0x01AE6DD8] = 0x0FFEE5668<1a2970> JNI call-in frame: sp = 0x01AE6DE0, pc = 0x07FDCB96DE314<1a2970> JNI native method frame: cp = 0x0, arg0EA = 0x01AE6E88, ...<1a2970> Object pushes starting at 0x01AE6E08 for 12 slots

(gdb) x/200 0x0000000001AE6D600x1ae6d60: 0x00000000 0x00000000 0x01b9a718 0x000000000x1ae6d70: 0xb43a6344 0x00007fdc 0x01ae6d9a 0x000000000x1ae6d80: 0x01b9a6f8 0x00000000 0xb43a630a 0x00007fdc0x1ae6d90: 0x01ae6db8 0x00000000 0xffef46d0 0x000000000x1ae6da0: 0x01b9a6d8 0x00000000 0xb43a62db 0x00007fdc0x1ae6db0: 0x01ae6dd8 0x00000000 0x00001234 0x000000000x1ae6dc0: 0x00000000 0x00000000 0xb96de314 0x00007fdc0x1ae6dd0: 0x01ae6e00 0x00000000 0xffee5668 0x000000000x1ae6de0: 0x00000000 0x00000000 0x00020000 0x000000000x1ae6df0: 0x00000060 0x00000000 0x00000007 0x000000000x1ae6e00: 0x01ae6e8a 0x00000000 0xffee5668 0x000000000x1ae6e10: 0x86ef0828 0x00000000 0x86f317d0 0x000000000x1ae6e20: 0x86f317d0 0x00000000 0x00000000 0x000000000x1ae6e30: 0x00000000 0x00000000 0x86f2ed50 0x000000000x1ae6e40: 0x86ef3548 0x00000000 0x86ef34b8 0x000000000x1ae6e50: 0x86ef0240 0x00000000 0x86ef0fb0 0x00000000

DDR

Bjørn Vårdal

GDB vs DDRGDB


DDR Interactive”The tool that I use the most and that saves me the most time.”

J9 Service Team Lead

“We can implement commands to look for known symptoms, which drastically reduces our triage work.”

J9 Service Team Member


• We are open-sourcing the J9 JVM

• DDR coming along for the ride

• http://openj9.org

Bjørn Vårdal

OpenJ9

http://openj9.org/


Reliable Shared Components for High Performance Language Runtimes

http://www.eclipse.org/omrhttps://github.com/eclipse/omr

https://developer.ibm.com/open/omr

Bjørn Vårdal

Eclipse OMR

http://www.eclipse.org/omr

https://github.com/eclipse/omr

https://developer.ibm.com/open/omr


Agenda• DDR in J9• Capabilities• How DDR is implemented

• OMR ddrgen• Generating the DDR resources

• DDR in a new runtime• Adding DDR capabilities to Base9• Loading a Base9 core into DDR Interactive


• Print structure at address!j9object 0x12345678!j9method 0xABCDEF00

• Other printing commands!stackslots 0xABCDEF0

• Utility functions!classforname!findpattern!whatis

• Data analysisgccheckdeadlock

Bjørn Vårdal

What can DDR do?>


• Print structure at address!j9object 0x12345678!j9method 0xABCDEF00

• Other printing commands!stackslots 0xABCDEF0

• Utility functions!classforname!findpattern!whatis

• Data analysisgccheckdeadlock

Bjørn Vårdal

What can DDR do?> gccheck

Starting GC CheckChecking HEAP...done (919 ms).Checking CLASS HEAP...done (296 ms).Checking REMEMBERED SET...done (54 ms).Checking UNFINALIZED...done (3 ms).Checking FINALIZABLE...done (3 ms).Checking OWNABLE_SYNCHRONIZER...done (1 ms).Checking STRING TABLE...done (516 ms).Checking CLASS LOADERS...done (4 ms).Checking JNI GLOBAL REFS...done (1 ms).Checking JNI WEAK GLOBAL REFS...done (1 ms).Checking JVMTI OBJECT TAG TABLES...done (3 ms).Checking VM CLASS SLOTS...done (1 ms).Checking MONITOR TABLE...done (7 ms).Checking VM THREAD SLOTS...done (208 ms).Checking THREAD STACKS...done (25 ms).Done (2058ms)

• Structure layout information in the process memory

• DDR data generator generates blob and code

• Detectable RAS structure in memory at runtime

Bjørn Vårdal IBM Runtime Technologies 32

How does DDR work?

Runtime

DDR InteractiveDDR data generator

DDR blob

Core dump

DDR blob

Generated code

RAS RAS


DDR ComponentsOpenJ9Eclipse OMR

J9 JVM

DDR data generator

DDR Interactive

RAS


RAS StructureRuntime memory

VM structure

ThreadsPort libraryetc.



RAS Structure

Eye catcher: “J9VMRAS”Bit pattern: 0xAA55AA55

...DDR Structure blob…VM…

VM structure




DDR blob

Structure layoutsField offsetsBuild flags

RAS Structure



VM structure



DDR Structure BlobRuntime memory

DDR blob

Structure layoutsField offsetsBuild flags

RAS Structure



VM structure



OMR ddrgenCompile

with debug symbols

.dbg / .pdb

macroFile

ddrgen

blob.dat

superset.out

Load into memory at

runtime

Scan source files for

build flags

./getMacros.sh . libdwarf


DDR Class Generation

*.javasuperset.out

DDR Interactive(j9ddr.jar)

Generate Java structure classes

superset.out Generate Java pointer classes

*.java


DDR Interactive startup

Generated Java classes

DDR Interactive

DDR blob from core file

DDR ClassLoaderTemplate Field offsets

Load structure classes


• DDR capabilities – Analyzing the JVM in a core file

• RAS structure – A detectable starting point

• OMR ddrgen – Creating the DDR blob and superset

• DDR Interactive – Using the blob and superset

Bjørn Vårdal

Summary


DDR in a new Runtime


• Small tutorial runtime with ~10 opcodes

• For illustrating basic OMR JITBuilder functionality

• For demonstrating basic DDR functionality?• My fork: https://github.com/bjornvar/Base9/tree/ddr

Bjørn Vårdal

Base9

https://github.com/bjornvar/Base9/tree/ddr


• Quick prototype

• ~1 hour of hacking

• Mostly copied from J9 JVM

Bjørn Vårdal

DDR in Base9


RAS Structure in J9JVM runtime memory

DDR blob

Structure layoutsBuild flags

RAS Structure

Eye catcher: “J9VMRAS”Bit pattern: 0xAA55AA55...DDR Structure blob…JavaVM…

JavaVM structure



RAS Structure in Base9Base9 runtime memory

DDR blob


Generated by running ddrgen on Base9

RAS Structure

Eye catcher: “J9VMRAS”Bit pattern: 0xAA55AA55...DDR Structure blob…ExecutionContext

Copied the structure from the J9 source

ExecutionContext

StackString tableetc.


Base9 Code Changes


RAS Structure in Base9Base9 runtime memory

DDR blob


RAS Structure

Eye catcher: “J9VMRAS”Bit pattern: 0xAA55AA55...DDR Structure blob…ExecutionContext

ExecutionContext

StackString tableetc.


DDR InteractiveSome changes were required:

• Disabled all J9 specific functionality (!stackslots, etc.)

• Told it to look for B9RAS instead of J9RAS

• Added build flags constants

IBM Runtimes 52

$

Bjørn Vårdal

DDR Interactive


What’s next?•Make DDR Interactive plug-and-play• Always support structure commands on DDR enabled core files

• Easy DDR generation process

• Release DDR Interactive and DDR generators• OMR• OpenJ9


Conclusion• DDR in J9• Capabilities• How DDR is implemented

• OMR ddrgen• Turning debug symbols into blob and superset

• DDR in Base9• Adding DDR capabilities to a new runtime• Loading a Base9 core into DDR Interactive


Q & A

secrets of building a debuggable runtime: learn how language implementors solve your runtime issues

Technology