are your v8 garbage collection logs speaking to you?joyee cheung -alibaba cloud(alibaba group)

Are Your V8 Garbage Collection Logs Speaking to You? Joyee Cheung, Alibaba Cloud

What Prompted us to Look into themNode.js at Alibaba

What Prompted Us to Look Into ThemWhy GC Logs?

v Java is the mainstream at Alibabav Chose Node.js for productivity, but it’s meaningless without availabilityv Usually not the bottleneck, but with a huge traffic, every optimization counts

v Garbage collection is important in long-running, data-intensive serversv High CPU usage: hurts responsivenessv High memory usage: affects other processesv Memory leaks: misled by developers, crashes the process(OOM)v Java developers know their VM and its GCv …

v More about Node.js at Alibabav https://www.linux.com/blog/can-nodejs-scale-ask-team-alibaba

What Prompted Us to Look Into ThemWhy GC Logs?

v Tools from V8/Chromium are more about optimizing frontend applications

CPU profiles aremore about function calls,less about objects

v Heap snapshots and timelines are more about allocations, less about collections

v Lack of high-level overview of the heap, too much details

Hard to use in staging/production environment

What Prompted Us to Look Into ThemWhy GC Logs?

v V8 provides garbage collection logs v Not documented, not available in the devtools, need to use command line argsv Higher-level than heap profiles, lower-level than the number of memory usuagev Help verify the fixes by displaying patterns of GCv Not silver bullet, but nice to have

v This talk is based on the versions of V8 used by Node v4 & v6 LTSv LTS plan stablizes the V8 versions for each LTS and its GC logs

An Introduction to V8 GCWhat is Garbage Collection

An Introduction to V8 GCHow Most GCs are triggered

An Introduction to V8 GCStop-The-World, Parallel, Concurrent GC

An Introduction to V8 GCGenerational GC

An Introduction to V8 GCHeap Organization

An Introduction to V8 GCHeap Limits(Default)

v deps/v8/src/heap/heap.h: Heap::Heap()

16MB1MB1.4GB0.7GB

512MB

v Can be configured via command line argumentsv --max_old_space_size, --max_semi_space_size, --max_executable_space_size

An Introduction to V8 GCNew Generation – Scavenge GC

An Introduction to V8 GCOld Generation – Mark-Sweep/Compact GC

An Introduction to V8 GCGC Pauses in V8

How to Get The GC Logs

v Plenty of options (node --v8-options| grep gc)v We’ll be focusing on

v --trace_gcv Print a line briefly describing when, how, what and why for each GC

v --trace_gc_verbosev Combined with --trace_gc, summarizes about each space and the

heap after each GCv --trace_gc_nvp

v Print name-value pairs describing how each step of GC goesv In case you do want to read the code

v deps/v8/src/heap, start with gc-tracer.cc

How to Read The GC Logs

v --trace_gcWHERE: [PID:Isolate]

WHEN: Time since the process has started

WHAT: Type of GC

HOW MUCH:Size of all objects(Size of memory allocated from OS)

HOW LONG:GC pause/Time spent on external memory

WHY

How to Read The GC Logs

v --trace_gc_verbose && --trace_gc

Memory V8 allocated from the OS

All memory reserved for this space (insidededicated pages)

Maintained by external C++ code

Adjust heap growing factor and the allocation limit that triggers the next incremental marking

How to Read The GC Logs

v --trace_gc_nvp ms=Mark-Sweep, s=Scavenge

What We’ve Found

v Tons of Scavenge GC in New Space eating up the CPUv Cache misses caused by inappropriate caching strategiesv Excessive deep clones/object merging

v Large Object Space keeps growingv Serialization/deserialization of huge RPC payload

v Code Space keeps growingv Bugs in templating engines

v Old Space keeps growingv Closures in long connections

Tools

v Our parserv Up on NPMv For people who already have

applications running in production

https://github.com/aliyun-node/v8-gc-log-parser

http://alinode.aliyun.com/blog(translation coming soon)

Visualization coming soon!

Tools

v --trace_gc_object_statsv Feed the output to https://mlippautz.github.io/v8-heap-stats/

v Requires Node >= v7v More about the memory usuage, less about GC

v --track_gc_object_statsv Combined with --log_timer_events, writes a log file to diskv Can be viewed with chrome://tracing (Incomplete???)v Pending PR: https://github.com/nodejs/node/pull/9304

THANKSTwitter/Github: @joyeecheung

joyeec9h3@gmail.comhttps://alinode.aliyun.com

An Introduction to V8 GCAccurate GC