perfomance tuning on go 2.0

© 2010 ThoughtWorks, Inc. All rights reserved.

Performance Tuning on Go 2.0http://www.thoughtworks-studios.com/go/

Yogi Kulkarni

© 2010 ThoughtWorks, Inc. All rights reserved.

GoContinuous integration Release management

© 2010 ThoughtWorks, Inc. All rights reserved.

© 2010 ThoughtWorks, Inc. All rights reserved.

Stage-2Unit Test

Run Unit Test Suite - 1

Run Unit Test Suite - 2

Stage-3Acceptance Test

Accetance Test Suite - 1

Acceptance Test Suite – 2

Stage-1Compile

CompileJob

Pipeline

Run Unit Test Suite - 3

© 2010 ThoughtWorks, Inc. All rights reserved.

Pipeline Dashboard

© 2010 ThoughtWorks, Inc. All rights reserved.

© 2010 ThoughtWorks, Inc. All rights reserved.

Pipeline Dependencies

© 2010 ThoughtWorks, Inc. All rights reserved.

The Problem

© 2010 ThoughtWorks, Inc. All rights reserved.

Slow page loads

Problem Investigation Setup Solution

© 2010 ThoughtWorks, Inc. All rights reserved.

Jobs taking long to get assigned to agents

Problem Investigation Setup Solution

© 2010 ThoughtWorks, Inc. All rights reserved.

Jobs getting rescheduled

Problem Investigation Setup Solution

© 2010 ThoughtWorks, Inc. All rights reserved.

Artifact uploads timing out

Problem Investigation Setup Solution

© 2010 ThoughtWorks, Inc. All rights reserved.

Server VM freezing

Problem Investigation Setup Solution

© 2010 ThoughtWorks, Inc. All rights reserved.

Lots of concurrent activity

© 2010 ThoughtWorks, Inc. All rights reserved.

What was happening under the hood?Profiler revealed the culprits

Problem Investigation Setup Solution

© 2010 ThoughtWorks, Inc. All rights reserved.

Thread blocks

© 2010 ThoughtWorks, Inc. All rights reserved.

Threads blocked on the database connection pool

© 2010 ThoughtWorks, Inc. All rights reserved.

So we introduced asynchronicity

Problem Investigation Setup Solution

© 2010 ThoughtWorks, Inc. All rights reserved.

Big mistake!

Problem Investigation Setup Solution

© 2010 ThoughtWorks, Inc. All rights reserved.

Find the root cause

Problem Investigation Setup Solution

© 2010 ThoughtWorks, Inc. All rights reserved.

© 2010 ThoughtWorks, Inc. All rights reserved.

Problems

Too many queries

Slow queries

Coarse-grained synchronized blocks

JRuby and Rails issues

© 2010 ThoughtWorks, Inc. All rights reserved.

Problems Solutions

Too many queries Caching

Slow queries Query tuning

Coarse-grained synchronized Reduce lock granularity

JRuby and Rails issues Hack and pray!

© 2010 ThoughtWorks, Inc. All rights reserved.

ObjectiveNo request should take > 100 ms

Most should be under 50 ms

Problem Investigation Setup Solution

© 2010 ThoughtWorks, Inc. All rights reserved.

Setup & Tools

Problem Investigation Setup Solution

© 2010 ThoughtWorks, Inc. All rights reserved.

Conservative hardware

Problem Investigation Setup Solution

© 2010 ThoughtWorks, Inc. All rights reserved.

“Performance Server”Dell 620 laptop

2 Cores, 2GB RAM

© 2010 ThoughtWorks, Inc. All rights reserved.

Large Dataset5GB merged database

Large config file with 100 pipelinesReal-world data

Problem Investigation Setup Solution

© 2010 ThoughtWorks, Inc. All rights reserved.

Extreme load70 agents

Each Job generating lots of console outputEach Job uploading 10 MB artifacts

Apache Bench / Httperf

Problem Investigation Setup Solution

© 2010 ThoughtWorks, Inc. All rights reserved.

Automation for fast feedbackCapistrano script

Feedback cycle down from hours to 10 min

Problem Investigation Setup Solution

© 2010 ThoughtWorks, Inc. All rights reserved.

Caching

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

Single application-wide cacheDomain objects

DTOsRails view fragments

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

© 2010 ThoughtWorks, Inc. All rights reserved.

EHCacheIn-memory only

Least-Recently-Used eviction policy

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

Transaction rollbacks == Incorrect cached dataIgnore cache puts in transactions

Invalidate caches on transaction commit only

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

Spring to the rescue

© 2010 ThoughtWorks, Inc. All rights reserved.

Fast deep-cloning of cached objectsJava Deep Cloning Library: http://robust-it.co.uk/clone/index.php

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

Locks

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

Be suspicious of synchronized methods

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

Lock on interned stringsFine granularityProfiler support

Also used as cache keys

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

© 2010 ThoughtWorks, Inc. All rights reserved.

Use ReadWriteLocks for highly contended code

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

Use “volatile” fields for lockless reads

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

Slow

© 2010 ThoughtWorks, Inc. All rights reserved.

Not synchronized

© 2010 ThoughtWorks, Inc. All rights reserved.

Acquire locks before starting transactionsTo enforce aggregate invariants

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

Stage(Building)

Job – 1(Completed)

Job – 2(Building)

Stage(Building)

Job – 1(Building)

Job – 2(Completed)

Thread 1

Thread 2

Stage(Building)

Job – 1(Building)

Job – 2(Building)

Stage(Building)

Job – 1(Completed)

Job – 2(Cmpleted)

© 2010 ThoughtWorks, Inc. All rights reserved.

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

Query Tuning

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

Use a large databaseUse the query analyzer to find missing indexes

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

Views are great… sometimes10x-20x faster for “TOP n” queries

Depends on database implementation

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

Query selectivity is important200 ms

Stages(70,000 rows)

Pipelines(20,000 rows)

BuildStateTransitions(4,000,000 rows)

© 2010 ThoughtWorks, Inc. All rights reserved.

2 ms

Introduce an n+1 query

© 2010 ThoughtWorks, Inc. All rights reserved.

JRuby on Rails Hacks

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

Single JRuby runtime + Rails multi-threaded mode

Much lower memory footprintBut completely uncharted territory

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

Severe thread blocks in JRubyReported to JRuby community

Fixed immediately in 1.5.0 & 1.5.1

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

Rails nested partials Mysterious lock contention

10 concurrent requests take 90 secs to completeStill a mystery!

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

Desperate situations, desperate measuresStatically inline partials at build time!

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

Routes subsystem is slowCache url_for

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

LoggingReplace Rails BufferedLogger with Java Log4jLogger

Turn off Rails logging in production

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

Fragment cachingGenerate a cache-key that incorprates all fields that are needed to

uniquely identify that object’s current stateLRU cache will evict it on state change

Problem Investigation Setup Solution

Caching Locks Query Tuning Jruby & Rails

© 2010 ThoughtWorks, Inc. All rights reserved.

“pipelineDashboardFragment|dev|2[compile,1,passed][test,1,building]”

Key-identifier Pipeline-name Stage detail Stage detail

© 2010 ThoughtWorks, Inc. All rights reserved.

“pipelineDashboardFragment|dev|1[compile,1,passed][test,1,building]”

“pipelineDashboardFragment|dev|2[compile,1,passed][test,1,passed]”

Key-identifier Pipeline-name Stage detail Stage detail

© 2010 ThoughtWorks, Inc. All rights reserved.

Lessons LearnedFind the root cause

Use a profilerMeasure before and after each fix

Problem Investigation Setup Solution

© 2010 ThoughtWorks, Inc. All rights reserved.

How we got from that to thisINSERT IMAGE

Threads after performance tuning

© 2010 ThoughtWorks, Inc. All rights reserved.

http://www.thoughtworks-studios.com/go/