Scaling Ruby for Scaling Ruby for Enterprise ApplicationsEnterprise ApplicationsMinneBar 2013MinneBar 2013

by Joel by Joel Jensen and Jensen and John John MaloneMalone

ProfilingProfiling

DatabasesDatabases

QueuesQueues

ToolsTools

TestingTesting

Legacy CodeLegacy Code

MentalityMentality

State on the WireState on the Wire

Server ScalingServer Scaling

Lessons LearnedLessons Learned

Throughput IncreaseThroughput Increase

Vertical ScalingVertical Scaling

How we got there...How we got there...

Best Buy Epic AppBest Buy Epic App

TopologyTopology

ProfilingProfilingNew Relic is our weapon of choice.New Relic is our weapon of choice.

Stuff we usedStuff we used

Thread profilingThread profiling

Monitoring DaemonsMonitoring Daemons

Custom Custom instrumentationinstrumentation

Developer ModeDeveloper Mode

Deployment eventsDeployment events

Slow query logSlow query log

Developer ModeDeveloper Mode

Developer ModeDeveloper ModeQuery profilingQuery profiling

MongoDBMongoDB

+Drivers of our language of choice+Drivers of our language of choice

+Speed+Speed

+Flexibility+Flexibility

+Partition Tolerance+Partition Tolerance

+Availability+Availability

-Consistency-Consistency

DatabasesDatabases

MysqlMysql

+Drivers for our language+Drivers for our language

-Flexibility-Flexibility

+Consistency+Consistency

DatabasesDatabases

CAP theoremCAP theorem

QueuesQueues

0MQ0MQ+Speed (fastest you +Speed (fastest you can get)can get)

+Very simple +Very simple design, great for design, great for complex systemscomplex systems

+Drivers in all major +Drivers in all major languageslanguages

-Hard to use-Hard to use

-Poor drivers-Poor drivers

-No UI-No UI

RabbitMQRabbitMQ

+Speed+Speed

+Great web based +Great web based UIUI

Drivers in all major Drivers in all major languageslanguages

ToolsTools

ChefChef

Automated, Automated, scripted server scripted server provisioningprovisioning

Crucial for Crucial for modern systemsmodern systems

TestingTesting

Integration test is the MOST important for usIntegration test is the MOST important for us

Automate huge batchesAutomate huge batches

Automate your metricsAutomate your metrics

Test with real dataTest with real data

Rspec and CucumberRspec and Cucumber

Legacy Legacy CodeCode

MentalityMentality

Hack the PlanetHack the Planet

Be a hacker (ala Be a hacker (ala hackers and painters)hackers and painters)

You are allowed to failYou are allowed to fail

DRYDRY

Abstraction Abstraction everywhereeverywhere

TestsTests

No burnoutsNo burnouts

Component ArchitectureComponent ArchitectureBefore State on the WireBefore State on the Wire

State on the WireState on the Wire

Beat the ClockBeat the Clock

Component => Input data, work on it, save it, Component => Input data, work on it, save it, enqueueenqueue

Atomic Units of WorkAtomic Units of Work

Design of latency, serialize the messageDesign of latency, serialize the message

Minimize database writesMinimize database writes

Daemons gemDaemons gem

Component ArchitectureComponent ArchitectureAfter State on the WireAfter State on the Wire

State on the Wire State on the Wire ImprovementsImprovements

60% increase in throughput60% increase in throughput

90% decrease in database I/O90% decrease in database I/O

85% decrease in database size85% decrease in database size

Vertical ScalingVertical Scalingec2instances.infoec2instances.info

Throwing hardware at Throwing hardware at the problemthe problem

Needed solid profilingNeeded solid profiling

Realized we were running our CPU hotRealized we were running our CPU hot

Moved the Epic subsystem onto a High CPU Moved the Epic subsystem onto a High CPU instanceinstance

Moved Bouncer subsystem onto a High Moved Bouncer subsystem onto a High Memory instanceMemory instance

Vertical Scaling Vertical Scaling ContinuedContinued

8 cores were now put into play8 cores were now put into play

200% increase in throughput200% increase in throughput

Daemons began autoscalingDaemons began autoscaling

Leave a core for your OSLeave a core for your OS

Horizontal ScalingHorizontal Scaling

Componentized architecture allows us to throw Componentized architecture allows us to throw more machines at the problemmore machines at the problem

Future state - autoscaling based upon queue Future state - autoscaling based upon queue lengthlength

Chef stands machines up and down based on Chef stands machines up and down based on CloudWatch queue length monitoringCloudWatch queue length monitoring

Lessons LearnedLessons Learned

You can’t improve what You can’t improve what you can’t measureyou can’t measure

Install good monitoring Install good monitoring and alertingand alerting

Don’t ask for Don’t ask for permissionpermission

Say no to premature Say no to premature optimizationoptimization

1 million operations a 1 million operations a day means 12 ops per day means 12 ops per secondsecond

Thank YouThank YouAaron GaalswykAaron Gaalswyk

Anjali RamanAnjali Raman

Arun BatchuArun Batchu

Arvind NaikArvind Naik

Carl MuellerCarl Mueller

Chad SearsChad Sears

Geoff LoryGeoff Lory

Jayaraj SomasekharanJayaraj Somasekharan

John MurphyJohn Murphy

Josh FribergJosh Friberg

Kevin MathenyKevin Matheny

Laurie HeavierLaurie Heavier

Logan PoelhmanLogan Poelhman

Lynette LinamanLynette Linaman

Mike HelferMike Helfer

Mohit PadalkarMohit Padalkar

Niyaz AhamedNiyaz Ahamed

Shelley BerubeShelley Berube

Trent OgrenTrent Ogren

Yury GavrikYury Gavrik