Systems ThinkingPatterns for scaling unstable systems

@_siddharth_ram

Siddharth Ram Chief Architect, Small Business Group

Intuit

Intuit Confidential and Proprietary 2

Intuit’s Mission To improve our customers’ financial lives so profoundly …

they can’t imagine going back to the old way

Consumers Small Businesses Accounting Professionals

Intuit Confidential and Proprietary 6

Recognized as One of the World’s Leading Companies

FORTUNE 100 BEST COMPANIES TO WORK FOR

2013%

MOST ADMIRED: SOFTWARE INDUSTRY MOST INNOVATIVE COMPANIES

2014%2015

31 Ranked #

12 Years in a Row

2004 2005 2006 2007 2008 2009 2010 2011 2012 2015

WORLD’S MOST ADMIRED COMPANIES

2013 2014

14 Years in a Row

2 Ranked #

• #1 Business management solution for small businesses and accountants worldwide

• More than 5M paid subscribers

• Available via Web, native tablets/mobile apps

• Available in 140 countries

• Translated into 12 languages

About QuickBooks

1. Optimism bias

Actual Perform

anceEstimated PerformanceO

vere

stim

atio

n

Und

eres

timat

ion

Test

Sco

re

Quartiles

Unskilled and Unaware of It:

How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments

The Dunning-Kruger Effect

2. Wirth’s lawThe rate of slowdown of software

is faster than the rate of speedup of hardware

3. Hofstadter’s lawIt always takes longer than you expect, even when you take into account Hofstadter’s law

4. ComplexityComplexity should be useful

-Larry Wall

5. The End to End Principle

http://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf

Reliability from unreliable parts

Stability vs Scale

Scal

e

Stability

Com

plex

ity

Pattern 0

Attitude towards failures

Everything always works

I do my job. Others need to do theirs

Everything is broken

My service works even when my dependent services are broken

Blind Faith

Trust but verify

Infrastructure is broken

Software is broken

My service works when there are infrastructure

problems

My Service works even when my hardware &

software dependencies are broken

God’s country

Dunning-KrugerOptimism bias

Circuit Breakers Destructive Testing

Fallacies of Distributed computing

The network is reliable

• Transport cost is zero

• It is easy to coordinate nodes in a network

• The network is homogenous

• There is one administrator

• Bandwidth is infinite

• Topology does not change

• The network is secure

• Latency is zero

Hofstradter’s law

Pattern 1

Scale out , Scale up

Scale out AND scale up

Wirth’s law

Pattern 1A:

Understand How to Scale

Horizontal duplication

Serv

ices

Shar

ding

(0,0,0) Monolith

(1,0,0) Read replicas

(0,0,1) Sharding

(0,1,0) Services

(1,1,1) Infinite Scale

C Axis Organizational

Scale

How to

Scale

Pattern 2

Everyone is Full Stack

The rule of least power

Atwoods Law

The less powerful the language, the more you can do with the data stored

in that language

Occam’s Razor Simpler is better

Any application that can be written in JavaScript, will eventually be written in

JavaScript.

4 Tier Applications - add NodeJS

Clients

NodeJS

ClientsClients

Java

Data

Atwoods Law

Useful C

omplexity

Full S

tack

Full S

tack

DevOps - own it end to endYou built it, you run it - https://queue.acm.org/detail.cfm?id=1142065

Giving developers operational responsibilities has greatly enhanced the quality of the services, both from a customer and a technology point of view. The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. This customer feedback loop is essential for improving the quality of the service

– Werner Vogels

Pattern 3:

Performance is a feature

Pattern 3a:

Understand what/how to measure

• Time to first byte

• Start render

• onLoad()

Perform

ance

What do you measure?

• Average

• Median

• TP

• RUM

Perform

ance

How do you measure?

Pattern 3b:

Design your edge architecture

Service 1

CDN/ Edge Server

Firewall Gateway

Facade

Service 2

Service 2 Cluster Router

Perform

ance

Pattern 4

Search & Destroy Statefulness

State kills. Kill State

Session and state destroy the ultimate value promised by multi-tenancy within Internet (SaaS, commerce, and so on)

applications.

- 50 Principles for Scaling Web Sites

Don’t simplify by creating unnecessary complexity

Kill Stat

e

Pattern 5

Loose Couple to lose coupling

Pattern 5a Service Decoupling

unnecessary complexity

Tight

Coupling

FacadeFacadeCache

{ X-identity:

X-i18n: X-role:

X-permission:

}HTTP Headers

{ X-identity:

X-i18n: X-role:

X-permission: X-TX-Count:

}

12

3

Roles Permissions

Identity Loos

e Cou

pling

Pattern 5a Bus & Event Sourcing

Loos

e Cou

pling

Async

The “Bus”

Highly Reliable…?

Loos

e Cou

pling

The “Bus”

Unreliable, high

throughput

recovery logic

End to End Principle

Loos

e Cou

pling

Unreliable, high

throughput

recovery logic

Event SourceMessage

Bus

ConsumersConsumersConsumers

Loos

e Cou

pling

Embrace failures

Embrace useful complexity

Understand Scaling patterns

Know how to measure performance

Be full Stack

Destroy State

Use async and event sourcing

Plan your edge architecture

culture

Technology