desert code camp 2016.1 - stateful distributed systems

Statefulness in Distributed

Systems

Amazon is hiring!

To learn more about our Dev Centers: http://bit.ly/phxdevcenters To learn more about current opportunities, email: phx@amazon.com

Our mission is to provide the most innovative, scalable, and reliable systems in the

world.

@JoeRawlings

https://goo.gl/O9PAUK

Outline

OutlineWhat’s that smell?

OutlineWhat’s that smell?

What’s the meaning of this?

OutlineWhat’s that smell?

What’s the meaning of this?

Pics or it didn’t happen.

OutlineWhat’s that smell?

What’s the meaning of this?

Pics or it didn’t happen.

Swallowing the red pill.

OutlineWhat’s that smell?

What’s the meaning of this?

Pics or it didn’t happen.

Swallowing the red pill.

Challenge accepted!

What’s that smell?

Stateful Services

DESIGN SMELL

Stateful Services

ANTI-PATTERNDESIGN SMELL

Stateful Services

Stateful Services

Stateful Services

WEB SERVER

TIER

CLIENT TIER

DATABASE TIER

WEB SERVER

TIER

CLIENT TIER

DATABASE TIER

WEB SERVER

TIER

CLIENT TIER

DATABASE TIER

WEB SERVER

TIER

CLIENT TIER

DATABASE TIER

What’s the meaning of this?

Stateful Services

Stateful ServicesYou’re already doing it!

Required to do something useful

Stateful ServicesYou’re already doing it!

Required to do something useful

Stateful Services

(most of the time)

You’re already doing it!

Required to do something useful

Stateful Services

(most of the time)

You’re already doing it!

stateful processing systems share similar concerns

Required to do something useful

Stateful Services

(most of the time)

You’re already doing it!

stateful processing systems share similar concerns

(This is where the fun is)

Stateful Processing

Stateful ProcessingData locality

data intensive systems

Stateful ProcessingData locality

data intensive systems

Stateful Processing

Strong Consistency

Data locality

data intensive systems

Stateful Processing

Strong Consistency

Data locality

High performance

Stateless Services

Stateless Services

does not care what has happened

Stateless Services

does not care what has happened

does not care what has changed

Stateful Processing

Needs to care what has happened

Stateful Processing

Needs to care what has happenedNeeds To care what has changed

Stateful Processing

Needs to care what has happenedNeeds To care what has changed

Divorce Process lifecycle from data lifecycle

Stateful Processing

CLIENT TIER

WEB SERVER

TIER

DATABASE TIER

Stateful Processing

service

CLIENT TIER

Stateful Processing

service

CLIENT TIER

Storage TIER

Stateful Processing

service

CLIENT TIER

Storage TIER

Stateful Processing

service

CLIENT TIER

Storage TIER

Stateful Processing

service

CLIENT TIER

Storage TIER

Stateful Processing

service

CLIENT TIER

Storage TIER

Stateful Processing

service

CLIENT TIER

Storage TIER

Stateful Processing

service

CLIENT TIER

Storage TIER

Stateful Processing

service

CLIENT TIER

Storage TIER

Stateful Processing

service

CLIENT TIER

Storage TIER

Stateful Processing

service

CLIENT TIER

Storage TIER

Pics Or It Didn’t Happen

https://github.com/graphite-project/whisper https://github.com/graphite-project/carbon

https://www.elastic.co/products/elasticsearch http://lucene.apache.org/

Swallowing The Red Pill

Stateful Processing

Gossip protocols

Stateful Processing

Stateful Processing

service

CLIENT TIER

Storage TIER

Stateful Processing

service

CLIENT TIER

Storage TIER

Gossip Protocols

Gossip ProtocolsUsage

Gossip ProtocolsUsagedissemination - event / background

Gossip ProtocolsUsagedissemination - event / backgroundanti-entropy - data repair

Gossip ProtocolsUsagedissemination - event / backgroundanti-entropy - data repairaggregation - compute across the network

Gossip ProtocolsUsagedissemination - event / backgroundanti-entropy - data repairaggregation - compute across the network

SWIM

Gossip ProtocolsUsagedissemination - event / backgroundanti-entropy - data repairaggregation - compute across the network

SWIMscalable weakly-consistent infection-style process group membership protocol - membership list (non-faulty processes)

SWIM Protocol

SWIM Protocolfailure detection

SWIM Protocolfailure detection1. ping random process from member list

SWIM Protocolfailure detection1. ping random process from member list

A. Receives Ack, Great! end

SWIM Protocolfailure detection1. ping random process from member list

A. Receives Ack, Great! endB. does not receive ack, goto 2

SWIM Protocolfailure detection1. ping random process from member list

A. Receives Ack, Great! endB. does not receive ack, goto 2

2. send ping request to k processes from member list

SWIM Protocolfailure detection1. ping random process from member list

A. Receives Ack, Great! endB. does not receive ack, goto 2

2. send ping request to k processes from member list3. k processes try to ping “failed” process

SWIM Protocolfailure detection1. ping random process from member list

A. Receives Ack, Great! endB. does not receive ack, goto 2

2. send ping request to k processes from member list3. k processes try to ping “failed” process

A. K processes respond back to original process about status

SWIM Protocolfailure detection1. ping random process from member list

A. Receives Ack, Great! endB. does not receive ack, goto 2

2. send ping request to k processes from member list3. k processes try to ping “failed” process

A. K processes respond back to original process about status

dissemination

SWIM Protocolfailure detection1. ping random process from member list

A. Receives Ack, Great! endB. does not receive ack, goto 2

2. send ping request to k processes from member list3. k processes try to ping “failed” process

A. K processes respond back to original process about status

disseminationmulticast failure (vol. leave) updates

SWIM Protocolfailure detection1. ping random process from member list

A. Receives Ack, Great! endB. does not receive ack, goto 2

2. send ping request to k processes from member list3. k processes try to ping “failed” process

A. K processes respond back to original process about status

disseminationmulticast failure (vol. leave) updatesmulticast new members

Gossip protocols

Stateful Processing

consensusGossip protocols

Stateful Processing

Stateful Processing

service

CLIENT TIER

Storage TIER

Stateful Processing

service

CLIENT TIER

Storage TIER

http://zookeeper.apache.org/

Paxos

PaxosProposers, Acceptors, Learners

PaxosProposers, Acceptors, Learners

Proposers:

PaxosProposers, Acceptors, Learners

Proposers: Ask Acceptors to Approve Proposals

PaxosProposers, Acceptors, Learners

Proposers: Ask Acceptors to Approve ProposalsAsk Acceptors to accept a proposal & Version

PaxosProposers, Acceptors, Learners

Proposers:

Acceptors:

Ask Acceptors to Approve ProposalsAsk Acceptors to accept a proposal & Version

PaxosProposers, Acceptors, Learners

Proposers:

Acceptors:

Ask Acceptors to Approve ProposalsAsk Acceptors to accept a proposal & Version

Do not have to approve or accept

PaxosProposers, Acceptors, Learners

Proposers:

Acceptors:

Ask Acceptors to Approve ProposalsAsk Acceptors to accept a proposal & Version

Do not have to approve or acceptOnly Approve/Accept what has been proposed

PaxosProposers, Acceptors, Learners

Proposers:

Acceptors:

Ask Acceptors to Approve ProposalsAsk Acceptors to accept a proposal & Version

Do not have to approve or acceptOnly Approve/Accept what has been proposed

LEARNERS:

PaxosProposers, Acceptors, Learners

Proposers:

Acceptors:

Ask Acceptors to Approve ProposalsAsk Acceptors to accept a proposal & Version

Do not have to approve or acceptOnly Approve/Accept what has been proposed

LEARNERS: Proposed Value is chosen when a majority of Acceptors have accepted the value

PaxosProposers, Acceptors, Learners

PaxosProposers, Acceptors, Learners

Phase 1:

PaxosProposers, Acceptors, Learners

Phase 1:1) Proposers propose a value

PaxosProposers, Acceptors, Learners

Phase 1:1) Proposers propose a value2) Acceptors approve value

PaxosProposers, Acceptors, Learners

Phase 1:1) Proposers propose a value2) Acceptors approve value

1) not approve or accept smaller values

PaxosProposers, Acceptors, Learners

Phase 1:1) Proposers propose a value2) Acceptors approve value

1) not approve or accept smaller values2) send back highest (accepted) version

PaxosProposers, Acceptors, Learners

Phase 1:1) Proposers propose a value2) Acceptors approve value

1) not approve or accept smaller values2) send back highest (accepted) version

Phase 2:

PaxosProposers, Acceptors, Learners

Phase 1:1) Proposers propose a value2) Acceptors approve value

1) not approve or accept smaller values2) send back highest (accepted) version

Phase 2:1) proposer Receives approval from majority

PaxosProposers, Acceptors, Learners

Phase 1:1) Proposers propose a value2) Acceptors approve value

1) not approve or accept smaller values2) send back highest (accepted) version

Phase 2:1) proposer Receives approval from majority2) sends accept! to acceptors with highest

version

PaxosProposers, Acceptors, Learners

Phase 1:1) Proposers propose a value2) Acceptors approve value

1) not approve or accept smaller values2) send back highest (accepted) version

Phase 2:1) proposer Receives approval from majority2) sends accept! to acceptors with highest

version

Phase 3:

PaxosProposers, Acceptors, Learners

Phase 1:1) Proposers propose a value2) Acceptors approve value

1) not approve or accept smaller values2) send back highest (accepted) version

Phase 2:1) proposer Receives approval from majority2) sends accept! to acceptors with highest

version

Phase 3: 1) acceptors notify all learners

Gossip protocolsconsensus

Stateful Processing

consistent hashing

Gossip protocolsconsensus

Stateful Processing

Stateful Processing

service

CLIENT TIER

Storage TIER

Stateful Processing

service

CLIENT TIER

Storage TIER

Stateful Processing

service

CLIENT TIER

Storage TIER

Consistent Hashing

Consistent HashingDistributed Hash Tables

Consistent HashingDistributed Hash TablesConvenient, Fast O(1)

Consistent HashingDistributed Hash TablesConvenient, Fast O(1)Great for lookups given a key

Consistent HashingDistributed Hash TablesConvenient, Fast O(1)Great for lookups given a key

Issue

Consistent HashingDistributed Hash TablesConvenient, Fast O(1)Great for lookups given a key

IssueHorizontal scaling

Consistent HashingDistributed Hash TablesConvenient, Fast O(1)Great for lookups given a key

IssueHorizontal scalingFault Tolerance

Consistent HashingDistributed Hash TablesConvenient, Fast O(1)Great for lookups given a key

IssueHorizontal scalingFault Tolerancehash(object) (mod n)

n = number of slots

Consistent HashingDistributed Hash TablesConvenient, Fast O(1)Great for lookups given a key

IssueHorizontal scalingFault Tolerancehash(object) (mod n)Remapping keys to more slots

n = number of slots

Consistent HashingDistributed Hash TablesConvenient, Fast O(1)Great for Key / Value lookups

Solution

K = keys, n = number of slots

Consistent HashingDistributed Hash TablesConvenient, Fast O(1)Great for Key / Value lookups

Solutionconsistent hashing

K = keys, n = number of slots

Consistent HashingDistributed Hash TablesConvenient, Fast O(1)Great for Key / Value lookups

Solutionconsistent hashingDuring resize, k / n keys are remapped

K = keys, n = number of slots

Consistent Hashing

Challenge Accepted!

Challenges

Stateful Processing

Challengesresult composition

Stateful Processing

Challengesresult compositionwork distribution

Stateful Processing

Challengesresult compositionwork distributioncode deployments

Stateful Processing

Challengesresult compositionwork distributioncode deploymentsunbounded data structures

Stateful Processing

Challengesresult compositionwork distributioncode deploymentsunbounded data structuresmemory management

Stateful Processing

Challengesresult compositionwork distributioncode deploymentsunbounded data structuresmemory managementpersistence strategies

Stateful Processing

Challengesresult compositionwork distributioncode deploymentsunbounded data structuresmemory managementpersistence strategiesconcurrency

Stateful Processing

Key Takeaways

Key Takeaways

Stateful and Stateless systems co-exist

Key Takeaways

Stateful and Stateless systems co-exist

process lifecycle vs data lifecyCle

Key Takeaways

Stateful and Stateless systems co-exist

process lifecycle vs data lifecyCle

Stateful services have their place

Key Takeaways

Stateful and Stateless systems co-exist

process lifecycle vs data lifecyCle

Stateful services have their place

Interesting opportunities

Key Takeaways

Stateful and Stateless systems co-exist

process lifecycle vs data lifecyCle

Stateful services have their place

Interesting opportunities

But not all is rosy

Dynamo: Amazon’s Highly Available Key-value Store http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf

Papers We Love

The Chubby lock service for loosely-coupled distributed systemsresearch.google.com/archive/chubby-osdi06.pdf

Paxos made simplehttp://research.microsoft.com/en-us/um/people/lamport/pubs/paxos-simple.pdf

Time, Clocks, and the Ordering of Events in a Distributed Systemhttp://research.microsoft.com/en-us/um/people/lamport/pubs/time-clocks.pdf

The Google File Systemhttp://research.google.com/archive/gfs-sosp2003.pdf

Resources

Caitie McCaffrey’s talks - https://goo.gl/8MSdRz

Apache Foundation - http://apache.org/

graphite - https://graphiteapp.org/

elastic - https://www.elastic.co/

Thanks! Q&A

Amazon is hiring!To learn more about our Dev Centers: http://bit.ly/phxdevcenters To learn more about current opportunities, email: phx@amazon.com