desert code camp 2016.1 - stateful distributed systems

Statefulness in Distributed

Systems

Amazon is hiring!

To learn more about our Dev Centers: http://bit.ly/phxdevcenters To learn more about current opportunities, email: [email protected]

Our mission is to provide the most innovative, scalable, and reliable systems in the

world.

@JoeRawlings

https://goo.gl/O9PAUK

Outline

OutlineWhat’s that smell?


What’s the meaning of this?



Pics or it didn’t happen.




Swallowing the red pill.




Swallowing the red pill.

Challenge accepted!

What’s that smell?

Stateful Services

DESIGN SMELL

Stateful Services

ANTI-PATTERNDESIGN SMELL

Stateful Services

Stateful Services

WEB SERVER

TIER

CLIENT TIER

DATABASE TIER

Stateful Services

Stateful ServicesYou’re already doing it!

Required to do something useful

Stateful ServicesYou’re already doing it!


Stateful Services

(most of the time)

You’re already doing it!


Stateful Services

(most of the time)


stateful processing systems share similar concerns


Stateful Services

(most of the time)


stateful processing systems share similar concerns

(This is where the fun is)

Stateful Processing

Stateful ProcessingData locality

data intensive systems

Stateful ProcessingData locality


Stateful Processing

Strong Consistency

Data locality


Stateful Processing

Strong Consistency

Data locality

High performance

Stateless Services

Stateless Services

does not care what has happened

Stateless Services

does not care what has happened

does not care what has changed

Stateful Processing

Needs to care what has happened

Stateful Processing

Needs to care what has happenedNeeds To care what has changed

Stateful Processing

Needs to care what has happenedNeeds To care what has changed

Divorce Process lifecycle from data lifecycle

Stateful Processing

CLIENT TIER

WEB SERVER

TIER

DATABASE TIER

Stateful Processing

service

CLIENT TIER

Stateful Processing

service

CLIENT TIER

Storage TIER

Pics Or It Didn’t Happen

https://github.com/graphite-project/whisper https://github.com/graphite-project/carbon

https://www.elastic.co/products/elasticsearch http://lucene.apache.org/

Swallowing The Red Pill

Stateful Processing

Gossip protocols

Stateful Processing

Stateful Processing

service

CLIENT TIER

Storage TIER

Gossip Protocols

Gossip ProtocolsUsage

Gossip ProtocolsUsagedissemination - event / background

Gossip ProtocolsUsagedissemination - event / backgroundanti-entropy - data repair

Gossip ProtocolsUsagedissemination - event / backgroundanti-entropy - data repairaggregation - compute across the network


SWIM


SWIMscalable weakly-consistent infection-style process group membership protocol - membership list (non-faulty processes)

SWIM Protocol

SWIM Protocolfailure detection

SWIM Protocolfailure detection1. ping random process from member list


A. Receives Ack, Great! end


A. Receives Ack, Great! endB. does not receive ack, goto 2



2. send ping request to k processes from member list



2. send ping request to k processes from member list3. k processes try to ping “failed” process




A. K processes respond back to original process about status





dissemination





disseminationmulticast failure (vol. leave) updates





disseminationmulticast failure (vol. leave) updatesmulticast new members

Gossip protocols

Stateful Processing

consensusGossip protocols

Stateful Processing

Stateful Processing

service

CLIENT TIER

Storage TIER

http://zookeeper.apache.org/

PaxosProposers, Acceptors, Learners


Proposers:


Proposers: Ask Acceptors to Approve Proposals


Proposers: Ask Acceptors to Approve ProposalsAsk Acceptors to accept a proposal & Version


Proposers:

Acceptors:

Ask Acceptors to Approve ProposalsAsk Acceptors to accept a proposal & Version


Proposers:

Acceptors:


Do not have to approve or accept


Proposers:

Acceptors:


Do not have to approve or acceptOnly Approve/Accept what has been proposed


Proposers:

Acceptors:



LEARNERS:


Proposers:

Acceptors:



LEARNERS: Proposed Value is chosen when a majority of Acceptors have accepted the value


Phase 1:


Phase 1:1) Proposers propose a value


Phase 1:1) Proposers propose a value2) Acceptors approve value



1) not approve or accept smaller values



1) not approve or accept smaller values2) send back highest (accepted) version




Phase 2:




Phase 2:1) proposer Receives approval from majority




Phase 2:1) proposer Receives approval from majority2) sends accept! to acceptors with highest

version





version

Phase 3:





version

Phase 3: 1) acceptors notify all learners

Gossip protocolsconsensus

Stateful Processing

consistent hashing

Gossip protocolsconsensus

Stateful Processing

Stateful Processing

service

CLIENT TIER

Storage TIER

Consistent Hashing

Consistent HashingDistributed Hash Tables

Consistent HashingDistributed Hash TablesConvenient, Fast O(1)

Consistent HashingDistributed Hash TablesConvenient, Fast O(1)Great for lookups given a key


Issue


IssueHorizontal scaling


IssueHorizontal scalingFault Tolerance


IssueHorizontal scalingFault Tolerancehash(object) (mod n)

n = number of slots


IssueHorizontal scalingFault Tolerancehash(object) (mod n)Remapping keys to more slots

n = number of slots

Consistent HashingDistributed Hash TablesConvenient, Fast O(1)Great for Key / Value lookups

Solution

K = keys, n = number of slots


Solutionconsistent hashing



Solutionconsistent hashingDuring resize, k / n keys are remapped


Consistent Hashing

Challenge Accepted!

Challenges

Stateful Processing

Challengesresult composition

Stateful Processing

Challengesresult compositionwork distribution

Stateful Processing

Challengesresult compositionwork distributioncode deployments

Stateful Processing

Challengesresult compositionwork distributioncode deploymentsunbounded data structures

Stateful Processing

Challengesresult compositionwork distributioncode deploymentsunbounded data structuresmemory management

Stateful Processing

Challengesresult compositionwork distributioncode deploymentsunbounded data structuresmemory managementpersistence strategies

Stateful Processing

Challengesresult compositionwork distributioncode deploymentsunbounded data structuresmemory managementpersistence strategiesconcurrency

Stateful Processing

Key Takeaways

Key Takeaways

Stateful and Stateless systems co-exist

Key Takeaways


process lifecycle vs data lifecyCle

Key Takeaways



Stateful services have their place

Key Takeaways




Interesting opportunities

Key Takeaways




Interesting opportunities

But not all is rosy

Dynamo: Amazon’s Highly Available Key-value Store http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf

Papers We Love

The Chubby lock service for loosely-coupled distributed systemsresearch.google.com/archive/chubby-osdi06.pdf

Paxos made simplehttp://research.microsoft.com/en-us/um/people/lamport/pubs/paxos-simple.pdf

Time, Clocks, and the Ordering of Events in a Distributed Systemhttp://research.microsoft.com/en-us/um/people/lamport/pubs/time-clocks.pdf

The Google File Systemhttp://research.google.com/archive/gfs-sosp2003.pdf

http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf

http://research.google.com/archive/chubby-osdi06.pdf

http://research.microsoft.com/en-us/um/people/lamport/pubs/paxos-simple.pdf

http://research.microsoft.com/en-us/um/people/lamport/pubs/time-clocks.pdf

http://research.google.com/archive/gfs-sosp2003.pdf

Resources

Caitie McCaffrey’s talks - https://goo.gl/8MSdRz

Apache Foundation - http://apache.org/

graphite - https://graphiteapp.org/

elastic - https://www.elastic.co/

https://goo.gl/8MSdRz

http://apache.org/

https://graphiteapp.org/

https://www.elastic.co/

Thanks! Q&A

Amazon is hiring!To learn more about our Dev Centers: http://bit.ly/phxdevcenters To learn more about current opportunities, email: [email protected]

desert code camp 2016.1 - stateful distributed systems

Engineering