1  

Couchbase  Server  for  Speed  and  Scale  with  Interac8ve  Applica8ons  

Perry  Krug  Presenter  Title  

2  

Agenda  (just  an  outline)  

•  Needs  for  Interac8ve  Apps  –  Speed  

•  Speed  of  access  •  Speed  of  app  development  

–  Scale  •  Built  to  scale  

–  Unique  architectural  principals  •  Clustered  system  

–  Simple  for  applica8ons  –  Automa8c  sharding  of  data  across  cluster  

•  Single  index  view  –  No  need  to  define  sharding  keys  –  Automa8c,  online  transfer  of  data  across  cluster  

3  

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.

EXAMPLE  OF  AN  INTERACTIVE  APP  

4  

Instagrowth:  Android  Launch  

•  Instagram  gained  nearly  1  million  users  overnight  when  they  expanded  to  Android  

Example

5  

1 Instagram =

7.5M MAU*

Example

6  

Draw  Something  -­‐  Social  Game  

•  50  million  downloads  in  50  days  •  That’s  almost  5  Instagrams.  In  2  months.  

7  

Goes  Viral  3  Weeks  AZer  Launch  

19  17  15  13  11  9  7  5  3  3/1  28  26  24  22  20  18  16  14  12  10  8  2/6  

Draw  Something  by  OMGPOP  Daily  Ac8ve  Users  (millions)  

21  

2  

4  

6  

8  

10  

12  

14  

16  

8  

 Database  Opera8ons  

19  17  15  13  11  9  7  5  3  3/1  28  26  24  22  20  18  16  14  12  10  8  2/6  

Draw  Something  by  OMGPOP  Daily  Ac8ve  Users  (millions)  

21  

2  

4  

6  

8  

10  

12  

14  

16  

By  March  19,  there  were    over  30,000,000  downloads  of  the  app,  

over  5,000  drawings  being  stored  per  second,  over  2,200,000,000  drawings  stored,  

over  105,000  database  transac8ons  per  second,  and  over  3.3  terabytes  of  data  stored.  

9  

Social  Gaming  on  Couchbase  

iki  

10  

Speed  

•  Modern,  interac8ve  applica8ons  require  speed  – Web  pages  are  dynamic,  interac8ve  and  may  incrementally  fetch  content  for  display  

– Mobile  applica8ons  cannot  carry  all  of  the  content,  so  they  need  to  be  able  to  fetch  data  on  demand  

•  Couchbase  Speed  via…  – Managed,  built  in  cache  

•  memcached  compa8ble,  built  on  memcached  

–  Protocol  design  •  Client  can  directly  access  data  objects  within  the  cluster  •  No  quorum  reads  needed  to  determine  version,  always  consistent  

11  

Scale  

•  Modern  app  deployments  are  always  on,  available  24x7x365  –  Some  applica8ons  may  have  a  lifecycle  which  requires  scaling  down,  in  addi8onal  to  scaling  up  

•  Couchbase  scales  via…  –  Automa8cally  sharding  data  among  the  nodes  of  the  cluster  and  atomically  transferring  vbuckets  

–  Coordina8ng  these  changes  with  client  libraries  accessing  the  cluster  

12  

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.

CLUSTERING  ARCHITECTURE  

13  

                           

                           

         

COUCHBASE  CLIENT  LIBRARY      

Basic  Opera8on  

 Docs  distributed  evenly  across  servers  in  the  cluster  

 Each  server  stores  both  ac/ve  &  replica  docs    Only  one  server  ac8ve  at  a  8me  

 Client  library  provides  app  with  simple  interface  to  database  

 Cluster  map  provides  map  to  which  server  doc  is  on    App  never  needs  to  know  

  App  reads,  writes,  updates  docs  

  Mul8ple  App  Servers  can  access  same  document  at  same  8me  

 

                           

Doc  4  

Doc  2  

Doc  5  

SERVER  1  

Doc  6  

Doc  4  

SERVER  2  

Doc  7  

Doc  1  

SERVER  3  

Doc  3  

User  Configured  Replica  Count  =  1  

Read/Write/Update  

         

COUCHBASE  CLIENT  LIBRARY      

Read/Write/Update  

Doc  9  

Doc  7  

Doc  8   Doc  6  

Doc  3  

DOC  

DOC  

DOC  

DOC  

DOC  

DOC  

DOC  

DOC  

DOC  

DOC  

DOC  

DOC  

DOC  

DOC  

DOC  

Doc  9  

Doc  5  

DOC  

DOC  

DOC  

Doc  1  

Doc  8   Doc  2  

Replica  Docs   Replica  Docs   Replica  Docs  

Ac8ve  Docs   Ac8ve  Docs   Ac8ve  Docs  

CLUSTER  MAP      

CLUSTER  MAP      

APP  SERVER  1   APP  SERVER  2  

COUCHBASE  SERVER    CLUSTER  

14  

                           

                           

Add  Nodes  

  Two  servers  added  to  cluster    One-­‐click  opera8on  

  Docs  automa8cally  rebalanced  across  cluster    Even  distribu8on  of  

docs    Minimum  doc  movement  

  Cluster  map  updated  

  App  database  calls  now  distributed  over  larger  #  of  servers  

User  Configured  Replica  Count  =  1  

Read/Write/Update   Read/Write/Update  

Doc  7  

Doc  9  

Doc  3  

Ac8ve  Docs  

Replica  Docs  

Doc  6  

         

COUCHBASE  CLIENT  LIBRARY      CLUSTER  MAP  

   

APP  SERVER  1  

         

COUCHBASE  CLIENT  LIBRARY      CLUSTER  MAP  

   

APP  SERVER  2  

                           

                           

                           

Doc  4  

Doc  2  

Doc  5  

SERVER  1  

Doc  6  

Doc  4  

SERVER  2  

Doc  7  

Doc  1  

SERVER  3  

Doc  3  

Doc  9  

Doc  7  

Doc  8   Doc  6  

Doc  3  

DOC  

DOC  

DOC  

DOC  

DOC  

DOC  

DOC  

DOC  

DOC  

DOC  

DOC  

DOC  

DOC  

DOC  

DOC  

Doc  9  

Doc  5  

DOC  

DOC  

DOC  

Doc  1  

Doc  8   Doc  2  

Replica  Docs   Replica  Docs   Replica  Docs  

Ac8ve  Docs   Ac8ve  Docs   Ac8ve  Docs  

SERVER  4   SERVER  5  

Ac8ve  Docs   Ac8ve  Docs  

Replica  Docs   Replica  Docs  

COUCHBASE  SERVER    CLUSTER  

15  

                           

                           

Fail  Over  Node  

  App  servers  happily  accessing  docs  on  Server  3  

  Server  fails    App  server  requests  to  server  3  fail    Cluster  detects  server  has  failed  

  Promotes  replicas  of  docs  to  ac/ve    Updates  cluster  map  

  App  server  requests  for  docs  now  go  to  appropriate  server  

  Typically  rebalance    would  follow    

User  Configured  Replica  Count  =  1  

Doc  7  

Doc  9  

Doc  3  

Ac8ve  Docs  

Replica  Docs  

Doc  6  

         

COUCHBASE  CLIENT  LIBRARY      CLUSTER  MAP  

   

APP  SERVER  1  

         

COUCHBASE  CLIENT  LIBRARY      CLUSTER  MAP  

   

APP  SERVER  2  

                           

                           

                           

Doc  4  

Doc  2  

Doc  5  

SERVER  1  

Doc  6  

Doc  4  

SERVER  2  

Doc  7  

Doc  1  

SERVER  3  

Doc  3  

Doc  9  

Doc  7   Doc  8  

Doc  6  

Doc  3  

DOC  

DOC  

DOC  DOC  

DOC  

DOC  

DOC   DOC  

DOC  

DOC  

DOC   DOC  

DOC  

DOC  

DOC  

Doc  9  

Doc  5  DOC  

DOC  

DOC  

Doc  1  

Doc  8  

Doc  2  

Replica  Docs   Replica  Docs   Replica  Docs  

Ac8ve  Docs   Ac8ve  Docs   Ac8ve  Docs  

SERVER  4   SERVER  5  

Ac8ve  Docs   Ac8ve  Docs  

Replica  Docs   Replica  Docs  

COUCHBASE  SERVER    CLUSTER  

16  

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.

SINGLE  INDEX  VIEW  

17  

Couchbase  Server  Views  

•  Couchbase  has  a  method  of  building  and  maintaining  views  over  the  set  of  documents  within  a  bucket  

•  Can  be  used  for:  –  Secondary  indexes  –  Aggrega8ons  and  stats  –  Analyzing  datasets  

•  Built  for  speed  –  Allow  relaxed  consistency  for  speed,  or  consistent  if  required    

map:  function(doc) { emit(doc.name, [doc.city, doc.salary]); }  reduce:  function (key, values, rereduce) { var sum = 0; for(i=0; i < values.length; i++) { sum = sum + values[i]; } return(sum); }

18  

Views  are  B+Trees  Over  the  Data  

bits 0

Key: "f"Sum: 17

0 0 0 bits 0

Key: "f"Sum: 17

0 0 0

bits 0

Key: "f"Sum: 17

0 0 0 bits 0

Key: "f"Sum: 17

0 0 0 bits 0

Key: "f"Sum: 17

0 0 0 bits 0

Key: "f"Sum: 17

0 0 0

bits 0

Key: "f"Sum: 17

0 0 0 bits 0

Key: "f"Sum: 17

0 0 0 bits 0

Key: "f"Sum: 17

0 0 0 bits 0

Key: "f"Sum: 17

0 0 0 bits 0

Key: "f"Sum: 17

0 0 0 bits 0

Key: "f"Sum: 17

0 0 0 bits 0

Key: "f"Sum: 17

0 0 0 bits 0

Key: "f"Sum: 17

0 0 0

Key: "a"Value: 4

vBucket: 1

Key: "b"Value: 2

vBucket: 2

Key: "c"Value: 0

vBucket: 3

Key: "d"Value: 6

vBucket: 4

Key: "e"Value: 4

vBucket: 5

Key: "f"Value: 4

vBucket: 6

Key: "j"Value: 4

vBucket: 10

Key: "k"Value: 4

vBucket: 11

Key: "l"Value: 4

vBucket: 12

Key: "m"Value: 4

vBucket: 13

Key: "n"Value: 4

vBucket: 14

Key: "o"Value: 4

vBucket: 15

Key: "p"Value: 4

vBucket: 16

Key: "q"Value: 4

vBucket: 17

Key: "r"Value: 4

vBucket: 18

Key: "s"Value: 4

vBucket: 19

Key: "g"Value: 4

vBucket: 7

Key: "h"Value: 4

vBucket: 8

Key: "i"Value: 4

vBucket: 9

Key: "t"Value: 4

vBucket: 20

Key: "u"Value: 4

vBucket: 21

Key: "v"Value: 4

vBucket: 22

Key: "w"Value: 4

vBucket: 23

Key: "x"Value: 4

vBucket: 24

Key: "y"Value: 4

vBucket: 25

Key: "z"Value: 4

vBucket: 26

Views  are  Built  Atop  Incremental  Map  Reduce  •  Distributed  throughout  the  cluster  •  Query  API  abstracts  the  cluster  topology,  and  automa8cally  gathers  the  results  •  Rebuilds  subsets  of  the  B+Tree  based  on  updates  

19  

bits 0

Key: "m"Sum: 92

0 0 0 bits 0

Key: "z"Sum: 17

0 0 0

bits 0

Key: "f"Sum: 21

0 0 0 bits 0

Key: "m"Sum: 71

0 0 0

bits 0

Key: "d"Sum: 10

0 0 0 bits 0

Key: "f"Sum: 11

0 0 0 bits 0

Key: "i"Sum: 24

0 0 0 bits 0

Key: "m"Sum: 47

0 0 0

Key: "a"Value: 4

vBucket: 1

Key: "b"Value: 2

vBucket: 2

Key: "c"Value: 0

vBucket: 3

Key: "d"Value: 6

vBucket: 4

Key: "e"Value: 4

vBucket: 5

Key: "f"Value: 4

vBucket: 6

Key: "j"Value: 4

vBucket: 10

Key: "k"Value: 4

vBucket: 11

Key: "l"Value: 4

vBucket: 12

Key: "m"Value: 4

vBucket: 13

Key: "g"Value: 4

vBucket: 7

Key: "h"Value: 4

vBucket: 8

Key: "i"Value: 4

vBucket: 9

20  

Data  is  Distributed  Using  vbuckets,  but…  

•  Couchbase  Server  has  1024  vbuckets  stored  throughout  the  cluster  

21  

B-­‐Superstar:  a  new  B+Tree  ready  for  vbuckets  

Only  one  B+Tree  per  node  with  masking  of  ac8ve  vbucket  Data.  

S8ll  allows  concurrent  access  and  even  

materializing  tree  during  rebalance  

Read  more  on  Damien’s  blog:  hmp://damienkatz.net/2012/05/stabilizing_couchbase_server_2.html  

22  

Perry  Krug  

Thanks  Q&A