Powering Microservices with Docker, Kubernetes & Kafka Andrew Morgan MongoDB Product Marketing @andrewmorgan

#MDBW16

Agenda

What, Why, How?

Microservices 01 Kubernetes, Mesos,…

Orchestration 03 Docker Kafka

Containers 02

Why, How?

MongoDB 04 Who, Why?

Use Cases 06 When to Use 05

Microservices

#MDBW16

Why Use Microservices? (tl;dr WebScale)

Team = Components No committees

Empower Build MVP quickly

Speed Simple components

Isolated impact

Maintain Rapid iterations React to market

Change Product Team

Scale

#MDBW16

1990s & Earlier

Monolithic

Cross-Team

Tight Coupling

Small Change

Huge Impact

System Retests

#MDBW16

2000s

SOA

Coordination

Looser Coupling

System Retests

#MDBW16

2010s

Microservices

Decoupled

Independent Dev

Isolated Impact

#MDBW16

Microservices Example

Twitter Ingest Google+

Ingest

Snapchat Ingest

Feed merge

Facebook Ingest

#MDBW16

Microservices Example

Twitter Ingest

Snapchat Ingest

Feed merge

Facebook Ingest

#MDBW16

Microservices Example

Twitter Ingest

Snapchat Ingest

Feed merge

Facebook Ingest

#MDBW16

Microservices Example

Twitter Ingest

Snapchat Ingest

Feed merge

Facebook Ingest

Whatsapp Ingest

#MDBW16

Microservices Example

Twitter Ingest

Snapchat Ingest

Feed merge

Facebook Ingest

Whatsapp Ingest

Snapchat Ingest Snapchat

Ingest

#MDBW16

Alternate Microservices Example

• Much larger Microservices 1.  User Account 2.  Product Catalog 3.  Inventory 4.  Orders

• Best practice is each microservice has its own database

“ Conway’s Law 1967

Any organization that designs a system will inevitably produce a design whose structure is a copy of the organization's communication structure.”

#MDBW16

Development Teams

Containers

#MDBW16

Containers – Powering Microservices

Real world shipping containers • Road, rail & sea • Contents untouched • Ubiquitous & standardized • Simple • Contents protected • Constraints

#MDBW16

Containers – Powering Microservices

Software containers • 1 image -> Many containers

•  Laptop, DC, cloud •  Dev, QA, production, support

• Simple, efficient •  Isolation • Constraints

#MDBW16

VMs vs. Containers

VM VM VM

Bare Metal

Host Operating System

Hypervisor

Guest OS

Libraries

Apps

Service

Guest OS

Libraries

Apps

Service

Guest OS

Libraries

Apps

Service

Container Container Container

Bare Metal

Host Operating System

Docker Engine

Libraries

Libraries

Apps

Libraries

Apps

Service Service Service

#MDBW16

VMs vs. Containers

VM VM VM

Bare Metal

Host Operating System

Hypervisor

Guest OS

Libraries

Apps

Service

Guest OS

Libraries

Apps

Service

Guest OS

Libraries

Apps

Service

Container Container Container

Bare Metal

Host Operating System

Docker Engine

Libraries

Libraries

Apps

Libraries

Apps

Service Service Service

#MDBW16

VMs vs. Containers

VM VM VM

Bare Metal

Host Operating System

Hypervisor

Guest OS

Libraries

Apps

Service

Guest OS

Libraries

Apps

Service

Guest OS

Libraries

Apps

Service

Container Container Container

Bare Metal

Host Operating System

Docker Engine

Libraries

Libraries

Apps

Libraries

Apps

Service Service Service

#MDBW16

Docker

• Simple to use • 100K+ images on Docker Hub • Build images from images • Platforms

•  Linux, OS X, Windows •  Laptop, VM, Cloud,… •  Cloud services

#MDBW16

Run MongoDB

docker run -d mongo

#MDBW16

Security

Opportunities •  Isolation • Constrain resources • Narrow roles • Keep images/containers current • root: so restrict access

#MDBW16

Title only

#MDBW16

Title only

#MDBW16

Microservice Architectures Built on Containers

Many small, focused containers -> sophisticated services • Well defined APIs •  Independent languages & libraries • Modular: easy maintenance + reuse •  Fault tolerant • Scalable

#MDBW16

Connecting the Microservices – Apache Kafka

Producer 987

1 2 3 ...

Topic A

Consumer

#MDBW16

Connecting the Microservices – Apache Kafka

Producer

987

1 2 3 ...

Topic A

Consumer

Producer Consumer

#MDBW16

Connecting the Microservices – Apache Kafka

Producer

987

1 2 3 ... Partition 0

Topic A

Consumer

Producer Consumer

435

1 2 3 ... Partition 1

#MDBW16

Connecting the Microservices – Apache Kafka

Producer

LEADER

Topic A / Partition 0

Broker 1

FOLLOWER

Topic A / Partition 1

FOLLOWER

Topic A / Partition 0

Broker 2

LEADER

Topic A / Partition 1

#MDBW16

Connecting the Microservices – Apache Kafka

Producer

Producer

Producer

987

1 2 3 ...

Partition 0

435

1 2 3 ...

Partition 1

732

1 2 3 ...

Partition N

Topic A

Topic B

765

1 2 3 ...

Partition 0

New ß Old

Consumer

Consumer

Orchestration

#MDBW16

Orchestration

Automated deployment, connecting, and maintenance of multiple containers •  Provision hosts •  Containers

•  Instantiate •  Reschedule •  Link •  Scale Out/In

•  Expose services

#MDBW16

Kubernetes

Created by Google, feature-rich and widely adopted •  Deployment and ‘replication’ •  On-line scale out/in •  Rolling upgrades •  High Availability •  Persistence •  Ports •  Load balancing •  Google Compute Engine

#MDBW16

Apache Mesos

10,000s of physical servers; used by Twitter, Airbnb & Apple • Code (“frameworks) vs. declarative •  Less feature rich than Kubernetes • Kubernetes as a Mesos framework •  Foundation for distributed systems

•  Apache Aurora, Chronos, Marathon

#MDBW16

Choosing an Orchestration Framework

• What you have: •  Skills? •  DevOps frameworks? •  Number of hosts? •  Bare metal, VMs, or cloud?

•  Lifecycle •  Features

•  Automated High Availability? •  Grouping and load balancing? •  As a service??

MongoDB

#MDBW16

Why MongoDB is a Good Fit For Microservices

Scalability Monitoring &

Automation

Redundancy Flexible Data Model

Simplicity

#MDBW16

Orchestrating MongoDB Using Kubernetes

Distributed, stateful application

•  Persistent volumes

•  External IP addresses for internal comms

•  Init MongoDB replica set

•  Monitor

•  Backup

Volumename: mongo-persistent-storage1

pdName: mongodb-disk1

mongodb-disk1

Containername: mongo-node1image: mongocommand: mongod –replSet my_replica_setcontainerPort: 27017volumeMounts:

name: mongo-persistent-storage1mountPath: /data/db

DockerHubRegistry

Podlabels: [name: mongo-node1; instance: rod]

ReplicationControllername: mongo-rc1labels: [name: mongo-rc]replicas: 1 selector: [name: mongo-node1]

Service:LoadBalancername: mongo-svc-a labels: [name: mongo-svc-a]ports: [port: 27017, targetPort: 27017]]selector: [name: mongo-node, instance: rod]Ex

ternalIP

Address

104.1.1.1

App

104.1.1.1:27017

ReplicationControlername: mongo-rc-europelabels: [name: mongo-europe]replicas: 1 selector: [name: mongo-node]

Volumename:

mongo-persistent-storagepdName: mongodb-disk-europe

mongodb-disk-europe

Containername: mongo-nodeimage: mongocommand: mongod –replSet my_replica_setcontainerPort: 27017volumeMounts:

name: mongo-persistent-storagemountPath: /data/db

Podlabels: [name: mongo-node]

Service:LoadBalancername: mongo-svc-europelabels: [name: mongo-svc-europe]ports: [port: 27017, targetPort: 27017]]selector: [name: mongo-node]

ReplicationControlername: mongo-rc-asialabels: [name: mongo-asia]replicas: 1 selector: [name: mongo-node]

Volumename: mongo-persistent-storage

pdName: mongodb-disk-asia

mongodb-disk-asia

Containername: mongo-nodeimage: mongocommand: mongod –replSet my_replica_setcontainerPort: 27017volumeMounts:

name: mongo-persistent-storagemountPath: /data/db

Podlabels: [name: mongo-node]

Service:LoadBalancername: mongo-svc-asialabels: [name: mongo-svc-asia]ports: [port: 27017, targetPort: 27017]]selector: [name: mongo-node]

ReplicationControlername: mongo-rc-uslabels: [name: mongo-us]replicas: 1 selector: [name: mongo-node]

Volumename: mongo-persistent-storage

pdName: mongodb-disk-us

mongodb-disk-us

Containername: mongo-nodeimage: mongocommand: mongod –replSet my_replica_setcontainerPort: 27017volumeMounts:name: mongo-persistent-storagemountPath: /data/db

Podlabels: [name: mongo-node]

Service:LoadBalancername: mongo-svc-uslabels: [name: mongo-svc-us]ports: [port: 27017, targetPort: 27017]]selector: [name: mongo-node]

When to use Microservices

#MDBW16

When to use Microservices

Use Cases

#MDBW16

MongoDB & Microservices in the Wild

#MDBW16

References

• Enabling Microservices – Containers & Orchestration Explained

https://www.mongodb.com/collateral/microservices-containers-and-

orchestration-explained

• Microservices: The Evolution of Building Modern Applications

https://www.mongodb.com/collateral/microservices-the-evolution-of-

building-modern-applications