apache mesos and mesosphere: live webcast by ceo and co-founder florian leibert
DESCRIPTION
In this slide deck you will learn: - How Twitter used Mesos to eliminate the "fail whale" - Why Mesos is important to companies of any size - How Mesosphere Mesos reduces operational complexity - How to use Mesos to increase developer velocity - How Mesos helps you reduce total cost of ownership and ultimately increase the competitiveness of your business - How to use Mesos on your own datacenter, or virtualized in a public cloud like Amazon Web Services (AWS) and Google Compute Engine (GCE) You will also gain knowledge about how Apache Mesos is being used by well-known brands, including Twitter, Airbnb, and Hubspot to improve their operational effectiveness and save money. Learn how Twitter has scaled its service, running Mesos on tens of thousands of nodes. Learn how Airbnb has reduced its operational complexity, running it's entire data stack on Mesos. And learn how Hubspot has figured out how to use Mesos to reduce it's Amazon Web Services bill by 65%.TRANSCRIPT
![Page 1: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/1.jpg)
Brand GuidelinesVersion 110.21.2013
Software for the hyperscale datacenter
Florian Leibert, CEO & Founder [email protected]
![Page 2: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/2.jpg)
Imagine if…
![Page 3: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/3.jpg)
All your servers in your datacenter
and cloud
![Page 4: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/4.jpg)
were pooled together
![Page 5: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/5.jpg)
So they behave like one big computer
![Page 6: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/6.jpg)
as easy as
![Page 7: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/7.jpg)
…and building
new datacenter apps is as
easy as building an app for one
machine PowerBook G4
![Page 8: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/8.jpg)
Fantasy?
![Page 9: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/9.jpg)
![Page 10: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/10.jpg)
Brand GuidelinesVersion 110.21.2013
![Page 11: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/11.jpg)
Mesos Users Today
Sigmoid Analytics
![Page 12: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/12.jpg)
![Page 13: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/13.jpg)
Today’s Legacy Datacenter
![Page 14: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/14.jpg)
Provision VMs in the cloud or on physical servers
Today’s Legacy Datacenter
![Page 15: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/15.jpg)
Install Hadoop on a static set of machines
Installing an Application with Static Partitioning
![Page 16: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/16.jpg)
Install Web Server on a static set of machines
Installing an Application with Static Partitioning
![Page 17: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/17.jpg)
Scale up Hadoop manually
Resizing an Application with Static Partitioning
![Page 18: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/18.jpg)
What if your Laptop was operated like your Data
![Page 19: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/19.jpg)
Issues with Statically Partitioned Data Centers
Complex Machine sprawl, manual resize/scale LimitedNo software failure handling, “black box”
Inefficient Static partitioning, overhead
Not Developer-FriendlyLong time to roll out software, development starts at the machine level
![Page 20: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/20.jpg)
Aggregation
Mesosphere aggregates resources, makes a data center look like one big computer
Mesosphere runs on top of a VM or on bare metal
![Page 21: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/21.jpg)
Applications in the Cloud Era
Cloud Era: Big apps, small servers
Client-Server Era: Small apps, big servers
Server
Virtualization
App App App AppApp
Aggregation
Serv Serv Serv Serv
![Page 22: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/22.jpg)
From Static Partitioning to Elastic Sharing
Static Partitioning
Elastic Sharing
WEB HADOOPCACHE
WASTED
FREEFREEHADOOP
WEB
CACHE
WASTED WASTED100% —
100% —
![Page 23: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/23.jpg)
Applications are Changing
![Page 24: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/24.jpg)
Deployments
![Page 25: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/25.jpg)
Mesos Facts
Scales to 10,000s of nodes Top-level Apache project Twitter and Airbnb are major users and contributors APIs for C++, Python, JVM, Go Pluggable CPU, memory, IO isolation Packages and commercial support through Mesosphere Highly available, scalable, elastic
![Page 26: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/26.jpg)
Apache Mesos Features
Multi-tenancy
Improved resource utilization
Resource Isolation
Fault-tolerance, HA
Scalability + Elasticity
26
![Page 27: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/27.jpg)
Mesos Slave
Hadoop task-tracker Mesos Executor
Task #1 Task #2 ./ruby XYZ
Mesos Slave
Docker Executor Docker Executor
java -jar XYZ.jar ./xyz
Mesos Master Mesos Master Mesos Master
Hadoop scheduler
Marathon scheduler
Zookeeperquorum
![Page 28: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/28.jpg)
Mesos Slave
Hadoop task-tracker Mesos Executor
Task #1 Task #2 ./ruby XYZ
Mesos Slave
Docker Executor Docker Executor
java -jar XYZ.jar ./xyz
Mesos Master Mesos Master Mesos Master
Hadoop scheduler
Marathon scheduler
Zookeeperquorum
28
![Page 29: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/29.jpg)
Mesos Slave
Hadoop task-tracker Mesos Executor
Task #1 Task #2 ./ruby XYZ
Mesos Slave
Docker Executor Docker Executor
java -jar XYZ.jar ./xyz
Mesos Master Mesos Master Mesos Master
Hadoop scheduler
Marathon scheduler
Zookeeperquorum
29
![Page 30: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/30.jpg)
Mesos Slave
Hadoop task-tracker Mesos Executor
Task #1 Task #2 ./ruby XYZ
Mesos Slave
Docker Executor Docker Executor
java -jar XYZ.jar ./xyz
Mesos Master Mesos Master Mesos Master
Hadoop scheduler
Marathon scheduler
Zookeeperquorum
30
![Page 31: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/31.jpg)
Resource Offers and Launching a Task
1. resourceOffers()2. launchTasks()
3. launchTask() 4. statusUpdate()
5. statusUpdate()
Mesos Slave
Executor
Mesos Master
Scheduler
Task Task
![Page 32: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/32.jpg)
Response times and overhead are significantly Ti
me
to p
rovi
sion
(sec
onds
)
1
100
10000
Bare metal VM Container
Inspired by Tomas Barton’s Mesos talk at InstallFest in Prague
![Page 33: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/33.jpg)
Mesos provides fine-grained resource isolation
Compute Node
Mesos Slave Process
Hadoop task-tracker Mesos Executor
Task #1 Task #2 ruby XYZ
Container(Cgroups)
Executor
![Page 34: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/34.jpg)
Mesos provides fine-grained resource isolation
Compute Node
Mesos Slave Process
Hadoop task-tracker
Task #1 Task #2
Container(Cgroups)
Task #3
![Page 35: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/35.jpg)
Mesos provides componentized resource
Mesos Slave Process
Mesos Containerizer
CGroups CPU isolator
CGroups Memory isolator
Launcher
Container foo
Task baz
Containerizer API
Executor bar
![Page 36: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/36.jpg)
Mesos provides pluggable resource isolation
External Containerizer
External Containerizer API
Mesos Slave Process
External Containerizer Program
Container foo
MySQL
Containerizer API
Ubuntu 13.10
Container bar
Ruby
Centos 6.4
github.com/mesosphere/deimos
![Page 37: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/37.jpg)
From Static Partitioning to Elastic Sharing
Static Partitioning
Elastic Sharing
WEB HADOOPCACHE
WASTED
FREEFREEHADOOP
WEB
CACHE
WASTED WASTED100% —
100% —
![Page 38: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/38.jpg)
Mesos has no single point of failure
Tasks keep running!
Framework
Masters
![Page 39: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/39.jpg)
Master node can fail-over
Tasks keep running!
Framework
Masters
![Page 40: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/40.jpg)
Slave processes can fail-over
Tasks keep running!
Compute Node
Mesos Slave Process
Mesos Executor Mesos Executor
![Page 41: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/41.jpg)
The UNIX Operating System Stack
SSHd
Linux, BSD
MySQLApache
Kernel
ApplicationsMemcached
Init, Upstart, Systemd Init System
![Page 42: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/42.jpg)
The Mesos Stack
Memcached
Mesos
RedisRails
Kernel
ApplicationsElasticsearch
Marathon Init System
![Page 43: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/43.jpg)
Mesosphere Demo
![Page 44: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/44.jpg)
Marathon
![Page 45: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/45.jpg)
What is Marathon?
“Init Daemon” for the data center
•Runs any Linux binary without modification (e.g. Rails, Tomcat, …)
•Cluster-wide process supervisor
!
Private PaaS
•Service discovery
•Automated software and hardware failure handling
•Deployment and scaling
![Page 46: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/46.jpg)
Marathon Design Goals
Simplify Fewer things to manage = fewer sources of error All machines have the same configuration AutomateNobody likes to get paged at nightAutomatically respond to hardware & software failures
Improve efficiency Humans are bad at estimating resource requirements Use software-controlled elastic resource sharing
Self-serve API Developers want to have control over their apps Give them direct access to cluster resources
![Page 47: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/47.jpg)
Marathon Key Features
REST/JSON API
Easy to use web interface
Authentication & SSL
Highly available - no single point of failure
Placement constraints (nodes, racks, etc.)
Service discovery & load balancing via HAProxy
Event system for integration with 3rd party components, like load balancers
!
![Page 48: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/48.jpg)
Marathon Workflow
3.foo.com1.foo.com 2.foo.com
Rails Rails
Mesos
Marathon
POST /v2/apps
![Page 49: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/49.jpg)
Marathon Workflow
1.foo.com 3.foo.com2.foo.com
Rails Rails
Mesos
Marathon
POST /v2/apps
Play
Play
![Page 50: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/50.jpg)
Marathon API - Launching Self-Contained Apps
•Command to start the app •URL(s) to the app archive/configuration •Environment variables
!POST /v2/apps { “id”: “Play”, “uris”: [“http://downloads.mesosphere.io/tutorials/PlayHello.zip”] “cmd”: “./Hello-*/bin/hello -Dhttp.port=$PORT”, “env”: {“SECRET”: “password123”} }
![Page 51: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/51.jpg)
Marathon API - Launching Dockers
•Starting with Mesos 0.19 containers are 1st class citizens •Deimos is the Docker containerizer !POST /v2/apps { “id”: “Cassandra”, “container”: { “image”: “docker:///mesosphere/cassandra:2.0.6”, “options”: [“-v”, “/mnt:/mnt:rw”, “-e”, “CLUSTER_NAME=prod”] } }
![Page 52: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/52.jpg)
Marathon API - Integration with Deimos
Mesos Master
Marathon
Launch 3x Redis
Mesos Slave
DeimosTask 1
Task 2docker run ...
Deimos
docker run ...
Mesos Slave
Deimos
docker run ...
Task 3
![Page 53: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/53.jpg)
Marathon API - Scaling Apps
•Just tell Marathon how many you want! !!
PATCH /v2/apps/Play { “instances”: 4 }
![Page 54: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/54.jpg)
![Page 55: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/55.jpg)
![Page 56: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/56.jpg)
![Page 57: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/57.jpg)
Marathon Service Discovery with HAProxy
Apps available on localhost & known port
HAProxy updates via Marathon REST API
HAProxy runs on every cluster node
Configurable policies
Rails
HAProxy
Rails
HAProxy
SQL
HAProxy
Search Search
HAProxy
Cache
Marathon
![Page 58: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/58.jpg)
Mesosphere Products & Services
Infrastructure • Try out Mesos on Amazon Web Services
https://elastic.mesosphere.io • On premise and cloud provisioning tools • Linux Packages • Mesosphere Plugins and Tools • Professional Services
• Training, Installation, Support Contracts
![Page 59: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.vdocuments.mx/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/59.jpg)
Brand GuidelinesVersion 110.21.2013
Thank you.