![Page 1: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/1.jpg)
Building Robust Systems With Consul
![Page 2: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/2.jpg)
I’m Mitchell HashimotoAlso known as @mitchellh
![Page 3: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/3.jpg)
HashiCorpTowards a Software Managed Datacenter
![Page 4: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/4.jpg)
Vagranthttp://www.vagrantup.com
Packerhttp://www.packer.io
SERFhttp://www.serfdom.io
Consulhttp://www.consul.io
![Page 5: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/5.jpg)
Consul
![Page 6: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/6.jpg)
Take a Step BackTaking a look at the big picture.
![Page 7: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/7.jpg)
Node
Service Service Service
![Page 8: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/8.jpg)
Hypervisor
Node Node Node
S S S S S S S S S
![Page 9: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/9.jpg)
Hypervisor
Node Node Node
Container
S S Container S Container
S S S S S S
![Page 10: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/10.jpg)
Hypervisor
Node Node Node
Container
S S Container S Container
S S S S S S
![Page 11: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/11.jpg)
Modern OpsMore everything, more problems.
![Page 12: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/12.jpg)
• Where is service foo?• Is service foo healthy/available?• What is service foo’s
configuration?• Where is the service foo leader?
![Page 13: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/13.jpg)
Meta:
What happens when the thing that answers these questions is unavailable?
![Page 14: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/14.jpg)
![Page 15: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/15.jpg)
Robust SystemsStem from the ability to answer these questions.
![Page 16: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/16.jpg)
• Start services in any order• Destroy services with confidence• Restart servers safely• Reconfigure services easily
Practical Goals
![Page 17: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/17.jpg)
• Where is service foo?• Is service foo healthy/available?• What is service foo’s
configuration?• Where is the service foo leader?
![Page 18: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/18.jpg)
Where is service foo?
Maybe here: 127.0.0.1Maybe close: 10.0.1.35Maybe there: foo.foohost.com
![Page 19: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/19.jpg)
Is service foo healthy/available?
Yes: Great!No: Avoid or handle gracefully.
![Page 20: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/20.jpg)
What is service foo’s configuration?
Access information, supported features, enabled/disabled.
![Page 21: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/21.jpg)
What is my configuration?
Expect it to be modifiable.
![Page 22: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/22.jpg)
Where is the service foo leader or best choice?
Locality, master/slave, versions.
![Page 23: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/23.jpg)
Meta: Is the thing answering these questions stable/available?
Critical infrastructure component, you want “yes” as often as possible.
![Page 24: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/24.jpg)
Robust! Can find services, can avoid and handle unhealthy services, can be configured externally, and can trust that it can retrieve all of this information.
![Page 25: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/25.jpg)
• Start services in any order• Destroy services with confidence• Restart servers safely• Reconfigure services easily
Practical Goals
![Page 26: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/26.jpg)
Consul
![Page 27: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/27.jpg)
Solution AttemptsIn a world… before Consul...
![Page 28: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/28.jpg)
Manual/Hardcoded• Doesn’t scale with services/nodes• Not resilient to failures• Localized visibility/auditability• Manual locality of services
![Page 29: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/29.jpg)
Config Mgmt Problem• Slow to react to changes• Not resilient to failures• Not really configurable by
developers• Locality, monitoring, etc. manual
![Page 30: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/30.jpg)
LB Fronted Services• Introduces different SPOF• How does LB find service
addresses/configure?• Solves some problems, though.
![Page 31: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/31.jpg)
ZooKeeper• Complicated• Heavy clients• Building block, very manual
![Page 32: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/32.jpg)
Consul
![Page 33: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/33.jpg)
Service Discovery
Where is service foo?
![Page 34: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/34.jpg)
Service Discovery$ dig web-frontend.service.consul. +short10.0.3.8910.0.1.46
$ curl http://localhost:8500/v1/catalog/service/web-frontend[{ “Node”: “node-e818f1”, “Address”: “10.0.3.89”, “ServiceID”: “web-frontend”, …}]
![Page 35: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/35.jpg)
Service Discovery
• DNS is legacy-friendly. No application changes required.
• HTTP returns rich metadata.
![Page 36: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/36.jpg)
Failure Detection
Is service foo healthy/available?
![Page 37: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/37.jpg)
Failure Detection
![Page 38: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/38.jpg)
Failure Detection
• DNS won’t return non-healthy services or nodes.
• HTTP has endpoints to list health state of catalog.
![Page 39: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/39.jpg)
Key/Value Storage
What is the config of service foo?
![Page 40: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/40.jpg)
Key/Value Storage$ curl –X PUT –d ‘bar’ http://localhost:8500/v1/kv/footrue
$ curl http://localhost:8500/v1/kv/foo?rawbar
![Page 41: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/41.jpg)
Key/Value Storage
• Highly available storage of configuration.
• Turn knobs without big configuration management process.
![Page 42: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/42.jpg)
Multi-Datacenter
![Page 43: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/43.jpg)
Multi-Datacenter$ dig web-frontend.singapore.service.consul. +short10.3.3.3310.3.1.18
$ dig web-frontend.germany.service.consul. +short10.7.3.4110.7.1.76
![Page 44: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/44.jpg)
Multi-Datacenter$ curl http://localhost:8500/v1/kv/foo?raw&dc=asiatrue
$ curl http://localhost:8500/v1/kv/foo?raw&dc=eufalse
![Page 45: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/45.jpg)
Multi-Datacenter
• Local by default• Can query other datacenters
however you may need to
![Page 46: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/46.jpg)
Web UI
![Page 47: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/47.jpg)
Web UI
• Node, service, health check, and K/V management and visibility for every datacenter in a single UI.
![Page 48: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/48.jpg)
OperationsConsul Availability / Scalability
![Page 49: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/49.jpg)
The Meta Question
![Page 50: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/50.jpg)
Architecture
![Page 51: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/51.jpg)
Server Cluster• 3, 5, 7 servers• (n/2) + 1 for
availability• Replicated writes• Automatic leader
election, leader forwarding.
![Page 52: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/52.jpg)
Lightweight Clients• Ephemeral state• Health checks• Optional (but
recommended). Legacy machines don’t need them.
• Automatic request forwarding to servers.
![Page 53: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/53.jpg)
Cheap Gossip• Health check and
membership info.• Very cheap• No guaranteed
reliability, but only used for data that can be lost
• (See Serf)
![Page 54: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/54.jpg)
Multi-DC• Independent server
clusters• Request forwarding• WAN gossip for
membership
![Page 55: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/55.jpg)
General Points: Servers
• (n+1)/2 servers for write avail• More servers means higher write latency
because of replication. Throughput marginally affected.
• Can leave/add at will, keeping in mind min. node requirement.
![Page 56: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/56.jpg)
General Points: Clients• Clients can be removed/added at will
without issue.• Clients don’t currently affect read/write
throughput in a meaningful way.• Although technically optional, they’re
highly recommended for delegated health checks.
![Page 57: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/57.jpg)
Throughput
• On virtualized cloud systems with spinning disks: thousands of reads and writes per second
• Practically won’t hit read/write limit
![Page 58: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/58.jpg)
Scalable and available. Consul’s architecture makes it incredibly scalable and highly unlikely to become unavailable.
![Page 59: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/59.jpg)
Robust SystemsConsul configured, monitored, discovered
![Page 60: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/60.jpg)
• Consul KV for configuration.• Consul DNS for service
coupling/discovery.• Consul Health Checks for
monitoring.
![Page 61: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/61.jpg)
Consul KV: Configuration
![Page 62: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/62.jpg)
Consul KV: Configuration$ envconsul –reload myapp/config bin/myapp…
![Page 63: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/63.jpg)
Consul KV: Configuration
• envconsul turns K/V into environmental variables and restarts on change.
• No application changes!
![Page 64: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/64.jpg)
Consul DNS: Service Discovery$ envconsul myapp/config envELASTICSEARCH_HOST=elasticsearch.service.consul.POSTGRESQL_HOST=master.postgresql.service.consul.REDIS_HOST=redis.service.consul.
![Page 65: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/65.jpg)
Consul DNS: Service Discovery
• Configuration to point to other services uses DNS.
• No application changes!
![Page 66: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/66.jpg)
Consul Health Checks: Monitoring$ cat /etc/consul.d/web.json{ “check”: { “name”: “http”, “script”: “curl localhost:80”, “interval”: “5s” }}
![Page 67: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/67.jpg)
Consul Health Checks: Monitoring
![Page 68: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/68.jpg)
Consul Health Checks: Monitoring
• Simple shell scripts (UNIXy)• Logged output• Won’t show as result in service
discovery queries if failing.
![Page 69: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/69.jpg)
Robust! Add/remove services, reconfigure services, see global state of services without complicated logic. And without modifying application code.
![Page 70: Mitchell Hashimoto: Building Robust Systems w/ Service Discovery & Configuration](https://reader036.vdocuments.mx/reader036/viewer/2022070303/5493323eb479594c5c8b468a/html5/thumbnails/70.jpg)
Thank You
http://www.consul.io