server monitoring (scaling while bootstrapped)

37
By Ajibola Aiyedogbon Server Monitoring (scaling while bootstrapped)

Upload: ajibola-aiyedogbon

Post on 19-Feb-2017

149 views

Category:

Technology


0 download

TRANSCRIPT

Page 1: Server Monitoring  (Scaling while bootstrapped)

By Ajibola Aiyedogbon

Server Monitoring(scaling while bootstrapped)

Page 2: Server Monitoring  (Scaling while bootstrapped)

About meCo-founder Amebo App

Mobile Developer (Jobberman, GTBank WP, etc)

DevOps enthusiast

Page 3: Server Monitoring  (Scaling while bootstrapped)

Before before...1 server for everything

-1 users

J2ME only

What throughput!

Cloudinary as CDN

Deployment fails

High costs, ignorance is very expensive

Page 4: Server Monitoring  (Scaling while bootstrapped)

Now5+ servers

Hundreds of thousands of users

Multi Platform apps

18000 req/min throughput

Cloudflare as CDN

Deployments with zero downtime

Managed costs

Page 5: Server Monitoring  (Scaling while bootstrapped)

Scaling rhymes with Failing!

Page 6: Server Monitoring  (Scaling while bootstrapped)

Server Stack

Page 7: Server Monitoring  (Scaling while bootstrapped)

Server Stack1 load balancer (layer 4, high availability, failover*)

3 web servers (vertically & horizontally scalable)

1 database server (replication*, redundancy*)

1 staging server

$65 monthly serving over 100 million requests

Cloudflare secret weapon, caches static requests (70%).

Page 8: Server Monitoring  (Scaling while bootstrapped)

Technology Stack

Page 9: Server Monitoring  (Scaling while bootstrapped)

Technology StackHaproxy (load balancer)

Nginx, Php-fpm (web server, php interpreter)

Phalcon, Php-Resque (framework, scheduler)

Redis, MongoDB, MariaDB (in-memory cache, datastores)

Git (BitBucket), Packer, Ansible (server provisioning, code provisioning)

SetCronJob, CloudFlare, Fastly (3rd party)

Page 10: Server Monitoring  (Scaling while bootstrapped)

Why Iaas not Paas?All about the pricing page!

Bandwidth costs too high

Code optimizations are hidden behind computing power

Mission critical? Offload to PaaS selectively, e.g. Parse EOL, death by acquisition...

Page 11: Server Monitoring  (Scaling while bootstrapped)

Why Monitor?

Page 12: Server Monitoring  (Scaling while bootstrapped)

Don’t end up like these guys...

Page 13: Server Monitoring  (Scaling while bootstrapped)

Why monitor?Get Visibility

Improve usability & stability

Complicated technology stacks with hard to trace errors

Mission critical

More sleep!

Page 14: Server Monitoring  (Scaling while bootstrapped)

What to monitor apart from everything?

Page 15: Server Monitoring  (Scaling while bootstrapped)

Server Metrics (infrastructure)Ram usage, spikes

Bandwidth usage, highs vs lows

CPU usage over time, peak usage

Disk I/O

Open source vs Saas

Free mostly

Page 16: Server Monitoring  (Scaling while bootstrapped)
Page 17: Server Monitoring  (Scaling while bootstrapped)
Page 18: Server Monitoring  (Scaling while bootstrapped)

Server Metrics (services)Haproxy stats

Nginx Stats

Mysql performance etc

Service *something* status

Page 19: Server Monitoring  (Scaling while bootstrapped)

Application ErrorsCatch all exception php

User defined errors

3rd party Library errors

Page 20: Server Monitoring  (Scaling while bootstrapped)

Tech Stack (Application Performance Monitoring)

Request throughput

Resource usage

Service Health

Database monitoring

Infrastructure bottlenecks

Failure Alerts

Code Errors

High level overview with deep dive

Page 21: Server Monitoring  (Scaling while bootstrapped)

Log Tracking

Better way to tail -f

Http stack errors & anomalies

Multiple log files from diff services

Manual tailing is difficult

Get pre configured graphs based on logs

All server traffic is logged, access_log

Page 22: Server Monitoring  (Scaling while bootstrapped)
Page 23: Server Monitoring  (Scaling while bootstrapped)
Page 24: Server Monitoring  (Scaling while bootstrapped)

Client Errors (Mobile)

Client side stack traces post deployment

Valuable version & device insight

Very handy at debug time & post

Catch all errors …. mostly

Memory leaks & stack traces

3rd party library errors or platform errors

Page 25: Server Monitoring  (Scaling while bootstrapped)
Page 26: Server Monitoring  (Scaling while bootstrapped)

Open Source vs Proprietary

Vendor lockin

Community support

DIY vs training

Industry standards & experience

Fault tolerance

Enterprise customer experience

Page 27: Server Monitoring  (Scaling while bootstrapped)

3rd Party vs Native monitoring toolsCore business?

Pricing again!

Support lifecycle and responsiveness

Product version, beta or 5.0?

Dashboard simplicity

Security implications? firewalled? https? localhost only? Install certs?

Page 28: Server Monitoring  (Scaling while bootstrapped)

Too many alerts….!

Page 29: Server Monitoring  (Scaling while bootstrapped)

What now?Congratulations, you reward is more work!

Customize alerts

Fix errors

Webhooks

Send to slack

Ignore at own risk

Page 30: Server Monitoring  (Scaling while bootstrapped)

Be like this guy….or not!

Page 31: Server Monitoring  (Scaling while bootstrapped)

Graphs on graphs on graphs on graphsInformation overload is real

Customize dashboard

Overviews only

Deep dive early to be familiar with dashboard

Page 32: Server Monitoring  (Scaling while bootstrapped)

What Next? Setup BugSnag

Page 33: Server Monitoring  (Scaling while bootstrapped)
Page 34: Server Monitoring  (Scaling while bootstrapped)
Page 35: Server Monitoring  (Scaling while bootstrapped)
Page 36: Server Monitoring  (Scaling while bootstrapped)

ConclusionWhy Monitor

What to Monitor

How to monitor

Pricing

Dashboards

Discuss your stack with peers

Page 37: Server Monitoring  (Scaling while bootstrapped)

Thank You@Ajibz