© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Monitoring with Icinga2Jordan Jethwa | Web Technologist, Online Experience Management

© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Thank you

Icinga Development, QA, and Support§ Commitment to Open Source values§ Technology§ Release schedules

Community§ Mailing list, IRC, and Monitoring Portal users

© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Online Experience Management

About us§ DevOps!§ Develop platform code that runs on Adobe Experience Manager (AEM)§ Support and maintain infrastructure§ Automation, monitoring, and metrics

Stakeholders§ Content authors across multiple worldwide business units§ Teams with integrations, etc

Priorities§ Accelerate content development activities§ Reduce technical and maintenance debt§ Provide a unified and consistent experience

3

© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Online Experience Management (continued)

Adobe@Adobe§ Showcase technologies from Adobe§ Collaborate with engineering teams

§ Innovate

Environments§ Complex

§ Landscapes (dev -> prod)§ Platforms (legacy vs new, intranet, etc)§ Differing SLAs

4

© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Ops day to day stack slice

5

© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Icinga2 Case StudyOnline Experience Management

6

© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Move to DevOps (~2 years ago)

An opportunity to evaluate, modernize, and experiment§ Methodology§ Technology stack

Scalability§ An ever growing infrastructure§ More hands in the cookie jar: Dev + Ops

§ More frequent code and infrastructure changes

Flexibility§ All the things

© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Why Icinga2?

A few other evaluations§ Nagios - NOPE!§ New Relic, Datadog – Cons: Hosted, Java Agent byte-code injection

§ Sensu – Cons: Set up and maintenance

Icinga2§ Object-based configuration format (macros, assignments, conditionals, and functions!)

§ Dependencies§ Metrics and built-in graphite export§ Icinga Web 2§ Compatible with Monitoring Plugins (https://www.monitoring-plugins.org/)§ Icinga Exchange (https://exchange.icinga.org/)

§ REST API – Manage objects, retrieve information, perform actions like rescheduling checks

© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Current Icinga2 stats

9

220+hosts

2100+ checks

5m intervals

~ 1sexecution time

1 server (failover)

© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Configuration Design

Focus on maintainability, flexibility, and automationStructure:

§ Tree-based§ One file per configuration object (mostly)

Groups:§ Not just for visualization!

Templates:§ Apply a set of generic attributes to multiple objects

Assignments – Match object attributes:§ HostGroups and ServiceGroups§ Custom variables

10

/etc/icinga2/conf.d/|—- api-users.conf|—- app.conf|—- commands/|—- downtimes/|—- groups/|—-—-—- hostGroups/|—-—-—- serviceGroups/|—- hosts/|—-—-—- platform0/|—-—-—-—-—- myhost.conf|—-—-—-—-—- myhost/|—-—-—-—-—-—-—- myhost_specific0.conf|—- notifications/|—- services/|—-—-—- ssh.conf|—-—-—- platform0_specific/|—-—-—-—-—-—-—- platform_specific0.conf|—- templates/|—-—-—- hosts/|—-—-—- notifications/|—- users.conf

© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Configuration Design (continued)

Keep it simple and modularObjects:

§ Rely on inheritance§ assign and ignore§ Human readable

Host object automation:

§ Current: Script generates from CMDB but, CMDB entry is manual L§ Target: Chef

11

+-----------------------------+

|object Host "example_host" { |

| import "rhel6-5" |

| import "aem6-1" |

| |

| display_name "Example Host"|

| address = "192.168.0.22" |

| |

| vars.landcape = "prod" |

| vars.location = "eu-west-1"|

|} |

+-----------------------------+

© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Guess the chart

12

0

10

20

30

40

50

60

70

80

90

100

© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Millions of notifications go unread everyday

13

0

10

20

30

40

50

60

70

80

90

100

Notification fatigue

Productivity Notifications

You can make a differenceReduce notifications:

§ Notification intervals based on SLAs§ Use dependencies!§ Scheduled downtimes§ Route using user groups

Notification alternatives/additions:§ Slack (https://github.com/jjethwa/icinga2-slack-

notification)§ PagerDuty

§ Push notifications (Pushover, etc)

© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Performance Data

14

It’s all about metricsEnable performance data writer:

§ No

Enable Graphite Carbon writer:§ Pa

Grafana:§ Visualization – See trends, correlate data§ Dashboards – Impress execs J

Bonus round:§ Icinga Web 2 graphite module

(https://github.com/findmypast/icingaweb2-module-graphite)

# icinga2 feature enable perfdata

# icinga2 feature enable graphite

© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Docker

Zero to Icinga2 in less than a minute§ x§ debian:jessie

Resources:§ Docker Hub: https://hub.docker.com/r/jordan/icinga2/§ GitHub: https://github.com/jjethwa/icinga2

Contributing a patch:§ Submit an issue describing your proposed change to the repo in question.§ I will respond to your issue promptly.§ Fork the desired repo, develop and test your code changes.§ Submit a pull request.

15

# docker pull jordan/icinga2

# sudo docker run -p 80:80 -t jordan/icinga2:latest

© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

Upcoming activities

§ Host object generation - Replace current CMDB script§ Plug n Play Icinga2 satellites - Gather metrics from different office networks

16