monitutor

Mar 8, 2016 1 / 26

Icinga Camp BerlinIcinga Camp BerlinMarch 1March 1stst 2016 2016

Icinga meets education: MoniTutorIcinga meets education: MoniTutorMark SwillusMark Swillus

Camp Berlin 2016Camp Berlin 2016 22Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor

Motivation

Creative use of Icinga

Technical details

Future of MoniTutor

Lessons Learned + Q&A

MoniTutor

Monitoring madness


FH-Aachen

Electrical Engineering and Information Technology

Prof. Andreas Claßen

IT-Infrastructure, Fault tolerant systems


Applying concepts in virtual environment

Hands-On lessons

Complex problems, scalability, no overview, no state information

Lab environment

ITSclient

ITSserver

X.500LDAP

X.500LDAP

Online resources

Script

Lecture

Scenario

Hypervisor

Student


Hands-On lessonsScenario life cycle

Basic Setup Server configuration Client config.

✔ Iface.-config

✔ Internet access

✔ Client reaches Server

✔ slapd installed

✔ slapd running

✔ Port 389 listening

✔ ldapsearch xyz successful

Mile

sto

ne

1

Mile

sto

ne

2

Mile

sto

ne

3

✔ libdnss-ldap installed

✔ pam configured

✔ nsswitch configured

✔ Remote-login successful

Service-check


Hands-On lessons

Lab environment

ITSclient

ITSserver

ITSclient

ITSserver

Lab environment Lab environment

ITSclient

ITSserver

State information

Central overview

Automated checks

Remote access

Data repository

Interactive help

NAT

student1 student2 student3


MoniTutor

Student Monitoring in workshop environments

Multiplatform

Supports arbitrary, self-defined scenarios

Penetrates firewalls and NAT

Completely free and open source


Use of Icinga

Command pipe

(icinga.cmd)

Templates

(inheritance)

Database

(DBIDO)

Custom vars


Icinga & Web2py

Define structure(Hosts, Milestones, Checks)

Program parameters


hosts,services

Icinga & Web2py


Deploy


Icinga & Web2py


Deploy

Monitoring

Lab environment

Execute Check2

Send result to Icinga3

Deploy Check Program1

Prg1

MoniTutor DB

student1

Prg1


Icinga & Web2py


Deployment

Monitoring

Display progress

WebGui / student login1

Extract status2

Reconstruct Structure3

Display4

servicestatus hoststatus


Deployment

Scenario / workshop

Milestone1 Milestone

2Check1: Host: “itsserver”

Check2: Host: “itsclient”

Check3: Host: “itsserver”

...

...

Templates

(inheritance)


Deployment

Templates

(inheritance)

Icinga Documentation:

Apply For and Custom Attribute Override

Defining dictionaries.

object Host "cisco-catalyst-6509-34" { import "generic-host" display_name = "Catalyst 6509 #34 VIE21" address = "127.0.1.4" /* "GigabitEthernet0/2" is the interface name, * and key name in service apply for later on */ vars.interfaces["GigabitEthernet0/2"] = { /* define all custom attributes with the * same name required for command parameters/arguments * in service apply (look into your CheckCommand definition) */ iftraffic_units = "g" iftraffic_community = IftrafficSnmpCommunity iftraffic_bandwidth = 1 vlan = "internal" qos = "disabled" } vars.interfaces["GigabitEthernet0/4"] = { iftraffic_units = "g" //iftraffic_community = IftrafficSnmpCommunity iftraffic_bandwidth = 1 vlan = "renote" qos = "enabled" } vars.interfaces["MgmtInterface1"] = { iftraffic_community = IftrafficSnmpCommunity vlan = "mgmt" interface_address = "127.99.0.100" #special management ip }}

apply Service "if-" for (interface_name => interface_config in host.vars.interfaces) { import "generic-service" check_command = "iftraffic" display_name = "IF-" + interface_name /* use the key as command argument (no duplication of values in host.vars.interfaces) */ vars.iftraffic_interface = interface_name vars.iftraffic_units = interface_config.iftraffic_units vars.iftraffic_community = interface_config.iftraffic_community vars += interface_config if (interface_config.iftraffic_bandwidth == "") { vars.iftraffic_bandwidth = 1 } if (interface_config.vlan == "") { vars.vlan = "not set" } if (len(interface_config.iftraffic_community) == 0 || len(vars.iftraffic_community) == 0) { vars.iftraffic_community = IftrafficSnmpCommunity } notes = "Interface check for " + interface_name + " (units: '" + \\

interface_config.iftraffic_units + "') in VLAN '" + vars.vlan + \\ "' with ' QoS '" + vars.qos + "'"

notes_url = "http://foreman.company.com/hosts/" + host.name action_url = "http://snmp.checker.company.com/" + host.name + "/if-" + interface_name}


Deployment

Templates

(inheritance)

Icinga Documnetation:

Apply For and Custom Attribute Override

Defining dictionaries.

///Created by monitutor

template Service "Install_Configure_LDAP_Server" {

vars.scenario_milestones["ldap01"] = {

display_name = "LDAP Hands-On Lab"

hidden = "false"

milestones["Install_Configure_LDAP_Server"] = {

name = "Install_Configure_LDAP_Server"

display_name = "Install & Configure LDAP Server"

description = "Basic LDAP Server install & configure steps \\

that students have to do.<br>"

order = "0"

hidden = false

}

}

}


Deployment

Templates

(inheritance)paul2

paul2_itsserver monitutor-system

+ basic host configuration

itsserver

+ itsserver specific configuration

Monitutor-check

+ basic service configuration

paul2_Check_ldap_utils_

Package_Installed

host

service

Check_ldap_utils_Package_Installed

+ check_ldap… specific configuration

Install_Configure_LDAP_Server

+ Milestone Install… specific configuration Dictionary


Deployment


Monitoring

Command pipe

(icinga.cmd)

MoniTutor DB

Daemon Init

paul2connect

Username, hostname

QueueHandlerClientThread

Read

Write

{“user-host”:Priority_Queue()} Send check

Interpreter, program, params,

servicename

/tmp/monitutor

Checkprogram?Request code

program name

Server Client


Monitoring

Command pipe

(icinga.cmd)

MoniTutor DB

Daemon Init

paul2

QueueHandlerClientThread

Read

Write

{“user-host”:Priority_Queue()} Send code

Interpreter, program, params,

servicename, code

/tmp/monitutor

Check

Server Client

Execute

Send Result

servicename, output,

exit code….

ResultHandler

standard libraries only

DBIDO

Request code

program name


Display


DisplaySELECT varname, varvalue FROM icinga_customvariables WHERE varname = 'scenario_milestones' LIMIT 3;

varname | varvalue ---------------------+------------------------------------------------------------------------------------------------------------------------ scenario_milestones | {"ldap01":{"display_name":"LDAP Hands-On Lab","hidden":"false","milestones":{"Install_Configure_LDAP_Server":{"descr... scenario_milestones | {"ldap01":{"display_name":"LDAP Hands-On Lab","hidden":"false","milestones":{"Install_Configure_LDAP_Server":{"descr... scenario_milestones | {"ldap01":{"display_name":"LDAP Hands-On Lab","hidden":"false","milestones":{"Install_Configure_LDAP_Server":{"descr...(3 rows)

milestone_queue={}for milestone in milestones: if milestone.get("name") not in milestone_queue: milestone_queue[milestone.get("name")] = {"order": milestone.get("order"), "display_name": milestone.get("display_name"), "description": milestone.get("description"), "hidden": milestone.get("hidden"), "services": []}

milestone_queue[milestone.get("name")]["services"].append(service[0].service_id)

return dict(milestones=sorted(milestone_queue.items(), key=lambda x: x[1]["order"]))


Future of MoniTutor

Further development

One semester testing (fault tolerant systems)

Release in summer (Free & OpenSource)

Other environments

Online-Hands-On lessons (?)


Lessons learned


Mark SwillusStudent FH [email protected]: 40A5 A4F1 D548 6054 4D7F CE08 F22C 2E99 B2FF 4A4C

Prof. Dr. Andreas ClaßenFH Aachen

[email protected] Tel.: +49.241.6009 52128