simply monitor a grid site with nagios

EGEE-II INFSO-RI-031688

Enabling Grids for E-sciencE

www.eu-egee.org

EGEE and gLite are registered trademarks

Simply monitor a grid site with NagiosJ. Casey, CERN

E. Imamagic, SRCE

ISGC 2008

ISGC 2008 / Simply monitor a grid site with Nagios 2



Overview

• Nagios• Nagios-based grid monitoring• Site monitoring prototype• Demo• Current status• Future work• Conclusions




Nagios

• Open source monitoring framework– widely used & actively developed

• Host and service problems detection and recovery• Provides wide set of basic sensors

– easy to develop custom sensors

• Centralized vs. distributed deployment• High configurability

– service dependencies, fine-grained notification options

• Web interface– status view, administration




Nagios-based Grid Monitoring

• Monitoring CRO-GRID Infrastructure (2004-2006)– Globus Toolkit Pre-WS & WS, UNICORE, other services– active recovery of services– http://www.cro-ngi.hr

• Monitoring EGEE resources in Central Europe (CE)– core services since mid 2006– all CE sites for 1st line support since September 2006– http://nagios.ce-egee.org

• Grid Services Monitoring (GSM) WG– site monitoring prototype, mid 2007– http://crnjak.srce.hr/nagios (egee.srce.hr)– https://pps-monitoring.cern.ch/nagios (CERN-PPS)




Site Monitoring Prototype

Nagios

…Site

nodes

Site BDII CE SE LFC

NCG

NPM

Remote gatherers

Standard probes

Credential refresh

MyProxy

VOMS

Long-lived MyProxy certificate

Nagios configuration

Refresh

proxy

Get VOMS proxy

Probe wrapper

VOMS proxy certificate

Service checks

Get remote results

Probe descriptions

SAM

…

Get site’s & nodes

information

Get nodes information

Live node checks

Publisher

External applications

Get Nagios results

Nagios web

interface

Site admins

Get site status

Issue alarms

Monitoring server




Grid Probes

• Provided by SRCE, CERN, OSG• Security facilities & services

– CA distribution, Certificate lifetime, MyProxy

• Monitoring & information services – R-GMA, BDII, MDS, GridICE

• Job management services – Globus Gatekeeper, RB, WMS, WMProxy, Job matching

• File management services – GridFTP, SRM, DPNS, LFC, FTS




Standard Components

• Specifications defined by GSM WG• Probe wrapper

– enables integration of standardized probes– Grid Monitoring Probes Specification– https://twiki.cern.ch/twiki/bin/view/LCG/

GridMonitoringProbeSpecification

• Publisher & remote gatherers– integration with other tools– Grid Monitoring Data Exchange Standard– https://twiki.cern.ch/twiki/bin/view/LCG/

GridMonitoringDataExchangeStandard




Nagios Config Generator

• Uses multiple information sources– SAM, BDII, active heuristic checks

• Modular approach– plugging in additional information sources– integration with other monitoring systems (e.g. LEMON)

• User-defined rules– configuration tuning for non-standard grid sites

• Standalone configuration– integration with existing Nagios server




Remote gLite UI

• Avoid installation of grid middleware on Nagios server– execute grid probes on existing gLite UI– use Nagios Remote Plugin Executor (NRPE)

…Site

nodes

Site BDII CE SE LFC

Nagios

Nagios server

Standard probes

gLite UI

NRPE

Service checks




SAM

Standard probes

NPM




Current Status

• Three sets of standard probes integrated– SRCE, CERN, OSG

• Two external monitoring systems– SAM, ENOC DownCollector

• Several deployments– CERN-PPS, SRCE, NIKHEF, PIC, IN2P3, ScotGrid

• RPMs in apt and yum repository• Installation and configuration manual• More info

https://twiki.cern.ch/twiki/bin/view/LCG/GridServiceMonitoringInfo




Future Work

• NCG development– providing configuration for multiple sites (regional monitoring)– providing configuration for multiple VOs

• Integration with global monitoring systems– ActiveMQ messaging system– Operations Automation Team mandate

• Enabling “on-host” check via NRPE– process, logs, ports, files, etc

• Probe description & site topology databases definition




Conclusions

• Nagios – highly configurable monitoring framework with notifications,

service dependencies, …– widely used by site admins

• Grid extensions– integration with existing infrastructure (user certificates, VOMS,

GOCDB, SAM)– probes for key grid services

• Implementation of GSM WG specifications– probe wrapper, publisher & remote gatherers– easy integration with existing probes and monitoring systems




Thank You!

Questions?

simply monitor a grid site with nagios

Documents