zenoss core4 event management paper
DESCRIPTION
Zenoss Event ManagementTRANSCRIPT
-
Event Management for Zenoss Core 4
January 2013
Jane Curry
Skills 1st Ltd
www.skills-1st.co.uk
JaneCurrySkills1stLtd2CedarChaseTaplowMaidenheadSL60EU01628782565
www.skills1st.co.uk
-
SynopsisThispaperisintendedasanintermediateleveldiscussionoftheZenosseventsysteminZenossCore4.TheeventarchitecturehaschangeddramaticallyinZenoss4frompreviousversions.
ItisassumedthatthereaderisalreadyfamiliarwiththeZenossEventConsoleandwithbasicnavigationaroundtheZenossGraphicalUserInterface(GUI).ItlooksinsomedetailatthearchitecturebehindtheZenosseventsystemthedaemonsandhowtheyareinterrelatedanditlooksatthestructureofaZenosseventandtheeventlifecycle.
ZenosscanreceiveeventsfrommanysourcesinadditiontoZenossitself.EventsfromWindows,UnixsyslogsandSimpleNetworksManagementProtocol(SNMP)TRAPsareallexaminedindetail.
TheprocessbywhichanincomingeventisconvertedintoaparticularZenosseventisknownaseventmappingandthereareanumberofdifferentpossibletechniquesforperformingthatconversion.Thesewillallbeexploredalongwiththecreationofneweventclasses.
Onceaneventhasbeenreceived,classifiedandstoredbyZenoss,automationmayberequired.Alertingtousersbyemailandpageisdiscussed,asarebackgroundactionstoruncommandsorgenerateTRAPs.
LogginganddebuggingtechniquesarediscussedinsomedetailsasistheJSONAPIforextractingdataoutofZenoss.
ThispaperwaswrittenusingZenossCore4.2.3
ThepaperisacompaniontexttotheZenoss4EventManagementWorkshop.
NotationsThroughoutthispaper,texttobytyped,filenamesandmenuoptionstobeselected,arehighlightedbyitalics;importantpointstotakenoteofareshowninbold.
Pointsofparticularnotearehighlightedbyanicon.
2 EventManagementforZenossCore4Skills1stLtd 23January2013
-
Table of Contents1Introduction..........................................................................................................................62Zenosseventarchitecture....................................................................................................6
2.1EventConsole...............................................................................................................62.2EventManagersettings.............................................................................................102.3Eventdatabasetables...............................................................................................11
2.3.1Zenoss2.xand3.x...............................................................................................112.3.2Zenoss4................................................................................................................14
2.4Neweventdaemons....................................................................................................202.4.1RabbitMQ.............................................................................................................202.4.2zeneventserver.....................................................................................................222.4.3zeneventd.............................................................................................................222.4.4zenactiond...........................................................................................................232.4.5memcached...........................................................................................................23
2.5OtherdatabaserelatedchangesinZenoss4............................................................242.6Eventlifecycle............................................................................................................25
2.6.1Eventgeneration.................................................................................................272.6.2Applicationofdevicecontext..............................................................................292.6.3Eventclassmapping...........................................................................................292.6.4Applicationofeventcontext...............................................................................302.6.5Eventtransforms.................................................................................................302.6.6Databaseinsertionsanddeduplication............................................................312.6.7Resolution............................................................................................................322.6.8Ageingandarchiving..........................................................................................34
3EventsgeneratedbyZenoss..............................................................................................343.1zenping........................................................................................................................353.2zenstatus.....................................................................................................................363.3zenprocess...................................................................................................................363.4zenwin.........................................................................................................................373.5zenwinperf...................................................................................................................373.6zenperfsnmp................................................................................................................373.7zencommand...............................................................................................................38
4Syslogevents......................................................................................................................384.1Configuringsyslog.conf.............................................................................................394.2Zenossprocessingofsyslogmessages.......................................................................40
5ZenossprocessingofWindowseventlogs.........................................................................485.1ManagementusingtheWMIprotocol.......................................................................485.2ManagementofWindowssystemsusingsyslog.......................................................51
6EventMapping...................................................................................................................516.1Workingwitheventclassesandeventmappings....................................................52
6.1.1Generatingtestevents........................................................................................546.2Regexineventmappings...........................................................................................55
23January2013 EventManagementforZenossCore4Skills1stLtd 3
-
6.3Rulesineventmappings............................................................................................576.4Otherelementsofeventmappings...........................................................................58
7Eventtransforms...............................................................................................................587.1Differentwaystoapplytransforms...........................................................................597.2Understandingfieldsavailableforeventprocessing...............................................60
7.2.1EventProxies.......................................................................................................637.2.2EventDetails.......................................................................................................66
7.3Transformexamples...................................................................................................687.3.1CombininguserdefinedfieldsfromRegexwithtransform.............................687.3.2Applyingeventanddevicecontextinrelationtotransforms..........................69
8Testinganddebuggingaids..............................................................................................718.1Logfiles.......................................................................................................................71
8.1.1zeneventd.log.......................................................................................................718.1.2zeneventserver.log...............................................................................................728.1.3Otherlogfiles......................................................................................................75
8.2UsingzendmdtorunPythoncommands..................................................................758.2.1ReferencinganexistingZenosseventforuseinzendmd.................................758.2.2UsingzendmdtounderstandattributesforanEventSummaryProxy...........79
8.3UsingthePythondebuggerintransforms................................................................839ZenossandSNMP..............................................................................................................87
9.1SNMPintroduction.....................................................................................................879.2SNMPonLinuxsystems............................................................................................889.3ZenossSNMParchitecture........................................................................................91
9.3.1Thezentrapdaemon............................................................................................919.4InterpretingMIBs......................................................................................................93
9.4.1zenmibexample...................................................................................................949.4.2AfewcommentsonimportingMIBswithZenoss.............................................99
9.5TheMIBBrowserZenPack......................................................................................1009.5.1ModifyingZenossCore4.2tomaketheMIBBrowserZenPackwork..........102
9.6MappingSNMPevents............................................................................................1039.6.1SNMPeventmappingexample........................................................................103
10EventTriggersandNotifications.................................................................................10810.1ZenosspriortoV4...................................................................................................10810.2Zenoss4architecture.............................................................................................10910.3Triggers...................................................................................................................11010.4Notifications............................................................................................................111
10.4.1emailNotifications..........................................................................................11310.4.2PageNotifications...........................................................................................11810.4.3CommandNotifications..................................................................................11810.4.4TRAPNotifications.........................................................................................120
10.5NotificationSchedules............................................................................................12210.6Usingzenactiond.log..............................................................................................12310.7TheeffectofdeviceProductionState....................................................................125
11AccessingeventswiththeJSONAPI...........................................................................126
4 EventManagementforZenossCore4Skills1stLtd 23January2013
-
11.1Definitions...............................................................................................................12611.2UnderstandingtheJSONAPI...............................................................................12711.3UsingtheJSONAPI..............................................................................................130
11.3.1Bashexamples.................................................................................................13011.3.2Pythonexamples.............................................................................................134
12Conclusions.....................................................................................................................13913AppendixA.....................................................................................................................143
13.1getevents.py............................................................................................................14313.2zensendevent..........................................................................................................148
14References.......................................................................................................................152
23January2013 EventManagementforZenossCore4Skills1stLtd 5
-
1 IntroductionZenossisanOpenSource,multifunctionsystemsandnetworkmanagementtool.Thereisafree,Coreoffering(whichhasmostthingsyouneed),andachargeableoffering,ZenossResourceManager,whichhasextraaddongoodiessuchashighavailabilityconfigurations,distributedmanagementservers,servicemanagementandeventcorrelation;italsoincludesasupportcontract.
Zenossoffersconfigurationdiscovery,includinglayer3topologymaps,availabilitymonitoring,problemmanagementandperformancemanagement.ItisdesignedaroundtheITILconceptofaConfigurationManagementDatabase(CMDB),theZenossStandardModel.ZenossisbuiltusingthePythonbasedZopewebapplicationserverandusestheobjectorientedZopeObjectDatabase(ZODB)astheCMDB,usedtostorePythonobjectsandtheirstates.Zenoss3usedZEO,asalayerbetweenZopeandtheZODB;inZenoss4theZODBdataisstoredinaMySQLdatabase.
TherelationalMySQLdatabaseisalsousedtoholdcurrentandhistoricalevents.PerformancedataisheldinRoundRobinDatabase(RRD)files.
ThedefaultprotocolsformonitoringaretypicallyagentlesstheSimpleNetworkManagementprotocol(SNMP),WindowsManagementInstrumentation(WMI)andcollectingeventsfromsyslogs.Itisalsopossibletomonitordevicesusingtelnet,sshandtouseNagiosplugins.
Zenossprovidesdocumentationathttp://community.zenoss.org/community/documentation.ThereisalsoawealthofinformationontheZenosswebsiteinvariousforums,FAQs,andtheWiki.AusefulbookisavailablefromPACKTPublishing,ZenossCore3.xNetworkandSystemMonitoringbyMichaelBadger,whichprovidesmuchofthesameinformationastheZenossAdministrationGuidebutinamuchclearerformatwithplentyofscreenshots.AlthoughthisisaZenoss3text,itstillprovidesgoodbasicinformation.
ThispaperisanattempttoexpandontheeventinformationintheZenossCore4AdministrationGuidebydrawingonmyownexperienceandthecollectedwisdomofseveralZenossemployeesandcontributorsfromthecommunity.
2 Zenoss event architecture2.1 Event ConsoleWhenaneventarrivesatZenoss,itisparsed,associatedwithaneventclassificationandthentypically(butnotalways),itisinsertedintotheevent_summarytableofthezenoss_zepdatabase.EventscanthenbeviewedbyusersusingtheEventConsoleoftheZenossGraphicalUserInterface(GUI).
6 EventManagementforZenossCore4Skills1stLtd 23January2013
-
ThereareanumberwaystoaccesstheEventConsole.ThemainEventConsoleisreachedfromthetopEVENTS>EventConsolemenu.ThedefaultistoshoweventswithaseverityofInfoorhigher,sortedfirstbyseverityandthenbytime(mostrecentfirst).Eventsareassigneddifferentseverities:
Name Number Colour
Critical 5 Red
Error 4 Orange
Warning 3 Yellow
Info 2 Blue
Debug 1 Grey
Cleared 0 Green
AlleventsalsohaveaneventStatefield.Zenoss3eventStatehadthreepossiblevaluesNew,AcknowledgedandSuppressed.Zenoss4hasenhancedthesedefinitionssowenowhave:
Name Number Description
New 0 Neweventnoprevioussimilarevent
Acknowledged 1 Acknowledgedbyuserorrule
Suppressed 2 Typicallyfrombeyondasinglepointoffailure
Closed 3 Closedbyauser
Cleared 4 Closedbyarule
Dropped 5 Discardednotsavedinthedatabase
Aged 6 Autoclosedduetoage/severity
NotethatClosed,ClearedandAgedeventsallhavethesamestatusiconintheEventConsole.
Bydefault,NewandAcknowledgedeventsareshownintheEventConsole.AnyeventwhichhasbeenAcknowledgedhasatickinitsstatuscolumn.ASuppressedeventisnotshownbydefaultbutcanbefilteredinifdesired;ithasasnowflakeicon.Zenossbuildsaninternaltopologyofthenetworkitismanaging(usingnmap).Ifaneventisreceivedforadevicethatthetopologymapknowsisunreachable,theeventisautomaticallysuppressed.ThusZenosshasabuiltinmechanismforpinpointingfailuredevicesandsuppressingthefloodofeventsfrombehindsuchfailurepoints.
Eventscanbesortedbyclickingonadesiredcolumnheader;clickingagainsortsinthereverseorder.Tochangetheorderofcolumns,simplydragacolumnheader.
23January2013 EventManagementforZenossCore4Skills1stLtd 7
-
Thereisafilterboxaboveeachcolumnheadertohelpselectrelevantevents.Mostfiltersareamatchforapartialtextstring(youdon'tneedtosupplywildcards).Datefieldsprovideacalendaricontoselectanearliestdate.Thecountfieldpermitsyoutoenterarange,forexampletoshoweventswithcount>10,use10:(ifyoutypesomethingillegalinthecountfilteritwillsupplyhelpfortherequiredsyntax).
Toselectfieldstodisplay,hoverthemouseattheendofaheadertoseethedownarrowforsorting;thethirdoptiononthedropdownmenuistoconfigurethefieldstodisplay.
FromtheEventConsole,oneormoreeventscanbeselectedbyclickingonthelinebecarefulnottoclicksomethingthatisalink(likethedevicenameoreventclass).TheiconsatthetopleftcanbeusedtoAcknowledge,Close,MaptoanEventClass,UnacknowledgeorReOpen.The+iconattheendofthisrowoficonscanbeusedtogeneratetestevents.
Doubleclickaneventtoshowthedetailsofanevent.Thisshowsbothstandardfieldsandanyuserdefinedfieldsorganisedunderseveralgroupingswhichcanbeexpandedandcontracted.AnyAcknowledge,CloseorReOpenwillbeshownatthebottom,includingwhoperformedtheaction.Freeformnotescanalsobeloggedhere.
8 EventManagementforZenossCore4Skills1stLtd 23January2013
Figure1:ZenossEventConsole
-
Thesummaryandmessagefieldsarefreeformtextfields.Thesummaryfieldallowsupto255characters;themessagefieldallowsupto4096characters.Thesefieldsusuallycontainsimilardata.Fordetailsofotherfields,seesection7.1.2oftheZenossCore4Administrationguide.
Bydefault,theEventConsoleisrefreshedeveryminute.ThedropdownbesidetheRefreshbuttonallowsyoutochangetheintervalortorefreshmanually.
23January2013 EventManagementforZenossCore4Skills1stLtd 9
Figure2:EventdetailsshowingAcknowledgementandaddednote
-
EventConsolesarealsoavailableatvariousplacesintheGUIwhichhavefiltersalreadyapplied:
Fromadevice'sdetailpage,selectEventsinthelefthandmenu
Foradeviceclass,clicktheDETAILSlinkandthenEventsinthelefthandmenu
ForaLocation,GrouporSystem,clicktheDETAILSlinkandthenEventsinthelefthandmenu
FromanEventClass,selectEventsinthelefthandmenu
PriortoV4,ZenosseventswereeitherOpenorClosed.OpeneventswerestoredintheMySQLeventsdatabaseinthestatustable.Whenaneventwasclosed,itwasmovedtothehistorytableoftheeventsdatabase.
WithZenoss4thereisasignificantchange.TheMySQLdatabaseforeventsiscalledzenoss_zepandithasfarmoretables,includingevent_summaryandevent_archive.Openeventswillbestoredintheevents_summarytable.Beawarethattheevents_summarytablewillalsoholdclosed,clearedandagedeventsthiscatchesoutmanypeoplemigratingfromolderversionsofZenosstoZenoss4.ChecktheStatusfilterintheEventConsoletoshowClosed,ClearedandAgedevents(theyallhavethesamestatusicon).Closed,ClearedandAgedeventsmaybeautomaticallymovedtotheevent_archivetablebasedonage(after3days,bydefault).
2.2 Event Manager settingsFromtheADVANCED>Settingsmenu,chooseEventsinthelefthandmenutosetupvariousparametersthatcontroltheeventssubsystem,includinghoweventsareagedandfinallypurged.
Figure3onpage11showslargelydefaultsettings.EventsofseverityWarningandbelowwillbeAgedafter240minutes(4hours).After4320minutes(3days)eventswithstatusofClosed,ClearedorAgedwillbeArchived(movedtotheevents_archivetable).After7daysArchivedeventswillbedeletedentirely(notethislastsettingis90daysbydefaultandcanresultinaverylargedatabase).
Seechapter7oftheZenossCore4AdministratorsGuideformoreinformation.
10 EventManagementforZenossCore4Skills1stLtd 23January2013
-
2.3 Event database tables 2.3.1 Zenoss 2.x and 3.xTheeventsarchitecturewasthesameforversions2and3andwasrelativelysimple.Eventsweregeneratedfromsomewhere.ThezenhubdaemonprocessedthemandusuallythensavedthemeitherinthestatustableoftheMySQLeventsdatabaseorcouldsendthemtothehistorytable.
ThedatabasefieldsofthestatusandhistorytablesmatchedthedetailsseeninanEventConsoleandifyouwroterulesandtransformstoprocessevents,theywerebasedonthesesamefieldnames.
TheeventsdatabaseiscreatedautomaticallywhenZenossisinstalledandcantypicallybeaccessedbythezenossuserwithapasswordofzenossseeFigure4.
23January2013 EventManagementforZenossCore4Skills1stLtd 11
Figure3:EventManagerparametersforageingandarchiving
-
TheformatofeachofthesetablesandthevalidfieldsforaZenosseventcanbeseenbyexaminingtheZenossdatabasesetupfilein$ZENHOME/Products/ZenEvents/db/zenevents.sql,where$ZENHOMEwillbe/opt/zenossforaCore4.2ZenossonRedHat/CentOS(theonlycurrentlysupportedplatform).
12 EventManagementforZenossCore4Skills1stLtd 23January2013
Figure4:ZenosseventsdatabasepriortoZenoss4
-
zenevents.sqlalsodefinesthehistorytableinasimilarfashion.
Afurtherfourtablesaredefinedforheartbeat,alert_state,loganddetail.ThedetailtablecanbeusedtoextendthedefaulteventfieldstoincludeanyinformationthattheZenossadministratorrequiresforanevent.
23January2013 EventManagementforZenossCore4Skills1stLtd 13
Figure5:Definitionofstatuseventfieldsinzenevents.sqlpriortoZenoss4
-
IfyouareusingZenosspriortoversion4,gettheolderversionofthisZenossEventManagementpaperfromhttp://www.skills1st.co.uk/papers/jane/zenoss_event_management_paper.pdf.
2.3.2 Zenoss 4WithZenoss4eventsarestillheldinaMySQLdatabasewhichisnowcalledzenoss_zepanditiscreatedwhenZenossisinstalled.Aswithearlierversions,thezenossusercanaccessthisdatabasewithapasswordofzenoss.
NotethatwithZenoss4.2.3,ifinstalledwiththecoreautodeployscript,thenthepasswordfortheMySQLzenossuserischangedtoarobust,randompasswordthatisthensavedin$ZENHOME/etc/global.conf.Permissionsfor$ZENHOME/etcanditscontentsareallsettofullaccessforthezenossuserandnoaccessforanyoneelse.
14 EventManagementforZenossCore4Skills1stLtd 23January2013
Figure6:zenevents.sqlshowingheartbeat,alert_state,loganddetailtableszenoss2and3only
-
Inpassing,notethatinadditiontothezenoss_zepdatabase,theirisalsoazodbandazodb_sessiondatabase.TheZopedatabase(ZODB)thatstoresalltheobjects(devices,deviceclasses,processes,networks,etc)isnowinMySQL.
Examiningthetablesofthezenoss_zepdatabaseiswherethingsdivergesignificantlyfrompreviousversions.
23January2013 EventManagementforZenossCore4Skills1stLtd 15
Figure7:AccessingMySQLdatabaseswithZenoss4
-
Themaintablesarenowevent_summaryandevent_archivebutthestructureismorecomplicated.Someofthedataisheldinseparatetableswithpointerstothemfromthemaintables.Theseinclude:
agent event_class event_class_key event_group event_key monitor
16 EventManagementforZenossCore4Skills1stLtd 23January2013
Figure8:TablesintheZenoss4zenoss_zepdatabase
-
Thedetailsoftheevent_summarytableisshownbelow.Theeventarchivetableisverysimilarwithjustthetwofingerprint_hashfieldsomitted.
23January2013 EventManagementforZenossCore4Skills1stLtd 17
Figure9:Fieldsintheevent_summarytableinZenoss4
-
TheeagleeyedwillalsospotthatsomeofthefieldnameshavechangedfromthoseinFigure5.eventClassintheoldversionbecomesevent_classinV4;firstTimeinFigure5becomesfirst_seeninthelaterversionandthereareanumberofothersimilar,subtlechanges.
Asmentionedabove,someofthedataisheldinseparatetablessoagent_id,event_class_id,event_class_key_id,event_group_id,event_key_idandmonitor_keyarelinkstoseparatetableswiththecorrespondingdata.
Somedatahaschangedfairlysubtly:
Old New
evid uuid
eventState status_id
eventClassMapping event_class_mapping_uuid
severity severity_id
stateChange status_change
firstTime first_seen
lastTime last_seen
count event_count
facility syslog_facility
priority syslog_priority
ntevid nt_event_code
ownerid current_user_uuid/current_user_name
clearid clear_fingerprint_hash/cleared_by_event_uuid
Allreferencestothedevicehavechangedsignificantly.deviceisreplacedbythefourfields,element_uuid,element_type_id,elementidentifierandelement_titlewhilstthecomponentfieldisreplacedbyelement_sub_uuid,element_sub_type_id,element_sub_identifierandelement_sub_title.
dedupidhasbecomefingerprintandfingerprint_hash.
OtherfieldswithdevicecontextsuchasprodState,DeviceClass,Location,Systems,DeviceGroups,ipAddress,monitorandDevicePrioritywillnowbefoundfromthetags_jsonfield;theyarealsoavailableintheeventdetails.
PriortoZenoss4therewasaseparatelogtablewhoseroleisnowtakenbythenotes_jsonfieldoftheevent_summarytable.
Eventdetailsratherthanbeinginaseparatetable,arenowreachedfromdetails_json.
update_timehasbeenaddedthelasttimeaneventwasupdated.
18 EventManagementforZenossCore4Skills1stLtd 23January2013
-
suppid(whichwasneverused)hasdisappearedintheZenoss4schema.managerhasalsodisappearedfromZenoss4.
Thesetablesarecreatedbythefilesin$ZENHOME/share/zeneventserver/sql/mysql.
Someoftheseeventfieldsareparticularlypertinentdependingonhowtheeventwasgenerated:
Syslogeventspopulatethefacilityandpriorityfields
Windowseventspopulatethentevidfield
SNMPTRAPspopulateatleastcommunityandoidfieldsintheeventdetail.TheyalsousetheeventdetailtoprovideanyvariablespassedbyanSNMPTRAP.
TheagentfielddenoteswhichZenossdaemongeneratedorprocessedtheincomingevent;forexample,zentrap,zeneventlog,zenping.
23January2013 EventManagementforZenossCore4Skills1stLtd 19
Figure10:Partofthe001.sqlfilethatdefinesMySQLtablesinthezenoss_zepdatabaseforZenoss4
-
FundamentallyZenossadministratorsshouldnotbeaccessingthezenoss_zepdatabasedirectly.Zenosshaveprovidedaninternaleventmappingsothat,largely,administratorscancontinuetousethesameeventattributenamesashavebeenusedpreviously.Thiseventproxymappingwillbediscussedinmoredetaillater.Ingeneral,thispaperwillusetheoldnamesunlessexplicitlystatedotherwise.
Ifyoudoneedtoaccesseventdatainthedatabasetables,perhapsforreportingonevents,itispossiblewiththeJSONAPI(alsomoreonthislater).
2.4 New event daemonsPriortoZenoss4mostoftheworkofprocessinganeventwasperformedbythezenhubdaemonwhichalsohaslotsofotherrolestofulfil.Eventprocessingcouldbecomeaseverebottleneck.Zenoss4hasintroducedseveralnewsubsystemsanddaemonstodramaticallyimprovethethroughputofeventprocessing.
2.4.1 RabbitMQAMessageQueueingarchitecturehasbeenimplementedtospeedupprocessingandtoofferanAPIsothatZenossandotherapplicationproviderscaninteractwithevents.ItisalsousedbythenewJobarchitecture.ItusestheAdvancedMessageQueueingProtocol(AMQP)standard,andtheopensourceRabbitMQimplementationinparticular,fortheeventpipeline.
WhenZenossisinstalledtheRabbitMQsubsystemisalsoinstalledandconfiguredwithavhostofzenoss,userzenoss,passwordzenoss.TherabbitmqctlutilitycanprovideinformationaboutthestateoftheMQenvironment;notethatrabbitmqctlcommandsmustberunbytherootuser.
Aneasywaytoseequeuesbuildingupistotemporarilystopzeneventdandtheraweventsqueuewillthenbuildrapidly.
20 EventManagementforZenossCore4Skills1stLtd 23January2013
Figure11:Usingtherabbitmqctlutilitytoshowqueuesforthe/zenossvhost
-
rabbitmqctlonitsownorwithinsufficientargumentsprovidestheusagehelp.rabbitmqctlreportgivesagoodoverallviewofthesubsystem.
IftheZenossserverisrenamedthenyoumustclearandrebuildqueuesbeforethezenhubandzenjobsdaemonswillrestart.Toresolvethis,issuethefollowingcommandsastherootuser(althoughanydataqueuedatrestarttimewillbelost):export VHOST="/zenoss"export USER="zenoss"export PASS="zenoss"rabbitmqctl stop_apprabbitmqctl resetrabbitmqctl start_apprabbitmqctl add_vhost "$VHOST"rabbitmqctl add_user "$USER" "$PASS"rabbitmqctl set_permissions -p "$VHOST" "$USER" '.*' '.*' '.*'
Seesection14.8oftheZenossCore4AdministratorsGuideforthisinformation.
NotethatwithZenossCore4.2.3installedusingtheautodeployscript,orifthesecure_zenoss.shscripthasbeenrunstandalone,thenthepasswordinthethirdlineabovewillhavebeenchanged.Examine$ZENHOME/etc/global.conffortheamqppasswordandsubstituethatvalue,ratherthanusingzenossasthepassword.
ProvidedtheRabbitMQsubsystemisrunning,anymissingqueuewillautomaticallyberecreatedwhenZenossisrestarted.
Tosimplyhavethequeuesrecreated,startasthezenossuser:zenossstopsu(tobecomerootuser)rabbitmqctldelete_vhost/zenossrabbitmqctladd_vhost/zenossrabbitmqctladd_userzenosszenoss#mightcreateanerrorzenossrabbitmqctlset_permissionsp/zenosszenoss'.*''.*''.*'rabbitmqctllist_vhosts (shouldhavezenossagain)rabbitmqctlp/zenosslist_queues(shouldbenone)exit (backtozenossuser)zenossstartsurabbitmqctlp/zenosslist_queues(shouldbeseveral)
Thereisafurtherscriptavailableatgist,writtenbycluther,toresetRabbitMQhttps://gist.github.com/4192854.
TwoutilitiesareavailableforthezenossusertogetRabbitMQinformation:zenqdump
dumpstheeventsinaqueue,convertingthebinaryblobs(whichishowtheeventsareactuallystored)intohumanreadabletext.
Notethatthezenqdumputilityhasparametersforuserandpasswordforauthentication,thatdefaulttozenoss/zenoss(youcanfindthiscodein$ZENHOME/lib/python/zenoss/protocols/amqpconfig.py).InZenoss4.2.3,passwordsarelikelytohavebeenimprovedoninstallationsothesimplecommandshownabove
23January2013 EventManagementforZenossCore4Skills1stLtd 21
-
willfail.Examine$ZENHOME/etc/global.conffortheparametersamqpuserandamqppasswordandsupplythosevalues.Forexample:
zenqdumpuzenosspuy+680bEubHgdPow8Tfhzenoss.queues.zep.rawevents
Thezenqutilityhasthreedifferentoptionstomanageaqueue:zenq count zenq purge zenq delete
Thecountparametergivesacontinualoutputoftimestampandqueuelength.
Thepurgeparameterpurgeseventsfromaqueue.ThiscommandissafewhenZenossisrunning.
ThedeleteparameterdeletesthequeueandshouldnotbeusedwhenZenossisrunning.
zenqdoesnothaveauthenticationparameters.
2.4.2 zeneventserverAnewJavadaemon,zeneventserver(alsoknownaszep),hasbeencreated.Itsroleistopresenteventstotheuserinterfaceandotherclients,andtomanagetheflowofdatabetweentheRabbitMQqueuesandtheMySQLdatabase.DataispresentedtoclientsviaJSONcalls.
2.4.3 zeneventdzeneventdisanewPythondaemonwhoseresponsibilityistotakedatafromtheincomingraweventqueue,classifyit(iftheeventdoesnotalreadyhaveaclass),adddevicecontextandeventcontext,andperformanytransforms.ItthenoutputstothezeneventsqueuesothatthezeneventserverdaemoncanmanageitsprogresstotheMySQLdatabase,totheuserinterfaceandforalertingaction.
22 EventManagementforZenossCore4Skills1stLtd 23January2013
-
2.4.4 zenactiond zenactiondhasbeencompletelyrewrittenforZenoss4.Itisresponsibleforexecutingactionsassociatedwithnotificationssuchaspaging,email,executingbackgroundcommandsandraisingnotificationTRAPs.zenactiondwillperiodicallyinspectthesignalqueueforsignalmessages,dumpthemintoitsshareofmemcachedandsubsequentlyactonthemessagesasinstructedintheassociatednotification.
2.4.5 memcachedPriortoZenoss4eachofthedaemonshaditsowncache.Thiscouldbeawastefulallocationofmemory.WithZenoss4,amemcachedsubsystemisintroducedwhichprovidessharedL2memorycacheforalldaemons,offeringmuchbetterperformance.
memcachedisconfiguredin/etc/sysconfig/memcached.Thedefaultistoconfigure64Mbformemcached(whichisnotpreallocated;itisonlyusedasnecessary).Thisshouldbeincreasedtoatleast1Gbonproductionsystemswithmorethan100devices(andrun/etc/init.d/memcachedrestart).Alsoensurethatmemcachedisenabledin$ZENHOME/etc/zope.conf.
23January2013 EventManagementforZenossCore4Skills1stLtd 23
Figure12:Zenoss4eventarchitecture
-
2.5 Other database-related changes in Zenoss 4Notdirectlyrelatedtotheeventssubsystem,buttheZopedatabase(ZODB)thatusedtobeheldin$ZENHOME/var/Data.fsandaccessedbythezeoctldaemon,isnowstoredinthesameMySQLinstanceaszenoss_zep(andZEOhasgone).
ThezodbdatabaseisthemainZopedatabaseandthereisalsoazodb_sessiondatabasewhichholdsuserpreferencesthinkofzodb_sessionasanexpandedsetofuser'scookies;ifnecessary,itcanbedeletedanditwillberecreatedautomatically.
ZODBiswherealltheobjectdataisstoredrelatingtodevices,components,processes,services,networks,MIBs,etc.Theeventprocessingdaemonsneedaccesstothezodbdatabasetoenricheventswithdeviceandcomponentinformation.
Zopeobjectsareknownaspickles,typicallyastringrepresentationofencodeddata(ablob)inotherwords,treattheZODBdatabaseasablackbox(justasData.fswas).AJSONinterfaceisprovidedtoaccessdataintheZODBandthezendmdtoolstillworksinexactlythesamewayasinpreviousversionsofZenoss,despitetheZODBnowbeinginMySQL.
24 EventManagementforZenossCore4Skills1stLtd 23January2013
-
ToprovideaccesstothethezodbMySQLdatabase,aRelStoragesubsystemisusedasahighperformancebackendtoZODB.RelStoragemayalsousememcachedtofurtherenhanceperformance.
TheolderversionsofZenossdidnotdomuchbywayofindexingtheeventsdatabase.WithZenoss4holdingZODBdataaswellaseventsdatainMySQL,aneffectiveindexingmechanismwasrequiredsotheLucenepackageisusedfromApache.Luceneisahighperformance,fullfeaturedtextsearchenginelibrarywrittenentirelyinJava.Itisusedtoholdindexesforbothzodbandzenoss_zep.
2.6 Event life cycleThelifecycleofaneventhaseightphases:
Eventgeneration
Devicecontextadditionalinformationaboutthedevicethatgeneratedtheevent
Eventclassmappingtodistinguishonetype(class)ofeventfromanother
Eventcontextadditionalinformationpertinenttoaclassofevent
23January2013 EventManagementforZenossCore4Skills1stLtd 25
-
Eventtransformmanipulationofeventfields
Databaseinsertionanddeduplication
Resolution
Ageingandarchiving
ProcessingofaneventdependsontheeventclassthataneventisassignedtothevalueofitseventClassfield.Adescriptionofeachofthesephaseswillbegivenhere:subsequentsectionsofthepaperprovidemoredetailsofsomeareas.
InFigure14,thefirstsixphasesoftheeventlifecycleareshown.Theblue,dashedpathshowstheprogressofaninternallygeneratedZenossevent,whichdoesnotpassthroughaneventmappingphase.AneventClassfieldisproducedbythedaemonthatgeneratedtheevent.Itsonlywaytoapplyatransformisasaclasstransform.
ThepurplepathshowstheprogressofaneventthatisgeneratedexternallytoZenoss.TheinitialparsingdaemonmustprovideaneventClassKeyfieldwhichisthenused,alongwithotherfields,inaneventclassmappingRuleand/orRegex,whichinturnprovidesaneventClassfield.Aftermapping,theeventmaypassthroughbothaneventclasstransformandaneventmappingtransform.
26 EventManagementforZenossCore4Skills1stLtd 23January2013
Figure14:Eventlifecycle,generationtodatabaseinsertion
-
AnareathathaschangedfairlysignificantlyinZenoss4isthemechanismforresolvingandageingevents.PriortoVersion4,aneventwasfundamentallyopen(whichalsoencompassedeventStateofAcknowledgedandSuppressedaswellasNew)andsuchaneventresidedinthestatustableoftheeventsdatabase;alternatively,aneventwasClosed,inwhichcaseitwasmovedtothehistorytableoftheeventsdatabase.
WithZenoss4,thepossiblevaluesofeventStatehavebeenexpandedtoinclude:
Name Number Description
New 0 Anewevent
Acknowledged 1 Acknowledgedbyuserortransform
Suppressed 2 Eventtypicallybeyondasinglepointoffailure
Closed 3 Eventresolvedbyauser
Cleared 4 Eventresolvedbyanautomaticrule
Dropped 5 WouldneverreachtheMySQLdatabase
Aged 6 Eventautomaticallyclosedaccordingtotheseverityandlastseentimeoftheevent.
Thesearewelldescribedinchapter7oftheZenossCore4AdministrationGuide.Thehugedifferencehereisthatthenewevent_summarytableintheMySQLdatabasewillprobablyhaveClosed/Cleared/Agedeventsinit.Theevent_archivetablehaseventsthathavebeenautomaticallyagedoutbasedontheirseverityandage.
2.6.1 Event generationFundamentally,eventswilleitherbegeneratedbyZenossitselfintheprocessofdiscovery,availabilityandperformancechecking,oreventswillbegeneratedoutsideZenossandcapturedbyspecialisedZenossdaemons.
23January2013 EventManagementforZenossCore4Skills1stLtd 27
-
Zenossdaemon Exampleofwheneventgenerated
zenping pingfailureoninterface
zendisc newdevicediscovered
zenstatus TCP/UDPserviceunavailable
zenprocess processunavailable
zenwin Windowsservicefailed
zenwinperf WMIperformancedatacollectionfailure/threshold
zencommand sshperformancedatacollectionfailure/threshold
zenperfsnmp SNMPperformancedatacollectionfailure/threshold
zenmodeler Configurationdatachangedonzenmodelerpoll
Table2.1.:EventsgeneratedbyZenossitself
Zenossdaemon Exampleofwheneventgenerated
zensyslog processessyslogeventsreceivedonUDP/514(default)
zeneventlog processesWindowseventsreceivedusingWMI
zentrap processesSNMPTRAPsreceivedonUDP/162
Table2.2.:ExternaleventscapturedbyspecialisedZenossdaemons
EventsgeneratedinternallybyZenossneednofurtherprocessingtointerprettheevent.ThedaemonthatgeneratestheeventparsesthenativeinformationandassignsavaluetotheeventClassfieldandanyotherrelevantfieldssuchascomponent,summary,messageandagent.TypicallytheeventClassKeyfieldwillbeblank.SomeZenossdaemonspopulatetheeventKeyfield(forexampleanInterfacediscoveryeventwillpopulatetheeventKeyfieldwiththeIPaddressofthediscoveredinterface).
EventsthatareinitiallygeneratedoutsideZenossarecapturedbyzensyslog,zeneventlogorzentrap.ThesedaemonseachhaveaparsingmechanismtointerpretthenativeeventintotheZenosseventformat.ThePythoncodeforthezensyslogandzentrapparsingisin$ZENHOME/Products/ZenEvents.(Bydefault,$ZENHOMEwillbe/opt/zenoss).SyslogProcessing.pydecodessyslogevents;zentrap.pydecodesSNMPTRAPs.
ThedaemonsforprocessingWindowsWMIdatausedtobeastandardpartoftheCorecodebutwithZenoss4thishasmovedtoaZenosssuppliedZenPackZenPacks.zenoss.WindowsMonitor.zenwin,zenwinperfandzeneventlogcanallbefoundunderthatZenPack'sbasedirectory.
Typically,theexternaleventparsingmechanismsdonotdeliveravalueforeventClass;rathertheydeliveravaluefortheeventClassKeyfield,alongwithvaluesforsome
28 EventManagementforZenossCore4Skills1stLtd 23January2013
-
otherfieldssuchascomponent,summary,messageandagent.Itisthenthejoboftheeventmappingphasetodistinguishtheeventclass.
2.6.2 Application of device contextEarlyintheeventprocessinglifecycle,thezeneventddaemonappliesdevicecontexttotheevent.ThismeansthatsevenfieldsoftheeventarepopulatedbydeterminingthedevicethatgeneratedtheeventandthenlookingupthefollowingvaluesforthedeviceintheZODBdatabase:
prodState DevicePriority Location DeviceClass DeviceGroups Systems ipAddress(mayhavealreadybeenassigned)
2.6.3 Event class mappingEventclassmappingtendsonlytobeapplicabletoeventsthatoriginateoutsidetheZenosssystem.ItistheprocessbywhichaneventisassignedavalueforitseventClassfieldand,potentially,otherfields.
Typically,theeventgenerationphasewilldeliveraneventwithafewfieldspopulated;generallythisdoesnotincludetheeventClassfieldbutdoesincludetheeventClassKeyfield.OftentheZenossparsingdaemon(suchaszensyslog),willusethesameeventClassKeyforseveraldifferentnativeevents.Forexample,aneventClassKeyofdropbearisusedforseveralloginsecurityevents.Thecomponent,summary,messageandagentfieldsmayalsobepopulated.
Theeventclassmappingphaseexaminestheevent(suchasitis,sofar)andthenusesanumberofteststodeterminetheeventClasstoassigntothisevent:
1. AneventClassKeyfieldmustexistformappingtobesuccessful.
2. APythonRulecanbewrittentotestanyavailablefieldoftheeventoranyavailableattributeofthedevicefromwhichtheeventcame.SuchrulescanbecomplexPythonexpressions,includinglogicalANDsandORs.Iftheruleissatisfied,theincomingevent'seventClassfieldwillbegiventheclassassociatedwiththatmapping.Iftheruleisnotsatisfied,thismappingisdiscarded,theclassisnotassociated,andthenextmappingwillbetestedforamatch.ARuledoesnothavetoexistinamappinginstance.
3. IftheRuleissatisfied(ordoesnotexist),themappingcanthenuseaRegexPythonregularexpressiontoparsetheevent'ssummaryfield,checkingforparticularstrings.TheRegexcanalsoassignpartsofthesummaryfieldtonew,
23January2013 EventManagementforZenossCore4Skills1stLtd 29
-
userdefineddetailfieldsoftheevent.IfaRuleexistsandissatisfied,theclassmappingwillapply,eveniftheRegexisnotsatisfied;anyuserdefinedfieldsintheRegexwillnotbecreatediftheRegexdoesnotmatch.IfaRuledoesnotexistthentheRegexmustbesatisfiedforthemapping(andanytransform)toapply.
4. TheGUIdialoguethatdefinesthemappingspecifiestheeventClassKey,theRule,theRegexandanyTransform.AsequencenumberisalsoavailablesothatifmultipleincomingeventshavethesameeventClassKeythenthesequencenumberdefinestheorderinwhichthevariousmappingswillbeapplied,lowestnumberfirst.ThefirstRule/Regexmappingcombinationthatmatcheswillbeapplied.
Eventclassmappingisexecutedbythezeneventddaemon.
2.6.4 Application of event contextEventcontextisdefinedbytheConfigurationProperties(zProperties)ofanevent.Eventcontextcanbedefinedattheeventclasslevel,foraneventsubclass,orattheeventmappinglevel.Aswithallobjectorientedattributes,thevaluesareinheritedbychildobjectssoapplyingeventcontexttoaclassautomaticallysetsitforanysubclassesandsubclassmappings.Thethreeeventcontextattributesare:
zEventAction status|history|dropdefaultisstatus
zEventClearClasses bydefaultthisisanemptyPythonlistofstrings
zEventSeverity Originalbydefault
Eventcontextisappliedintheeventlifecycle,afterRuleandRegexprocessingbutbeforeanyeventtransforms.Thus,thezEventActionzPropertycanspecifyhistorybutaneventtransformcouldoverridethatactionbysettingtheevt._actionvaluetostatus.
NotethatthestatusandhistoryvaluesreflecttheolddatabasetablespriortoZenoss4.statusnowmapstoaneventStateofNewandhistorymapstoaneventStateofClosed;bothwillbestoredintheevent_summarydatabasetable.
Eventcontextisappliedbythezeneventddaemon.
2.6.5 Event transformsEventtransformscanbespecifiedforaneventclassmappingorforaneventclass(orsubclass).AtransformiswritteninPythonandcanbeusedtomodifyanyavailablefieldsofeithertheeventorthedevicethatgeneratedtheevent.Itcanalsocreateuserdefinedfields.
FromZenoss2.4,cascadingeventtransformsmeanthatclasstransformsareappliedfromeverylevelintheappropriateclasshierarchy,followedbyanytransformforan
30 EventManagementforZenossCore4Skills1stLtd 23January2013
-
appliedeventmapping.PriortoZenoss2.4,eitheramappingtransformwasapplied,oraclasstransform,butnotboth.Classtransformswereonlyappliedtotheexactclass,notfromtheeventclasshierarchy.
AtransforminaneventmappingwillonlybeexecutedoncetheeventClassKeyhasbeenmatched,andtheRulehasbeensatisfied(ifitexists).IfaRuledoesnotexist,anyRegexhastobesatisfiedforthetransformtobeexecuted.
Eventtransformsareexecutedbythezeneventddaemon.
2.6.6 Database insertions and de-duplicationZenosseventsarenowstoredinaMySQLdatabasecalledzenoss_zep(usedtobeevents).Themaintablesfortheeventlifecyclearetheevent_summarytableforrecentevents,theevent_archivetableforoldevents.
Somefieldsoftheeventareonlyassignedatdatabaseinsertiontimetheyarenotavailableateventmappingoreventtransformtime.Theseinclude:
count eventState evid stateChange dedupid eventClassMapping firstTime lastTime
ItistheJavazeneventserverdaemonthatisresponsibleforgettingeventsintothedatabase.
Zenossautomaticallyappliesaduplicationdetectionrulesothatifaduplicateeventarrives,thentherepeatcountofanexistingeventwillbeincremented.duplicateisdefinedashavingthefollowingfieldsthesame:
device component eventClass eventKey severity
IftheeventdoesnotpopulatetheeventKeyfield,thenthesummaryfieldmustalsomatch.Thededupidfieldiscreatedbyconcatenatingtheabovefieldstogether,separatedbythepipe(verticalbar)symbol.Thusanexamplededupidmightbe:zenoss.skills-1st.co.uk|su|/Security/Su||5|FAILED SU (to root)jane on /dev/pts/1
wherethedeviceiszenoss.skills1st.co.uk,componentissu,eventClassis/Security/Su,theeventKeyisunset,severityis5(Critical),andthesummaryisFAILEDSU(toroot)janeon/dev/pts/1.
InZenoss4,thededupidfieldisalsoknownasthefingerprint.
23January2013 EventManagementforZenossCore4Skills1stLtd 31
-
Whenaneweventisreceivedbythesystem,thededupidisconstructedbythezeneventddaemon.Transformsmaymodifyeithercomponentfieldsofthefingerprintormaydirectlymodifythededupidfield.
Whenzeneventservercomestoinserttheeventinthedatabase,ifitmatchesthededupidforanyactiveevent,theexistingeventisupdatedwithpropertiesoftheneweventoccurrence,theevent'scountisincrementedbyone,andthelastTimefieldisupdatedtobethecreatedtimeoftheneweventoccurrence.
NotethatthisisasubtlebutsignificantchangefrompriorversionsofZenossastheexistingeventisupdatedwithpropertiesofthenewevent;olderversionsofZenosssimplyupdatedthecountandlastTimefields.Forexample,ifthefingerprintincludesaneventKeysodoesnotincludethesummary,theresultingeventwillnowshowthesummaryofthelatestreceivedduplicateevent.
Iftheincomingeventdoesnotmatchthededupidofanyactiveevents,thenitisinsertedintotheactiveeventtablewithacountof1,andthefirstTimeandlastTimefieldsaresettothecreatedtimeofthenewevent.
2.6.7 ResolutionResolutionofaproblemrepresentedbyaneventcanhappeninseveralways:
Auserclosestheevent(eventState=Closed)
TheeventcontextzEventActionzPropertyforaneventclassisdrop(theeventisdiscarded).Forexample,eventclass/Ignore.
TheeventcontextzEventActionzPropertyforaneventclassishistory(eventState=Closed).Forexample,eventclass/Archive.
Atransformsetsevt._actionto'drop'(theeventisdiscarded)
Atransformsetsevt._actionto'history'(eventState=Closed)
Anotherclearingeventarrivesthatclearstheinitialevent(eventState=Cleared)
TheEventManagersettingshaveseverityandlastSeenparametersthatdenotewhicheventswillbeautomaticallyaged(eventState=Aged)
Alltheaboveeventswillstillbeintheevent_summarytableoftheMySQLdatabase.TheEventManagerparameterforEventArchiveThresholdistheonlyautomaticactionthatmoveseventsfromevent_summarytoevent_archiveanditwillmovealleventswitheventStateofClosed,ClearedandAged.
Themoreinterestingformsofeventresolutioninvolvecorrelationofevents;therearetwodifferentmechanisms.Thebasicprincipleisthatgoodnewsclearsbadnews.
ThefirstclearingmechanismisthatanyeventwithaseverityofClearwillsearchtheevent_summarytableforsimilaractiveeventsandsettheireventStatetoCleared(notClosed).
TheZenossCore4AdministratorsGuidedefinesthisautoclearfingerprintas:
32 EventManagementforZenossCore4Skills1stLtd 23January2013
-
IfcomponentUUIDexists:
componentUUID
eventClass
eventKey(canbeblank)
IfcomponentUUIDdoesnotexist:
device
component(canbeblank)
eventClass
eventKey(canbeblank)
Thiscanbealittleconfusing.TheEventConsoleshowsacomponentfield.ItdoesnotshowacomponentUUIDfield.StrictlythecomponentfieldintheEventConsoleshowstheelement_sub_identifierfieldfromtheMySQLdatabasetablethenameofthecomponent.SomeeventsgenerateacomponentUUID(UniversallyUniqueIdentifier)andsomedonot.InspectingtheeventinthedatabaseorusingtheJSONinterfaceistheonlywaytodeterminewhetherthisuniquecomponentidfieldexistsornot.Ifitdoesexistthenitshouldalso,byimplication,denotethedevicethatthecomponentbelongsto,hencethedevicefieldisunnecessary.(VersionsofZenosspriorto4didnothaveacomponentUUID;similarwasdefinedashavingthesameeventClass,deviceandcomponentfields.)
EitherwayinCore4,theeventClassandtheeventKeyfieldsaresignificant.IfthecomponentUUIDdoesnotexistthenitistheelement_sub_identifier(componentname)thatmustmatch,alongwiththedevicename(element_identifierintheMySQLtable).
ThesecondautomaticclearingmechanismextendstheautoclearfingerprintdefinitionofeventClass.TheeventcontextofaneventclassincludeszEventClearClasseswhichisalistofothereventclassesthatthisgoodnewseventwillclear,inadditiontoitsownclass.Theotherconditionsoftheautoclearfingerprintremainthesame.
Notethatthesameeffectcanbeachievedinatransformbyassigningalistofclassnamestoevt._clearClasses.
Alleventswiththesameautoclearfingerprintarecleared,notjustthemostrecent.
TheclearingeventwillautomaticallyhaveitseventStatesettoClosed,provideditmatchesoneormorebadnewsevents.Ifitdoesnotmatchanyeventsthentheclearingeventisdroppedandwillnotbepersistedtothezenoss_zepdatabase.Thisistoavoidfillingupthedatabasewithredundantgoodnewsevents.
Whencorrelationtakesplacesomeoftheexistingbadnewseventfieldsareupdated;stateChangebecomesthetimewhentheeventwasresolved;clearidispopulatedwiththeevidfieldoftheclearing,goodnewsevent.
Thisautomaticresolutionofeventsisperformedbythezeneventserverdaemon.
23January2013 EventManagementforZenossCore4Skills1stLtd 33
-
2.6.8 Ageing and archivingMaintenanceisrequiredonthetablesofthezenoss_zepdatabaseorthediskwillsimplyfillupeventually.ThreemechanismsareprovidedbytheEventManager:
Bydefault,eventswithseveritylessthanErrorwillbeAgedafteranEventAgeingThresholdof4hours;thatis,theeventStatewillbesettoAged(strictlythevalue6).
Bydefault,theEventArchiveThresholdis4320minutes(3days).ThismeansanyeventwitheventStateofClosed,ClearedorAgedwillbemovedfromtheevent_summarytabletotheevent_archivetableofthezenoss_zepdatabase.
TheDeleteArchivedEventsOlderThan(days)parameteris90bydefault.Thisistheonlyparameterthatautomaticallydeletesdata.Itisnotpossibletofinetunethistodelete,say,lowerseverityeventsafterdifferentintervals.
Zenosspriortoversion4providedautility,$ZENHOME/Products/ZenUtils/ZenDeleteHistory.pywhichcoulddeleteeventsselectivelybasedonageandseverity.ThisutilityisnotshippedwithZenoss4andcurrentlyhasnoequivalentfunction.
DeletingdatafromtheoldhistorytableinZenoss3usedtobeveryslow.InZenoss4,theevent_archivetableispartitioned,byday,ratherthanbeingonehugefile.Thismeansthatdeletingdataissimplyamatterofdroppingpartitionfiles.Thiscanbeseenfromthemysqlinterfacewith:
showcreatetableevent_archive;
3 Events generated by ZenossInthecourseofdiscovery,availabilitymonitoringandperformancemonitoring,Zenossmaygenerateeventstorepresentachangeinthecurrentstatus.AlthoughmanyeventsarebadnewsitshouldberecognisedthateventscanalsobegoodnewsInterfaceUp,Thresholdnolongerbreached,etc.
EventsgeneratedbyZenossaredependentonthevariouspollingintervalsconfigured.Toexaminethedefaultparameters,usetheADVANCED>Collectorsmenu.Clickonlocalhost(thecollectorontheZenosssystem).NotethatearlyversionsofZenossusedthetermandmenuoptionMonitorsratherthanCollectors.
34 EventManagementforZenossCore4Skills1stLtd 23January2013
-
Parameterstonoteparticularlyare:
SNMPPerformanceCycleInterval 300secs(5mins)
ProcessCycleInterval 180secs(3mins)
StatusCycleInterval 60secs(1min)
WindowsServiceCycleInterval 60secs(1min)
PingCycleTime 60secs(1min)
ModelerCycleInterval 420mins(12hours)
3.1 zenpingThemostbasiclevelofavailabilitycheckingistopingpoll.Thezenpingdaemonwill,bydefault,pingpolleachinterface,everyminute.Aninterfacedowneventisgeneratedwhenthepingfailstogetaresponse.Thiseventisautomaticallyclearedwhenasimilarpingissuccessful;meantime,whileaninterfaceremainsdown,thecountfieldoftheeventisincreased.
Thezenpingdaemoncandetectwhenthenetworkpathtoadeviceisbroken,forexampleifasinglepointoffailurerouterisdown.WithZenoss4thisisachievedusingnmap;withearlierversions,ZenossbuiltaninternaltopologybasedonqueryingroutingtableswithSNMP.
Ifaneventisreceivedforanisolatedelement,aneventisgeneratedwithaneventStatefieldofSuppressedandthesummaryfieldreportsnotonlytheinterfaceforwhichthepingfailed,butalsothecausaldevice;forexample:
ip10.191.101.1isdown,failedatbino.skills1st.co.uk
23January2013 EventManagementforZenossCore4Skills1stLtd 35
Figure15:DefaultparametersforlocalhostCollector
-
Allotherdeviceavailabilitymonitoringisdependentonpingaccess.Onceapinghasfailed,SNMP,process,TCP/UDPserviceandwindowsservicemonitoringwillallbesuspendeduntilpingaccessisrestored.Thecountfieldofthehigherlevelmonitoringeventswillnotincreaseuntilpingaccessisresumed.
Alsonotethatifthereisnopingaccess,noperformanceinformationwillbecollected.Ifadevicereallydoesnotsupportping,perhapsbecauseoffirewallrestrictions,thenensurethatthezPropertyzPingMonitorIgnoreissettoTrue;thiswillpermitSNMPandsshavailabilitymonitoringandperformancedatacollection.
Thelogfileforzenpingiszenping.login$ZENHOME/log.
3.2 zenstatusThezenstatusdaemoncanbeconfiguredtocheckforaccesstovariousTCPand/orUDPportsonbothWindowsandUnixarchitectures.Bydefault,itcheckseveryminute.Zenosscomeswithahugenumberofservicespreconfigured;thesecanbeexaminedfromtheINFRASTRUCTURE>IpServicesmenu.Bydefault,theonlyservicemonitorsthatareactiveareforsmtpandhttp;therestaresetwithmonitoringdisabled.
Aswithpingpolling,agoodnewsserviceeventforadeviceautomaticallyclearsasimilarbadnewseventandthecountfieldoftheeventincreaseswhilsttheserviceremainsdown.
Thelogfileforzenstatusiszenstatus.login$ZENHOME/log.
3.3 zenprocesszenprocessmonitorsWindowsandUnixsystemsforthepresenceofprocesses.InaUnixcontext,thiswouldbewhethertheprocessappearsinapseflisting;inaWindowscontext,theprocessmustappearintheWindowsTaskManager(andnotethatthischeckiscasesensitiveonbotharchitectures).Monitoringisevery3minutes,bydefault.
ConfigurationofprocessmonitoringforadeviceissimilarasforservicestheINFRASTRUCTURE>Processesmenuprovidesawaytoconfigureprocessestobemonitored.Zenoss4comeswithdefinitionspreconfiguredforalltheZenossprocesses.
ProcessmonitoringisactuallyachievedusingtheHostResourcesManagementInformationBase(MIB)ofSNMP,byretrievingthehrSWRuntable.ThismeansthatifSNMPaccesstoadeviceisbroken,therewillbenoprocessinformation.
Aswiththeotheravailabilitydaemons,goodnewseventsclearbadnewseventsandthecountfieldincreasesonsubsequentfailedpolls.
Thelogfileforzenprocessiszenprocess.login$ZENHOME/log.
36 EventManagementforZenossCore4Skills1stLtd 23January2013
-
3.4 zenwinThezenwindaemonshipswiththeZenPacks.zenoss.WindowsMonitorZenPackwithZenoss4(itwasastandardpartoftheCorecodeinearlierversions).ItmonitorsWindowsservices(notTCP/UDPservices).ThesecanbeexaminedfromtheINFRASTRUCTURE>WindowsServices.Bydefault,noneofthesemonitorsareactive.
zenwinusestheWindowsManagementInstrumentation(WMI)interfacetoaccessservicesontheremotesystemeveryminute,bydefault.ThezPropertiesforadevice(ordeviceclass)mustbeconfiguredtoallowaccesstoWMIbeforewindowsservicepollingcanbesuccessful.
Aswithpingpolling,agoodnewswindowsserviceeventforadeviceautomaticallyclearsasimilarbadnewseventandthecountfieldincreasesonsubsequentfailedpolls.
Thelogfileforzenwiniszenwin.login$ZENHOME/log.
3.5 zenwinperfzenwinperfisanewdaemonforZenoss4whichisalsopartoftheZenPacks.zenoss.WindowsMonitorZenPack.WithearlierversionsofZenoss,manyusersdeployedtheexcellentcommunityWMIDataSourceandWMIWindowsPerformanceZenPackstoachievesomethingverysimilartothisnewdaemon.
zenwinperfprovidesperformancemonitoringofinterfaces,filesystems,memory,CPUandpagingusingtheWMIprotocol.Defaultthresholdsareconfiguredforsomemetricswhichthengenerateeventswhenexceeded.ItcanbeextendedbytheusertomonitorotherperfmonmetricsusingtheWMIprotocol.
Dataisgatheredevery5minutes.
Thelogfileforzenwinperfiszenwinperf.login$ZENHOME/log.
3.6 zenperfsnmpzenperfsnmppollseachdeviceevery5minutes,bydefault.ItcancollectbothSNMPperformanceinformationandstatusinformationforprocesses.EvenifSNMPperformancemonitoringisnotconfigured,zenperfsnmpchecksthattheSNMPagentisavailable.
Within5minutesofanSNMPpollfailure,ansnmpagentdowneventshouldbegenerated.Withinafurther3minutesthereshouldbeanUnabletoreadprocessesondevice..event,ifprocessmonitoringisconfigured.Notealsothatthecountfieldforindividualmissingprocesseventsshouldstopincreasing.WhileSNMPaccesstothedeviceremainsbroken,thecountfieldfortheUnabletoreadprocessesondevice..eventwillincreaseevery3minutes.
23January2013 EventManagementforZenossCore4Skills1stLtd 37
-
Thelogfileforzenperfsnmpiszenperfsnmp.login$ZENHOME/log.
3.7 zencommandThezencommanddaemonperformsmonitoringbasedonrunningcommands,typicallyoveransshconnection.Likezenperfsnmpandzenwinperfitusesperformancetemplatestomonitormetricsandcangenerateaneventifathresholdisbreached.
Thelogfileforzencommandiszencommand.login$ZENHOME/log.
4 Syslog eventsTheUnixsyslogmechanismispervasivethroughoutallversionsofUnix/Linuxalthoughslightlydifferentversionsandformatsexist.TherearealsoopensourceimplementationsofsyslogforWindowssystemsandmanynetworkingdevicesalsosupportthesyslogconcept.
Typicallysystemmessagesareoutputtooneormorelogfilessuchas/var/log/messages.Thesyslogsubsystemcanalsobeconfiguredtosendsyslogmessagestoacentralsyslogratherthanholdingfilesoneachsystem.ThewellknowndefaultportforforwardingsyslogmessagesisUDP/514.
Astandardsyslogsystemisconfiguredbythesyslog.conffile,typicallyin/etc.Anewerversionofsyslogisimplementedonsomesystems,syslogng,whichhasgreaterfilteringcapabilities.Thesyslogngconfigurationfileistypically/etc/syslogng/syslogng.conf.
AnothervariationisrsyslogdwhichistypicallyshippedwithnewerRedHat/CentOSSuSEsystems,configuredthrough/etc/rsyslog.conf.
Asyslogmessageincludesapriorityandafacility.Theprioritiesare:
0 emerg1 alert2 crit3 err4 warning5 notice6 info7 debugFacilitiesinclude:
auth (4) authpriv(10)
cron (9) daemon(3)
ftp(11) kern(0)
lpr(6) mail(2)
38 EventManagementforZenossCore4Skills1stLtd 23January2013
-
news (7) syslog(5)
user (1) uucp(8)
Thesedefinitionscanbefoundinsyslog.h(typicallyin/usr/include/sys).Bothpriorityandfacilityareencodedinasingle32bitintegerwherethebottom3bitsrepresentpriorityandtheremaining28bitsareusedtorepresentfacilities.
Forexample,ifthefacility/prioritytagis,thiswouldbe00010110inbinary,wherethebottom110representsapriorityof6(info)andthetop00010representsafacilityof2=mail.
4.1 Configuring syslog.conf AnydevicethatisgoingtoreportsyslogeventstoZenossmusthaveitssyslog.conffileconfiguredwiththedestinationaddressoftheZenosssystem.Theoriginalsyslog.confpermitsfilteringbasedonpriorityandfacilityso,acatchallstatementtosendalleventstotheZenosssystem,wouldbe: *.debug @
Thisalsoworksforrsyslogd.SeeFigure16foranrsyslog/syslogexamplethatforwardstozen42.class.example.orgallfacilitieswithpriorityofnoticeandabovebutallcronmessagesarefilteredout;authprivmessageswillbeforwardedwithseverityinfoandabove.
23January2013 EventManagementforZenossCore4Skills1stLtd 39
Figure16ConfigurationfileforrsyslogsendingselectedeventstoZenossserver
-
syslogng.confrequiresatleastasource,adestinationandalogstatement.syslogngofferssuperiorfilteringovertheoriginalsyslogsooneormorefilterstatementsmayalsobepresent.
4.2 Zenoss processing of syslog messagesTocollectsyslogmessageswithZenoss,thezensyslogprocessautomaticallystartsonportUDP/514andcollectsanysyslogmessagesdirectedfromothersystems.zensyslogthenparsesthesemessagesintoZenossevents.Youmustensurethatthesyslog.conffileontheZenosssystemdoesnotenablecollectingremotesyslogsorthesyslogdandzensyslogprocesseswillclashoverwhogetsUDP/514(itispossibletoreconfigureeitherdaemon,ifrequired).
40 EventManagementforZenossCore4Skills1stLtd 23January2013
Figure17:syslogng.conftosendalleventstoZenosssystemat10.0.0.131(nofilteringactive)
-
Toexaminetheincomingsyslogmessagesandtheparsingthatzensyslogperforms,thelevelofzensyslogloggingcanbeincreased.
1. UsetheINFRASTRUCTURE>Settings>Daemonsmenu.
2. Clicktheeditconfiglinkforthezensyslogdaemon.
3. ChangethefollowingparametersandclickSave:
logorig selectthis
logseverity Debug
4. Inspecttheunderlyingconfigurationfilein$ZENHOME/etc/zensyslog.conf.
5. Thelogoriglinesaystologtheoriginalincomingsyslogmessage;itwillbein$ZENHOME/log/origsyslog.log.Notethatthisparameterisuniquetozensyslogandisusefulfordebugging.
6. ThelogseveritylineisagenericZenossdaemonparameter;avalueof10isthemaximumDebuglevel.
7. Don'tforgettoSavethischange
8. UsetheRestartlinktorecyclezensyslog.Alternatively,asthezenossuser,issuethecommand:
zensyslog restart9. Examinethezensysloglogfilein$ZENHOME/log/zensyslog.log
10.Anewincomingeventstartswithalineshowinghostnameandipaddress,eg.host=zen241.class.example.org, ip=172.16.222.241
11.Thenext2linesshowtherawmessageandthedecodingforfacilityandpriority.
12.LinesstartingwithtagshowthezensyslogparsingprocessasitteststheincominglineagainstvariousPythonregularexpressions,hopefullyendingwithatagmatchline.
13.Ifamatchissuccessful,aneventClassKeymaybedetermined
14.ThelastlineforaparsedeventshouldbeaQueueingevent.
23January2013 EventManagementforZenossCore4Skills1stLtd 41
-
Wheneverdifferentnativeeventlogsystemsareintegratedthereisalmostinevitablyamismatchofseverities.Thefollowingtabledemonstratesthis.
Zenoss syslogpriority Windows
Critical(red)(5) emerg(0) Error(1)
Error(orange)(4) alert(1) Warning(2)
Warning(yellow)(3) crit(2) Informational(3)
Info(blue)(2) err(3) Securityauditsuccess(4)
Debug(grey)(1) warning(4) Securityauditfailure(5)
Clear(green)(0) notice(5)
info(6)
debug(7)
Table4.1.:EventseveritiesforZenoss,syslogandWindows
NotethatthenumericvalueofZenosseventseveritydecreasesaseventsgetlesscriticalbutthatthepriorityofsyslogeventsincreasesaseventsgetlesscritical.
DefaultmappingfromsyslogprioritytoZenosseventseverity,isperformedby$ZENHOME/Products/ZenEvents/SyslogProcessing.pysearchfordefaultSeverityMaparoundline187inCore4.2.Theresultisthat:
syslogpriority
-
Outofthebox,allsyslogeventsmaptotheZenosseventclassof/Unknown.
SyslogProcessing.pyisthecodethatparsesanyincomingsyslogmessageandgeneratesaZenossevent.
ThefirstsectionhasaseriesofPythonregularexpressionstomatchagainsttheincomingsyslogline.Eachexpressionischeckedinturnuntilamatchisfound.Ifnomatchisfoundthenanentrygoesto$ZENHOME/log/zensyslog.logwithparseTagfailed.
ThemainbodyofSyslogProcessing.pystartsbyassigningvaluesfromtheincomingeventtoZenosseventclassfields,asfollows:
23January2013 EventManagementforZenossCore4Skills1stLtd 43
Figure19:SyslogProcessing.pyregularexpressionstomatchsyslogtags
-
def process(self, msg, ipaddr, host, rtime): evt = dict(device=host, ipAddress=ipaddr, firstTime=rtime, lastTime=rtime, eventGroup='syslog')
Atthisstage,noaccountofduplicatesistakensothefirstTimeandlastTimefieldsarebothsettothetimestampontheincomingevent.NotethattheZenosseventGroupfieldishardcodedatthisstagetosyslog.
parsePRIisthePythonfunctioncalledtoparseoutthesyslogpriorityandfacility.
ThedefaultSeverityMapfunctioniscalledfromwithintheparsePRIfunctiontosettheseverityfieldoftheZenossevent.
44 EventManagementforZenossCore4Skills1stLtd 23January2013
Figure20:SyslogProcessing.pyprocessmainroutine
-
Next,theparseHEADERfunctioniscalledtoextractthetimestampandhostnamefromtheincomingevent.ThedeviceandipAddressfieldsoftheZenosseventaresetattheendofthisfunction.
23January2013 EventManagementforZenossCore4Skills1stLtd 45
Figure21:SyslogProcessing.pyparsingofpriority,facilityandseverity
-
TheparseTagfunctioniscalledtoparseoutthesyslogtag,usingtheregexexpressionsatthebeginningofthefile.IfnomatchexiststhenaparseTagfailedmessageislogged.TheendofthefunctionreturnstheremainderoftheincomingmessageintheZenosseventsummaryfield.
46 EventManagementforZenossCore4Skills1stLtd 23January2013
Figure22:SyslogProcessing.pyprocessingtheheaderinformation
-
ThecruxofeventprocessinginZenossistoderiveaneventClassKeythisisdonewiththebuildEventClassKeyfunction.
23January2013 EventManagementforZenossCore4Skills1stLtd 47
Figure23:SyslogProcessing.pyparsingthesyslogtag
-
NotethatiftheeventhasthecomponentfieldpopulatedthenthatisusedastheeventClassKeyaftercheckingforapreexistingeventClassKeyandforanntevidfield.
5 Zenoss processing of Windows event logs5.1 Management using the WMI protocol
Zenosspriortoversion4shippedWindowsmonitoringaspartoftheCorecode.Zenoss4shipsWindowssupportwiththeZenPacks.zenoss.WindowsMonitorZenPackwhichhasaprerequisiteofZenPacks.zenoss.PySamba.TheseareZenossprovidedCoreZenPacks.
IfaWindowsdevicesupportsSNMPthenitisperfectlypossibletousethatprotocol,especiallyasmostWindowsSNMPagentsalsosupporttheHostResourcesMIBsosomesysteminformationisavailableinadditiontothestandardMIB2networktypeinformation.
TheZenossWindowsZenPacksintroducethe/Server/Windows/WMIdeviceclasswhichhasbothWMImodelerpluginsandWMIperformancetemplatesassociatedwithit.Targetdevicesshouldbeaddedtothisclassorsubclassesthereof.ThisallowsmonitoringusingtheWindowsManagementInstrumentation(WMI)protocol.AuseridandpasswordneedtobeconfiguredontargethoststopermitWMIaccessfromthe
48 EventManagementforZenossCore4Skills1stLtd 23January2013
Figure24:SyslogProcessing.pydeterminingtheEventClassKey
-
Zenossserver;italsomeansthatfirewallsbothontheWindowsdevicesandanyinterveningnetworkfirewalls,mustbeconfiguredtopermitWMIaccess.TheZenossServermustthenbeconfiguredwithmatchingWindowszProperties(zWinUserandzWinPassword)forthetargetdevices/deviceclasses.ThereareafewotherWindowsspecificConfigurationPropertiesseeFigure25.ThesezPropertiescanbechangedforadeviceclassorforaspecificdevice.
ZenPacks.zenoss.WindowsMonitorprovidesthreenewdaemons: zenwin monitorswindowsservicesusingWMI zenwinperf collectsperformancedatausingtheWMIprotocol zeneventlog retrievesWindowseventloginformationusingWMI
ThethreezWinPerf...zPropertiesfinetunetheconfigurationofthezenwinperfdaemon;thezWinEventlogparametermustbeTruetocollectWindowseventsfromatargetdevice.
ThezWinEventlogMinSeveritypropertydefinestheleastseriousseverityeventsthatwillbeforwardedfromWindowstoZenoss.Notethatthenumericdenotationofwindowseventseveritiesandtheirnamesandsupportcurrency,havechangedoverthelifeofZenoss.SeeTable4.1onpage42forcurrentvalidseverities.AlsonotethatifyouchangethisparameteryouarepresentedwithalistofZenossseverities,notWindowsstyleseverities;againrefertotheearliertableforatranslation.IfyouwanttoincludeallWindowsseverities,includingsecurityauditfailure(5),youneedtoselecttheClearseverityinthedropdownmenuwhenchangingzWinEventlogMinSeverity.
ThezWinEventlogClausewasintroducedduringthelifetimeofZenoss3tohelpfiltereventsfromWindowsdevices.ConsulttheZenossCore4AdministratorsGuide,chapter
23January2013 EventManagementforZenossCore4Skills1stLtd 49
Figure25zPropertiesforWindowstargets
-
6.6.6fordocumentationandexamples.Thisparameterisratherobtuse.FundamentallyaWindowsQueryLanguage(WQL)queryisconstructedtoberunbyzeneventlog:
SELECT*FROM__InstanceCreationEventWHERETargetInstanceISA'Win32_NTLogEvent'ANDTargetInstance.EventType
-
ManyWindowseventlogeventsareautomaticallymappedtoeventclassesbuttheymayhavealowseverity(suchasDebug)andtheymayhavetheirzEventActioneventzPropertysettohistorysothattheydonotappearinthestatustableoftheeventsdatabase.
5.2 Management of Windows systems using syslogThereisalsoasyslogutilityavailableforWindowssystemsfromDatagramConsultingathttp://syslogserver.com.TheclientutilityisSyslogAgentandismadeavailableundertheGNUlicense.SyslogserverutilitiesforWindowsarealsoavailableaschargeableproducts.ThismeansthatWindowseventlogscanalsobecollectedwiththezensyslogdaemon.
NotethattheSyslogagentiscapableofbeingconfiguredtomonitorWindowsapplicationlogfiles,inadditiontothestandardWindowseventlogs.Whenmonitoringthestandardeventlogs,therearebetterfilteringcapabilitieswithSyslogthenwithzeneventlog.
6 Event Mapping
ZenosseventsarecategorisedintoahierarchyofeventClasses,manyofwhicharedefinedoutoftheboxbutwhichcaneasilybemodifiedoraugmented.TheprocessofEventClassMappingisaboutassociatinganincomingeventwithaparticularZenossEventClass(settingitseventClassfield)and,potentially,modifyingotherfieldsofthateventbyusinganeventtransform.
Eventclassesandsubclassesaretreatedidenticallyfromthepointofviewofeventclassmapping.Theclasshierarchycanbeusefulinthateventcontext,asimplementedbyeventzProperties(zEventSeverity,zEventAction,zEventClearClasses),followsthenormalrulesforobjectinheritanceifzEventActionissettodropontheeventclass/Ignore,thenanysubclassesof/Ignorewillalsoinheritthatproperty.
NotableoutoftheboxeventzPropertiesarethat/Ignoreclassesandsubclassesdropincomingeventstotally;/ArchiveclassesandsubclassesautomaticallysettheeventStatefieldtoClosed.
Mosteventclasseshaveoneormoremappingsassociatedwiththemtheseareknownasinstances.Notethataneventdoesnothavetohaveanymappingsassociated,inwhichcaseaneventofthatclasswillonlyappearinanEventConsoleifthedaemonthatgeneratestheevent,assignstheeventclassatthattime(/Perfeventsmaywellcomeintothiscategory,forexample).Outoftheboxeventclassmappingsaredefinedin$ZENHOME/Products/ZenModel/data/events.xml.TheycanbeinspectedfromtheZenossGUIbyselectingtheEVENTS>EventClassesmenu.
23January2013 EventManagementforZenossCore4Skills1stLtd 51
-
MostoutoftheboxeventclassmappingssimplymatchontheeventClassKeyfieldwhichispopulatedbythenativeeventparsingmechanism(suchaszensyslog,zeneventlog,zentrap).ThesemechanismsmaygenerateseveraldifferenteventswiththesameeventClassKeyfield;thusothertechniquesareneededtodistinguishbetweensucheventsandpotentiallytoseparatethemintodifferenteventclasses.
Thesequencenumberinaneventmappinggivestheorderinwhichmappingsaretestedagainsttheincomingeventlowestnumbersaretestedfirst.Dependingonwhichmappingactuallymatches(ifany)willdeterminetheresultingeventClassoftheevent.
6.1 Working with event classes and event mappingsEventsareorganisedinanobjectorientedhierarchy;thusattributesassignedtoaparenteventclassareinheritedbyachildeventsubclass.
NeweventclassescanbedefinedbynavigatingtoaneventclassandusingthedropdownmenualongsideSubClassestoAddNewOrganizer.Thenamesuppliedisthenameoftheneweventclass.Forexample,drilldowntothe/SecurityeventclassandcreateanewsubclasscalledSu.
Anyeventwhichdoesnotmaptoaneventclassisthegiventheclassof/Unknown.ThesimplestwaytomapsuchaneventistostartfromanexistingeventintheEventConsole.Thefollowingscenarioexplainsthis,creatinganeweventclassmappingcalledsuwhichmapsanincomingeventtotheeventclass/Security/Su.
1. GenerateasyslogauthenticationfailureeventattheZenosssystem.
2. OpenanEventConsolethatshowstheeventandinspectitsdetails.
3. SelecttheeventandusetheReclassifyEventiconatthetopoftheconsole.Selectyournew/Security/Suclassfromthedropdownlist.Youshouldbeshowntheeventclassmappingpanel.ClickthelefthandEditmenu.
4. YoushouldfindthatthenameoftheneweventclassmappingissettosuandtheEventClassKeyissettosu(notelowercasesinbothcases).TheeventClassKeyfieldisactuallyderivedfromthecomponentfieldoftheincomingeventinSyslogProcessing.py(aroundline289).ThesummaryfieldoftheeventshouldhavebeencopiedintothemappingExamplebox.
5. AddatextstringtotheExplanationboxsuchasAutoaddedbyeventmapping.
6. AddatextstringtotheResolutionboxsuchasThisisadummyresolution.
7. OpenaZenossGUIwindowthatshowsallSuevents(youmayfinditusefultohaveseveralbrowsertabsopentofocusondifferentaspectsoftheZenossGUI).SelectalltheSueventsandClosethem.
8. GenerateanewSuevent.
9. CheckthedetailsoftheneweventintheEventConsole.TheeventshouldhavemappedtoeventClass/Security/Su.TheseverityshouldbeInfo(blue).The
52 EventManagementforZenossCore4Skills1stLtd 23January2013
-
detailsoftheeventshouldshowtheeventClassMappingfieldsetto/Security/Su/su.
Anyexistingeventmappingcanbemodifiedinasimilarfashion.
Wheneveryouchangeaneventmapping,itisadvisabletoclearanyexistingeventsofthatcategorybeforetestingthenewconfiguration.
Whenyouareworkingwitheventmappings,don'tforgettheEventmenuwhichfiltersanEventConsolebyEventClass.
Itisusefultorefertoeventclassesusingthebreadcrumbpathseenatthetopofapage,suchas/Events/Security/Su.
23January2013 EventManagementforZenossCore4Skills1stLtd 53
Figure27:Editdialogueforeventclassmapping
-
6.1.1 Generating test eventsTesteventscanbecreatedfromtheEventConsoleusingthe+icon.
Alternatively,thecommandlinezensendeventcanbeused(youshouldensureyouarethezenossuser).Thistakesparameters:
d device p component k eventClassKey s severity c eventClass y eventKey i IPaddress h help o =(foranyotherattribute;canhavemultipleo) monitor collectorthiseventcamefrom port=PORT defaultis8081 server=SERVER defaultislocalhost auth=AUTH defaultisadmin:zenoss Theremainderofthelineaftertheseoptionsisusedforthesummaryfield
(strictlytheMessagefieldintheGUIdialoguepopulatestheeventsummaryfield)
ThecoreautodeployscriptdeliveredwithZenoss4.2.3hasnewfunctionalitytoincreasesecurityonaZenossinstallation.FormanyyearstheZenossuserofadminwithapasswordofzenosshasbeenconfiguredasstandard.Thenewinstallationscriptchangesthis,generatingarobustpasswordwhichisstoredinseveralconfigurationfilesin$ZENHOME/etc,includingglobal.confandhubpasswd.
54 EventManagementforZenossCore4Skills1stLtd 23January2013
Figure28:Dialoguetocreateatestevent
-
zensendeventisastandalonePythonutilityin$ZENHOME/binthatcommunicateswiththezenhubdaemon.Noteintheusagedescriptionabove,thatthedefaultauthparametervalueisadmin:zenoss;typicallythismeansthatzensendeventcommandswillfailwithanUnauthorizedmessageunlesstheauthparameterisaddedwiththecorrectuserandpassword,foundin$ZENHOME/etc/hubpasswd.
Adiscussiononmodifyingzensendeventtoautomaticallylookupthecorrectauthenticationparameters,canbefoundontheZenosswikiathttp://wiki.zenoss.org/Zensendevent_in_Zenoss_4.2.3ThecodeissuppliedinAppendixA.
6.2 Regex in event mappingsTheRegexelementofaneventclassmappingcanbeusedtoparsethesummaryfieldoftheincomingevent,whichispresentedbytheparsingdaemon(zensyslog,zeneventlog,zentrap).TheRegexelementusesthePythonformatforregularexpressionsandcanusethePythonnamedgroupsyntaxtonotonlycheckforliteralstringsbutalsotodefineregularexpressionsforvariablepartsofastring,andassociatethatvariablepartwithaname.VariablepartsofthestringarecapturedintoPythonnamedgroupsthismeansthat:
Youcanhaveoneexpressionmatchlotsofsimilarbutdifferentincomingevents
Thevariablepart(typicallybetweenthe(?Pand\S+))canbepassedtotherestoftheeventprocessingmechanismasanamedfieldoftheevent.
Thus,intheproductshippeddropbeareventmappingfor/Security/Login/Fail,theRegexisasfollows:
exitbeforeauth\(user'(?P\S+)',(?P\S+)fails\):Maxauthtriesreached
(?P\S+)willparsethecharactersafteruser'uptothenextsinglequoteandplacethatstringintotheeventKeyfieldoftheevent.Similarly(?P\S+)willparsethestringthatfollowsacommaandspaceandisendedbyspaceandfails,intoaneweventattributecalledfailures.
Matchingtheliteralstringrepresentingabracketrequiresthebackslashescapeorthebracketwillbeinterpretedasametacharacter.
TherestoftheeventsummarymustmatchtheliteraltextintheRegex;however,othertextcanappearbeyondtheendaftertriesreached.
TheExampleboxshouldshowsasampleeventsummarythatismatchedbytheregularexpressionintheRegexbox.IfyouattempttoSavearegexthatdoesnotmatchtheexample,theregexfieldwillbeshowninred.
FormoreinformationonPythonregularexpressions,seehttp://docs.python.org/2/library/re.html.
23January2013 EventManagementforZenossCore4Skills1stLtd 55
-
SeeFigure29foranexampleofamorespecificmapping,su_root,fortheeventclass/Security/Su.Theregexisusedtoensurethatthesummaryhasthestringpam_unix(su:auth):authenticationfailure;followedbysomefixedandsomevariableelements.
pam_unix\(su:auth\):authenticationfailure;logname=(?P\S+)uid=(?P\d+)euid=(?P\d+)tty=(?P\S+)ruser=(?P\S+)rhost=\s+user=(?P\S+)
Theeventsummaryfieldcanbeparsedtogeneratenew,userdefinedfieldsfortheeventwhichwillbeshowninthedetailsoftheeventandcanbeusedinanysubsequenteventtransforms.
Additionally,theConfigurationPropertyofzEventSeverityhasbeensettoWarningforthismapping.
56 EventManagementforZenossCore4Skills1stLtd 23January2013
Figure29:EventmappingdialoguewithRegexforauthenticationfailure
Figure30Eventdetailsforauthenticationfailureeventshowingneweventfieldscreatedbytheregex
-
TheRegexelementisonlyusedifboththeeventClassKeyandtheRule(ifany)aresatisfied.IftheRulefails,theRegexwillnotbetested,norwillanynamedgroup,userdefinedfieldsbegenerated.IfaRuledoesnotexistandtheRegexdoesnotmatch,theuserdefinedfieldswillnotbegeneratedandtheeventclassmappingtothiseventclasswillfail.Noeventtransformswilltakeplace.IfaRuledoesexistandissatisfiedbuttheRegexfailsthenanyuserdefinedfieldswillnotbegeneratedbuttheeventclassmappingwillbesuccessfulandanymappingtransformwilltakeplace.
6.3 Rules in event mappingsTheRuleelementofaneventclassmappingusesPythonexpressionstotestanyinstantiatedfieldoftheincomingeventagainstavalue.ExpressionscanbecomplexincludingPythonmethodcallsandlogicalANDsandORs.Thedefaulteventfieldsthataredefined,aregiveninAppendixD3oftheZenossCore4AdministrationGuide.Notethatsomeofthesefieldsarenotactuallyavailableateventmappingtimenotablyevid,stateChange,count,dedupid,firstTime,lastTimeandeventClassMapping.
TheRuleelementcanalsousePythonexpressionstotestforvaluesofattributesofthedevicethatgeneratedtheevent.SomeofthemethodsandattributesthatareavailablefordevicesaredocumentedinAppendixD2oftheZenossCore4AdministrationGuide,underthesectiononTALESexpressions(TemplateAttribute
23January2013 EventManagementforZenossCore4Skills1stLtd 57
Figure31:Eventmappinglinetest,showingcomplexRuletestingeventanddeviceattributes
-
LanguageExpressionSyntaxispartofZope.ZopeistheapplicationserverthatZenossisbuilton).
TheRuleelementwillonlybeusediftheeventClassKeyfieldinthemappinghasachievedamatchwiththeincomingevent.Afterthat,ifaRuleexists,itmustbesatisfiedbeforethismapping(andhenceclass)isapplied.
6.4 Other elements of event mappings TheExampleelementofaneventclassmappingisasamplestringthatisusefulwhenconstructingaRegex.TheRegexwillturnrediftheRegexdoesnotmatchtheExamplestringwhentheSavebuttonisused.
TheExplanationandResolutionelementsofaneventclassmappingarestringsthatcanbeconfiguredtoprovidefurtherinformationtoZenossusers.Theyappearintheeventdetail.Notethattheseelementscanonlybeliteralstrings;theycannotuseeitherstandardoruserdefinedfieldsfromtheevent.
ThecombinationofeventClassKey,RuleandRegexdeterminetheeventclassthatwillbeassociatedwiththeincomingeventandwhattransforms(ifany)willtakeplace.Theremaystillbemultiplecombinationsofthesethatsatisfyanygivenincomingevent.Ifso,theSequencemenuisusedtodecidetheprecedenceofevaluationofmatchingeventmappings.Themappingswillbetestedfromthelowesttothehighestsequencenumber.Onceamatchisfound,anysubsequentmappings(withhighersequencenumbers)willbeignored.Generally,amappingwithmorespecificmatchingcriteriawillhavealowersequencenumber.
Intheexamplesaboveforthe/Security/Suclass,thegenericsumappinghassequencenumber1andthemorespecificsu_rootmappinghassequence0.
Aparticularexampleofeventmappingsthatusesequencenumbers,istheeventclassmappingcalleddefaultmappingwhichmusthaveaneventClassKeyofdefaultmapping.Thereareatleast6mappings,allcalleddefaultmapping,outofthebox.Eachmapstoadifferentclass.AdefaultmappingisaspecialcasethatisusedbytheeventmappingprocessifnomatchcanbefoundfortheeventClassKeyfield(notethatiftheeventClassKeyfielddoesnotexistthennomappingatallwillbeapplied).InthecasewhereaneventClassKeymatchisnotfound,themappingprocessreevaluateslookingforamatchwiththespecialeventClassKeyofdefaultmapping.Itispossibletocreatenewmappings,eitherwiththenameofdefaultmappingor,indeed,withadifferentname,providedtheeventClassKeyisdefaultmapping.Thesequencenumbersofallsuchdefaultmappingsshouldbeadjustedtoprioritisethesedefaultmappings.
7 Event transformsTransformscanbeusedtomodifyfieldsofanevent,createnew,userdefinedfieldsorfieldscanberetrievedfromeventsalreadyintheMySQLdatabase.
58 EventManagementforZenossCore4Skills1stLtd 23January2013
-
7.1 Different ways to apply transformsYoucanhavesimpleassignmentsoffieldvaluesorsetthembasedoncomplexPythonprograms.Thetransformmechanismcanbeappliedintwoways:
eventclasstransforms
eventclassmappingtransforms
PriortoZenoss2.4,aneventclasstransformwasonlyusedforeventsinserteddirectlytothatexacteventclassbytheparsingmechanism(zenping,zenperfsnmp,zencommand,AddEventwithEventClassspecified,etc).Ifatransformexistedinaneventclassmappingthatwasused,theeventclasstransformwasnotused.
Zenoss2.4introducedcascadingeventtransforms.Thischangedthingsintwoways.Givenaneventclass/Toptestwithasubclassof/T1,ifaneventarrivesthatalreadyhasclass/Toptest/T1,thentheToptesttransformwillbeapplied,followedbytheT1transform.Ifaneventarrivesthatdoesnothaveapreallocatedclassbutwhoseeventclassisdeterminedtobe/Toptest/T1,bytheRule/Regexoftheeventclassmapping,t1,thentransformswillbeappliedintheorder:
Toptestclass>T1class>t1eventclassmapping
Itisperfectlypossibleforatransformtouseuserdefinedeventfieldsinstantiatedbyearliertransforms;however,beveryawarethatifanystatementinatransformfails(perhapsbecauseafielddoesn'texist),thentheprocessingofthattransformwillstopatthatpointandnofurtherstatementswillbeexecuted.Anyfurthertransformswillbeexecuted(atleastuntilanerrorisreached).
AlltransformsareexecutedoncetheRuleandRegexelementsofamappinghavebeensuccessfullytestedandafterdeviceandeventcontexthavebeenapplied.Thus,attransformtime,mostofthestandardeventfieldsareavailable,exceptthosepopulatedatdatabaseinsertionstime(evid,stateChange,eventState,dedupid,count,eventClassMapping,firstTimeandlastTime).AnyuserdefinedfieldscreatedbytheRegexarealsoavailable.
Eventclasstransformscanbeusefulonthe/Unknownclasstoselectivelychangetheclassforeventsthatwouldotherwisebe/Unknown.
Notethatifatransformtriestoreferenceafieldofaneventthatdoesnotyetexist(likecount)thenthatlineofthetransformandanysubsequentlineswillbeignored.Suchanerrorwillnottriggeranyerrormessagesinthetransformdialogue.Transformsareimplementedbythezeneventddaemonsoinspecttheendof$ZENHOME/log/zeneventd.logtoseetheerrormessagereportingtheabsenceoftheattribute.
AclasstransformisconfiguredfromtheActioniconatthebottomofthelefthandmenuforaneventclass.
23January2013 EventManagementforZenossCore4Skills1stLtd 59
-
AmappingtransformisspecifiedaspartofthesameeventmappingdialoguethatdefinestheRuleandRegexfields.Ineachcase,ifthePythonsyntaxisincorrect,whenyouusetheSavebutton,thenthetransformisalldisplayedinredtext,indicatinganerror.
Figure31onpage57showedaneventmappingcalledlinetestwhichincludesatransformtocreateseveraluserdefinedeventfields,somebasedonvaluesfromtheeventandsomewithvaluesfromthedevicethatgeneratedtheevent.Theeventsummaryfieldissettoastringconstructedfromliteraltext,standardeventfieldsanduserdefinedfields.
evt.myDevId=device.idevt.mySnmpSysLoc=device.snmpLocationevt.mySnmpSysContact=device.snmpContactevt.mySnmpStatus=device.getSnmpStatusString()evt.summary="Problemis%sondevice%s.Pleasecall%s"%(evt.summary,
evt.myDevId,evt.mySnmpSysContact)Mostoftheuserdefinedfieldsareassignedtosimpleattributesofeithertheeventorthedevice;forexample,device.snmpContact.ThelinebeforetheenddemonstratesusingaPythonmethodtogetvalues;forexampledevice.getSnmpStatusString()(notethe()attheendthisisthecluethatitisamethodratherthananattribute).
7.2 Understanding fields available for event processingSohowdoesoneworkoutwhatattributesandmethodsareavailable?TheZenossCore4AdministrationGuidedocumentstheTALESEventAttributesinAppendixD3butthisisonlyastartingpoint.
Similarly,AppendixD2documentsTALESDeviceAttributesandmethodsbutthisinformationisveryincomplete.
Whenzeneventdisprocessinganevent,strictlyitisworkingonanumberofPythondictionariesthatmakeupaZepRawEventProxyobjectclass.Rememberfromthearchitecturesectionthatzeneventdtakeselementsfromtheraweventsqueue,processesthemandoutputstheresulttothezeneventsqueuetobefurtherprocessedbythezeneventserverdaemon(Figure12,Zenoss4eventarchitecture).Themessagesontheraweventqueue(likeallotherqueuemessages)areblobsofbinarydata.
Thereareanumberofmodulesin$ZENHOME/lib/python/zenoss/protocolsthatmanipulatethismessagedatausingGoogleprotobufsasadatainterchangeformatforthestructuredqueuemessagedata.
$ZENHOME/Products/ZenEvents/events2containsthreePythonfilesthatarecrucialforunderstandingthedetailsofhowzeneventdprocessestherawevent:
processing.py
fields.py
proxy.py
60 EventManagementforZenossCore4Skills1stLtd 23January2013
-
$ZENHOME/Products/ZenEvents/zeneventd.pyhasanumberofpipelinesthataneventpassesthrough.Theireffectcanbeseenbeanalysingzeneventd.logiftheDebuglogginglevelisturnedon.
processing.pycontainsthecodetoimplementeachofthepipelinestagesexecutedbyzeneventd.Therearemethodstoprocessesarawevent,adddeviceandeventcontext,processruleandregextoestablishaneventclass,andtoperformtransforms.Thereisalsoamethodtogeneratethefingerprintfield.
23January2013 EventManagementforZenossCore4Skills1stLtd 61
Figure32EventPipelineProcessorobjectclassinzeneventd.py
-
62 EventManagementforZenossCore4Skills1stLtd 23January2013
Figure33EventFieldobjectclassin$ZENHOME/Products/ZenEvents/events2/fields.py
-
$ZENHOME/Products/ZenEvents/events2/fields.pycontainsobjectclassdefinitionsfor:
EventField
TheEventFieldattributesmatchupwiththebaseMySQLdatabasefieldsinzenoss_zep.
TheActor,DetailandTagfieldsaredefinedassubclassesoftheobject
EventSummaryField
Hastheadditionalfieldsthatarepopulatedwhentheeventisinsertedintothezenoss_zepdatabaseevent_summarytable.
ZepRawEventField
HasthesamefieldsasEventFieldbutalsohasclear_event_classasthatisneededbythezeneventdprocessingpipelinesasitispartoftheeventcontext.
Notethatthedefinitionsinfields.pyarenothelpfulwhendecidingwhatattributesareavailabletotransforms;thesearethefieldsonefindsinthezenoss_zepdatabase.
7.2.1 Event Proxies$ZENHOME/Products/ZenEvents/events2/proxy.pyisthekeytounderstandingwhatattributesareavailablewhenwritingrulesandtransforms.proxy.pyprovides
23January2013 EventManagementforZenossCore4Skills1stLtd 63
Figure34EventSummaryFieldandZepRawEventFielddefinitions
-
translationsbetweenencodedformatsofeventsandahumanreadableJSON(JavaScriptObjectNotation)format.
Asfaraspossible,theattributespresentedbyaproxyarethesameinZenoss4astheywereinpreviousversions.
64 EventManagementforZenossCore4Skills1stLtd 23January2013
Figure35EventProxydefinitionin$ZENHOME/Products/ZenEvents/events2/proxy.py
-
AnEventProxyisseveralPythondictionaries:
Themainbodyoftheeventisadictionarycalled_event
Adetailsdictionary
An_tagsdictionary
Adictionaryfor_clearClasses
Adictionaryfor_readOnlyattributes
TherearealargenumberofPython@propertydecoratorconstructswhosepurposeistopresentanattributeusingamethod,forexample:
@propertydefdevice(self):returnself._event.actor.element_identifier
definesanattributecalleddevicewhichisdeliveredbyamethodthatreturnsthevalueoftheevent'sactor'selement_identifier.deviceisthefieldthatwehave(havealwayshad)tomanipulateintransforms.
The@propertydefinitionsattheendofFigure35showsimplerdefinitionsthatreturnthevalueofabasicfieldofanevent(usingtheEventFielddefinitionsdefinedinfields.py).
WhenauserviewseventdetailsusingtheZenossGUIoraccessesdatafromfromtheevent_summarytableofthezenoss_zepdatabaseusingtheJSONAPI,theeventdatapresentedisanEventSummaryProxy,whichisaJSONformat.TheEventSummaryProxyinheritsfromtheEventProxybutalsohasattributesthatareaddedondatabaseinsertion:
evid
stateChange
clearid
firstTime
lastTime
count
ownerid
eventState
TheEventSummaryProxywasoriginallydesignedwithanideaofkeepingalleventdata,treatingduplicatesasmultipleoccurrenceswithintheEventSummaryProxy;howeverthescalabilitywasnotfeasibleso,inpractisethefieldsofaneventareinthezero'thelementofanEventSummaryoccurrencelist.
23January2013 EventManagementforZenossCore4Skills1stLtd 65
-
proxy.pyalsodefinesaclassforZepRawEventProxywhichinheritsfromEventProxy.TheadditionalpropertiesforZepRawEventProxyarefor_ClearClasses,_actionandeventClassMapping.
Itistheattributesdefinedinproxy.pyfortheZepRawEventProxyobjectclassthatareavailableforuseinrulesandtransforms.
7.2.2 Event DetailsSowhathappenstoauserdefinedeventattributegenerated,say,bythevarbindsthatcomeinonanSNMPTRAP?
RememberthattheEventProxyhasanumberofdictionaries,includingadetailsdict