pan109098 - going splunking: using splunk for server log
TRANSCRIPT
PAN109098- GoingSplunking:UsingSplunkforserverloganalyticsinBlackboardLearn7/24/17(Monday)11:00AM- 284-285
PRESENTEDBY:CHRISBRAYBLACKBOARDADMINISTRATORATUNIVERSITYOFARKANSAS|IANGOHSOFTWAREENGINEERATJOHNSHOPKINSUNIVERSITY
Statementsregardingourproductdevelopmentinitiatives, includingnewproductsandfutureproductupgrades,updatesorenhancementsrepresentourcurrent intentions,butmaybemodified,delayedorabandonedwithout priornotice andthere isnoassurancethatsuchoffering,upgrades,updatesorfunctionalitywillbecomeavailableunlessanduntiltheyhavebeenmadegenerallyavailabletoourcustomers.
TheProblem
TheProblem
Goodmorning,
We’reseeinganerrormessagewhentryingtoreviewthefullresultsofaquizinthe“PrinciplesofEconomics– SU17.2”(ORG.SA.PRINCIPLESOFECONOMICS-SU17.2)communitysite. TheerrormessageisshownbelowandoccurswhenstudentsattempttoviewthefullresultsviaMyGradesandwhenmeandtheinstructor trytoviewtheattemptviatheFullGradeCenter. ThequizinquestionisModule5Graded.
Pleaseadvise. Thanks!
blackboard/apis/assessment/OrderingAnswerAttemptForreference,theErrorIDis4e1f842a-6f34-4657-8c76-d3d50bbb73a5.Tuesday,June27,20178:38:44AMEDT
Admins- Howoftendoyougetthisemail?
TheProblem
• Whatstepsdoyounormallytaketoanalyzetheissue?
•
AskTheAudience
TheProblem
1. Manylogs
2. LogFormats– lackconsistentformats,timeformats
3. MultipleServers– needtocentralizelogs
4. Expertise/Access– whocanaccessthelogs,whounderstandswhichlogdoeswhat?
Splunk
WhatisSplunk?https://www.splunk.com/
• “GoogleforLogfiles”-- HelgeKlein/https://helgeklein.com/blog/2014/09/splunk-work/
• “Schemaonthefly”
• Splunk EnterpriseCoreFeatures:
– CollectandIndexData
– Search,AnalyzeandVisualize
– Monitor,AlertandReport
• ProvidesPremiumSolutions:Security, ITServiceIntelligence,UserBehaviorAnalytics
Components ofSplunk
• Forwarders - Splunk softwarecaningestallkindsofdatatypesandsources.File-baseddatacanbesentvia forwarders thatresidedirectlyonthedatasources
• Indexer - AnindexeristheSplunk instancethatindexesdata.Theindexertransformstherawdataintoeventsandstorestheeventsintoanindex.
• SearchHead- Inadistributedsearchenvironment,thesearchheadistheSplunkinstancethatdirectssearchrequeststoasetofsearchpeersandmergestheresultsbacktotheuse
Managed&SelfHostedBlackboardEnvironments
ChrisBray/U.Arkansas–ManagedHosting• logsarezippedandsentoverdaily
• Splunk forwardingagenttransferslogsintolocalinstanceofSplunk
I.Goh/JohnsHopkins University- SelfHosted• useForwarderstosendlogstolocal instanceofSplunk
• prodwebservers(andourdev,testserversaswell)sendreal-timedata
WhataboutSAAS?
OfferingELK(Elasticsearch,Logstash,andKibana)?• PossiblylinktosomeoneelsedoingaDevCon onELK?
Alternatives toSplunk
• MS SystemCenterOperationsManager(SCOM) (Windows)
• Nagios (OpenSource)
• ELK(Elasticsearch,Logstash,andKibana)
BlackboardLogs– whatdoyousend,howlongisitkept/indexed
C.Bray /U.Arkansas- Manage Hosted• Blackboard
• tomcat/access-logs
• bb-email.log
• bb-authentication.log
I.Goh /JohnsHopkins University- SelfHosted• MicrosoftIIS(sourcetype: iis)
• Blackboard
• bb-services-log.txt (sourcetype:bb_services - mightbejust acustomlabelforus)
• tomcat/bb-access-log- txt(sourcetype:access_combined_wcookie)
• tomcat/stdout-stderr- log(sourcetype: log4j)
• Productionindexkeepssixmonthsofdata
Examples
SearchExamples
HeretherebeExampleshttp://bbadmin.uark.edu
ReportExamples
• StartupTimesofwebapp servers
• Search:host=hostname*source=“tomcat\\stdout-stderr-*.log""Blackboardapplicationserverreadytoacceptrequests”
• Report:
ReportExamples
• Geolocation:visualizewheretheBbStudentappishittingusfrom(beforewegetthemedu.comrequests)
• Search:host=hostname*"/webapps/Bb-mobile-bb_bb60/customAuthSuccess"|iplocation c_ip |geostats countBYc_ip
• Report:
AlertExamples
• Useareal-timealerttomonitoreventsoreventpatternsastheyhappen.Youcancreatereal-timealertswithper-resulttriggeringorrollingtimewindowtriggering
• WouldnotbeusefulintheManageHostedsituationiflogswillbedelayed(duetotransfer)
• JHU:weuseSCOMforalerts
• Example:theStartupTimesearchcouldbeusedtotriggeranemail/pager(orevenaPhilipsHuelight!)
UsesofSplunk outsideofIT
• UNLV– usingSplunk toanalyzelearningdataw/machinelearning
• https://www.splunk.com/en_us/resources/video.UzaWVuNjE60_AMjGA_NfnDfE2FGoIIFB.html#
• “Thediscoveryandminingofsuch(LMS)logsledhimtobuildadatadictionarythatenabledhimtoidentifytheevents,classifythem,andgaininsightsintotheactionsstudentswerelikelytotakeandwhichonespredictedtheirachievement“
Splunk Resources
GettingStartedwithSplunk
• https://www.splunk.com/en_us/download.html
• WhenyoudownloadSplunk Enterpriseforthefirsttime,yougetanEnterpriseTriallicensefor60days.ThisEnterpriseTriallicenseincludesallofthefeatures,butlimitstheamountofdatathatyoucanindexeachday.Thedailylimitis500MB.
• SearchTutorial-http://docs.splunk.com/Documentation/Splunk/latest/SearchTutorial/WelcometotheSearchTutorial
• QuickReferenceGuide- https://www.splunk.com/content/dam/splunk2/pdfs/solution-guides/splunk-quick-reference-guide.pdf
• Splunk Community- https://www.splunk.com/en_us/community.html
• SplunkLive!- http://splunklive.splunk.com/ (it’safreeevent!)
Splunk Free
• http://docs.splunk.com/Documentation/Splunk/6.3.2/Admin/MoreaboutSplunkFree
– Splunk Freeisdesignedforpersonal,adhocsearchandvisualizationofITdata.YoucanuseSplunkFreeforongoingindexingofsmallvolumes(<500MB/day)ofdata.Additionally,youcanuseitforshort-termbulk-loadingandanalysisoflargerdatasets--Splunk Freeletsyoubulk-loadmuchlargerdatasetsupto3timeswithina30dayperiod.Thiscanbeusefulforforensicreviewoflargedatasets.
Don’tforgettoratethissessionintheBbWorldapp.
Oh?TheSolution…
TheProblem
Goodmorning,
We’reseeinganerrormessagewhentryingtoreviewthefullresultsofaquizinthe“PrinciplesofEconomics– SU17.2”(ORG.SA.PRINCIPLESOFECONOMICS-SU17.2)communitysite. TheerrormessageisshownbelowandoccurswhenstudentsattempttoviewthefullresultsviaMyGradesandwhenmeandtheinstructor trytoviewtheattemptviatheFullGradeCenter. ThequizinquestionisModule5Graded.
Pleaseadvise. Thanks!
blackboard/apis/assessment/OrderingAnswerAttemptForreference,theErrorIDis4e1f842a-6f34-4657-8c76-d3d50bbb73a5.Tuesday,June27,20178:38:44AMEDT
Admins- Howoftendoyougetthisemail?
TheProblem– SolvingitwithSplunk 1
1. Searchfor‘4e1f842a-6f34-4657-8c76-d3d50bbb73a5’
TheProblem– SolvingitwithSplunk 2
2. Searchfor‘java.lang.NoClassDefFoundError’, overthelastsevendays
TheProblem– SolvingitwithSplunk 345
3. Seeit’sonlyappearingononehost(webapp)server
4. Takeserveroutofloadbalancer,restartservices.
5. Monitorforanyotherjava.lang.NoClassDefFoundError afterrestart