perfSONAR:ALookAhead

AndrewLake,ESnetMarkFeit,Internet2

http://www.perfsonar.netOctober16,2017

Overview

• Introduction• perfSONAR 4.0Recap• perfSONAR 4.0.2(November2017)• perfSONAR 4.1(Early2018)

WhatisperfSONAR?• perfSONARisatoolto:

• Set(hopefullyraise)networkperformanceexpectations• Findnetworkproblems(“softfailures”)• Helpfixtheseproblems

• Allinmulti-domainenvironments• Over2000publichostsonmanydifferentnetworks• http://stats.es.net/ServicesDirectory/

• Theseproblemsareallharderwhenmultiplenetworksareinvolved• FocusonResearchandEducation(R&E)Networking,1Gbpslinksorhigher

• perfSONAR providesastandardwaytopublishactiveandpassivemonitoringdata• Thisdataisinterestingtonetworkresearchersaswellasnetworkoperators

3

perfSONAR Bundles• perfsonar-tools• Justthemeasurementtools:iperf,iperf3,nuttcp,pScheduler client,owamp

• perfsonar-testpoint• Tools+pScheduler,mesh-config,LookupServiceregistration

• perfsonar-core• testpoint +esmond (forstoringresults)

• perfsonar-toolkit• Perfsonar-core+Web UI,scripts toapplytuningandsecuritysettings• Availableasafullsuiteoftoolsfor CentOSandDebian• AvailableasCentOS7ISO

October16,2017 ©2014,http://www.perfsonar.net 4

perfSONAR 4.0• ReleasedApril17,2017

• Majorreleasewithmanynewfeatures:• pScheduler – All-newsystemforschedulingandsupervisingmeasurements• Graphs – Cleanerdisplayofseveraldatatypes• MaDDash 2.0– Addedalertingfeatures.• OperatingSystems– SupportforCentOS7andDebian 8

• 75%+ofregisteredsystemson4.0ornewer

perfSONAR 4.0.1

• ReleasedAugust15,2017

• Minorreleasewithprimarilybugfixesandsmallchanges:• FullUbuntu 16andDebian 9support• AbilitytousecustompScheduler port(defaultis443)• ImprovedpSchedulerlimits

• Feedbackcontinuestobepositiveandconstructive• Keepitcoming!

ImportantDates• October17,2017

• perfSONAR 3.5end-of-life• Web100end-of-life

• November2017• perfSONAR4.0.2finalrelease Currentlyinbeta

• Q12018• perfSONAR 4.1released.CentOS 6packageswillnotbereleasedforperfSONAR 4.1

• Q32018(6monthsafter4.1release)• perfSONAR 4.0endoflife• CentOS 6versionsofperfSONAR nolongersupported

*Alldatessubjecttochange

Newin4.0.2

pScheduler:StreamingArchiving

• Per-runexecutionmodelledtohighsystemloadonhigh-volumesystems.

• Archiverpluginsnowoperateinstreamingmode:• Plugincomponentsstartedonceandfedmultipleresults• PoolsizemanagedbypSchedulerbasedonutilization

• Significantloadreductionseenwhendeployed.

pScheduler:SNMPPlugins

• DevelopedbyUniversityofMichiganITS• Collectdevicedatainparallelwithothertests• Sendresultstomanagementsystem(preferstrapsasinput)

• Tests• snmp-get One-shotfetchofSNMPMIBvalues• snmp-getbgm Long-termfetchanddeltadetection• snmp-set SetSNMPvalues(“extras”bundle)

• Tools— pysnmp,net-snmp-set• Archiver– snmptrap Generatetrapsfromresultdata

pScheduler:jq Support

• Toolforin-linemanipulationofJSON•Manybuilt-infunctionsandconstructsfortransformation,decisionmakingandoutput.•Addsend-userprogrammabilityinconfigurationandtaskspecificationinsteadofinstalledcode.• Infoanddocs:https://stedolan.github.io/jq

pScheduler:jq Support• Archiving— TransformresultJSONonthewaytoanyarchiverforsystemsthatrequirecustominput.• LimitSystem•Makepass/faildecisionsonproposedtasks• TransformdownloadedJSONCIDRliststoplain-textformatwithoutdependingonanothersystem.

• Library— CollectionofstandardfunctionsusefulwithpScheduler.• LookforaworkshopinearlyDecember.

pScheduler:Contexts

• Contextsallowmodificationoftheexecutioncontextsoftoolsbeforetheyarerun.• Thinkchroot forotherthings.• Sameprocessandexecutioncontextallthewaythrough

• Currentlyoneplugin:LinuxNetworkNamespace.

•Otherscanbeaddedbywritingplugins.

pScheduler:OtherChanges

• Toolsfortroubleshootinglimitconfigurationfiles

•BettersearchingintheRESTAPI• ImprovesperformanceofMeshconfig,futureapplications

• Theusualraftofminorenhancementsandfixes

What’sPlannedfor4.1

pScheduler:ResourceManagement

•ConfigurablerangesofTCPandUDPports

•Assignedandusedduringtesting

•Greatlysimplifieson- andoff-hostfirewallconfigurations

pScheduler:JobControl

•Currentimplementationrequiresthatmeasurementsruntocompletion.

•Stoprunsmidstream•Preemptpendingruns(DetailsTBD)•UsefulforgettingCLItaskstorunsooner

pScheduler:Remotely-SupervisedTasks

• Remotely-supervisedtaskswithlocalrunarchiving• NodeAsupervisesataskonB,retrievesandarchivesresults.• Solvesreachabilityproblemsforsomeusecases

NodeA

NodeB

NodeC

Archive

Firewall

pScheduler:TaskRewriting

•Makeon-the-flychangestotasksatingesttodothingsthelimitsystemcan’t•SimilartoApache’sURLrewriter

•Partofthelimitsystem•Workwillbedonewithjq transforms

TWAMP

• Two-WayActiveMeasurementProtocol(RFC5357)• SimilartoOWAMP• Doesacombinedone-wayandRTTmeasurement• RTTmeasurementsmoreaccurateinthefaceofclockvariation• twampd runningtoprocessmeasurementrequests• twping todoameasurementfromCLI• Streamingmeasurementswithsomethingsimilartopowstream

• DevicesfromseveralmanufacturersincludeTWAMPsupport• Manymorenon-perfSONARtargetsforlatencymeasurements

pScheduler:TWAMP

• WillbeintegratedwithpScheduler• pscheduler-tool-twping• pscheduler-tool-twpstream (orpscheduler-tool-twostream)

• AbletoproduceresultsforlatencyandRTT• WillbeshownintheregularperfSONARgraphsastheothertools

• WorkmadepossiblethankstopriorworkbyRobShearmanatBrocadeandDuncanEastoe.

pScheduler:MoreNewPlugins

•Networktrafficcapture•HTTPresponsetime

•Others•Weneedyourideas.Whatdoyouwanttomeasure?•Weneedyourparticipation. Pluginscanbewrittenbythirdparties.

pScheduler:BWCTLRetirement

•BWCTLdaemonwillnolongerberun•Backward-compatibilitycodewillberemovedfrompScheduler

•Checkyourfirewallsfor443• SomesitesonlyhaveBWCTLportsopen• Systemswerefallingback

Docker Support• GettingperfSONAR runningincontainersisapopulartopiconuserlist• CurrentlyhavebetaDocker imagesoftoolsandtestpoint bundles

• https://hub.docker.com/u/perfsonar/dashboard/

• EvaluatingbestwaytobundlethesesinceperfSONAR ismanyservicesandDocker ingeneralwantsyoutorunoneservicepercontainer• StartingwithDocker duetopopularity,maylookatothercontainertechnologiesinfuture(andothersaremorethanwelcometodoitnow)

pSConfig:Re-imaginingMeshConfig

• MajorfeatureforperfSONAR 4.1willberefactoringofMeshConfigintopSConfig• TheintroductionofMeshConfig helpedshapealotoftoday’sdeployments• perfSONAR haschangedalotthepastfewyearsandMeshConfighasn’tkeptupineveryarea.• JustaspScheduler wastransformativeatthetestschedulinglayerin4.0,wehopepSConfig willbeequallyastransformativeforthetestconfigurationlayerin4.1

Howpeopleusemeshconfig today• MeshConfig generallyusedfor:• Mediumtolargeinternaldeployments(e.g.ESnet)• Mediumtolargecollaborations(e.g.WLCG)

• Madeiteasiertodeploytestsonlotsofhostsandvisualizeit…whichhasbeengoodbutinterestingside-effects• Over-testingnotuncommonmakingithardtoseepatterns

• Testdesignishardandprobablyimpossibletocompletelysolvebutcanhopefullyhelp• Clearersyntaxsotakeadvantageofthingslike“disjoint”topologiesandauto-discoveryfeatures.• Easiertowritetoolstoauto-generatetestsbasedontopologicalknowledge,etc

October16,2017 ©2017,http://www.perfsonar.net 27

CurrentMeshConfigAgent

October16,2017 ©2017,http://www.perfsonar.net 28

CurrentMeshConfigGUIAgent

October16,2017 ©2017,http://www.perfsonar.net 29

Codechangeswhenweaddplugin ornewoptions

pSConfig

• MeshConfig renamedtopSConfig• SingleJSONconfigurationformat• Noneedtochangeanycodewhenaddingnewtestorarchiver,willvalidateagainstpscheduler ormaddash checkprofiles• Terminologysuchastask,testandarchiver tobemadeconsistentwithhowpscheduler usesthem• CLIforbuildingmeshestoavoidsyntaxerrorsandprovidereal-timeguidanceonoptions.• Threepackages:

• psconfig-publisher– ToolstohelpeditandpublishtaskJSONfiles• psconfig-pscheduler – Daemontocreatepscheduler tasksfromJSONfiles• psconfig-maddash – Daemontoconfiguremaddash basedonJSONfiles

October16,2017 ©2017,http://www.perfsonar.net 30

October16,2017 ©2017,http://www.perfsonar.net 31

Newplugin =Nocodechanges!

October16,2017 ©2017,http://www.perfsonar.net 32

Newplugin =Nocodechanges!

BackwardCompatibility

• psconfig packageswillconvertanyconfig filestonewformat• Ifrunningoldmeshconfig agent…• psconfig-publisherpackagewillbeabletogenerateoldjson• Canuserewriterulesanduser-agentheadertoserve-upoldversion

• Ifrunningnewpsconfig agent…• Option1:Understandjustthenew(couldditchalotofoldcodesooner)• Option2:Detectformatandparseeach

• MeshConfig GUI• Generatebothandserve-upcorrectbasedonuser-agentheader

October16,2017 ©2017,http://www.perfsonar.net 33

GettingReadyfor4.1

• DroppingsupportforthefollowingOSes• CentOS 6• Debian 7

• YouneedtoupgradeifyouareononeoftheOSes andyouwant4.1• Keepinmind4.0supportdropped6monthsafter4.1release

• BWCTLsupportwillalsobegoingaway,thusbackwardwith3.Xcompatibilitygoesaway

Conclusion

• 4.0.2comingsoonwithsomenewfeatures• 4.1comingnextyearwithsomebiggerfeatures• Wouldlovefeedbackonplannedfeatureset

• UpgradetoCentOS 7,Debian 9orUbuntu 16ifyouhavenotalreadysincesupportgoesawaymiddleof2018• Wearelookingatretransmitbehavior,wouldlovehelpifyouhaveanyCentOS 7boxesinthewild

• Thankyouforyourtime!

EmailListsandReferenceMaterials

• AnnouncementLists:•https://mail.internet2.edu/wws/subrequest/perfsonar-announce

• UsersList(developersalsomonitor):•https://mail.internet2.edu/wws/subrequest/perfsonar-users

MailingLists…

37

UsefulURLs• http://docs.perfsonar.net/• http://www.perfsonar.net/• https://www.youtube.com/channel/UCjK-P49pAKK9hUrrNbbe0Sg• http://fasterdata.es.net/• http://fasterdata.es.net/performance-testing/network-troubleshooting-tools/

• https://github.com/perfsonar• https://github.com/perfsonar/project/wiki

38