cloudera user group chicago - cloudera manager: apis & extensibility

Download Cloudera User Group Chicago - Cloudera Manager: APIs & Extensibility

Post on 06-May-2015

326 views

Category:

Technology

0 download

Embed Size (px)

DESCRIPTION

This presentation was delivered by Cloudera systems engineer Matt Harris at the Chicago Cloudera User Group (CUG) meeting on December 3, 2013.

TRANSCRIPT

  • 1.ClouderaManagerAPIs& Extensibility Ma#HarrisSystemsEngineer December2013 1CONFIDENTIAL-RESTRICTED

2. ClouderaManager End-to-EndAdministraGonforCDH1 Monitor 2 Diagnose 3 Integrate 4 ManageEasilydeploy,congure&opGmizeclustersMaintainacentralviewofallacGvityEasilyidenGfyandresolveissuesUseClouderaManagerwithexisGngtools22013Cloudera,Inc.AllRightsReserved. 3. IntegraGngwithyourITMgmttools DatacenterOpera*onsVariousop*onsofintegra*ngClouderaManagerintoyourexis*ng Installa;on, DatacenterOpera*ons/ToolsMonitoring Deployment Aler;ng Tools tools Tools e.g.Orion, ClouderaManagerAPI e.g.Chef, e.gNagios, Tivoli,BMC Puppetetc. SNMPetc. IntroducedinCM4(June2etc. 012) Installa*on&deployment Monitoring SNMPAlerts IntroducedinCM4.5(Feb2013) HadoopOpera*ons Andmore Cloudera Monitoringtsquery(Feb2013) Manager User-denedtriggers/alarms(newforC5!) Serviceextensibility(newforC5!)32013Cloudera,Inc.AllRightsReserved. 4. ClouderaManager(CM)API 4APIaccesswasanewfeatureintroducedinClouderaManager4.0,providingprogrammaGcaccessto clusteroperaGons(suchasconguraGonandrestart)andmonitoringinformaGon(suchashealthand metrics). TheCMAPIisanHTTPRESTAPI,usingJSONserializaGon.TheAPIisservedonthesamehostandportas theCMwebUI,anddoesnotrequireanextraprocessorextraconguraGon.APIusershavethesame privilegesastheydointhewebUIworld. Docs&Examples h#p://cloudera.github.io/cm_api/ h#ps://github.com/cloudera/cm_api Java/Pythonclients h#p://blog.cloudera.com/blog/2013/05/how-to- automate-your-hadoop-cluster-from-java/ 2013Cloudera,Inc.AllRightsReserved. 5. ExamplesofintegraGonwithCMAPI Installa;on&Deployment Chef Puppet DellCrowbar h#p://blog.cloudera.com/blog/2013/08/how-to-deploy-hadoop-clusters-automaGcally-with-dell-crowbar-and-cloudera-manager/StackIQ h#p://web.stackiq.com/blog/bid/312064/StackIQ-Cluster-Manager-now-integrated-with-ClouderaWANdisconon-stopNNsetupSeveralothercustomers/partnersleveragingtheAPIsaspartoftheirinstall&deployment processMonitoring&Aler;ng OracleEnterpriseManager(viaBigDataAppliance) Nagios h#ps://github.com/cloudera/cm_api/tree/master/nagios h#ps://github.com/harisekhon/nagios-plugins/blob/master/ check_hadoop_cloudera_manager_metrics.plSNMPalertsintegraGonwithIBMNetcoolDevelop&Contributeyourplug-insusingCloudera ManagerAPI 52013Cloudera,Inc.AllRightsReserved. 6. ClouderaManagerMonitoringviatsquery IntroducedaspartofCM4.5release(Feb2013)GreatwaytoaddinteresGngcharts(above&beyondwhatisprovidedbydefault)andmonitor metricsthatarerelevanttoyourclustersThetsquerylanguage susedto ManagerGme-seriesdiatastore specifystatementsforretrievingGme-seriesdatafromtheCloudera Example:HowdoIcomparealldiskIOforalltheDataNodesthatbelongtoaspecicHDFSservice? selectbytes_read,bytes_wriZenwhereroleType=DATANODEandserviceName=hdfs1RetrievedGme-seriesdatacanbeplo#edviavariousopGonsline,bar,sca#er,heatmaps,tablelist etc.Extendingthisconcepttocreateuser-denedtriggers/alarms(newforC5!). Moredetails h#p://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera- Manager-DiagnosGcs-Guide/cm5dg_chart_Gme_series_data.html62013Cloudera,Inc.AllRightsReserved. 7. ExamplesofClouderaManagertsquery Example1:HowdoItrackthe aggregateClusterDiskIO? selectdt0(read_bytes_disk_sum), dt0(write_bytes_disk_sum)where category=CLUSTERandclusterId= $CLUSTERID Example2:HowdoIcompareCPU usageacrosshosts?selectdt0(total_cpu_user)/getHostFact(numCores,1)*100, dt0(total_cpu_system)/getHostFact(numCores,1)*100, dt0(total_cpu_nice)/getHostFact(numCores,1)*100, dt0(total_cpu_iowait)/getHostFact(numCores,1)*100, dt0(total_cpu_irq)/getHostFact(numCores,1)*100, dt0(total_cpu_so`_irq)/getHostFact(numCores,1)*100Create&Contributeyourtsqueries! h#ps://github.com/cloudera/cm_charGng_scrapbook 72013Cloudera,Inc.AllRightsReserved. 8. ClouderaManagerServiceExtensibility IntroducedinC5 SGllinBeta!Someaspects(espciallyParcelmgmt)availableinCM4.xExample:CollaboraGonwithSyncsorttodeployDMX-hlibrariesSinglemanagementconsoleforCDH,non-CDHservicesandISV applicaGonsSimilarlookandfeelasexisGngservicesEasytowrite(Java-free!)FlexibleIndependentreleasecycle 2013Cloudera, Inc. All Rights Reserved. 9. AnalogyfromOperaGngSystems(OS)world ISVsviewofOSSystemsManagement Package MgmtProcess/ Resource MgmtSecurity MgmtCoreOSkernel92013Cloudera,Inc.AllRightsReserved.Data Access Mgmt 10. BringingISVAppstoCDH ISVsviewofHadoopClouderaManager ParcelsResource MgmtSecurity MgmtCDKAPIsCoreHadoop/CDHkernel102013Cloudera,Inc.AllRightsReserved. 11. IntegraGngintotheClouderaProductPorpolio FeaturesExamples-Abilitytoeasilypackageanddistributebinaries/ jarsviaParcels-InformaGca -SyncsortResource Mgmt-AbilitytodeployapplicaGonsasstand-alone processesorviaYARN*ontheHadoopgrid -ResourceisolaGonofclusterresources-SAS -0xData -AccumuloSecurity MgmtClouderaManagerDescrip;onPackage MgmtISVs- SupportforKerberosMgmt - RolebasesaccesscontrolforTables/Viewsin Hive/ImpalaviaSentryDataAccess Mgmt-HDFSandHBaseAPIabstracGonand simplicaGonSystemsMgmt Manage MonitorNon-CDHApps Accumulo, Spark,Giraph etc.-Deployandupgrade(rolling)servicesandpkgs -ManageconguraGons -ProacGvehealthchecks -TrackresourceuGlizaGon -CustommetricschartsDiagnose-DistributedlogcollecGonandsearching -TagandtrackkeyeventsIntegrate-AccessoperaGonaltoolsviaAPI -SurfaceoverallclustermetricstoISVdashboard*SupportforYARNplannedaspartofCM5.xinFY14112013Cloudera,Inc.AllRightsReserved. 12. So..Howdoesitwork? AJSONlethatdescribesyourservice Setofcontrolscripts PackagedasaJARle Aspromised,Java-free2013Cloudera, Inc. All Rights Reserved. 13. Example:ClouderaManagerExtensions-Spark2013Cloudera, Inc. All Rights Reserved. 14. ClouderaManagerExtensions 2013Cloudera, Inc. All Rights Reserved. 15. ClouderaManagerExtensions:Spark2013Cloudera, Inc. All Rights Reserved. 16. ClouderaManagerExtensions:Spark2013Cloudera, Inc. All Rights Reserved. 17. ClouderaManagerExtensions:Spark2013Cloudera, Inc. All Rights Reserved. 18. TheCode #!/bin/bashname:spark, roles:[{ name:"master",CMD=$1 MASTER_PORT=startRunner:{ program:"scripts/control.sh", args:["start_master",case$CMDin (start_master) exec$SPARK_HOME/scripts/spark-start.shmaster""./params.proper;es"] },;;parameters:[{(*) echo"$;mestampDon'tunderstand[$CMD]"name:"master_port", type:"port",;; esacdefault:7077 }], congWriter:{ generators:[{ lename:"params.proper;es" }] }]2013Cloudera, Inc. All Rights Reserved. 19. NextSteps DocumentaGon&SDKaspartofC5Beta2 orlater(denitelybeforeGA!) WorkingwithselectISVs(SAS,Syncsort, 0xDataetc.)aspartofBetatofurtherne- tunethisfeature Develop&ContributeyourClouderaManagerservice extensibilityplug-ins! 2013Cloudera, Inc. All Rights Reserved. 20. Horizontal ExtensionSecurity ISVs0xDataSASSyncsortInformaticaRevolutionAPIOps Apps Capacity MgrService ExtensibilityVertical ExtensionVisionofCMExtensibilitySLA MgrCost OptimizerCDHCM SNMP APIOracle OEM20NagiosDellChef/ Puppet2012Cloudera,Inc.AllRightsReserved.AccumuloSparkGiraph 21. Q&A2013Cloudera, Inc. All Rights Reserved.