pre-con ed: better big data analytics with ca workload automation for hadoop
TRANSCRIPT
![Page 1: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/1.jpg)
BetterBigDataAnalyticswithCAWorkloadAutomationforHadoop
SimonPearsonCATechnologiesSeniorEngineeringServicesArchitectSession#:MFX31E
@peasi02#CAWORLD
WorkloadAutomation– OffTheFloorSessions
![Page 2: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/2.jpg)
2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.
Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.
ForInformationalPurposesOnlyTermsofthisPresentation
![Page 3: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/3.jpg)
3 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Abstract
BigData.BigDeal.Theneedtominemassivesetsofinformationforuniqueinsightsintocustomerbehaviours,competitiveplaysandmarketfluctuationshastransformedbigdatainitiativesintoimperative,business-criticalpriorities.
Theopen-sourceApache™Hadoop®platformhasrapidlyemergedasthedominantmeansbywhichbusinessesprocess,analyseandextractinsightsfromtheirgrowingsetsofdata.
WhileHadoopdoesincludeabasicschedulerthatdeliverssomeautomation,itisfocusedprimarilyonjobsthatrunonHadoopclusters,anddoesn’tintegratewellwithotherworkloadautomationengines.
CAWorkloadAutomationAdvancedIntegrationforHadoopmakesitpossibletointegrateHadoopwithtraditionaljobs.
SimonPearsonCATechnologiesSeniorEngineeringServicesArchitect
![Page 4: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/4.jpg)
4 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Agenda
INSTALLATION
CONFIGURATION
SUPPORTEDHADOOPJOBTYPES
EXAMPLES– DEFININGHADOOPJOBS
ANDFINALLY……
1
2
3
4
5
![Page 5: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/5.jpg)
5 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InstallationDownloadCAWAAIforHadoopfromsupport.ca.com
![Page 6: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/6.jpg)
6 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InstallationCheckthatyouhaveasupportedHadoopdistribution
![Page 7: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/7.jpg)
7 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InstallationSupportingtheseCAWorkloadEngines
FormoreinformationaboutusingAdvancedIntegrationforHadoop,refertothedocumentationforyourschedulingengine:
•CAWorkloadAutomationAEandWorkloadControlCenter•CAWorkloadAutomationDE•CAWorkloadAutomationESP
![Page 8: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/8.jpg)
8 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InstallationHavethecorrectversionofthesystemagentinstalled
![Page 9: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/9.jpg)
9 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InstallationOrelsethiswillhappen
![Page 10: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/10.jpg)
10 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InstallationMounttheISOandruntheinstaller
![Page 11: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/11.jpg)
11 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InstallationTheHadoopAIinstallerruns………
![Page 12: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/12.jpg)
12 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ConfigurationCreateagentdefinitionfortheHadoopplugin
![Page 13: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/13.jpg)
13 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ConfigurationCreateconnectionprofile
ThisidentifiestheDBIwanttoextractdatafrom
![Page 14: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/14.jpg)
14 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ConfigurationSecurityProfile– IDandPWforlogintosourceDBforSqoop jobs
![Page 15: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/15.jpg)
15 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ConfigurationSecurityProfile– AccessrighttoHadoopHDFSfilesystem
![Page 16: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/16.jpg)
16 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ConfigurationSecurityProfile– IDwithrightstoexecuteHivejobs
![Page 17: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/17.jpg)
17 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
SupportedHadoopJobTypes• HDFS- FileSystemactions
• Hive- Scriptexecution
• Pig- Scriptexecution
• Sqoop – DatabaseImport/Exportoperations
• Oozie – ExecuteOozie Workflows
![Page 18: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/18.jpg)
18 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- HDFSOperationssupported– CreateDir,CreateFile,DeleteFile,DeleteDir
![Page 19: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/19.jpg)
19 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- HDFSExample– Deletefile
HadoopconnectionandHDFSsecurityprofileselectedfromthosedefinedinpreviousstep
![Page 20: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/20.jpg)
20 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- HDFSExample– MakeDirectory
ThefourHDSFoperationsSupportedcanbeselectedfromthedropdownlist
![Page 21: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/21.jpg)
21 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- HDFSViewtheresultsviathejoblog
![Page 22: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/22.jpg)
22 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- SqoopOperationssupported:
![Page 23: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/23.jpg)
23 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- SqoopSimpleexample
![Page 24: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/24.jpg)
24 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- SqoopSpoolFile
![Page 25: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/25.jpg)
25 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- PigJobDefinition
![Page 26: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/26.jpg)
26 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- PigScripttoexecuteandsampledata
![Page 27: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/27.jpg)
27 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- PigSpoolFile
![Page 28: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/28.jpg)
28 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
IntegratingHadoopwithExternalWorkflows
![Page 29: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/29.jpg)
29 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
AndFinally…..WhatifyouarerunningCAWAAE?
![Page 30: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/30.jpg)
30 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
AndFinally…..ThesameHadoopjobtypesareavailableintheQuickEditor
![Page 31: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/31.jpg)
31 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
AndFinally…..HadoopjobtypesalsoavailableinApplicationEditor
![Page 32: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/32.jpg)
32 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Questions?
![Page 33: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/33.jpg)
SimonPearson
@cainc
Slideshare.net/CAinc
Linkedin.com/company/ca-technologies
CA.com
![Page 34: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop](https://reader031.vdocuments.mx/reader031/viewer/2022030316/587413571a28abcb5b8b4cc3/html5/thumbnails/34.jpg)
34 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Thankyou
Stayconnectedatcommunities.ca.com