bringing together esb and big data -...

Bringing Together ESB and Big Data

2

Bringing Together ESB and Big Data

Table of Contents

Why ESB and Big Data? ...................................................................................3

Exploring the Promise of Big Data and ESB ........................................... 4

Moving Forward With ESB and Big Data ...................................................5

Choosing the Right ESB Solution .................................................................6

Conclusion .............................................................................................................7

Big data is one of the hottest topics across IT and business. IDC has called big data one of the four pillars of the IT industry’s next dominant platform, along with mobility, social networking and cloud services.1 McKinsey & Co. has characterized it as “the next frontier for innovation, competition and productivity.”2 And Gartner called big data one of the most important technology trends of 2012, projecting that by 2015, more than 4.4 million IT jobs will be created globally to support big data initiatives.3

1 IDCPredicts2012WillBetheYearofMobileandCloudPlatformWarsasITVendorsVieforLeadershipWhiletheIndustryRedefinesItself,IDC,December2011

2Bigdata:Thenextfrontierforinnovation,competitionandproductivity,McKinsey&Co.,May2011

3 In‘bigdata,’Gartnerseesbigjobs—4.4millionby2015,SearchSOA,Oct.24,2012

3

BigdataisoneofthehottesttopicsacrossITandbusiness.IDChascalledbigdataoneofthefourpillarsoftheITindustry’snextdominantplatform,alongwithmobility,socialnetworkingandcloudservices.McKinsey&Co.hascharacterizeditas“thenextfrontierforinnovation,competitionandproductivity.”AndGartnercalledbigdataoneofthemostimportanttechnologytrendsof2012,projectingthatby2015,morethan4.4millionITjobswillbecreatedgloballytosupportbigdatainitiatives.

Oneofthedrivingforcesbehindbigdataistheopportunityfororganizationstouseadvancedanalyticstotakeadvantageofthemassiveamountofunstructureddatabeingcreatedbothinsideandoutsidetheenterprise.Email,socialmedia,tweets,customerinteractions,collaborativesoftwareandotherapplicationscreateawealthofnewinformationaboutuserinterests,buyingpatterns,habitsandothertrendingdatathatorganizationscouldusetoboostbusinessperformance—iftheyareabletoharnessit.AccordingtoIDC,thenumberoffilesbeingcreatedandstoredismorethandoublingeverytwoyears,and90%ofthatisunstructureddata.4

However,inordertorealizethepromiseofbigdata,organizationsmusthaveinplaceanoverallstrategyfordatamanagementthatisbuiltonasoundfoundationofunifiedandintegratedmanagementofalldata.Suchanapproachmuststartwiththetransac-tionaldatathatiscriticaltostrategicbusinessinitiativessuchascustomerrelationshipmanagement(CRM),enterpriseresourceplanning(ERP),supplychainmanagement(SCM)andbusinessprocessmanagement(BPM).Centraltoanyunifiedapproachtointegrateddatamanagementistheabilitytocapture,analyzeandactupondatafromawiderangeofapplicationsacrossheterogeneousandhighlycomplexenvironments.Oneoftheoperationaltoolsthatcanenablethistypeofenterprisewidedataintegrationforbigdatainitiativesisanenterpriseservicebus(ESB).

Why ESB and Big Data?

AnESBprovidesaflexibleconnectivitylayertoimplementtheinteractionandcommu-nicationsbetweenheterogeneousapplications.WithanESBinplace,organizationscanreliabilityandsecurelyintegratedistributedsystemsandremotelocationswhilereducingthenumber,sizeandcomplexityofapplicationinterfaces.WithcertainESBproducts,specificallythosefromTalend,organizationscanuseasingleuserinterfaceandcentral-izedconsoletospeedthedeploymentandupgradeprocessforWebservices,RESTapplications,dataservicesandmessagingroutes.

4ExtractingValuefromChaos,IDCDigitalUniverseStudy,June2011

4

HowdoesanESBenablebigdatainitiatives?Severalways:

1. Helps address the issue of variety, including the integration of structured and unstructured data.Real-timeinformationcancomefromawiderangeofsources,includingsocialmedia,email,mobiledevicesandevenembeddedsensorsinenergymeters,carsandotherobjects.Butfordata-drivenorganizationstoaccess,analyzeandactuponthatdata,itmustbeintegratedintotheexistingapplicationenviron-ment,includingyourERP,CRMandSCMsystems.

2. Helps address complex technological challenges.Toachieveasuccessfulbigdatainitiative,organizationshavetoacquire,integrateandmanageseveralbigdatatechnologies,suchasHadoop,MapReduce,NoSQL,Pig,Scoop,HiveOozieandothers.Integratinglargedatasetsofstructuredandunstructureddatacanbeanexpensivecustom-codingprojectthatwillalsobeextremelydifficulttomaintainandmanage.Conventionaldatamanagementtoolsfailwhentryingtointegrate,searchandanalyzebigdatasets.

Ifyouthinkabouthowbigdatacanimpactanorganization’sabilitytodelivercompetitiveadvantagethroughreal-timeanalyticsandactionableinformation,itiseasytoseetheadvantagesofaunifieddataintegrationplatformthatincludesanESB.Foranyorderthatcomesin—say,throughaSiebelCRMorSAPERPapplication—therewillbeaflowofdataandmovementacrosstheESB.Butwhatiftheorganizationhasaccesstoadditionaldatathatmightnotnecessarilybecapturedinthecustomerorderprocess?Ifithas,forexample,datacapturedthroughFacebook,Twitterorevenanemailinter-actionwiththatsamecustomer,thatinformationcouldbeusedtoupsell,offeranewproductorimprovethecustomerexperienceinanynumberofways.Thecombiningoftheworldofstructureddatawiththeworldofunstructureddataistherealpromiseofbigdata.

Exploring the Promise of Big Data and ESB

Formostorganizations,fulfillingthepromiseofbigdataissomethingthatisstillonthehorizon,particularlywhenyouconsiderthechallengeofcombiningbigdatainitiativesandunstructureddatawithexistingmission-criticalapplicationsplatforms.Bigdataisacomplextopic,bothasatechnologyinitiativeandasabusinessinitiative.Whilemostorganizationsareawareofbigdata,theyarenotnecessarilysurehowtodeployitandtakeadvantageofit.Assuch,manyenterpriseshaveproductteamsthatareinvesti-gatingsolutionsforintegratingbigdatawithexistingenterpriseapplications.

5

Asorganizationsexplorethepotentialforbigdata,theyarefindingdramaticandexcitingusecasesacrossawiderangeofindustriesandapplications,fromsentimentanalysistopredictiveanalytics,riskmanagement,fraudprevention,retailbanking,digitalmarket-ing,channeloptimization,targetedadvertisingandmany,manyothers.Herearejustacoupleofexamples:

n Retail:Oneofthebigdatausecasesthatmanypeoplecanrelatetoisinretail,wherethereareanumberofopportunitiesfororganizationsthatcanrapidlyutilizeadvancedanalyticstotargetspecificcustomerbehavior.If,forexample,retailersknowthatcertaintypesoftoys,clothingorelectronicshaveaparticularappealtoshopperswhogotostoresonBlackFriday,thedayafterThanksgiving,theycanadjusttheirinventories,structuretheirdiscountsandcreatepromotionalprogramsaroundspecificcustomerinterests.Theirpromotionscouldchangeduringthecourseoftheday,basedonobservedcustomerbehavior,andmarketingincentivescouldbepushedouttocustomersviasocialmediaastheyareshopping.

n Financialservices:Financialservicesfirms,withtheirmassiveamountsofdataandthehighvelocityoftheirapplications,arefindingwidespreadusecasesforbigdatainitiatives.Forexample,manyofthesefirmsarealreadycorrelatingdeepanalyticswithaccountingdata,positiontrackingandordermanagementsystemstodealwithpotentialroguetradingproblems.Theyarealsocorrelatingdatafromawiderangeofsources—manyofwhichareunrelatedtooneanother—tocatchfraudulentactivitiesmorequicklythantheycouldusingcurrentsystemsandmethodologies.

Whatthesescenarioshaveincommonistheopportunityfororganizationstodriveincrementalrevenue,enhanceprofitabilityandimproveoperationsandagilitybystrength-eningtheirabilitytoaccess,integrate,analyzeandactupondatacomingfromawiderangeofdisparateandheterogeneoussources.AndthatiswhereanoperationaltoolsuchasanESBcomestogetherwithbigdataanalyticstoenablenewandinnovativewaysofdoingbusiness.

Moving Forward With ESB and Big Data

OneofthebestwaystogetstartedwithESBandbigdataistoworkwithanopensourceESBsolutionsuchasTalend’s,whichoffersafullyunifiedandintegratedplatformthatcanaddressallofyourdatamanagementchallengesforstructuredandunstructured

6

data.Youwilltypicallyfindthatsuchopensourcesolutionsarelessexpensivethanproprietarysolutionsandprovideothersignificantadvantages,including:

1. Less vendor tie-in:Eventhoughasolutionmaybeopensource,it’snotcompletelyopen,andyouwillbebuyingintoyourvendor’sdevelopmentenvironment,whichwillincludeproprietaryextensions.TheextensionsandsupportofaparticularenvironmentareoneofthereasonsyouwouldwanttochooseavendorsuchasTalend,whichoffersafullysupportedandintegratedplatform.Butitisalsolessproprietarythanworkingwithanon-open-sourcevendor.IfyoudevelopintheopensourceTalendplatform,allofyourcodewillbereusablewithoutlicense.Youaremuchlesstiedintotheidiosyncrasiesofaparticularvendor.

2. Faster time to innovation:Theopensourcecommunitytendstoinnovatemuchmorequicklythantraditionalproprietaryvendors.Thecommunityiscommittedtofindingsolutionstoproblems,notprotectinganinstalledbase.Ifthereisarequirementtochangethesourcecodetoaddressanewopportunity,youwillgetitmuchmorequicklyinanopensourceenvironment.

3. Community support:Likewise,ifthereareanynewfeaturesthatarerequiredorbugsthatneedtobefixed,Talendhasalargenumberofcontributorsthatworktogetherinacoordinatedandmanagedcommunitythatprovidestraining,supportandunfetteredaccesstoinformation.EverythingthathappensintheTalendenviron-mentiscompletelytransparent.

Choosing the Right ESB Solution

ThemostimportantfactorinchoosingtherightESBsolutionistomakesurethatitcomesfromavendorthatoffersacompletedatamanagementandintegrationplat-formthatincorporatesESBwithmasterdatamanagement(MDM),BPM,dataqualityanddataintegration.Oneofyourkeygoalswithbigdataistogeta360-degreeviewofthecustomer,takingintoaccountalloftheinformationthatmightimpactabuyingdecisionoracustomerexperience.Forexample,sayacustomerhascreatedanonlineaccountwithyoure-commerceportalandhasbeenaddingcommentsor“likes”toacertainclassofproduct.Thisinformationcanbeusedtomarkettothatcustomerthenexttimeshevisitsyoursite,paysabillorreceivesaninvoice.

TalendistheonlyvendorthatoffersacompletedataintegrationsolutionthatincludesESB,MDM,BPM,dataqualityanddataintegrationonasingleplatform.Itprovidesanopensourceapproachandflexibleintegrationplatformforbigdata,whichenables

7

organizationstoeasilyconnectandanalyzedatafromdisparatesystemstohelpdriveandimprovebusinessperformance.Talend’ssolutionsaddressallbigdatachallengesandopportunities,includingbigdataintegration,qualityandmanipulation,aswellasprojectgovernanceandadministration.

Conclusion

Asmostorganizationsknowalltoowell,managingdataisoneofthemostchallenging—andimportant—aspectsofdoingbusinesstoday.TheproliferationofheterogeneousITenvironmentsandmission-criticalapplicationshasdemandedsolutions,suchasESB,thatsimplifyintegrationandenableinnovation.Withouttheabilitytosimplyandcost-efficientlycommunicateandenablecross-developmentacrosstheseapplications,organizationswouldbecrippledandinnovationwouldbestifled.

Bringingallofthepotentialofunstructureddataintothealreadycomplexworldofdataintegrationisnosmallchallenge.It’snosmallopportunityeither.Infact,aswe’veseen,thepotentialofbigdatawillbeprofound,affectingallaspectsofourlivesandourbusinesses.

Butforbigdatatoreachthatpotential—andforunstructureddatatomakethatkindofgame-changingimpact—organizationshavetohavetherightdatamanagementtechnologyfoundationinplace,onethatisbuiltonaunifiedapproachandincludescriticaltools,suchasESB,toenableagilityandflexibilityacrossapplications,systemsanddatatypes.Onlythenwillthepromiseofbigdatatransformintotherealityofbigdata.