microservices with clojure: develop event-driven, scalable ... · chapter 3, microservices for...

MicroserviceswithClojure

Developevent-driven,scalable,andreactivemicroserviceswithreal-timemonitoring

AnujKumar

BIRMINGHAM-MUMBAI

Allrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem,ortransmittedinanyformorbyanymeans,withoutthepriorwrittenpermissionofthepublisher,exceptinthecaseofbriefquotationsembeddedincriticalarticlesorreviews.

Everyefforthasbeenmadeinthepreparationofthisbooktoensuretheaccuracyoftheinformationpresented.However,theinformationcontainedinthisbookissoldwithoutwarranty,eitherexpressorimplied.Neithertheauthor,norPacktPublishingoritsdealersanddistributors,willbeheldliableforanydamagescausedorallegedtohavebeencauseddirectlyorindirectlybythisbook.

PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthecompaniesandproductsmentionedinthisbookbytheappropriateuseofcapitals.However,PacktPublishingcannotguaranteetheaccuracyofthisinformation.

CommissioningEditor:RichaTripathiAcquisitionEditor:AiswaryaNarayananContentDevelopmentEditor:AkshadaIyerTechnicalEditor:AbhishekSharmaCopyEditor:SafisEditingProjectCoordinator:PrajaktaNaikProofreader:SafisEditingIndexer:FrancyPuthiryGraphics:JasonMonteiroProductionCoordinator:DeepikaNaik

Firstpublished:January2018

Productionreference:1230118

PublishedbyPacktPublishingLtd.LiveryPlace35LiveryStreetBirminghamB32PB,UK.

ISBN978-1-78862-224-0

www.packtpub.com

Tomymother,Mrs.InduSrivastava,myfather,Mr.DilipKumar,andtomylovelywife,Aishwarya,fortheircontinuoussupportandencouragement.AllthetimethatIhavespentonthisbookshouldhavebeenspentwiththem.FortripsthatwecanceledandforweekendsthatIspentatmydesk.

Tomyfamily,teachers,andcolleagues.Theyhaveextendedtheircontinuoussupport,providedcriticalfeedback,andmadeitpossibleformetofocusonthisbook.

mapt.io

Maptisanonlinedigitallibrarythatgivesyoufullaccesstoover5,000booksandvideos,aswellasindustryleadingtoolstohelpyouplanyourpersonaldevelopmentandadvanceyourcareer.Formoreinformation,pleasevisitourwebsite.

Whysubscribe?SpendlesstimelearningandmoretimecodingwithpracticaleBooksandVideosfromover4,000industryprofessionals

ImproveyourlearningwithSkillPlansbuiltespeciallyforyou

GetafreeeBookorvideoeverymonth

Maptisfullysearchable

Copyandpaste,print,andbookmarkcontent

PacktPub.comDidyouknowthatPacktofferseBookversionsofeverybookpublished,withPDFandePubfilesavailable?YoucanupgradetotheeBookversionatwww.PacktPub.comandasaprintbookcustomer,youareentitledtoadiscountontheeBookcopy.Getintouchwithusatservice@packtpub.comformoredetails.

Atwww.PacktPub.com,youcanalsoreadacollectionoffreetechnicalarticles,signupforarangeoffreenewsletters,andreceiveexclusivediscountsandoffersonPacktbooksandeBooks.

Contributors

AbouttheauthorAnujKumaristheco-founderandchiefarchitectofFORMCEPT,adataanalyticsstartupbasedinBangalore,India.Hehasmorethan10yearsofexperienceindesigninglarge-scaledistributedsystemsforstorage,retrieval,andanalytics.

Hehasbeeninindustryhacking,mainlyintheareaofdataintegration,dataquality,anddataanalyticsusingNLPandmachinelearningtechniques.HehaspublishedresearchpapersatACMconferences,gotafewpatentsgranted,andhasspokenatTEDx.

PriortoFORMCEPT,hehasworkedwiththeOracleServerTechnologiesdivisioninBangalore,India.

Iwouldliketothankmytechnicalreviewer,MichaelVitz,forhisvaluablefeedbackandthePackteditorialteamforanexcellentfeedbacklooptocomeupwithgoodqualitycontent.IwouldalsoliketothankmyteachersandFORMCEPTteammembers,whohavehelpedmeonvarioustopicscoveredinthisbook.Andespecially,Iwouldliketothankmyparents,mywife,andmyentirefamilyfortheircontinuousencouragement.

AboutthereviewerMichaelVitzhasmanyyearsofexperiencebuildingandmaintainingsoftwarefortheJVM.Currently,hismaininterestsincludemicroserviceandcloudarchitectures,DevOps,theSpringFramework,andClojure.

AsaseniorconsultantforsoftwarearchitectureandengineeringatINNOQ,hehelpsclientsbybuildingwell-craftedandvalue-providingsoftware.

HealsoisthewriterofacolumnintheGermanmagazine,JavaSPEKTRUM,wherehepublishesarticlesaboutJVM,infrastructure,andarchitecturaltopicseverytwomonths.

PacktissearchingforauthorslikeyouIfyou'reinterestedinbecominganauthorforPackt,pleasevisitauthors.packtpub.comandapplytoday.Wehaveworkedwiththousandsofdevelopersandtechprofessionals,justlikeyou,tohelpthemsharetheirinsightwiththeglobaltechcommunity.Youcanmakeageneralapplication,applyforaspecifichottopicthatwearerecruitinganauthorfor,orsubmityourownidea.

TableofContents

PrefaceWhothisbookisfor

Whatthisbookcovers

TogetthemostoutofthisbookDownloadtheexamplecodefiles

Conventionsused

GetintouchReviews

1. MonolithicVersusMicroservicesDawnofapplicationarchitecture

Monolithicarchitecture

MicroservicesDatamanagement

Whentousewhat

MonolithicapplicationstomicroservicesIdentifyingcandidatesformicroservices

Releasecycleandthedeploymentprocess

Summary

2. MicroservicesArchitectureDomain-drivendesign

Boundedcontext

Identifyingboundedcontexts

Organizingaroundboundedcontexts

ComponentsHexagonalarchitecture

MessagingandcontractsDirectmessaging

Observermodel

Servicecontracts

ServicediscoveryServiceregistry

Servicediscoverypatterns

DatamanagementDirectlookup

Asynchronousevents

Combiningdata

Transactions

AutomatedcontinuousdeploymentCI/CD

Scaling

Summary

3. MicroservicesforHelpingHandsApplicationDesign

Usersandentities

Userstories

Domainmodel

MonolithicarchitectureApplicationcomponents

Deployment

Limitations

MovingtomicroservicesIsolatingservicesbypersistence

Isolatingservicesbybusinesslogic

Messagingandevents

Extensibility

WorkflowsforHelpingHandsServiceproviderworkflow

Serviceworkflow

Serviceconsumerworkflow

Orderworkflow

Summary

4. DevelopmentEnvironmentClojureandREPL

HistoryofClojure

ClojurebuildtoolsLeiningen

ClojureprojectConfiguringaproject

Runningaproject

Runningtests

Generatingreports

Generatingartifacts

ClojureIDE

Summary

5. RESTAPIsforMicroservicesIntroducingREST

RESTfulAPIsStatuscodes

Namingconventions

UsingRESTfulAPIsviacURL

RESTAPIsforHelpingHandsConsumerandProviderAPIs

ServiceandOrderAPIs

Summary

6. IntroductiontoPedestalPedestalconcepts

Interceptors

Theinterceptorchain

ImportanceofaContextMap

CreatingaPedestalserviceUsinginterceptorsandhandlers

Creatingroutes

Declaringrouters

Accessingrequestparameters

Creatinginterceptors

Handlingerrorsandexceptions

Logging

Publishingoperationalmetrics

Usingchainproviders

Usingserver-sentevents(SSE)CreatinginterceptorsforSSE

UsingWebSocketsUsingWebSocketwithPedestalandJetty

Summary

7. AchievingImmutabilitywithDatomic

DatomicarchitectureDatomicversustraditionaldatabase

Developmentmodel

Datamodel

Schema

UsingDatomicGettingstartedwithDatomic

Connectingtoadatabase

Transactingdata

UsingDatalogtoquery

Achievingimmutability

Deletingadatabase

Summary

8. BuildingMicroservicesforHelpingHandsImplementingHexagonalArchitecture

Designingtheinterceptorchainandcontext

CreatingaPedestalproject

DefininggenericinterceptorsInterceptorforAuth

Interceptorforthedatamodel

Interceptorforevents

CreatingamicroserviceforServiceConsumerAddingroutes

DefiningtheDatomicschema

Creatingapersistenceadapter

Testingroutes

CreatingamicroserviceforServiceProviderAddingroutes

DefiningDatomicschema

Testingroutes

CreatingamicroserviceforServicesAddingroutes

DefiningaDatomicschema

Testingroutes

CreatingamicroserviceforOrderAddingroutes

Testingroutes

CreatingamicroserviceforLookupDefiningtheElasticsearchindex

Creatingqueryinterceptors

Usinggeoqueries

Gettingstatuswithaggregationqueries

CreatingamicroserviceforalertsAddingroutes

CreatinganemailinterceptorusingPostal

Summary

9. ConfiguringMicroservicesConfigurationprinciples

Definingconfigurationparameters

Usingconfigurationparameters

UsingOmniconfforconfigurationEnablingOmniconf

IntegratingwithHelpingHands

ManagingapplicationstateswithmountEnablingmount

IntegratingwithHelpingHands

Summary

10. Event-DrivenPatternsforMicroservicesImplementingevent-drivenpatterns

Eventsourcing

UsingtheCQRSpattern

IntroductiontoApacheKafkaDesignprinciples

GettingKafka

UsingKafkaasamessagingsystem

UsingKafkaasaneventstore

UsingKafkaforHelpingHandsUsingKafkaAPIs

InitializingKafkawithMount

IntegratingtheAlertServicewithKafka

UsingAvrofordatatransfer

Summary

11. DeployingandMonitoringSecuredMicroservicesEnablingauthenticationandauthorization

IntroducingTokensandJWT

CreatinganAuthserviceforHelpingHandsUsingaNimbusJOSEJWTlibraryforTokens

CreatingasecretkeyforJSONWebEncryption

CreatingTokens

Enablingusersandrolesforauthorization

CreatingAuthAPIsusingPedestal

MonitoringmicroservicesUsingELKStackformonitoring

SettingupElasticsearch

SettingupKibana

SettingupLogstash

UsingELKStackwithCollectd

Loggingandmonitoringguidelines

DeployingmicroservicesatscaleIntroducingContainersandDocker

SettingupDocker

CreatingaDockerimageforHelpingHands

IntroducingKubernetes

GettingstartedwithKubernetes

Summary

OtherBooksYouMayEnjoyLeaveareview-letotherreadersknowwhatyouthink

Preface

Themicroservicearchitectureissweepingtheworldasthedefactopatternforbuildingscalableandeasy-to-maintainweb-basedapplications.Thisbookwillteachyoucommonpatternsandpractices,showingyouhowtoapplythemusingtheClojureprogramminglanguage.ItwillteachyouthefundamentalconceptsofarchitecturaldesignandRESTfulcommunication,andshowyoupatternsthatprovidemanageablecodethatissupportableindevelopmentandatscaleinproduction.ThisbookwillprovideyouwithexamplesofhowtoputtheseconceptsandpatternsintopracticewithClojure.

Whetheryouareplanninganewapplicationorworkingonanexistingmonolith,thisbookwillexplainandillustratewithpracticalexampleshowteamsofallsizescanstartsolvingproblemswithmicroservices.Youwillunderstandtheimportanceofwritingcodethatisasynchronousandnon-blocking,andhowPedestalhelpsusdothis.Later,thebookexplainshowtobuildReactivemicroservicesinClojure,whichadheretotheprinciplesunderlyingtheReactiveManifesto.Wefinishoffbyshowingyouvarioustechniquestomonitor,test,andsecureyourmicroservices.Bytheend,youwillbefullycapableofsettingup,modifying,anddeployingamicroservicewithClojureandPedestal.

WhothisbookisforIfyouarelookingforwardtomigrateyourexistingmonolithicapplicationstomicroservicesortakingyourfirststepsintomicroservicearchitecture,thenthisbookisforyou.YoushouldhaveaworkingknowledgeofprogramminginClojure.However,noknowledgeofRESTfularchitecture,microservices,orwebservicesisexpected.

WhatthisbookcoversChapter1,MonolithicVersusMicroservices,introducesmonolithicandmicroservicearchitectureanddiscusseswhentousewhat.Italsocoversthepossiblemigrationplansofmovingfrommonolithicapplicationstomicroservices.

Chapter2,MicroservicesArchitecture,coversthebasicbuildingblocksofmicroservicearchitectureanditsrelatedfeatures.Itdiscusseshowtosetupmessagingandcontracts,andmanagedataflowsamongmicroservices.

Chapter3,MicroservicesforHelpingHandsApplication,introducesasampleHelpingHandsapplicationanddescribesthestepsthatwillbetakenintherestofthebooktobuildtheapplicationusingmicroservices.Further,thechaptercomparesandcontraststhebenefitsofusingamicroservices-basedarchitecturecomparedwithamonolithicone.

Chapter4,DevelopmentEnvironment,coversClojureandREPLatahighlevelandintroducestheconceptsofLeiningenandBoot—thetwomajorbuildtoolsforanyClojureproject.TheemphasiswillbeonLeiningenwithabasicintroductiontoBootonhowtosetupaClojureprojectforimplementingmicroservices.

Chapter5,RESTAPIsforMicroservices,coversthebasicsoftheRESTarchitecturalstyle,variousHTTPmethods,whentousewhat,andhowtogivemeaningfulnamestoRESTfulAPIsofmicroservices.ItalsocoversthenamingconventionsforRESTAPIsusingtheHelpingHandsapplicationasanexample.

Chapter6,IntroductiontoPedestal,coverstheClojurePedestalframeworkindetailwithalltherelevantfeaturesprovidedbyPedestal,includinginterceptorsandhandlers,routes,WebSockets,server-sentevents,andchainproviders.

Chapter7,AchievingImmutabilitywithDatomic,givesanoverviewoftheDatomicdatabasealongwithitsarchitecture,datamodel,transactions,andDatalogquerylanguage.

Chapter8,BuildingMicroservicesforHelpingHands,isastep-by-step,hands-on

guidetobuildandtestmicroservicesfortheHelpingHandsapplicationusingthePedestalframework.

Chapter9,ConfiguringMicroservices,coversthebasicsofmicroservicesconfigurationanddiscusseshowtocreateconfigurablemicroservicesusingframeworkssuchasOmniconf.Italsoexplainsthestepstomanagetheapplicationstateeffectivelyusingavailablestate-managementframeworkssuchasMount.

Chapter10,Event-DrivenPatternsforMicroservices,coversthebasicsofevent-drivenarchitecturesandshowshowtouseApacheKafkaasamessagingsystemandeventstore.Further,itdiscusseshowtouseApacheKafkabrokersandsetupconsumergroupsfortheeffectivecoordinationofmicroservices.

Chapter11,DeployingandMonitoringSecuredMicroservices,coversthebasicsofmicroservicesauthenticationusingJWTandhowtosetupareal-timemonitoringsystemusingtheELKStack.ItalsoexplainsthebasicconceptsofcontainersandorchestrationframeworkssuchasKubernetes.

TogetthemostoutofthisbookTheJavaDevelopmentKit(JDK)isrequiredtorunanddevelopapplicationsusingClojure.YoucangettheJDKfromhttp://www.oracle.com/technetwork/java/javase/downloads/index.html.Itisalsorecommendedthatyouuseatexteditororanintegrateddevelopmentenvironment(IDE)ofyourchoiceforimplementation.SomeoftheexamplesinthechaptersrequireLinuxasanoperatingsystem.

Downloadtheexamplecodefiles

Youcandownloadtheexamplecodefilesforthisbookfromyouraccountatwww.packtpub.com.Ifyoupurchasedthisbookelsewhere,youcanvisitwww.packtpub.com/supportandregistertohavethefilesemaileddirectlytoyou.

Youcandownloadthecodefilesbyfollowingthesesteps:

1. Loginorregisteratwww.packtpub.com.2. SelecttheSUPPORTtab.3. ClickonCodeDownloads&Errata.4. EnterthenameofthebookintheSearchboxandfollowtheonscreen

instructions.

Oncethefileisdownloaded,pleasemakesurethatyouunziporextractthefolderusingthelatestversionof:

WinRAR/7-ZipforWindowsZipeg/iZip/UnRarXforMac7-Zip/PeaZipforLinux

ThecodebundleforthebookisalsohostedonGitHubathttps://github.com/PacktPublishing/Microservices-with-Clojure.Wealsohaveothercodebundlesfromourrichcatalogofbooksandvideosavailableathttps://github.com/PacktPublishing/.Checkthemout!

ConventionsusedThereareanumberoftextconventionsusedthroughoutthisbook.

CodeInText:Indicatescodewordsintext,databasetablenames,foldernames,filenames,fileextensions,pathnames,dummyURLs,userinput,andTwitterhandles.Hereisanexample:"ThepersistenceprotocolServiceDBconsistsofupsert,entity,anddeletefunctions."

Ablockofcodeissetasfollows:

"query":{

"term":{

"status":"O"

Whenwewishtodrawyourattentiontoaparticularpartofacodeblock,therelevantlinesoritemsaresetinbold:

(defnhome-page

[request]

(log/counter::home-hits1)

(ring-resp/response"HelloWorld!"))

Anycommand-lineinputoroutputiswrittenasfollows:

%leinrun

Noname|Hello,World!

%leinrunClojure

Clojure|Hello,World!

Bold:Indicatesanewterm,animportantword,orwordsthatyouseeonscreen.Forexample,wordsinmenusordialogboxesappearinthetextlikethis.Hereisanexample:"ClickontheCreateavisualizationbutton."

Warningsorimportantnotesappearlikethis.

Tipsandtricksappearlikethis.

GetintouchFeedbackfromourreadersisalwayswelcome.

Generalfeedback:Emailfeedback@packtpub.comandmentionthebooktitleinthesubjectofyourmessage.Ifyouhavequestionsaboutanyaspectofthisbook,pleaseemailusatquestions@packtpub.com.

Errata:Althoughwehavetakeneverycaretoensuretheaccuracyofourcontent,mistakesdohappen.Ifyouhavefoundamistakeinthisbook,wewouldbegratefulifyouwouldreportthistous.Pleasevisitwww.packtpub.com/submit-errata,selectingyourbook,clickingontheErrataSubmissionFormlink,andenteringthedetails.

Piracy:IfyoucomeacrossanyillegalcopiesofourworksinanyformontheInternet,wewouldbegratefulifyouwouldprovideuswiththelocationaddressorwebsitename.Pleasecontactusatcopyright@packtpub.comwithalinktothematerial.

Ifyouareinterestedinbecominganauthor:Ifthereisatopicthatyouhaveexpertiseinandyouareinterestedineitherwritingorcontributingtoabook,pleasevisitauthors.packtpub.com.

ReviewsPleaseleaveareview.Onceyouhavereadandusedthisbook,whynotleaveareviewonthesitethatyoupurchaseditfrom?Potentialreaderscanthenseeanduseyourunbiasedopiniontomakepurchasedecisions,weatPacktcanunderstandwhatyouthinkaboutourproducts,andourauthorscanseeyourfeedbackontheirbook.Thankyou!

FormoreinformationaboutPackt,pleasevisitpacktpub.com.

MonolithicVersusMicroservices

"Theoldorderchangethyieldingplacetonew"

-AlfredTennyson

Awell-designedmonolithicarchitecturehasbeenthekeytomanysuccessfulsoftwareapplications.However,microservices-basedapplicationsaregainingpopularityintheageoftheinternetduetotheirinherentpropertyofbeingautonomousandflexible,theirabilitytoscaleindependently,andtheirshorterreleasecycles.Inthischapter,youwill:

LearnaboutthebasicsofmonolithicandmicroservicesarchitecturesUnderstandthemonolithic-firstapproachandwhentostartusingmicroservicesLearnhowtomigrateanexistingmonolithicapplicationtomicroservicesCompareandcontrastthereleasecycleanddeploymentmethodologyofmonolithicandmicroservices-basedapplications

DawnofapplicationarchitectureEversinceAdaLovelace(https://en.wikipedia.org/wiki/Ada_Lovelace)wrotethefirstalgorithmforAnalyticalEngine(https://en.wikipedia.org/wiki/Analytical_Engine)inthe19thcenturyandAlanTuring(https://en.wikipedia.org/wiki/Alan_Turing)formalizedtheconceptsofalgorithmandcomputationviatheTuringmachine(https://en.wikipedia.org/wiki/Turing_machine),softwarehasgonethroughmultiplephasesinitsevolution,bothintermsofhowitisdesignedandhowitismadeavailabletoitsendusers.Theearliersoftwarewasdesignedtorunonasinglemachineinasingleenvironment,andwasdeliveredtoitsendusersasanisolatedstandaloneentity.Intheearly1990s,asthefocusshiftedtoapplicationsoftware,theindustrystartedexploringvarioussoftwarearchitecturemethodologiestomeetthedemandsofchangingrequirementsandunderlyingenvironments.Oneofthesoftwarearchitecturesthatwaswidelyadoptedwasmultitierarchitecture,whichclearlyseparatedthefunctionsofdatamanagement,businesslogic,andpresentation.Whentheselayerswerepackagedtogetherinasingleapplication,usingasingletechnologystack,runningasasingleprogram,itwascalledamonolithicarchitecture,stillinusetoday.

Withtheadventoftheinternet,softwarestartedgettingofferedasaserviceovertheweb.Withthischangeindeploymentandusage,itstartedbecominghardtoupgradeandaddfeaturestosoftwarethatadoptedamonolithicarchitecture.Technologystartedchangingrapidlyandsodidprogramminglanguages,databases,andunderlyinghardware.Companiesthatwereabletodisintegratetheirmonolithicapplicationsintoloosely-coupledservicesthatcouldtalktoeachotherwereabletoofferbetterservices,betterintegrationpoints,andbetterperformancetotheirusers.Theywerenotonlyabletoupgradetothelatesttechnologyandhardware,butalsoabletooffernewfeaturesandservicesfastertotheirusers.Theideaofdisintegratingamonolithicapplicationintoloosely-coupledservicesthatcanbedeveloped,deployed,andscaledindependentlyandcantalktootherservicesoveralightweightprotocol,wascalledmicroservices-basedarchitecture(https://en.wikipedia.org/wiki/Microservices).

CompaniessuchasNetflix,Amazon,andsoonhavealladoptedamicroservices-basedarchitecture.IfyoulookatGoogleTrendsintheprecedingscreenshot,youcanseethatthepopularityofmicroservicesisrisingdaybyday,butthisdoesn'tmeanthatmonolithicapplicationsareobsolete.Thereareapplicationsthatarestillsuitedformonolithicarchitecture.Microserviceshavetheiradvantages,butatthesametimetheyarehardtodeploy,scale,andmonitor.Inthischapter,wewilllookatbothmonolithicandmicroservices-basedarchitectures.Wewilldiscusswhentousewhatandalsotalkaboutwhenandhowtomigratefromamonolithictoamicroservices-basedarchitecture.

MonolithicarchitectureMonolithicarchitectureisanall-in-onemethodologythatencapsulatesalltherequiredservicesasasingledeployableartifact.Itworksonasingletechnologystackandisdeployedandscaledasasingleunit.Sincethereisonlyonetechnologystacktomaster,itiseasytodeploy,scale,andsetupamonitoringinfrastructureformonolithicapplications.EachteammemberworksononeormorecomponentsofthesystemandfollowsthedesignprincipleofSeparationofConcerns(SoC)(https://en.wikipedia.org/wiki/Separation_of_concerns).Suchapplicationsarealsoeasiertorefactor,debug,andtestinasinglestandalonedevelopmentenvironment.

Applicationsbasedonmonolithicarchitecturemayconsistofoneormoredeployableartifactsthatarealldeployedatthesametime.SuchamonolithicarchitectureisoftenreferredtoasaDistributedMonolith.

Forexample,averycommonmonolithicapplicationisawordprocessingapplication;MicrosoftWordisinstalledviaasingledeployableartifactandisentirelybuiltonMicrosoft.NETFramework(https://www.microsoft.com/net/).Therearevariouscomponentswithinwordprocessingapplication,suchastemplates,import/export,spell-checker,andsoon,thatworktogethertohelpcreateadocumentandexportittheformatofchoice.

Monolithicarchitectureappliesnotonlytostandaloneapplications,butalsotoclient-serverbasedapplicationsthatareprovidedasaserviceovertheweb.Suchclient-serverbasedapplicationshaveaclearlydefinedmultitierarchitecturethatprovidestherelevantservicestoitsendusersviaauserinterface.

Theuserinterfacetalkstoapplicationendpointsthatcanbeprogrammedusingwell-definedinterfaces.

Atypicalclient-serverapplicationmayadoptathree-tierarchitecturetoseparatethepresentation,businesslogic,andpersistencelayerfromeachother,asshownintheprecedingdiagram.Componentsofeachlayertalkstrictlytothecomponentsofthelayerbelowthem.Forexample,thecomponentsofthepresentationlayermaynevertalktothepersistencelayerdirectly.Iftheyneedaccesstodata,therequestwillberoutedviathebusinesslogiclayerthatwillnotonlymovethedatabetweenthepersistencelayerandthepresentationlayer,butalsodotherequiredprocessingtoservetherequest.Adoptingsuchacomponent-basedlayeredarchitecturealsohelpsinisolatingtheeffectofchangetoonlythecomponentsofdependentlayersinsteadoftheentireapplication.Forexample,changestothecomponentsofthebusinesslogiclayermayrequireachangeinthedependentcomponentsofthepresentationlayerbutcomponentsofthepersistencelayermayremainintact.

EventhoughamonolithicapplicationisbuiltonSoC,itisstillasingleapplicationonasingletechnologystackthatprovidesallrequiredservicestoitsusers.Anychangetosuchanapplicationrequirestobecompatiblewithalltheencapsulatedservicesandunderlyingtechnologystack.Inadditiontothat,itisnotpossibletoscaleeachserviceindependently.Anyscalingrequirementismetbydeployingmultipleinstancesoftheentiresystemasasingleunit.Ateam

workingonsuchamonolithicapplicationscalesovertimeandhastoadapttonewertechnologiesasawhole,whichisoftenchallengingduetotherapidlychangingtechnologylandscape.Iftheydonotchangewiththetechnology,theentiresoftwarebecomesobsoleteovertimeandisdiscardedduetoincompatibilitywithnewersoftwareandhardware,orashortageoftalent.

MicroservicesMicroservicesareafunctionalapproachwellappliedtosoftware.Ittriestodecomposetheentireapplicationfunctionallyintoasetofservicesthatcanbedeployedandscaledindependently.Eachservicedoesonlyonejobanddoesitwell.Ithasitsowndatabase,decidesitsownschema,andprovidesaccesstodatasetsandservicesthroughwell-definedapplicationprogramminginterfacesthatarebetterknownasAPIs,oftenpairedwithauserinterface.APIsfollowasetcommunicationprotocols,butservicesarefreetochoosetheirowntechnologystackandcanbedeployedonhardwareofchoice.

Inamicroserviceenvironment,asshownintheprecedingdiagram,therearenolayerslikeinmonoliths;instead,eachserviceisorganizedaroundaboundedcontext(https://en.wikipedia.org/wiki/Domain-driven_design#Bounded_context)thataddsabusinesscapabilitytotheapplicationasawhole.Newcapabilitiesinsuchanapplicationareaddedasnewservicesthataredeployedandscaledindependently.Eachuserrequestinamicroservices-basedapplicationmaycalloneormoreinternalmicroservicetoretrievedata,processit,andgeneratetherequiredresponse,asshowninthefollowingdiagram.Suchsoftwareevolvesfasterandhaslowtechnologydebt.Theydonotgetmarriedtoaparticulartechnologystackandcanadoptanewtechnologyfaster:

DatamanagementInamicroservices-basedapplication,databasesareisolatedforeachbusinesscapabilityandaremanagedbyonlyoneserviceatatime.AnyrequestthatneedsaccesstothedatamanagedbyanotherservicestrictlyusestheAPIsprovidedbytheservicemanagingthedatabase.Thismakesitpossibletonotonlyusethebestdatabasetechnologyavailabletomanagethebusinesscapability,butalsotoisolatethetechnologydebttotheservicemanagingit.However,itisrecommendedforthecallingservicetocacheresponsesovertimetoavoidtightcouplingwiththetargetserviceandreducethenetworkoverheadofeachAPIcall.

Forexample,aservicemanaginguserinterestsmightuseagraphdatabase(https://en.wikipedia.org/wiki/Graph_database)tobuildanetworkofusers,whereasaservicemanagingusertransactionsmightusearelationaldatabase(https://en.wikipedia.org/wiki/Relational_database)duetoitsinherentACID(https://en.wikipedia.org/wiki/ACID)propertiesthataresuitablefortransactions.ThedependentserviceonlyneedstoknowtheAPIstoconnecttotheservicefordataandnotthetechnologyoftheunderlyingdatabase.

Thisiscontrarytoamonolithiclayeredarchitecture,wheredatabasesareorganizedbybusinesscapability,whichmaybeaccessedbyoneormorepersistencemodulesbasedontherequest.Iftheunderlyingdatabaseisusingadifferenttechnology,theneachofthemodulesaccessingthedatabaseshavetocomplywiththesametechnology,thusinheritingthecomplexityofeachdatabasetechnologythatithasaccessto.

Databaseisolationshouldbedoneatthedatabaselevelandnotatthedatabasetechnologylevel.Avoiddeployingmultipleinstancesofthesamerelationaldatabaseorgraphdatabaseasmuchaspossible.Instead,trytoscalethemondemandandusetheisolationcapabilityofthesesystemstomaintainseparatedatabaseswithinthemforeachservice.

Theconceptofmicroservicesisverysimilartoawell-knownarchitecturecalled

service-orientedarchitecture(SOA)(https://en.wikipedia.org/wiki/Service-oriented_architecture).Inmicroservices,thefocusisonidentifyingtherightboundedcontextandkeepingthemicroservicesaslightweightaspossible.Insteadofusingacomplexmessage-orientedmiddleware(https://en.wikipedia.org/wiki/Message-oriented_middleware)suchasESB(https://en.wikipedia.org/wiki/Enterprise_service_bus),asimplemodeofcommunicationisusedthatisoftenjustHTTP.

"ArchitecturalStyle[ofMicroservices]isreferredtoasfine-grainedSOA,perhapsserviceorientationdoneright"

-MartinFowleronmicroservices

WhentousewhatThemonolithiclayeredarchitectureisoneofthemostcommonarchitecturesinuseacrossthesoftwareindustry.Monolithicarchitecturesarewellsuitedfortransaction-orientedenterpriseapplicationsthathavewell-definedfeatures,changelessoften,andhavecomplexbusinessmodels.Forsuchapplications,transactionsandconsistencyareofprimeimportance.Theyrequireadatabasetechnologywithbuilt-insupportforACIDpropertiestostoretransactions.Ontheotherhand,microservicesaresuitedbetterforSoftware-as-a-Service,internet-scaleapplicationsthatarefeature-firstapplicationswitheachfeaturefocusedonasinglebusinesscapability.Suchapplicationschangerapidlyandarescaledpartiallyperbusinesscapabilityondemand.Transactionsandconsistencyinsuchapplicationsarehardtoachieveduetomultipleservices,ascomparedtomonolithsthatareimplementedassingleapplications.

Itisrecommendedtostartwithawelldesigned,modularmonolithicapplicationirrespectiveofthedomaincomplexityortransactionalnature.Generally,allapplicationsstartasamonolithicapplicationthatcanbedeployedfasterasasingleartifactandlatersplitintomicroserviceswhentheapplication'scomplexitybeginstooutweightheproductivityoftheteam.

Theproductivityoftheteammaystartdecreasingwhenchangestothe

monolithicapplicationstartaffectingmorethanonecomponent,asshownintheprecedingdiagram.Thesechangesmaybearesultofanewfeaturebeingaddedtotheapplication,adatabasetechnologyupgrade,ortherefactoringofexistingcomponents.Anychangesmadetotheapplicationmustkeeptheentireteamin-sync,especiallythedeploymentteam,ifthereareanychangesrequiredinthedeploymentprocesses.Communicatingsuchchangesinalargeteamoftenresultsinacoordinationnightmare,multiplechangerequests,andin-turn,reducestheoverallproductivityoftheteamworkingontheapplication.

Productivityalsodependsontheinitialchoicesmadewithrespecttothetechnologystackanditsflexibilityofimplementation.Forexample,ifanewfeaturerequiresalibrarythatisreadilyavailablewithadifferenttechnologystackoraprogramminglanguage,itbecomeschallengingtoadoptasitdoesnotconformtotheexistingtechnologystackoftheapplicationcomponents.Insuchcases,theteamendsupimplementingthesamefeaturesetforthecurrenttechnologystackfromscratch,andthatinturnreducesproductivityandfurtheraddstothetechnologydebt.

Beforestartingwithmicroservices,firstsetupbestdesignprinciplesamongteammembers.Next,trytoevaluatetheexistingmonolithwithregardtocomponentsandtheirinteraction.Ifrefactoringcanhelpreducethedependencybetweenthecomponents,dothatfirstinsteadofdisintegratingyourapplicationintomicroservices.

MonolithicapplicationstomicroservicesMostapplicationsstartasamonolith.Amazon(http://highscalability.com/amazon-architecture)startedwithamonolithicPerl(https://en.wikipedia.org/wiki/Perl)/C++(https://en.wikipedia.org/wiki/C%2B%2B)application,andTwitter(http://highscalability.com/blog/2013/7/8/the-architecture-twitter-uses-to-deal-with-150m-active-users.html)startedwithamonolithicRails(https://en.wikipedia.org/wiki/Ruby_on_Rails)application.Bothorganizationshavenotonlygonethroughmorethanthreegenerationsofsoftwarearchitecturalchanges,buthavealsotransformedtheirorganizationalstructuresovertime.Today,allofthemarerunningonmicroserviceswithteamsorganizedaroundservicesthataredeveloped,deployed,scaled,andmonitoredbythesameteamindependently.Theyhavemasteredcontinuousintegrationandcontinuousdeliverypipelineswithautomateddeployment,scaling,andmonitoringofservicesforreal-timefeedbacktotheteam.

IdentifyingcandidatesformicroservicesThetop-mostchallengeinmigratingfromamonolithicapplicationtomicroservicesistoidentifytherightcandidatesformicroservices.Awellstructuredandmodularizedmonolithicapplicationalreadyhaswell-definedboundaries(boundedcontexts)thatcanhelpdisintegratetheapplicationintomicroservices.Forexample,theUser,Orders,andInterestmodulesalreadyhavewell-definedboundariesandaregoodcandidatestocreatemicroservicesfor.Iftheapplicationdoesnothavewell-definedboundaries,thefirststepistorefactortheexistingapplicationtocreatesuchboundedcontextsformicroservices.Eachboundedcontextmustbetiedtoabusinesscapabilityforwhichaservicecanbecreated.

Anotherapproachinidentifyingtherightcandidatesformicroservicesistolookatthedataaccesspatternsandassociatedbusinesslogic.Ifthesamedatabaseisbeingupdatedbymultiplecomponentsofamonolithicapplication,thenitmakessensetocreateaservicefortheprimarycomponentwithassociatedbusinesslogicthatmanagesthedatabaseandmakesitaccessibletootherservicesviaAPIs.Thisprocesscanberepeateduntildatabasesandtheassociatedbusinesslogicaremanagedbyoneandonlyoneservicethathasasmallsetofresponsibilities,modeledaroundabusinesscapability.

Forexample,amonolithicapplicationconsistingofUser,Interest,andOrderscomponentscanbemigratedintomicroservicesbypickingonecomponentatatimeandcreatingamicroservicewithanisolateddatabase,asshownintheprecedingdiagram.Tostartwith,firstpicktheonewiththeleastdependency,theUsermodule,andcreatetheUserServiceservicearoundit.AllothercomponentscannowtalktothisnewUserServiceforUserManagement,includingauthentication,authorization,andgettinguserprofiles.Next,picktheOrdersmodulebasedontheleastdependencylogic,andcreateaservicearoundit.Finally,picktheInterestmoduleasitisdependentonboththeUserandOrdersmodules.Sincewehavethedatabasesisolated,wecanalsoswapoutthedatabaseforInterestwithmaybeagraphdatabasethatisefficienttostoreandretrieveuserinterestsduetoitsinherentcapabilityofstoringrelationshipsasagraph.

Inadditiontoorganizingyourmicroservicesaroundbusinesscapabilitiesanddatabaseaccesspatterns,lookforcommonareas,suchasauthentication,authorization,andnotification,thatcanbeperfectedonceasaserviceandcanbeleveragedbyoneormore

microserviceslater.

ReleasecycleandthedeploymentprocessOnceamonolithicapplicationisdisintegratedintomicroservices,thenextstepistodeploythemintoproduction.Monolithicapplicationaremostlydeployedasasingleartifact(JARs,WARs,EXEs,andmore)thatarereleasedafterextensivetestingbythequalityassurance(QA)team.Typically,developersworkonvariouscomponentsoftheapplicationandreleaseversionsfortheQAteamtopickandvalidateagainstthespecification,asshownundertheOrgStructureofmonolithicarchitectureinthefollowingdiagram.Eachiterationmayinvolvetheadditionorremovaloffeaturesandbugfixes.Thereleasegoesthroughmultipledevelopers(dev)andQAteamiterationsuntiltheQAteamflagsoffthereleaseasstable.OncetheQAteamflagsofftherelease,thereleasedartifactishandedovertotheITopsteamtodeployitinproduction.Ifthereareanyissuesinproduction,theITopsteamasksthedevteamtofixthem.Oncetheissuesarefixed,thedevteamtagsanewreleaseforQAthatagaingoesthroughthesamedev-QAiterationsbeforebeingmarkedasstableandeventuallyhandedovertoIT/ops.Duetothisprocess,anyreleaseforamonolithicapplicationsmayeasilytakeuptoamonth,oftenthreemonths.

Ontheotherhand,formicroservices,teamsareorganizedintogroupsthatfully

ownaservice.Theteamisresponsiblefornotonlydevelopingtheservice,butalsoforputtingtogetherautomatedtestcasesthatcantesttheentireserviceagainsteachchangesubmittedfortheservice.Sincetheserviceistobetestedinisolationforitsfeatures,itisfastertorunentiretestsuitesfortheserviceforeachchangesubmittedbythedevelopers.Additionally,theteamitselfcreatesdeployablebinariesoftenpackagedintocontainers(https://en.wikipedia.org/wiki/Linux_containers),suchasDocker(https://en.wikipedia.org/wiki/Docker_(software)),thatarepublishedtoacentralrepositoryfromwheretheycanbeautomaticallydeployedintoproductionbysomewell-knowntools,suchasKubernetes(https://en.wikipedia.org/wiki/Kubernetes).Theentiredevelopmenttoproductiontimelineiscutshorttodays,oftenhours,astheentiredeploymentprocessisautomated.WewilllearnmoreaboutdeployingmicroservicesinproductionandhowtousethesedeploymenttoolsinPart-4,thelastpartofthisbook.

Thereisareasonwhyalotofmicroserviceprojectsfailandonlyafewsucceed.Migratingfromamonolithicarchitecturetomicroservicesmustnotonlyfocusonidentifyingtheboundedcontexts,butalsotheorganizationalstructureanddeploymentmethodologies.Teamsmustbeorganizedaroundservicesandnotprojects.Eachteammustowntheservicerightfromdevelopmenttoproduction.Sinceeachteamownstheresponsibilityfortesting,validation,anddeployment,theentireprocessshouldbeautomatedandtheorganizationmustmasterit.Developmentanddeploymentcyclesmustbeshortwithimmediatefeedbackviafine-grainedmonitoringofthedeployedmicroservices.

Automationiskeyforanysuccessfulmicroservicesproject.Testing,deployment,andmonitoringmustbeautomatedbeforemovingmicroservicestoproduction.

SummaryInthischapter,welearnedaboutmonolithicandmicroservicesarchitecturesandwhymicroservicesarebecomingpopularintheindustry,especiallywithweb-scaleapplications.Welearnedabouttheimportanceofdatabaseisolationwithmicroservicesandhowtomigrateamonolithicapplicationtomicroservicesbyobservingthedatabaseaccesspattern.Wealsodiscussedtheimportanceofthemonolith-firstapproachandwhentomovetowardsmicroservices.Weconcludedwithacomparisonofmonolithicandmicroservicesarchitectureswithregardtothereleasecycleanddeploymentprocess.

Thenextchapterofthisbookwilltalkaboutmicroservicearchitectureindetail;wewilllearnmoreaboutdomain-drivendesignandhowtoidentifytherightsetofmicroservices.InChapter3,MicroservicesforHelpingHandsApplication,thelastchapterofPart-1,wewillpickareal-lifeusecaseformicroservicesanddiscusshowtodesignitusingtheprinciplesofmicroservicearchitecture.

MicroservicesArchitecture

"Gathertogetherthethingsthatchangeforthesamereasons.Separatethosethingsthatchangefordifferentreasons."

-RobertMartin,SingleResponsibilityPrinciple

Softwarearchitectureplaysakeyroleinidentifyingthebehaviorofthesystembeforeitisbuilt.Awell-designedsoftwarearchitectureleadstoflexible,reusable,andscalablecomponentsthatcanbeeasilyextended,verified,andmaintainedovertime.Sucharchitecturesevolveovertimeandhelppavethewayfortheadoptionofnext-generationarchitectures.Forexample,awell-designedmonolithicapplicationthatisbuiltontheprinciplesofSeparationofConcern(SoC)iseasiertomigratetomicroservicesthananapplicationthatdoesnothavewell-definedcomponents.Inthischapter,youwill:

LearnasystematicapproachtodesigningmicroservicesusingtheboundedcontextLearnhowtosetupcontractsbetweenmicroservicesandisolatefailuresLearnhowtomanagedataflowsandtransactionsamongmicroservicesLearnaboutservicediscoveryandtheimportanceofautomateddeployment

Domain-drivendesignIdealenterprisesystemsaretightlyintegratedandprovideallbusinesscapabilitiesasasingleunitthatisoptimizedforaparticulartechnologystackandhardware.Suchmonolithicsystemsoftengrowsocomplexovertimethatitbecomeschallengingtocomprehendthemasasingleunitbyasingleteam.Domain-drivendesignadvocatesdisintegratingsuchsystemsintosmallermodularcomponentsandassigningthemtoteamsthatfocusonasinglebusinesscapabilityinaboundedcontext(https://en.wikipedia.org/wiki/Domain-driven_design#Bounded_context).Oncedisintegrated,allsuchcomponentsaremadeapartofanautomatedcontinuousintegration(CI)processtoavoidanyfragmentation.Sincethesecomponentsarebuiltinisolationandoftenhavetheirowndatamodelsandschema,thereshouldbeawell-definedcontracttointeractwiththecomponentstocoordinatevariousbusinessactivities.

ThetermDomain-drivendesignwasfirstcoinedbyEricJ.Evansasthetitleofhisbookin2003.InPart-IV,Evanstalksabouttheboundedcontextandtheimportanceofcontinuousintegration,whichformsthebasisofanymicroservicesarchitecture.

BoundedcontextAdomainmodelisaconceptualmodelofabusinessdomainthatformalizesitsbehavioranddata.Asingleunifieddomainmodeltendstogrowincomplexitywithbusinesscapabilitiesandincreasesthecollaborationoverheadamongtheteamduetohighcoupling.Toreducecoupling,domain-drivendesignrecommendsdefiningamodelforeachbusinesscapabilitywithawell-definedboundarytoseparatethedomainconceptswithinthemodelfromtheonesoutside.Eachsuchmodelthenfocusesonthebehavioranddataconfinedtoasinglebusinesscapability,andthusgetsboundedbyasingleapplicationcontext,calledaboundedcontext.Monolithicapplicationstendtohaveaunifieddomainmodelfortheentirebusinessdomain,whereasformicroservices,domainmodelsaredefinedforeachidentifiedboundedcontext.

Forexample,insteadofdefiningasingleunifieddomainmodelforane-commerceapplication,itisbettertodividetheapplicationintoboundedcontextsofCustomer,Sales,andMarketinganddefineadomainmodelforeachofthesecontexts,asshownintheprecedingdiagram.Suchfocuseddomainmodelscanthenconquereachcontextbasedonbusinesscapabilities.Forexample,CustomerContextcanfocusonlyonuserandprofilemanagement,SalesContextcanhandleordersandtransactions,andMarketingContextcankeeptrackofuserinterestsforfocusedmarketing.

IdentifyingboundedcontextsOneofthemostchallengingtasksindesigningamicroservices-basedarchitectureistogettheboundedcontextrightforeachmicroservice.Itisaniterativeprocessthatrequiresathoroughunderstandingofbusinesscapabilitiesandbusinessdomain.Businesscapabilitiesmustnotbeconfusedwithbusinessfunctionsorprocesses.Businesscapabilitiestargetthewhatpartofabusinessandhaveanoutcome,whereasabusinessprocesstargetsthehowpart.Forexample,alertingisabusinesscapability,whereassendinganemailisabusinessprocess.Abusinesscapabilitymayincorporateoneormorebusinessprocesses.

Boundedcontextsmusttargetbusinesscapabilitiesandnotbusinessprocesses.Toidentifytherightboundedcontexts,itisrecommendedtostartwithamonolithicapplicationwithasingleunifiedmodelandanalyzeititerativelyovertimeforhighcouplingareas.Often,highcouplingareasaregoodtargetpointstosplitthedomainmodelintosub-domainmodelsthatcaninteractusingfixedcontractsandhelpreducetightcoupling.However,suchsub-domainsmustbefurthervalidatedagainstbusinesscapabilitiestomakesurethattheytargetonlyonebusinesscapability.

Microservicesmustbeorganizedaroundabusinesscapabilitywithinaboundedcontextandowntheirpresentation,businessdomain,andpersistencelayer.Theymusttakeresponsibilityfortheend-to-enddevelopmentstackincludingfunctions,datamodel,persistence,userinterface,andthecontracttoaccesstheserviceusingAPIs,oftenoverHTTP(S).

Organizingaroundboundedcontexts"Anyorganizationthatdesignsasystem(definedbroadly)willproduceadesignwhosestructureisacopyoftheorganization'scommunicationstructure"

-MelvynConway,1967

Generally,theorganizationstructurecontributesheavilytothedesignoftheapplication.Therefore,boundedcontextsshouldneverbeidentifiedonthebasisoftheexistingstructureoftheorganization.Instead,theorganizationmustbestructuredaroundboundedcontextssuchthattheentireteamcanworkonaserviceinisolation.

Atypicalorganization,workingonamonolithicapplication,isbuiltaroundlayersofpresentation,businesslogic,andpersistence,asshownintheprecedingdiagram.TheytendtohaveaseparateteamofUIdesignersandUI/UXexpertsforthepresentationlayer,ateamofbackenddeveloperstoimplementthedomainmodel,andateamofdatabaseadministratorstocreateadatabasefordeveloperstoaccess.Suchanorganizationstructureisidealforanapplicationwithasmallersetofbusinesscapabilities.

Oncetheapplicationgrows,anychangestotheapplicationrequirecommunicationtobemadeacrosstheteamofUIengineers,backenddevelopers,anddatabaseadministrators.Often,suchcommunicationleadstosomanyback-and-forthexchangesinvolvingdesigndocumentsandspecificationchangesthat

itbecomesoverwhelmingfortheteamstotranslatetherequirementscorrectlytotheimplementation.Suchcommunicationoverheadaddsdelaystotheprojectandbringsdowntheproductivityoftheentireteamworkingontheproductasawhole.

Boundedcontext,ifidentifiedcorrectly,solvesthisproblembylocalizingtheteamofUIdevelopers,backenddevelopers,anddatabaseadministratorstofocusonasinglebusinesscapability.Suchboundariesmakesurethatthecommunicationbetweentheteamsisboundedbyafixedcontractthatissetattheservicelevel.Thismakesitpossibletoreduceaconsiderablecommunicationoverhead,asanychangesmadetoaserviceareconfinedwithintheteamworkingontheservice.Forexample,thelocalizedteamofUserServiceandOrdersServicewillcommunicateonlytodiscusstheserviceAPIsthattheuserserviceisexposingforOrdersServicetogetthecustomerdetails.Sinceanychangestothecustomerschemaortheordersschemashouldnotimpacteachotherasperthedefinitionofboundedcontext,itisnotrequiredtocommunicatesuchchangestotheotherservice.

Components

Oncetheboundedcontextsareidentifiedformicroservicesandtheorganizationstructureisaligned,eachmicroservicemustbeconsideredasaproductthatistested,deployed,andscaledinisolationbythesameteamthatdevelopedit.Awell-designedmicroservicemustneverexposeitsinternaldatamodeltotheoutsideworlddirectly.Instead,itmustmaintainaservicecontractthatmapstoitsinternalmodelsuchthatitcanevolveovertimewithoutaffectingthedependentmicroservices.

Component-basedsoftwareengineering(https://en.wikipedia.org/wiki/Component-based_software_engineering)definesacomponentasareusablemodulethatisbasedontheprinciplesofSoCandencapsulatesasetofrelatedfunctionsanddata.Inthecontextofmicroservicesarchitecture,itisrecommendedtoimplementeachserviceasacomponentthatisindependentlyswappableanddeployablewithoutaffectinganyothermicroservices.

HexagonalarchitectureHexagonalarchitecture(http://alistair.cockburn.us/Hexagonal+architecture),alsoknownastheportsandadapterspattern,aimstodecouplebusinesslogicfromotherpartsofthecomponent,especiallythepersistenceandserviceslayers.Acomponent,builtontheportsandadapterspattern,exposesasetofportstowhichoneormoreadapterscanbeaddedasnecessary.Forexample,totestandverifythecorebusinesslogicinisolation,amockdatabaseadaptercanbepluggedinandlaterreplacedwitharuntimedatabaseadapterinproduction.

Aportisanentrypointthatisprovidedbythecorebusinesslogictointeractwithotherpartsofthecomponent.Anadapterisanimplementationofaport,andtheremaybemorethanoneadapterdefinedforasingleportbasedontherequirement.Forexample,aRESTadapterisusedtoacceptrequestsfromexternalusersorothermicroservicescomponents.ItinternallycallstheserviceAPIportdefinedbythecorebusinesslogicthatperformsthatrequestedoperationandgeneratesaresponse.Similarly,adatabaseadapterisusedbythecorebusinesslogictointeractwiththeexternaldatabaseviaitsdatabaseport,asshowninthefollowingdiagram:

Basedontheirapplicabilityandusage,theRESTadapteranddatabaseadapterareoftenreferredtoasprimaryandsecondaryadaptersrespectively.Similarly,theserviceAPIportanddatabaseportarereferredtoasprimaryandsecondaryportsrespectively.Primaryportsarecalledbytheprimaryadaptersandtheyactasthemaininterfacebetweenthecorebusinesslogicanditsusers,whereassecondaryportsandsecondaryadaptersareusedbythecorebusinesslogictogenerateeventsorinteractwithexternalservices,likeadatabase.Primaryadaptershelpvalidateservicerequestswithrespecttoaserviceschemaandcallcorebusinesslogicfunctions,whereasthecorebusinesslogiccallsthefunctionsofsecondaryadapterstohelptranslatetheapplicationschematotheexternalserviceschema,likethatofadatabase.Primaryadaptersareinitializedwiththeapplication,butreferencestothesecondaryadaptersarepassedtothecorebusinesslogicviadependencyinjection.

Thenamehexagonalarchitectureisderivedfromthestructureofacomponentthathassixports,butthatisnotarule.Theideaofrepresentingthearchitectureasahexagonisjusttoremovethenotionofone-dimensionallayeredarchitectureandhaveroomtoinsertportsandadaptersasrequired.

Messagingandcontracts

Inmonolithicapplications,messagingbetweencomponentsismostlyachievedusingfunctioncalls,whereasformicroservices,itisachievedusinglightweightmessagingsystems,oftenHTTP(S).Usingalightweightmessagingsystemisoneofthemostpromisingfeaturesofmicroservicesandmakesiteasiertoadoptandscale,ascomparedtoservice-orientedarchitecture(SOA)thatusesacomplexmessagingsystemwithmultipleprotocols.Microservicesaremoreaboutkeepingtheendpointssmartandthecommunicationchannelsassimpleaspossible.

Inamicroservicesarchitecture,oftenmultiplemicroservicesneedtointeractwitheachothertoachieveaparticulartask.Theseinteractionscanbeeitherdirect,viarequest-response-based(https://en.wikipedia.org/wiki/Request-response)communication,orthroughalightweightmessage-orientedmiddleware(MOM)(https://en.wikipedia.org/wiki/Message-oriented_middleware).Directmessagingissynchronous,thatis,therequesterwaitsfortheresponsetobereturned,whereasamessage-orientedmiddlewareisprimarilyusedforasynchronouscommunication.

DirectmessagingIndirectmessaging,eachrequestissentdirectlytothemicroserviceonitsAPIendpoint.Suchrequestsmaybeinitiatedbyusers,applications,orbyothermicroservicesthatintegratewiththetargetmicroservicetocompleteaparticulartask.MostlytheendpointstohandlesuchrequestsareimplementedusingREST(https://en.wikipedia.org/wiki/Representational_state_transfer),whichallowsresourceidentifierstobeaddresseddirectlyviaHTTPbasedAPIswithasimplemessagingstyle.

RESTisanarchitecturalstylewithpredefinedoperationsbasedonHTTPrequestmethods,suchasGET,PUT,POST,andDELETE.ThestatelessnatureofRESTmakesitfast,reliable,andscalablewithmultiplecomponents.

Forexample,inatypicalmicroservices-basede-commerceapplication,anadministrativeusermaywishtocreate,update,ormanageusersviatheUserService'sRESTendpoint.Inthatcase,theusercandirectlycalltheRESTAPIoftheUserService,asshowninthefollowingdiagram:

Similarly,amobileapplicationmaywanttoqueryuserinterestsviaInterestService'sRESTendpointofthee-commerceapplication.SincetheInterestServicedependsonboththeUserServiceandtheOrdersService,itmayfurtherinitiatetwoseparatecallstotheRESTendpointofthesetwoservicestogetthedesiredresponsewhichitcanmergewiththeuserinterestsdataandgeneratetheresponsefortherequestingapplication.Mostly,allthesekindsofrequest-responsearesynchronousinnatureasthesenderexpectsaresponsewiththeresultsinthesamecall.Asynchronousrequestsaremostlyusedtosendalertsormessagestorecordanoperation,andinthosecases,thesenderdoesn'twaitfortheresponseandexpectstheactiontobetakeneventually.

Avoidbuildinglongchainsofsynchronouscallsasthatmayleadtohighlatencyandanincreasedpossibilityoffailureduetomultiplehopsofintermediaterequestsandnetworkround-tripsamongtheparticipatingservices.

MessageformatsusedwithRESTendpointsaremostlytext-basedmessageformatssuchasJSON,XML,orHTML,assupportedbytheendpointimplementation.BinarymessageformatssuchasThrift(https://en.wikipedia.org/wiki/Apache_Thrift),ProtoBuf(https://en.wikipedia.org/wiki/Protocol_Buffers),andAvro(https://avro.ap

ache.org/)arealsopopularduetotheirwidesupportacrossmultipleprogramminglanguages.

Usedirectmessagingonlyforsmallermicroservices-baseddeployments.Forlargerdeployments,itisadvisabletogowithAPIgatewaysthatactasthemainentrypointforallclients.SuchAPIgatewayshelpmonitorrequestsformicroservicesandalsoassistinthemaintenanceandupgradeoperations.

ObservermodelTheobservermodelusesamessagebroker(https://en.wikipedia.org/wiki/Message_broker)atitscoretosendmessagesamongmicroservices.Amessagebrokerprovidescontentandtopic-basedroutingusingthepublish-subscribepattern(https://en.wikipedia.org/wiki/Publish-subscribe_pattern),whichmakesthesenderandreceiverindependentofeachother.Allobservingmicroservicessubscribetooneormoretopicsthroughwhichtheycanreceivethemessagesandalsoconnecttotopicsonwhichtheycanpublishthemessagesforotherobservers.Allinteractionsdoneviamessagebrokersareasynchronousinnatureanddonotblockthesender.Thishelpsinscalingboththepublishersandsubscribersindependently.Amicroservicesarchitecturethatisbuiltononlyasynchronousandnon-blockinginteractionsusingmessagebrokerscalesverywell.

Messagebrokersarealsousedtomanageworkloadsinscenarioswheretherateofpublishedmessagesishigherthantherateatwhichthesubscriberisabletoprocessthemessages.Messagebrokersalsoprovidereliablestorage,multipledeliverysemantics(atleastonce,exactlyonce,andsoon),andalsotransactionmanagementthatisusefulfordatamanagementacrossmicroservices.BinarymessageformatssuchasThrift,ProtoBuf,andAvroarepreferredovertextformatsformessagebrokers.

Intheobservermodel,alltherequestsgeneratedbyusers,applications,ormicroservicesarepublishedonatopictowhichoneormoremicroservicescansubscribeandreceivethemessageforprocessing,asshownintheprecedingdiagram.Thegeneratedresultscanalsobewrittenbacktoatopicthatcanbelaterpickedbyanothermicroservice,whichmayeitherreporttheresponsebacktotheapplicationorpersistitwithinadatastore.Ifasubscriberfails,themessagebrokercanreplaythemessage.Similarly,ifallsubscribersarebusy,themessagebrokercanaccumulatethemessagesuntiltheyareprocessedbythesubscribers.

Theobservermodelhelpstoachievebetterscalabilityascomparedtodirectmessagingattheexpenseofasinglepointoffailureofthemessagebroker.

Servicecontracts

Contractsarerequiredforentitiestointeractwitheachother.Theydefinethemessageformatandmediumofcommunicationfortheparticipatingentities.Inamonolithicenvironment,itissimplerforcomponentstointeractwiththetargetcomponentsusingtheinterfacesandfunctionsexposedbythem.Sinceafunctionclearlydefinesitsmessageformatasinputparameters,themessagepassingbetweenthecomponentscanbedonebyjustafunctioncallwiththerequiredinputparameters.Functioncallsarefurthersimplifiedinamonolithduetoacommonunderlyingtechnologystack.Contractsarealsoeasiertomaintainformonolithicapplicationsbecauseanychangedonetothecontractistestedandverifiedacrosscomponentsforcompatibility.Suchapplicationsarealsoversionedasawholeandnotper-component.Thefollowingtablecomparesandcontrastsmonolithicandmicroservicearchitecturalstyles:

Monolithicarchitecture Microservices

Entity Components Servicesbycapability

EndpointInterfacesandfunctions

RESTURIs(HTTP)/Thrift/Avro/Protobuf

Medium Functioncalls

HTTP/publish-subscribeviamessagebroker(observer)

Contract Functiondefinition

APIspecification(Swagger,RAML)/messageserialization(ThriftIDL,AvroSchema)

Version Singleversion Separateversionsforeachservice

Technology Single Polyglot

Ascomparedtoamonolith,inamicroservices-basedenvironment,thereisa

servicedeployedforeachbusinesscapabilitythatmayormaynotbeusingthesametechnologystack.Insuchcases,servicecontractsbecomeabsolutelymandatoryformicroservicestounderstandthemessageformatsandcommunicationmediumacceptedbyothermicroservicesandinteractwiththem.Moreover,thesemessageformatsneedtobelanguage-agnostictoallowanymicroservicetocommunicateirrespectiveofthetechnologystackinwhichtheyareimplemented.

Microservicesshouldneverexposetheirinternaldatamodeldirectlyasapartofamessagecontracttotheexternalworld.Theinternaldatamodelmustbedecoupledfromtheexternalservicecontractandthereshouldbeawaytoconvertandvalidatethecontractatentryandexit.Thishelpstoevolvethedatamodelofamicroserviceinisolationwithoutaffectingthecontractwithothermicroservices.Ifthereisachangerequiredintheservicecontract,itmustbeversionedandeachversionofthecontractmustbesupportedbythemicroserviceaslongasitisinusebyanyexternalservice.Aversionshouldbediscardedonlywhentherearenootherservicesusingtheobsoleteversionandthereisnolongeraneedtorollbacktheservicetoitspreviousversion.

Avoidmultipleversionsofservicecontracts(andmessageformats)asmuchaspossible.Chooseaflexiblemessageformatthatcanevolveovertimewithoutbreakingpreviousversions.

MicroservicesthatexposeRESTAPIsprimarilyusetheRESTAPIdefinitionandHTTPverbs(https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Request_methods)todefinewell-formedURIs.ServicecontractsforREST-basedAPIsaredefinedusingframeworkssuchasSwagger(https://swagger.io/)andRAML(https://raml.org/).MicroservicesthatusetheobserverpatterntendtoacceptmessagesinThrift,Avro,orProtoBufformats.Eachoftheseframeworkshasawaytodefinelanguage-agnosticspecificationsandsupportsmostofthepopularprogramminglanguages.

ServicediscoveryAlltheAPIsexposedbyamicroserviceareaccessibleviatheIPaddressandportofthehostmachineonwhichthemicroserviceisdeployed.SincemicroservicesaredeployedinavirtualmachineoracontainerthathasadynamicIP,itisquitepossiblethattheIPsandportsthatareallocatedtothemicroserviceAPIsmaychangeovertime.Therefore,IPaddressesandportsofservicesshouldneverbehardcodedbythedependingmicroservice.Instead,thereshouldbeacommondatabaseofalltheservicesthatareactiveforthecurrentapplication.Suchadatabaseofservicesiscalledtheserviceregistryinmicroservicesarchitectureandisalwayskeptuptodatewiththelocationofthemicroservices.ItkeepsthedetailsofalltheactivemicroservicesincludingtheircurrentIPaddressandport.Microservicesthenquerythisserviceregistrytodiscoverthecurrentlocationoftherequiredmicroservicesandconnecttothemdirectly.

ServiceregistryTheserviceregistryactsasadatabaseofmicroservices.ItmusthaveadedicatedstaticIPaddressorafixedDNSnamethatmustbeaccessiblefromalltheclients,asshowninthefollowingdiagram.Sincealltheclientsdependonaserviceregistrytolookupthetargetservices,italsobecomesasinglepointoffailurefortheentiremicroservicesarchitecture.Therefore,theimplementationoftheserviceregistrymustbeextremelylightweightandshouldsupporthighavailabilitybydefault.SomecommontoolsthatcanbeusedasaserviceregistryareApacheZookeeper(http://zookeeper.apache.org/),etcd(https://github.com/coreos/etcd),andconsul(https://www.consul.io/):

Tokeeptheregistryuptodate,microservicesshouldeitherimplementthestartup/shutdowneventtoregister/deregisterwiththeserviceregistrythemselvesorthereshouldbeanexternalserviceconfiguredtokeeptrackofservicesandkeeptheregistryuptodate.Someorchestrationtools,suchasKubernetes(https://kubernetes.io/),supportserviceregistryoutoftheboxandmaintaintheregistryfortheentireinfrastructure.

Clientsmustcachethelocationoffrequentlyusedmicroservicestoreducedependencyontheserviceregistry,butthelocationmustbesyncedperiodicallywiththeserviceregistryforup-to-dateinformation.

ServicediscoverypatternsMicroservices-basedapplicationsmayoftenscaletosuchalargenumberofservicesthatitmaynotbefeasibleforeachmicroservicetokeepatrackofallotheractiveservicelocations.Insuchscenarios,theserviceregistryhelpsindiscoveringmicroservicestoperformaparticulartask.Thereareprimarilytwopatternsforservicediscovery—client-sidediscoveryandserver-sidediscovery,asshowninthefollowingdiagram.

Intheclient-sidediscoverypattern,theresponsibilityfordeterminingthelocationofservicesbyqueryingtheserviceregistryisontheclient.Therefore,theserviceregistrymustbeaccessibletotheclienttolookupthelocationoftherequiredservices.Also,eachclientmusthaveservicediscoveryimplementationbuilt-inforthispatterntowork.

Ontheotherhand,intheserver-sidediscoverypattern,theresponsibilityforconnectingwiththeserviceregistryandlookingupthelocationofservicesisofarouteroragatewaythatactsasaloadbalanceraswell.Clientsjustneedtosendarequesttoarouterandtheroutertakescareofforwardingtherequesttotherequiredservice.OrchestrationtoolssuchasKubernetessupportserver-sidediscoveryusingproxies.

Theserver-sidediscoverypatternmustbepreferredforalarge-scaledeployment.Itcanalsobeusedasacircuitbreakertopreventresourceexhaustionbycontrollingthenumberofopenrequeststoaservicethathasencounteredconsecutivefailuresorisnotavailable.

DatamanagementInamicroservices-basedarchitecture,thedatamodelandschemamustnotbesharedamongboundedcontexts.Eachmicroservicemustimplementitsowndatamodelbackedbyadatabasethatisaccessibleonlythroughtheserviceendpoints.Microservicesmayalsopublisheventsthatcanbeconsideredasalogofthechangestheserviceappliestoitsisolateddatabase.Keepingapplicationdatauptodateacrossmicroservicesmayalsoaddtothenetworkoverheadanddataduplication.

DirectlookupAlthoughmicroserviceshavetheirownisolatedpersistence,anapplicationimplementedusingmicroservicesmayneedtosharedataamongasetofservicestoperformtasks.Inamonolithicenvironment,sincethereisacommondatabase,itiseasiertosharedataandmaintainconsistencyusingtransactions.Inamicroservicesenvironment,itisnotrecommendedtoprovidedirectaccesstothedatabasemanagedbyaservice,asshowninthefollowingdiagram:

Forexample,whenauserplacesaneworder,theOrderServicemayneedaccesstothedeliveryaddress;thatis,theuseraddressfromtheUserService.Similarly,oncetheorderisplaced,theUserServicewouldliketoupdatethetotalordersplacedtilldatebytheuserinitsdatabase.SincetheuserdatabaseismaintainedbytheUserServiceandtheOrdersDatabaseismaintainedbytheOrdersService,thesetwoserviceswillgettherequireddetailsviatheAPIsexposedbytheotherserviceonly.Theyshouldnotbeallowedtodirectlyupdateoraccessdatabasesmaintainedbyanotherservice.Thishelpsinalwaysmaintainingasinglecopyoftheuserandorderdatabasesthatisnotaccessibletoanyothermicroservicedirectly.

AsynchronouseventsGettingdataviaserviceendpointssynchronouslymaybecomeoverwhelmingforservicesthatmaintainawidelyuseddatabase,liketheusersdatabase.Therefore,itisrecommendedforservicestomaintainaread-onlycacheforsuchdatabasesandkeepituptodateasynchronouslyusingevents,asshowninthefollowingdiagram:

Forexample,insteadoflookinguptheaddressorordercountusingserviceendpointssynchronously,servicessuchasUserServiceandOrdersServicecanpublishtheeventsofinterestonamessagequeueinorderofoccurrence.TheUserServicecanthenreceivetheorderseventfromtheOrdersServiceviatheMessageBrokerandupdateitsdatabasewiththeorderscountorcacheit.Similarly,theOrdersServicecanreceiveanyaddressupdateeventfromtheUserService,keeptheaddressuptodatefortheuserwithinitscache,andrefertoitasandwhenrequiredtogenerateordersforusers.

Microservicesshouldalwayshaveanisolateddatabase,butitisnotrecommendedtocreateseparateservicestoisolateimmutabledatabasessuchasgeolocations,PINcodes,domainknowledge,andsoon.Sincethesedatabasedonotchangethatoften,itisfair

enoughtoshareandcachetheseacrossmicroservices.

CombiningdataInamonolithicenvironment,combiningdataiseasy;youneedtojustjointwotablestocreatetherequiredview.Inmicroservices,datasetsaredistributedacrossmicroservicesandcombiningthemrequiresmovingthedataacrossmicroservices,whichmayinvolvesignificantnetworkandstorageoverhead.Italsobecomeschallengingtokeepthecombineddatauptodate.Therearemultiplewaystosolvetheproblemofcombiningdataorjoinsinamicroservicesarchitecturebasedonthescopeoftherequest.

Forexample,ifyouwishtobuildanordersummarypageforaparticularuser,youneedtogetonlythatuser'sdatafromtheUserServiceandalltheordersforthatuserfromtheOrdersService.Thesecanbeobtainedindependentlyandjoinedattherequestingserviceleveltogeneratetheordersummary,asshownintheprecedingdiagram.Thesekindsofjoinworkwellfor1:Njoins.

Real-timejoinsworkwellforlimiteddatasets,butitisexpensivetocombinedatainrealtimeforeachrequest.ImaginetensofthousandsofsimilarrequestshittingtheOrderSummaryServiceeverysecond.Insuchscenarios,services

shouldinsteadkeepdenormalized(https://en.wikipedia.org/wiki/Denormalization)combineddatainacachethatiskeptuptodateusingtheeventsgeneratedbythesourceservices.Theservicecanthenrespondtotherequestsbyjustlookingupthisdenormalizeddatacacheinrealtime.Thisapproachscaleswellattheexpenseofdatabeingnearrealtime.Thedatainthecachemightbeoffbythetimesourceservicegeneratestheeventandtargetservicepicksitupandmakeschangestoitscache.

Forexample,asshownintheprecedingdiagram,anInterestServicemayreceiveuserinterestsviaitsAPIendpoint,butitmayneedtheuserandorderdetailsfromtheUserandOrdersservicesrespectively.Insteadofdirectlylookingupdetailsforeachuserinterest,theInterestServicemaysubscribetotheeventsgeneratedbytheuserandordersserviceandinternallykeepadenormalizedcacheviewofinterestdatathatisreadilyavailablewithalltherequireddetailsofusersandorders.

TransactionsEachmicroservicecanuseadatabaseofitschoice.ThechosendatabasesmayormaynothavetheACIDproperty(https://en.wikipedia.org/wiki/ACID)andsupporttransactions.Thisisoneofthereasonswhydistributedtransactionsarehardtoimplementwithmicroservices.However,businesstransactionsinvolvingchangesacrossmultiplebusinessentitiescannotbeomittedentirely,andthereforemicroservicesimplementdistributedtransactionsbyusingdataworkflows,asshowninthefollowingdiagram:

Microservicespublisheventswhenevertheymakeachangetothedatabase.Theeventscontainthetypeofchangealongwithimmutabledataaboutthebusinessentitiesthatwereaffectedbythischange.Otherservicesthenlistentotheseeventsasynchronouslyandperformthechangesstrictlyintheorderinwhicheventswerepublished.Asingletransactionmaycontainoneormoreeventsthatmayresultincascadingeventsgeneratedbythemicroservicesthatareaffectedbyit.Duetotheasynchronousnatureoftheeventflow,theconsistencyachievedacrossmicroservicesinthiscaseiseventual(https://en.wikipedia.org/wiki/Eventual_consistency).

Ifatransactionfails,theservicethatencountersthefailuregeneratescompensatoryeventstonullifythechangesmadeacrossmicroservicesthathavealreadyprocessedthetransactioneventsinthechain.Thecompensatoryeventsflowbackwardstowardstheoriginofthetransaction,asshownintheprecedingdiagram.Compensatoryeventsareidempotentinnatureandretrieduntiltheysucceed.

ThetransactionpatternformicroservicesisinspiredbySagas(http://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdf)andwasproposedbyHectorGarcia-MolinaandKennethSalem,aspublishedinanACMpaperin1987.Sagasmaybeimplementedasasetofworkflows,whereateachstepoffailureacompensatingactionistriggeredtobringthesystembacktoitsoriginalstateasitwasbeforetheworkflowwastriggered.

AutomatedcontinuousdeploymentThecorephilosophyofamicroserviceenvironmentmustbebasedontheYoubuildit,yourunit(https://queue.acm.org/detail.cfm?id=1142065)model;thatis,theteamworkingonthemicroservicemustownitendtoend,rightfromdevelopmenttodeployment.Theinfrastructurerequiredfortheteamtointegrate,test,anddeployanychangesmustbecompletelyautomated.Thismakesitpossibletobuildacontinuousintegrationandcontinuousdelivery(CD)pipeline,whichisthebackboneofmicroservices-basedarchitectures.

CI/CDThetermCI/CDcombinesthepracticesofCIandCDtogetherasaseamlessprocess.Inatypicalmicroservices-basedsetup,theteamcontinuouslyworksonenhancingthefeaturesofthemicroserviceandfixingtheissuesthatareencounteredinproduction.

Theentirecyclefromdevelopmenttodeploymenthasthreemajorphases,asshowninthefollowingdiagram:

Inthefirstphase,theteamcommitsthechangestoaversioncontrolrepository.Theversioncontrolrepositoryusedformicroservicesshouldbecommonforalltheservicesandapplications.Consolidatingalltheimplementationsinasingleversioncontrolsystemhelpsinautomationandrunningapplication-wideintegrationandacceptancetests.Someversioncontrolservicesalsosupportafine-grainedcollaborationsystemthatallowsthedeveloperstonotonlysharetheirchangeswithagroupofdevelopers,butalsohaveactivereviewsandafeedbacksystemwithintheteambeforethechangesarecommitted.

VersioncontrolsystemslikeGitareidealforadistributedteamworkingonmultiplemicroservicesduetoitsinherentfeatures.ServicelikeGitHubandBitbucketaresomecloudhosting

providersforGitthatalsohavethecapabilitytobuildtriggersforCI/CDsystems.

Inthesecondphase,aCI/CDsystemsuchasJenkinsisusedtobuildthechangesandruntheunittestsforthemicroserviceforwhichthechangeiscommitted.Runningthetestsforeachchangerequesthelpsindetectingissuesbeforethechangesareintegratedwiththerestoftheapplication.Italsohelpstodetectanyregressions(https://en.wikipedia.org/wiki/Software_regression)thatmighthavebeenintroducedduetotherecentchanges.Ifthereareanytestfailures,analertissentbacktotheteam,especiallythecommitterwhosubmittedthechangerequest.

Theteamthenfixesthetestsandsendsthechangerequestagaintotheversioncontrolsystemthatin-turntriggersthebuildtoretestthechanges.Thisprocessrepeatsuntilallthetestssucceed.Oncethetestssucceed,theCI/CDsystemmergesthechangeswiththemainlineandpreparesareleaseartifactfortheservice.Artifactsformicroservicesareoftenpackagedascontainersthatarereadilydeployablebyorchestrationtools.Packagingmicroservicesinacontaineralsohelpsinautomatingthedeploymentandscalingoftheserviceson-demand.

Dockerisonepreferredtechnologyforpackagingmicroservices.IthasseamlessintegrationwithmultipleorchestrationtoolssuchasKubernetesandMesos(http://mesos.apache.org/).

Inthethirdphase,theCI/CDsystempublishesthereleasestoacentralrepositoryandinstructstheorchestrationtoolstopickthelatestversionofthemicroservicethatcontainstherecentchanges.Theorchestrationenginethenpullsthelatestreleasefromtherepositoryanddeploysitinproduction.Alltheinstancesinproductionaremonitoredbyautomatedtoolsthatgeneratealertsfortheteamifthereareanyissues.Ifthereareanyissuesencounteredbytheteam,theteamfixestheissuesandsubmitsthechangerequesttotheversioncontrolsystemthattriggersthebuildandtheentireprocessrepeatstopushthechangestoproduction.

Generally,theorchestrationenginedoesarollingupgradeoftheservicewhiledeployingupdates,butsometeamsprefertodotheA/Btestingbyupgradingonlyasubsetofdeployedserviceinstancesandroll-outonlywhenthetestssucceedforthatsubset.SuchdeploymentsareoftenreferredtoasBlueGreenDeployment(h

ttps://martinfowler.com/bliki/BlueGreenDeployment.html).

Suchanautomatedenvironmenthelpstheteamcutshorttheentiredevelopment-to-deploymentcyclefrommonthstodaysanddaystohours.LargecompaniessuchasGoogle,Facebook,Netflix,Amazon,andsoonarenowabletopushmultiplereleasesinadayduetosuchautomatedenvironmentsandrobusttestingprocesses.

ScalingTheArtofScalability(http://theartofscalability.com/)bookusesascalecubemodeltodescribethreeprimaryscalingpatternsforanapplication,asshowninthefollowingdiagram.Thex-axisofthecuberepresentshorizontalscaling;thatis,deployingthesameinstanceoftheapplicationbyjustcloningthemandfront-endingbyaloadbalancertodistributetheloadevenlyamongtheinstances.Thisscalingpatternisquitecommonforhandlingahighnumberofservicerequests.Thez-axisofthecubeaddressesscalingbydatapartitioning.Inthiscase,eachapplicationinstancedealswithonlyasubsetofdata.Thisscalingpatternisparticularlyusefulforapplicationswherethepersistencelayerbecomesabottleneck:

They-axisofthecubeaddressesscalingbysplittingtheapplicationbyfunctionorservice.Thispatternrelatesdirectlytothemicroservicespattern.Thefaceofthecubecreatedusingthexy-axiscombinesthebestpracticesofscalingformicroservices-basedarchitecture.Microservicesareidentifiedbysplittinganapplicationbyboundedcontexts(y-axis)andscaledbycloningeachinstance(x-axis).Formicroservices,cloningisdonebydeployingmultipleinstancesofaservicecontainer.

SummaryInthischapter,welearnedaboutdomain-drivendesignandtheimportanceofidentifyingtherightboundedcontextformicroservices.Welearnedaboutthehexagonalarchitectureformicroservicesandvariousmessagingpatterns.Wealsodiscusseddatamanagementpatternsformicroservicesandhowtosetupserviceregistriestodiscovermicroservices.Weconcludedwiththeimportanceofautomatingtheentiremicroservicedeploymentcycleincludingtesting,deployment,andscaling.Inthenextchapter,wewillintroduceareal-lifeusecaseformicroservicesandlearnhowtodesignanapplicationusingtheconceptslearnedinthischapter.

MicroservicesforHelpingHandsApplication

"Welearnbyexampleandbydirectexperiencebecausetherearereallimitstotheadequacyofverbalinstruction."

-MalcolmGladwell,Blink:ThePowerofThinkingWithoutThinking

Microservicesaregainingpopularityforinternet-scaleapplicationsthattargetconsumers.Inthischapter,youwilllearnhowtoapplyprinciplesofmicroservicesarchitecturetodesignasimilar,internet-scale,fictitiousapplicationcalledHelpingHandsthatconnectshouseholdserviceproviderswithconsumersofservicessuchashomecleaning,appliancerepair,pestcontrol,andsoon.Inthischapter,youwill:

LearnhowtogatherrequirementsandcaptureuserstoriestodesigntheHelpingHandsapplicationLearntheimportanceofmonolithic-firstdesignLearnhowtomovetowardsamicroservices-baseddesignLearnhowtouseevent-drivenarchitecturewithmicroservices

DesignOneofthebestwaystodesignasoftwaresystemistocapturethebusinessdomain,itsusers,andtheirinteractionwiththesystemasauserstory(https://en.wikipedia.org/wiki/User_story).Userstoriesareaninformalwayofcapturingtherequirementsofasoftwaresystem.Inuserstories,thefocusisontheendusersandtheinteractionsthatarepossiblebetweentheusersandthesystem.

UsersandentitiesThefirststepinwritinguserstoriesfortheHelpingHandsapplicationistounderstandtheusersandentitiesofthesystem.Primarily,therearetwousersofthesystem—ServiceConsumersandServiceProviders,asshowninthefollowingdiagram.ServiceConsumerssubscribetooneormoreservicesprovidedbytheServiceProviders.Thecoreentityoftheapplicationistheservice.Aserviceisanintangible,temporal,andlimitedassetthatprovidersownandprovidetotheconsumerson-demandataprice.

ServiceProvidersregisteroneormoreserviceswiththesystemthatcanbesubscribedtobytheconsumers.Allservicesareregisteredwithavailabletimeslotsandthedurationforwhichtheycanbeoffered.EachservicedurationandtimeslothasanassociatedpricethattheServiceConsumerhastopaytomakeuseoftheservice.ServiceProvidersarealsoresponsibleformaintainingtheavailabilitystatusoftheserviceswiththesystem.ServiceConsumerscansearchforservicesavailablefromasetofprovidersandcanpickaprovideroftheirchoice.Oncechosen,theServiceConsumercanscheduleaservicefromtheServiceProviderbasedonavailability.

Userstories

ThenextstepistolisttheuserstoriesthatwillbesupportedbytheHelpingHandsapplication.Herearetheuserstoriesfortheapplication:

AsaServiceConsumer,IcancreateanaccountsothatIcansearchforservicesandbookthemAsaServiceConsumer,IcansearchforrequiredservicessothatIcanbookoneformytaskAsaServiceConsumer,IcansubscribetooneormoreservicessothatIcangetmytaskdoneonaregularbasisAsaServiceConsumer,IwouldliketoratetheservicesofferedsothatotherscanbenefitfromthefeedbackandchoosethebestservicesofferedforaparticulartaskAsaServiceConsumer,IwanttogetnotificationssothatIcangetremindedoftheservicescheduleAsaServiceProvider,IcancreateanaccountsothatIcanregisteroneormoreservicesforconsumersAsaServiceProvider,IwanttoregisteroneormoreservicessothatIcangetservicerequestsAsaServiceProvider,IwanttospecifytheservicelocationareasothatIcangetonlyservicerequeststhatareneartomyplaceAsaServiceProvider,IwanttospecifythepriceandavailabilitysothatIcangetonlyservicerequeststhatarefeasibletoserveandtheonesIaminterestedinAsaServiceProvider,IwanttogetnotificationswhenaservicerequestisplacedsothatIcanattendtoit

Apartfromuserstories,therearesomenon-functionalrequirements(https://en.wikipedia.org/wiki/Non-functional_requirement)aswellthatmustbeaddressedbytheHelpingHandsapplication:

Alltheimplementationsmustbetrackedandversionedinarevisioncontrol

system.TheHelpingHandsapplicationwillusetheexternalhostingserviceGitHubtotrackthecodebase.Allthedependenciesmustbeexplicitlydeclared.TheHelpingHandsapplicationwillbeimplementedusingClojure(https://clojure.org/)withLeiningen(https://leiningen.org/)fordependencymanagement.Allservicesmusthaveauthenticationandauthorizationbuiltin.Configurationsmustbespecifiedexternallyandnothardcodedintheapplication.Alleventsmustbeloggedtounderstandthestateoftheapplicationandmonitoritinproduction.

Non-functionalrequirementsareapartoftwelve-factormethodology(https://12factor.net/),whichcoverstwelvedifferentaspectsthatmustbeaddressedbytheapplication.Part-3andPart-4ofthisbookaddresssomeoftheseimportantaspectsfortheHelpingHandsapplicationindetail.

DomainmodelBasedontheuserstories,ServiceOrderandService(catalog)arethetwocoredomainsoftheHelpingHandsapplication.Apartfromthesetwodomains,useraccounts,provideraccounts,andnotificationsarethegenericdomainsthatformtheHelpingHandsapplication.ServiceConsumer,ServiceProvider,Service,andServiceOrderareentitiesoftheHelpingHandsapplication.Theseareshownalongwiththeirfieldsinthefollowingdiagram:

EachServiceConsumerhasanIDassignedinthesystemandhasName,Address,Mobile,andEmaildefinedassomeoftheattributes.TheGeoLocationofeachconsumerisderivedfromtheaddressinformation.EachServiceProvideralsohasanIDassignedinthesystemwithName,Mobile,

ActiveSince,andOverallRatingassomeoftheattributes.AServiceisregisteredbytheServiceProviderandhasauniqueIDdefinedinthesystem.ItisregisteredagainstaServiceTypeandhasoneormoreServiceAreasdefined.BasedontheServiceAreas,thesystemderivestheGeoLocationBoundarywheretheserviceisofferedtotheconsumers.AServicealsohasanHourlyCostsetbytheServiceProviderandkeepsanOverallRatingbasedonthepreviousorders.

AServiceConsumersubscribestotheServiceandplacesaServiceOrderfortheServiceProvidertofulfill.EachServiceOrderhasanIDdefinedinthesystemandhasanassociatedServiceID,ProviderID,andConsumerID.EachorderentryhasanassociatedTimeSlotbasedonwhichtheCostisdeterminedbythesystem.TheServiceOrderalsohasastatusthatisupdatedbytheServiceProvideroncetheorderiscompleted.TheorderalsokeepsaRatingbetween1and5,asratedbytheconsumerfortheserviceprovided.

TheHelpingHandsapplicationalsoalertstheServiceConsumerandtheServiceProviderwithrespecttotherelatedServiceOrder.AnychangeinStatusisnotifiedtoboththeparticipantsviaSMSsentovertheregisteredmobilenumber.TheServiceConsumercanalsosubscribetoServiceandreceiveallupdateswithrespecttoservicestatus,availability,costchanges,andsoon.

MonolithicarchitectureTheHelpingHandsapplicationcanbedesignedusingathree-layeredarchitectureofpresentation,businesslogic,andpersistence.Basedonthedomainmodel,therecanbefourmaintablesintheHelpingHandsapplicationdatabasecorrespondingtoeachentity.Therewillbeasingledatabasethatwillstoreallthedatainthedesignatedtable.Thedatabasemustbeaccessibletoallthecomponentsofthesystem.Thebusinesslogiclayerwillhavewell-definedcomponentsbasedontheprincipleofSeparationofConcerns(SoC).ComponentswilladdressalluserstoriesfortheHelpingHandsapplication.

Applicationcomponents

Toaddresstheuserstories,therewillbethreemaincomponentsandtwohelpercomponents,asshowninthefollowingdiagram.TheRegistrationComponentwillmanagealltheuseraccountsandrelatedCRUDoperations.TheServiceComponentwillhandleallservice-relatedoperationssuchascreate,update,andlookup.TheOrderComponentwillhelpplacetheorders,searchforthehistoricalorder,andalsoratetheservices.ItwillalsohelptomaintainastatusfortheserviceorderandgeneraterelevantalertsintheformofSMSandemailusinganexternalservice:

Therewillbetwohelpercomponents—GeoLocationandAlerting.ThesecomponentswillhelpconnecttheapplicationtoexternalservicestogetgeolocationtagsandsendalertsintheformofSMSandemailviaprovidedAPIs.Components,theirresponsibilities,andrelateddatabasesaresummarizedinthefollowingtable:

Component Responsibilities Component

RegistrationComponent

Create/update/deleteaccountforserviceconsumersCreate/update/deleteaccountforserviceproviders

GeoLocationComponenttogetthelongitudeandlatitudebasedontheaddress

ServiceComponent

Create/update/deleteservicesSearchforservicesbykeywords,types,location,andsoon

GeoLocationComponenttogetthelongitudeandlatitudebasedonservicearea

OrderComponent

Create/updateordersSearchforordersbykeywords,types,timespan

AlertingComponenttogeneratealertswithrespecttoorders

GeoLocationComponent

Lookuplongitudeandlatitudebyaddress ExternalService(API)

AlertingComponent

SendemailandSMSalerts ExternalService(API)

Componentssuchasauthenticationanduserssuchasadministratorsandsoonhavebeenintentionallyomittedfromtheexplanationtofocusonlyonthecorecomponentsandfeaturesoftheapplication.

DeploymentTheHelpingHandsapplicationcanbedeployedasasingleartifactinanapplicationserver.Oncetheservicebecomespopularitmustbescaledtohandleincomingservicelookupsandorderrequests.Theeasiestwaytoscaletheapplicationforincomingrequestsistodeploymultipleinstancesoftheapplicationandfrontenditwithaloadbalancer,asshowninthefollowingdiagram.Theloadbalancerthenhelpstodistributetherequestsamongthedeployedinstancesoftheapplication.Sincealltheinstancesusethesamedatabase,allofthemarealwaysin-syncandhaveaccesstoconsistentdata.Consistencyisachievedduetotheinherentcapabilitiesofthedatabaseusedfortheapplication;itsupportsACIDtransactions:

LimitationsAlthoughamonolithicarchitecturefortheHelpingHandsapplicationfulfillsthepurpose,ithassomeinherentlimitations.Thereisahighcouplingofthecomponentswiththeconsumersandproviderstable.Anychangeinthesetwotableswillaffectalmostallthecomponentsofthesystemandwillrequireredeploymentoftheentireapplication.

TheHelpingHandsapplicationalsodependsontwoexternalservices.Supposeoneoftheseservicesshutsdownorthereisarequirementtomovetoabetterservice;thecorrespondinggeolocationoralertingcomponentwillbechangedandwillresultintheredeploymentofalltheinstancesoftheapplication,eventhoughtherewasnochangeinthecorefunctionalityandservicesoftheapplication.Thisaddstothedeploymentoverheadforsimplechangesaswell.

Sincetheentireapplicationisdeployedasasingleartifact,scalinganapplicationscalesallthecomponentsoftheapplicationequally.Forexample,toscalewithincomingorderandservicelookuprequests,theRegistrationComponentisunnecessarilyscaledwiththeorderandservicecomponents.Thisalsoincreasestheloadonthedatabasethatishandlingalltheincomingrequestsfromthecomponents.Often,requestsfromonecomponentcanaffectthedatabaseperformanceforothercomponentsaswellandreducetheperformanceoftheentireapplication.

AnotherlimitationofthecurrentmonolithicarchitectureoftheHelpingHandsapplicationisitsdependencyonasingledatabasetechnology.Inpractice,afuzzysearchforservicesusingtagsandlookupusinggeolocationscanbesupportedbetterbydatabasessuchasElasticsearch(https://www.elastic.co/products/elasticsearch)ascomparedtorelationaldatabasessuchasMySQL(https://www.mysql.com/).Relationaldatabasesarebettersuitedfortransactionaloperationssuchascreatingserviceordersandmaintaininguseraccounts.Withthecurrentarchitecture,thereisonlyonedatabasetechnology,andthataffectstheefficiencyoftheapplicationandmakesitlessflexible.

MovingtomicroservicesThelimitationsofamonolithicarchitectureforHelpingHandscanbeaddressedbyseparatingoutthecomponentsalongwiththedatabaseasamicroservice.Theseservicescanthenmakeinformedchoicesaboutthetechnologystackanddatabasethatsuitthemwell.Theseservicescanbedeveloped,changed,anddeployedinisolationaspertheconceptsofamicroservices-basedarchitecture.Toidentifytheboundedcontextforthecomponentsofanexistingmonolithicapplication,itisrecommendedtolookatthedatabaseaccesspatternandrelatedbusinesslogicfirst,isolatethem,andthenlookatthepossibilitiestoisolatethecomponentsfurtherbasedonbusinesscapabilities.

IsolatingservicesbypersistenceIntheexistingmonolithicapplicationofHelpingHands,theconsumersandprovidersdatabasetablesareaccessedbyallthecorecomponentsofthesystem,asshowninthefollowingdiagram.Thesetablesareprimecandidatesforbeingwrappedaroundaserviceandisolatedinaseparatedatabasethatisaccessibleonlytothecorrespondingservicedirectly.AllotherservicesmusttalktotheServiceConsumerserviceandtheServiceProviderserviceforanydetailsinsteadofdirectlyaccessingtheconsumersandprovidersdatabases.

Sincethereisaseparateservicecreatedtohandletherequestsforconsumersandproviders,thereisnoneedtohaveaservicecorrespondingtotheRegistrationComponent.TheServiceConsumerserviceandServiceProviderservicecannowhandlealltherequeststoregister,modify,ordeleteconsumersandproviders,respectively.Similarly,theserviceandorderservicescannowhandlealltherequestsrelatedtoservicesandorders,respectively,byisolatingthecorrespondingdatabases.TheorderservicecannowtalktoServiceConsumer,ServiceProvider,andServicetogettherequireddetailsfortheorder.

TheHelpingHandsapplicationwillbeusingacombinationoftheDatomic(http://www.datomic.com/)andElasticsearch(https://www.elastic.co/products/elasticsearch)databasesforvariousmicroservices.Part-3ofthisbookdiscussesthepersistencelayerindetail,andthelastchapterofPart-2introducesDatomic.

IsolatingservicesbybusinesslogicOncepersistence-basedservicesareisolated,thenextstepistoevaluateexistingcomponentsformicroserviceswithrespecttobusinesslogic.ApartfromdroppingtheRegistrationComponentinfavorofseparateservicesforconsumerandprovider,anewservicecalledlookupcanbecreatedtoconsolidateallthesearchoperationsintooneserviceandallowuserstosearchacrossapplicationentities,asshowninthefollowingdiagram.Sincedatabasesofconsumers,providers,services,andorderscannotbesharedwithlookupservices,itcankeepadenormalized(https://en.wikipedia.org/wiki/Denormalization)viewofthesedatabasescontainingonlythefieldsthatneedtobesearched.

Geolocation-basedquerieswillalsobelimitedtolookupservices,sothereisnoneedtomaintainaseparategeolocationservice;instead,theLookupServiceitselfcanqueryforthegeolocation.

Sincegeolocationsrarelychange,theLookupServicecancachethemandmaintainadatabaseofwell-knownandalreadyqueriedgeolocationsaswellforbetterperformance.

TheAlertingComponentcanbeisolatedasaseparateserviceasitwillberequiredbymultipleservices,includingOrder,ServiceConsumer,andService

Provider,tosendalertstotheusers.AlertsmaybesentviaSMSoremail,andtheAlertingServicecanuseexternalservicestosendthealerts.Sincealertsmustnotbeoverwhelmingforusers,theAlertingServicecangroupbyuserallthealertsthatarerequestedinashortperiodoftimeandsendthemasasinglenotificationmessage.

Donotattempttoaggressivelystartdisintegratingyourcomponentsintomicroservices.Focusonthebusinesscapabilitiesandnotonfeaturesandusecases.Forexample,insteadofcreatingaseparateserviceforsendingemailsandsendingSMS,itisrecommendedtocreateasingleAlertingServicewithbothcapabilities.

MessagingandeventsThenextstepfortheHelpingHandsapplicationistodefinetheinteractionsbetweentheidentifiedmicroservices.Microservicescaneitherinteractbydirectlysendingthemessagestootherserviceendpointssynchronouslyortheycansubscribetotheeventsgeneratedbyothermicroservicesandreceivethemessagesasynchronously.Asynchronousmessagesrelyontheunderlyingmessagebrokeranditsdurability.Messagebrokersnotonlyhelptoscaletheapplicationbyholdingthemessagesyettobeprocessedinthequeue,butalsosupportdurabledeliveries.Evenifaservicefails,itcanberestoredandallowedtostartprocessingpendingmessagesfromthepointwhereitleftoff.CombiningbothsynchronousandasynchronousmessagepatternsfortheHelpingHandsapplicationgivesusaflexibleandperformantarchitecturetoaccomplishagiventask,asshowninthefollowingdiagram:

AlltheservicesoftheHelpingHandsapplicationmustpublishchangelogeventsrelatedtothebusinessentitiesonamessagequeuethatcanbereadby

anyservicethatsubscribestoit.Thepublishedeventsmustberetainedbythemessagequeueforaconfiguredamountoftime,beyondwhichtheeventsmaybediscarded.Forexample,allthecoreservices—ServiceConsumer,ServiceProvider,Service,andOrder—publisheventsontheirdesignatedmessagequeuesattheallocatedtopic.

TheLookupServicemustsubscribetoalltheeventspublishedbytheConsumer,Provider,Service,andOrderservicestomaintainalocaldenormalizeddatabasetosupportsearchqueries.Itmustaddgeolocationdetailsbyqueryingtheexternalserviceandcachingtheresultslocally.Anychangesdonetotheconsumers,providers,services,andordersdatabasesmustbecommunicatedtotheLookupServiceviaevents,asynchronously.TheLookupServicemayalsopublisheventstoitsdesignatedmessagequeueforotherservicestoconsume.Theseeventsareoftenusefultoanalyzethenumberofsearchqueriesreceived,trendingservices,andsoon.

ServicessuchasAlertingarebestsuitedforsuchasynchronousmessages.TheAlertingServiceshouldnotonlyrelyonthemessagebrokerforvariousdeliverysemantics,suchasat-leastorexactly-oncedeliverybutmustalsoreadbatchesofalerts,combinealertsforthesameuserandsendthemasasingleconsolidatedalert.

ServicessuchasOrderServicemayalsorelyondirectmessagestoretrievedetailsoftheconsumer,provider,andtheservicebeforeregisteringanorderfortheuser.Oncetheorderisregistered,achangelogeventmustbepublishedbytheOrderServicefortheLookupServicetomaketheorderavailabletobesearched.

Eventlogsarealsousefultosetupadeepmonitoringandreportinginfrastructureformicroservices.Part-4ofthisbookdescribesthemonitoringandreportingpatternforthemicroservicesarchitecturethatisbasedoneventlogs.

ExtensibilityAmicroservices-basedarchitecturefortheHelpingHandsapplicationnotonlymakesiteasiertodeployandscalebutalsomakesithighlyextensible.Forexample,bydeployingaseparateLookupServiceforsearchoperations,itisnowpossibletouseadatabasesuchasElasticsearchonlyforsearchoperationsandDatomicforallothermicroservicesthatrequireconsistenttransactions.

Inthefuture,ifthereisabettertechnologyavailable,itwillbeeasiertodeployserviceswithanewertechnology.Newertechnologymayalsocomewithhardwarechallenges.Forexample,adatabasesuchasMapD(https://www.mapd.com/)runsonGPUs(https://en.wikipedia.org/wiki/Graphics_processing_unit).Tousesuchdatabases,microservicesneedtorunonspecializedhardware.Sincemicroservicescanbedeployedinisolationonthesameoranentirelyseparatemachine,itispossibletodeployservicesthatneedGPUsonmachinesthatsupportGPUswithoutaffectingthewaytheyinteractwithotherservices.Thisisoneoftheadvantagesofmicroservices—youarenotboundbythetechnologyorunderlyinghardwareandanychangesdonearelocalizedwithintheboundedcontextoftheservice.

Dataanalyticsisnowanintegralpartofanyweb-basedapplication.Itnotonlyhelpsusunderstandtheusagepatternsoftheapplication,butalsohelpsprovidebetterservicestotheusers.Bygeneratingeventlogsforallthechangesdonetoentities,itisalsopossibletofurtherextendtheHelpingHandsapplicationtoanalyzeusagepatterns.Forexample,onecanlistentoeventsgeneratedbytheOrderServiceandstudyserviceusagepatternsbydemography.Itshouldalsobepossibletoanalyzethepopularityofservicesbylocationandprovidecustomizedofferstousersasnotifications.

WorkflowsforHelpingHandsDataworkflowsarethebackboneofanymicroservices-basedarchitecture.Theydefinethesequenceofmessagesandeventsthataregeneratedamongtheservicestoaccomplishadesiredtask.Aworkflowmayconsistofbothsynchronousandasynchronousmessages.

Workflowsshowninthissectionareonlyforexplanatorypurposesanddonotconformtotheexactsemanticsofsequencediagrams(https://en.wikipedia.org/wiki/Sequence_diagram).Detailsofauthentication,authorization,validation,anderrorconditionshavebeenomittedintentionally.

ServiceproviderworkflowTheserviceproviderworkflowconsistsofServiceProvider,LookupService,andAlertingService.TheServiceProviderComponentexposesendpointsforuserstocreateandupdateserviceprovidersfortheHelpingHandsapplication,asshowninthefollowingdiagram:

Forthecreateoperation,theServiceProviderservicefirstvalidatestheinputrequestfortherequiredparametersandprivileges,thenstartsatransactiontoaddanewserviceproviderwithinthesystem.AnewIDfortheserviceproviderisautomaticallygeneratedbytheServiceProvider.Oncethetransactionissuccessful,itemitsacreateeventonitseventlogqueuefortheLookupServiceandothersubscriberstopickup,andpublishesacreatealertonthemessagequeueoftheAlertingServiceforittopickandsendanalerttotheinterestedusers.

Toupdatetheserviceproviders,theusersendstheupdaterequesttotheServiceProvideranditinitiatesanewtransactiontoupdateitslocaldatabase.Oncethetransactionissuccessful,itemitsanupdateeventonitseventlogqueueforthe

LookupServicetopick-up.Updateoperationsarenotsentasalertsornotificationstousers.Ifthisisarequirement,itcanbeenabledbypublishingamessagefortheAlertingServiceonitsrequestqueue.

ServiceworkflowTheserviceworkflowdescribesthemessageinteractionamongtheServiceComponent,ServiceProvider,LookupService,andAlertingService,asshowninthefollowingdiagram.TheserviceexposesendpointsforuserstocreateandupdateservicesfortheHelpingHandsapplication.Eachservicemusthaveaserviceprovideralreadydefinedfortheapplication:

Tocreateanewservice,theusercallstheAPIendpointforServicewiththerequiredparameters.TheproviderIDmustbespecifiedtocreateanewservice.OncetheServiceComponentreceivestherequest,itfirstsendsadirectmessagetotheServiceProvidertovalidatethespecifiedproviderIDandmakesurethattheproviderisalreadyregistered.Iftheproviderisalreadyregistered,theServiceProviderreturnstheproviderdetailssynchronouslytotheService.

Oncetheproviderisvalidated,theServiceComponentstartsatransactiontoaddanewserviceintoitslocaldatabase.AnewIDfortheserviceisautomaticallygeneratedbytheServiceComponent.Oncethetransactionissuccessful,theServiceComponentemitsacreateeventonitseventlogqueuefortheLookupServiceandothersubscriberstopickup.TheLookupService

pullsthecreateevent,looksupthegeolocationcorrespondingtotheserviceareas,andupdatestheservicedetailsinitslocaldenormalizeddatabase.TheServiceComponentalsopublishesanewservicealertonthemessagequeueoftheAlertingServicetosendanalerttotheinterestedusers.

Toupdatetheservicedetails,theusersendsanupdaterequesttotheServiceComponentwiththeserviceID.Sincetheupdateoperationisperformedonlyforexistingservices,theserviceproviderisnotvalidatedinthiscase.TheServiceComponentinitiatesatransactiontoupdatetheservicedetailsandupdatesthespecifiedfields.Oncethetransactionissuccessful,itemitsanupdateeventontheeventlogqueuefortheLookupServicetopickup.TheLookupServicethenupdatesthelocaldatabasewiththechangesandalsoupdatesthegeolocationsfortheserviceareasiftheyarepartofthechangelog.Alertsarenotsentforserviceupdates.

ServiceconsumerworkflowTheserviceconsumerworkflowdescribesthemessageinteractionamongtheServiceConsumer,Lookup,andAlertingService.TheServiceConsumercomponentexposesendpointsforuserstocreateandupdateserviceconsumersfortheHelpingHandsapplication,asshowninthefollowingdiagram.

Tocreateanewserviceconsumer,theusersendsarequestwithvalidparameterstotheServiceConsumerComponent.Thecomponentthenvalidatestherequestandiftherequestisvalid,itstartsatransactiontoaddanewconsumertotheapplication.Foreachnewconsumer,anIDisautomaticallygeneratedbyServiceConsumerComponent.

Oncethetransactionissuccessful,theServiceConsumerComponentemitsacreateeventonitseventlogqueuefortheLookupServiceandothersubscriberstopickup.Afterreceivingthecreateevent,theLookupServicelooksupthegeolocationcorrespondingtotheaddressoftheconsumerandthenstorestheconsumerdetailsalongwiththegeolocationinitslocaldenormalizeddatabase.TheServiceConsumercomponentalsopublishesanewconsumeralertonthemessagequeueoftheAlertingServicetosendanalerttointerestedusers:

Toupdatetheconsumerdetails,theusersendstheupdaterequesttotheServiceConsumerComponentalongwiththeconsumerID.TheServiceConsumerComponentinitiatesatransactiontoupdatetheconsumerdetailsandupdatesthespecifiedfields.Oncethetransactionissuccessful,itemitsanupdateeventontheeventlogqueuefortheLookupServicetopickup.TheLookupServicethenupdatesthelocaldatabasewiththechangesandalsoupdatesthegeolocationfortheconsumeraddressifitisapartofthechangelog.Alertsarenotsentforconsumerupdates.

OrderworkflowTheorderworkflowinvolvesmostoftheservicesintheHelpingHandsapplication.TheOrderServicereceivestherequestfromtheconsumertocreateaneworderforthechosenservice.TheconsumersendsthecreaterequesttotheOrderServicewiththeproviderandservicedetailsalongwiththetimeslot.TheOrderServicethenvalidatestheorderdetailsbysendingadirectsynchronousrequesttotheServiceConsumer,ServiceProvider,andService.Ifthedetailsarevalid,thatis,theserviceisalreadyregisteredbythespecifiedproviderandtheconsumerisavaliduserinthesystem,thentheOrderServicesendsadirectsynchronousrequesttotheLookupServicetomakesurethattheserviceisavailableinthevicinityoftheconsumerlocationbasedonthegeolocationoftheconsumeraddressandtheservicearea.

Iftheserviceisfeasiblefortheconsumeraddress,theOrderServicestartsatransactionandcreatesaneworderinthesystem,asshowninthefollowingdiagram:

Oncethetransactionissuccessful,theOrderServiceemitsacreateeventonitseventlogqueuefortheLookupService;itreceivestheevents,updatestheorderlistwithinitslocaldatabase,andmakesitavailableforuserstosearch.TheOrderServicealsopublishesaneworderalertonthemessagequeueoftheAlertingServicetosendanalerttoboththeconsumerandtheprovideroftheservice.

Toupdatetheordertimeslot,status,orrating,theusersendsanupdaterequesttotheOrderService.SinceanupdatemessagerequiresanexistingorderID,theOrderServicedoesnotvalidatetheproviderandconsumerfortherequest,itjustmakessurethatitisanexistingorder.Inadditiontovalidatinganexistingorder,theOrderServicealsosendsadirectsynchronousrequesttotheLookupServicetomakesurethattherequestedtimeslotandservicearestillavailablefortheconsumer.Iftheyareavailable,thentheOrderServicestartsa

transactionandupdatesitslocaldatabase.Oncethetransactionissuccessful,itemitstheupdateeventonitseventlogqueuefortheLookupServiceandalsosendsanalerttotheAlertingServicebypublishinganupdateorderalert.Anychangemadetotheorderissentasanalerttoboththeconsumerandprovideroftheserviceorder.

Inpractice,therewillbeseparateservicesandworkflowsforauthenticationandauthorization.Thoseworkflowshavebeenintentionallyomittedfromthischaptertofocusonlyonthecoreservicesandworkflows.Part-4ofthisbookdescribesauthenticationandauthorizationservicespatternsindetail.

SummaryInthischapter,wedesignedanapplicationcalledHelpingHands,usingthebestsoftwareengineeringpractices.Westartedwithamonolithicarchitectureandarguedwhyamicroservices-basedarchitectureiswellsuitedfortheHelpingHandsapplication.Inthenextpartofthisbook,wewillfirsttakealookatthebasicdevelopmenttoolsandlibrariesthatwewillbeusingtobuildourHelpingHandsapplication.

DevelopmentEnvironment

"Themechanicthatwouldperfecthisworkmustfirstsharpenhistools."

-Confucius

Thedevelopmentenvironmentconsistsoftoolsandlibrariesthatareusefultoimplement,debug,andmakechangestosoftwaresystems.Theefficiencyofadevelopmentteamishighlydependentonthedevelopmentenvironmentandtechnologystackathand.Inthischapter,youwilllearnhowtosetupadevelopmentenvironmentformicroservicesusingtheClojureecosystem.Thischapterwillhelpyouto:

LearnthehistoryofClojureandfunctionalprogrammingLearntheimportanceofREPLLearnhowtobuildyourapplicationusingClojurebuildtoolsLearnaboutwellknownintegrateddevelopmentenvironments(IDEs)

ClojureandREPLClojure(https://clojure.org/)isadialectoftheLisp(https://en.wikipedia.org/wiki/Lisp_(programming_language)programminglanguageandprimarilyrunsonaJavavirtualmachine(JVM).TheothertargetimplementationsincludeClojureCLR(https://github.com/clojure/clojure-clr),whichrunsonCommonLanguageRuntime(CLR),andClojureScript,whichcompilestoJavaScript.AlthoughClojureusesaJVMasitsunderlyingruntimeengine,itemphasizesafunctionalprogramminglanguagewithimmutabilityatitscore.AlldatastructuresofClojureareimmutable.SinceClojureisadialectofLisp,italsotreatscodeasdataandisknowntobehomoiconic.ItssyntaxisbuiltonS-expressions(https://en.wikipedia.org/wiki/S-expression)thatarefirstparsedasadatastructureandthentranslatedintoconstructsoftheJavaprogramminglanguagebeforebeingcompiledintoJavabytecode.Clojurealsosupportsmetaprogrammingwithmacro(https://en.wikipedia.org/wiki/Macro_(computer_science)).

ClojureintegrateswellwithexistingJavaapplicationsduetoitsprimarysupportforunderlyingJVMsanditsentireecosystem.AllexistinglibrariesthatrunonaJVMintegrateseamlesslywithClojure,includingMaven,Java'sbuildsystem.

HistoryofClojureClojureisafunctionalprogramminglanguagethattreatsfunctionsasfirst-classobjects;thatis,itsupportspassingfunctionsasargumentstootherfunctionsandalsoreturningthemasvaluesfromotherfunctions.Italsosupportsanonymousfunctions,assigningfunctionstovariablesandstoringthemindatastructures.

ThetimelineintheprecedingdiagramshowstheevolutionoffunctionalprogrammingthatledtothedevelopmentofClojure.TheconceptoffunctionalprogrammingoriginatedfromaformalsystemcalledLambdacalculusthatwasintroducedbyAlonzoChurch(https://en.wikipedia.org/wiki/Alonzo_Church)in1930.Churchproposedauniversalmodelofcomputationthatwasbasedonfunctionabstractionanditsapplicationusingvariablebindingandsubstitution.Church'smodellaidthefoundationoffunctionalprogramminglanguagesthatareknowntoday,includingClojure.

Almostthreedecadeslaterin1958,JohnMcCarthy(https://en.wikipedia.org/wiki/John_McCarthy_(computer_scientist))introducedtheLispprogramminglanguage,whichwashighlyinfluencedbythenotationsofLambdacalculus.Itwasdistinctiveduetoitsfullyparenthesizedprefixnotation.Itssourcecodewasmadeoflists,hencethenameLIStProcessor.LISPintroducedmanyotherconceptsincludingtreedatastructure,dynamictyping,high-orderfunctions,andread-eval-print-loop(REPL).TwowidelypopularLispdialectsareScheme(https://en.wikipedia.org/wiki/Scheme_(programming_language))andCommonLisp(https://en.wikipedia.org/wiki/Common_Lisp),bothinventedbyGuySteele(https://en.wikipedia.org/wiki/Guy_L._Steele_Jr.).Schemewasintroducedin1970followedbyCommonLispin1984.ClojureisalsooneofthedialectsofLispthatwasintroducedin2007byRichHickey.

InadditiontobeingadialectofLISP,ClojurealsochoseJVMasitsruntimeduetoitsinherentadvantagesandmaturedecosystem.Javawasfirstintroducedin

1994-1995byJamesGosling(https://en.wikipedia.org/wiki/James_Gosling).Javaisanobject-orientedlanguagewithafocusonconcurrency.Javabecamewidelypopularinenterprisessoonafteritsreleaseduetoitsefficientdependencymanagementandabilitytowriteonce,andrunanywhere(https://en.wikipedia.org/wiki/Write_once,_run_anywhere).Javaalloweddeveloperstowritetheircodeonce,compileitintobytecode,andthenrunitonallplatformsthatsupportJVMwithoutrecompilingthecode.

ClojureinheritedallthecapabilitiesofJVMwiththefocusonimmutabilityandLISP-likeparenthesizedprefixnotation.Itisbuiltontheconceptsofimmutabilityandpersistentdatastructures,whichmakesithighlyconcurrent.ItusestheEpochalTimeModel,whichorganizesprogramsbyidentities,whereeachidentityisdefinedasaseriesofimmutablestatesovertime.Duetoitshighlyconcurrentnature,Clojureisbestsuitedtobuildingmassivelyparallelsoftwaresystemsthatarerobustandutilizemodernmultiprocessorhardwaretothefullest.

TheEpochaltimemodelwasintroducedbyRichHickeyinhiskeynotetalkonArewethereyet?(https://www.infoq.com/presentations/Are-We-There-Yet-Rich-Hickey)attheJVMLanguagesSummitin2009.HerevisiteditinhiskeynotetalkonEffectivePrograms-10YearsofClojure(https://www.youtube.com/watch?v=2V1FtfBDsLU)atClojure/Conjin2017.

REPLRead-eval-print-loopisaninteractiveprogrammingenvironmentthattakesexpressionsasinputsfromtheuser,parsesthemintoadatastructurefortheprogramminglanguage,evaluatesthem,andprintstheresultbacktotheuser.Oncedone,itreturnstothereadstateandwaitsforuserinput,thusformingaloop,asshowninthefollowingdiagram.REPLwaspioneeredbyLispandisnowavailableforotherprogramminglanguagesaswell:

REPLisalsothedefaultinteractiveprogrammingenvironmentforClojure.ItcompilesanexpressionintoJavabytecode,evaluatesit,andreturnstheresultoftheexpression.TouseClojureREPL,downloadaClojurerelease(1.8.0)fromClojureDownloads(https://repo1.maven.org/maven2/org/clojure/clojure/1.8.0/clojure-1.8.0.zip)andextractit.SinceClojurerunsonJVM,youneedtofirstmakesurethatyouhaveJava1.6+orhigherinstalledonyourmachine.TovalidateyourJavaversion,usethejavacommandwiththe-versionoption.ThisbookusesJava1.8forallexamples:

%java-version

javaversion"1.8.0_121"

Java(TM)SERuntimeEnvironment(build1.8.0_121-b13)

JavaHotSpot(TM)64-BitServerVM(build25.121-b13,mixedmode)

TouseClojureREPL,firstunziptheClojure1.8release:

%unzipclojure-1.8.0.zip

Archive:clojure-1.8.0.zip

creating:clojure-1.8.0/

inflating:clojure-1.8.0/clojure-1.8.0-slim.jar

inflating:clojure-1.8.0/clojure-1.8.0.jar

inflating:clojure-1.8.0/pom.xml

inflating:clojure-1.8.0/build.xml

inflating:clojure-1.8.0/readme.txt

inflating:clojure-1.8.0/changes.md

inflating:clojure-1.8.0/clojure.iml

inflating:clojure-1.8.0/epl-v10.html

Intheunzippedclojure-1.8.0folder,theclojure-1.8.0.jarJAR(https://en.wikipedia.org/wiki/JAR_(file_format))isanexecutableJARthatcontainstheClojurecompiler.ItisalsousedtostartClojureREPLusingthejavacommand:

%cdclojure-1.8.0

%java-jarclojure-1.8.0.jar

Clojure1.8.0

user=>(+123)

user=>

TheprecedingmethodofstartingtheREPLwiththeClojureJARfileworkswelluntilthelaststablereleaseofClojure1.8atthetimeofwritingthisbook.WithClojure1.9andonwards,itisrecommendedtousetheClojurecljtool(https://clojure.org/guides/deps_and_cli)orabuildtoolsuchasLeiningen(https://leiningen.org/)tosetupaprojectandstartREPLfortheprojectwiththerequiredlibraries.

repl.it(https://repl.it)isanonlineutilitythatallowyoutoaccessClojureREPLonlineoverthewebbrowser.

ClojurebuildtoolsBuildtoolsareofprimeimportanceforanyprogramminglanguage.Theynotonlyhelptogenerateadeployableartifactfortheapplication,butalsomanagethedependenciesoftheapplicationthroughoutitsdevelopmentlifecycle.LeiningenandBoot(http://boot-clj.com/)aretwowidelyusedbuildtoolsforClojure.SinceClojureisahostedlanguageforJVM,ClojurebuildtoolsprimarilygenerateJARsasdeployableartifactsforallClojureprojects.

LeiningenLeiningenisabuildandprojectmanagementtoolthatiswritteninClojureandiswidelyusedacrossClojureprojects.ItdescribesaClojureprojectusinggenericClojuredatastructures.ItintegrateswellwiththeMavenrepositoryfromtheJavaworldandtheClojars(https://clojars.org/)repositoryofClojurelibraries,fordependencymanagementandreleases.Leiningenhasbuilt-insupportforpluginstoextenditsfunctionality.ItalsoprovidesanoptiontodesignapplicationtemplatesthatcanhelpcreateaClojureapplicationcodelayoutwithasingleleincommand.Toseparateprojectconfigurationfromdevelopment,test,andproduction,italsohastheconceptofprofiles,whichcanbeusedtochangetheprojectconfigurationatbuildtimebyusingitwiththeleincommand.

Tosetuplein,downloadthelatestLeinscript(https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein)fromtheLeiningenGitHubrepository(https://github.com/technomancy/leiningen),makeitexecutableusingthechmodcommand,andmakesurethatthescriptisavailableonyourpath,forexample,bycopyingitto~/binor/usr/local/bininLinux.Oncetheleinscriptisavailableonyourpath,runittodownloadthelatestself-installpackageandsetuptheenvironment.LeindownloadstheexecutableJARforLeiningenfirsttime:

#Downloadthelatestscript(latestversion)

%wgethttps://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein

#Makeitexecutable

%chmoda+xlein

#Runleintosetup

DownloadingLeiningen...

LeiningenisatoolforworkingwithClojureprojects.

#Validateleinversion

%leinversion

Leiningen2.8.0onJava1.8.0_121JavaHotSpot(TM)64-BitServerVM

#Downgradetoversion2.7.1,i.e.theversionusedinthebook

%leindowngrade2.7.1

#Validateleinversion

%leinversion

Leiningen2.7.1onJava1.8.0_121JavaHotSpot(TM)64-BitServerVM

Notethattheleinscriptalwaysdownloadsthemostrecentlyreleasedversion.Onceleinissetup,youcanstartREPLdirectlyfromtheleincommand:

%leinrepl

nREPLserverstartedonport43952onhost127.0.0.1-nrepl://127.0.0.1:43952

REPL-y0.3.7,nREPL0.2.12

Clojure1.8.0

JavaHotSpot(TM)64-BitServerVM1.8.0_121-b13

Docs:(docfunction-name-here)

(find-doc"part-of-name-here")

Source:(sourcefunction-name-here)

Javadoc:(javadocjava-object-or-class-here)

Exit:Control+Dor(exit)or(quit)

Results:Storedinvars*1,*2,*3,anexceptionin*e

user=>(+123)

user=>

BootBootisanalternativebuildsystemforClojurethatisgainingpopularity.BoottreatsbuildscriptsasClojureprogramsthatcanbeextendedusingpods(https://github.com/boot-clj/boot/wiki/Pods).PodsalsohelptoisolateclasspathsforbetterdependencymanagementandusesmultipleClojureruntimes.InsteadoftheprofilesandpluginsofLeiningen,Bootusesthecommonconceptoftasks(https://github.com/boot-clj/boot/wiki/Tasks).TasksareusedacrosstheBootprogramtomodifythebuildenvironmentasperthecontext.

TosetupBoot,downloadthelatestboot.shscript(https://github.com/boot-clj/boot-bin/releases/download/latest/boot.sh)fromtheBootGitHubrepository(https://github.com/boot-clj/boot),makeitexecutableusingthechmodcommand,andmakesurethatthescriptisavailableonyourpath,forexample,bycopyingitto~/binor/usr/local/bininLinux.OncetheBootscriptisavailableonyourpath,runittosetuptheenvironment,asshownhere:

#Downloadbootscript

%curl-fsSLoboothttps://github.com/boot-clj/boot-

bin/releases/download/latest/boot.sh

#Makeitexecutable

%chmod755boot

#Runboottosetup

Downloadinghttps://github.com/boot-clj/boot/releases/download/2.7.2/boot.jar...

Runningforthefirsttime,BOOT_VERSIONnotset:updatingtolatest.

Retrievingmaven-metadata.xmlfromhttps://repo.clojars.org/(3k)

Retrievingboot-2.7.2.pomfromhttps://repo.clojars.org/(2k)

Retrievingboot-2.7.2.jarfromhttps://repo.clojars.org/(3k)

#http://boot-clj.com

#SatOct2119:43:24IST2017

BOOT_CLOJURE_NAME=org.clojure/clojure

BOOT_VERSION=2.7.2

BOOT_CLOJURE_VERSION=1.8.0

OnceBootissetup,youcanstartREPLdirectlyfromthebootcommand:

%bootrepl

nREPLserverstartedonport33140onhost127.0.0.1-nrepl://127.0.0.1:33140

REPL-y0.3.7,nREPL0.2.12

Clojure1.8.0

JavaHotSpot(TM)64-BitServerVM1.8.0_121-b13

Exit:Control+Dor(exit)or(quit)

Commands:(user/help)

Docs:(docfunction-name-here)

(find-doc"part-of-name-here")

FindbyName:(find-name"part-of-name-here")

Source:(sourcefunction-name-here)

Javadoc:(javadocjava-object-or-class-here)

Examplesfromclojuredocs.org:[clojuredocsorcdoc]

(user/clojuredocsname-here)

(user/clojuredocs"ns-here""name-here")

boot.user=>(+123)

boot.user=>

ThisbookwillfocusonLeiningenasthebuildtoolfortheClojureprojects,butitisimportanttoknowthatLeiningenisnottheonlybuildtoolavailableforClojure.BootisalsoanoptionthatcanbeusedinsteadofLeiningen.

ClojureprojectAClojureprojectisadirectorythatcontainssourcefiles,testfiles,resources,documentation,andprojectmetadata.SourcefilesareprimarilyfromClojure,butaprojectmaycontainJavasourcefilesaswell.LeiningenhasadefaultprojecttemplatethatcanbeusedtoquicklycreateaClojureprojectstructureusingtheleinnew<project-name>command:

#Createanewproject'playground'

%leinnewplayground

Generatingaprojectcalledplaygroundbasedonthe'default'template.

Thedefaulttemplateisintendedforlibraryprojects,notapplications.

Toseeothertemplates(app,plugin,etc),try`leinhelpnew`.

#Showthe'playground'projectdirectorystructure

%treeplayground

playground

├──CHANGELOG.md

├──doc

│└──intro.md

├──LICENSE

├──project.clj

├──README.md

├──resources

├──src

│└──playground

│└──core.clj

└──test

└──playground

└──core_test.clj

6directories,7files

EachClojureprojectcontainsaprojectmetadatafile,project.clj,whichdefinesalltheprojectdependencies,profiles,andpluginsthatarerequiredfortheproject.Forexample,theproject.cljfileoftheplaygroundprojectliststheprojectmetadataanddependenciesbasedonthedefaultprojecttemplateofLeiningen,asshownhere:

(defprojectplayground"0.1.0-SNAPSHOT"

:description"FIXME:writedescription"

:url"http://example.com/FIXME"

:license{:name"EclipsePublicLicense"

:url"http://www.eclipse.org/legal/epl-v10.html"}

:dependencies[[org.clojure/clojure"1.8.0"]])

ThedefprojectisaClojuremacrothatisdefinedbyLeiningenandactsasacontainerforallprojectmetadatadirectives.Thedefaulttemplateomitsthe

defaultconfigurationdirectives.

ConfiguringaprojectAgoodprojectconfigurationmustdefineseparateprofilesfordevelopment,test,andproduction.Itshouldalsohavedirectivestotesttheimplementation,checkcodequality,andgeneratetestreportsanddocumentationfortheapplication.Eachprojectconfigurationshouldalsohavetheentrypointintotheapplicationdefinedusingthe:maindirective,whichpointstothenamespacethatcontainsthemainfunction,asshownhere:

:description"PlaygroundProject"

:url"http://example.com/playground"

:dependencies[[org.clojure/clojure"1.8.0"]]

:mainplayground.core

:profiles{:provided{:dependencies[[org.clojure/tools.reader"0.10.0"]

[org.clojure/tools.nrepl"0.2.12"]]}

:uberjar{:aot:all:omit-sourcetrue}

:doc{:dependencies[[codox-theme-rdash"0.1.1"]]

:codox{:metadata{:doc/format:markdown}

:themes[:rdash]}}

:dev{:resource-paths["resources""conf"]

:jvm-opts["-Dconf=conf/conf.edn"]}

:debug{:jvm-opts

["-server"

(str"-agentlib:jdwp=transport=dt_socket,"

"server=y,address=8000,suspend=n")]}})

Projectconfigurationshouldexplicitlylistsourceandtestfilelocationsusingthe:source-pathsand:test-pathsdirective.IfbothClojureandJavasourcecodefilesarepresentintheproject,thenitisrecommendedtoorganizethesourcefileundersrc/cljforClojureandsrc/jvmforJava.Similarly,testfilescanbekeptundertest/cljandtest/jvmforClojureandJava,respectively.Areferenceproject.cljfilewiththerequiredprojectconfigurationisshowninthefollowingcodesnippet:

:mainplayground.core

:source-paths["src/clj"]

:java-source-paths["src/jvm"]

:test-paths["test/clj""test/jvm"]

:resource-paths["resources""conf"]

Projectresourcefiles,suchasstartup,shutdownscripts,andmore,canbekeptintheresourcesdirectoryandtheprojectconfigurationfilescanbekeptintheconfdirectory.Boththedirectoriesmustbespecifiedunderthe:resource-pathsdirective,asshownintheprecedingsnippet.Togeneratedocumentationandtestreports,andcheckcodequality,thereareanumberofthird-partypluginsavailablethatcanbeaddedtotheprojectconfiguration.Forexample,Codox(https://github.com/weavejester/codox)canbeusedtogenerateAPIdocumentation,Cloverage(https://github.com/cloverage/cloverage)canbeusedtotestcodecoverage,andtest2junit(https://github.com/ruedigergad/test2junit)canbeusedtogeneratetestreports.Addthesepluginsintheconfigurationfile,asshownhere:

:resource-paths["resources""conf"]

:plugins[[:lein-codox"0.10.3"]

;;CodeCoverage

[:lein-cloverage"1.0.9"]

;;Unittestdocs

[test2junit"1.2.2"]]

:codox{:namespaces:all}

:test2junit-output-dir"target/test-reports"

:uberjar{:aot:all:omit-sourcetrue}

:themes[:rdash]}}

:dev{:resource-paths["resources""conf"]

:jvm-opts["-Dconf=conf/conf.edn"]}

:debug{:jvm-opts

["-server"

(str"-agentlib:jdwp=transport=dt_socket,"

"server=y,address=8000,suspend=n")]}})

SomeofthesepluginsmayalsorequireextraLeiningendirectivestobedefinedforthem.Dependenciesthatarecommonacrossprofilesmustbelistedunderthe:dependenciesdirectiveofthedefprojectmacro,andtherestofthedependenciesmustbelistedundertherespectiveprofiles,asshownintheprecedingconfiguration.

TheLeiningenGitHubrepositoryhasasample.project.cljfile(https://github.com/technomancy/leiningen/blob/master/sample.project.clj)thatlistsallsupportedprojectdirectivesforaClojureprojectthatismanagedbyLeiningen.ThisfileactsasdetaileddocumentationforallthefeaturesofLeiningen.

Runningaproject

Ifthe:maindirectiveisdefinedintheproject.cljfileoftheproject,thentheprojectcanberunbydirectlycallingtheleinruncommand.leinrunexpectsthenamespacespecifiedunder:maindirectivetocontaintheClojuremainfunction.Forexample,theplayground.corenamespacemusthaveamainfunctiondefinedforleinruntowork,asshowninthefollowingimplementation.Sincethe:source-pathsparameterintheupdatedconfigurationofproject.cljpointstosrc/cljinsteadofthedefaultsrc/,thecore.cljsourcefile,asshowninthefollowingcode,mustresidewithinthesrc/clj/playground/directoryasperthenamespaceandconfiguredsourcepaths:

(nsplayground.core

(:gen-class))

(defnfoo

"Idon'tdoawholelot."

(printlnx"|Hello,World!"))

(defn-main

[&args]

(foo(or(firstargs)"Noname")))

Ifthemainfunctionisdefinedintheplayground.corenamespace,thenitcanbeexecutedusingleinrun:

%leinrun

Noname|Hello,World!

%leinrunClojure

Runningtests

Torunallthetestsdefinedunderthe:test-pathsdirective,runtheleintestcommand.SincethedefaulttemplateofLeiningenhasonlyonetestdefinedanditalwaysfails,therewillbeatestfailurereported,asshowninthefollowingcodesnippet.Sincethe:test-pathsparameterintheupdatedconfigurationofproject.cljpointstotest/cljinsteadofthedefaulttest/,thegeneratedcore_test.cljtestsourcefilemustresidewithinthetest/clj/playground/directoryasperthenamespaceandconfiguredtestpathsforteststorun,asshownhere:

%leintest

leintestplayground.core-test

leintest:onlyplayground.core-test/a-test

FAILin(a-test)(core_test.clj:7)

FIXME,Ifail.

expected:(=01)

actual:(not(=01))

Ran1testscontaining1assertions.

1failures,0errors.

Testsfailed.

Generatingreports

AgoodprojectdeliverablemustincludetestandcoveragereportswithassociatedAPIdocumentation.Basedontheconfiguredpluginsunderthepluginsdirectiveofproject.clj,leincanbeusedtogeneratevariousreportsaspertheplugindocumentation.Theexecutionofeachofthepluginsusingtheleincommandandtheircorrespondingresultsareshownhere:

%leintest2junit

Usingtest2junitversion:1.2.2

RunningTests...

Writingoutputto:target/test-reports

Creatingdefaultbuild.xmlfile.

Testing:playground.core-test

>1failures,0errors.

Testsfailed.

%leinwith-profiles+testcloverage

Loadingnamespaces:(playground.core)

Testnamespaces:(playground.core-test)

Loadedplayground.core.

Instrumentednamespaces.

Testingplayground.core-test

FAILin(a-test)(core_test.clj:7)

FIXME,Ifail.

expected:(=01)

actual:(not(=01))

1failures,0errors.

Rantests.

Producedoutputinplayground/target/coverage.

HTML:target/coverage/index.html

|-----------------+---------+---------|

|Namespace|%Forms|%Lines|

|-----------------+---------+---------|

|playground.core|17.65|60.00|

|-----------------+---------+---------|

|ALLFILES|17.65|60.00|

|-----------------+---------+---------|

Errorencounteredperformingtask'cloverage'withprofile(s):

'base,system,user,provided,dev,test'

Suppressedexit

%leinwith-profiles+doccodox

GeneratedHTMLdocsinplayground/target/doc

Asaresultoftheexecutionofthecommandsshownintheprecedingsnippet,theoutputoftest2junitisgeneratedunderthetarget/test-reports/xml/playground.core-test.xmlfile,whichisastandardJUnit(https://en.wikipedia.org/wiki/JUnit)testreport.Coveragereportsaregeneratedundertarget/coverage/,whichcontainsanindex.htmlfilethatcanbeviewedusingawebbrowsertoseeadetailedreportandreviewtheinstrumentedcodepathsthatwerecoveredwiththetestcases.Thedocumentationfortheprojectisgeneratedunderthetarget/docfile,whichcontainstheindex.htmlfilethatcanbeviewedinawebbrowsertoreviewthegenerateddocs.

Generatingartifacts

ClojureartifactsaregeneratedasrunnableJARs.TogeneratearunnableJAR,runtheleinuberjarcommand,asshowninthefollowingsnippet.Theartifactcanbeconfiguredusingtheuberjarprofileinproject.clj:

%leinuberjar

Compilingplayground.core

Createdtarget/playground-0.1.0-SNAPSHOT.jar

Createdtarget/playground-0.1.0-SNAPSHOT-standalone.jar

%java-jartarget/playground-0.1.0-SNAPSHOT-standalone.jar

Noname|Hello,World!

%java-jartarget/playground-0.1.0-SNAPSHOT-standalone.jarClojure

ClojureIDE

Anintegrateddevelopmentenvironment(IDE)isasoftwareapplicationthatprovidesutilitiesforprogrammerstodevelop,build,anddebugsoftwareapplications.Itconsistsofacodeeditor,buildautomationtool,andadebugger.ClojurehasagrowingnumberofIDEsavailable,outofwhichEmacs(https://en.wikipedia.org/wiki/Emacs)andVim(https://en.wikipedia.org/wiki/Vim_(text_editor))standout.AlthoughthisbookdoesnotcoverIDEs,whereveroneisreferredto,usesEmacsastheIDE.BothEmacsandVimaretexteditorsthatneedadditionalpluginstosupportClojure.EmacsneedsCIDER(https://github.com/clojure-emacs/cider),whereasVimgetsREPLsupportusingFireplace(https://github.com/tpope/vim-fireplace).

SomeotherClojureIDEsthatarewidelyusedare:

Eclipse(https://www.eclipse.org/home/index.php)withCounterclockwise(http://doc.ccw-ide.org/)LightTable(http://lighttable.com/)SublimeText(https://www.sublimetext.com/)withSublimeREPL(https://github.com/wuub/SublimeREPL)Atom(https://atom.io/)withnrepl(https://atom.io/packages/nrepl)Cursive(https://cursive-ide.com/)

Summary

Inthischapter,wefocusedonthedevelopmentenvironmentfortheHelpingHandsapplication.SinceClojureisourlanguageofchoiceforimplementation,wefirstlookedatthehistoryofClojureandLispandunderstoodwhyitiswellsuitedforourusecase.WealsolookedattheREPLenvironmentandtwobuildtoolsforClojure—LeiningenandBoot.Further,wedefinedareferenceLeiningenprojectconfigurationforourapplicationandlearnedhowtorunanapplicationandtestit.Wealsolearnedhowtogeneratedocumentationandreports,andhowtocreateadeployableartifact.Attheend,webrieflylookedattheClojureIDEsthatcanmakeourapplicationdevelopmentworkeasy.

Inthenextchapter,wewilllearnaboutRESTspecification.WewilllearnhowtodefineRESTAPIsformicroservicesintheHelpingHandsapplicationthatcanhelpwithdirectmessagingamongtheservices.

RESTAPIsforMicroservices

"Comingtogetherisabeginning;keepingtogetherisprogress;workingtogetherissuccess."

-HenryFord

OneofthemodesofinteractionamongmicroservicesisdirectmessagingviaAPIs.RESTAPIsareoneofthewaystomakethedirectmessagingpossibleamongmicroservices.RESTAPIsenableinteroperabilityamongmicroservicesirrespectiveofthetechnologystackinwhichtheyareimplemented.Inthischapter,wewillcoverthefollowingtopics:

TheconceptofRESTHowtodefineRESTURIswithappropriatemethodswithstatuscodesHowtouseREST-basedHTTPURIsusingutilitiessuchascURLRESTAPIsfortheHelpingHandsapplication

IntroducingRESTRESTstandsforrepresentationalstatetransfer,anddefinesanarchitecturalstylefordistributedhypermediasystems.Itfocusesoncreatingindependentcomponentimplementationsthatarestatelessandscalewell.ApplicationsthatsupportRESTdonotstoreanyinformationabouttheclientstateontheserverside.SuchapplicationsrequireclientstomaintainthestatethemselvesandusetheREST-styleimplementations,suchasHTTPAPIsexposedbytheapplication,totransferthestatebetweenthemandtheserver.ClientswhousetheRESTAPImayquerytheserverforthelateststateandrepresentthesamestateattheclientside,thuskeepingtheclientandserverinsync.

Thestateofanapplicationisdefinedbythestateoftheentitiesofthesystem.EntitiescanberelatedtotheconceptofresourcesinRESTarchitecture.AresourceisakeyabstractionofinformationinREST.Itisaconceptualmappingtoasetofentitiesthatisdefinedbytheresourceidentifier.AnyinformationthatcanbenamedorbethetargetoftheRESTresourceidentifiercanbearesource.Forexample,aconsumerisaresourceforaServiceConsumerserviceandanorderisaresourceforanOrderservice.

ResourceidentifiersinRESTaredefinedasAPIendpointsthataremappedtoHTTPURIs.TheURIspointtooneormoreresourcesandallowthecallersto

performoperationsontheresources,asshownintheprecedingimage.AlltheoperationssupportedbyRESTarestateless;thatis,noneoftheclientstateispersistedontheserverside.ThestatelessnatureoftheRESTAPIshelpsincreatingservicesthatscalehorizontallywithoutdependingonsharedstate.

RESTstyleissuitedtomicroservicesfordirectcommunicationandbuildingsynchronousAPIstogetaccesstotheentities.EachentitythatismanagedbyamicroservicecanbedirectlymappedtotheconceptofaresourceinREST.

TheconceptofRESTwasdefinedbyRoyFieldingin2000asapartofhisPhDdissertationonArchitecturalStylesandtheDesignofNetwork-basedSoftwareArchitectures(https://www.ics.uci.edu/~fielding/pubs/dissertation/fielding_dissertation_2up.pdf).ARESTfulsystemconformstosixguidingprinciples(https://en.wikipedia.org/wiki/Representational_state_transfer#Architectural_constraints)ofclient-server,statelessness,cacheability,layeredsystem,code-on-demand,anduniforminterface.

RESTfulAPIs

WebserviceAPIsthatconformtoRESTarchitectureiscalledRESTfulAPIs.MicroservicesmostlyimplementtheHTTP-basedRESTfulAPIsthatarestatelessandhaveabaseURIandamediatype(https://en.wikipedia.org/wiki/Media_type)fortherepresentationofresources.ItalsosupportspredefinedstandardoperationsthataremappedtoHTTPmethodssuchasGET,POST,PUT,DELETE,andmore.

Forexample,asshowninthefollowingtable,theOrderservicemaydefineanAPI/orderstogetaccesstotheordersthatitmaintains.ItcansupportaGETmethodtolookupalltheordersorgetaspecificorderbyspecifyingtheorderID.ItcanalsoallowclientstocreatenewordersbyusingthePOSTmethodorcreateanorderwithaspecificIDbyusingthePUTmethod.Similarly,itcansupportthePUTmethodtoupdateorderdetailsandtheDELETEmethodtodeleteanorderbyspecifyingtheorderIDexplicitly.

URI HTTPmethod Operation Description

GEThttps://server/orders GET Read Getsalltheorders

https://server/orders/1GET Read Getsthedetailsoforder

withIDas1

https://server/ordersPOST Create Createsaneworderand

returnstheID

https://server/orders/100PUT Create Createsaneworderwith

IDas100

https://server/orders/2PUT Update Updatesanexistingorder

withIDas2

DELETE

https://server/orders/1DELETE Delete Deletesanexistingorder

withIDas1

GET,PUT,POST,andDELETEarethemostwidelyusedHTTPmethodsforRESTfulAPIs.OthermethodsincludeHEAD,OPTIONS,PATCH,TRACE,andCONNECT(https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Request_methods).EachrequestthatissenttotheURIsgeneratesaresponsethatmaybeHTML,JSON,XML,oranydefinedformat.JSONisthepreferredformatformicroservicesthathandlethecoreoperationsoftheapplication.Forexample,theresponsesgeneratedbytheRESTfulAPIsoftheOrderservicewillcontainaJSONobjectoforderdetailsoraJSONarrayoforders,whereeachorderisrepresentedasaJSONobjectwithkey-valuepairscontainingtheorderdetails.

Statuscodes

Statuscodesareimportantforclientstounderstandtheoutcomeoftherequestandtakerequiredactionontheresponse.Successfulrequestsreturnaresponsewiththerequiredrepresentationofresources,butfailedrequestsgeneratearesponserepresentingtheerrorinstead.Statuscodeshelpclientstobewellawareofthecontentoftheresponseandtakenecessaryactionattheclientside.

AllmicroserviceswithRESTfulAPIsmustsupportappropriateHTTPstatuscodes(https://en.wikipedia.org/wiki/List_of_HTTP_status_codes)andsendthemtotheclientbasedontheoutcomeoftherequestedoperation.SomeofthestatuscodesthatmustbeimplementedbyallAPIsofmicroservicesare:

Statuscode Usage

Standardresponsecodeiftherequestwassuccessful.AGETrequestmustreturntheresourcedetailsintheresponse,andaPOSTrequestmustreturntheoutcomeoftheresponse.

201CreatedSentwhentherequestresultsinthecreationofaresource.TheresponsemustcontaintheURIoftheresource.

400Bad

Request

Client-sideerrorresponsecodeiftherequestcontainsinvalidorinsufficientparameters.Theresponsemustincludedetailsoftheissueswithrespecttorequestparametersforclienttocorrect.

Unauthorized

Sentwhenauthenticationisrequiredtoaccessaresourceandeitherclientrequestisnotauthenticatedordoesnotcontainauthentication/authorizationtokensasexpectedbytheserver.

Forbidden

Sentwhentheclientrequestisvalidbutitisnotallowedtoaccesstheresource,possiblyduetoinsufficientprivileges.

404Not

Found Sentwhentherequestedresourceisnotfoundontheserver.

405Method

NotAllowedSentwhentherequestedmethodisnotsupportedbytheresource.Forexample,aresourcemaysupportonlytheGETmethod.Inthatcase,sendingaPUTrequestmayresultinaresponsewiththiserrorcode.

500Internal

ServerError

Genericerrorresponsecodesentbytheserverwhentherequestfailswithanunknownerrorthatisnothandledbytheserverandthereisnootherappropriateerrormessagetosend.

TheprecedingstatuscodesmustbeimplementedfortheRESTAPIs.ThereareotherstatuscodesaswellthatmaybeusedbytheAPIs.Toreviewallthedefinedstatuscodes,takealookattheHTTPstatuscodesguide.

Namingconventions

RESTAPIsmustbeorganizedaroundresourcesofthesystemandmustbeeasytounderstandbyjustlookingattheURIs.URIsmustfocusononeresourcetypeatatimewithoperationsmappedtoHTTPrequestmethods.TheymayhaveoneormorerelatedresourcesthatcanbefurthernestedintheURI.

GoodexamplesofURIsare/users,/consumers,/orders,and/services,not/getusers,/showorders,andmore.AbasicruletodefineAPIURIsistotargetnounsasresourcesandverbsasHTTPmethods.Forexample,insteadofcreatingaURIas/getusers,itisrecommendedtohavetheURIas/userswiththeHTTPmethodasGET.Similarly,insteadof/showorders,theURIGET/ordersmustbecreatedtoprovidealltheordersintheresponsetotheclient.GuidelinesfornamingURIsarefurtherexplainedinthefollowingdiagram:

ItisrecommendedtoalwaysuseSSL(https://en.wikipedia.org/wiki/Transport_Layer_Security)byonlyusingHTTPS-basedURIsforalltheAPIs.TheAPIsmustbeversionedtoupgradethemovertimewithoutbreakingtheintegrationwithexistingservices.TospecifytheversionoftheAPI,amajorversioncanbeusedasaprefixtoalltheAPIendpoints.Sub-versionsmustnotbeaddedtotheURI.Ifrequired,theycanbespecifiedusingacustomheaderparameter.

Versionscanalsobeintroducedviaamediatype.Itisperfectlyfine

toincludesub-versionsaswellwiththemediatype.Versionsmayincludebreakingchangesaswellfortheclients,butitisrecommendedtokeepthechangesasbackwardcompatibleaspossible.

URIsmayalsocontainapplicationqualifiersthatsegmenttheAPIendpointsbasedonthetargetcomponentsorbusinessoperations.EachURImustbefocusedonasingleresourcetype,andthatnamemustbeusedinitspluralform.Toretrievespecificresourcesofthespecifiedtype,anIDmustbeappendedtotheURI.HerearesomeoftheexamplesofURIsthatfollowtheeasy-to-understandnamingconvention:

URI DescriptionGEThttps://server/v1/orders GetsalltheordersGEThttps://server:9988/v1/orders GetsalltheordersGEThttps://server/v1/tech/orders GetsalltheordersGEThttps://server/v1/orders/7 GetsanorderwithID7POSThttps://server/v1/orders CreatesaneworderandreturnstheID

PUThttps://server/v1/orders/17UpdatestheorderID17,createsnewifnotpresent

https://server/v1/orders/7/feedbacks GetsallfeedbacksfororderID7GET

https://server/v1/orders/7/feedbacks/1 GetsthefeedbackwithID1fororderID7DELETEhttps://server/v1/orders/3 DeletestheorderwithID3PUThttps://server/v1/orders/7/rate UpdatestheratingfororderID7

GEThttps://server/v1/orders?

fields=ts,id,name

Getsalltheorderswithonlyts,idandnamefieldoftheordersintheresponse

GEThttps://server/v1/orders?

status=open&sort=-ts,name

Getsalltheordersthathavestatusasopen,sortedbytsindescendingorderandnameinascendingorder

UsingRESTfulAPIsviacURLcURLisacommand-lineutilitytotransferdatausingvariousprotocols,includingHTTPandHTTPS.cURLiswidelyusedtotryRESTfulAPIsusingitscommand-lineinterface.Forexample,let'stakealookattheAPIsprovidedbyMetaWeather(https://www.metaweather.com/api/)toqueryalocationandgetitsweatherinformation.SinceitisapublicAPIthatmaintainstheweatherinformationofwell-knownplaces,itallowsonlyGETrequeststogetthedetailsoftheresourcesanddoesnotallowcreatingorupdatingresources.

Togettheweatherofalocation,theMetaWeatherapplicationrequirestheresourceIDasdefinedbythesystem.TogettheresourceID,MetaWeatherprovidesasearchAPItolookuptheresourcesbyasearchquery.ThesearchAPIusestheGETmethodwiththequeryastheURLparameterintherequestandreturnstheJSONresponsewithresourcesthatmatchthequery,asshownhere:%curl-XGET"https://www.metaweather.com/api/location/search/?query=bangalore"[{"title":"Bangalore","location_type":"City","woeid":2295420,"latt_long":"12.955800,77.620979"}]

ThewoeidfieldintheresponseistheresourceIDrequiredbytheweatherAPIthatcanbeusedtogettheweatherdetailsofthequeriedlocation:

%curl-XGET"https://www.metaweather.com/api/location/2295420/"

"title":"Bangalore",

"location_type":"City",

"woeid":2295420,

"latt_long":"12.955800,77.620979",

"timezone":"Asia/Kolkata",

"time":"2017-10-24T21:06:34.146240+05:30",

"sun_rise":"2017-10-24T06:11:13.036505+05:30",

"sun_set":"2017-10-24T17:56:02.483163+05:30",

"timezone_name":"LMT",

"parent":{

"title":"India",

"location_type":"Country",

"woeid":23424848,

"latt_long":"21.786600,82.794762"

"consolidated_weather":[

"id":6004071454474240,

"weather_state_name":"HeavyCloud",

"weather_state_abbr":"hc",

"wind_direction_compass":"NE",

"created":"2017-10-24T15:10:08.268840Z",

"applicable_date":"2017-10-24",

"min_temp":18.458000000000002,

"max_temp":29.392000000000003,

"the_temp":30.0,

"wind_speed":1.9876466767411649,

"wind_direction":51.745585344396069,

"air_pressure":969.60500000000002,

"humidity":63,

"visibility":11.150816730295077,

"predictability":71

"sources":[

"title":"BBC",

"slug":"bbc",

"url":"http://www.bbc.co.uk/weather/",

"crawl_rate":180

TheMetaWeatherapplicationAPIsalsosupporttheOPTIONSmethodthatcanbeusedtofindoutthedetailsoftheAPI.Forexample,sendingtherequesttothesamesearchAPIwiththeOPTIONSHTTPmethodprovidestherequireddetailsintheresponse:

%curl-XOPTIONS"https://www.metaweather.com/api/location/search/?query=bangalore"

{"name":"LocationSearch","description":"","renders":["application/json"],"parses":

["application/json","application/x-www-form-urlencoded","multipart/form-data"]}

TheMetaWeatherapplicationalsorespondswiththeappropriatestatuscodeandmessageintheresponseforaninvalidresourceID:

%curl-v-XGET"https://www.metaweather.com/api/location/0/"

*Trying172.217.26.179...

*Connectedtowww.metaweather.com(172.217.26.179)port443(#0)

*compression:NULL

*ALPN,serveracceptedtousehttp/1.1

>GET/api/location/0/HTTP/1.1

>Host:www.metaweather.com

>User-Agent:curl/7.47.0

>Accept:*/*

<HTTP/1.1404NotFound

<x-xss-protection:1;mode=block

<Content-Language:en

<x-content-type-options:nosniff

<strict-transport-security:max-age=2592000;includeSubDomains

<Vary:Accept-Language,Cookie

<Allow:GET,HEAD,OPTIONS

<x-frame-options:DENY

<Content-Type:application/json

<X-Cloud-Trace-Context:507eec2981a7028e50e596fcb651acb7;o=1

<Date:Tue,24Oct201716:07:36GMT

<Server:GoogleFrontend

<Content-Length:23

*Connection#0tohostwww.metaweather.comleftintact

{"detail":"Notfound."}

RESTAPIsforHelpingHandsTheHelpingHandsapplicationhasConsumerandProviderservicesthatexposetheRESTAPIstomanageconsumersandprovidersfortheapplication.EachserviceproviderintheapplicationcanregisteroneormoreservicesthataremanagedbyServiceAPIs.Apartfromtheseservices,thereisanOrderservicethatmanagesalltheordersplacedbytheconsumersfortheserviceandservedbytheprovidersoftheservice.TheHelpingHandsapplicationalsoprovidesanapplication-wideLookupservicethatprovidesasingleAPItolookupservicesandordersbyvicinity.AllAPIsprovidedbytheHelpingHandsmicroservicesalsohandleerrorswithappropriateerrormessagesintheresponseandthecorrespondingstatuscode.

ConsumerandProviderAPIs

TheAPIsprovidedbytheConsumerandProviderservicestargettheconsumersandprovidersresourcesofthesystem,respectively:

URI Method Params Description

/consumers POST Details(JSON) CreatesanewconsumerandreturnstheconsumerID

/consumers/1 PUTDetailstobeupdated(JSON)

Updatesthedetailsofanexistingconsumer

/consumers GETFields(CSV),sort(CSV),page

Getsalltheconsumersbasedonrequestparams

/consumers/1 DELETE - Deletesthespecifiedconsumer

/providers POST Details(JSON) CreatesanewproviderandreturnstheproviderID

/providers/1 PUTDetailstobeupdated(JSON)

Updatesthedetailsofanexistingprovider

/providers/1/star PUT - Incrementsthestarsfortheproviderbyone

/providers GETFields(CSV),sort(CSV),page

Getsalltheprovidersbasedonrequestparams

/providers/1 DELETE - Deletesthespecifiedprovider

ServiceandOrderAPIs

TheAPIsprovidedbytheServiceandOrderservicestargettheproviderservicesandordersresourcesofthesystem,respectively:

URI Method Params Description

/services POST Details(JSON) CreatesanewserviceandreturnstheserviceID

/services/1 PUTDetailstobeupdated(JSON)

Updatesthedetailsofanexistingservice

/services/1/star PUT - Incrementsthestarsfortheservicebyone

/services GETFields(CSV),sort(CSV),page

Getsalltheservicesbasedonrequestparams

/services/1 DELETE - Deletesthespecifiedservice

/orders POST Details(JSON) CreatesaneworderandreturnstheorderID

/orders/1 PUTDetailstobeupdated(JSON)

Updatesthedetailsofanexistingorder

/orders GETFields(CSV),sort(CSV),page

Getsalltheordersbasedonrequestparams

/orders/1 DELETE - Deletesthespecifiedorder

APIsforauthenticationandauthorizationhavebeenintentionallyleftoutfromthediscussion,butthesewillbeaddressedinPart-4ofthisbook.

Summary

Inthischapter,welearnedabouttheconceptofRESTandhowtodesignRESTfulAPIswithaneasy-to-understandnamingconvention.Wealsolearnedaboutvariousrequestmethodsandstatuscodes,andwhentousewhat.Attheend,werevisitedtheHelpingHandsapplicationtolistdowntheRESTAPIsthatwillberequiredfortheapplicationacrossmicroservices.

Inthenextchapter,wewilltakealookataClojureframeworkcalledPedestal(http://pedestal.io/)thatwewillbeusingtodesignmicroservicesfortheHelpingHandsapplication.PedestalwillalsobeusedtoexposeRESTAPIsforvariousmicroservicesoperations.

IntroductiontoPedestal

"Aconceptualframeworkisa'framethatworks'toputthoseconceptsintopractice."

-PaulHughes

Microservicesareimplementedusingaparticulartechnologystack,whichservesasingle-boundedcontextandprovidesservicesaroundit.Theservicesexposedfortheexternalworldmustbescalableandsupportdirectmessagingaswellasasynchronousrequests.Pedestal(http://pedestal.io/)isonesuchClojureframeworkthatfitsinwelltocreatereliableandscalableservicesformicroservice-basedapplications.ItalsofitsinwellwiththeClojurestackoftheHelpingHandsapplication.Inthischapter,youwill:

LearnaboutthebasicconceptsofPedestalLearnhowtodefinePedestalroutesandinterceptorsLearnhowtohandleerrorswithPedestalinterceptorsLearnhowtopublishoperationalmetricswithPedestalLearnhowtouseServer-sentEventsandWebSocketswithPedestal

PedestalconceptsPedestalisanAPI-firstClojureframeworkthatprovidesasetoflibrariestobuildreliableandhighlyconcurrentservicesthataredynamicinnature.Itisanextensibleframeworkthatisdata-drivenandimplementedusingprotocols(https://clojure.org/reference/protocols)toreducethecouplingbetweenitscomponents.Itfavorsdataoverfunctionsandfunctionsovermacros.Itallowsforthecreationofdata-drivenroutesandhandlersthatcanapplyadifferentbehavioratruntimebasedonincomingrequests.Thismakesitpossibletocreatehighlyflexibleanddynamicservicesthatarewellsuitedformicroservice-basedapplications.Italsosupportsthebuildingofscalableasynchronousservicesusingserver-sentevents(SSE)andWebSockets:

ThePedestalarchitectureisbasedontwomainconcepts,InterceptorsandContextMap,andtwosecondaryconcepts,ChainProvidersandNetworkConnectors.AllthecorelogicofthePedestalframeworkhasbeenimplementedasInterceptors,buttheHTTPconnectionhandlerhasbeenseparatedtocreateaninterfaceforChainProviderthatsetsuptheinitialContextMapandqueueofInterceptorstostarttheexecution.PedestalincludesaservletchainprovideroutoftheboxthatworkswithalltheHTTPserversthatworkwithservlets.ItalsosetsuptherequiredkeysintheContextMapthatareexpectedbytheInterceptorsasperthecontract.PedestalapplicationsarenotlimitedtojustHTTP.Customchainproviderscanbewrittentosupportotherapplicationprotocolsaswell.Theycanalsoworkwithdifferenttransportprotocols,suchasreliableUDPbasedonthetargetchainproviderandtheunderlyingnetworkconnector.

Pedestalinitiallyhadtwoseparateparts—thePedestalapplication

andPedestalserver.PedestalapplicationwasaClojureScript-basedfrontendframeworkthathasbeendiscontinued.NowthefocusisonlyonPedestalservertobuildreliableservicesandAPIs.

InterceptorsPedestalinterceptorsarebasedonasoftwaredesignpatterncalledinterceptor.Interceptorisaserviceextensionthatregisterseventsofinterestwiththeframeworkandisinvokedbytheframeworkwhenthoseeventsoccurwithinthecontrolflow.Oncetheinterceptorisinvoked,itexecutesitsfunctionality,andthenthecontrolflowreturnstotheframework.MostofPedestalcorelogicismadeupofoneormoreinterceptorsthatcanbecomposedtogethertobuildachainofinterceptors.

InterceptorinPedestalisdefinedbyaClojureMap(https://clojure.org/reference/data_structures#Maps)thatcontains:name,:enter,:leave,and:errorkeysasshowninthefollowingdiagram.The:namekeycontainsanamespacedkeyword(https://clojure.org/reference/namespaces)fortheinterceptorandisoptional.The:enterand:leavekeysspecifyaClojurefunctionthattakesaContextMapasinputandreturnsaContextMapasoutput.Eitherthe:enteror:leavekeymustbedefinedfortheinterceptortoberegisteredwiththePedestalframework.Thefunctionspecifiedbythe:enterkeyiscalledbythePedestalframeworkwhenthedataflowsintotheinterceptor.Thefunctionspecifiedby:leaveiscalledwhentheresponseisbeingreturnedbytheinterceptor:

Thefunctionspecifiedby:erroriscalledwhenanexceptioneventistriggeredbytheinterceptorexecutionoriftheexecutionfailswithanerror.Tohandletheexceptionevent,the:errorkeyspecifiesaClojurefunctionthattakestwoarguments—aContextMapandanex-info(https://clojuredocs.org/clojure.core/ex-info)exceptionthatwasthrownbytheinterceptor.IteitherreturnsaContextMap,

optionally,withtheexceptionreattachedforotherinterceptorstohandle,orthrowsanexceptionforthePedestalframeworktohandle.

TheinterceptorchainInterceptorsinPedestalcanbecomposedasaninterceptorchainthatfollowstheChainofResponsibility(https://en.wikipedia.org/wiki/Chain-of-responsibility_pattern)designpattern.Eachinterceptordoesexactlyonejobandwhencomposedtogetherasaninterceptorchain,theyachieveabiggertaskconsistingofoneormorejobs.

TheChainofResponsibilitypatternhelpswithnavigatingthecompositestructureoftheinterceptorchain.ThecontrolflowwithintheinterceptorchainiscontrolledbyaContextMap.SinceaContextMapitselfispassedasinputtoeachinterceptor,interceptorscanoptionallyadd,remove,orreorderinterceptorsinthechain.Thisisoneofthereasonswhymostofthemodulesofawebframework,suchasRouting,ContentNegotiation,RequestHandlers,andmore,arealsoimplementedasinterceptorsbyPedestal.

Interceptorfunctionsfor:enterand:leavemustreturnContextMapasavaluefortheexecutionflowtocontinuewiththenextinterceptor.Ifthefunctionsreturnanil(https://clojure.org/reference/data_structures#nil)value,aninternalservererrorisreportedbythePedestalframeworkandtheexecutionflowterminates.Aninterceptormayreturnacore.async(https://clojure.github.io/core.async/)channelinsteadoftheContextMap.Inthatcase,thechannelistreatedlikeapromisetodelivertheContextMapinthefuture.OncethechanneldeliverstheContextMap,thechainexecutorclosesthechannel:

Thechainexecutorcallseachinterceptorinthecore.asynclibrary'sgoblock(https://clojure.github.io/core.async/#clojure.core.async/go),sooneinterceptormaybecalledonadifferentthreadthanthenextbutallbindingsareconveyedtoeachinterceptor.InscenarioswheretheinterceptormaytakealongtimetoprocesstherequestormakeanexternalAPIcall,itisrecommendedtouseagoblocktosendachannelasthereturnvalueandletPedestalcontinuewiththeexecutionasynchronously.WhenPedestalreceivesachannelasanoutput,ityieldstheinterceptorthreadandwaitsforavaluetobeproducedbythechannel.Onlyonevalueisconsumedfromthechannel,anditmustbeaContextMap.

Asshownintheprecedingdiagram,whileexecutingtheinterceptorchain,all:enterfunctionsarecalledintheorderofinterceptorslistedinthechain.Onceallthe:enterfunctionsoftheinterceptorsarecalled,theresultingContextMapissentthroughthe:leavefunctionoftheinterceptorsbutinreverseorder.Sinceanyoftheinterceptorsinthechaincanreturnasynchronously,Pedestalcreateswhatitcallsavirtualcallstackofinterceptors.Itkeepsaqueueofinterceptorsforwhichithastocallthe:enterfunctionandalsomaintainsastackofinterceptorsforwhichthe:enterfunctionhasbeencalled,butthe:leavefunctionispending.KeepingastackallowsPedestaltocallthe:leavefunctioninthereverseorderofthe:enterfunctioncallsoftheinterceptorsinthechain.BothofthesequeuesandstacksarekeptintheContextMapandareaccessibletotheinterceptors.

SinceinterceptorshaveaccesstotheContextMap,theycanchangetheexecutionplanfortherestoftherequestbymodifyingthesequenceofinterceptorsinthechain.Theycannotonlyenqueueadditionalinterceptorsbutcanalsoterminatetherequesttoskipalltheremaininginterceptorsinthechain.

ImportanceofaContextMap

AcontextisjustaClojureMap(https://clojure.org/reference/data_structures#Maps)thatistakenasinputbytheinterceptorandalsogeneratedasanoutput.ItcontainsallthevaluesthatcontrolaPedestalapplication,includinginterceptors,chains,executionstacks,queues,andcontextvaluesthatmaybegeneratedbyinterceptorsorrequiredbyinterceptorsremainingintheinterceptorchain.AContextMapalsocontainsasequenceofpredicatefunctions.Ifanyoneofthepredicatefunctionreturnstrue,thechainisterminated.AlltherequiredfunctionsandsignalkeysarealsodefinedwithintheContextMaptoenableasynccapabilitiesandfacilitatetheinteractionbetweentheplatformandinterceptors.ThetableshownherelistssomeofthekeysthattheinterceptorchainkeepsintheContextMap:

Key Type Description

:bindingsMap(var->value)

Installedusingwith-bindings(https://clojuredocs.org/clojure.core/with-bindings)priortoexecution

:io.pedestal.interceptor.chain/error Exception Mostrecentexceptionthattriggerederrorhandling

:io.pedestal.interceptor.chain/execution-

id Opaque UniqueIDfortheexecution

:io.pedestal.interceptor.chain/queueQueueofinterceptors

Interceptorstobeexecutednext

:io.pedestal.interceptor.chain/stackStackofinterceptors

Interceptorslefttobeexecuted

:io.pedestal.interceptor.chain/terminators

Collectionofpredicates

Checksforvalidterminationpredicatesaftereach:enterfunction

Ifthe:bindingsmapisalteredbyaninterceptorandreturnedintheoutputContextMap,thenPedestalwillinstallthenewbindingsasthreadlocalbindingspriortotheexecutionofthenextinterceptorinthechain.The:io.pedestal.interceptor.chain/queuecontextkeycontainsalltheinterceptorsthatarelefttobeexecuted.Thefirstinterceptorinthequeueisthenextoneconsideredtobeexecutedbycallingthe:enterfunction.Thiskeymustbeusedonlyfordebuggingpurposes.Tomakeanychangestothequeueorexecutionflow,enqueue,terminate,orterminate-when(http://pedestal.io/api/pedestal.interceptor/io.pedestal.interceptor.chain.html#var-enqueue)intheinterceptorchainmustbecalledinsteadofchangingthevalueofthiskey.

Theterminationpredicatesspecifiedbythe:io.pedestal.interceptor.chain/terminatorskeysarecheckedforatruepredicateaftereach:enterfunctioncall.Ifthereisavalidpredicatefound,Pedestalskipsallotherremaininginterceptors':enterfunctionsandbeginsexecutingthe:leavefunctionofinterceptorsinthestacktoterminatetheexecutionflow.

The:io.pedestal.interceptor.chain/stackcontextkeycontainstheinterceptorsforwhichthe:enterfunctionhasalreadybeencalledbutthe:leavefunctionispending.Theinterceptoratthetopofthestackisexecutedfirsttomakesurethatthe:leavefunctioniscalledinthereverseorderof:enterfunctioncalls.

Inadditiontothekeysaddedbytheinterceptorchain,ContextMapmaycontainkeysthatareaddedbyotherinterceptorsaswell.Forexample,aservletinterceptor(http://pedestal.io/reference/servlet-interceptor),providedbyPedestaloutofthebox,addsservlet-specifickeystotheContextMap,suchas:servlet-request,:servlet-response,:servlet-config,and:servlet.WhenworkingwithHTTPserver,ContextMapalsohasthe:requestand:responsekeys,whichhaverequest(http://pedestal.io/reference/request-map)andresponse(http://pedestal.io/reference/response-map)mapsassignedtothem,respectively.

PedestalextendsbeyondjustHTTPservices.YoucanextendservicestoKafka-like(https://kafka.apache.org/)systemsandalsousedifferentprotocolssuchasSCTP,ReliableUDP,UDT,andmore.Thepedestal.servicePedestalmoduleisacollectionofHTTP-specificinterceptors.

CreatingaPedestalservicePedestalprovidesaLeiningen(https://leiningen.org/)templatenamedpedestal-servicetocreateanewprojectwiththerequireddependenciesanddirectorylayoutforaPedestalservice.Tocreateanewprojectusingthetemplate,usetheleincommandwiththetemplatenameandaprojectnameasshownhere:

#Createanewproject'pedestal-play'withtemplate'pedestal-service'

%leinnewpedestal-servicepedestal-play

Retrievingpedestal-service/lein-template/0.5.3/lein-template-0.5.3.pomfromclojars

Retrievingpedestal-service/lein-template/0.5.3/lein-template-0.5.3.jarfromclojars

Generatingapedestal-serviceapplicationcalledpedestal-play.

Theleincommandwillcreateanewdirectorywiththespecifiedprojectnameandaddalltherequireddependenciestotheproject.cljfile.Itwillalsoinitializetheserver.cljandservice.cljfileswiththecodetemplateforasamplePedestalservice.Thecreatedprojectdirectorytreeshouldlookliketheoneshownhere:

#Showthe'pedestal-play'projectdirectorystructure

%treepedestal-play

pedestal-play

├──Capstanfile

├──config

│└──logback.xml

├──Dockerfile

├──README.md

├──src

│└──pedestal_play

│├──server.clj

│└──service.clj

└──test

└──pedestal_play

└──service_test.clj

5directories,8files

Toruntheproject,justusetheleinruncommandanditwillcompileandstartthesampleservicedefinedbythetemplateatport8080.Totesttheservice,openthehttp://localhost:8080URLandhttp://localhost:8080/aboutinabrowserandobservetheresponse.ThefirstURLreturnstheresponseasHelloWorld!,whereasthesecondURLreturnsClojure1.8.0-servedfrom/about:

BoththeendpointscanalsobeaccessedfromcURLasshownhere:

%curl-vhttp://localhost:8080

*RebuiltURLto:http://localhost:8080/

*Trying127.0.0.1...

*Connectedtolocalhost(127.0.0.1)port8080(#0)

>GET/HTTP/1.1

>Host:localhost:8080

>Accept:*/*

<HTTP/1.1200OK

<Date:Fri,03Nov201706:30:00GMT

<Strict-Transport-Security:max-age=31536000;includeSubdomains

<X-Frame-Options:DENY

<X-Content-Type-Options:nosniff

<X-XSS-Protection:1;mode=block

<X-Download-Options:noopen

<X-Permitted-Cross-Domain-Policies:none

<Content-Security-Policy:object-src'none';script-src'unsafe-inline''unsafe-eval'

'strict-dynamic'https:http:;

<Content-Type:text/html;charset=utf-8

<Transfer-Encoding:chunked

*Connection#0tohostlocalhostleftintact

HelloWorld!

%curl-vhttp://localhost:8080/about

*Trying127.0.0.1...

*Connectedtolocalhost(127.0.0.1)port8080(#0)

>GET/aboutHTTP/1.1

>Host:localhost:8080

>Accept:*/*

<HTTP/1.1200OK

<Date:Fri,03Nov201706:30:09GMT

<Strict-Transport-Security:max-age=31536000;includeSubdomains

<X-Frame-Options:DENY

<X-Content-Type-Options:nosniff

<X-XSS-Protection:1;mode=block

<X-Download-Options:noopen

<X-Permitted-Cross-Domain-Policies:none

<Content-Security-Policy:object-src'none';script-src'unsafe-inline''unsafe-eval'

<Content-Type:text/html;charset=utf-8

<Transfer-Encoding:chunked

*Connection#0tohostlocalhostleftintact

Clojure1.8.0-servedfrom/about

Toruntheapplicationindevmode,startaREPLsessionviaCIDER(https://cider.readthedocs.io/en/latest/up_and_running/#launch-an-nrepl-server-and-client-from-emacs),andcallthepedestal-play.server/run-devfunctiontostarttheserverindevmode.

UsinginterceptorsandhandlersInthepedestal-playproject,theservice.cljsourcefiledefinestwointerceptors,about-pageandhome-page,whichareusedfortheservice.Additionally,itusestwoHTTP-specificinterceptors,body-params(http://pedestal.io/api/pedestal.service/io.pedestal.http.body-params.html#var-body-params)andhtml-body(http://pedestal.io/api/pedestal.service/io.pedestal.http.html#var-html-body),whichareprovidedoutoftheboxbythepedestal.servicemodule.Thepedestal-play.servicenamespacesnippetisshownherewiththeinterceptordeclarations:

(nspedestal-play.service

(:require[io.pedestal.http:ashttp]

[io.pedestal.http.route:asroute]

[io.pedestal.http.body-params:asbody-params]

[ring.util.response:asring-resp]))

;;UsedbyGET/about

(defnabout-page

[request]

(ring-resp/response(format"Clojure%s-servedfrom%s"

(clojure-version)

(route/url-for::about-page))))

;;UsedbyGET/

(defnhome-page

[request]

(defcommon-interceptors[(body-params/body-params)http/html-body])

InsteadoftakingaContextMapasinputandreturningtheContextMap,theabout-pageandhome-pageinterceptorstaketherequestmap(http://pedestal.io/reference/request-map)asanargumentandreturntheresponsemap(http://pedestal.io/reference/response-map).SuchinterceptorsarecalledhandlersandaretreatedasfunctionsbythePedestalframework.HandlersinthePedestalframeworkdonothaveaccesstoContextMap;therefore,theycannotchangethesequenceofinterceptorsinthechainandthatisthereasontheycanonlybeusedattheendoftheinterceptorchaintoconstructtheresponsemap.Theresponsefunction,usedbythehome-pageandabout-pagehandlers,isautilityfunctionprovidedbytheRingframework(https://github.com/ring-clojure/ring)tocreateaRingresponsemapwith200status,noheaders,andagivenbodycontent.

Thebody-paramsfunctionreturnsaninterceptorthatparsestherequestbodybasedonitsMIMEtype(https://en.wikipedia.org/wiki/Media_type)andaddstherelevantkeystotherequestmapwiththecorrespondingbodyparameters.Forexample,itwilladda:form-paramskeywithalltheformparametersfortherequestswiththecontenttypeofapplication/x-www-form-urlencoded.Theinterceptorspecifiedbythehtml-bodyvaraddstheContent-Typeheaderparametertotheresponsewiththetypeastext/html;charset=UTF-8.Sincehtml-bodyactsontheresponse,thefunctionforthisinterceptorisdefinedforthe:leavekey,whereasforbody-paramsthefunctionisdefinedforthe:enterkeyofitsinterceptormap.

AlltheinterceptorsinPedestalimplementtheIntoInterceptorprotocol(http://pedestal.io/api/pedestal.interceptor/io.pedestal.interceptor.html#var-IntoInterceptor).PedestalextendstheIntoInterceptorprotocoltoaMap,Function,List,Symbol,oraVarandallowsinterceptorstobedefinedinanyoftheseextendedforms.ThemostcommonwayofdefininganinterceptorisaClojureMap(https://clojure.org/reference/data_structures#Maps)with:enter,:leave,and:errorkeys.Itcanbedefinedasafunction,aswell,tobeconsideredahandler,suchastheabout-pageandhome-pageofthepedestal-playproject.

Creatingroutes

InterceptorsinPedestalareattachedtoroutesthatdefinetheendpointsforclientstointeractwiththeapplication.PedestalprovidesthreeprominentwaysofdefiningRoutes(http://pedestal.io/reference/routing-quick-reference#_routes)—verbose,table,andterse.AllthreearedefinedasClojuredatastructureswithverbosebeingdefinedasaMap(https://clojure.org/reference/data_structures#Maps),tableasaset(https://clojure.org/reference/data_structures#Sets),andterseasavector(https://clojure.org/reference/data_structures#Vectors).Thesethreeformatsaredefinedforconvenienceonly.Pedestalinternallyconvertsallroutedefinitionsintotheverboseformbeforeprocessingthemusingtheconvenientfunctionexpand-routes(http://pedestal.io/api/pedestal.route/io.pedestal.http.route.html#var-expand-routes).TheverbosesyntaxisalistofMapswithkeywordsthatdefinearoute.Alistofkeywordsandtheirdescriptionsareshowninthefollowingtable:

Keyword Samplevalue Description:route-name

:pedestal-

play.service/about-page Uniquenamefortheroute:app-name :pedestal-play Optionalnamefortheapplication:path /about/:id PathURI

:method :getHTTPverb;canbe:get,:put,:post,:delete,:any,andmore

:scheme :http Optional,suchas:httpand:https:host somehost.com Optionalhostname:port 8080 Portnumber

:interceptorsVectorofinterceptors

InterceptorMapswith:name,:enter,:leave,and:errorkeys

:query-

constraints

{:name#".+":search#"

[0-9]+"} Constraintsonqueryparameters,ifany

Additionally,thereareafewmorekeysderivedfromthe:pathparameterof

verbosesyntax,asfollows:

Keyword Samplevalue Description:path-re #"/\Qabout\E/([^/]+)" Regexusedtomatchthepath:path-parts ["about":id] PartsofthePathURI:path-params [:id] Pathparameters:path-constraints {:id"([^/]+)"} Constraintsforpathparameters,ifany

Thepedestal.routePedestalmodulecontainstheimplementationforbothRoutesandRouters.RoutersarespecialinterceptorsthattakeRoutesasinputinverboseformandprocesstheincomingrequeststotheRoutesbasedontheimplementation.Routersmakesurethattherequestsareprocessedthroughtheinterceptorsaspertheinterceptorchain.AnychangeintheinterceptorchainbyinterceptorsishandledbyRoutersefficiently.

Inthepedestal-playproject,theservice.cljsourcefiledefinestworoutes,GET/andGET/about,whichareboundedbytheinterceptorchainwithhome-pageandabout-pagehandlersattheendrespectively,asshowninthefollowingcode:

;;TabularRoutes

(defroutes#{["/":get(conjcommon-interceptors`home-page)]

["/about":get(conjcommon-interceptors`about-page)]})

The:app-name,:host,:port,and:schemekeyscanbespecifiedasmapsthatapplytoalltheroutesspecifiedinthelistofroutes,asshowninthefollowingcode:

(defroutes#{{:app-name"PedestalPlay":host"localhost":port8080:scheme:http}

["/":get(conjcommon-interceptors`home-page)]

["/about":get(conjcommon-interceptors`about-page)]})

Toseetheverbosesyntaxfortheroutes,usetheexpand-routes(http://pedestal.io/api/pedestal.route/io.pedestal.http.route.html#var-expand-routes)functionasshowninthefollowingREPLsession.Theoutputoftheexpand-routes(http://pedestal.io/api/pedestal.route/io.pedestal.http.route.html#var-expand-routes)functioniswhatispassedtothePedestalrouter.The:route-nameintheverboseformatisderivedfromthenameofthelastinterceptorinthechainorthesymbolattheendofthechainthatresolvestoafunction,suchas:pedestal-play.service/home-page:

;;REPL

pedestal-play.server>(io.pedestal.http.route/expand-routespedestal-

play.service/routes)

({:app-name"PedestalPlay",

:route-name:pedestal-play.service/home-page,

:scheme:http,

:host"localhost",

:port8080,

:path"/",

:method:get,

:path-re#"/\Q\E",

:path-parts[""],

:path-params[],

:interceptors

[{:name:io.pedestal.http.body-params/body-params,

:enter#function[io.pedestal.interceptor.helpers/on-request/fn--9231],

:leavenil,

:errornil}

{:name:io.pedestal.http/html-body,

:enternil,

:leave#function[io.pedestal.interceptor.helpers/on-response/fn--9248],

:errornil}

{:namenil,

:enter#function[io.pedestal.interceptor/eval157/fn--158/fn--159],

:leavenil,

:errornil}]}

{:app-name"PedestalPlay",

:route-name:pedestal-play.service/about-page,

:scheme:http,

:host"localhost",

:port8080,

:path"/about",

:method:get,

:path-re#"/\Qabout\E",

:path-parts["about"],

:path-params[],

:interceptors

[{:name:io.pedestal.http.body-params/body-params,

:enter#function[io.pedestal.interceptor.helpers/on-request/fn--9231],

:leavenil,

:errornil}

{:name:io.pedestal.http/html-body,

:enternil,

:leave#function[io.pedestal.interceptor.helpers/on-response/fn--9248],

:errornil}

{:namenil,

:enter#function[io.pedestal.interceptor/eval157/fn--158/fn--159],

:leavenil,

:errornil}]})

ThesameRoutescanbedefinedinatersesyntaxaswell,usingthevector(https://clojure.org/reference/data_structures#Vectors)ofnestedvectors.Eachvectordefinesanapplication,optionally,withanapplicationname,scheme,host,andport.Eachapplicationdeclarationhasoneormorenestedvectorsthatdefinetheroutes.Eachvectoraddsapathsegmentrepresentingthehierarchicaltreestructureofroutes,asshowninthefollowingcode:

(defroutes

`[["PedestalPlay":http"localhost"8080

["/"{:gethome-page}

^:interceptors[(body-params/body-params)http/html-body]

["/about"{:getabout-page}]]]])

Eachroutevectorcontainsapathsegment,suchas/about,interceptormetadatamap,constraintsmap,verbmap,andchildroutevectors,ifany.Interceptorsdefinedintheinterceptormetadatamapareappliedtoeveryroutedefinedintheverbmap,andtheverbkeyintheverbmapcontainsthevalueofverb-specifichandlerfunctionsoralistofinterceptors.

DeclaringroutersRoutersarefunctionsthatareaddedasinterceptorstothechaintoanalyzetherequestsbasedonthedefinedRoutes.Pedestalcreatesarouterbasedonthevaluesof:io.pedestal.http/routesand:io.pedestal.http/routerkeys,asspecifiedintheservicemap(http://pedestal.io/reference/service-map).TheservicemapcontainsallthedetailsforPedestaltocreateaserviceincludingarouter,routes,chain-providerproperties,andmore.Itactsasabuilder(https://en.wikipedia.org/wiki/Builder_pattern)forPedestalservices.

Pedestalprovidesthreebuilt-inroutersthatcanbespecifiedusingthe:map-tree,:prefix-tree,and:linear-searchkeywords.The:map-treeisthedefaultrouterthathasconstanttimecomplexitywhenappliedtoallroutesthatarestatic.Itfallsbacktoprefix-treeifanyrouteshavepathparametersorwildcards.

Ifthevalueof:io.pedestal.http/routerisspecifiedasafunctionthenthatfunctionisusedtoconstructarouter.Thefunctionmusttakeoneargument,thatis,thecollectionofroutesinverboseformat,andmustreturnarouterthatsatisfiesrouterprotocols(http://pedestal.io/api/pedestal.route/io.pedestal.http.route.router.html#var-Router).

Accessingrequestparameters

ServletChainProvider(http://pedestal.io/api/pedestal.service/io.pedestal.http.impl.servlet-interceptor.html)attachesarequestmaptotheContextMapwiththe:requestkeybeforethefirstinterceptorisinvoked.Therequestmapcontainsalltheform,query,andURLparametersspecifiedbytheclientoftheAPI.Theseparamsaredefinedasmapsofkey-valuepairs,whereeachkeyrepresentstheparameterspecifiedbytheclient.Alltheparamsareoptionalandarepresentonlyiftheyarespecifiedbytheclientintherequest.Hereisalistofthekeysthatarequestmapmaycontain:

Key Usedfor:path-

params PresentifanypathparametersarespecifiedandfoundbytheRouter

:query-

params

Presentifthequery-params(http://pedestal.io/api/pedestal.route/io.pedestal.http.route.html#var-query-params)interceptorisused(default)

:form-

params

Presentifthebody-params(http://pedestal.io/api/pedestal.service/io.pedestal.http.body-params.html#var-body-params)interceptorisusedandtheclientsendstherequestwithapplication/x-www-form-urlencodedasthecontenttype

:json-

params

Presentifthebody-paramsinterceptorisusedandtheclientsendstherequestwithapplication/jsonasthecontenttype

params

Presentifthebody-paramsinterceptorisusedandtheclientsendstherequestwithapplication/ednasthecontenttype

:params Mergedmapofpath,query,andrequestparameters

Apartfromparameters,therequestmapalsohasthesekeysthatarealwayspresentandcanbeusedbyinterceptorsinthechain:

Key Type Usedfor

:async-

supported? Boolean Trueifthisrequestsupportsasynchronousoperations

:body ServletInputStream Bodyoftherequest

:headers Map Requestheaderssentbytheclientwithallnamesconvertedtolowercase

:path-info String Requestpath,belowthecontextpath;alwayspresent,atleast/

:protocol String Nameandversionoftheprotocolwithwhichtherequestwassent

:query-

string String Thepartoftherequest'sURLafterthe?character

:remote-

addr String IPAddressoftheclient(orthelastproxytoforwardtherequest)

:request-

method KeywordHTTPverbused,inlowercaseandinkeywordformasdeterminedbythemethod-paraminterceptor(default)

:server-

name String Hostnameoftheservertowhichtherequestwassent

:server-

port Int Portnumbertowhichtherequestwassent

:scheme String Thenameoftheschemeusedfortherequest,suchashttp,https,orftp

:uri String RequestURIfromtheprotocolnameuptothequerystring

Toreviewtherequestmap,addanewdebug-pagehandlertothepedestal-playprojectandmapittotheroute/debugasshowninthefollowingcodesnippet.Thehandlerdebug-pagejustreturnstherequestmapintheresponsewithonlytheparameterkeysofinterest.ItalsoconvertsitintoaJSONstringusingtheCheshire(https://github.com/dakrone/cheshire)library:(defndebug-page[request](ring-resp/response

(cheshire.core/generate-string(select-keysrequest[:params:path-params:query-params:form-params]))))

;;CommonInterceptorsusedforallroutes(defcommon-interceptors[(body-params/body-params)http/html-body])

(defroutes#{{:app-name"PedestalPlay":host"localhost":port8080:scheme:http}["/":get(conjcommon-interceptors`home-page)]["/about":get(conjcommon-interceptors`about-page)]["/debug/:id":post(conjcommon-interceptors`debug-page)]})

ThecURLrequesttothe/debugroutenowprovidestheentirerequestmapthatcanbeinspectedforthepath,query,andformparamsasshowninthefollowingexample:curl-XPOST-d"formparam=1""http://localhost:8080/debug/1?qparam=1"{"params":{"qparam":"1","formparam":"1"},"path-params":{"id":"1"},"query-params":{"qparam":"1"},"form-params":{"formparam":"1"}}

CreatinginterceptorsAPedestalinterceptorcanbedefinedasamapwiththekeys:name,:enter,:leave,and:error.Forexample,aninterceptormsg-playcanbedefinedforthe/hellorouteofthepedestal-playprojecttochangethequeryparameternametouppercaseatthetimeofentryusingthe:enterfunctionandappendingagreetingatthetimeofexitusingthe:leavefunction.Itisfollowedbythehandlerhello-pagethatreadsthequeryparametersandaddsaHellogreeting.Takealookatthefollowingexample:

;;Handlerfor/helloroute

(defnhello-page

[request]

(ring-resp/response

(let[resp(clojure.string/trim(get-inrequest[:query-params:name]))]

(if(empty?resp)"HelloWorld!"(str"Hello"resp"!")))))

(defmsg-play

{:name::msg-play

:enter

(fn[context]

(update-incontext[:request:query-params:name]clojure.string/upper-case))

:leave

(fn[context](update-incontext[:response:body]

#(str%"Goodtoseeyou!")))})

;;CommonInterceptorsusedforallroutes

(defroutes#{{:app-name"PedestalPlay":host"localhost":port8080:scheme:http}

["/about":get(conjcommon-interceptors`about-page)]

["/debug/:id":post(conjcommon-interceptors`debug-page)]

["/hello":get

(conjcommon-interceptors`msg-play`hello-page)]})

ThecURLrequesttothe/helloroutewiththenamequeryparameternowprovidestheresultasexpectedwithboth:enterand:leaveeventsfiringforthemsg-playinterceptor:

%curl"http://localhost:8080/hello?name=clojure"

HelloCLOJURE!Goodtoseeyou!

Ifthequeryparameterisnotspecified,itreturnstheHelloWorldgreetingaspertheimplementationofthehello-pageinterceptor:

%curl"http://localhost:8080/hello?name="

HelloWorld!Goodtoseeyou!

HandlingerrorsandexceptionsInterceptorsmightthrowanexceptionduetoanerrorintheimplementation.Forexample,trycallingthe/helloroutewithnoqueryparameter.Itfailsandthrowsanexceptionduetotheusageoftheupper-casefunctiononanilvalueofthe:nameparameter.Theexceptionisthrownbythe:enterfunctionofthemsg-playinterceptorbutthereisno:errorfunctiondefinedfortheinterceptortohandletheexception.SucherrorsmustbehandledgracefullyanderrorsmustbereportedtothecallerusingappropriateHTTPstatuscodes.Inthiscase,iftherequiredparameter:nameisnotdefined,thentherouteshouldreturnaresponsewithaHTTP400BadRequest,alongwithameaningfulmessageforthecaller.

PedestalunifieserrorhandlingforbothsynchronousinterceptorsthatreturnContextMap,andasynchronousinterceptorsthatreturnachannel.Itcatchesallexceptionsthrownwithinaninterceptorandbindsittothe:io.pedestal.interceptor.chain/errorkeyintheContextMap.Oncetheerrorisattachedtothekey,Pedestalstartslookingforthenextinterceptorinthechainthathasan:errorfunctionattachedtoit.

Tohandletheexceptionwiththe/hellorouteofthepedestal-playproject,an:errorfunctioncanbedefinedforthemsg-playinterceptor.The:errorfunctioncanthencatchanyexceptionthrownbytheinterceptorandassociateanappropriateresponsewiththeContextMap.Takealookatthefollowingexample:

(defmsg-play

{:name::msg-play

:enter

(fn[context]

(update-incontext[:request:query-params:name]

clojure.string/upper-case))

:leave

(fn[context](update-incontext[:response:body]

#(str%"Goodtoseeyou!")))

:error

(fn[contextex-info]

(assoccontext:response{:status400:body"Invalidname!"}))})

ThecURLrequestforthe/helloroutenowprovidesanappropriateresponsewiththecorrectstatuscodeandamessage,asshowninthefollowingexample:

%curl-i"http://localhost:8080/hello"

HTTP/1.1400BadRequest

Invalidname!

The:errorfunctionreceivestwoarguments,ContextMapandanex-infoexception.ItcaneitherreturnaContextMaptocatchtheerrororupdatethe:io.pedestal.interceptor.chain/errorkeywiththeexceptiontolookforahandler.Itcanalsore-throwtheexceptionorthrowanewexceptiontosignalsomethingwentwrongwhilehandlingtheexception.Inbothcases,Pedestalwillstartlookingforahandleroftheexception.Pedestalkeepstrackofalltheexceptionsthatwereoverriddenbyaddingtheminsequencetothekey:io.pedestal.interceptor.chain/suppressedofContextMap.

Pedestalalsoprovidesanerror-dispatch(http://pedestal.io/api/pedestal.interceptor/io.pedestal.interceptor.error.html#var-error-dispatch)macrotobuilderror-handling(http://pedestal.io/reference/error-handling#_error_dispatch_interceptor)interceptorsthatusepatternmatchingtoselectaclause.

LoggingThePedestalmodulepedestal.logcontainscomponentsforloggingandalsoreportingruntimeoperationalmetrics.PedestalusesLogback(https://logback.qos.ch/)forlogginganditcanbeconfiguredbycreatingalogback.xmlfileintheprojectconfigdirectory.Natively,logback-classicimplementsSLF4J(https://www.slf4j.org/),whichisusedbyPedestalaswellforlogging.Pedestalimplementseachlogginglevel—trace,debug,info,warn,anderror—asmacrosthattakekey-valuepairsasparameters,printedusingtheprfunction(https://clojuredocs.org/clojure.core/pr).TologanexceptionviaPedestallogger,the:exceptionkeymustbeusedwithajava.lang.Throwableobjectasavalueassignedtoit.

ThedefaultprojecttemplateofPedestalcontainsalogback.xmlfileintheconfigdirectoryalongwiththerelevantdependenciesaddedintheproject.cljfileforrequiredloggerimplementations.Thedefaultloggingconfigurationofthepedestal-playprojectlogsthelogbackconfigurationintheconsoleandalsointhelogfile,asmentionedinthelogback.xmlfile.Takealookatthefollowingexample:%leinrun18:53:30,595|-INFOinch.qos.logback.classic.LoggerContext[default]-CouldNOTfindresource[logback.groovy]18:53:30,595|-INFOinch.qos.logback.classic.LoggerContext[default]-CouldNOTfindresource[logback-test.xml]18:53:30,595|-INFOinch.qos.logback.classic.LoggerContext[default]-Foundresource[logback.xml]at[file:/pedestal-play/config/logback.xml]...18:53:30,686|-INFOinch.qos.logback.core.joran.action.AppenderAction-Abouttoinstantiateappenderoftype[ch.qos.logback.core.rolling.RollingFileAppender]18:53:30,688|-INFOinch.qos.logback.core.joran.action.AppenderAction-Namingappenderas[FILE]18:53:30,691|-INFOinch.qos.logback.core.joran.action.NestedComplexPropertyIA-Assumingdefaulttype[ch.qos.logback.classic.encoder.PatternLayoutEncoder]for[encoder]property18:53:30,711|-INFOin

c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@2017577360-Archivefileswillbelimitedto[64MB]each.18:53:30,712|-INFOinc.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@2017577360-Nocompressionwillbeused18:53:30,713|-INFOinc.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@2017577360-Willusethepatternlogs/pedestal-play-%d{yyyy-MM-dd}.%i.logfortheactivefile18:53:30,716|-INFOinch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@1e75bef1-Thedatepatternis'yyyy-MM-dd'fromfilenamepattern'logs/pedestal-play-%d{yyyy-MM-dd}.%i.log'.18:53:30,716|-INFOinch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@1e75bef1-Roll-overatmidnight.18:53:30,719|-INFOinch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@1e75bef1-SettinginitialperiodtoTueNov0718:53:30IST201718:53:30,719|-WARNinch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@1e75bef1-SizeAndTimeBasedFNATPisdeprecated.UseSizeAndTimeBasedRollingPolicyinstead18:53:30,721|-INFOinch.qos.logback.core.rolling.RollingFileAppender[FILE]-Activelogfilename:logs/pedestal-play-2017-11-07.0.log...

Oncetheserverisstarted,thedefaultPedestalloggerlogseachrouteaccessrequestasanINFOmessageinthelog:Creatingyourserver...INFOorg.eclipse.jetty.server.Server-jetty-9.4.0.v20161208INFOo.e.j.server.handler.ContextHandler-Startedo.e.j.s.ServletContextHandler@62765e11{/,null,AVAILABLE}INFOo.e.jetty.server.AbstractConnector-StartedServerConnector@58fef400{HTTP/1.1,[http/1.1,h2c]}{0.0.0.0:8080}INFOorg.eclipse.jetty.server.Server-Started@4829msINFOio.pedestal.http-{:msg"GET/hello",:line80}

Pedestal'sservletinterceptor(http://pedestal.io/reference/servlet-interceptor)providesa

defaulterrorhandlerthatlogstheHTTPrequestsandalsoexceptions,ifany.Italsoemitstheexceptionstacktraceintheresponsebodyindevelopmentmode.Pedestal'sloggingisbackedbytheLoggerSourceprotocol,whichcanbeimplementedforcustomloggers.

PublishingoperationalmetricsOperationalmetricsareusefultounderstandtheusageandperformanceoftheapplicationbyobservingitsruntimestateasreportedbythemetrics.PedestalprovidesaloggingcomponentthatusestheMetricslibrary(http://metrics.dropwizard.io/3.2.3/)topublishthemetricstoJMX(https://en.wikipedia.org/wiki/Java_Management_Extensions)bydefaultviaMetricRegistry(http://metrics.dropwizard.io/3.1.0/getting-started/#the-registry).TheprotocolimplementedbyPedestalmetricsisMetricRecorder,whichcanbeimplementedforcustommetricsimplementation.Bydefault,MetricRecorderprovidesfourtypesofrecordersasshowninthefollowingtable:

Metricrecorder Usage

Gauge Usedfortheinstantaneousmeasurementofavalue

Counter Usedtoincrement/decrementasinglenumericmetric

Histogram Usedtomeasurethestatisticaldistributionofvalues(min,max,mean,median,percentiles)

Meter Usedtomeasuretherateofatickingmetric

Tocountthenumberofrequestsreceivedforeachrouteofthepedestal-playproject,acountercanbeaddedandincrementedeverytimethecorrespondinghandleriscalled:

[io.pedestal.log:aslog]

(defnabout-page

[request]

(log/counter::about-hits1)

(ring-resp/response(format"Clojure%s-servedfrom%s"

(clojure-version)

(route/url-for::about-page))))

(defnhome-page

[request]

(log/counter::home-hits1)

(defndebug-page

[request]

(log/counter::debug-hits1)

(ring-resp/response

(cheshire.core/generate-string

(select-keysrequest[:params:path-params:query-params:form-params]))))

(defnhello-page

[request]

(log/counter::hello-hits1)

(ring-resp/response

(let[resp(clojure.string/trim(get-inrequest[:query-params:name]))]

(if(empty?resp)"HelloWorld!"(str"Hello"resp"!")))))

Bydefault,thecounterwillbepublishedviaJMXandcanbelookedupusingtheJVMmonitoringtoolJConsole(https://en.wikipedia.org/wiki/JConsole).TopublishthemetricstoJMX,startthePedestalapplicationinREPLusingpedestal-play.server/run-devandaccessvariousroutesdefinedforthepedestal-playapplication.Next,openJConsoleandconnecttotheclojure.mainprocessformetrics.Itwillstartlistingtheroutemetricsundertheio.pedestal.metricsMBean(https://en.wikipedia.org/wiki/Java_Management_Extensions#Managed_beans).Takealookatthefollowingscreenshot:

UsingchainprovidersPedestalprovidesservletinterceptoraschainprovidersforHTTP-basedwebapplicationsoutofthebox.Itconnectsanyservletcontainertotheinterceptorchain.Bydefault,thePedestalapplicationtemplateusestheJettywebserver(https://www.eclipse.org/jetty/),butitalsohassupportforchainprovidersthatworkwithserversotherthanJettyaswell,suchasImmutant(http://immutant.org/)andTomcat(https://tomcat.apache.org/).

Tostarttheapplicationwiththedefaultserverandchainprovider,thatis,forJetty,runleinrunwithinthepedestal-playapplicationandobservethelogmessages.ItshowsthelogsfortheJettyserver,thatis,theserverinuse:

Creatingyourserver...

INFOorg.eclipse.jetty.server.Server-jetty-9.4.0.v20161208

INFOo.e.j.server.handler.ContextHandler-Started

o.e.j.s.ServletContextHandler@62765e11{/,null,AVAILABLE}

INFOo.e.jetty.server.AbstractConnector-StartedServerConnector@4789995a{HTTP/1.1,

[http/1.1,h2c]}{0.0.0.0:8080}

INFOorg.eclipse.jetty.server.Server-Started@4755ms

INFOio.pedestal.http-{:msg"GET/about",:line80}

INFOio.pedestal.http-{:msg"GET/hello",:line80}

Touseadifferentchain-provider,say,Immutant,changetheservicemaptouseImmutantasthechain-provider.Takealookatthefollowingcode:

(defservice{:env:prod

::http/routesroutes

::http/resource-path"/public"

;;Either:jetty,:immutantor:tomcat

::http/type:immutant

::http/port8080

Also,changetheproject.cljfiletousetheImmutantimplementationasfollows:

(defprojectpedestal-play"0.0.1-SNAPSHOT"

:description"FIXME:writedescription"

:url"http://example.com/FIXME"

:dependencies[[org.clojure/clojure"1.8.0"]

[io.pedestal/pedestal.service"0.5.3"]

;;Removethislineanduncommentoneofthenextlinesto

;;useImmutantorTomcatinsteadofJetty:

;;[io.pedestal/pedestal.jetty"0.5.3"]

[io.pedestal/pedestal.immutant"0.5.3"]

;;[io.pedestal/pedestal.tomcat"0.5.3"]

[ch.qos.logback/logback-classic"1.1.8":exclusions[org.slf4j/slf4j-

[org.slf4j/jul-to-slf4j"1.7.22"]

[org.slf4j/jcl-over-slf4j"1.7.22"]

[org.slf4j/log4j-over-slf4j"1.7.22"]]

:min-lein-version"2.0.0"

:resource-paths["config","resources"]

:main^{:skip-aottrue}pedestal-play.server)

Now,theleinrunlogsshowUndertow(http://undertow.io/)beingused,thatis,thewebserverusedbyImmutantlibrariesfortheweb.TheroutesandloggingworkexactlysameaswiththeJettywebserver:

INFOorg.xnio-XNIOversion3.4.0.Beta1

INFOorg.xnio.nio-XNIONIOImplementationVersion3.4.0.Beta1

WARNio.undertow.websockets.jsr-UT026010:Bufferpoolwasnotseton

WebSocketDeploymentInfo,thedefaultpoolwillbeused

INFOorg.projectodd.wunderboss.web.Web-Registeredwebcontext/

INFOio.pedestal.http-{:msg"GET/about",:line80}

INFOio.pedestal.http-{:msg"GET/hello",:line80}

Pedestalisnotjustlimitedtowebapplications.InterceptorsinPedestalcanbeusedinmessageprocessinganddataflowapplications,aswell,andarenotlimitedtoonlyrequest/reply-basedwebservices.

Usingserver-sentevents(SSE)Server-sentevents(SSE)(https://www.w3.org/TR/eventsource/)isastandardthatenablesefficientserver-to-clientstreamingusingatwo-partimplementation.ThefirstpartistheEventSourceAPIthatisimplementedattheclientsidetoinitiatetheSSEconnectionwiththeserver,andthesecondpartisthepushprotocolthatdefinestheeventstreamdataformatthatisusedfortheserver-to-clientcommunication.

TheEventSourceAPIofSSEisdefinedasapartoftheHTML5standardbyW3C(https://en.wikipedia.org/wiki/World_Wide_Web_Consortium)andisnowsupportedbyallthemodernwebbrowsers:

SSEsareprimarilyusedtopushreal-timenotifications,updates,andcontinuousdatastreamsfromservertoclientonceaninitialconnectionhasbeenestablishedbytheclient.Generally,thenotificationsandupdatesarepulledbytheclientbysendingarequesttotheAPIsorpolling(https://en.wikipedia.org/wiki/Polling_(computer_science))theserverforupdates.Pollingrequiresanewconnectiontobeestablishedbetweentheclientandserverforeachrequesttopullthenotificationsandupdates,asshownintheprecedingdiagram.SSEinsteadfocusesonthepushmodelinwhichtheconnectionisestablishedoncebytheclientandislong-lived.Allthenotifications,updates,anddatastreamsarepushedbytheserveroverthe

sameclientconnection.Ifaconnectionislost,theclientautomaticallyreconnectswiththeservertokeeptheconnectionactive.TheflowbetweentheserverandtheclientisfurtherillustratedintheprecedingdiagramforbothpollingandSSEmodesofinteraction.

ThePedestalservicecomponentincludessupportforSSEaswell.Itsendsallitseventsasapartofasingleresponsestreamthatiskeptaliveovertimebysendingeventsand/orperiodicheartbeatdata.Iftheresponsestreamisclosedortheconnectionsisinterrupted,theclientcansendarequesttoreopenitandcontinuetoreceivetheeventsnotificationsfromtheserver.

CreatinginterceptorsforSSEAninterceptorforSSEcanbecreatedusingthestart-event-stream(http://pedestal.io/api/pedestal.service/io.pedestal.http.sse.html#var-start-event-stream)functionprovidedbythePedestalservicecomponent.Ittakesafunctionasinputandreturnsaninterceptor.Thefunctionexpectedbythestart-event-streamfunctioniscalledbyPedestaloncetheinitialHTTPconnectionisestablishedwiththeclient;theHTTPresponseispreparedandPedestalnotifiestheclientthatanSSEstreamisstarting.Ittakesachannelasinput(https://clojure.github.io/core.async/),andContextMap.Tosendtheeventstotheclient,thefunctionjustpublishesthemonthechannelprovidedasanargumenttothefunction.Inadditiontothechannelprovidedasanargumenttothefunction,ContextMapalsocontainsachannelassociatedwiththe:response-channelkey.ThischannelisdirectlyconnectedwiththeresponseOutputStreamandmustnotbeusedtosendeventstotheclient.

TocreateaninterceptorforSSEinthepedestal-playproject,defineafunctionsse-stream-readyandpassitasanargumenttothestart-event-stream.Thestart-event-streamreturnsaninterceptorthatisassignedtoaroute/eventsthatinitiatesSSE.Thefunctionsse-stream-readyreadsarequestparametercounterforthenumberofmessagestobesenttotheclientanddefaultstofive.ItpublishesaMaponthechannelevent-chwithtwokeys,:nameand:data,thatcontainastringvalue.Itisrecommendedtousenamedeventsastheyarehelpfulforclientstotakeappropriateactionbasedonthenameoftheevent.Oncetherequirednumberofeventsaresent,itsendsacloseeventandclosesthechannel.Onceitclosesthechannel,Pedestalcleansuptheconnection.Takealookatthefollowingcode:

[io.pedestal.http.sse:assse]

[ring.util.response:asring-resp]

[clojure.core.async:asasync]))

(defnsse-stream-ready

"Startssendingcountereventstoclient."

[event-chcontext]

(let[count-num(Integer/parseInt

(or(->(context:request)

:query-params:counter)"5"))]

(loop[countercount-num]

(async/put!

event-ch{:name"count"

:data(strcounter",T:"

(.getId(Thread/currentThread)))})

(Thread/sleep2000)

(if(>counter1)

(recur(deccounter))

(async/put!event-ch{:name"close":data"Iamdone!"})

(async/close!event-ch))))))

(defroutes#{{:host"localhost":port8080:scheme:http}

["/about":get(conjcommon-interceptors`about-page)]

["/debug/:id":post(conjcommon-interceptors`debug-page)]

["/hello":get

(conjcommon-interceptors`msg-play`hello-page)]

["/events":get

[(sse/start-event-streamsse-stream-ready)]]})

Theroute/eventsisdefinedundertheroutesofthepedestal-playapplicationthatareusedtoreceiveeventsoverSSE.Ithasaninterceptorassignedthatiscreatedbycallingthestart-event-streamfunctionwiththesse-stream-readyfunctionthatpublishestheevents.Whenarequestreachesthisinterceptor,itpausestheinterceptorexecutionandsendsHTTPresponseheaderstotheclientstatingthataneventstreamisstarting,andinitiatesatimedheartbeattokeeptheconnectionalive.Oncetheconnectionisestablished,itcallsthesse-stream-readyfunctionwiththechannelandthecurrentContextMap.

TotrytheSSEendpoint,runthepedestal-playapplicationusingleinrunandusecURLtosendthegetrequesttothe/eventsendpoint.Takealookatthefollowingexample:

%curl-i-XGET"http://localhost:8080/events"

HTTP/1.1200OK

Date:Wed,08Nov201707:07:51GMT

X-Frame-Options:DENY

X-XSS-Protection:1;mode=block

X-Download-Options:noopen

Strict-Transport-Security:max-age=31536000;includeSubdomains

X-Permitted-Cross-Domain-Policies:none

Cache-Control:no-cache

X-Content-Type-Options:nosniff

Content-Security-Policy:object-src'none';script-src'unsafe-inline''unsafe-eval'

Content-Type:text/event-stream;charset=UTF-8

Connection:close

event:count

data:5,T:22

event:count

data:4,T:22

event:count

data:3,T:22

event:count

data:2,T:22

event:count

data:1,T:22

event:close

data:Iamdone!

Bydefault,itsendsfiveevents,butthatcanbecontrolledusingtherequestqueryparametercounter.Takealookatthefollowingexample:

%curl-XGET"http://localhost:8080/events?counter=2"

event:count

data:2,T:22

event:count

data:1,T:22

event:close

data:Iamdone!

TheSSEinterceptorsendsapartialHTTPresponsetotheclientasapartoftheconnectioninitializationprocessitself,therefore,anydownstreaminterceptorsarenotallowedtochangetheContextMapandtheresponsemap.Theycanonlyexaminethem.

PedestalsupportstheuseofLast-Event-ID(https://www.w3.org/TR/eventsource/#last-event-id)aswell,whichallowstheclienttoreconnectandresumefromthepointwhereitgotdisconnected.BasedontheSSEspec(https://www.w3.org/TR/eventsource/),PedestalsupportsassigningastringIDtotheSSEstreamthatcanbereferredtobytheclientintheLast-Event-IDheadertoresume.

UsingWebSocketsWebSocketsisacommunicationsprotocol(https://en.wikipedia.org/wiki/Communication_protocol)thatprovidesfull-duplex(https://en.wikipedia.org/wiki/Duplex_(telecommunications)#FULL-DUPLEX)communicationchannelsoverasingleTCPconnectionbetweenclientandserver.ItallowsclientstosendmessagestotheserverandreceiveservereventsoverthesameTCPconnectionwithoutpolling.ComparedtoServer-SentEvents(SSE)(https://www.w3.org/TR/eventsource/),WebSocketssupportfull-duplexcommunicationbetweenclientandserverinsteadofaone-waypush.Also,SSEsareimplementedoverHTTP,whichisanentirelydifferentTCPprotocolcomparedtoWebSocket.Althoughbothprotocolsaredifferent,theybothdependontheTCPlayer.

RFC6455(https://tools.ietf.org/html/rfc6455)statesthatWebSocketisdesignedtoworkoverHTTPports80and443aswellastosupportHTTPproxiesandintermediaries,thusmakingitcompatiblewiththeHTTPprotocol.Toachievecompatibility,theWebSockethandshakeusestheHTTPUpgradeheader(https://en.wikipedia.org/wiki/HTTP/1.1_Upgrade_header)tochangefromtheHTTPprotocoltotheWebSocketprotocol.

Server-senteventsareusefultosendnotificationsoralertsfromtheserverasandwhentheyoccur.Iftheapplicationrequiresaninteractivesessionbetweentheclientandtheserver,thenWebSocketsmustbepreferred.

UsingWebSocketwithPedestalandJettyPedestalprovidesout-of-the-boxsupportforJettyWebSocketsasapartofitspedestal.jettymodule.TocreateandregisteraWebSocketendpoint,Pedestalprovidestheadd-ws-endpointsfunctionthatacceptsaServletContextHandlerandaMapofWebSocketpathstotheactionMap.BasedontheprovidedWebSocketpaths,itproducesthecorrespondingservletsandaddsthemtothecontextoftheservletcontainer,thatis,Jettyinthiscase.TheservletcontainerthenmakestheWebSocketpathsavailablefortheclientstoconnecttousingtheWebSocketprotocol.TheWebSocketendpointsarecommunicatedtotheJettycontainerusingthe:context-configuratorkeyofthemapassignedtothe::http/container-optionskeyofPedestal'sservicemap.

Thepedestal-playprojectdefinesaws-pathsmapthatcontainsasingleWebSocketpath,/chat.Theactionsdefinedforthe/chatpathare:on-connect,:on-text,:on-binary,:on-error,and:on-close.Thestart-ws-connectionfunction,providedbyPedestal,acceptsafunctionoftwoarguments—theJettyWebSocketsessionanditspairedcore.asyncchannel—andreturnsafunctionthatcanbeusedasan:on-connectactionhandler.Forotheractions,thesamplepedestal-playprojectjustlogsamessage.Takealookatthefollowingexample:

[io.pedestal.http.sse:assse]

[io.pedestal.http.jetty.websockets:asws]

[ring.util.response:asring-resp]

[clojure.core.async:asasync]))

;;Atomtoholdclientsessions

(defws-clients(atom{}))

(defnnew-ws-client

"Keepstrackofallclientsessions"

[ws-sessionsend-ch]

(async/put!send-ch"Welcome!")

(swap!ws-clientsassocws-sessionsend-ch))

(defws-paths

{"/chat"{:on-connect(ws/start-ws-connectionnew-ws-client)

:on-text(fn[msg]

(log/info:msg(str"Client:"msg)))

:on-binary(fn[payloadoffsetlength]

(log/info:msg"BinaryMessage!":bytespayload))

:on-error(fn[t]

(log/error:msg"WSErrorhappened":exceptiont))

:on-close(fn[num-codereason-text]

(log/info:msg"WSClosed:"

:reasonreason-text))}})

::http/routesroutes

::http/type:jetty

::http/port8080

;;Optionstopasstothecontainer(Jetty)

::http/container-options

{:h2c?true

:h2?false

:ssl?false

:context-configurator#(ws/add-ws-endpoints%ws-paths)}})

Inthepedestal-playexample,thefunctionprovidedtothestart-ws-connectionfunctionisnew-ws-client,afunctionthatsendsaWelcome!messagetoeachnewclientandkeepstrackofclientsessions.Thepedestal-playexamplealsodefinesacoupleofutilitymethods,send-and-close!andsend-message-to-all!,tosendmessagesfromtheserversidetotheclientsconnectedtotheWebSocket.Takealookatthefollowingexample:

(defnsend-and-close!

"Utilityfunctiontosendmessagetoaclientandclosetheconnection"

[message]

(let[[ws-sessionsend-ch](first@ws-clients)]

(async/put!send-chmessage)

(async/close!send-ch)

(swap!ws-clientsdissocws-session)

(log/info:msg(str"ActiveConnections:"(count@ws-clients)))))

(defnsend-message-to-all!

"Utilityfunctiontosendmessagetoallclients"

[message]

(doseq[[^org.eclipse.jetty.websocket.api.Sessionsessionchannel]

@ws-clients]

(when(.isOpensession)

(async/put!channelmessage))))

Totestthe/chatWebSocket,startthepedestal-playapplicationinREPLusingthepedestal-play.server/run-devfunction:

pedestal-play.server>(defsrv(run-dev))

Creatingyour[DEV]server...

INFOorg.eclipse.jetty.server.Server-jetty-9.4.0.v20161208

INFOo.e.j.server.handler.ContextHandler-Started

o.e.j.s.ServletContextHandler@15d129a1{/,null,AVAILABLE}

INFOo.e.jetty.server.AbstractConnector-StartedServerConnector@766b5ac6{HTTP/1.1,

[http/1.1,h2c]}{0.0.0.0:8080}

INFOorg.eclipse.jetty.server.Server-Started@220932ms

#'pedestal-play.server/srv

pedestal-play.server>

Now,opentheJavaScriptconsoleinawebbrowserandstarttheWebSocketsessionusingthefollowingcommands:

//connectstothe'/chat'endpointwith'ws'protocol

w=newWebSocket("ws://localhost:8080/chat")

//logallthemessagesreceivedfromserveronconsole

w.onmessage=function(e){console.log(e.data);}

//messagetobeshownwhenserverclosestheconnection

w.onclose=function(e){

console.log("Theconnectiontotheserverhasclosed.");}

//sendamessagetoserver

w.send("HellofromtheClient-1!");

AnymessagethatissentfromtheclientisreceivedbytheserverandloggedontheREPL.Similarly,anymessagesentbytheserverusingthefunctionsend-message-to-all!isbroadcastedtoalltheactiveclientconnections.ToclosetheWebSocketconnection,callthesend-and-close!functionthatwillpickthefirstclientconnection,sendamessage,andcloseit.Italsologsthenumberofactiveclientconnectionsasshowninthefollowingcode:

INFOpedestal-play.service-{:msg"Client:HellofromtheClient-1!",:line113}

INFOpedestal-play.service-{:msg"Client:HellofromtheClient-2!",:line113}

pedestal-play.server>(pedestal-play.service/send-message-to-all!"HellofromPedestal

Server!")

pedestal-play.server>(pedestal-play.service/send-and-close!"GoodbyefromPedestal

Server!")

INFOpedestal-play.service-{:msg"ActiveConnections:1",:line102}

INFOpedestal-play.service-{:msg"WSClosed:",:reasonnil,:line119}

pedestal-play.server>(pedestal-play.service/send-and-close!"GoodbyefromPedestal

Server!")

INFOpedestal-play.service-{:msg"ActiveConnections:0",:line102}

INFOpedestal-play.service-{:msg"WSClosed:",:reasonnil,:line119}

ThefollowingscreenshotcapturestheWebSocketinteractionsamongaREPLsessionandtwobrowserclients,connectedviaaJavaScriptconsole:

Summary

Inthischapter,welearnedabouttheconceptsofthePedestalframeworkandhowtousethemtocreateAPIs.WealsolearnedhowtologusefulinformationfordebuggingandmonitoringtheruntimestateoftheapplicationusingJMXmetrics.WealsolookedatvariouswebserverpluginsthatcanbeusedwithPedestal.Finally,welookedathowPedestalcanbeusedforSSEsandWebSocketsforclient-serverinteraction.

Inthenextchapter,wewilltakealookataDatomicdatabasethatwillbeusedforpersistencebythemicroservicesoftheHelpingHandsapplication.DatomiciswritteninClojureandfitsinwellwiththeHelpingHandsapplication,whichrequirestransactionsandtemporalqueries.

AchievingImmutabilitywithDatomic

"Mostofthebiggestproblemsinsoftwareareproblemsofmisconception."

-RichHickey

Microservicesdependontheunderlyingdatabasetoreliablystoreandretrievedata.OftenapplicationslikeHelpingHandsneedtostoreusertransactionsconsistentlyalongwithuserlocationsthatmaychangeovertime.Insteadofupdatingtheuserlocationpermanentlyandlosingthehistoryofthechanges,agoodapplicationmustmaintainthechangeindatasothatitcanbequeriedovertime.Suchrequirementsexpectthedatastoredinthedatabasetobeimmutable.Datomic(http://www.datomic.com/)isonesuchdatabasethatnotonlyprovidesdurabletransactionsbutalsohastheconceptofimmutabilitybuiltintoitscoresothatuserscanquerythestateofthedatabaseoveraperiodoftime.DatomicisalsowritteninClojure,whichisthetechnologystackofchoicefortheHelpingHandsapplication.Inthischapter,youwilllearnaboutthefollowing:

DatomicarchitectureanditsdatamodelHowtostoreandretrievedataasfactswithDatomicDatalogquerylanguagetoretrievefactsHowtoqueryimmutablefactswithanexample

DatomicarchitectureDatomicisadistributeddatabasethatsupportsACID(http://docs.datomic.com/acid.html)transactionsandstoresdataasimmutablefacts.Datomicisfocusedonprovidingarobusttransactionmanagertokeeptheunderlyingdataconsistent,adatamodeltostoreimmutablefacts,andaqueryenginetohelpretrievedataasfactsovertime.Insteadofhavingitsownstorage,itreliesonanexternalstorageservice(http://docs.datomic.com/storage.html)tostorethedataondisk.

DatomicversustraditionaldatabaseAtypicaldatabaseisimplementedasamonolithicapplicationthatcontainsthestorageengine,queryengine,andthetransactionmanagerallpackagedasasingleapplicationtowhichclientsconnecttostoreandretrievedata.Datomic,ontheotherhand,takesaradicalapproachofseparatingouttheTransactionManager(Transactor)asaseparateprocesstohandleallthetransactionsandcommitthedatatoanunderlyingStorageServicethatactsasapersistencestoreforallthedatamanagedbyDatomic.Ahigh-levelarchitectureofDatomicanditscomparisonwithtraditionaldatabasesisshownhere:

ClientsofDatomicarecalledpeersandhavetheapplicationcodeandpeerlibrary(http://docs.datomic.com/integrating-peer-lib.html)thatconnectwiththeTransactortostoredataconsistently.Peersalsoquerytheunderlyingstorageservicefordataandmaintainacachetoreducetheloadontheunderlyingstorageservice.PeersalsoreceiveupdatesfromtheTransactoraswellandaddtothecache.PeersinDatomicarethickclientsthatcanbeconfiguredtocache(http://docs.datomic.com/caching.html)thedatain-memory,oruseexternalcachingsystemslikeMemcached(https://en.wikipedia.org/wiki/Memcached)tostoretheobjects.Thecachemaintainedbypeersalwayscontainsimmutablefactsthatarealwaysvalid.

DatomicalsohasaPeerServerthatallowslightweightclientslikethatof

traditionaldatabasestoconnecttoitdirectlytoquerythedatabase.Itactsasacentralqueryprocessorforalltheclientsconnectedtoit.Datomicprovidesaconsole(http://docs.datomic.com/console.html)aswell,whichhasagraphicaluserinterfacetomanageschema,examinetransactions,andexecutequeries.

Datomicisdesignedfortransactionaldataandmustbeusedtostoreuserprofiles,orders,inventorydetails,andmore.Itshouldnotbeusedforhigh-throughputusecasessuchasthosefoundinIoT(https://en.wikipedia.org/wiki/Internet_of_things),whichrequiresatime-seriesdatabasetostoreincomingdatawithhighvelocity.

DevelopmentmodelDatomicprovidestwodevelopmentmodels(http://docs.datomic.com/clients-and-peers.html)—PeerandClient.BoththemodelsrequireaTransactor(http://docs.datomic.com/transactor.html)toberunningtohandlethetransactionsandstorethedataconsistently.IntheClientmodel,aPeerServer(http://docs.datomic.com/peer-server.html)isrequiredinadditiontotheTransactortocoordinatethestorageandqueryrequestsfromtheclients.

Boththedevelopmentmodelsandtheparticipatingcomponentsareshowninthefollowingdiagram:

ClientsarelightweightinthecaseofaclientmodelasalltheinteractionwiththeTransactorandstorageengineishandledbythePeerServeronbehalfoftheclient,butitaddsanadditionalhopofrequestsasalltherequestsareroutedthroughPeerServerinsteadofdirectlyconnectingwiththeTransactorandthestorageengine.DatomicprovidesaseparatePeerLibraryandClientLibrary(http://docs.datomic.com/project-setup.html)forpeerandclientmodes,respectively.

DatamodelDatomicstoresdataasfactswitheachfactbeingafive-tuple(https://en.wikipedia.org/wiki/Tuple).ThesefactsarecalledDatoms.EachdatomconsistsofanEntityID,Attribute,andValuethatformthefirstthreepartsofthefive-tuple.Thefourthpartdefinesthetimestampatwhichthefactwascreatedandholdstrue.Thefifthpartconsistsofabooleanvaluethatdetermineswhetherthedefineddatomisanadditionorretractionofafact.Multipledatomsofthesameentitycanberepresentedasanentitymapwithanattributeandvalueaskey-valuepairs.TheEntityIDfortheentitymapisdefinedusingthekey:db/id:

Intheprecedingexample,therearefourattributes—order/name,:order/status,:order/rating,and:order/contactdefinedfortheorderentitywiththeID1234.

SchemaEachdatabaseinDatomichasanassociatedschemathatdefinesalltheattributesthatcanbeassociatedwiththeentities.Italsodefinesthetypeofvalueeachattributecancontain.Theattributesarethemselvestransactedasdatoms,thatis,theyarealsoconsideredasentitieswithassociatedattributesthataredefinedbyDatomic.TheattributessupportedbyDatomicforschemadefinitionsareasshowninthefollowingtable:

Attribute Type Description

:db/identNamespacedkeyword

Uniquenameofanattributeintheform<namespace>[.<nested-namespace>]/<name>,suchas:order/name.Namespacesareusefultopreventcollisionsbuttheycanbeomitted.:dbisarestrictednamespaceusedbyDatomicinternally.

:db/valueType Keyword

Definesthetypeofvalue.Supportedtypesare:

:db.type/keyword

:db.type/string

:db.type/boolean

:db.type/long

:db.type/bigint

:db.type/float

:db.type/double

:db.type/bigdec

:db.type/ref

:db.type/instant

:db.type/uuid

:db.type/uri

:db.type/bytes

:db/cardinality

Specifieswhethertheattributeissingle-valuedormulti-valued.Possiblecardinalitytypesare:

Keyword :db.cardinality/one

:db.cardinality/many

Datomicalsodefinessomeoptionalschema(http://docs.datomic.com/schema.html)attributessuchas:db/doc,:db/unique,:db/index,:db/fulltext,:db/isComponent,and:db/noHistory.The:db/ident,:db/valueType,and:db/cardinalityattributesaremandatory.

Datomicalsosupportsindexes(http://docs.datomic.com/indexes.html)thatcanbeenabledforanattributeusingthe:db/indexschemaattribute.Internally,DatomicmaintainsfourindexesthatcontaindatomsorderedbyEAVT,AEVT,AVET,andVAET,whereEisentity,Aisattribute,Visvalue,andTistransaction.

UsingDatomicDatomiccanbedownloadedfreelyfromitsGetDatomic(http://www.datomic.com/get-datomic.html)website.TostartwithDatomic,downloadtheDatomicfreeeditionthatincludesamemorydatabaseandembeddedDatalogqueryengine.ThefreeeditionisalsolimitedtotwosimultaneouspeersandembeddedstoragethatshouldbegoodenoughtotryoutDatomicfeaturesandworkwithitsdatamodel.

GettingstartedwithDatomicTosetupDatomic,downloadandextractthefreeedition'sdatomic-free-x.x.xxxx.xx.zipfile.ThefreeversionofDatomicdoesnotrequireanylicensekey.Forotherversions,registrationismandatorytoobtainalicensekeywhichmustbeaddedtotransactorpropertiesforDatomictowork.DatomicdistributioncontainstwoJARs,datomic-free-x.x.xxxx.xx.jaranddatomic-transactor-free-x.x.xxxx.xx.jar.Thedatomic-freeJARfilecontainsapeerlibraryanddatomic-transactorcontainstheimplementationofthetransactor.ThedistributionalsocontainsabinfolderthathasalltherequiredscriptstostartDatomiccomponentsasshownhere:datomic-free-0.9.5561.62├──bin├──CHANGES.md├──config├──COPYRIGHT├──datomic-free-0.9.5561.62.jar├──datomic-transactor-free-0.9.5561.62.jar├──lib├──LICENSE├──pom.xml├──README├──resources├──samples└──VERSION

5directories,8files

Tostartwiththefreeedition,createanewClojureLeiningenprojectandaddthedatomic-freedependency.Tobuilduponthepedestal-playproject,addthedependenciestotheexistingprojectconfigurationfileproject.cljofprojectpedestal-playasshownhere:(defprojectpedestal-play"0.0.1-SNAPSHOT":description"FIXME:writedescription":url"http://example.com/FIXME":license{:name"EclipsePublicLicense":url"http://www.eclipse.org/legal/epl-v10.html"}

:dependencies[[org.clojure/clojure"1.8.0"][io.pedestal/pedestal.service"0.5.3"][frankiesardo/route-swagger"0.1.4"]

;;Removethislineanduncommentoneofthenextlinesto;;useImmutantorTomcatinsteadofJetty:[io.pedestal/pedestal.jetty"0.5.3"];;[io.pedestal/pedestal.immutant"0.5.3"];;[io.pedestal/pedestal.tomcat"0.5.3"]

;;DatomicFreeEdition[com.datomic/datomic-free"0.9.5561.62"]

[ch.qos.logback/logback-classic"1.1.8":exclusions[org.slf4j/slf4j-api]][org.slf4j/jul-to-slf4j"1.7.22"][org.slf4j/jcl-over-slf4j"1.7.22"][org.slf4j/log4j-over-slf4j"1.7.22"]]:min-lein-version"2.0.0":resource-paths["config","resources"]...:main^{:skip-aottrue}pedestal-play.server)

Thedependencyofcom.datomic/datomic-freewillpulltherequireddependencyfromClojars.Datomicdistributionalsoprovidesabin/maven-installscripttoinstalltheJARshippedinthedistributioninthelocalMaven(https://maven.apache.org/)repositoryfromwhereLeiningencanpullitfortheproject.

Now,startaREPLusingleinreplorjack-inusingtheEmacsCIDERplugintostartusingDatomicAPIs.Thenamespacerequiredtoaccessthepeerlibraryisdatomic.api.IncludethenamespaceinaREPLsessionasshownhere:%leinreplnREPLserverstartedonport33835onhost127.0.0.1-nrepl://127.0.0.1:33835REPL-y0.3.7,nREPL0.2.12Clojure1.8.0JavaHotSpot(TM)64-BitServerVM1.8.0_121-b13Docs:(docfunction-name-here)(find-doc"part-of-name-here")

Source:(sourcefunction-name-here)Javadoc:(javadocjava-object-or-class-here)Exit:Control+Dor(exit)or(quit)Results:Storedinvars*1,*2,*3,anexceptionin*e

pedestal-play.server=>(require'[datomic.api:asd])nilpedestal-play.server=>

TheDatomicfreeeditioncomeswithin-memorystorage.Thedatastoredwiththein-memorystorageisonlyavailableforthelifetimeoftheapplicationprocesswhenworkingwiththeDatomicfreeedition.

Connectingtoadatabase

Toconnecttoadatabase,first,definethedatabaseURIandcreateadatabaseusingthecreate-databasefunctionofthedatomic.apinamespace.IttakesasinputadatabaseURIthatdefinesthestorageenginetobeused,thatis,memforin-memoryandthedatabasetobecreated,thatis,hhorderforHelpingHandsorders.Itreturnstrueifthedatabaseiscreatedsuccessfullyasshownhere:

pedestal-play.server>(require'[datomic.api:asd])

pedestal-play.server>(defdburi"datomic:mem://hhorder")

#'pedestal-play.server/dburi

pedestal-play.server>(d/create-databasedburi)

Oncethedatabaseiscreated,connecttoitusingtheconnectfunctionprovidedbythedatomic.apinamespace.Itreturnsadatomic.peer.LocalConnectionobjectthatcanbeusedtotransactwiththedatabase.Takealookatthefollowingexample:

pedestal-play.server>(defconn(d/connectdburi))

#'pedestal-play.server/conn

pedestal-play.server>conn

#object[datomic.peer.LocalConnection0x299180eb

"datomic.peer.LocalConnection@299180eb"]

TransactingdataDatomicneedstoknowabouttheattributestobeusedfortheentitiesinthedatabasebeforehand.Bothattributes(schema)andfacts(data)aretransactedasdatomsusingthetransactfunctionofthedatomic.apinamespace.Datomscanbetransactedindividuallyorclubbedtogetherasapartofsingletransactionbywrappingtheminavector(https://clojure.org/reference/data_structures#Vectors).Forexample,alltheattributesforhhordercanbetransactedusingasingletransactionbyspecifyingalltheentitymapstogetherwithinavectorandpassingthatvectorasanargumenttothetransactfunctionasshowninthefollowingexample:

(defresult

(d/transactconn[{:db/ident:order/name

:db/valueType:db.type/string

:db/cardinality:db.cardinality/one

:db/doc"DisplayNameofOrder"

:db/indextrue}

{:db/ident:order/status

:db/doc"OrderStatus"}

{:db/ident:order/rating

:db/valueType:db.type/long

:db/doc"Ratingfortheorder"}

{:db/ident:order/contact

:db/doc"ContactEmailAddress"}]))

#'pedestal-play.server/result

Ifthe:db/idkeyisnotdefinedasapartoftheentitymap,itisaddedbyDatomicautomatically.ThetransactfunctiontakesasparameteraDatomicconnectionandavectorofoneormoredatomstotransact.Itreturnsapromise(https://en.wikipedia.org/wiki/Futures_and_promises)thatcanbedereferencedtoseethetransacteddatoms,aswellasthebeforeandafterstateofthedatabase.Takealookatthefollowingexample:

pedestal-play.server>(pprint@result)

{:db-beforedatomic.db.Db@6c5316fa,

:db-afterdatomic.db.Db@351b51d3,

:tx-data

[#datom[1319413953431250#inst"2017-11-22T13:01:30.632-00:00"13194139534312true]

#datom[6310:order/name13194139534312true]

#datom[63402313194139534312true]

#datom[63413513194139534312true]

#datom[6362"DisplayNameofOrder"13194139534312true]

#datom[6344true13194139534312true]

#datom[6410:order/status13194139534312true]

#datom[64402313194139534312true]

#datom[64413513194139534312true]

#datom[6462"OrderStatus"13194139534312true]

#datom[6510:order/rating13194139534312true]

#datom[65402213194139534312true]

#datom[65413513194139534312true]

#datom[6562"Ratingfortheorder"13194139534312true]

#datom[6610:order/contact13194139534312true]

#datom[66402313194139534312true]

#datom[66413513194139534312true]

#datom[6662"ContactEmailAddress"13194139534312true]

#datom[0136513194139534312true]

#datom[0136413194139534312true]

#datom[0136613194139534312true]

#datom[0136313194139534312true]],

:tempids

{-922330166810959814363,

-922330166810959814264,

-922330166810959814165,

-922330166810959814066}}

Oncetheattributesaredefinedforthehhorderdatabase,theycanbeassociatedwiththeentities.Toaddaneworder,transactwiththeregisteredattributesasshowninthefollowingexample:

(deforder-result

(d/transactconn[{:db/id1

:order/name"CleaningOrder"

:order/status"Done"

:order/rating5

:order/contact"abc@hh.com"}

{:db/id2

:order/name"GardeningOrder"

:order/status"Pending"

:order/rating4

:order/contact"def@hh.com"}]))

#'pedestal-play.server/order-result

Forexample,twoorderswithIDs1and2havebeenaddedtothehhorderdatabase,whichcannowbequeriedviaits:db/idorotherdefinedattributevalues.

UsingDatalogtoquery

Datomicofferstwowaystoretrievethedatafromthedatabase—pull(http://docs.datomic.com/pull.html)andquery(http://docs.datomic.com/query.html).ThequerymethodofretrievingfactsfromDatomicdatabasesusesanextendedformofDatalog(https://en.wikipedia.org/wiki/Datalog).Toquerythedatabase,theqfunctionofdatomic.apineedstoknowthestateofthedatabasetorunthequeryon.Thecurrentstateofthedatabasecanberetrievedusingthedbfunctionofdatomic.api.Takealookatthefollowingexample:

pedestal-play.server>(d/q'[:find?e?n?c?s

:where[?e:order/rating5]

[?e:order/name?n]

[?e:order/contact?c]

[?e:order/status?s]]

(d/dbconn))

#{[1"CleaningOrder""abc@hh.com""Done"]}

EachDatomicquerymusthaveeithera:findand:whereclauseora:findand:inclausepresentasapartofthequeryconstruct.Aquery,whengivenasetofclauses,scansthroughthedatabaseforallthefactsthatsatisfythegivenclausesandreturnsalistoffacts.Datomicquerygrammar(http://docs.datomic.com/query.html#grammar)definesallthepossiblewaystoquerythedatabase.Herearesomeoftheexamplestoquerythehhorderdatabase:

;;ReturnsonlytheentityIDoftheentitiesmatchingtheclause

(d/q'[:find?e

:where[?e:order/rating5]]

(d/dbconn))

#{[1]}

;;findalltheentitieswiththethreeattributesandentityID

(d/q'[:find?e?n?c?s

:where[?e:order/name?n]

(d/dbconn))

#{[1"CleaningOrder""abc@hh.com""Done"][2"GardeningOrder""def@hh.com"

"Pending"]}

;;using'or'clause

(d/q'[:find?e?n?c?s

:where(or[?e:order/rating4][?e:order/rating5])

[?e:order/name?n]

(d/dbconn))

#{[1"CleaningOrder""abc@hh.com""Done"][2"GardeningOrder""def@hh.com"

"Pending"]}

;;usingpredicates

(d/q'[:find?e?n?c?s

:where[?e:order/rating?r]

[?e:order/name?n]

[?e:order/status?s]

[(<?r5)]]

(d/dbconn))

#{[2"GardeningOrder""def@hh.com""Pending"]}

AchievingimmutabilityAllthefactspresentinaDatomicdatabaseareimmutableandarevalidforanygiventimestamp.Forexample,thestatusofanorderwith:db/id2inthecurrentstateofthedatabaseissetasPending.Takealookatthefollowingexample:

(d/q'[:find?e?s

:where[?e:order/status?s]]

(d/dbconn))

#{[1"Done"][2"Pending"]}

Now,trytoupdatethevalueofthe:order/statusattributefororderID2toDonebytransactingwithits:db/id.Takealookatthefollowingexample:

;;updatethestatusattributeto'Done'fororderID'2'

(defstatus-result(d/transactconn[{:db/id2:order/status"Done"}]))

#'pedestal-play.server/status-result

;;querythelateststateofdatabase

(d/q'[:find?e?s:where[?e:order/status?s]](d/dbconn))

#{[2"Done"][1"Done"]}

Aftertransacting,thestatusoftheorderID2nowshowstheupdatedstatusinthecurrentstateofthedatabase.AlthoughthequeryshowsthatthestatusoforderID2isnowDone,DatomicdoesnotoverwritethevalueofthestatusfororderID2in-place.Instead,itaddsanewdatomwiththerecenttransactiontimestamp.Wheneverthequeryisexecutedwiththecurrentstateofthedatabase,thatis,usingthedbfunctionofdatomic.api,italwaysreturnsthefactswiththemostrecenttimestamp.

Toretrievethepreviousstatusoftheorder2,usethestateofthedatabasebeforethetransactionthatupdatedthestate.Thereturnvalueoftransactcontainsa:db-beforekeythatcanbeusedtorunthesamestatusqueryonthedatabasestatebeforethetransaction.Takealookatthefollowingexample:

;;querythestatusonpreviousstate

(d/q'[:find?e?s

:where[?e:order/status?s]]

(@status-result:db-before))

#{[1"Done"][2"Pending"]}

TheresultreturnsthepreviousstatusoforderID2,thatis,Pending.ImmutabilityisoneofthemostpowerfulfeaturesofaDatomicdatabaseandisveryusefulto

trackthechangesinthedatabase.

Deletingadatabase

Todeleteanexistingdatabase,usethedelete-databasefunctionofthedatomic.apinamespace.IttakesasinputthetargetdatabaseURIandreturnstrueifthedeletionsucceedsasshowninthefollowingexample:

pedestal-play.server>(d/delete-databasedburi)

DatomichasaDayofDatomic(http://www.datomic.com/training.html)seriesthatprovidesin-depthdetailsaboutDatomicdatabaseswithdetailedexamplesandtutorialstolearnfrom.

Summary

Inthischapter,welearnedaboutDatomicarchitectureandhowitisradicallydifferentfromtraditionaldatabases.Welearnedaboutitsdatamodelandhowitstoresdatoms.WealsolearnedhowtoretrievefactswithDatomicAPIsanditsDatalog-basedqueryengine.Wealsolookedatitsimmutabilityconstructsandhowtoquerydatabasesincurrentaswellashistoricalstates.

Inthenextpartofthisbook,wewillfocusontheimplementationofmicroservicesfortheHelpingHandsapplication,whichwillusePedestalasthebaseframeworktodesignAPIsandDatomicforpersistence.

BuildingMicroservicesforHelpingHands

"It'snottheideas;it'sdesign,implementationandhardworkthatmakethedifference."

-MichaelAbrash

Identifyingboundedcontextisthefirststeptowardsbuildingasuccessfulmicroservices-basedarchitecture.Designingforscaleandimplementingthemwiththerighttechnologystackisthenextandthemostcrucialstepinbuildingamicroservices-basedapplication.Thischapterbringstogetherallthedesigndecisionstakeninthefirstpartofthebook(Chapter2,MicroservicesArchitectureandChapter3,MicroservicesforHelpingHandsApplication)anddescribesthestepstoimplementthemusingthePedestalframework(Chapter6,IntroductiontoPedestal).Inthischapter,youwilllearnhowto:

ImplementHexagonaldesignformicroservicesCreatescalablemicroservicesforHelpingHandsusingPedestalImplementworkflowsformicroservicesusingthePedestalinterceptorchainImplementthelookupserviceofHelpingHandstosearchforservicesandgeneratereports

ImplementingHexagonalArchitectureHexagonalArchitecture(http://alistair.cockburn.us/Hexagonal+architecture),asshowninthefollowingdiagram,aimstodecouplethebusinesslogicfromthepersistenceandtheservicelayer.Clojureprovidestheconceptofaprotocol(https://clojure.org/reference/protocols)thatcanbeusedtodefinetheinterfaces,thatactasportsofHexagonalArchitecture.Theseportscanthenbeimplementedbytheadapters,resultinginadecoupledimplementationthatcanbeswappedbasedontherequirement.ExecutionoftheseadapterscanthenbetriggeredviaPedestalinterceptorsbasedonthebusinesslogic.

Designingtheinterceptorchainandcontext

TheinterceptorchainmustbedefinedforeachmicroserviceoftheHelpingHandsapplicationseparately.Eachinterceptorchainmayconsistofinterceptorsthatauthenticatetherequest,validatethedatamodels,andapplythebusinesslogic.InterceptorscanalsobeaddedtointeractwiththePersistencelayerandtogenerateeventsaswell.ThelistofprobableinterceptorsthatmaybeapartofHelpingHandsservicesinclude:

Auth:UsedtoauthenticateandauthorizerequestsreceivedbytheAPIendpointsexposedbythemicroservice.Validation(datamodel):Usedtovalidatetherequestparametersandmaptheexternaldatamodelwiththeinternaldatamodelasexpectedbythebusinesslogicandunderlyingpersistentstore.Businesslogic:Oneormoreinterceptorstoimplementthebusinesslogic.TheseinterceptorsprocesstherequestparametersreceivedbytheAPIendpoints.Persistence:PersistthechangesusingoneormoreadaptersthatimplementthePortProtocoldefinedforthemicroservicedatamodeltobepersisted.Events:Generateeventsasynchronouslybothforothermicroservicesaswellasformonitoringandreporting.Theseinterceptorsareaddedattheendofthechaintogeneratechangelogeventsofthepersistentstoreforothermicroservicestoconsume.

PedestalcontextisaClojuremapthatcontainsallthedetailsrelatedtotheinterceptorchain,requestparameters,headers,andmore.Thesamecontextmapcanalsobeusedtosharedatawithotherinterceptorsinthechain.Insteadofaddingakeytothecontextmapdirectly,itisrecommendedtokeepaparentkey,suchastx-data,thatcontainsamapofkeysthatarerelatedtothedatabeingprocessedbythemicroservice.Itmayalsocontainthevalidateduserdetailsforothermicroservicestoconsume.

CreatingaPedestalprojectTostartwiththeimplementation,createaprojectbythenameofhelping-handsandinitializeitwithaPedestaltemplateasdiscussedearlierinChapter6,IntroductiontoPedestal.Oncetheprojecttemplateisinitialized,updatetheproject.cljfileandotherconfigurationparametersasperthedevelopmentenvironmentsetupoftheplaygroundapplicationofChapter4,DevelopmentEnvironment.Onceconfigured,theproject.cljfileshouldcontainalltherequireddependenciesandplugins,asshownhere:

(defprojecthelping-hands"0.0.1-SNAPSHOT"

:description"HelpingHandsApplication"

:url"https://www.packtpub.com/application-development/microservices-clojure"

;;Removethislineanduncommentoneofthenextlinesto

;;useImmutantorTomcatinsteadofJetty:

[io.pedestal/pedestal.jetty"0.5.3"]

;;[io.pedestal/pedestal.immutant"0.5.3"]

;;[io.pedestal/pedestal.tomcat"0.5.3"]

;;CodeCoverage

;;Unittestdocs

;;IfyouuseHTTP/2orALPN,usethejava-agenttopullinthecorrectalpn-boot

dependency

;:java-agents[[org.mortbay.jetty.alpn/jetty-alpn-agent"2.0.5"]]

:dev{:aliases{"run-dev"["trampoline""run""-m""helping-

hands.server/run-dev"]}

:dependencies[[io.pedestal/pedestal.service-tools"0.5.3"]]

:jvm-opts["-Dconf=config/conf.edn"]}

:uberjar{:aot[helping-hands.server]}

:themes[:rdash]}}

:debug{:jvm-opts

["-server"(str"-agentlib:jdwp=transport=dt_socket,"

"server=y,address=8000,suspend=n")]}}

:main^{:skip-aottrue}helping-hands.server)

Theprojectdirectorystructureshouldcontaintherequiredfiles,asshownhere:

├──config

├──Dockerfile

├──README.md

├──src

│├──clj

││└──helping_hands

││├──core.clj

││├──persistence.clj

││├──server.clj

││└──service.clj

│└──jvm

└──test

├──clj

│└──helping_hands

│├──core_test.clj

│└──service_test.clj

└──jvm

9directories,11files

Notethetwonewsourcefiles,core.cljandpersistence.clj,thathavebeencreatedmanuallyandaddedtotheprojectstructurealongwithacore_test.cljfilefortestcases.Thisprojectactsasatemplateforeachmicroservicethatfurtherextendstheservice.cljfilewiththeimplementationofroutesanddirectmessagingendpoints.Initializationoftheapplicationhappensincore.cljalongwiththeimplementationofinterceptorsandbusinesslogic.Thepersistencelayer,alongwithitsprotocol,isdefinedinpersistence.clj.ThenextstepistostartdefiningthegenericinterceptorsforHelpingHandsmicroservices.

DefininggenericinterceptorsInadditiontothebusinesslogic,eachmicroserviceneedstoauthenticaterequests,validateincominginputparameters,mapexternaldatamodelstointernaldatamodelsforbusinesslogic,andgenerateeventsforothermicroservicesbasedontheactiontaken.AllofthesecapabilitiescanalsobeimplementedasaPedestalinterceptorforbetterflexibilityandreducedmaintenanceoverheadduetotheseparationofconcernofeachPedestalinterceptor.

InterceptorforAuthMicroservicesthatallowuserstoregisterservices,lookupservices,andcreateordersmustintegratewiththeAuthservicetomakesurethattherequestsreceivedbytheservicearegenuineandthesenderisauthorizedtoperformtherequestedtask.InsteadofembeddingAuthlogicinallthemicroservices,itisrecommendedtoseparateitoutasamicroservicewithwhichallotherservicescaninteractviadirectmessagingtoauthenticatethesenderoftherequest.

TheinteractionwiththeAuthservicecanbeembeddedwithinaPedestalinterceptorthatfrontendstheinterceptorchainforallthesecuredAPIs.ThisinterceptorshouldcapturetheauthenticationandauthorizationdetailsintherequestandsendthemtotheAuthservicetovalidate.IftheAuthservicefailstovalidatetherequest,theinterceptorshouldterminatethechainandreturnaHTTP401Unauthorizedresponse,elseitshouldaddthesenderprofiledetailstotherequestandforwardittothenextinterceptorinthechain.

Forexample,authisaninterceptorthatlooksforatokenintherequestheader.Ifitispresent,itlooksuptheuserdetailsandpopulatesthemunderthe:userkeywordofthePedestalcontextmap.Ifatokenisnotpresentintheheader,itaddsaHTTP401Unauthorizedresponsewithamessageinthebodyandterminatesthechain:

(nshelping-hands.service

(:require[cheshire.core:asjp]

[io.pedestal.http:ashttp]

[io.pedestal.interceptor.chain:aschain]

(defauth

{:name::auth

:enter

(fn[context]

(let[token(->context:request:headers(get"token"))]

(if-let[uid(and(not(nil?token))(get-uidtoken))]

(assoc-incontext[:request:tx-data:user]uid)

(chain/terminate

(assoccontext

:response{:status401

:body"Authtokennotfound"})))))

:error

(fn[contextex-info]

(assoccontext

:body(.getMessageex-info)}))})

;;Tabularroutes

(defroutes#{["/":get(conjcommon-interceptors`auth`home-page)]})

Asofnow,forsimplicity,let'sassumethattheget-uidfunctionjustreturnsaresponsewithuidasafieldthathasafixedvalue,hhuser,thatcanbepickedbyinterceptors,suchashome-page,toconstructtheresponsefurtherdowntheinterceptorchain:

(nshelping-hands.service

(defnhome-page

[request]

(ring-resp/response

(if-let[uid(->request:tx-data:user(get"uid"))]

(jp/generate-string{:msg(str"Hello"uid"!")})

(jp/generate-string{:msg(str"HelloWorld!")}))))

(defn-get-uid

"TODO:IntegratewithAuthService"

[token]

(when(and(string?token)(not(empty?token)))

;;validatetoken

{"uid""hhuser"}))

Now,trytorequesttheroute/withandwithoutatoken.ItwillreturntheHTTP200OKresponsewiththemessagecontainingthefixeduserhhuser;whereas,withoutatoken,itwillreturnaHTTP401Unauthorizedresponse.Inthelattercase,executionofthehome-pageinterceptorisskippedentirelyincludingboththe:enterand:leavefunctions:

%curl-i-H"token:1234"http://localhost:8080

HTTP/1.1200OK

{"msg":"Hellohhuser!"}

%curl-ihttp://localhost:8080

HTTP/1.1401Unauthorized

Authtokennotfound

TheAuthserviceandrelatedinterceptorareexplainedindetailinPart-4ofthisbookunderChapter11,DeployingandMonitoringSecuredMicroservices.

InterceptorforthedatamodelOncetherequestisauthenticatedandauthorizedbytheAuthinterceptor,itshouldbevalidatedagainstthedatamodelofthemicroservice.ThisvalidationlogiccanalsobeimplementedasaPedestalinterceptorthatcanreviewtheinputparametersspecifiedwiththerequestandmakesurethatitconformstothedatamodel.Iftherequestparametersarevalid,thisinterceptorshouldforwardtherequesttothenextinterceptorinthechainwiththerequireddetails,elseitshouldterminatethechainandreturnaHTTP400BadRequestresponse.

Thedatamodelvalidationinterceptorcanalsoaddadditionaldetailstotherequest.Forexample,iftherequestcontainsjusttheserviceID,thisinterceptorcanpullintheservicedetailsandserviceproviderdetailsandaddittothelistoftheparametersfortherestofthechaintoprocess.ItcanalsovalidatethepresenceandabsenceofthespecifiedserviceID.

Forexample,tocreateanorder,arequestmustcontaintheserviceIDforwhichtheorderistobecreated.Thedatamodelinterceptor,inthiscase,canfirstvalidatethattherequestcontainstheserviceIDandifitdoes,itcanvalidatetheserviceIDwiththeexternalmicroservicethatmanagestheServicedatabaseandpullinadditionaldetails:(defn-get-service-details"TODO:GettheservicedetailsfromexternalAPI"[sid]{"sid"sid,"name""HouseCleaning"})

(defdata-validate{:name::validate

:enter(fn[context](let[sid(->context:request:form-params:sid)](if-let[service(and(not(nil?sid))(get-service-detailssid))](assoc-incontext[:request:tx-data:service]service)(chain/terminate(assoccontext

:response{:status400:body"InvalidServiceID"})))))

:error(fn[contextex-info](assoccontext:response{:status500:body(.getMessageex-info)}))})

;;Tabularroutes(defroutes#{["/":post(conjcommon-interceptors`auth`data-validate`home-page)]})

Now,thePOST/endpointusesbothauthanddata-validateinterceptorstomakesurethatbeforetherequesthitsthehome-pageinterceptor,ithasbeenauthenticatedandcontainstherequiredservicedetails.ThebehavioroftheendpointisshownhereusingcURLrequests:%curl-i-XPOSThttp://localhost:8080HTTP/1.1401Unauthorized...

Authtokennotfound

%curl-i-XPOST-H"token:1234"http://localhost:8080HTTP/1.1400BadRequest...

InvalidServiceID

%curl-i-XPOST-H"token:1234"-d"sid=1"http://localhost:8080HTTP/1.1200OK...

{"msg":"Hellohhuser!"}

ValidationinterceptorscoveredinthischapteruseClojurecoreconditionalfunctionstoachievethedesiredvalidation.WiththeClojure-1.9.0release,itisrecommendedtomovetoClojurespec(ht

tps://clojure.org/guides/spec)forallsuchvalidations.

InterceptorforeventsHelpingHandsmicroservices,suchasServiceProviderandServiceConsumer,generatechangelogeventsfortheLookupservicetopickupandupdatethelocaldatabaseforServiceConsumerstolookup.ThisrequirescertaineventstobegeneratedeverytimeachangeispushedtothelocaldatabaseoftheServiceProviderandServiceConsumerservice.TheseeventscanbedeterminedbyaninterceptorthatispresentinthechainrightbeforethePersistenceinterceptorthatpersiststhechangestothedatabase.Apartfromchange-logevents,eachmicroservicemayalsopublisheventsformonitoringandreporting.Alltheseeventscanbegeneratedbythesameinterceptor.

Theeventsinterceptorgeneratesalltheeventsrequiredbyothermicroservicesandtomonitortheentireapplication.Chapter10,EventDrivenPatternsforMicroservices,andChapter11,DeployingandMonitoringSecuredMicroservices,talkaboutusingexternalframeworkssuchasKafkatopublishandconsumeeventsforcoordinationandbuildinganevent-drivendatapipelineformicroservices.

CreatingamicroserviceforServiceConsumer

TheServiceConsumermicroserviceexposesAPIsforenduserstoregisterasaconsumeroftheHelpingHandsapplication.Tolookupregisteredservicesandplaceanorder,theuseroftheapplicationmustberegisteredasaConsumer.AspertheworkflowofServiceConsumerdefinedinChapter3,MicroservicesforHelpingHandsApplication,itrequiresthefollowingAPIstocreatenewconsumers,getconsumerprofiles,andupdateconsumerdetails:

URI Description

/consumers/:id/?

flds=name,address

Getsthedetailsoftheconsumerwiththespecified:idifthe:idisspecified,elseitgetsthedetailsoftheauthenticatedconsumer.Optionally,itacceptsaCSVoffieldstobereturnedintheresponse.

/consumers/:id CreatesanewconsumerwiththespecifiedID.POST/consumers CreatesanewconsumerandreturnstheID.DELETE

/consumers/:id DeletestheconsumerwiththespecifiedID.

AddingroutesPedestalroutesareaddedforeachoftheidentifiedAPIs.TheinterceptorchainforeachAPIconsistsofAuth,Validation,BusinessLogic,andEventinterceptorsthatauthorizetheincomingrequests,validatetherequiredparameters,applythebusinesslogic,andgeneratetherelevantevents,respectively,forothermicroservicesandmonitoringframeworks.

Sincetheinterceptorchainofeachrouteendswithacommongen-eventsinterceptor,a:route-namemustbedefinedforeachroutetomakesurethattheyreceiveauniquename.If:route-nameisnotspecified,PedestalwilltrytoassignthenameofthelastinterceptortoeachrouteandwillfailwiththeRoutenamesarenotuniqueexception.Addtheroutesfortheconsumerserviceintheservice.cljfile,asshowninthefollowingcodesnippet:(nshelping-hands.consumer.service(:require[helping-hands.consumer.core:ascore][io.pedestal.http:ashttp][io.pedestal.http.route:asroute][io.pedestal.http.body-params:asbody-params]))

;;Tabularroutes(defroutes#{["/consumers/:id":get(conjcommon-interceptorsàuth`core/validate-id`core/get-consumer`gen-events):route-name:consumer-get]["/consumers/:id":put(conjcommon-interceptorsàuth`core/validate-id`core/upsert-consumer`gen-events):route-name:consumer-put]["/consumers":post(conjcommon-interceptorsàuth`core/validate`core/create-consumer`gen-events):route-name:consumer-post]

["/consumers/:id":delete(conjcommon-interceptors`auth`core/validate-id`core/delete-consumer`gen-events):route-name:consumer-delete]})

Chapter11,DeployingandMonitoringSecuredMicroservices,discussestheimportanceofeventsgeneratedbythemicroservicesforreal-timemonitoringandgeneratingalerts.

DefiningtheDatomicschema

TheServiceConsumermicroserviceusesDatomicasthelocaldatabasetostoretheconsumerdetails.Theschemafortheconsumerdatabaseconsistsofthefollowingattributes:

:db/ident :db/valueType :db/cardinality :db/index :db/fulltext

:consumer/id :db.type/string :db.cardinality/one true false

:consumer/name :db.type/string :db.cardinality/one true true

:consumer/address :db.type/string :db.cardinality/one true true

:consumer/mobile :db.type/string :db.cardinality/one false -

:consumer/email :db.type/string :db.cardinality/one true -

:consumer/geo :db.type/string :db.cardinality/one false -

CreatingapersistenceadapterPersistenceprotocolfortheConsumerserviceconsistsofupsert,entity,anddeletefunctionsthatareimplementedbyeachadapterofthisport.AdaptersimplementingthepersistenceprotocolarethenusedwithinthePedestalinterceptortocreate,update,query,anddeleteConsumerdetails.Theimplementationoftheprotocolandcorrespondingrecordisshowninthefollowingcodesnippetthatmustbeaddedtothepersistence.cljsourcefile:

(nshelping-hands.consumer.persistence

"PersistencePortandAdapterforConsumerService"

(:require[datomic.api:asd]))

;;--------------------------------------------------

;;ConsumerPersistencePortforAdapterstoPlug-in

;;--------------------------------------------------

(defprotocolConsumerDB

"Abstractionforconsumerdatabase"

(upsert[thisidnameaddressmobileemailgeo]

"Adds/Updatesaconsumerentity")

(entity[thisidflds]

"Getsthespecifiedconsumerwithallorrequestedfields")

(delete[thisid]

"Deletesthespecifiedconsumerentity"))

;;--------------------------------------------------

;;DatomicAdapterImplementationforConsumerPort

;;--------------------------------------------------

(defn-get-entity-id

[connid]

(->(d/q'[:find?e

:in$?id

:where[?e:consumer/id?id]](d/dbconn)(strid))

ffirst))

(defn-get-entity

[connid]

(let[eid(get-entity-idconnid)]

(->>(d/entity(d/dbconn)eid)seq(into{}))))

(defrecordConsumerDBDatomic[conn]

ConsumerDB

(d/transactconn

(vector(into{}(filter(compsome?val)

{:db/idid

:consumer/idid

:consumer/namename

:consumer/addressaddress

:consumer/mobilemobile

:consumer/emailemail

:consumer/geogeo})))))

(entity[thisidflds]

(when-let[consumer(get-entityconnid)]

(if(empty?flds)

consumer

(select-keysconsumer(mapkeywordflds)))))

(delete[thisid]

(when-let[eid(get-entity-idconnid)]

(d/transactconn[[:db.fn/retractEntityeid]]))))

Forexample,theConsumerDBprotocoldefinestheportforthePersistencelayeroftheConsumerservicethatisimplementedbytheConsumerDBDatomicrecord(https://clojure.github.io/clojure/clojure.core-api.html#clojure.core/defrecord)thatactsasanadaptertomanagetheconsumerdatabasewithintheDatomicdatabase.Thehelping-hands.consumer.persistencenamespacealsoprovidesacreate-consumer-databaseutilityfunctiontoinitializethedatabaseandupdatetheschemaifthedatabaseiscreatedforthefirsttime,asshownhere:

(defncreate-consumer-database

"Createsaconsumerdatabaseandreturnstheconnection"

;;createandconnecttothedatabase

(let[dburi(str"datomic:mem://"d)

db(d/create-databasedburi)

conn(d/connectdburi)]

;;transactschemaifdatabasewascreated

(whendb

(d/transactconn

[{:db/ident:consumer/id

:db/doc"UniqueConsumerID"

:db/unique:db.unique/identity

:db/indextrue}

{:db/ident:consumer/name

:db/doc"DisplayNamefortheConsumer"

:db/indextrue

:db/fulltexttrue}

{:db/ident:consumer/address

:db/doc"ConsumerAddress"

:db/indextrue

:db/fulltexttrue}

{:db/ident:consumer/mobile

:db/doc"ConsumerMobileNumber"

:db/indexfalse}

{:db/ident:consumer/email

:db/doc"ConsumerEmailAddress"

:db/indextrue}

{:db/ident:consumer/geo

:db/doc"Latitude,LongitudeCSV"

:db/indexfalse}]))

(ConsumerDBDatomic.conn)))

Thecreate-consumer-databasefunctionacceptsadatabasenameasinput,suchasconsumer,andcreatesaDatomicdatabaseURI.ThedatabaseURIprefixdatomic:mem://signifiesanin-memorydatabaseofDatomicthatisusedinthisexample.Thecreate-consumer-databasefunctiontriestocreateadatabaseandifitsucceeds,ittransactstheschemafortheconsumerdatabase.ItcreatesaconnectiontothedatabaseandreturnsitwrappedasaConsumerDBDatomicrecordthatcanthenbeusedtoupsertconsumers,retrievethem,anddeletethembasedontherequirement.

CreatinginterceptorsThehelping-hands.consumer.corenamespacedefinesalltheinterceptorsthatareusedfortheroutesoftheConsumermicroservice.Validationinterceptorsvalidatetheinputparametersandmakesurethatalltherequiredfieldsarepresentintherequest.Theyalsopreparetheinputparametersforthebusinesslogicinterceptorsbycreatinga:tx-datakeywithalltherequiredparameters.

Validationinterceptorsmayalsoperformtransformations,suchaschangingtheCSVofthefldsparametertoavectoroffieldnamesthatisrequiredasaninputfortheentityfunctiondefinedbyConsumerDBprotocol.TheyterminatetherequestwithaHTTP400BadRequeststatusiftherequiredparametersarenotpresent.TheyalsodefinetheerrorhandlertocatchtheexceptioninthechainandreportthemtotheclientwithaHTTP500InternalServerErrorstatus.Theimplementationofthevalidationinterceptorisshowninthefollowingcodesnippet:

(nshelping-hands.consumer.core

"InitializesHelpingHandsConsumerService"

[clojure.string:ass]

[helping-hands.consumer.persistence:asp]

[io.pedestal.interceptor.chain:aschain])

(:import[java.ioIOException]

[java.utilUUID]))

;;delaythecheckfordatabaseandconnection

;;tillthefirstrequesttoaccess@consumerdb

(def^:privateconsumerdb

(delay(p/create-consumer-database"consumer")))

;;--------------------------------

;;ValidationInterceptors

;;--------------------------------

(defn-prepare-valid-context

"Appliesvalidationlogicandreturnstheresultingcontext"

[context]

(let[params(merge(->context:request:form-params)

(->context:request:query-params)

(->context:request:path-params))]

(if(and(not(empty?params))

;;anyoneofmobile,emailoraddressispresent

(or(params:id)(params:mobile)(params:email)(params:address)))

(let[flds(if-let[fl(:fldsparams)]

(maps/trim(s/splitfl#","))

(vector))

params(assocparams:fldsflds)]

(assoccontext:tx-dataparams))

(chain/terminate

(assoccontext

:body(str"OneofAddress,emailand"

"mobileismandatory")})))))

(defvalidate-id

{:name::validate-id

:enter

(fn[context]

(if-let[id(or(->context:request:form-params:id)

(->context:request:query-params:id)

(->context:request:path-params:id))]

;;validateandreturnacontextwithtx-data

;;orterminatedinterceptorchain

(prepare-valid-contextcontext)

(chain/terminate

(assoccontext

:body"InvalidConsumerID"}))))

:error

(fn[contextex-info]

(assoccontext

(defvalidate

{:name::validate

:enter

(fn[context]

(if-let[params(->context:request:form-params)]

(chain/terminate

(assoccontext

:body"Invalidparameters"}))))

:error

(fn[contextex-info]

(assoccontext

Thehelping-hands.consumer.corenamespacealsodefinestheinterceptorstogetconsumerdetails,performupsertoperations,createaconsumerwithageneratedID,anddeletetheconsumer,asshownhere:

;;--------------------------------

;;BusinessLogicInterceptors

;;--------------------------------

(defget-consumer

{:name::consumer-get

:enter

(fn[context]

(let[tx-data(:tx-datacontext)

entity(.entity@consumerdb(:idtx-data)(:fldstx-data))]

(if(empty?entity)

(assoccontext:response{:status404:body"Nosuchconsumer"})

(assoccontext:response{:status200

:body(jp/generate-stringentity)}))))

:error

(fn[contextex-info]

(assoccontext

(defupsert-consumer

{:name::consumer-upsert

:enter

(fn[context]

id(:idtx-data)

db(.upsert@consumerdbid(:nametx-data)

(:addresstx-data)(:mobiletx-data)

(:emailtx-data)(:geotx-data))]

(if(nil?@db)

(throw(IOException.

(str"Upsertfailedforconsumer:"id)))

(assoccontext

:body(jp/generate-string

(.entity@consumerdbid[]))}))))

:error

(fn[contextex-info]

(assoccontext

(defcreate-consumer

{:name::consumer-create

:enter

(fn[context]

;;generatearandomIDifitisnotspecified

id(str(UUID/randomUUID))

tx-data(if(:idtx-data)tx-data(assoctx-data:idid))

;;createconsumer

db(.upsert@consumerdbid(:nametx-data)

(if(nil?@db)

(throw(IOException.

(str"Upsertfailedforconsumer:"id)))

(assoccontext

(.entity@consumerdbid[]))}))))

:error

(fn[contextex-info]

(assoccontext

(defdelete-consumer

{:name::consumer-delete

:enter

(fn[context]

db(.delete@consumerdb(:idtx-data))]

(if(nil?db)

(assoccontext:response{:status404:body"Nosuchconsumer"})

(assoccontext:response{:status200:body"Success"}))))

:error

(fn[contextex-info]

(assoccontext

TestingroutesTheroutesdefinedfortheConsumerserviceallowuserstocreateanewconsumer,queryforitsproperties,anddeleteit.Asofnow,fortheAuthinterceptor,assumethat123isavalidtokenthatissentintheheaderofeachrequest.Tostartwith,createaconsumerusingthePUT/consumers/:idroutewiththeIDsetas1,asshownhere:%curl-i-H"token:123"-XPUT-d"name=ConsumerA"http://localhost:8080/consumers/1HTTP/1.1200OK...

{"consumer/id":"1","consumer/name":"ConsumerA"}

Itcreatesanewconsumerandreturnstheconsumerentity.Toaddanewfield,usethesameAPIwiththesameconsumerIDandspecifythenewfields.Itwilldoanupsertoperationontheentityandaddthenewfields,asshownhere:

curl-i-H"token:123"-XPUT-d"email=user1@helpinghands.com"

http://localhost:8080/consumers/1

HTTP/1.1200OK

{"consumer/id":"1","consumer/name":"ConsumerA","consumer/email":"user1@helpinghands.com"}

Togettheentity,usetheGET/consumers/:idroute.IftheconsumerIDisnotfound,itreturnsaHTTP404NotFoundresponse:

%curl-i-H"token:123""http://localhost:8080/consumers/1"

HTTP/1.1200OK

{"consumer/id":"1","consumer/name":"ConsumerA","consumer/email":"user1@helpinghands.com"}

%curl-i-H"token:123""http://localhost:8080/consumers/1?

flds=consumer/name,consumer/email"

HTTP/1.1200OK

{"consumer/name":"ConsumerA","consumer/email":"user1@helpinghands.com"}

%curl-i-H"token:123""http://localhost:8080/consumers/2"

HTTP/1.1404NotFound

Nosuchconsumer

ThePOST/consumersroutecanalsobeusedtocreateaconsumerwitharandomID:

%curl-i-H"token:123"-XPOST-d"name=ConsumerX&email=userx@helpinghands.com"

http://localhost:8080/consumers

HTTP/1.1200OK

{"consumer/id":"b46cdbbb-06a1-4375-9287-

2230e3ad8ded","consumer/name":"ConsumerX","consumer/email":"userx@helpinghands.com"}

%curl-i-H"token:123""http://localhost:8080/consumers/b46cdbbb-06a1-4375-9287-

2230e3ad8ded"

HTTP/1.1200OK

{"consumer/id":"b46cdbbb-06a1-4375-9287-

2230e3ad8ded","consumer/name":"ConsumerX","consumer/email":"userx@helpinghands.com"}

Todeleteaconsumer,usetheDELETE/consumers/:idroutewithanexistingconsumerID:

%curl-i-H"token:123"-XDELETE"http://localhost:8080/consumers/b46cdbbb-06a1-

4375-9287-2230e3ad8ded"

HTTP/1.1200OK

Success

%curl-i-H"token:123""http://localhost:8080/consumers/b46cdbbb-06a1-4375-9287-

2230e3ad8ded"

HTTP/1.1404NotFound

Nosuchconsumer

CreatingamicroserviceforServiceProvider

TheServiceProvidermicroserviceexposesAPIsforenduserstoregisterasaserviceprovideroftheHelpingHandsapplication.ServiceproviderscanregisteroneormoreserviceswiththeHelpingHandsapplicationthattheyarewillingtofulfillifanorderisplacedagainstit.AspertheworkflowofServiceProviderdefinedinChapter3,MicroservicesforHelpingHandsApplication,thefollowingAPIsarerequiredtocreatenewproviders,getproviderprofiles,andupdateproviderdetails:

URI Description

/providers/:id/?

flds=name,mobile

Getsthedetailsoftheserviceproviderwiththespecified:idifthe:idisspecified,elseitgetsthedetailsoftheauthenticateduserregisteredasaserviceprovider.Optionally,itacceptsaCSVoffieldstobereturnedintheresponse.

PUT/providers/:id CreatesanewproviderwiththespecifiedID.POST/providers CreatesanewproviderandreturnstheID.PUT

/providers/:id/rate Addstothelatestratingsfortheprovider.DELETE

/providers/:id DeletestheproviderwiththespecifiedID.

Addingroutes

RoutesfortheServiceProvidermicroserviceareverysimilartotheConsumerservice.TheProviderserviceadditionallyprovidesadedicatedroutetoregisteraratingfortheProvider,asshownhere:

(defroutes#{["/providers/:id"

:get(conjcommon-interceptors`auth`core/validate-id

`core/get-provider`gen-events)

:route-name:provider-get]

["/providers/:id"

:put(conjcommon-interceptors`auth`core/validate-id

`core/upsert-provider`gen-events)

:route-name:provider-put]

["/providers/:id/rate"

`core/upsert-provider`gen-events)

:route-name:provider-rate]

["/providers"

:post(conjcommon-interceptors`auth`core/validate

`core/create-provider`gen-events)

:route-name:provider-post]

["/providers/:id"

:delete(conjcommon-interceptors`auth`core/validate-id

`core/delete-provider`gen-events)

:route-name:provider-delete]})

TheServiceProvidermicroserviceusesDatomicasthelocaldatabasetostoretheproviderdetails.Theschemafortheproviderdatabaseconsistsofthefollowingattributes:

:provider/id :db.type/string :db.cardinality/one true false

:provider/name :db.type/string :db.cardinality/one true true

:provider/mobile :db.type/string :db.cardinality/one false -

:provider/since :db.type/long :db.cardinality/one false -

:provider/rating :db.type/float :db.cardinality/many false -

Persistenceprotocolconsistsofupsert,entity,anddeletefunctions,similartotheConsumerservice,thataredefinedinthepersistence.cljsourcefile,asfollows:

(nshelping-hands.provider.persistence

"PersistencePortandAdapterforProviderService"

;;--------------------------------------------------

;;ProviderPersistencePortforAdapterstoPlug-in

;;--------------------------------------------------

(defprotocolProviderDB

"Abstractionforproviderdatabase"

(upsert[thisidnamemobilesincerating]

"Adds/Updatesaproviderentity")

(entity[thisidflds]

"Getsthespecifiedproviderwithallorrequestedfields")

(delete[thisid]

"Deletesthespecifiedproviderentity"))

;;--------------------------------------------------

;;DatomicAdapterImplementationforProviderPort

;;--------------------------------------------------

(defn-get-entity-id

[connid]

(->(d/q'[:find?e

:in$?id

:where[?e:provider/id?id]](d/dbconn)(strid))

ffirst))

(defn-get-entity

[connid]

(defrecordProviderDBDatomic[conn]

ProviderDB

(upsert[thisidnamemobilesincerating]

(d/transactconn

{:db/idid

:provider/idid

:provider/namename

:provider/mobilemobile

:provider/sincesince

:provider/ratingrating})))))

(entity[thisidflds]

(when-let[provider(get-entityconnid)]

(if(empty?flds)

provider

(select-keysprovider(mapkeywordflds)))))

(delete[thisid]

Thehelping-hands.provider.persistencenamespacealsoprovidesacreate-provider-databaseutilityfunctiontoinitializethedatabaseandupdatetheschemaifthedatabaseiscreatedforthefirsttime:

(defncreate-provider-database

"Createsaproviderdatabaseandreturnstheconnection"

(whendb

(d/transactconn

[{:db/ident:provider/id

:db/doc"UniqueProviderID"

:db/indextrue}

{:db/ident:provider/name

:db/doc"DisplayNamefortheProvider"

:db/indextrue

:db/fulltexttrue}

{:db/ident:provider/mobile

:db/doc"ProviderMobileNumber"

:db/indexfalse}

{:db/ident:provider/since

:db/doc"ProviderActiveSinceEPOCHtime"

:db/indexfalse}

{:db/ident:provider/rating

:db/valueType:db.type/float

:db/cardinality:db.cardinality/many

:db/doc"Listofratings"

:db/indexfalse}]))

(ProviderDBDatomic.conn)))

Thehelping-hands.provider.corenamespacedefinesalltheinterceptorsthatareusedfortheroutesoftheProvidermicroservice.Validationinterceptorsvalidatetheinputparametersandmakesurethatalltherequiredfieldsarepresentintherequest.ValidationinterceptorsforProviderroutesworkexactlyinthesamewayasthatoftheConsumermicroserviceroutes.Additionally,theyvalidatethedatatypeoftheratingandsincefieldstomakesurethatthebusinessmodelgetsthevaluesintheformatexpectedbytheDatomicdatabase.Theimplementationofvalidationinterceptorsisshowninthefollowingcodesnippet:

(nshelping-hands.provider.core

"InitializesHelpingHandsProviderService"

[helping-hands.provider.persistence:asp]

[java.utilUUID]))

;;tillthefirstrequesttoaccess@providerdb

(def^:privateproviderdb

(delay(p/create-provider-database"provider")))

;;--------------------------------

(defn-validate-rating-ts

"Validatestheratingandtimestamp"

[context]

(let[rating(->context:request:form-params:rating)

since_ts(->context:request:form-params:since)]

(let[context(if(not(nil?rating))

(assoc-incontext[:request:form-params:rating]

(Float/parseFloatrating))context)

context(if(not(nil?since_ts))

(assoc-incontext[:request:form-params:since]

(Long/parseLongsince_ts))context)]

context)

(catchExceptionenil))))

[context]

(->context:request:path-params))

ctx(validate-rating-tscontext)

params(if(not(nil?ctx))

(assocparams

:rating(->ctx:request:form-params:rating)

:since(->ctx:request:form-params:since)))]

(not(nil?ctx))

;;anyoneofidormobile

(or(params:id)(params:mobile)))

(vector))

(chain/terminate

(assoccontext

:body(str"ID,mobileismandatory"

"andrating,sincemustbeanumber")})))))

(defvalidate-id

{:name::validate-id

:enter

(fn[context]

(chain/terminate

(assoccontext

:body"InvalidProviderID"}))))

:error

(fn[contextex-info]

(assoccontext

(defvalidate

{:name::validate

:enter

(fn[context]

(chain/terminate

(assoccontext

:error

(fn[contextex-info]

(assoccontext

Otherinterceptors,suchasget-provider,upsert-provider,create-provider,anddelete-provider,aresimilartointerceptorsdefinedfortheroutesoftheConsumerservice,asshownhere:

;;--------------------------------

(defget-provider

{:name::provider-get

:enter

(fn[context]

entity(.entity@providerdb(:idtx-data)(:fldstx-data))]

(if(empty?entity)

(assoccontext:response{:status404:body"Nosuchprovider"})

:error

(fn[contextex-info]

(assoccontext

(defupsert-provider

{:name::provider-upsert

:enter

(fn[context]

id(:idtx-data)

db(.upsert@providerdbid(:nametx-data)

(:mobiletx-data)(:sincetx-data)

(:ratingtx-data))]

(if(nil?@db)

(throw(IOException.

(str"Upsertfailedforprovider:"id)))

(assoccontext

(.entity@providerdbid[]))}))))

:error

(fn[contextex-info]

(assoccontext

(defcreate-provider

{:name::provider-create

:enter

(fn[context]

;;generatearandomIDifitisnotspecified

id(str(UUID/randomUUID))

tx-data(if(:idtx-data)tx-data(assoctx-data:idid))

;;createprovider

db(.upsert@providerdbid(:nametx-data)

(:mobiletx-data)(:sincetx-data)

(:ratingtx-data))]

(if(nil?@db)

(throw(IOException.

(str"Upsertfailedforprovider:"id)))

(assoccontext

(.entity@providerdbid[]))}))))

:error

(fn[contextex-info]

(assoccontext

(defdelete-provider

{:name::provider-delete

:enter

(fn[context]

db(.delete@providerdb(:idtx-data))]

(if(nil?db)

(assoccontext:response{:status404:body"Nosuchprovider"})

(assoccontext:response{:status200:body"Success"}))))

:error

(fn[contextex-info]

(assoccontext

TestingroutesTheroutesdefinedfortheProviderserviceallowuserstocreateanewprovider,queryforitsproperties,anddeleteit.Asofnow,fortheAuthinterceptor,assumethat123isavalidtokenthatissentintheheaderofeachrequest.Tostartwith,createaproviderusingthePUT/providers/:idroutewithIDsetas1:%curl-i-H"token:123"-XPUT-d"name=ProviderA"http://localhost:8080/providers/1HTTP/1.1200OK...

{"provider/id":"1","provider/name":"ProviderA"}

Toaddarating,usethePUT/providers/:id/rateroutewiththesameIDasthatofProviderA:

%curl-i-H"token:123"-XPUT-d"rating=5.0"http://localhost:8080/providers/1/rate

HTTP/1.1200OK

{"provider/id":"1","provider/name":"ProviderA","provider/rating":[5.0]}

Togettheprovider,usetheGET/providers/:idroute.IftheproviderIDisnotfound,itreturnsaHTTP404NotFoundresponse:

%curl-i-H"token:123""http://localhost:8080/providers/1"

HTTP/1.1200OK

{"provider/id":"1","provider/name":"ProviderA","provider/rating":[5.0]}

HTTP/1.1404NotFound

Nosuchprovider

Todeleteaprovider,usetheDELETE/providers/:idroutewithanexistingproviderID:

%curl-i-H"token:123"-XDELETE"http://localhost:8080/providers/1"

HTTP/1.1200OK

Success

HTTP/1.1404NotFound

Nosuchprovider

CreatingamicroserviceforServices

TheServicemicroservicemanagesthelistofservicesofferedbytheserviceprovidersviatheHelpingHandsapplication.ItexposesAPIsforserviceproviderstoregisterservicesthatareofferedbythem.AspertheworkflowofService,definedinChapter3,MicroservicesforHelpingHandsApplication,thefollowingAPIsarerequiredtocreateanewservice,getservicedetails,andupdateservicedetails.EachservicemustalreadyhavetheserviceproviderregisteredwiththeHelpingHandsapplicationviatheServiceProvidermicroservice.Sinceonlyanexistingserviceprovidercanregisteraservice,theserviceproviderIDisretrievedbytheAuthtokenreceivedwiththerequestcallingtheServiceAPItocreateanewservice:

URI Description

/services/:id/?

flds=name,mobile

Getsthedetailsoftheservicewiththespecified:idifthe:idisspecified.Optionally,itacceptsaCSVoffieldstobereturnedintheresponse.

PUT/services/:id CreatesanewservicewiththespecifiedID.POST/services CreatesanewserviceandreturnstheID.PUT

/services/:id/rate Addstothelatestratingsfortheservice.DELETE

/services/:id DeletestheservicewiththespecifiedID.

Addingroutes

RoutesforServiceareverysimilartotheProviderservice.Italsodefinesroutestocreate,modify,rate,anddeleteservices.TheroutetocreateaserviceexpectsaproviderIDasamandatoryparametertomakesurethateachserviceisassociatedwithaprovideratthetimeofcreation.Also,anychangeinproviderIDusingthePUT/services/:idrouteisvalidatedagainsttheProviderservicetomakesurethatthespecifiedproviderexistsandisregisteredwiththeHelpingHandsapplication.Theroutesareshowninthefollowingcodesnippet:

;;Tabularroutes

(defroutes#{["/services/:id"

:get(conjcommon-interceptors`auth`core/validate-id-get

`core/get-service`gen-events)

:route-name:service-get]

["/services/:id"

`core/upsert-service`gen-events)

:route-name:service-put]

["/services/:id/rate"

`core/upsert-service`gen-events)

:route-name:service-rate]

["/services"

`core/create-service`gen-events)

:route-name:service-post]

["/services/:id"

:delete(conjcommon-interceptors`auth`core/validate-id-get

`core/delete-service`gen-events)

:route-name:service-delete]})

DefiningaDatomicschemaTheServicemicroserviceusesDatomicasthelocaldatabasetostoretheservicedetails.ServicemaintainsaproviderIDaswell.Althoughintheschemaitisdefinedastype:db.type/string;ifacommondatabaseisusedfortheproviderandservicethenitisrecommendedtodefinetheproviderIDoftype:db.type/reftomakebetteruseofDatomicentityreferences.Theschemafortheservicedatabaseconsistsofthefollowingattributes:

:service/id :db.type/string :db.cardinality/one true false

:service/type :db.type/string :db.cardinality/one true true

:service/provider :db.type/string :db.cardinality/one false -

:service/area :db.type/string :db.cardinality/many true true

:service/cost :db.type/float :db.cardinality/one false -

:service/rating :db.type/float :db.cardinality/many false -

:service/status :db.type/string :db.cardinality/one false -

ThereisalsoaGeoLocationfieldmentionedinthedatamodeloftheServicedatabaseinChapter3,MicroservicesforHelpingHandsApplication.ThisfieldisaderivedfieldthatiscomputedandstoredbytheLookupserviceinsteadofbeingstoredintheServicedatabase.Thisfieldisusedonlyforgeolocation-basedqueriesthattheLookupservicewillbehandling.

PersistenceprotocolServiceDBconsistsofupsert,entity,anddeletefunctions,similartotheProviderservice,thataredefinedinthepersistence.cljsourcefile,asshownhere:

(nshelping-hands.service.persistence

"PersistencePortandAdapterforService"

;;--------------------------------------------------

;;ServicePersistencePortforAdapterstoPlug-in

;;--------------------------------------------------

(defprotocolServiceDB

"Abstractionforservicedatabase"

(upsert[thisidtypeproviderareacostratingstatus]

"Adds/Updatesaserviceentity")

(entity[thisidflds]

"Getsthespecifiedservicewithallorrequestedfields")

(delete[thisid]

"Deletesthespecifiedserviceentity"))

;;--------------------------------------------------

;;DatomicAdapterImplementationforServicePort

;;--------------------------------------------------

(defn-get-entity-id

[connid]

(->(d/q'[:find?e

:in$?id

:where[?e:service/id?id]](d/dbconn)(strid))

ffirst))

(defn-get-entity

[connid]

(defrecordServiceDBDatomic[conn]

ServiceDB

(upsert[thisidtypeproviderareacostratingstatus]

(d/transactconn

{:db/idid

:service/idid

:service/typetype

:service/providerprovider

:service/areaarea

:service/costcost

:service/ratingrating

:service/statusstatus})))))

(entity[thisidflds]

(when-let[service(get-entityconnid)]

(if(empty?flds)

service

(select-keysservice(mapkeywordflds)))))

(delete[thisid]

Thehelping-hands.service.persistencenamespacealsoprovidesacreate-service-databaseutilityfunctiontoinitializethedatabaseandupdatetheschemaifthedatabaseiscreatedforthefirsttime,asshownhere:

(defncreate-service-database

"Createsaservicedatabaseandreturnstheconnection"

(whendb

(d/transactconn

[{:db/ident:service/id

:db/doc"UniqueServiceID"

:db/indextrue}

{:db/ident:service/type

:db/doc"TypeofService"

:db/indextrue

:db/fulltexttrue}

{:db/ident:service/provider

:db/doc"AssociatedServiceProviderID"

:db/indexfalse}

{:db/ident:service/area

:db/doc"ServiceAreas/Locality"

:db/indextrue

:db/fulltexttrue}

{:db/ident:service/cost

:db/doc"HourlyCost"

:db/indexfalse}

{:db/ident:service/rating

:db/indexfalse}

{:db/ident:service/status

:db/doc"StatusofService(A/NA/D)"

:db/indexfalse}]))

(ServiceDBDatomic.conn)))

CreatinginterceptorsThehelping-hands.service.corenamespacedefinesalltheinterceptorsthatareusedfortheroutesoftheServicemicroservice.SincebusinessmodelinterceptorsofServiceareexactlythesameasthoseoftheProviderserviceanddependonlyontheValidationinterceptors,let'sfocusonlyontheinterceptorsthatvalidatetheinputparametersforServiceroutes.Theimplementationofthevalidationinterceptorisshowninthefollowingcodesnippet:

(nshelping-hands.service.core

"InitializesHelpingHandsServiceService"

[helping-hands.service.persistence:asp]

[java.utilUUID]))

;;tillthefirstrequesttoaccess@servicedb

(def^:privateservicedb

(delay(p/create-service-database"service")))

;;--------------------------------

(defn-validate-rating-cost

"Validatestheratingandcost"

[context]

(let[rating(->context:request:form-params:rating)

cost(->context:request:form-params:cost)]

(let[context(if(not(nil?rating))

(assoc-incontext[:request:form-params:rating]

(Float/parseFloatrating))context)

context(if(not(nil?cost))

(assoc-incontext[:request:form-params:cost]

(Float/parseFloatcost))context)]

context)

(catchExceptionenil))))

[context]

ctx(validate-rating-costcontext)

(assocparams

:cost(->ctx:request:form-params:cost)))]

(not(nil?ctx))

(params:id)(params:type)(params:provider)

(params:area)(params:cost)

(contains?#{"A""NA""D"}(params:type))

(provider-exists?(params:provider)))

(vector))

(chain/terminate

(assoccontext

:body(str"ID,type,provider,areaandcostismandatory"

"andrating,costmustbeanumberwithtype"

"havingoneofvaluesA,NAorD")})))))

(defvalidate-id

{:name::validate-id

:enter

(fn[context]

(chain/terminate

(assoccontext

:body"InvalidServiceID"}))))

:error

(fn[contextex-info]

(assoccontext

(defvalidate-id-get

{:name::validate-id-get

:enter

(fn[context]

(params:id))

(vector))

(chain/terminate

(assoccontext

(chain/terminate

(assoccontext

:error

(fn[contextex-info]

(assoccontext

(defvalidate

{:name::validate

:enter

(fn[context]

(chain/terminate

(assoccontext

:error

(fn[contextex-info]

(assoccontext

ForService,thevalidationrulealsoincludesvalidatingagivenproviderIDagainstanexternalProviderServicetomakesurethattheprovideroftheserviceisalreadyregistered.Todoso,thevalidationinterceptorofServicemakesanexternalAPIcallasynchronouslyviatheprovider-exists?functionandpassesPedestalcontexttobusinesslogicinterceptorsonlywhenitfindsthattheproviderIDisvalid.

clj-http(https://github.com/dakrone/clj-http)isaClojurelibrarythatiswidelyusedtocreateHTTPclientstomakeAPIcallstoexternalservices.TomakeaGETcalltoanexternalservicelikethatofGET/providers/:id,seeclj-httpGET(https://github.com/dakrone/clj-http#get).

Testingroutes

TheroutesdefinedforServiceallowuserstocreateanewservice,queryforitsproperties,anddeleteit.Asofnow,forAuthinterceptor,assumethat123isavalidtokenthatissentintheheaderofeachrequest.HereareasampleofcURLrequeststocreate,query,anddeleteaserviceofferedbytheHelpingHandsapplication:

;;Createanewservicewithrequiredparameters

%curl-i-H"token:123"-XPUT-d"type=A&provider=1&area=bangalore&cost=250"

http://localhost:8080/services/1

HTTP/1.1200OK

{"service/id":"1","service/type":"A","service/provider":"1","service/area":

["bangalore"],"service/cost":250.0}

;;GetservicepropertiesbyID

%curl-i-H"token:123"http://localhost:8080/services/1

HTTP/1.1200OK

{"service/id":"1","service/type":"A","service/provider":"1","service/area":

["bangalore"],"service/cost":250.0}

;;Deletetheservice

%curl-i-H"token:123"-XDELETEhttp://localhost:8080/services/1

HTTP/1.1200OK

Success

;;Validateservicenolongerexists

%curl-i-H"token:123"http://localhost:8080/services/1

HTTP/1.1404NotFound

Nosuchservice

CreatingamicroserviceforOrder

TheOrderServicereceivestherequestfromServiceConsumerstocreateaneworderfortheserviceprovidedbyaparticularserviceprovider.ItexposesthefollowingAPIsforconsumerstocreateaneworder,getorderdetails,andgetthelistofordersplacedbythem.ItalsoallowstheconsumerstoratetheOrderbasedonthequalityofservicereceived.TocreateanewOrder,APIsexpectaserviceIDandproviderIDtobespecifiedalongwiththerequireddetailssuchastimeslot,andmore.TheconsumerIDispickedfromtheAuthtokenthatisreceivedasapartofrequestheaders.TheIDsspecifiedforServiceandServiceProvidermustalreadyberegisteredwiththeHelpingHandsapplicationviatheServiceandServiceProvidermicroservices.CreationofanOrderalsomakessurethattherequestedserviceisofferedwithinthevicinityoftheconsumerbasedonthegeolocationoftheconsumerandservicebeingrequested:

URI DescriptionGET/orders/?

flds=cost,status Getsalltheordersplacedbytheauthenticatedconsumer.

/orders/:id/?

flds=name,mobile

Getsthedetailsoftheorderwiththespecified:idandplacedbytheauthenticatedconsumer.Optionally,itacceptsaCSVoffieldstobereturnedintheresponse.

PUT/orders/:idCreatesaneworderwiththespecifiedIDfortheauthenticatedconsumer.

POST/ordersCreatesaneworderfortheauthenticatedconsumerandreturnstheID.

/orders/:id/rate

Addstothelatestratingsfortheorder.ConsumerIDoftheordermustmatchtheauthenticatedconsumerID.

DELETE

/orders/:id

DeletestheorderwiththespecifiedIDfortheauthenticatedconsumer.ConsumerIDoftheordermustmatchtheauthenticatedconsumerID.

Addingroutes

TheOrderServiceallowsconsumerstocreateneworders,modifythem,rate,anddeletethem.ItalsoallowsanauthenticatedusertogetalltheordersplacedbytheauthenticateduserID.Theroutesrequiredforthisserviceareshowninthefollowingcodesnippet:

;;Tabularroutes

(defroutes#{["/orders/:id"

:get(conjcommon-interceptors`auth`core/validate-id-get

`core/get-order`gen-events)

:route-name:order-get]

["/orders"

:get(conjcommon-interceptors`auth`core/validate-all-orders

`core/get-all-orders`gen-events)

:route-name:order-get-all]

["/orders/:id"

`core/upsert-order`gen-events)

:route-name:order-put]

["/orders/:id/rate"

`core/upsert-order`gen-events)

:route-name:order-rate]

["/orders"

`core/create-order`gen-events)

:route-name:order-post]

["/orders/:id"

:delete(conjcommon-interceptors`auth`core/validate-id-get

`core/delete-order`gen-events)

:route-name:order-delete]})

TheOrdermicroserviceusesDatomicasthelocaldatabasetostoretheorderdetails.Theschemafortheorderdatabaseconsistsofthefollowingattributes:

:db/ident :db/valueType :db/cardinality :db/index :db/fulltext :db/unique

:order/id :db.type/string :db.cardinality/one true false :db.unique/identity

:order/service :db.type/string :db.cardinality/one false - -

:order/provider :db.type/string :db.cardinality/one false - -

:order/consumer :db.type/string :db.cardinality/one false - -

:order/cost :db.type/float :db.cardinality/one false - -

:order/start :db.type/long :db.cardinality/one false - -

:order/end :db.type/long :db.cardinality/one false - -

:order/rating :db.type/float :db.cardinality/many false - -

:order/status :db.type/string :db.cardinality/one false - -

PersistenceprotocolOrderDBconsistsofupsert,entity,anddeletefunctions,similartotheProviderservice.Additionally,itdefinesanordersfunctionthatcanlistalltheordersofthegivenauthenticatedconsumerID,asshownhere:

(nshelping-hands.order.persistence

"PersistencePortandAdapterforOrder"

;;--------------------------------------------------

;;OrderPersistencePortforAdapterstoPlug-in

;;--------------------------------------------------

(defprotocolOrderDB

"Abstractionfororderdatabase"

(upsert[thisidserviceproviderconsumer

coststartendratingstatus]

"Adds/Updatesanorderentity")

(entity[thisidflds]

"Getsthespecifiedorderwithallorrequestedfields")

(orders[thisuidflds]

"Getsalltheordersoftheauthenticateduserwithallorrequestedfields")

(delete[thisid]

"Deletesthespecifiedorderentity"))

;;--------------------------------------------------

;;DatomicAdapterImplementationforOrderPort

;;--------------------------------------------------

(defn-get-entity-id

[connid]

(->(d/q'[:find?e

:in$?id

:where[?e:order/id?id]](d/dbconn)(strid))

ffirst))

(defn-get-entity

[connid]

(defn-get-entity-uid

[connuid]

(->>(d/q'[:find?e

:in$?id

:where[?e:order/consumer?id]](d/dbconn)(struid))

(into[])flatten))

(defn-get-all-entities

[connuid]

(let[eids(get-entity-uidconnuid)]

(map#(->>(d/entity(d/dbconn)%)seq(into{}))eids)))

(defrecordOrderDBDatomic[conn]

OrderDB

(upsert[thisidserviceproviderconsumer

coststartendratingstatus]

(d/transactconn

{:db/idid

:order/idid

:order/serviceservice

:order/providerprovider

:order/consumerconsumer

:order/costcost

:order/startstart

:order/endend

:order/ratingrating

:order/statusstatus})))))

(entity[thisidflds]

(when-let[order(get-entityconnid)]

(if(empty?flds)

(select-keysorder(mapkeywordflds)))))

(orders[thisuidflds]

(when-let[orders(get-all-entitiesconnuid)]

(if(empty?flds)

orders

(map#(select-keys%(mapkeywordflds))orders))))

(delete[thisid]

Theget-all-entitiesfunctionqueriestheDatomicdatabaseforalltheordersthathave:order/consumersettothegivenconsumerIDthatisprovidedasaparametertotheendpointthatrequestsalltheordersfortheconsumer.Italsoallowstopickonlythespecifiedfieldsacrosstheorders.Thehelping-hands.order.persistencenamespacealsoprovidesacreate-order-databaseutilityfunctiontoinitializethedatabaseandupdatetheschemaifthedatabaseiscreatedforthefirsttime,asshownhere:

(defncreate-order-database

"Createsaorderdatabaseandreturnstheconnection"

(whendb

(d/transactconn

[{:db/ident:order/id

:db/doc"UniqueOrderID"

:db/indextrue}

{:db/ident:order/service

:db/doc"AssociatedServiceID"

:db/indexfalse}

{:db/ident:order/provider

:db/doc"AssociatedServiceProviderID"

:db/indexfalse}

{:db/ident:order/consumer

:db/doc"AssociatedConsumerID"}

{:db/ident:order/cost

:db/doc"HourlyCost"

:db/indexfalse}

{:db/ident:order/start

:db/doc"StartTime(EPOCH)"

:db/indexfalse}

{:db/ident:order/end

:db/doc"EndTime(EPOCH)"

:db/indexfalse}

{:db/ident:order/rating

:db/indexfalse}

{:db/ident:order/status

:db/doc"StatusofOrder(O/I/D/C)"

:db/indexfalse}]))

(OrderDBDatomic.conn)))

CreatinginterceptorsThehelping-hands.order.corenamespacedefinesalltheinterceptorsthatareusedfortheroutesoftheOrdermicroservice.TheAuthinterceptoristhegenericinterceptorthatreadsthetokenandupdatestheuserIDfield:uidfortheOrderroutestogetalltheordersforanauthenticateduser.Forsimplicityoftheimplementation,theAuthinterceptorassumesthatthetokenpassedintheheaderissettotheconsumerID.

ThevalidationinterceptorforOrderroutesvalidatesbothserviceIDandtheproviderIDoftheordertomakesurethatbothproviderandserviceareregisteredwiththeHelpingHandsapplicationandthesameproviderprovidesthespecifiedservice.Theservice-exists?,provider-exists?,andconsumer-exists?functionsvalidatetheservice,provider,andconsumer,respectively.Additionally,thevalidationinterceptorchecksfortherightvalueofstatusandthevalueofrating,cost,start,andendtobeoftypenumber,asshowninthefollowingcodesnippet.TheimplementationofthesefunctionsaresameasthatoftheConsumer,Order,andServicemicroservicesimplementationsexplainedearlier:

[context]

ctx(validate-rating-cost-tscontext)

(assocparams

:cost(->ctx:request:form-params:cost)

:start(->ctx:request:form-params:start)

:end(->ctx:request:form-params:end)))]

(not(nil?ctx))

(params:id)(params:service)(params:provider)

(params:consumer)(params:cost)(params:status)

(contains?#{"O""I""D""C"}(params:status))

(service-exists?(params:service))

(provider-exists?(params:provider))

(consumer-exists?(params:consumer)))

(vector))

(chain/terminate

(assoccontext

:body(str"ID,service,provider,consumer,"

"costandstatusismandatory.start/end,"

"ratingandcostmustbeanumberwithstatus"

"havingoneofvaluesO,I,DorC")})))))

(defvalidate-id

{:name::validate-id

:enter

(fn[context]

(chain/terminate

(assoccontext

:body"InvalidOrderID"}))))

:error

(fn[contextex-info]

(assoccontext

(defvalidate-id-get

{:name::validate-id-get

:enter

(fn[context]

(params:id))

(vector))

(chain/terminate

(assoccontext

(chain/terminate

(assoccontext

:error

(fn[contextex-info]

(assoccontext

(defvalidate-all-orders

{:name::validate-all-orders

:enter

(fn[context]

(if-let[params(->context:tx-data)]

;;GetuserIDfromauthuid

(assoc-incontext[:tx-data:flds]

(if-let[fl(->context:request:query-params:flds)]

(vector)))

(chain/terminate

(assoccontext

:error

(fn[contextex-info]

(assoccontext

Thevalidate-id-getinterceptorisusedfortheGET/orders/:idrequestsandvalidatesonlytheorderIDandtheorderfieldsparameter.Similarly,thevalidate-all-ordersinterceptorisusedwiththeGET/ordersroutetogetalltheordersoftheauthenticatedconsumer.TheimplementationofinterceptorsforthebusinesslogicoftheOrderserviceissimilartothatofthepreviousimplementationofConsumer,Provider,andService.Additionally,theOrderservicedefinesinterceptorstogetallordersoftheauthenticatedconsumerthatusetheordersfunctionoftheOrderDBprotocol,asshownhere:

(nshelping-hands.order.core

"InitializesHelpingHandsOrderService"

[helping-hands.order.persistence:asp]

[java.utilUUID]))

;;tillthefirstrequesttoaccess@orderdb

(def^:privateorderdb

(delay(p/create-order-database"order")))

(defget-all-orders

{:name::order-get-all

:enter

(fn[context]

entity(.orders@orderdb(:uidtx-data)(:fldstx-data))]

(if(empty?entity)

(assoccontext:response{:status404:body"Nosuchorders"})

:error

(fn[contextex-info]

(assoccontext

Testingroutes

TotesttheroutesoftheOrderservice,createoneormoreordersbythesameconsumerIDandtrytoqueryalltheordersforthesameconsumerID.Forsimplicity,theroutesassumethevalueofthetokenspecifiedintheheaderastheconsumerIDfortheGET/ordersroute.HereisalistofcURLrequeststodemonstratetheprocessofcreating,querying,anddeletingtheorders:

;;AddanorderforConsumerwithID1

%curl-i-H"token:1"-XPUT-d"service=1&provider=1&consumer=1&cost=500&status=O"

http://localhost:8080/orders/1

HTTP/1.1200OK

{"order/id":"1","order/service":"1","order/provider":"1","order/consumer":"1","order/cost":500.0,"order/status":"O"}

;;AddanotherorderforConsumerwithID1

%curl-i-H"token:1"-XPUT-d"service=2&provider=2&consumer=1&cost=250&status=O"

HTTP/1.1200OK

{"order/id":"2","order/service":"2","order/provider":"2","order/consumer":"1","order/cost":250.0,"order/status":"O"}

;;AddanorderforConsumerwithID2

%curl-i-H"token:2"-XPUT-d"service=1&provider=1&consumer=2&cost=250&status=I"

HTTP/1.1200OK

{"order/id":"3","order/service":"1","order/provider":"1","order/consumer":"2","order/cost":250.0,"order/status":"I"}

;;GetallordersofconsumerwithID1

%curl-i-H"token:1"http://localhost:8080/orders

HTTP/1.1200OK

[{"order/id":"1","order/service":"1","order/provider":"1","order/consumer":"1","order/cost":500.0,"order/status":"O"},

{"order/id":"2","order/service":"2","order/provider":"2","order/consumer":"1","order/cost":250.0,"order/status":"O"}]

;;GetallordersofconsumerwithID2

HTTP/1.1200OK

[{"order/id":"3","order/service":"1","order/provider":"1","order/consumer":"2","order/cost":250.0,"order/status":"I"}]

;;GetallordersofconsumerwithID1withspecificfields

%curl-i-H"token:1""http://localhost:8080/orders?flds=order/service,order/status"

[{"order/service":"1","order/status":"O"},{"order/service":"2","order/status":"O"}]

;;DeleteorderwithID2

%curl-i-H"token:123"-XDELETEhttp://localhost:8080/orders/2

HTTP/1.1200OK

Success

;;MakesurethatOrderwithID2nolongerexists

curl-i-H"token:123"-XDELETEhttp://localhost:8080/orders/2

HTTP/1.1404NotFound

Nosuchorder

;;CheckordersforconsumerwithID1doesnotlistOrderwithID2now

curl-i-H"token:1""http://localhost:8080/orders?flds=order/service,order/status"

HTTP/1.1200OK

[{"order/service":"1","order/status":"O"}]%

;;DeletetheorderwithID3thatwastheonlyorderforconsumerwithID2

%curl-i-H"token:123"-XDELETEhttp://localhost:8080/orders/3

HTTP/1.1200OK

Success

;;MakesuretherearenootherordersleftforconsumerwithID2

HTTP/1.1404NotFound

Nosuchorders

CreatingamicroserviceforLookupTheLookupserviceisusedtosearchforservicesbytype,geolocation,andavailability.ItsubscribestotheeventsgeneratedbytheConsumer,Provider,Service,andOrdermicroservicesasanObserverandkeepsadenormalizeddatasetthatisfastertoqueryforrequiredServices.TheLookupservicealsoaddslongitudeandlatitudefortheServicesandConsumersthatisderivedfromtheiraddressandserviceareaorlocality.Togetthelongitudeandlatitudefromtheservicearea,itdependsonanexternalAPI.TheLookupserviceprovidesthefollowingAPIsforconsumerstosearchforaserviceandalsofiltertheservicesbytype,ratings,andproviders:

URI Description

GET/lookup/?q=queryFiltersalltheservicesbasedonthespecifiedquery.

GET/lookup/?q=query&type=typeFiltersalltheservicesofagiventypebasedonthespecifiedquery.

GET/lookup/geo/?

tl=40.73,-74.1&br=40.01,-71.12

Looksuptheservicebythegivenlatitude-longitudesetfortop-left(tl)andbottom-right(br)boundingboxpointsorwithinaradiusofapre-defineddistancefromtheconsumerlocation.

GET/validate/:service/?

tl=40.73,-74.1&br=40.01,-71.12

Validatesifthespecified:serviceIDiswithintheboundingboxregionspecifiedbythetop-left(tl)andbottomright(br)boundingboxpointsorwithinaradiusofapre-defineddistancefromtheconsumerlocation.

GET/status

Getsthecurrentstatusoftheeventsincludingthenumberofordersplacedovertime,trendingordertypes,trendinglocation,toppreferredserviceproviders,andmore.

GET/status/consumer/:idGetsthecurrentstatusoftheeventswithkeydatapointsforthespecifiedconsumerID.

GET/status/provider/:id GetsthecurrentstatusoftheeventswithkeydatapointsforthespecifiedproviderID.

GET/status/service/:typeGetsthecurrentstatusoftheeventswithkeydatapointsforthespecifiedtypeofservices.

TheLookupservicedoesnotprovideanyAPIstoupdatethedataasitmaintainsonlyadenormalizedviewofeventsreceivedfromConsumer,Provider,Service,andOrderservices.Ifthereareanychangesrequiredinthedata,theymustbedonewiththeAPIsexposedbytheircorrespondingmicroservicesthatmanageit.TheLookupservicewillthenreceivethechangeeventsandreflectthechangesinitslocaldatabase.

DefiningtheElasticsearchindex

TheLookupmicroserviceusesElasticsearchasthelocaldatabasetostorealltheeventsthataredenormalizedacrossdatabasesmaintainedbyConsumer,Provider,ServiceandOrdermicroservices.Elasticsearchprovidessub-secondresponseforthesearchqueriesandalsosupportsgeolocation-basedqueries.ItalsosupportsaggregationandanalyticsoutoftheboxtogeneratevariousreportsfortheHelpingHandsapplication.HereisamappingfortheElasticsearchindexthatisrequiredfortheLookupservice:

Field Type Mapping Analyzer Descriptionoid string - keyword OrderIDcid string - keyword UsedforconsumerID

pid string - keywordUsedforserviceproviderID

sid string - keyword UsedforserviceIDstype string - keyword Usedforservicetype

locality string - standardUsedtostorethelocalityoftheorder

geo string(lat,long) geo_point - Usedforgeolocation-basedqueries

cost float - - Usedtostorethetotalcostoftheorder

ts_startdate

(yyyyMMDD'T'HH:mm:ss) - - Startdate-timeoftheorder

ts_enddate

(yyyyMMDD'T'HH:mm:ss) - - Enddate-timeoftheorder

rating float

Ratinggivenforthe

- - order

status string - keywordStatusoftheorder,oneofO,I,D,C

TheeventsarestoredintoElasticsearchwiththeprecedingindexschemainLookupindex.TheeventsarereceivedbytheLookupServiceviatheKafkatopicthatitsubscribesto.ThedetailsofKafka,theconceptoftopicsandhowtosubscribetoitforevents,havebeenexplainedinChapter10,Event-DrivenPatternsforMicroservices.Inthischapter,thefocuswillbeonhowtoquerytheElasticsearchindexthathasthefieldsaspertheschemaoftheLookupindex.

CreatingqueryinterceptorsAlltheroutesoftheLookupserviceareoftype:getastheyareusedonlytoquerythedata.SincethedataresideswithElasticsearch,therequestsneedstobemappedtorelevantElasticsearchqueriestogettherequiredresult.InterceptorstoqueryElasticsearchdatareadtherequiredfieldsfromthe:tx-datafieldofthePedestalcontextassetbythevalidationinterceptors.TheimplementationofthevalidatorthatwrapsthequeriestoElasticsearchisthesameasthatofotherservicesexplainedearlier.

ElasticsearchdefinesaQueryDSL(https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html)thatmustbeusedtoquerythedataagainsttheElasticsearchindex.TheQueryDSLisbasedonJSONandinvolvescreatingaJSONstructureofqueryclausesthatarewrappedwithinaqueryorafiltercontext(https://www.elastic.co/guide/en/elasticsearch/reference/current/query-filter-context.html).Queryclausesmaybeoftypeleafqueryclausesorcompoundqueryclauses.

Leafqueryclauseslookforaparticularvalueinaparticularfield,asshowninthefollowingcodesnippetofanElasticsearchquery.Forexample,queryingforalltheordersthathavestatusopen;thatis,O.Thetermquery(https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-term-query.html),usedtogettheopenorders,queriesonlyforexactmatches.Sincethestatusfieldofthelookupschemahasthemappingdefinedasthatoftypekeyword,itsupportsexactmatchesviatheKeywordAnalyzer(https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-keyword-analyzer.html)ofElasticsearch:{"query":{"term":{"status":"O"}}}

Compoundqueryclausesareusedtowraponeormorequeryclausethatmaybeoftypeleaforcompound,asshowninthefollowingcodesnippet.Forexample,queryingforalltheordersthathavestatusdone;thatis,D,andhavearatingof

fourormorethanthat.ElasticsearchprovidesaBoolQuery(https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html)tocreateBooleancombinationsofoneormorequeriestoformacompoundclause.Forstatus,itusesatermqueryandforratingitusesarangequery(https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html)ofElasticsearchthatarecompoundedbyamustclausethatmakesboththeconditionstobesatisfiedforanordertobereturnedasaresponseofthisquery:{"query":{"bool":{"must":[{"term":{"status":"D"}},{"range":{"rating":{"gte":4}}}]}}}

ElastischisaClojureclientthatcanbeusedtocreateElasticsearchindexesandquerythem.ElasticsearchalsoprovidestheJavaRESTClient(https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/index.html)andJavaAPIs(https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/index.html)thatcanalsobeusedwithClojure.

Usinggeoqueries

Geoqueries(https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-queries.html)ofElasticsearchallowfindingtherecordsusingaboundingbox,distancefromagivengeopoint,ortherecordslyingwithinthepolygonmadeupofgeopointsspecifiedasaboundingregion.TheHelpingHandsapplicationrequiresgeoqueriesfortwoofitsroutes,GET/lookup/geoandGET/validate/:service.

Thefirstrouteallowsconsumerstolookupavailableserviceswithinthespecifiedboundingboxofalatitudeandlongitudepairthatcanbeselectedviaamapbydrawingarectangle.Alternatively,consumerscanalsolookupaservicewithin,say,a5kmradiusoftheirlocationspecifiedbytheirgeolocation.Similarly,thesecondroutevalidatesthattheselectedserviceID:servicelieswithintheallowedlimitsofaconsumergeolocationboundary.Boththeroutesinternallyuseeitheraboundingboxquery(https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-geo-bounding-box-query.html)oradistancequery(https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-geo-distance-query.html)ofElasticsearch,asshownhere:

"query":{

"bool":{

"must":{

"match_all":{}

"filter":{

"geo_bounding_box":{

"geo.location":{

"top_left":{

"lat":13.17,

"lon":77.38

"bottom_right":{

"lat":12.73,

"lon":77.88

{"query":{"bool":{"must":{"match_all":{}},"filter":{"geo_distance":{

"distance":"5km","geo.location":{"lat":12.97,"lon":77.59}}}}}}

Geoqueriesbasedonbounding-boxanddistancerequiresfieldstohavethegeo_point(https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html)mappingdefined.Tolookupbydefiningashape,fieldsmusthavegeo_shape(https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-shape.html)mappingdefined.

GettingstatuswithaggregationqueriesElasticsearchalsosupportsaggregations(https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html)thatprovideaggregatedresultsbasedonthegivenqueries.AggregationsareusefulfortheHelpingHandsapplicationtogetthestatusoftheordersandgenerateanalyticsreportsthatcanbeusedtounderstandtheusageoftheapplication.

Forexample,totakealookattheordersreceivedeverymonth,adatehistogramaggregationcanbecreatedonthets_startfield:

"aggs":{

"monthly_orders":{

"date_histogram":{

"field":"ts_start",

"interval":"month"

Similarly,togetthestatsonratingsreceivedacrossorderssofar,statsaggregationcanbeusedontheratingfield,asshowninthefollowingexample.Statsreturnsthecount,min,max,avg,andsumofthevaluesofthespecifiedfield;thatis,ratinginthiscase:

"aggs":{

"rating_stats":{

"stats":{

"field":"rating"

Toknowthecurrentstatusoftheordersacrosstheapplication,termsaggregation(https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html)canbeusedonthestatusfield,asshowninthefollowingexample.Termsaggregationreturnsthecurrentcountofalltheorderstatusesacrossthesystemtogetareportofhowmanyordersareopen,inprogress,done,orclosed:

"aggs":{

"order_status":{

"terms":{

"field":"status"

ElasticsearchisalsousedtobuildamonitoringsystemfortheHelpingHandsapplication.SuchamonitoringsystemreliesheavilyonaggregationqueriesofElasticsearchtobuildadashboardtounderstandtheruntimestateofthesystem.ThemonitoringsystemfortheHelpingHandsapplicationisdescribedinPartIV,Chapter11,DeployingandMonitoringSecuredMicroservices.

Creatingamicroserviceforalerts

TheAlertserviceisusedtosendemailalertsandSMS.Alertscanbegeneratedatvariouslevelsbyothermicroservices.Forexample,successfulcreationofaconsumer,provider,service,oranordermayrequireanemailtobesenttotherelevantstakeholders.Similarly,alertsmayberequiredwheneverthereisachangeinthestatusoftheorderoraratingisreceived.TheAlertservicedoesnotmaintainalocaldatabase,itjustgenerateseventsforeachsuccessfulalertsentthatcanbetrackedformonitoringpurposes.ThefollowingtableliststheendpointsfortheAlertservice:

URI Params Description

/alerts/emailto,cc,subject,body

Sendsanalertviaemailtooneormorerecipients.

POST/alerts/sms to,body SendsanalertviaSMStooneormorerecipients.

Addingroutes

Mostly,theAlertservicewilllistenforeventsasanObserverandwillnotreceiverequeststosendalertsviaroutes.Ifitisrequiredtosendalertssynchronously,the/alerts/emailand/alerts/smsroutescanbeused,asdefinedinthefollowingcodesnippet:

(defroutes#{["/alerts/email"

`core/send-email`gen-events)

:route-name:alert-email]

["/alerts/sms"

`core/send-sms`gen-events)

:route-name:alert-sms]})

CreatinganemailinterceptorusingPostalPostal(https://github.com/drewr/postal)isaClojurelibrarythatallowssendingemail.ItrequiresSMTPconnectiondetailsandthemessageasamapcontainingtherequireddetailsofto,from,cc,subject,andbodytosendasanemail.PostalcanbeusedwithinthePedestalinterceptortosendanemailiftherequiredfieldsarevalidatedandpresentinthecontext,asshownhere:

(nshelping-hands.alert.core

"InitializesHelpingHandsAlertService"

[postal.core:aspostal]

[helping-hands.alert.persistence:asp]

[java.utilUUID]))

;;--------------------------------

[context]

(let[params(->context:request:form-params)]

(not(empty?(:toparams)))

(not(empty?(:bodyparams))))

(let[to-val(maps/trim(s/split(:toparams)#","))]

(assoccontext:tx-data(assocparams:toto-val)))

(chain/terminate

(assoccontext

:body"Bothtoandbodyarerequired"})))))

(defvalidate

{:name::validate

:enter

(fn[context]

(chain/terminate

(assoccontext

:error

(fn[contextex-info]

(assoccontext

;;--------------------------------

(defsend-email

{:name::send-email

:enter

(fn[context]

msg(into{}(filter(compsome?val)

{:from"admin@helpinghands.com"

:to(:totx-data)

:cc(:cctx-data)

:subject(:subjecttx-data)

:body(:bodytx-data)}))

result(postal/send-message

{:host"smtp.gmail.com"

:port465

:ssltrue

:user"admin@helpinghands.com"

:pass"resetme"}

;;sendemail

(assoccontext:response

{:status200

:body(jp/generate-stringresult)})))

:error

(fn[contextex-info]

(assoccontext

PostaldependsontheunderlyingSMTPservertoacceptthecredentialsandallowthird-partyclientstosendemails.ServicessuchasGmailmayrestrictthird-partyclientstouseusernameandpassword.

Tosendalerts,itisrecommendedtouseexternalservicessuchasAmazonSES,AmazonSNS,andmoreastheyarereliabletouseandfollowapay-per-usemodel.

SummaryInthischapter,wefocusedonthestep-by-stepimplementationofHelpingHandsmicroservicesusingthePedestalframework.WelearnedhowtoimplementHexagonalArchitectureusingClojureprotocols(https://clojure.org/reference/protocols)andPedestalinterceptors.WealsoimplementedtherequiredmicroservicesfortheHelpingHandsapplicationinPedestal.Inthenextchapter,wewilllearnhowtoconfigureourmicroservicesandmaintaintheruntimestateoftheapplicationthatincludesconnectionwithpersistentstorageandmessagequeuestostoredataandsendevents,respectively.

ConfiguringMicroservices

""Ican'tchangethedirectionofthewind,butIcanadjustmysailstoalwaysreachmydestination.""

-JimmyDean

Microservicesmustbeconfigurabletoadapttotheenvironmentinwhichtheyaredeployed.Theymustsupportexternalconfigurationparametersthatcanbespecifiedatruntimetoconfigurethemaspertheenvironmentinwhichtheyaredeployed.Oncetheconfigurationparametersaredefined,amicroservicemustbeabletoeffectivelypropagatetheconfigurationsacrossitsmodules.Theseconfigurationparametersmightthenbeusedtoinitializedatabaseconnectionsormaintainotherapplicationstatesthatmustbesharedacrossthemodulesofamicroservice.Allthemodulesmusthaveaccesstotheexactsamestateatruntime.Thischapterprovideseffectivesolutionstobuildsuchconfigurableservicesthatcanmanagetheirruntimestateseffectively.Inthischapter,youwilllearnhowtodothefollowing:

ApplyconfigurationprinciplestobuildhighlyconfigurableservicesUseOmniconfforconfigurationValidateconfigurationatstartupandregisteritforuseatruntimeUseamountlibrarytocomposeandmanageapplicationstates

ConfigurationprinciplesAlltheapplicationparametersthatarerelatedtotheenvironmentandaffecttheapplicationstatemustbemadeconfigurable.Forexample,theconnectionstringforaDatomicdatabasethatisusedbyHelpingHandsservicescanbemadeconfigurablesothatitcanbeupdatedexternallytopointtoaspecificinstanceofDatomicinproduction.Configurationparametersalsomakeitpossibletotesttheapplicationinvariousenvironments.Forexample,ifaDatomicconnectionstringismadeconfigurableforHelpingHandsservices,itcanbeusedtotesttheserviceswithanin-memoryinstanceofDatomicinlocaldevelopmentenvironmentsandlaterchangedtopointtotheproductioninstanceofDatomiconcetheyaredeployed.

Definingconfigurationparameters

Applicationsmustsupportmultiplewaysofdefiningtheconfigurationparameters.Usingcommand-lineargumentsisoneofthemostcommonwaysofspecifyingtheconfigurationparametersfortheapplication.Environmentvariablesandexternalconfigurationfilescanalsobespecifiedforapplicationstopicktheconfigurationparametersatruntime.SinceClojureusesJVMasitsruntimeengine,applicationsbuiltinClojurecanacceptconfigurationparametersasJavapropertiesaswell.

Applicationsthatacceptconfigurationparametersfrommultiplesourcesmustdecideonthepreferenceofvarioussources.Forexample,configurationparametersspecifiedatthecommandlineasJavapropertiescanoverwritethevaluesdefinedbytheenvironmentvariablesthatinturncanoverridethedefaultconfigurationdefinedintheconfigurationfile.

UsingconfigurationparametersOneoptiontoprovideaccesstoconfigurationparametersistoloadthematstartupandpassthemasargumentstothefunctions.Inthiscase,everytimeanewconfigurationparameterisadded,itmayresultinthechangeofthefunctionsignaturethatcanaffectallthefunctionsdependentonit.

Configurationparametersmustnotbetieddirectlytotheargumentsofthefunctionbecauseconfigurationparametersthatareloadedatstartuptimemaynotchangethroughoutthelifecycleoftheservice.Instead,configurationparameterscanbereadonceatstartupandkeptasimmutableconstantsthatcanbedirectlyaccessedbyallthefunctionsthatrequireoneormoreconfigurationparameter.

Readingtheconfigurationparametersfromvarioussourcesandmakingthemaccessibledoesnotguaranteetheconfigurationwillbecorrectunlesstheyareusedbytheapplication.Forexample,iftheapplicationneedsaportnumberasaconfigurationparameter,itmustbeverifiedassoonasitisreadandmustbeashortpositivenumberwithamaximumvalueof65535.Ifthesechecksarenotperformedatthetimetheconfigurationsareread,theyaregoingtoresultinruntimeexceptionslaterduringtheapplicationlifecycle.Detectingsuchconfigurationissuesatalaterpointintimeiscostlyastheconfigurationneedstobeupdatedandtheapplicationneedstoberedeployedtopickupontheupdatedconfiguration.

UsingOmniconfforconfigurationOmniconf(https://github.com/grammarly/omniconf)isanopensourceconfigurationlibraryforClojureprojectsthatcanbeusedtoconfiguremicroservicesoftheHelpingHandsapplication(refertoChapter3,MicroservicesforHelpingHandsApplication,andChapter8,BuildingMicroservicesforHelpingHands).Omniconfnotonlyallowstheapplicationtodefinethepreferencewithrespecttovariousconfigurationsourcesbutalsotoverifythematstartup.Internally,itkeepsalltheconfigurationparametersstoredasanimmutableconstantthatcanbeaccessedasaregularClojuredatastructure.

Omniconfisoneoftheoptionsforconfigurationmanagement.Libraries,suchasEnviron(https://github.com/weavejester/environ),Config(https://github.com/yogthos/config),Aero(https://github.com/juxt/aero),andFluorine(https://github.com/reborg/fluorine)canalsobeusedforconfigurationmanagement.

EnablingOmniconf

ToenableanOmniconflibraryforanexistingproject,suchasHelpingHandsConsumerService,addtheOmniconfdependencytotheproject.cljfileandaddJVMopts,conftothedevprofilethatpointstotheconf.ednOmniconfconfigurationfile:

(defprojecthelping-hands-consumer"0.0.1-SNAPSHOT"

:description"HelpingHandsConsumerApplication"

;;DatomicFreeEdition

[com.datomic/datomic-free"0.9.5561.62"]

;;Omniconf

[com.grammarly/omniconf"0.2.7"]

:dev{:aliases{"run-dev"["trampoline""run""-m"

"helping-hands.consumer.server/run-dev"]}

:dependencies[[io.pedestal/pedestal.service-tools"0.5.3"]]

:uberjar{:aot[helping-hands.consumer.server]}

:themes[:rdash]}}

:debug{:jvm-opts

:main^{:skip-aottrue}helping-hands.consumer.server)

IntegratingwithHelpingHandsTheHelpingHandsservicesthatwereimplementedinthepreviouschapterusedafixedDatomicdatabaseURI,suchasdatomic:mem://consumer,fortheconsumerdatabasemanagedbytheconsumerservice.InsteadoffixingthenameofthedatabaseandDatomicURI,itmustbemadeconfigurablesothatitcanbechangedatthetimeofdeployment.Forexample,considerascenariowhereyouwishtoruntwoinstancesofConsumerservicebutwithseparateconsumerdatabases.ItwillnotbepossibletodosoiftheDatomicdatabaseURIishardcodedintheimplementationandnotmadeconfigurable.

Omniconfrequiresalltheconfigurationparameterstobedefinedviatheomniconf.core/definefunction.FortheConsumerserviceoftheHelpingHandsapplication,addanewhelping-hands.consumer.confignamespaceandinitializetheconfigurationasshownhere:

(nshelping-hands.consumer.config

"DefinesConfigurationfortheService"

(:require[omniconf.core:ascfg]))

(defninit-config

"Initializestheconfiguration"

[{:keys[cli-argsquit-on-error]:asparams

:or{cli-args[]quit-on-errortrue}}]

;;definetheconfiguration

(cfg/define

{:conf{:type:file

:requiredtrue

:verifieromniconf.core/verify-file-exists

:description"MECBOTconfigurationfile"}

:datomic

{:nested

{:uri{:type:string

:default"datomic:mem//consumer"

:description"DatomicURIforConsumerDatabase"}}}})

;;like-:some-option=>SOME_OPTION

(cfg/populate-from-envquit-on-error)

;;loadpropertiestopick-Dconffortheconfigfile

(cfg/populate-from-propertiesquit-on-error)

;;Configurationfilespecifiedas

;;EnvironmentvariableCONForJVMOpt-Dconf

(when-let[conf(cfg/get:conf)]

(cfg/populate-from-fileconfquit-on-error))

;;like-:some-option=>(java-Dsome-option=...)

;;reloadJVMargstooverwriteconfigurationfileparams

(cfg/populate-from-propertiesquit-on-error)

;;like-:some-option=>-some-option

(cfg/populate-from-cmdcli-argsquit-on-error)

;;Verifytheconfiguration

(cfg/verify:quit-on-errorquit-on-error))

(defnget-config

"Getsthespecifiedconfigparamvalue"

[&args]

(applycfg/getargs))

Inthisexample,thereisamandatoryconfigurationparameter,:conf,definedtobeof:filethatmustpointtotheconf.ednconfigurationfile.Thereisalsoaverifierattachedtoitthatvalidatesthepresenceoftheconf.ednfilebasedonthedefinedlocation.Also,thereisa:datomicconfigurationparameterdefinedthatisnestedandhasa:uriparameter,definedasstringtype,withadefaultvalueofdatomic:mem://consumerthatpointstoanin-memoryDatomicdatabase.

Afterdefiningtheconfigurationparameters,theimplementationchecksfortheconfparametervaluebyfirstloadingtheJVMproperties.The-DconfJVMpropertypointstotheconf.ednfileasdefinedinthedevprofileofproject.clj.Theconfigurationparametersarereadinthesequenceofenvironmentvariables,propertiesfile,andcommandline,eachoverwritingthevaluesdefinedbytheprevioussourceaspertheloadingsequence.

Aget-configutilitymethodisalsodefinedwithinthesamenamespaceforothermodulestolookuptheconfigurationparametersthatareloadedbythisnamespaceusingOmniconf.Toloadtheconfigurationatstartup,calltheinit-configmethodattheapplicationentrypoint,thatis,helping-hands.consumer.server,asshownhere:

(nshelping-hands.consumer.server

(:gen-class);for-mainmethodinuberjar

(:require[io.pedestal.http:asserver]

[helping-hands.consumer.config:ascfg]

[helping-hands.consumer.service:asservice]))

(defnrun-dev

"Theentry-pointfor'leinrun-dev'"

[&args]

(println"\nCreatingyour[DEV]server...")

;;initializeconfiguration

(cfg/init-config{:cli-argsargs:quit-on-errortrue})

(->service/service;;startwithproductionconfiguration

;;Wireupinterceptorchains

server/default-interceptors

server/dev-interceptors

server/create-server

server/start))

(defn-main

"Theentry-pointfor'leinrun'"

[&args]

(println"\nCreatingyourserver...")

(server/startrunnable-service))

Now,ifyoutrytoruntheConsumerserviceindevmodeatREPL,Omniconfwillloadtheconfiguration,verifyit,andmakeitavailableviathehelping-hands.consumer.config/get-configmethod.Sincewehaven'tcreatedtheconf.ednfileinthespecifiedconf/location,theverifiershouldfailattheinitializationstepitself,asshownhere:

helping-hands.consumer.server>(defserver(run-dev))

CompilerExceptionjava.io.FileNotFoundException:config/conf.edn(Nosuchfileor

directory),compiling:(form-init3431514182044937086.clj:118:44)

Addaconf.ednfileundertheconfigdirectoryanddefineonlytheDatomicURIconfiguration,asshownhere:

{:datomic{:uri"datomic:mem://consumer"}}

Now,theconfigurationisvalid,asitfindsthedefinedconf.ednfile.Inthiscase,asshownintheREPLsessioninthefollowingcodesnippet,Omniconfwilldumptheloadedconfigurationthatcanthenbeverifiedtomakesurealltheconfigurationparametersareloadedasexpected.Notethattherequired:confparameterisnotdefinedintheconf.ednfile,butitisdefinedastheJVMpropertythatisalsoreadinthesequence:

;;startserviceindevmode

Omniconfconfiguration:

{:conf#object[java.io.File0x27a8b5d3"config/conf.edn"],

:datomic{:uri"datomic:mem://consumer"}}

#'helping-hands.consumer.server/server

;;trylookinguptheconfigurationparameter

helping-hands.consumer.server>(helping-hands.consumer.config/get-config:datomic)

{:uri"datomic:mem//consumer"}

helping-hands.consumer.server>(helping-hands.consumer.config/get-config:datomic

"datomic:mem//consumer"

Now,trychangingtheDatomicURlparameterintheconfig.ednfiletodatomic:mem://consumer-sample.Itwilloverwritethedefaultvalueandcanthenbe

usedbytheapplicationusingtheget-configmethodasshownhere:

{:conf#object[java.io.File0x60e62394"config/conf.edn"],

:datomic{:uri"datomic:mem://consumer-sample"}}

#'helping-hands.consumer.server/server

helping-hands.consumer.server>(helping-hands.consumer.config/get-config:datomic)

{:uri"datomic:mem://consumer-sample"}

helping-hands.consumer.server>(helping-hands.consumer.config/get-config:datomic

"datomic:mem://consumer-sample"

ThepersistencenamespaceoftheConsumerservicecannowreadthedatabaseURIdirectlyfromtheconfigurationinsteadofexpectingitasanargumentofthecreate-consumer-databasefunction:

(:require[datomic.api:asd]

[helping-hands.consumer.config:ascfg]))

(defncreate-consumer-database

"Createsaconsumerdatabaseandreturnstheconnection"

(let[dburi(cfg/get-config[:datomic:uri])

(whendb

(d/transactconn

[{:db/ident:consumer/id

:db/doc"UniqueConsumerID"

:db/indextrue}

(ConsumerDBDatomic.conn)))

OmniconfworkswithboththeLeiningenandBootbuildtoolsofClojure.Formoredetailsandusageinformationofallthepossibleoptions,takealookattheexample-lein(https://github.com/grammarly/omniconf/tree/master/example-lein)andexample-boot(https://github.com/grammarly/omniconf/tree/master/example-boot)projectsofOmniconf.

ManagingapplicationstateswithmountOncetheconfigurationparametersaredefinedusingOmniconf,theyareaccessibleacrossthenamespacesasimmutabledata.Theconfigurationparametersareoftenusedtocreatestatefulobjects,suchasdatabaseconnections.Forexample,intheConsumerserviceproject,Omniconfmadeitpossibletocreateaconsumerdatabasebydirectlylookingupthe:datomic:uriconfigurationparameterwithinthecreate-consumer-databasefunction.

Thehelping-hands.consumer.persistence/create-consumer-databasefunctionhasasideeffectofdatabasebeingcreatedandalsoanewconnectionbeinginitializedtoconnecttothecreateddatabase.ThisconnectionhasastatethatmustbesharedacrossothernamespacesoftheHelpingHandsConsumerservicethatneedaccesstothedatabase.Inthecurrentimplementation,theconnectionwasinitializedatthefirstcalltothehelping-hands.consumer.core/consumerdbasshownhere:(nshelping-hands.consumer.core"InitializesHelpingHandsConsumerService"(:require[cheshire.core:asjp][clojure.string:ass][helping-hands.consumer.persistence:asp][io.pedestal.interceptor.chain:aschain])(:import[java.ioIOException][java.utilUUID]))

;;delaythecheckfordatabaseandconnection;;tillthefirstrequesttoaccess@consumerdb(def^:privateconsumerdb(delay(p/create-consumer-database)))

Insteadofcreatingastateusingdelay,thestatemanagementcanbehandledeffectivelyusingalibrary,suchasmount(https://github.com/tolitius/mount).Creatingapplicationstatesusingmountallowsforthereloadingoftheentireapplicationstateusingstartandstopfunctionsprovidedbythemount.corenamespace.The

mountlibraryalsohelpswithstatecompositionbyallowingtheapplicationtostartonlywithspecificstatesandatthesametimeswappingotherswithnewvalues.Italsosupportsruntimearguments.

Component(https://github.com/stuartsierra/component)isanotherClojurelibrarythatiswidelyusedtomanagethelifecycleofstatefulobjectsinaClojureproject.mountisanalternativetotheComponentlibrarywithkeydifferences(https://github.com/tolitius/mount/blob/master/doc/differences-from-component.md#differences-from-component),oneofthembeingComponent'srequirementoftheentireappbeingbuiltarounditscomponentobjectmodel.

Enablingmount

ToenablethemountlibraryfortheHelpingHandsConsumerserviceapplication,addamountdependencytotheproject.cljfileasshowninthefollowingcodesnippet.Also,createanewhelping-hands.consumer.statenamespacethatwillbeusedtodefinethestatesusingthemount.core/defstatefunction,whichcanbereferredtobyothernamespacesoftheprojecttogetaccesstothecurrentstateofthedefinedobject,suchastheDatomicdatabaseconnection:

(defprojecthelping-hands-consumer"0.0.1-SNAPSHOT"

:description"HelpingHandsConsumerApplication"

;;Omniconf

;;Mount

[mount"0.1.11"]

:main^{:skip-aottrue}helping-hands.consumer.server)

IntegratingwithHelpingHandsTointegratemountwiththeHelpingHandsConsumerserviceproject,createastateforaDatomicdatabaseconnectionwithinthehelping-hands.consumer.statenamespace,asshownhere:

(nshelping-hands.consumer.state

"InitializesStateforConsumerService"

(:require[mount.core:refer[defstate]:asmount]

[helping-hands.consumer.persistence:asp]))

(defstateconsumerdb

:start(p/create-consumer-database)

:stop(.stopconsumerdb))

The:startclauseiscalledatthetimeofstartupand:stopiscalledatshutdown.ThefunctionstopisdefinedfortheConsumerDBprotocol,asshowninthefollowingexampleunderthehelping-hands.consumer.persistencenamespace:

(:require[datomic.api:asd]

[helping-hands.consumer.config:ascfg]))

(defprotocolConsumerDB

"Abstractionforconsumerdatabase"

"Adds/Updatesaconsumerentity")

(entity[thisidflds]

"Getsthespecifiedconsumerwithallorrequestedfields")

(delete[thisid]

"Deletesthespecifiedconsumerentity")

(close[this]

"Closesthedatabase"))

(defrecordConsumerDBDatomic[conn]

ConsumerDB

(close[this]

(d/shutdowntrue)))

Next,addthestartupandshutdownhooksformountattheapplicationentrypointunderthehelping-hands.consumer.servernamespace.Theshutdownhookdefinedat

theapplicationentrypointcallsthe:stopclauseofmountthatmustcleanupalltheresourcesrelatedtothestatefulobject,thatis,theDatomicconnectionasshowninthefollowingexample:

(nshelping-hands.consumer.server

[mount.core:asmount]

[helping-hands.consumer.config:ascfg]

[helping-hands.consumer.service:asservice]))

(defnrun-dev

[&args]

;;initializestate

(mount/start)

;;Addshutdown-hook

(.addShutdownHook

(Runtime/getRuntime)

(Thread.mount/stop))

server/start))

(defn-main

[&args]

;;initializestate

(mount/start)

;;Addshutdown-hook

(.addShutdownHook

Oncemountissetuptoinitializethestateatstartup;thedefinedstateconsumerdbcannowbereferredtoacrossnamespacesanduseddirectlyforthehelping-hands.consumer.corenamespaceasshownhere:

(nshelping-hands.consumer.core

"InitializesHelpingHandsConsumerService"

[helping-hands.consumer.persistence:asp]

[helping-hands.consumer.state:refer[consumerdb]])

[java.utilUUID]))

;;tillthefirstrequesttoaccess@consumerdb

;;NOLONGERREQUIREDDUETOMOUNT

;;(def^:privateconsumerdb

;;(delay(p/create-consumer-database)))

;;Usethereferredstatefulconsumerdbdirectlyintheinterceptor

(defupsert-consumer

{:name::consumer-upsert

:enter

(fn[context]

id(:idtx-data)

db(.upsertconsumerdbid(:nametx-data)

:error...})

SummaryInthischapter,wefocusedonhowtobuildconfigurableapplicationsthatcanadaptaspertherequirementsanddependenciesathand.WelookedatanopensourceconfigurationutilitycalledOmniconfthatprovidesaneffectivewaytodefineandvalidateconfigurationparametersforClojureapplications.

WealsolookedathowtheruntimestateoftheapplicationcanbecomposedandsharedamongvariousnamespacesoftheClojureapplication.Welookedatanopensourcelibrarycalledmountthathelpsapplicationstomanageandcomposetheirstatesatruntimewithoutaffectingtheoverallstructureoftheimplementation.

Inthenextchapter,wewilllearnhowtoadoptevent-drivenarchitectureforHelpingHandsmicroservices.WewillalsolearnhowtobuilddataflowsforthemicroservicesofHelpingHands.

Event-DrivenPatternsforMicroservices

""Thesinglebiggestproblemwithcommunicationistheillusionthatithastakenplace.""

-GeorgeBernardShaw

Microservicesaddressasingleboundedcontextandaredeployedindependentlyononeormorephysicalmachinesthataredistributedacrossanetwork.Althoughtheyaredeployedinisolation,theyneedtointeractwitheachothertoaccomplishapplication-leveltasksthatmaycutacrossmultipleboundedcontexts.Thechoiceofcommunicationmediumandmethodhasagreatimpactontheperformanceanddurabilityoftheentiremicroservice-basedarchitecture.Eventsareoneofthemethodsofasynchronouscommunicationamongmicroservicestoexchangedataofinterest.Part-1ofthebookexplainstheimportanceoftheobservermodelandhowamessagebroker(https://en.wikipedia.org/wiki/Message_broker)helpsinsendingandreceivingeventsinamicroservicesarchitecture.Inthischapter,youwill:

Learnaboutevent-drivenpatternsforeffectivemessagingamongmicroservicesLearnhowtouseApacheKafkaasamessagebrokerformicroservicesLearnhowtouseApacheKafkaforEventSourcingLearnhowtointegrateApacheKafkawithHelpingHandsmicroservices

Implementingevent-drivenpatternsEvent-drivenpatternsaddresstheobservermodelofcommunicationtosendmessagesamongmicroservices.Themessagesaresentandreceivedthroughamessagebrokerthatactsasaconnectingbridgebetweenthesenderandthereceiver.Inamicroservicesarchitecture,thesemessagesmaybegeneratedaseventsasanoutcomeoftheactiontakenbythemicroservice.Messagesforwhichthesourcemicroservicedoesnotexpectaresponsefromthetargetservicecanbepublishedaseventsasynchronously,insteadofsendingthemoveraRESTAPIfordirectcommunication.Sinceitisnotadirectcommunication,aneventcanbepublishedonceandconsumedbymorethanonemicroservicethathassubscribedtoreceiveit.Moreover,thesenderdoesnotgetblockedbythereceiverforeacheventthatispublished.

Messagebrokersalsohelptobuildaresilientarchitectureforevent-drivencommunicationasreceiversneednotbeavailablewhiletheeventisbeingproducedandtheycanconsumethemessagesatwill.Incaseoffailures,thereceivercanberestartedanditcanstartconsumingtheeventswhereitleftoff.Duringthedowntime,themessagebrokeritselfactsasaqueueandcachesalltheeventsthatwerenotconsumedbythereceiverandmakesitavailabletothereceiverondemand.

Asynchronouscommunicationviaeventsalsodecouplesthesenderfromthereceiverthathelpsinscalingboththesidesindependently.Thisfeatureisofprimeimportanceinamicroservice-basedarchitectureasitallowsthemicroservicestobedeployedindependentlyofothermicroservicesfromwhichitconsumestheeventsviaamessage-broker.Event-drivenpatternsarealsousedforasynchronoustaskssuchasstoringauditlogstokeeptrackofruntimestateoftheapplicationandsendingalerts.

Theobservermodel,alongwithvariousdatamanagementpatterns,areexplainedinChapter2,MicroservicesArchitectureofthisbook.

EventsourcingEventscanbeusedtoadvertisethechangesinthestateoftheentitiesmanagedbyamicroservice.Insuchcases,eventscarrytheupdatedstateoftheentityincludingtheentityidentifierandthevaluesofthefieldsthathavechanged.Itisalsorecommendedtoincludeauniqueidentifierandversionnumberaswellwitheachupdateevent.Anyinterestedmicroservicecanthensubscribetotheseeventsviaamessagebrokerandreceivethechangestoupdatethestateoftheentitylocally.TheLookupServiceoftheHelpingHandsapplicationisonesuchservicethatlistenstoallthestatechangeeventsgeneratedbytheconsumer,provider,andordermicroservicestokeepanupdateddatasetforuserstolookup.

Oneofthemainadvantagesofpublishingallthestatechangesacrossmicroservicesasimmutableeventsistomakesurethatthestateoftheentiremicroservice-basedapplicationcanberebuiltbyjustprocessingtheseeventsinthesequenceinwhichtheyarepublished.Boththecurrentstateoftheapplicationaswellasthestateinthepastcanbereconstructedbyprocessingtheseeventsintheexactsamesequence.Abilitytoreplaytheeventsnotonlyhelpsinrebuildingthestateoftheapplication,butalsohelpsinauditinganddebugging.

Messagebrokers,suchasApacheKafka(https://kafka.apache.org/),allowpublishingmessagesdecoupledfromconsumingthemandeffectivelyactasastoragesystemthatmaintainsadurablelogofpublishedevents.Suchsystemsallowtheeventstobecapturedbymultiplesystemsatthesametime,asshownintheprecedingdiagram.Forexample,theeventspublishedbyServiceAandServiceBcanbeconsumedbyServiceC,butatthesametime,theseeventscanbebacked-upinabackupstoreorcapturedinatransactionalstoreforbuildingmachinelearningmodelsorcapturedforreal-timemonitoringandreportingoftheapplicationstateinrealtime.Sincethelogsretainedbythebrokersareimmutableanddurable,applicationsthataredevelopedin-futurecanalsoreplaytheeventstobuildthetemporalstateoftheapplicationandworkonit.

UsingtheCQRSpatternTheCommandQueryResponsibilitySegregation(CQRS)patternappliesthecommand-queryseparation(https://en.wikipedia.org/wiki/Command-query_separation)principlebysplittingtheapplicationintotwoparts,querysideandcommandside.Querysideisresponsibleforgettingthedatabyonlyqueryingthestateoftheapplication,whereascommandsideisresponsibleforchangingthestateofthesystembyperformingcreate,update,ordeleteoperationsonapplicationdatathatmayresultinupdationofoneormoredatabasesusedbytheapplication.AlthoughtheCQRSpatternisoftenusedinconjunctionwiththeevent-sourcingpattern,itneednotbetiedtoeventsandcanbeappliedtoanyapplicationwithorwithoutevents.TheCQRSpatternisrecommendedforapplicationsthatarenotbalancedwithrespecttoreadandwriteloads.

CQS(https://en.wikipedia.org/wiki/Command-query_separation)principlewasdevisedbyBertrandMeyer(https://en.wikipedia.org/wiki/Bertrand_Meyer),whereasCQRStermwasfirstcoinedbyGregYoungaspartofCQRSdocuments(https://cqrs.files.wordpress.com/2010/11/cqrs_documents.pdf).

TheCQRSpatternfitswellwithmicroservice-basedarchitectureastheentireapplicationissplitintoseparateservices,eachhavingtheirowndatamodelandpublishingthechangesinthestateoftheirdatamodelasevents.Butatthesametime,itisalsochallengingtokeeptheseseparatemodelsconsistent.Thisiswheresagasareuseful,whichsupporteventualconsistency.ThesagaspatternhasbeendescribedinChapter2,MicroservicesArchitecture.

FormoredetailsontheCQRSpatternanditsusage,readtheCQRSarticlebyMartinFowler(https://martinfowler.com/bliki/CQRS.html)andtheClarifiedCQRSarticlebyUdiDahan(http://udidahan.com/2009/12/09/clarified-cqrs/).

IntroductiontoApacheKafkaApacheKafkaisadistributedstreamingplatformthatallowsapplicationstopublishandsubscribetoastreamofrecords.ApacheKafkaisnotjustamessagequeue,italsoallowsapplicationstopublishtheeventsthatarethenstoredbyKafkaasanimmutableloginafault-tolerantway.Itallowstheproducersandconsumersoftheeventstoscalehorizontallywithoutaffectingeachother.SincetheeventsareloggedinthesamesequenceastheyarepublishedwithinKafka,itallowsconsumerstoreplaythelogfromanduptothedesiredpointtoreconstructviewsoftheapplicationstate.

DesignprinciplesKafkaisrunasaclusterofoneormoreserversthatactasmessagebrokers(https://en.wikipedia.org/wiki/Message_broker)inthesystem.Kafkacategorizesthestreamofrecordsundertopicsthatareusedbyproducersandconsumerstoproducerecordsandconsumethem,respectively.Eachrecordconsistsofakey-valuepairandatimestamp.

AtypicalKafkaclusterisshowninthefollowingdiagramalongwiththestructureofaKafkatopicanditspartitions.AtopicisacoreabstractionforastreamofrecordsinKafka.Producerspublishtherecordstoatopicthatcanhavezero,one,ormoreconsumerssubscribedtoit.Eachtopicconsistsofoneormorepartitionsthatareordered,immutablesequenceofrecordsthatisappendedtoacommitlogandstoredonadiskandreplicatedacrossserversforfault-toleranceanddurability.Eachrecordthatispublishedonatopicgetsassignedtoapartitionwithinwhichitisassignedasequentialidentifiernumberthatiscalledtheoffset.Theoffsetuniquelyidentifiesarecordwithinthepartitionandisalsousedasareferencebyconsumerstotracktheirstateofconsumption.Offsetsallowconsumerstoresetthemselvestoanearlierstatetoeitherreplaytherecordsorfast-forwardintimetoskiptherecords.AKafkaclusterretainsallthepublishedrecordsonlyforaconfigurableretentionperiod.Posttheconfiguredretentionperiod,therecordsarenolongeravailableforconsumersirrespectiveofwhethertheywereconsumedearlierornot:

Kafkaproducerspublishdatatooneormoretopicsandhavethefreedomtodecidethepartitionforthepublishedrecords.Kafkaconsumersalwaysjoinwithaconsumergrouplabelthatissharedwithonemoreconsumerthatmaybe

presentononeormoremachines.TheConsumerGroupplaysanimportantroleinthewaytheKafkaclusterdeliversthepublishedrecordstotheconsumers.EachrecordpublishedonaKafkatopicisdeliveredtoonlyoneconsumerwithinaConsumerGroup.Forexample,intheprecedingdiagram,therearetwoconsumergroups—ConsumerGroupAandConsumerGroupBandsixconsumerswiththreeconsumersineachgroup.Anyrecordpublishedonpartitions0,1,2,or3issenttoonlyoneconsumerwithineachgroup.Forexample,recordspublishedtoPartition-0issenttoC1ofConsumerGroupAandC2ofConsumerGroupB.Kafkabalancestheflowofrecordsacrosstheconsumersintheconsumergroups.

ApacheKafkaprovidestotalorderingoverrecordsonlywithinapartition.Usecasesthatrequiretotalorderingovertherecordspublishedonatopicmusthavethetopicconfiguredtohaveasinglepartition.Thisconfigurationalsolimitsthethroughputtoonlyoneconsumerprocessperconsumergroup.

GettingKafkaApacheKafkaisanopen-sourceplatformandcanbedownloadedfromitsdownloadpage(https://kafka.apache.org/downloads).Fortheexamplesinthisbook,downloadandextractthe1.0.0release(https://www.apache.org/dyn/closer.cgi?path=/kafka/1.0.0/kafka_2.11-1.0.0.tgz)fromoneofthemirrors,asshownhere:

#downloadKafka1.0.0withScala2.11(Recommended)

%wgethttp://redrockdigimark.com/apachemirror/kafka/1.0.0/kafka_2.11-1.0.0.tgz

#extractKafkadistribution

%tar-xvfkafka_2.11-1.0.0.tgz

#switchtoKafkadirectory

%cdkafka_2.11-1.0.0

ApacheKafkaServersrequireApacheZookeeper(https://zookeeper.apache.org/)forclustercoordination.KafkadistributionpackagesasinglenodeZookeeperinstanceforconvenience;however,itisrecommendedtosetupanexternalZookeeperclusterforproductiondeployments.StartasinglenodeZookeeperwiththedefaultpropertiesfilepackagedwithKafka,asshownhere:

#startZookeeper

bin/zookeeper-server-start.shconfig/zookeeper.properties

INFObindingtoport0.0.0.0/0.0.0.0:2181

(org.apache.zookeeper.server.NIOServerCnxnFactory)

OncetheZookeeperisstarted,eitherswitchtoanewterminalorputtheZookeeperprocessinthebackgroundtogettheaccesstothesameterminal.Next,startaKafkaserverwiththedefaultpropertiesfilepackagedwithKafka,asshownhere:

#startKafkaServer

bin/kafka-server-start.shconfig/server.properties

INFOKafkaversion:1.0.0(org.apache.kafka.common.utils.AppInfoParser)

INFOKafkacommitId:aaa7af6d4a11b29d(org.apache.kafka.common.utils.AppInfoParser)

INFO[KafkaServerid=0]started(kafka.server.KafkaServer)

Thedefaultserver.propertiesfilecontainsafixedbroker.idpropertythatissetto0andthelistenersconfiguredtoadefaultportof9092withlog.dirpointingto/tmp.ForasingleKafkaserver,thesesettingsarefine,buttostartmultipleKafkaserversonthesamemachine,thesepropertiesmustbechangedforeachserverto

avoidID,port,andcommitlogdirectoryclashes.Next,createatopicbythenameoftestwithasinglepartition,asshownhere:

#createatopic

%bin/kafka-topics.sh--create--zookeeperlocalhost:2181--replication-factor1--

partitions1--topictest

Createdtopic"test".

#Listandconfirmthattopicwascreated

%bin/kafka-topics.sh--list--zookeeperlocalhost:2181

ThecreatetopiccommandrequirestheaddresstotheZookeeperthatisusedbyKafkacluster.SinceZookeeperwasstartedwithdefaultproperties,itwouldhavetakenthe2181portifthatwasfreeonthemachine.Next,startaproducerinanewterminalandpublishsomemessages,asshownhere:

#startanewproducertoproducethemessagesontopic'test'

%bin/kafka-console-producer.sh--broker-listlocalhost:9092--topictest

>HelloKafka!

>HelloHelpingHands!

ThestartedproducerconnectstotheKafkaserveronport9092;thatis,thedefaultportofKafkaserver.Thestartedproducerisalsoconfiguredtopublishthemessagesonthetesttopicthatwascreatedearlier.Oncetheproducerisstarted,publishthetwomessages,asshownintheprecedingcodesnippet,andstartaconsumerforthesametopicinanewterminal,asshownhere:

#startanewconsumertoconsumethemessagesfromtesttopicfromthebeginning

%bin/kafka-console-consumer.sh--bootstrap-serverlocalhost:9092--topictest--from-

beginning

HelloKafka!

HelloHelpingHands!

Theconsumerreceivesthepublishedmessagesandprintsthemontheconsole.Now,gobacktotheproducerterminalandpublishonemoremessageandverifythatconsumerreceivesonlythenewmessage,asshownhere:

#publishanewmessage

>HelloKafka!

>HelloHelpingHands!

>KafkaWorks!

#validatethemessageontheconsumerterminal

%bin/kafka-console-consumer.sh--bootstrap-serverlocalhost:9092--topictest--from-

beginning

HelloKafka!

HelloHelpingHands!

KafkaWorks!

KafkaManager(https://github.com/yahoo/kafka-manager)isausefultoolthathelpsinmanagingoneormoreKafkaclustersusingasingleweb-basedinterface.

UsingKafkaasamessagingsystemKafkaprovidesbothqueuing(http://en.wikipedia.org/wiki/Message_queue)andpublish-subscribe(http://en.wikipedia.org/wiki/Publish-subscribe_pattern)constructsofamessagingsystembytheconceptofaConsumerGroup.ThemessagespublishedonatopicpartitionarebroadcastedtoaconsumerwithineachConsumerGroupandwithintheConsumerGroup,eachconsumerreceivesthemessagesfromadifferentpartitionofatopic.Therefore,thenumberofconsumerspresentintheConsumerGroupmustnotbemorethanthenumberofpartitionspresentinatopic.Iftheyaremorethanthenumberofpartitions,thentheywillbejustsittingidleandwillonlygetthemessageifoneoftheconsumersfailswithinthegroup.Forexample,todemonstratethemessagingcapabilitiesofKafka,startonemoreconsumerforthetesttopic,asshownhere:#startanotherconsumer%bin/kafka-console-consumer.sh--bootstrap-serverlocalhost:9092--topictest--from-beginning

Oncetheconsumerisstarted,anyothermessagethatissentfromtheproducerprocessisreceivedbyboththeconsumers.ThishappensbecauseboththeconsumersareapartofthedifferentConsumerGroupandasperdesign,Kafkawillbroadcastthepublishedmessagesonatopicacrossconsumergroups.SincecurrentsetuphasonlyoneconsumerintheConsumerGroup,bothconsumersreceivethemessage.Next,stopbothoftheconsumerprocessesandstartthemasapartofthesameConsumerGroup,asshownhere:

#startfirstconsumer

%bin/kafka-console-consumer.sh--bootstrap-serverlocalhost:9092--topictest--group

test--from-beginning

#startsecondconsumer

Now,themessagespublishedviaproducerarereceivedbyonlyoneoftheconsumersasboththeconsumersareapartofthetestgroupandthetesttopichasadefaultpartitioncountof1.Tryterminatingthefirstconsumerprocessthatisreceivingthemessagesandsendanewmessagefromtheproducer.Noticethatnow,themessageisbeingreceivedbythesecondconsumerthatbecomesactiveandstartsconsumingthemessagesfromthetopicpartitionwheretheterminated

consumerleftoff.

UsingKafkaasaneventstoreApacheKafkaisnotjustlimitedtoamessagingsystem,itcanalsobeusedasadurablestorageforimmutablerecordsandtobuildastreamingdatapipelineontopofit.Itiswellsuitedforusecasessuchaswebsiteactivitytracking,real-timemonitoring,logaggregation,andprocessingstreamsofdata.AnymessagesthatarepublishedonaKafkatopicarepersistedondiskandreplicatedacrossKafkaserversbasedontheconfigurationforfaulttolerance.SinceKafkaguaranteesthesequenceofmessageswithinatopicpartition,itallowsconsumerstocontroltheirreadpositionirrespectiveofotherconsumersofthesametopic.ThefollowingexampleshowshowtheeventspublishedbyaproduceraremadeavailabletotheconsumersbasedonthetopicandtheassociatedConsumerGroup:

#startproducerfor'test'topic

>HelloApacheKafka!

>HelloKafkaEvent!

#startconsumerin'test'consumergrouplisteningto'test'topic

HelloApacheKafka!

HelloKafkaEvent!

#consumefrombeginninginadifferentconsumergroup

#sinceitisadifferentconsumergroup,itreplaymessagesthatwere

#publishedearlieraswellastheywerenotcommittedbyanyother

#consumerinconsumergroup'test1'

test1--from-beginning

HelloKafka!

HelloHelpingHands!

KafkaWorks!

HelloApacheKafka!

HelloKafkaEvent!

#startingaconsumerwithoutthe'from-beginning'flag

#waitsfornewmessagesonlyanddoesnotreplaypreviousmessages

#startingaconsumerwithoutthe'from-beginning'flag

#waitsfornewmessagesonlyanddoesnotreplaypreviousmessages

#evenforthenewconsumergroup'test2'

Kafkaconsumerscandecidetheoffsetfromwheretheywishtostartconsumingthemessages.Intheprecedingexample,theconsumerwasstartedwitha--from-beginningflagthattellstheconsumertostartconsumingfromthefirstoffset,andthatiswhyeverytimetheconsumerisstartedwiththisflaginanewConsumerGroup,itwillreplaythereceivedmessagesineachConsumerGroupfromthebeginning,asshownintheprecedingexample.ThereplayofmessagesandoffsetretentionalsodependsontheretentionperiodsetusingKafkaserverproperties.

TheApacheKafkaconfiguration(https://kafka.apache.org/documentation/#configuration)pageprovidesalistofconfigurationparametersforApacheKafkaservers.

UsingKafkaforHelpingHands

TheHelpingHandsapplicationusesApacheKafkatoimplementtheobservermodelandsendasynchronouseventsamongmicroservices.ItisalsousedasaneventstoretocaptureallthestatechangeeventsgeneratedfrommicroservicesthatareconsumedbytheLookupservicetobuildaconsolidatedviewtoserverlookuprequests.TheAlertmicroserviceoftheHelpingHandsapplicationalsoreceivesthealerteventsviatheKafkatopicandsendsanemailasynchronously.

ApacheKafkaincludesfivecoreAPIs:

TheProducerAPIallowsapplicationstopublishstreamsofeventstooneormoretopicsTheConsumerAPIallowsapplicationstoconsumepublishedeventsfromoneormoretopicsTheStreamsAPIallowstransformingstreamsfrominputtopicsandpublishtheresultstooutputtopicsTheConnectAPIallowssupportforvariousinputandoutputsourcestocaptureanddumpstreamofeventsTheAdminClientAPIallowstopicsandservermanagementalongwithotherKafkamanagementoperations

TointegrateKafkawiththeHelpingHandsapplication,theProducerandConsumerAPIswillonlyberequiredtoproducestatechangeeventsandconsumethem.Therequiredtopicscanbecreatedexternallyusingkafka-topics.shscript,asshownintheprevioussection.ToincludetheproducerandconsumerclientAPIs,addtheprojectdependencyofkafka-clients,asshowninthefollowingexample,totheproject.cljfileofthecorrespondingmicroserviceproject:

(defprojecthelping-hands-alert"0.0.1-SNAPSHOT"

:description"HelpingHandsAlertApplication"

;;Omniconf

;;Mount

[mount"0.1.11"]

;;postalforemailalerts

[com.draines/postal"2.0.2"]

;;kafkaclients

[org.apache.kafka/kafka-clients"1.0.0"]

;;logger

[org.clojure/tools.logging"0.4.0"]

:main^{:skip-aottrue}helping-hands.alert.server)

UsingKafkaAPIsAlthoughClojurehasKafkawrappers(https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-Clojure)available,thisbookfocusesonusingApacheKafkaJavaAPIsdirectlyinsteadofaClojurewrapper.TocreateaKafkaconsumer,importtherequiredJavaAPIs,asshownhere:

(nshelping-hands.alert.channel

"InitializesHelpingHandsAlertChannelConsumer"

[clojure.tools.logging:aslog]

[helping-hands.alert.config:asconf]

[postal.core:aspostal])

(:import[java.utilCollectionsProperties]

[org.apache.kafka.common.serialization

LongDeserializerStringDeserializer]

[org.apache.kafka.clients.consumer

ConsumerConsumerConfigKafkaConsumer]))

Next,defineafunctioncreate-kafka-consumerthatinitializesaKafkaconsumerandreturnsit,asshownhere:

(defncreate-kafka-consumer

"CreatesanewKafkaConsumer"

(let[props(doto(Properties.)

(.putAll(conf/get-config[:kafka]))

(.putConsumerConfig/KEY_DESERIALIZER_CLASS_CONFIG

(.getNameLongDeserializer))

(.putConsumerConfig/VALUE_DESERIALIZER_CLASS_CONFIG

(.getNameStringDeserializer)))

consumer(KafkaConsumer.props)

_(.subscribeconsumer(Collections/singletonList

(get(conf/get-config[:kafka])"topic")))]

consumer))

TheKafkaconsumerrequiresasetofconfigurationstoconnecttotheKafkaserverandstartconsumingthepublishedmessages.ThisconfigurationcanberetrievedviaOmniconfconfiguration,asdiscussedinthepreviouschapter.Tosettherequiredconfigurationparameter,first,definetheconfigurationparameterforOmniconftopick,asshownhere:

(cfg/define

{:conf{:type:file

:requiredtrue

:verifieromniconf.core/verify-file-exists

:description"MECBOTconfigurationfile"}

:kafka{:type:edn

:default{"bootstrap.servers""localhost:9092"

"group.id""alerts"

"topic""hh_alerts"}

:description"KafkaConsumerConfiguration"}})

Thebootstrap.serversparametercanhavemorethanoneKafkaserverdefinedasacomma-separatedvaluewiththeformatofhost:port.Aspertheprecedingconfiguration,itusesasingleKafkaservertoconnecttothatisrunninglocallyonport9092.Also,ittakesasinputagroup.idthatspecifiestheConsumerGroupfortheconsumerandatopictowhichitsubscribesformessages.Oncetheconsumeriscreated,itcanbeusedtolistenformessagesontheconfiguredtopic,asshownhere:

(defncapture-records

"Consumetherecordsusinggivenconsumer"

[consumerresult]

(whiletrue

(doseq[record(.pollconsumer1000)]

(swap!resultconjrecord))

(Thread/sleep5000)))

Thecapture-recordsfunctiontakesasinputaconsumerthatiscreatedbythecreate-kafka-consumerfunction,asdefinedearlier.Thisfunctionisasamplefunctionthatalsotakesanatom(https://clojuredocs.org/clojure.core/atom)asasecondparameterthatjustcapturesthereceivedmessagesthatcanbereferredtolater.Totakealookatthereceivedmessages,theycanalsobeloggedorpassedontoafunctionasaparametertotakefurtheractiononit.Also,notethatthisfunctionhasanever-endingwhileloop,sothismustbecalledonathreadotherthantheapplicationexecutionthreadtomakesurethattheapplicationexecutionthreadisnotblocked.Totestthefunction,usetheREPL,asshowninthefollowingexample,thatinitializestheconsumerthatconnectstothesameKafkaserverthatwasstartedatthecommandlineusingthekafka-server-start.shscript:

;;initializetherequirednamespaces

helping-hands.alert.server>(require'[helping-hands.alert.channel:aschannel])

helping-hands.alert.server>(require'[helping-hands.alert.config:asconf])

;;initializetheconfigurationforOmniconftopick

helping-hands.alert.server>(conf/init-config{:cli-args[]:quit-on-errortrue})

{:conf#object[java.io.File0x5fa5f9a0"config/conf.edn"],

:kafka

{"bootstrap.servers""localhost:9092",

"group.id""alerts",

"topic""hh_alerts"}}

;;createaconsumer

helping-hands.alert.server>(defconsumer(channel/create-kafka-consumer))

#'helping-hands.alert.server/consumer

;;createanatomtocollecttheresponses

helping-hands.alert.server>(defrecords(atom[]))

#'helping-hands.alert.server/records

;;waitforrecords

helping-hands.alert.server>(channel/consume-recordsconsumerrecords)

Oncetheconsumerislisteningformessages,ontheterminal,startanewproducerforthesamehh_alertstopicthattheconsumerislisteningtousingthekafka-console-producer.shscript,asshowninthefollowingexample.Notethatwedidn'tcreatethetopicbutitisstillavailable.ThereasonisthedefaultconfigurationofKafkaallowsittoauto-createthetopicifitdoesnotexist.Theothertopic,__consumer_offsets,isusedbyKafkatomanagethecommittedoffsets:

%bin/kafka-topics.sh--list--zookeeperlocalhost:2181

__consumer_offsets

hh_alerts

%bin/kafka-console-producer.sh--broker-listlocalhost:9092--topichh_alerts

>SampleAlert

>SampleAlert1

Publishacoupleofmessagesfromtheproducer,asshownintheprecedingexample,andchecktherecordsatominREPL.Itshouldhavereceivedthemessagesasfollows:

;;C-c-binCIDERbreakstheexecutiontogivebackthecontroltoREPL

;;lookupthepublishedmessages

helping-hands.alert.server>(pprint(map#(.value%)@records))

("SampleAlert""SampleAlert1")

helping-hands.alert.server>

InitializingKafkawithMountInthepreviousimplementation,theKafkaconsumerwascreatedbycallingthecreate-kafka-consumerfunction.Insteadofcreatingaconsumerbycallingthefunctionatruntime,itcanbecreatedandmanagedusingMount,asdiscussedinthepreviouschapter.TouseMount,makesurethattheMountdependencyisaddedtotheproject.cljfile,asshownhere:

(defprojecthelping-hands-alert"0.0.1-SNAPSHOT"

:description"HelpingHandsAlertApplication"

;;Omniconf

;;Mount

[mount"0.1.11"]

;;postalforemailalerts

[com.draines/postal"2.0.2"]

;;kafkaclients

[org.apache.kafka/kafka-clients"1.0.0"]

;;logger

:main^{:skip-aottrue}helping-hands.alert.server)

Next,asshowninthefollowingexample,createanamespaceanddefinethestateoftheKafkaconsumerthatcreatesaKafkaconsumertobeusedbytheAlertsmicroserviceandclosesitwhentheserviceisshutdown:

(nshelping-hands.alert.state

"InitializesStateforAlertService"

[helping-hands.alert.channel:asc]))

(defstatealert-consumer

:start(c/create-kafka-consumer)

:stop(.closealert-consumer))

Next,addthestartupandshutdownhooksforMountattheapplicationentrypoints,asshowninthefollowingexamplefortheAlertsmicroservice:

(nshelping-hands.alert.server

[helping-hands.alert.config:ascfg]

[helping-hands.alert.service:asservice]))

(defnrun-dev

[&args]

;;initializestate

(mount/start)

;;Addshutdown-hook

(.addShutdownHook

server/start))

(defn-main

[&args]

;;initializestate

(mount/start)

;;Addshutdown-hook

(.addShutdownHook

NotethatOmniconfconfigurationisinitializedbeforeMountisstartedtomakesurethattheconsumerisabletopicktheconfigurationparametersthatarereadbyOmniconf.OnceMountisinitialized,thekafka-consumerstatecanbeuseddirectlyacrossthenamespaces.Forexample,thefollowingREPLsessionshowshowtoinitializetheAlertserviceindevmodeandusethekafka-consumerstatemanagedbyMountthatisstartedwiththedevinstanceoftheservice:

;;startsserviceindevmode

;;alsoinitializesOmniconfand

;;setstheKafkaconsumerstate

helping-hands.alert.server>(defserver(run-dev))

{:alert

{:from"admin@helpinghands.com",

:host"smtp.gmail.com",

:port465,

:ssltrue,

:to"alerts@helpinghands.com",

:user"admin@helpinghands.com",

:creds<SECRET>},

:conf#object[java.io.File0x266875c9"config/conf.edn"],

:kafka

{"bootstrap.servers""localhost:9092",

"group.id""alerts",

"topic""hh_alerts"}}

#'helping-hands.alert.server/server

;;refertotheconsumerstatemanagedbymount

helping-hands.alert.server>(require'[helping-hands.alert.state:refer[alert-

consumer]])

;;createaatomtocapturerecords

helping-hands.alert.server>(defrecords(atom[]))

#'helping-hands.alert.server/records

;;lookformessagestocapture

helping-hands.alert.server>(helping-hands.alert.channel/capture-recordsalert-

consumerrecords)

Now,publishacoupleofmessagesfromthecommand-lineproducer,asshownhere:

>Hello

>HiMount!

Thesamemessagesarecapturedwithintherecordsatom,asshowninthefollowingexample.Atthetimeofshutdown,Mountwillstoptheconsumerandcleanuptheconnection:

;;C-c-binCIDERbreakstheexecutiontogivebackthecontroltoREPL

;;lookupthepublishedmessages

helping-hands.alert.server>(pprint(map#(.value%)@records))

("Hello""HiMount!")

helping-hands.alert.server>

IntegratingtheAlertServicewithKafka

TheAlertmicroserviceoftheHelpingHandsapplicationreceivesthealertmessagesovertheKafkatopicthatisusedtosendthealertinanemail.Tosendanemailassoonasthemessageisreceived,itcanbeintegratedwithintheloopthatlooksforamessagepublishedbytheproducer,asshownhere:

(defnconsume-records

"Consumetherecordsusinggivenconsumer"

[consumerresult]

(whiletrue

(doseq[record(.pollconsumer1000)]

(let[rmsg(jp/parse-string(.valuerecord))

msg(into{}(filter(compsome?val)

{:from(conf/get-config[:alert:from])

:to(getrmsg"to"(conf/get-config[:alert:to]))

:cc(rmsg"cc")

:subject(rmsg"subject")

:body(rmsg"body")}))

result(postal/send-message

{:host(conf/get-config[:alert:host])

:port(conf/get-config[:alert:port])

:ssl(conf/get-config[:alert:ssl])

:user(conf/get-config[:alert:user])

:pass(conf/get-config[:alert:creds])}

msg)])

(catchExceptione

(log/error"Failedtosendemail"e)))

(swap!resultconjrecord))

(Thread/sleep5000)))

Intheprecedingfunction,oncethemessageisreceived,itisparsedtogettheJSONwiththekeyssuchasto,cc,subject,andbodythatareusedtocreateanemailandsenditusingthePostallibrary(https://github.com/drewr/postal)thatwasdiscussedinpreviouschapters.Alltheexceptionsarecaughtandloggedforreview.Inthiscase,theproducerpublishesastringifiedJSONwiththerequiredkeys,asshownhere:

>{"to":"admin@helpinghands.com","subject":"UsageAlert","body":"Usagealertexceeded

threshold100kreq/sec"}

UsingAvrofordatatransfer

ThekeysandvaluespublishedonaKafkatopicmusthaveassociatedSerDes(https://en.wikipedia.org/wiki/SerDes).TheexamplesusedintheprevioussectionusedLongDeserializerforkeysandStringDeserializerfortheKafkaConsumer.Similarly,theKafkaProducerforthecorrespondingconsumerwilluseLongSerializerandStringSerializertopublishthekeyandvalue,respectively.SincemicroservicesmaybewritteninanyprogramminglanguageandmayneedtocollaboratewithotherservicesoverKafkatopics,thelanguage-dependentSerDesisnotagoodoption.

Avro(https://avro.apache.org/)isadataserializationformatthatislanguageagnosticandhassupportformostofthewell-knownprogramminglanguages(https://cwiki.apache.org/confluence/display/AVRO/Supported+Languages).Avrohasitsowndeclarativewayofdefiningtheschema(https://avro.apache.org/docs/current/)thatcanbemappedtothebusinessmodeldescribingtheentity.Oncetheschemaisdefined,themessageisencodedagainsttheschemaattheproducerendandthendecodedattheconsumerendusingthesameschema.Asfarastheschemaisaccessibletobothproducerandconsumer,theycancommunicateviaAvromessagesirrespectiveoftheprogramminglanguagetheyareimplementedin.

AvroClojurelibraryabracad(https://github.com/damballa/abracad)isawrapperoverAvroAPIsthatintegrateswellwiththeapplicationswritteninClojure.Touseabracad,includethedependency[com.damballa/abracad"0.4.13"]intheproject.cljfile.Oncethedependenciesareavailable,theSerDesforAvrocanbedefinedasshowninthefollowingexample,andcanbeusedinsteadofStringSerDesbytheKafkaproducerandconsumer:;;adoptedfromfranzy-avroproject;;https://github.com/ymilky/franzy-avro(deftypeKafkaAvroSerializer[schema]

Serializer

(configure[___])

(serialize[__data]

(whendata

(avro/binary-encodedschemadata)))(close[_]))

(deftypeKafkaAvroDeserializer[schema]

Deserializer

(configure[___])

(deserialize[__data]

(whendata

(avro/decodeschemadata)))(close[_]))

(defnkafka-avro-serializer"AvroserializerforApacheKafka.UseforserializingKafkakeysvalues.

Valueswillbeserializedaccordingtotheprovidedschema.

Ifnoschemaisprovided,adefaultEDNschemaisassumed.

Seehttps://avro.apache.org/

Seehttps://github.com/damballa/abracad"

[schema]

(KafkaAvroSerializer.(orschema(aedn/new-schema))))

(defnkafka-avro-deserializer"AvrodeserializerforApacheKafka.

UsefordeserializingKafkakeysandvalues.

Ifnoschemaisprovided,adefaultEDNschemaisassumed.

Seehttps://avro.apache.org/

Seehttps://github.com/damballa/abracad"

[schema]

(KafkaAvroDeserializer.(orschema(aedn/new-schema))))

SummaryInthischapter,welearnedabouttheimportanceofevent-drivenpatternsformicroservicesandhowwecanuseApacheKafkaasamessagebrokertobuildascalableanddurableevent-drivenarchitecture.Event-drivenarchitecturesarescalable,buttheyareincrediblyhardtodebugforissueswithoutbeingmonitoredinrealtime.Inthenextchapter,wewilllearnhowtosecuremicroservicesanddeploytheminproductionwithareal-timemonitoringsystem.

DeployingandMonitoringSecuredMicroservices

"Thesuccessofaproductiondependsontheattentionpaidtodetail."

-DavidO.Selznick

Microservicesmustbedeployedinisolationandmonitoredforusage.Monitoringthecurrentworkloadandprocessingtimealsohelpstotakeadecisiononwhentoscalethemuporscalethemdown.Anotherimportantaspectofmicroservices-basedarchitectureissecurity.Onewaytosecuremicroservicesistoalloweachoneofthemtohavetheirownauthenticationandauthorizationmodule.Thisapproachsoonbecomesaproblem,aseachmicroserviceisdeployedinisolation,anditbecomesincrediblyhardtoagreeoncommonstandardstoauthorizeauser.Also,inthiscase,theownershipofusersandtheirrolesgetsdistributedacrosstheservices.Thischapteraddressessuchissuesandprovidessolutionstosecure,monitor,andscalemicroservices-basedapplications.Inthischapter,youwilllearnthefollowingthings:

HowtoenableauthenticationandauthorizationformicroservicesHowtouseJSONWebToken(JWT)andJSONWebEncryption(JWE)HowtocreateanauthenticationservicethatworkswithJSONWebTokensHowtocaptureauditlogsandruntimemetricsforreal-timemonitoringHowtodeploymicroservicesusingDockercontainersWhyKubernetesisusefulformicroservices-baseddeployments

EnablingauthenticationandauthorizationAuthenticationistheprocessofidentifyingwhotheuseris,whereasauthorizationistheprocessofverifyingwhattheauthenticateduserhasaccessto.Themostcommonwayofachievingauthenticationisbyaskinguserstospecifytheirusernameandpasswordthatcanthenbevalidatedagainstthebackenddatabaseofusercredentials.

Thepasswordsshouldneverbestoredinplaintextinthebackenddatabase.Itisrecommendedtocomputeaone-wayhashofthepasswordandstorethatinstead.Toresetthepassword,thesystemcanjustgeneratearandompassword,storeitshash,andsharetherandompasswordinplaintextwiththeuser.Alternatively,auniqueURLcanbesenttotheusertoresetthepasswordthroughaformthatcanvalidateauser'sidentityviamethodssuchaspresetquestionsandanswersandone-timepassword(OTP).

Authenticatingtheusersisnotenoughforanapplicationiftheapplicationhasmultiplesecurityboundaries.Forexample,anapplicationmayrequireonlycertainuserstosendnotificationthroughthesystemandpreventallothers.Todoso,theapplicationmustcreateasecurityboundaryforitsresourcesthatisoftendefinedusingrolesthathaveoneormorepermissionsthatcanbevalidatedbytheapplicationbeforeallowingaccesstoitsresourcesandfeatures,suchasnotification.Rolesandpermissionsarethekeyfactorsofauthorizationthatallowanapplicationtocreatemultiplesecurityboundariesforitsresources.

IntroducingTokensandJWTInamonolithicenvironment,authenticationandauthorizationarehandledwithinthesameapplicationusingamodulethatvalidatestheincomingrequestsforrequiredauthenticationandauthorizationinformation,asshowninthefollowingdiagram;thismodulealsoallowstheauthorizeduserstodefinetherolesandpermissionsandassignthemtootherusersinthesystemtoallowthemaccesstothesecuredresources:

Monolithicapplicationsmayalsomaintainasessionstoreagainstwhicheachinstanceofthemonolithicapplicationcanvalidatetheincomingrequestanddetermineavalidsessionfortheuser.Often,suchsessioninformationisstoredinacookiethatissenttotheclientasatokenoncetheuserissuccessfullyauthenticated.Thecookieisthenattachedtoeachrequestbytheclientthatcanthenbevalidatedbytheserverforavalidsessionandassociatedrolestodeterminewhethertoallowordisallowaccesstotherequestedresource,asshownintheprecedingdiagram.

Inamicroservices-basedapplication,eachmicroserviceisdeployedinisolationandmustnothavetheresponsibilityofmaintainingaseparateuserdatabaseorsessiondatabase.Moreover,theremustbeastandardwayofauthenticatingandauthorizingtheusersacrossmicroservices.ItisrecommendedtoseparateouttheauthenticationandauthorizationresponsibilityasaseparateAuthservicethatcanowntheuserdatabasetoauthenticateandauthorizeusers.ThisalsohelpsinauthenticatingtheuseronceviaAuthserviceandthenauthorizingthemto

accesstheresourcesandrelatedservicesthroughothermicroservices.

SinceeachmicroservicemayuseitsowntechnologystackandhavenopriorknowledgeofAuthservice,thereshouldbeacommonstandardtovalidatetheauthenticatedusersacrossMicroservices.JSONWebTokens(JWT)isonesuchstandardthatconsistsofaheader,payload,andsignaturethatcanbeissuedasatokentotheuseraftersuccessfulauthentication.Userscanthensendthistokenwitheachrequesttoanymicroservicethatcanthenvalidateitandgrantaccesstotherequestedresources.

JWTcaneitherhavethecontentencryptedorsecuredusingdigitalsignatureormessageauthenticationcodes.JSONWebSignature(JWS)representsthecontentsecuredwithdigitalsignaturesorMessageAuthenticationCodes(MACs),whereasJSONWebEncryption(JWE)representstheencryptedcontentusingJSON-baseddatastructures.Ifthetokenisencrypted,itcanonlybereadwithakeythatwasusedtoencryptthetoken.ToreadaJWEtoken,servicesmustownthekeythatwasusedtoencryptthetoken.Insteadofsharingthekeyacrossmicroservices,itisrecommendedtosendthetokentotheAuthservicedirectlytodecryptthetokenandauthorizetherequestonbehalfoftheservice.ThismayresultinaperformancebottleneckandsinglepointoffailureduetoeachservicetryingtogetAuthservicefirstforauthorization.Thiscanbepreventedbycachingtheprevalidatedtokensateachmicroservicelevelforaconfigurableamountoftimethatcanbedecidedbasedontheexpirytimeofthe

token.

ExpirytimeisanimportantcriteriawhileworkingwithJWTs.JWTswithaverylargeexpirytimemustbeavoided,asthereisnowayfortheapplicationtologouttheuserorinvalidatethetoken.Anissuedtokenremainsvalidunlessanduntilitexpires.Asfarastheuserownsavalidtoken,theyareallowedtogainaccesstotheserviceswiththeissuedtoken.Topreventtheissueoflogout,oneoptionistoletmicroservicesalwaysvalidateatokenwiththeAuthservicethatmaintainsacacheofuserauthorizationdetailsthatarekeptinsyncwiththeuser'srolesandpermissions.EverytimeanAuthservicereceivesatoken,itcanvalidateitagainstthiscache,andifthereischangeinuserrolesoranyotherproperties,itcaninvalidatethetokenthatwillforcetheusertorequestforanewtoken,andthattokenwillnowhavetheupdatedrolesandtheauthorizationdetails.

Formoredetails,refertoJWTRFC-7519(https://tools.ietf.org/html/rfc7519),JWSRFC-7515(https://tools.ietf.org/html/rfc7515),andJWERFC-7516(https://tools.ietf.org/search/rfc7516).

CreatinganAuthserviceforHelpingHands

TheAuthserviceforHelpingHandscanbebuiltusingthesamepedestalprojecttemplateasthatofothermicroservicesofHelpingHands.Inthisexample,itusesJWEtocreateJWTtokensfortheusers.Tostartwith,createanewprojectwiththedirectorystructureasshowninthefollowingexample;itcontainsanewnamespacehelping-hands.auth.jwtthatcontainstheimplementationrelatedtoJWT—therestofthenamespacesareusedasdescribedintheprecedingchapters.

├──config

│├──conf.edn

├──Dockerfile

├──README.md

├──resources

├──src

│├──clj

││└──helping_hands

││└──auth

││├──config.clj

││├──core.clj

││├──jwt.clj

││├──persistence.clj

││├──server.clj

││├──service.clj

││└──state.clj

│└──jvm

└──test

├──clj

│└──helping_hands

│└──auth

│├──core_test.clj

│└──service_test.clj

└──jvm

UsingaNimbusJOSEJWTlibraryforTokens

TheAuthserviceprojectwilladditionallyuseaNimbus-JOSE-JWTlibrary(https://bitbucket.org/connect2id/nimbus-jose-jwt/wiki/Home)tocreateandvalidateJSONWebTokensandapermissions(https://github.com/tuhlmann/permissions)librarytoauthorizeusersagainstasetofrolesandpermissions.AddtheNimbus-JOSE-JWTandpermissionslibrarydependencies,asshowninthefollowingproject.cljfile:

(defprojecthelping-hands-auth"0.0.1-SNAPSHOT"

:description"HelpingHandsAuthService"

;;Omniconf

;;Mount

[mount"0.1.11"]

;;nimbus-joseforJWT

[com.nimbusds/nimbus-jose-jwt"5.4"]

;;usedforrolesandpermissions

[agynamix/permissions"0.2.2-SNAPSHOT"]

;;logger

[ch.qos.logback/logback-classic"1.1.8"

:exclusions[org.slf4j/slf4j-api]]

;;CodeCoverage

;;Unittestdocs

:dev{:aliases

{"run-dev"["trampoline""run""-m"

"helping-hands.auth.server/run-dev"]}

:dependencies

[[io.pedestal/pedestal.service-tools"0.5.3"]]

:uberjar{:aot[helping-hands.auth.server]}

:themes[:rdash]}}

:debug{:jvm-opts

:main^{:skip-aottrue}helping-hands.auth.server)

CreatingasecretkeyforJSONWebEncryptionTostartwiththeimplementationofJWTwithencryptedclaims,firstcreateaget-secretfunctiontogenerateasecretkeyforencryption.Also,addaget-secret-jwkfunctionthatisusedtocreateaJSONWebKey(https://tools.ietf.org/html/rfc7517)usingthesecretkeygeneratedbytheget-secretfunction,asshowninthefollowingcode:

(nshelping-hands.auth.jwt

"JWTImplementationforAuthService"

(:require[cheshire.core:asjp])

(:import[com.nimbusds.joseEncryptionMethod

JWEAlgorithmJWSAlgorithm

JWEDecrypterJWEEncrypter

JWEHeader$BuilderJWEObjectPayload]

[com.nimbusds.jose.crypto

AESDecrypterAESEncrypter]

[com.nimbusds.jose.jwkKeyOperationKeyUse

OctetSequenceKeyOctetSequenceKey$Builder]

[com.nimbusds.jwtJWTClaimsSetJWTClaimsSet$Builder]

[com.nimbusds.jwt.procDefaultJWTClaimsVerifier]

[com.nimbusds.jose.utilBase64URL]

[java.utilDate]

[javax.cryptoKeyGenerator]

[javax.crypto.specSecretKeySpec]))

(def^:conskhash-256"SHA-256")

(defonce^:privatekgen-aes-128

(let[keygen(KeyGenerator/getInstance"AES")

_(.initkeygen128)]

keygen))

(defonce^:privatealg-a128kw

(JWEAlgorithm/A128KW))

(defonce^:privateenc-a128cbc_hs256

(EncryptionMethod/A128CBC_HS256))

(defnget-secret

"Getsthesecretkey"

([](get-secretkgen-aes-128))

([kgen]

;;mustbecreatediffthekeyhasn't

;;beencreaedearlier.Createonceand

;;persistinanexternaldatabase

(.generateKeykgen)))

(defnget-secret-jwk

"GeneratesanewJSONWebKey(JWK)"

[{:keys[khashkgenalg]:asenc-impl}secret]

;;mustbecreatediffthekeyhasn't

;;beencreaedearlier.Createonceand

;;persistinanexternaldatabase

(..(OctetSequenceKey$Builder.secret)

(keyIDFromThumbprint(orkhashkhash-256))

(algorithm(oralgalg-a128kw))

(keyUse(KeyUse/ENCRYPTION))

(build)))

TheprecedingimplementationshowngeneratesakeyusingtheAES128-bitalgorithm.Thesecretkeygeneratedbytheget-secretfunctionmustbegeneratedonlyonceforthelifetimeoftheapplication.Therefore,itisrecommendedtostoreitinanexternaldatabasethatcanbesharedamongtheinstancesoftheAuthserviceonceitisscaledtomorethanoneinstance.

Nimbus-JOSE-JWTalsosupports256-bitalgorithms.For256-bitalgorithmstowork,JREneedsexplicitJavaCryptographyExtension(JCE)UnlimitedStrengthJurisdictionPolicyFiles(http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html).

Theget-secret-jwkfunctiontakesthesecretkeyasoneofitsinputparametersandgeneratesaJWK,asshowninthefollowingREPLsession;JWKconsistsofaKeyType(kty),PublicKeyUse(use),KeyID(kid),KeyValue(k),andAlgorithm(alg)parametersthataredefinedinJWKRFC-7517(https://tools.ietf.org/html/rfc7517):

;;requirethenamespace

helping-hands.auth.server>(require'[helping-hands.auth.jwt:asjwt])

;;createasecretkey

helping-hands.auth.server>(defsecret(jwt/get-secret))

#'helping-hands.auth.server/secret

;;createaJSONWebKey

helping-hands.auth.server>(defjwk(jwt/get-secret-jwk{}secret))

#'helping-hands.auth.server/jwk

;;dumptheJSONobjectofJWK

helping-hands.auth.server>(.toJSONObjectjwk)

{"kty""oct","use""enc","kid""F5UNJYT4A-GpngZwRMYfs8ZuCKsmRGt08Xo_dMQrY5w","k"

"CvTaCBfdEkAlXfuOnW7pnw","alg""A128KW"}

SinceJWKisjustarepresentationofthesecretkeyinaJSONformat,thesecretkeycanberetrievedfromtheJWKusingautilityfunction,enckey->secret,asshowninthefollowingimplementation:

(defnenckey->secret

"ConvertsJSONWebKey(JWK)tothesecretkey"

[{:keys[kkidalg]:asenc-key}]

(..(OctetSequenceKey$Builder.k)

(keyIDkid)

(algorithm(oralgalg-a128kw))

(keyUse(KeyUse/ENCRYPTION))

(build)

(toSecretKey"AES")))

Theenckey->secretfunctiontakesKeyID(kid)andKeyValue(k)asitsinputtocreatethesecretkeythatissameastheoneusedtocreatethesourceJSONWebKey.ThealgparameterisoptionalandfallsbacktothedefaultAES-128algorithmifitisnotspecified.ThefollowingREPLsessionshowshowtocreateasecretkeyfromJWKgeneratedearlierandvalidatethatitalwaysgeneratesthesameJWK:

;;JSONWebKey(JWK)generatedearlier

;;extractthesecretkey

helping-hands.auth.server>(defsecret-extracted(jwt/enckey->secret{:k(.getKeyValue

jwk):kid(.getKeyIDjwk)}))

#'helping-hands.auth.server/secret-extracted

;;generateJSONWebKeythatisexactlysameassource

helping-hands.auth.server>(.toJSONObject(jwt/get-secret-jwk{}secret-extracted))

CreatingTokensThenextstepistodefinethefunctionstocreateandreadJWT.SincetheJWTusedfortheHelpingHandsapplicationusesJWEtoencrypttheclaims,itisOKtoaddbothuserIDandrolesinformationwithinthepayloadthatcanbelaterretrievedfromavalidtokentoauthorizetheuser.

Thecreate-tokenandread-tokenfunctionsshowninthefollowingexampleprovideawaytocreateaJSONWebTokenandreadanexistingone,respectively.Thecreate-tokenfunctionusesautilityfunction—create-payload—tocreatetheclaimsetandthepayloadofJWT.ClaimsetsthatarerelevantforthecurrentexampleareissueTimethatdefinestheepochtimeofwhenthistokenwascreated,expirationTimethatsetsthetimebeyondwhichthetokenwillbeconsideredasexpired,anduserandrolescustomclaimsthatstoretheauthenticatedusernameandtherolesassignedtotheuseratthetimeofissuingthetoken.Formoredetailsontheavailableclaimsetoptions,takealookatJWTRFC-7519(https://tools.ietf.org/html/rfc7519).

(defn-create-payload

"CreatesapayloadasJWTClaims"

[{:keys[userroles]:asparams}]

(let[ts(System/currentTimeMillis)

claims(..(JWTClaimsSet$Builder.)

(issuer"Packt")

(subject"HelpingHands")

(audience"https://www.packtpub.com")

(issueTime(Date.ts))

(expirationTime(Date.(+ts120000)))

(claim"user"user)

(claim"roles"roles)

(build))]

(.toJSONObjectclaims)))

(defncreate-token

"Createsanewtokenwiththegivenpayload"

[{:keys[userrolesalgenc]:asparams}secret]

(let[enckey(get-secret-jwkparamssecret)

payload(create-payload{:useruser:rolesroles})

passphrase(JWEObject.

(..(JWEHeader$Builder.

(oralgalg-a128kw)

(orencenc-a128cbc_hs256))

(build))

(Payload.payload))

encrypter(AESEncrypter.enckey)

_(.encryptpassphraseencrypter)]

(.serializepassphrase)))

(defnread-token

"Decryptsthegiventokenwiththesaidalgorithm

ThrowsBadJWTExceptionistokenisinvalidorexpired"

[tokensecret]

(let[passphrase(JWEObject/parsetoken)

decrypter(AESDecrypter.secret)

_(.decryptpassphrasedecrypter)

payload(..passphrasegetPayloadtoString)

claims(JWTClaimsSet/parsepayload)

;;throwsexceptionifthetokenisinvalid

_(.verify(DefaultJWTClaimsVerifier.)claims)]

(jp/parse-stringpayload)))

ThefollowingREPLsessionshowsthestepstocreateandreadatokenandlaterwaitforittogetexpired.Notetheexceptionthrownbythelibrarythatcanbecapturedtomarktheeventoftokenexpiry:

;;generateanewtokenwiththeuserandroles

helping-hands.auth.server>(deftoken(jwt/create-token{:user"hhuser":roles#

{"hh/notify"}}secret))

#'helping-hands.auth.server/token

;;dumpthecompactserializationstring

helping-hands.auth.server>token

"eyJlbmMiOiJBMTI4Q0JDLUhTMjU2IiwiYWxnIjoiQTEyOEtXIn0.FiAelEg_R8We8xEF2xRxcC908BCoH1nRYvY3nV_jkqYO8JPp-

QukBw.86-

JKq6cYFH2rtFBOXiA6A.Pxz3ZzBGKX2Cd_sjtYdEwKDltzKQiolWSvrjPbLLGL8NlShcWWEIqkd7NL2WcXHukDa6zS4ANIWnee2hNWUraItqZFEY6N_RhXZVVXQvZJsqzeiueBxvxc1fj1LFUKsyR63oOwLd5ZIIT99ItrqaYPM88enMsjchsXYBJ_Tcb-

WR6R_KirmDBxCVjqFcg7OdWjjcKTP4FcUNIQU9G8fSnQ.pfLyW8ggXV8vQnidytJmMw"

;;readthetokenback

helping-hands.auth.server>(pprint(jwt/read-tokentokensecret))

{"sub""HelpingHands",

"aud""https://www.packtpub.com",

"roles"["hh/notify"],

"iss""Packt",

"exp"1515959756,

"iat"1515959636,

"user""hhuser"}

;;waitfor2mins(expirytimeasperimplementation)

;;tokenisnowexpired

helping-hands.auth.server>(pprint(jwt/read-tokentokensecret))

BadJWTExceptionExpiredJWTcom.nimbusds.jwt.proc.DefaultJWTClaimsVerifier.<clinit>

(DefaultJWTClaimsVerifier.java:62)

EnablingusersandrolesforauthorizationIdeally,theAuthservicemustbebackedbyapersistentstoretokeeptheusers,roles,andthesecretkeyfortheapplication.Forthesakeofsimplicityoftheexample,createasamplein-memorydatabaseinthehelping-hands.auth.persistencenamespace,asfollows:

(nshelping-hands.auth.persistence

"PersistenceImplementationforAuthService"

(:require[agynamix.roles:asr]

[cheshire.core:asjp])

(:import[java.securityMessageDigest]))

(defnget-hash

"CreatesaMD5hashofthepassword"

[creds]

(..(MessageDigest/getInstance"MD5")

(digest(.getBytescreds"UTF-8"))))

(defuserdb

;;Usedonyfordemonstration

;;TODOPersistinanexternaldatabase

{:secretnil

:roles{"hh/superadmin""*"

"hh/admin""hh:*"

"hh/notify"#{"hh:notify""notify/alert"}

"notify/alert"#{"notify:email""notify:sms"}}

:users{"hhuser"{:pwd(get-hash"hhuser")

:roles#{"hh/notify"}}

"hhadmin"{:pwd(get-hash"hhadmin")

:roles#{"hh/admin"}}

"superadmin"{:pwd(get-hash"superadmin")

:roles#{"hh/superadmin"}}}}))

(defnhas-access?

"Checksforrelevantpermission"

[uidperms]

(r/has-permission?

(->@userdb:users(getuid))

:roles:permissionsperms))

(defninit-db

"Initializestherolesforpermissionframework"

(r/init-roles(:roles@userdb))

userdb)

userdbcontainsthesamplein-memorydatabasethatconsistsofthe:secretkeythatisinitializedtoniland:usersand:rolesthatcontaininformationontheusers

androles,respectively.Roledefinitionfollowstheguidelinesofthepermissionlibraryanddefinestherolesandpermissionsaspertheusageinstructions(https://github.com/tuhlmann/permissions#usage)ofthelibrary.Roleshaveaslash,/,intheirnameandpermissionshaveacolon,:,asdefinedintheprecedingroledefinition.Roledefinitionsarerecursive,andonerolecanencapsulatebothrolesandpermissions.

Theinit-dbfunctionisusedtoinitializethedatabaseandtheroledefinitions.Thehas-access?isautilityfunctionthatcanbeusedtovalidatewhetherausercontainsagivensetofpermissionsornot.ThefollowingREPLsessiondescribestheuseofthehas-access?functionwithanexample:

;;requirethepersistencenamespace

helping-hands.auth.server>(require'[helping-hands.auth.persistence:asp])

;;sincethereisnosecretkeydefine,

;;initializethedatabasewithasecret-key

;;ifitdoesnotexist

helping-hands.auth.server>(let[db(p/init-db)]

;;ifkeydoesnotexist,initializeone

;;andupdatethedatabasewith:secretkey

(if-not(:secret@db)

(swap!db#(assoc%:secret(jwt/get-secret)))@db))

{:secret#object[javax.crypto.spec.SecretKeySpec0xebc150b

"javax.crypto.spec.SecretKeySpec@17ce8"],:roles{"hh/superadmin""*","hh/admin"

"hh:*","hh/notify"#{"notify/alert""hh:notify"},"notify/alert"#{"notify:email"

"notify:sms"}},:users{"hhuser"{:pwd#object["[B"0x1b46ced7"[B@1b46ced7"],:roles

#{"hh/notify"}},"hhadmin"{:pwd#object["[B"0x7b9083e6"[B@7b9083e6"],:roles#

{"hh/admin"}},"superadmin"{:pwd#object["[B"0x64083ac1"[B@64083ac1"],:roles#

{"hh/superadmin"}}}}

;;validatethat`hhuser`hasthe``hh:notify``permission

helping-hands.auth.server>(p/has-access?"hhuser"#{"hh:notify"})

;;validatepermissionsthatarenotdefined

helping-hands.auth.server>(p/has-access?"hhuser"#{"hh:admin"})

;;validatepermissionsthatareobtainedbyotherrolereferences

helping-hands.auth.server>(p/has-access?"hhuser"#{"hh:notify""notify:email"})

TheprecedingexampleexplicitlyinitializesthedatabaseatREPLandsetsthesecretkey.Insteadofexplicitlyinitializingthedatabase,itcanbedoneatthestartupitselfusingmount,asdiscussedinChapter9,ConfiguringMicroservices.Toallowmounttoinitializethestateofthedatabasewiththesecretkeyandmakeitavailableforothernamespaces,definethedatabasestateinthehelping-hands.auth.statenamespace,asfollows:

(nshelping-hands.auth.state

"InitializesStateforAuthService"

[helping-hands.auth.jwt:asjwt]

[helping-hands.auth.persistence:asp]))

(defstateauth-db

:start(let[db(p/init-db)]

;;ifkeydoesnotexist,initializeone

;;andupdatethedatabasewith:secretkey

(if-not(:secret@db)

(swap!db#(assoc%:secret(jwt/get-secret)))@db))

:stopnil)

Next,enablethestartandstopeventsbyaddingmount/startandmount/stopfunctionstotheserverstartupfunctionsinthehelping-hands.auth.servernamespace,asshowninthefollowingexample:

(nshelping-hands.auth.server

[helping-hands.auth.config:ascfg]

[helping-hands.auth.service:asservice]))

;;Thisisanadaptedservicemap,thatcanbestartedandstopped

;;FromtheREPLyoucancallserver/startandserver/stoponthisservice

(defoncerunnable-service(server/create-serverservice/service))

(defnrun-dev

[&args]

;;initializestate

(mount/start)

;;Addshutdown-hook

(.addShutdownHook

server/start))

(defn-main

[&args]

;;initializestate

(mount/start)

;;Addshutdown-hook

(.addShutdownHook

CreatingAuthAPIsusingPedestalThenextstepistodefineAPIsfortheAuthservicetoauthenticateandauthorizeusers.Addthe/tokensand/tokens/validateroutestothehelping-hands.auth.servicenamespace,asfollows:

(nshelping-hands.auth.service

(:require[helping-hands.auth.core:ascore]

[cheshire.core:asjp]

;;Defines"/"and"/about"routeswiththeirassociated:gethandlers.

;;Theinterceptorsdefinedaftertheverbmap(e.g.,{:gethome-page}

;;applyto/anditschildren(/about).

;;Tabularroutes

(defroutes#{["/tokens"

:get(conjcommon-interceptors

`core/validate`core/get-token)

:route-name:token-get]

["/tokens/validate"

:post(conjcommon-interceptors

`core/validate`core/validate-token)

:route-name:token-validate]})

;;Seehttp/default-interceptorsforadditionaloptionsyoucanconfigure

::http/routesroutes

::http/type:jetty

::http/port8080

;;Optionstopasstothecontainer(Jetty)

::http/container-options{:h2c?true

:h2?false

:ssl?false}})

TheGET/tokensroutelooksforuidandpwdparametersoravalidauthorizationheadertoprocesstherequest.Iftheuidandpwdparametersarespecifiedandtheyarevalid,aJWTtokenisissuedaspartoftheauthorizationheader.IfanexistingJWTisspecifiedasapartoftheauthorizationheaderintherequest,Authservicereturnstheusernameandtherolesassociatedwithit.

ThePOST/tokens/validaterouteexpectsaformparameter—perms—andavalidauthorizationheaderwithJWTtoauthorizetheuseragainstthegiven

permissions.ThisendpointisusedbyothermicroservicesoftheHelpingHandsapplicationtoauthorizetheuseragainstthepermissionsrequiredbythemicroservicestoprovideaccesstotheresourcesthatitmanages.Sincepermissionsandrolesaredefinedasstrings,administratorscaninitializetheAuthdatabasewithalltheexpectedrolesandpermissionsandassignthemtouserstoallowordisallowaccesstoservicesoftheapplication.

Theinterceptorsusedfortheroutesdefinedintheprecedingcodesnippetareimplementedinthehelping-hands.auth.corenamespace,asshowninthefollowingexample;thevalidateinterceptorpreparesthe:tx-dataparameterwithalltheavailablerequestparametersandalsovalidatesthepresenceofeitheruidandpwdoranauthorizationheader—ifoneofthemdoesnotexist,itreturnsaHTTP400BadRequestresponse:

(nshelping-hands.auth.core

"InitializesHelpingHandsAuthService"

[helping-hands.auth.jwt:asjwt]

[helping-hands.auth.persistence:asp]

[helping-hands.auth.state:refer[auth-db]]

(:import[com.nimbusds.jwt.procBadJWTException]

[java.ioIOException]

[java.textParseException]

[java.utilArraysUUID]))

;;--------------------------------

[context]

(->context:request:headers)

(if-let[pparams(->context:request:path-params)]

(if(empty?pparams){}pparams)))]

(if(or(and(params:uid)(params:pwd))

(params"authorization"))

(assoccontext:tx-dataparams)

(chain/terminate

(assoccontext

:body"InvalidCreds/Token"})))))

(defvalidate

{:name::validate

:enter

(fn[context]

(prepare-valid-contextcontext))

:error

(fn[contextex-info]

(assoccontext

Theget-tokeninterceptorlooksforavaliduidandpwd,andissuesaJWTifauthenticationissuccessful.Iftheuidandpwdarenotpresent,itlooksforavalidauthorizationheaderofaBearertypeand,ifthetokenisvalid,itreturnstheauthenticateduserIDandassignedrolesthatareassociatedwiththeuser:

(defn-extract-token

"Extractsuserandrolesmapfromtheauthheader"

[auth]

(select-keys

(jwt/read-token

(second(s/splitauth#"\s+"))(auth-db:secret))

["user""roles"]))

(defget-token

{:name::token-get

:enter

(fn[context]

uid(:uidtx-data)

pwd(:pwdtx-data)

auth(tx-data"authorization")]

(anduidpwd(Arrays/equals

(->auth-db:users(getuid):pwd)

(p/get-hashpwd)))

(let[token(jwt/create-token

{:roles(->auth-db:users(getuid):roles)

:useruid}(auth-db:secret))]

{:status200

:headers{"authorization"(str"Bearer"token)}}))

(andauth(="Bearer"(->(s/splitauth#"\s+")first)))

{:status200

:body(jp/generate-string(extract-tokenauth))})

(catchBadJWTExceptione

{:status401:body"Tokenexpired"})))

:else(assoccontext:response{:status401}))))

:error

(fn[contextex-info]

(assoccontext

Theimplementationofthevalidate-tokeninterceptorshowninthefollowingexampleauthorizestheuserassociatedwiththeJWTsentasanauthorization

headerandaCSVofpermissionsspecifiedasthepermsformparameter:

(defvalidate-token

{:name::token-validate

:enter

(fn[context]

auth(tx-data"authorization")

perms(if-let[p(tx-data:perms)]

(into#{}(maps/trim(s/splitp#","))))]

(if(andauth(="Bearer"(->(s/splitauth#"\s+")first)))

(if(p/has-access?((extract-tokenauth)"user")perms)

(assoccontext:response{:status200:body"true"})

(assoccontext:response{:status200:body"false"}))

(catchBadJWTExceptione

{:status401:body"Tokenexpired"}))

(catchParseExceptione

{:status401:body"InvalidJWT"})))

(assoccontext:response{:status401}))))

:error

(fn[contextex-info]

(assoccontext

Totesttheroutes,starttheAuthserviceusingtheleinruncommandorstartitwithinaREPLasshowninthefollowingexample;assoonastheapplicationisstarted,mountkicksinandinitializesasecretkeythatisusedtoissuetokensandalsoreadthemforauthorization:

helping-hands.auth.server>(defserver(run-dev))

{:conf#object[java.io.File0x979c2d2"config/conf.edn"]}

#'helping-hands.auth.server/server

helping-hands.auth.server>

Oncetheserverisupandrunning,usecURLtotryoutvariousscenarios,asshowninthefollowingexample.Iftherearenoauthenticationheadersorvalidcredentialsspecified,thevalidateinterceptorwillkickinandmarkitasabadrequest,asfollows:

%curl-i"http://localhost:8080/tokens"

Date:Sun,14Jan201820:49:48GMT

InvalidCreds/Token

%curl-i-XPOST-d"perms=notify:email""http://localhost:8080/tokens/validate"

InvalidCreds/Token

Ifthespecifiedcredentialsareinvalid,itwillthrowaresponsewithHTTP401Unauthorizedstatus,asshowninthefollowingexample:

%curl-i"http://localhost:8080/tokens?uid=hhuser&pwd=hello"

%curl-i-H"Authorization:Bearerabc"-XPOST-d"perms=notify:email"

"http://localhost:8080/tokens/validate"

InvalidJWT

Iftheparametersarevalid,endpointsworkasexpected,asshownforthehhuseruser:

%curl-i"http://localhost:8080/tokens?uid=hhuser&pwd=hhuser"

HTTP/1.1200OK

Authorization:Bearer

eyJlbmMiOiJBMTI4Q0JDLUhTMjU2IiwiYWxnIjoiQTEyOEtXIn0.YY_dMY8qoqfTHeZwGacsFY70tCaUvCjjPKYNFhuA2ppOD-

Deaj5zzw.BbK36SSYyuUeVVS9jpyIXw.6UNkLFMVF5Foj5qFX5vLdKcyOoU2G2eeVtHskSWBoZuBnnAwI1NGrPc3PvQqKF4QkzlrbFfOYD2Vxd4YqYmj8Hcb1qVUQD1QgtYKiStIMujH-

-ZRltPfy7m8VW1D31ToeqAYU1LLlXYSC1W3kSjZQiMFMU1LXkMqZVdmJyfQIL_SvizfWbYuZQPcyDCxG5-

XtVeG2r09vnvUybw8tKdafg.WML7xCZ-lZur1GXpNFNKrw

Transfer-Encoding:chunked

%curl-i-H"Authorization:Bearer

XtVeG2r09vnvUybw8tKdafg.WML7xCZ-lZur1GXpNFNKrw""http://localhost:8080/tokens"

HTTP/1.1200OK

{"user":"hhuser","roles":["hh/notify"]}

%curl-XPOST-i-H"Authorization:Bearer

XtVeG2r09vnvUybw8tKdafg.WML7xCZ-lZur1GXpNFNKrw"-d"perms=notify:email"

HTTP/1.1200OK

XtVeG2r09vnvUybw8tKdafg.WML7xCZ-lZur1GXpNFNKrw"-d"perms=notify:random"

HTTP/1.1200OK

XtVeG2r09vnvUybw8tKdafg.WML7xCZ-lZur1GXpNFNKrw"-d"perms=notify:email"

Tokenexpired

AuthservicecanbedeployedinisolationandconnectedthroughtheauthinterceptorofrestoftheservicesoftheHelpingHandsapplicationtoauthorizetheusers.Userscanobtainthetokenbycallingthe/tokensendpointofAuthservicedirectlyandusethesametokentoauthenticateandauthorizethemselveswithotherservices.

Buddy(https://github.com/funcool/buddy)isanotherClojurelibrarythathasaBuddySign(https://github.com/funcool/buddy-sign)librarythatcanalsobeusedtogenerateJSONWebTokens.

MonitoringmicroservicesAmicroservices-basedapplicationishighlyflexibleintermsofdeploymentandscaling.Itconsistsofmultipleservicesthatmayhaveoneormoreinstancesrunningonaclusterofmachinesacrossthenetwork.Insuchahighlydistributedandflexibleenvironment,itisofutmostimportancethateachinstanceofamicroserviceismonitoredinrealtimetogetaclearviewofthedeployedservices,theirperformance,andtocaptureissuesofinterestthatmustbereportedassoonastheyoccur.Sinceeachrequesttoamicroservice-basedapplicationmayspanouttooneormorerequestsamongmicroservices,thereshouldbeamechanismtotracktheflowofrequestsandalsolocatetheareasofbottleneckthatcanbeaddressedbyperformingarootcauseanalysisandoftenscalingtheservicesfurthertomeetthedemand.

Oneofthewaystosetupaneffectivemonitoringsystemistocollectallthemetricsacrosstheservicesandmachinesandstoretheminacentralizedrepository,asshownintheprecedingdiagram.Thiscentralizedrepositorycanthensupporttheanalysisofthecapturedmetricsandhelptogeneratealertsfortheeventsofinterestinrealtime.Acentralizedrepositoryalsohelpsinsettingupthereal-timeviewofthesystemtounderstandthebehaviorofeachserviceanddecidewhethertoscaleituporscaleitdown.Tosetupacentralizedrepositoryfortheapplication,themetricsneedtobeeitherpulledfromalltheservicesandphysicalmachinesorpushedtothecentralizedrepositorybythe

servicesrunningonthephysicalmachines.Bothpushandpullmodelsareusefultosetupaneffectivemonitoringsystemthatservesthesourceoftruthforthestateofthesystemaswellastheperformanceoftheenvironmentthatiscrucialforeffectiveutilizationoftheinfrastructureusedbythemicroservices-basedapplication.

Alltheservicesmustberesponsibletopushthemetricsrelatedtothestateoftheapplicationtoacommonchannel,suchasApacheKafka(https://kafka.apache.org/),onacommontopicthatcanthenbeusedtoaggregatealltheapplication-levelmetricsacrosstheservicesandstoretheminacentralizedrepository.Application-levellogsthatarewrittentothefileonthephysicalserversandtheapplication-levelmetricsthatarepublishedviamediumssuchasJMX(http://www.oracle.com/technetwork/articles/java/javamanagement-140525.html)canbepulledbyanexternalcollectorandlaterpushedtothecentralizedstorage.Tomonitortheperformanceoftheinfrastructure,externalcollectorsmustalsocapturethestatsofthephysicalmachine,includingCPUutilization,networkthroughput,diskI/O,andmore,whichcanalsobepushedtothecentralrepositorytogetaholisticviewoftheresourceutilizationacrosstheservicesoftheapplication.

UsingELKStackformonitoringElasticsearch(https://www.elastic.co/products/elasticsearch),Logstash(https://www.elastic.co/products/logstash),andKibana(https://www.elastic.co/products/kibana),oftenreferredtoasELKStackorElasticStack(https://www.elastic.co/elk-stack),providealltherequiredcomponentstosetupareal-timemonitoringinfrastructuretocapture,pull,andpushtheapplicationandmachine-levelmetricsintoacentralizedrepositoryandbuildamonitoringdashboardforreportingandalerts.ThefollowingmonitoringinfrastructurediagramexhibitswhereeachofthecomponentsoftheELKStackfitin.Collectd(https://collectd.org/)andApacheKafka(https://kafka.apache.org/)arenotapartofELKStack,butELKStackprovidesseamlessintegrationwiththeseoutofthebox:

Collectdhelpsincapturingallthemachine-levelstats,includingCPU,memory,disk,andnetwork.ThecaptureddatacanthenbepulledthroughLogstashandpushedintoElasticsearchtoanalyzetheoverallperformanceandutilizationoftheinfrastructureusedbytheservicesoftheapplication.LogstashcanalsounderstandthestandardsetofapplicationlogsandpulltheloggedeventsfromthelogfilesgeneratedonthemachineandpushittotheElasticsearchcluster.LogstashalsointegrateswellwithApacheKafkaandcanbeusedtocapturethe

applicationstateeventspublishedbytheservicesandpushthemdirectlytoElasticsearch.SinceElasticsearchactsasacentralrepositoryforallthelogs,events,andmachinestats,KibanacanbeuseddirectlyontopofElasticsearchtoanalyzethestoredmetricsandbuilddashboardsthatareupdatedinrealtimeasandwheneventsarriveinElasticsearch.Kibanacanalsobeusedtoperformrootcauseanalysisandgeneratealertsfortheintendedrecipients.

ELKStackisusefulformonitoring,butitisnottheonlyoption.ToolssuchasPrometheus(https://prometheus.io/)canalsobeusedformonitoring.Prometheussupportsadimensionaldatamodel,flexiblequerylanguageandefficienttimeseriesdatabasewithin-builtalerting.

SettingupElasticsearch

TosetupElasticsearch,downloadthelatestversionfromtheElasticsearchdownloadpage(https://www.elastic.co/downloads/elasticsearch)andextractit,asshowninthefollowingexample;thisbookusesElasticsearch6.1.1,thatcanbedownloadedfromthereleasepageof6.1.1(https://www.elastic.co/downloads/past-releases/elasticsearch-6-1-1):#downloadelasticsearch6.1.1tar%wgethttps://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.1.1.tar.gz--https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.1.1.tar.gzResolvingartifacts.elastic.co(artifacts.elastic.co)...184.73.156.41,184.72.218.26,54.235.82.130,...Connectingtoartifacts.elastic.co(artifacts.elastic.co)|184.73.156.41|:443...connected.HTTPrequestsent,awaitingresponse...200OKLength:28462503(27M)[application/x-gzip]Savingto:‘elasticsearch-6.1.1.tar.gz’

elasticsearch-6.1.1.tar.gz100%[==============================>]27.14M2.38MB/sin21s

...(1.30MB/s)-‘elasticsearch-6.1.1.tar.gz’saved[28462503/28462503]

#extractthedownloadedtarball%tar-xvfelasticsearch-6.1.1.tar.gz...

#makesurethatthesedirectoriesarepresent%tree-L1elasticsearch-6.1.1elasticsearch-6.1.1├──bin├──config├──lib

├──LICENSE.txt├──modules├──NOTICE.txt├──plugins└──README.textile

5directories,3files

AlthoughElasticsearchwillrunstraightoutoftheboxwiththebin/elasticsearchcommand,itisrecommendedtoreviewthefollowingimportantconfigurationsandsystemsettingsforaneffectiveElasticsearchcluster.ThesettingsmarkedasESConfigaregenericforallElasticsearchdeployments,whereastheonesmarkedasSystemSettingarefortheLinuxoperatingsystem.Theenvironmentvariable—$ES_HOME—referstotheextractedElasticsearchinstallationfolder,thatis,elasticsearch-6.1.1forthecommandshownintheprecedingcodesnippet.

Type ConfigLocation ConfigParameter Value

SystemSetting

/etc/security/limits.conf memlock unlimited

SystemSetting

/etc/security/limits.conf nofile 65536

SystemSetting

/etc/sysctl.conf vm.overcommit_memory 1

SystemSetting

/etc/sysctl.conf vm.max_map_count 262144

SystemSetting

/etc/fstab Commentswapconfig -

ESJVM $ES_HOME/config/jvm.options -Xmsand-Xmx 8g,16g,

andsoon

Options

ESConfig

$ES_HOME/config/elasticsearch.yml cluster.name <name>

ESConfig

$ES_HOME/config/elasticsearch.yml node.name <name>

ESConfig

$ES_HOME/config/elasticsearch.yml path.data

Oneormore<pathtokeep

indexes>

ESConfig

$ES_HOME/config/elasticsearch.yml path.logs<pathtolog

directory>

ESConfig

$ES_HOME/config/elasticsearch.yml bootstrap.memory_lock true

ESConfig

$ES_HOME/config/elasticsearch.yml network.host <ip_address>

ESConfig

$ES_HOME/config/elasticsearch.yml discovery.zen.ping.unicast.hosts

Oneormore<ip>:<port>

ESConfig

$ES_HOME/config/elasticsearch.yml discovery.zen.minimum_master_nodes<numberof

nodes>

Notethatsomeofthesystemsettingsshownintheprecedingtablemayrequireasystemrestartforthemtotakeintoeffect.Also,settingslikethatofswapspacemustbedoneonlyifElasticsearchistheonlycomponentrunningonthehost

operatingsystem.Onceallthesettingsareinplace,eachElasticsearchnodecanbestartedusingthefollowingcommand;eachnodewilljointheclusteriftheyhavethesameclusternameandareapartofunicasthostslistthatanodeisallowedtojoin:#changetotheextractedelasticsearchdirectory%cdelasticsearch-6.1.1

#startelasticsearch%bin/elasticsearch[2018-01-15T20:46:35,408][INFO][o.e.n.Node][]initializing......[2018-01-15T20:46:36,328][INFO][o.e.p.PluginsService][W6r6s1z]loadedmodule[aggs-matrix-stats][2018-01-15T20:46:36,329][INFO][o.e.p.PluginsService][W6r6s1z]loadedmodule[analysis-common][2018-01-15T20:46:36,329][INFO][o.e.p.PluginsService][W6r6s1z]loadedmodule[ingest-common]...[2018-01-15T20:46:36,330][INFO][o.e.p.PluginsService][W6r6s1z]loadedmodule[tribe][2018-01-15T20:46:37,491][INFO][o.e.d.DiscoveryModule][W6r6s1z]usingdiscoverytype[zen][2018-01-15T20:46:37,929][INFO][o.e.n.Node]initialized[2018-01-15T20:46:37,930][INFO][o.e.n.Node][W6r6s1z]starting...[2018-01-15T20:46:38,100][INFO][o.e.t.TransportService][W6r6s1z]publish_address{127.0.0.1:9300},bound_addresses{[::1]:9300},{127.0.0.1:9300}[2018-01-15T20:46:41,164][INFO][o.e.c.s.MasterService][W6r6s1z]zen-disco-elected-as-master([0]nodesjoined),reason:new_master{W6r6s1z}{W6r6s1zTQ96ULo2wq9Tm3w}{ykEtBVl9Sy62mkXOFo892g}{127.0.0.1}{127.0.0.1:9300}...[2018-01-15T20:46:41,196][INFO][o.e.n.Node][W6r6s1z]started[2018-01-15T20:46:41,239][INFO][o.e.g.GatewayService][W6r6s1z]recovered[0]indicesintocluster_state

Notethatthefirstnodethatisstartedisautomaticallyelectedasamasteroftheclustertowhichothernodescanjoin:%curlhttp://localhost:9200

{"name":"W6r6s1z","cluster_name":"elasticsearch","cluster_uuid":"g33pKv6XRTaj_yMJLliL0Q","version":{"number":"6.1.1","build_hash":"bd92e7f","build_date":"2017-12-17T20:23:25.338Z","build_snapshot":false,"lucene_version":"7.1.0","minimum_wire_compatibility_version":"5.6.0","minimum_index_compatibility_version":"5.0.0"},"tagline":"YouKnow,forSearch"}

OncetheElasticsearchnodeisupandrunning,totesttheinstancesendaGETrequesttothedefault9200portonthemachinewhereElasticsearchisrunningusingcURL,asshownintheprecedingexample.Itshouldreturnaresponsestatingtheversionofthenode.Verifythatitistherightversion,thatis,6.1.1,forthisexample.

FormoredetailsontheimportantsettingsofElasticsearchandSystem,takealookattheElasticsearchdocsforImportantSettings(https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html)andSystemSettings(https://www.elastic.co/guide/en/elasticsearch/reference/current/setting-system-settings.html).

TheprecedingconfigurationdiscussesatypicalclusterdeploymentforElasticsearch.ElasticsearchalsoprovidesaconceptofCrossClusterSearch(https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cross-cluster-search.html)thatallowsanynodetoactasafederatedclientacrossmultipleclustersofElasticsearch.

ThestepstakeninthissectionuseElasticsearchtarballtosetupElasticsearchcluster,butElasticsearchprovidesanumberofoptionstosetitupusingbinaries,includingRPMpackage,Debianpackage,MSIpackageforWindows,andaDockerimage.For

moredetails,refertotheinstallationinstructionsathttps://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#install-elasticsearch.

SettingupKibanaKibanaisthevisualizationanddashboardinterfaceforElasticsearch.ItallowsexploringdatausingitsDiscovermodule(https://www.elastic.co/guide/en/kibana/6.1/discover.html)andbuildingreal-timedashboards.Itincludesagoodnumberofvisualizationoptions(https://www.elastic.co/guide/en/kibana/current/visualize.html)thatallowuserstoaggregatedatastoredwithinElasticsearchandvisualizethemusingvariouscharts,suchasline,bar,area,maps,andtagclouds.KibanacanbeusedtobuildthemonitoringdashboardusingthevariousmetricsthatarecapturedwithinElasticsearch.

TosetupKibana,downloadthelatestversionfromtheKibanadownloadspage(https://www.elastic.co/downloads/kibana)andextractitasshowninthefollowingexample;thisbookusesKibana6.1.1,whichcanbedownloadedfromthereleasepageof6.1.1(https://www.elastic.co/downloads/past-releases/kibana-6-1-1):

#downloadKibana6.1.1tar

%wgethttps://artifacts.elastic.co/downloads/kibana/kibana-6.1.1-linux-x86_64.tar.gz

--https://artifacts.elastic.co/downloads/kibana/kibana-6.1.1-linux-x86_64.tar.gz

Resolvingartifacts.elastic.co(artifacts.elastic.co)...54.225.188.6,23.21.118.61,

54.235.82.130,...

Connectingtoartifacts.elastic.co(artifacts.elastic.co)|54.225.188.6|:443...

connected.

HTTPrequestsent,awaitingresponse...200OK

Length:64664051(62M)[application/x-gzip]

Savingto:‘kibana-6.1.1-linux-x86_64.tar.gz’

kibana-6.1.1-linux-x86_64.tar.gz100%[==============================>]61.67M2.06MB/s

...(1.10MB/s)-‘kibana-6.1.1-linux-x86_64.tar.gz’saved[64664051/64664051]

#extractthedownloadedtarball

%tar-xvfkibana-6.1.1-linux-x86_64.tar.gz

#makesurethatthesedirectoriesarepresent

%tree-L1kibana-6.1.1-linux-x86_64

kibana-6.1.1-linux-x86_64

├──bin

├──config

├──data

├──LICENSE.txt

├──node

├──node_modules

├──NOTICE.txt

├──optimize

├──package.json

├──plugins

├──README.txt

├──src

├──ui_framework

└──webpackShims

Next,configureKibanainstancebysettingthefollowingconfigurationparametersinthe$KIBANA_HOME/config/kibana.ymlfile.Theenvironmentvariable—$KIBANA_HOME—referstotheextractedKibanainstallationfolder,thatis,kibana-6.1.1-linux-x86_64forthecommandshownintheprecedingcodesnippet.

ConfigParameter Value Description

server.port 5601 Defaultserver.host <host_ip> KibanabindstothisIPaddress

server.basePath <base_prefix_URL>Shouldnotendwith`/`.UsedtomapproxyURLprefix,ifany

server.name <name> Displayname

elasticsearch.urlhttp://<es_host>:

<es_port>

ElasticsearchURLtoconnectto;defaultportis9200

kibana.index <name>IndexnameascreatedbyKibanainElasticsearch

pid.file <path_to_pid_file> PIDfilelocationlogging.dest <path_to_log_file> FiletowriteKibanalogs

Onceallthesettingsareinplace,startKibanausingthecommandshowninthefollowingexample;ensurethatElasticsearchisalreadyrunningandaccessibletotheKibananodeontheconfiguredelasticsearch.urlsetting,asdescribedintheprecedingtable.

#changetoextractedkibanadirectory

%cdkibana-6.1.1-linux-x86_64

#startkibana

%bin/kibana

log[16:41:31.409][info][status][plugin:kibana@6.1.1]Statuschangedfrom

uninitializedtogreen-Ready

log[16:41:31.443][info][status][plugin:elasticsearch@6.1.1]Statuschangedfrom

uninitializedtoyellow-WaitingforElasticsearch

log[16:41:31.461][info][status][plugin:console@6.1.1]Statuschangedfrom

log[16:41:31.484][info][status][plugin:metrics@6.1.1]Statuschangedfrom

log[16:41:31.646][info][status][plugin:timelion@6.1.1]Statuschangedfrom

log[16:41:31.650][info][listening]Serverrunningathttp://localhost:5601

log[16:41:31.668][info][status][plugin:elasticsearch@6.1.1]Statuschangedfrom

yellowtogreen-Ready

OnceKibanaisupandrunning,opentheURLhttp://localhost:5601,asloggedintheprecedingmessagestoopenKibanainterface,asshowninthefollowingscreenshot;itshouldshowtheKibanahomepagewithoptionstovisualizeandexploredata:

ThecurrentconfigurationofKibanaallowsuserstoexploreElasticsearchinaclosednetwork.SinceKibanaprovidesfullcontrolovertheunderlyingElasticsearchclusteranddatastoredwithinit,itisrecommendedthatyouenableSSLandalsotheload-balancingoptionasdefinedintheproductionconfiguration(https://www.elastic.co/guide/en/kibana/current/production.html)foruserstoconnectandaccessdashboards.

KibananotonlyallowsuserstoexploredatasetsalreadystoredwithinElasticsearchbutalsosupportsloadingdatasetsdirectlyintoElasticsearchviaitsuserinterface.Tolearnmoreaboutthis,

followtheLoadingSampleDatatutorialofKibana(https://www.elastic.co/guide/en/kibana/current/tutorial-load-dataset.html).

SettingupLogstash

Logstashallowsuserstocollect,parse,andtransformlogmessages.Itsupportsanumberofinput(https://www.elastic.co/guide/en/logstash/current/input-plugins.html)andoutput(https://www.elastic.co/guide/en/logstash/current/output-plugins.html)pluginsthatallowLogstashtocollectlogsfromavarietyofsources,parseandtransformthem,andthenwritetheresultstooneofthesupportedplugins.TosetupLogstash,downloadthelatestversionfromtheLogstashdownloadspage(https://www.elastic.co/downloads/logstash)andextractitasshowninthefollowingcodesnippet—thisbookusesLogstash6.1.1,whichcanbedownloadedfromthereleasepageof6.1.1(https://www.elastic.co/downloads/past-releases/logstash-6-1-1):

#downloadLogstash6.1.1tar

%wgethttps://artifacts.elastic.co/downloads/logstash/logstash-6.1.1.tar.gz

--https://artifacts.elastic.co/downloads/logstash/logstash-6.1.1.tar.gz

Resolvingartifacts.elastic.co(artifacts.elastic.co)...23.21.118.61,54.243.108.41,

184.72.218.26,...

Connectingtoartifacts.elastic.co(artifacts.elastic.co)|23.21.118.61|:443...

connected.

Length:109795895(105M)[application/x-gzip]

Savingto:‘logstash-6.1.1.tar.gz’

logstash-6.1.1.tar.gz100%[=================================>]104.71M1.09MB/sin81s

...(1.30MB/s)-‘logstash-6.1.1.tar.gz’saved[109795895/109795895]

%tar-xvflogstash-6.1.1.tar.gz

%tree-L1logstash-6.1.1

logstash-6.1.1

├──bin

├──config

├──CONTRIBUTORS

├──data

├──Gemfile

├──Gemfile.lock

├──lib

├──LICENSE

├──logstash-core

├──logstash-core-plugin-api

├──modules

├──NOTICE.TXT

├──tools

└──vendor

9directories,5files

ThefollowingtableliststheprimaryconfigurationsettingsthatarerequiredforLogstashandmustbeaddedtothe$LOGSTASH_HOME/config/logstash.ymlfile;theenvironmentvariable—$LOGSTASH_HOME—referstotheextractedLogstashinstallationfolder,thatis,logstash-6.1.1forthecommandshownintheprecedingcodesnippet.

ConfigParameter Value Description

node.name <name>

Nodenametoidentifythenodefromoutputinterface.Goodtohaveasahostname.

path.data <path_to_data>

OneormorepathswhereLogstashanditspluginkeepsthedataforanypersistenceneeds.

pipeline.workers1,2,3,4,andsoon

Workerstoexecutefilterandoutputstages.Ifdeployedonaseparatemachine,setthistonumberofCPUcores.

pipeline.output.workers1,2,andsoon

Numberofworkerstouseperoutputplugininstance.Defaultsto1.

path.config <path_to_config>Locationtofetchpipelineconfigurationformainpipeline.

http.host <host_ip>BindaddressformetricsRESTendpoint.

http.port <host_port>

BindportforthemetricsRESTendpoint.Alsoacceptsranges,suchas(9600-9700),topickthefirstavailableport.

path.logs <path_to_logs> PathwhereLogstashwillkeepthelogs.

Theprecedingtablelistsonlytheprimaryconfigurationparameters;formoredetailsandallthesupportedconfigurationparameters,refertoLogstashSettingsFileguide(https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html).Onceallthesettingsareinplace,testasampleLogstashpipelinethatusesstdin(https://www.elastic.co/guide/en/logstash/current/plugins-inputs-stdin.html)asitsinputplugintoreceivemessagesandstdout(https://www.elastic.co/guide/en/logstash/current/plugins-outputs-stdout.html)asitsoutputplugintoemitthereceivedmessages,asfollows:

#changetoextractedlogstashdirectory

%cdlogstash-6.1.1

#startlogstashpipelinebyspecifyingtheconfiguration

#atcommandlineusingthe-eflag

%bin/logstash-e'input{stdin{}}output{stdout{}}'

SendingLogstash'slogstologstash-6.1.1/logswhichisnowconfiguredvia

log4j2.properties

[2018-01-15T23:11:02,245][INFO][logstash.modules.scaffold]Initializingmodule

{:module_name=>"netflow",:directory=>"logstash-6.1.1/modules/netflow/configuration"}

[2018-01-15T23:11:03,872][INFO][logstash.runner]StartingLogstash

{"logstash.version"=>"6.1.1"}

[2018-01-15T23:11:04,415][INFO][logstash.agent]SuccessfullystartedLogstashAPI

endpoint{:port=>9600}

[2018-01-15T23:11:06,212][INFO][logstash.pipeline]Startingpipeline

{:pipeline_id=>"main","pipeline.workers"=>4,"pipeline.batch.size"=>125,

"pipeline.batch.delay"=>5,"pipeline.max_inflight"=>500,:thread=>"#<Thread:0x77cbc3e6

run>"}

[2018-01-15T23:11:06,305][INFO][logstash.pipeline]Pipelinestarted

{"pipeline.id"=>"main"}

Thestdinpluginisnowwaitingforinput:

[2018-01-15T23:11:06,413][INFO][logstash.agent]Pipelinesrunning{:count=>1,

:pipelines=>["main"]}

helloworld

2018-01-15T17:41:19.900Zfc-machinehelloworld

2018-01-15T17:41:21.707Zfc-machine

HelloLogstash!

2018-01-15T17:41:28.566Zfc-machineHelloLogstash!

HelloELK!

2018-01-15T17:41:32.255Zfc-machineHelloELK!

HelloHelpingHandsEvents!

2018-01-15T17:41:38.685Zfc-machineHelloHelpingHandsEvents!

Logstashmaytakeafewsecondstostartthepipeline,sowaitforthePipelinerunningmessagetobelogged.Oncethepipelineisrunning,typeamessageintheconsole,andLogstashwillechothesameontheconsoleappendedwiththecurrenttimestampandhostname.Thisisaverysimplepipelinethatdoesnotdoanytransformation,butLogstashallowstransformationstobeappliedonthereceivedmessagesbeforetheyareemittedtothesink.Similartothebasicpipelineshownintheprecedingtest,theLogstashpipelineconfigurationiscreatedforeachpipelinethatisrequiredtobeexecutedbyLogstashtocapture

logs,events,anddata,andstoretheminthetargetsinks.

LogstashpluginsareimplementedprimarilyinRuby(https://www.ruby-lang.org/en/).ThatiswhyallthejobconfigurationfilesforLogstashandtransformationconstructsusesyntaxofRubylanguage.

UsingELKStackwithCollectdCollectdisadaemonthatcanbeconfiguredtocollectmetricsfromvarioussourceplugins,suchasLogstash.AscomparedtoLogstash,Collectdisverylightweightandportable,butitdoesnotgenerategraphs.ItcanwritetoRRDfiles,though,thatneedaRRDTool(https://en.wikipedia.org/wiki/RRDtool)toreadthemandgenerategraphstovisualizetheloggeddata.Ontheotherhand,sinceCollectdiswritteninCprogramminglanguage(https://en.wikipedia.org/wiki/C_(programming_language)),itisalsopossibletouseittocollectmetricsfromembeddedsystemsaswell.

Collectdneedstobebuiltfromsource.First,downloadtheCollectd5.8.0versionandextractthesame:

#downloadCollectd5.8.0tar

%wgethttps://storage.googleapis.com/collectd-tarballs/collectd-5.8.0.tar.bz2

--https://storage.googleapis.com/collectd-tarballs/collectd-5.8.0.tar.bz2

Resolvingstorage.googleapis.com(storage.googleapis.com)...172.217.26.208,

2404:6800:4007:802::2010

Connectingtostorage.googleapis.com(storage.googleapis.com)|172.217.26.208|:443...

connected.

Length:1686017(1.6M)[application/x-bzip]

Savingto:‘collectd-5.8.0.tar.bz2’

collectd-5.8.0.tar.bz2100%[===============================>]1.61M2.67MB/sin0.6s

...(2.67MB/s)-‘collectd-5.8.0.tar.bz2’saved[1686017/1686017]

%tar-xvfcollectd-5.8.0.tar.bz2

%tree-L1collectd-5.8.0

collectd-5.8.0

├──aclocal.m4

├──AUTHORS

├──bindings

├──build-aux

├──ChangeLog

├──configure

├──configure.ac

├──contrib

├──COPYING

├──m4

├──Makefile.am

├──Makefile.in

├──proto

├──README

├──src

├──testwrapper.sh

└──version-gen.sh

Next,installCollectdtoabuilddirectory,asshowninthefollowingexample.Incasetheconfigurescriptrequestsformissingdependencies,installthembeforecontinuingthesetupaspertheFirststepswiki(https://collectd.org/wiki/index.php/First_steps)ofCollectd:

#changetoextractedcollectddirectory

%cdcollectd-5.8.0

#configurethetargetbuilddirectory

#givethefullyqualifiedpathasprefix

#$COLLECTD_HOMEpointstocollectd-5.8.0directory

%./configure--prefix=$COLLECTD_HOME/build

checkingbuildsystemtype...x86_64-unknown-linux-gnu

checkinghostsystemtype...x86_64-unknown-linux-gnu

checkinghowtoprintstrings...printf

checkingforgcc...gcc

checkingwhethertheCcompilerworks...yes

checkingforCcompilerdefaultoutputfilename...a.out

checkingforsuffixofexecutables...

#installcollectd

%sudomakeallinstall

#verifythebuilddirectories

%tree-L1build

├──bin

├──etc

├──include

├──lib

├──man

├──sbin

├──share

└──var

8directories,0files

#owntheentirecollectddirectory

#replace<user>withyourusername

%sudochown-R<user>:<user>.

OnceCollectdisinstalled,thenextstepistoupdatethebuild/etc/collectd.conffilewiththedesiredconfigurationsandplugins.Thefollowingisasamplecollectd.conffiletoenablecpu,df,interface,network,memory,syslog,load,andswapplugins;formoredetailsontheavailablepluginsandtheirconfiguration,refertoCollectdTableofPlugins(https://collectd.org/wiki/index.php/Table_of_Plugins).

#BaseConfiguration

#replaceallpathsbelowwithfullyqualified

#pathtotheextractedcollectd-5.8.0directory

Hostname"helpinghands.com"

BaseDir"/collectd-5.8.0/build/var/lib/collectd"

PIDFile"/collectd-5.8.0/build/var/run/collectd.pid"

PluginDir"/collectd-5.8.0/build/lib/collectd"

TypesDB"/collectd-5.8.0/build/share/collectd/types.db"

CollectInternalStatstrue

#Syslog

LoadPluginsyslog

LogLevelinfo

</Plugin>

#Otherplug-ins

LoadPlugincpu

LoadPlugindf

LoadPlugindisk

LoadPlugininterface

LoadPluginload

LoadPluginmemory

LoadPluginnetwork

LoadPluginswap

#Plug-inConfig

ReportByCputrue

ReportByStatetrue

ValuesPercentagefalse

</Plugin>

#replacedeviceandmountpoint

#withthedevicetobemonitored

#asshownbydfcommand

Device"/dev/sda9"

MountPoint"/home"

FSType"ext4"

IgnoreSelectedfalse

ReportByDevicefalse

ReportInodesfalse

ValuesAbsolutetrue

</Plugin>

Disk"/^[hs]d[a-f][0-9]?$/"

IgnoreSelectedfalse

UseBSDNamefalse

UdevNameAttr"DEVNAME"

</Plugin>

#reportallinterfaceexceptloandsit0

Interface"lo"

Interface"sit0"

IgnoreSelectedtrue

ReportInactivetrue

UniqueNamefalse

</Plugin>

ReportRelativetrue

</Plugin>

ValuesAbsolutetrue

</Plugin>

#sendsmetricstothisporti.e.

#configuredinlogstashtoreceive

#thelogeventstobepublished

Server"127.0.0.1""25826"

<Server"127.0.0.1""25826">

</Server>

</Plugin>

ReportByDevicefalse

ReportBytestrue

ValuesAbsolutetrue

</Plugin>

Oncetheconfigurationfileisinplace,startCollectddaemon,asshownhere:

#startcollectddaemonwithsudo

#someplug-insrequiresudoaccess

%sudobuild/sbin/collectd

#makesureitisrunning

%ps-ef|grepcollectd

anuj272081768001:21?00:00:00build/sbin/collectd

#verifysyslogtomakesurethatcollectdisup

%tail-f/var/log/syslog

Jan1601:40:01localhostcollectd[28725]:plugin_load:plugin"syslog"successfully

loaded.

Jan1601:40:01localhostcollectd[28725]:plugin_load:plugin"cpu"successfully

loaded.

Jan1601:40:01localhostcollectd[28725]:plugin_load:plugin"df"successfully

loaded.

Jan1601:40:01localhostcollectd[28725]:plugin_load:plugin"disk"successfully

loaded.

Jan1601:40:01localhostcollectd[28725]:plugin_load:plugin"interface"

successfullyloaded.

Jan1601:40:01localhostcollectd[28725]:plugin_load:plugin"load"successfully

loaded.

Jan1601:40:01localhostcollectd[28725]:plugin_load:plugin"memory"successfully

loaded.

Jan1601:40:01localhostcollectd[28725]:plugin_load:plugin"network"successfully

loaded.

Jan1601:40:01localhostcollectd[28725]:plugin_load:plugin"swap"successfully

loaded.

Jan1601:40:01localhostcollectd[28726]:Initializationcomplete,enteringread-

Next,createaLogstashpipelineconfigurationfile,$LOGSTASH_HOME/config/helpinghands.conf,toreceivethedatafromCollectdusingtheCollectdCodec(https://www.elastic.co/guide/en/logstash/current/plugins-codecs-collectd.html)pluginandsendittoElasticsearchusingitsoutputplugin(https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html):

input{

port=>25826

buffer_size=>1452

codec=>collectd{

id=>"helpinghands.com-collectd"

typesdb=>["/collectd-5.8.0/build/share/collectd/types.db"]

output{

elasticsearch{

id=>"helpinghands.com-collectd-es"

hosts=>["127.0.0.1:9200"]

index=>"helpinghands.collectd.instance-%{+YYYY.MM}"

Next,runtheLogstashpipelinetoreceivedatafromCollectdprocessoverUDP(https://en.wikipedia.org/wiki/User_Datagram_Protocol)andsendittoElasticsearch.Ensurethatthe25826portspecifiedintheUDPconfigurationabovematchestheportofthenetworkpluginofCollectdconfiguration.BeforerunningLogstash,verifythatElasticsearchandCollectdbotharerunning:

#changeto$LOGSTASH_HOMEdirectoryandrunlogstash

%bin/logstash-fconfig/helpinghands.conf

SendingLogstash'slogsto/logstash-6.1.1/logswhichisnowconfiguredvia

log4j2.properties

{:module_name=>"netflow",:directory=>"/logstash-6.1.1/modules/netflow/configuration"}

{:module_name=>"fb_apache",:directory=>"/logstash-

6.1.1/modules/fb_apache/configuration"}

[2018-01-16T02:03:58,410][INFO][logstash.runner]StartingLogstash

{"logstash.version"=>"6.1.1"}

[2018-01-16T02:03:58,935][INFO][logstash.agent]SuccessfullystartedLogstashAPI

endpoint{:port=>9600}

[2018-01-16T02:04:02,412][INFO][logstash.outputs.elasticsearch]Elasticsearchpool

URLsupdated{:changes=>{:removed=>[],:added=>[http://127.0.0.1:9200/]}}

[2018-01-16T02:04:03,777][INFO][logstash.outputs.elasticsearch]NewElasticsearch

output{:class=>"LogStash::Outputs::ElasticSearch",:hosts=>["//127.0.0.1:9200"]}

[2018-01-16T02:04:03,883][INFO][logstash.pipeline]Pipelinestarted

{"pipeline.id"=>"main"}

[2018-01-16T02:04:03,960][INFO][logstash.inputs.udp]StartingUDPlistener

{:address=>"0.0.0.0:25826"}

[2018-01-16T02:04:03,997][INFO][logstash.agent]Pipelinesrunning{:count=>1,

:pipelines=>["main"]}

[2018-01-16T02:04:04,030][INFO][logstash.inputs.udp]UDPlistenerstarted

{:address=>"0.0.0.0:25826",:receive_buffer_bytes=>"106496",:queue_size=>"2000"}

OnceLogstashstarts,observeElasticsearchlogsthatshowsthatLogstashhascreatedanewindexbasedonthehelpinghands.collectd.instance-%{+YYYY.MM}patternasconfiguredinLogstash'sElasticsearchoutputplugin.Notethattheindex

namewilldifferbasedonthecurrentmonthandyear.Maintainingatime-basedindexpatternisrecommendedforindexesthatstoretimeseriesdatasets.Itnotonlyhelpsinqueryperformancebutalsohelpsinbackupandcleanupbasedonthedataretentionpoliciesofanorganization.ThefollowingarethelogmessagesthatcanbeobservedinElasticsearchlogfilesforsuccessfulcreationoftherequiredindexforthedatacapturedbyLogstashfromCollectd:

[2018-01-16T02:04:12,054][INFO][o.e.c.m.MetaDataCreateIndexService][W6r6s1z]

[helpinghands.collectd.instance-2018.01]creatingindex,cause[auto(bulkapi)],

templates[],shards[5]/[1],mappings[]

[2018-01-16T02:04:15,259][INFO][o.e.c.m.MetaDataMappingService][W6r6s1z]

[helpinghands.collectd.instance-2018.01/9x0mla-mS0akJLuuUJELZw]create_mapping[doc]

[helpinghands.collectd.instance-2018.01/9x0mla-mS0akJLuuUJELZw]update_mapping[doc]

LetthepipelinerunandstorethemachinemetricscapturedviaCollectd–Logstash–Elasticsearchpipeline.Now,opentheKibanainterfaceinthebrowserusingtheURLhttp://localhost:5601andclickontheSetupindexpatternsbuttononthetop-rightcorner.Itwillautomaticallylistthenewlycreatedindexhelpinghands.collectd.instance-2018.01,asshowninthefollowingscreenshot:

Addtheindexpatternhelpinghands.collectd.instance-*,asshownintheprecedingscreenshot,toincludealltheindexescreatedforthemetricscapturedbyCollectd.ClickontheNextstepbuttonontheright-handsideandselectTimefilterfieldnameas@timestamp,asshowninthefollowingscreenshot:

Next,clickontheCreateindexpatternbutton,asshownintheprecedingscreenshot.ItwillshowthelistoffieldsthatKibanawasabletoretrievefromtheElasticsearchindexmapping(https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html).Now,clickonDiscoverintheleft-handsidemenu,anditwillshowthereal-timedashboardofallthemessagesbeingcaptured,asshowninthefollowingscreenshot:

Theleft-handsidepaneloftheDiscoverscreenlistsallthefieldsbeingcaptured.Forexample,clickonhostandpluginfieldstoseethehostsbeingmonitoredandallthepluginsforwhichthedatahasbeencapturedbyCollectdandsenttoElasticsearchviatheLogstashpipeline,asshowninthefollowingscreenshot:

Kibanadashboardallowsbuildingdashboardwithvariousvisualizationoptions.Forexample,totakealookattheCPUutilizationsinceCollectdstartedmonitoringit,performthefollowingsteps:

1. ClickonVisualizeintheleftpaneloftheKibanaapplication2. ClickontheCreateavisualizationbutton3. ClickonLinetochoosethelinechart4. Choosetheindexhelpinghands.collectd.instance-*patternfromthesection

ontheleft5. ClickontheAddafilter+optionbelowthesearchbaratthetop6. Selectfilterasplugin.keywordiscpuandsave7. ClickontheY-axisandchangeAggregationtoAverage8. SelectthefieldasValue9. Next,clickontheX-AxisunderBuckets10. ChoosetheAggregationasDateHistogram11. Itshouldbydefaultselectthe@timestampfield12. KeeptheintervalasAuto13. ClickontheapplychangesplayiconatthetopoftheMetricspanel

14. Itwillshowthechartontheright-handside,asshowninthenextscreenshot

15. Clickonthearrowiconatthebottomofthechartareatobringupthetable

Oncethesestepshavebeenperformed,youshouldbeabletoseetheCPUutilizationovertimeforthelast15minutes(default),asshowninthefollowingscreenshot.Thecreatedvisualization(https://www.elastic.co/guide/en/kibana/current/visualize.html)canbesavedandlateraddedtoadashboard(https://www.elastic.co/guide/en/kibana/current/dashboard.html)tobuildafull-fledgeddashboardtomonitorallthecapturedmetricsinrealtime:

LoggingandmonitoringguidelinesMicroservicesoftheHelpingHandsapplicationmustgeneratebothapplicationandauditlogsthatcanbecapturedbytheELKStack.Applicationlogscanbegeneratedbytheservicesusingthetools.logginglibraryofClojure(https://github.com/clojure/tools.logging)thatlogsamessageinastandardlog4j(https://logging.apache.org/log4j/2.x/)stylesyntax.Logstashworkswellwithmostofthecommonloggingformats(https://www.elastic.co/guide/en/logstash/6.1/plugins-inputs-log4j.html),butitisrecommendedtousestructuredlogginginstead.

StructuredlogsareeasiertoparseandloadwithinacentralizedrepositoryusingtoolssuchasLogstash.Librariessuchastimbre(https://github.com/ptaoussanis/timbre)supportstructuredloggingandalsoallowpublishingthelogsdirectlytoaremoteserviceinsteadofloggingtoafile.Inadditiontostructuredlogging,considerincludingpredefinedstandardtagswithinthelogmessages,asshowninthefollowingtable:

Name Event<service> Thenameoftheservice,suchashelping-hands.alert

<service>-

Loggedfromthemainfunctiononcetheapplicationisupandrunning

<service>-

Oncetheserviceisupandrunningandithasbeensuccessfullyinitializedwiththerequiredconfiguration

<service>-

Thelaststatementinthemainfunctionbeforetheapplicationexitsorisintheshutdownhook

<service>-

config Usedforconfiguration-relatedmessages<service>-

process Usedforprocessing-relatedmessages<service>-

exception Usedforruntimeexceptionhandlerandrelatedmessages

Tagsareparticularlyusefultofilterthelogmessagesoriginatingfromaservice

ofinterestandalsohelpstofurtherdrilldownthelogsviaspecificstateleveltags,suchaslogmessagesgeneratedatstartup,duringshutdown,orloggedasaresultofanexception.

Inadditiontothetags,itisalsorecommendedtoalwaysuseUTC(https://en.wikipedia.org/wiki/Coordinated_Universal_Time)whilegeneratinglogmessages.Sincealltheselogmessagesareaggregatedinacentralizedrepository,havingdifferenttimezonemakesitchallengingtoanalyzethem,astheywillbeoutofsyncduetothetimezoneofthehostmachinesthatmayberunningindifferenttimezones.

Althoughlogmessagesarequiteusefultodebugtheissuesandprovideinformationregardingthestateoftheapplication,theyaffecttheperformanceoftheapplicationdrastically.So,logjudiciouslyandasynchronouslyasmuchaspossible.Itisalsorecommendedtopublishthelogeventsasynchronouslytochannels,suchasApacheKafka,insteadofloggingtoafilethatrequiresdiskI/O.LogstashhasaninputpluginforKafka(https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html)thatcanreadeventsfromaKafkatopicandpublishittothetargetoutputpluginlikethatofElasticsearch(https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html).

Riemann(http://riemann.io/)isanalternativetoELKstack.Itisusedtomonitordistributedsystems,suchastheonesbasedonmicroservices-basedarchitecture.Riemannisincrediblyfastandcanbeusedtogeneratealertsinnearrealtimewithoutoverwhelmingtherecipient,usingitsrollupandthrottleconstructs(http://riemann.io/howto.html#roll-up-and-throttle-events).

UsingELKstacktocollecttheeventsand,atthesametime,streamingLogstasheventsviaRiemannusingLogstashRiemannoutputplugin(https://www.elastic.co/guide/en/logstash/current/plugins-outputs-riemann.html)makesitpossibletogeneratealertsinnearrealtimeandalsousethegoodnessofElasticsearchandKibanatoprovideareal-timemonitoringdashboardfordrill-downanalysis.

DeployingmicroservicesatscaleMicroservicesmustbepackagedasaself-containedartifactthatcanbereplicatedanddeployedusingasinglecommand.Theservicesshouldalsobelightweightwithshorterstarttimestomakesurethattheyareupandrunningwithinseconds.Itisrecommendedtopackagemicroserviceswithinacontainer(https://en.wikipedia.org/wiki/LXC)thatcanthenbedeployedfasterduetoitsinherentimplementationascomparedtosettingupabaremetalmachinewithahostoperatingsystemandrequireddependencies.Packagingmicroserviceswithincontainersalsomakesitpossibletomovefromdevelopmenttoproductionfasterandinanautomatedfashion.

IntroducingContainersandDockerLinuxContainers(LXC)isavirtualizationmethodattheoperatingsystemlevelthatmakesitpossibletorunmultipleisolatedLinuxsystems,alsoknownascontainers,onasinglehostOSusingasingleLinuxKernel(https://en.wikipedia.org/wiki/Linux_kernel).Theresourcesaresharedamongthecontainersusingcgroups(https://en.wikipedia.org/wiki/Cgroups)thatdonotrequirevirtualmachines.SinceeachcontainerreliesontheLinuxKernelofthehostOSthatisalreadyrunning,thestarttimeofcontainersismuchlowerascomparedtoavirtualmachinethatisrunbyaHypervisor(https://en.wikipedia.org/wiki/Hypervisor).

Docker(https://en.wikipedia.org/wiki/Docker_(software))alsoprovidesresourceisolationforthecontainersusingLinuxcgroups,kernelnamespaces(https://en.wikipedia.org/wiki/Linux_namespaces),andunionmountingoption(https://en.wikipedia.org/wiki/Union_mount)thathelpsittoavoidtheoverheadofstartingandmaintainingvirtualmachines.UsingDockercontainerformicroservicesmakesitpossibletopackagetheentireserviceanditsdependencieswithinacontainerandrunonanyLinuxserver.

Althoughitispossibletopackagetheentiremicroservice,includingthedatabasewithinaDockercontainer,itisrecommendedthatyoukeepthedatabaseoutoftheDockercontainer.Reasonbeingthatdatabasesmaynotbetheprimecandidatefordynamicscaleupandscaledownascomparedtotheservicesthemselves.

SettingupDockerDockerisasoftwarethathelpscreateDockerimagesthatcanbeusedtocreateoneormorecontainersandrunitonthehostoperatingsystem.TheeasiestwaytosetupDockeristousethesetupscriptprovidedbyDockerforthecommunityedition,asshownhere:

%wget-qO-https://get.docker.com/|sh

TheprecedingcommandwillsetupDocker,basedonthehostoperatingsystem.DockeralsoprovidesprebuiltpackagesforallthepopularoperationsystemsunderitsDownloadsection(https://www.docker.com/community-edition#/download).Onceinstalled,addthecurrentusertothedockergroup,asshownhere:

%sudousermod-aGdocker$USER

Youmayhavetostartanewloginsessionforgroupmembershiptobetakenintoaccount.Oncedone,Dockershouldbeupandrunning.TotestDocker,trylistingtherunningcontainersusingthefollowingcommand:

%dockerps-a

CONTAINERIDIMAGECOMMANDCREATEDSTATUSPORTSNAMES

Sincetherearenocontainersrunning,itwilljustlisttheheaders.Totestrunningacontainer,usethedockerruncommand,asshowninthefollowingexample.Itwilldownloadthehello-worldDockerimageandrunitinacontainer:

%dockerrunhello-world

Unabletofindimage'hello-world:latest'locally

latest:Pullingfromlibrary/hello-world

ca4f61b1923c:Pullcomplete

Digest:sha256:66ef312bbac49c39a89aa9bcc3cb4f3c9e7de3788c944158df3ee0176d32b751

Status:Downloadednewerimageforhello-world:latest

HellofromDocker!

Thismessageshowsthatyourinstallationappearstobeworkingcorrectly.

Togeneratethismessage,Dockertookthefollowingsteps:

1.TheDockerclientcontactedtheDockerdaemon.

2.TheDockerdaemonpulledthe"hello-world"imagefromtheDockerHub.

(amd64)

3.TheDockerdaemoncreatedanewcontainerfromthatimagewhichrunsthe

executablethatproducestheoutputyouarecurrentlyreading.

4.TheDockerdaemonstreamedthatoutputtotheDockerclient,whichsentit

toyourterminal.

Totrysomethingmoreambitious,youcanrunanUbuntucontainerwith:

$dockerrun-itubuntubash

Shareimages,automateworkflows,andmorewithafreeDockerID:

https://cloud.docker.com/

Formoreexamplesandideas,visit:

https://docs.docker.com/engine/userguide/

Theprecedingoutputisaresultoftheexecutionofthehello-worldimage.DumpingtheentiresetofstepsthatwereusedbyDockertogeneratethemessageishelpfultounderstandwhatasinglecommandsuchasdockerrundoesattheback.

Theprecedingcommanddownloadsthehello-worldimage,storesitlocally,andthenrunsitwithinacontainer.Theimagejustdumpsamessagewiththestepsthatwereusedtogeneratethemessageontheconsoleandexits.Since,thehello-worldimagewasusedforthefirsttime,itwasnotpresentonthelocalmachine,andthatisthereasonthedockercommanddownloadeditfirstfromremoteDockerRegistry(https://docs.docker.com/registry/).Tryrunningthesamecommandagain,and,thistime,itwilllocatetheimagewithinthelocalmachineanduseitstraightaway,asshowninthefollowingexample.Inthiscase,itjustdumpsthemessageandtheinstallationstepsasanoutputoftheexecutionofthehello-worldimageasbefore:

%dockerrunhello-world

HellofromDocker!

Thismessageshowsthatyourinstallationappearstobeworkingcorrectly.

Togeneratethismessage,Dockertookthefollowingsteps:

1.TheDockerclientcontactedtheDockerdaemon.

2.TheDockerdaemonpulledthe"hello-world"imagefromtheDockerHub.

(amd64)

3.TheDockerdaemoncreatedanewcontainerfromthatimagewhichrunsthe

executablethatproducestheoutputyouarecurrentlyreading.

4.TheDockerdaemonstreamedthatoutputtotheDockerclient,whichsentit

toyourterminal.

Totrysomethingmoreambitious,youcanrunanUbuntucontainerwith:

$dockerrun-itubuntubash

Shareimages,automateworkflows,andmorewithafreeDockerID:

https://cloud.docker.com/

Formoreexamplesandideas,visit:

https://docs.docker.com/engine/userguide/

TolisttheDockerimagesavailableonthelocalmachine,executethedockerimagescommandasshowninthefollowingexample;itlistsalltheavailableimages:

%dockerimages

REPOSITORYTAGIMAGEIDCREATEDSIZE

hello-worldlatestf2a91732366c7weeksago1.85kB

Similarly,tolisttheDockercontainersthatwerecreated,usethedockerps-acommand,asusedearlier.Thistime,itshouldlistthecontainersthatwerestartedusingthehello-worldimage,asshownhere:

%dockerps-a

e0e5678ef80ahello-world"/hello"5minutesagoExited(0)5minutesago

happy_rosalind

ea9815d87660hello-world"/hello"8minutesagoExited(0)8minutesago

fervent_engelbart

Formoredetailsontheavailablecommandsandoptions,refertotheDockerCLIcommandsreferenceguide(https://docs.docker.com/engine/reference/commandline/docker/).

FormoredetailsonDockersetupandconfigurationoptions,takealookattheDockerpostinstallationguide(https://docs.docker.com/engine/installation/linux/linux-postinstall/)

CreatingaDockerimageforHelpingHandsHelpingHandsservicescreatedinthepreviouschaptershaveaDockerfilecreatedasapartoftheprojecttemplate.Forexample,takealookatthedirectorystructureoftheAuthservice,asshowninthefollowingexample.TorunwithinaDockercontainer,the::http/hostkeyoftheservicedefinitionwithinthehelping-hands.auth.servicenamespacemustbesettoafixedIPaddressor0.0.0.0tobindtoalltheIPv4addressesavailablewithinthecontainer.

%tree-L1

├──config

├──Dockerfile

├──README.md

├──resources

├──src

├──target

└──test

5directories,4files

ChangethecontentoftheDockerfilefortheAuthservice,asshowninthefollowingexample.Itcopiesboththeconfigdirectoryandthestand-aloneJARfileoftheAuthserviceofHelpingHands.Ifthestand-aloneJARisnotpresentinthetargetfolder,createitusingtheleinuberjarcommandthatwillcreateastand-aloneJARinthetargetdirectoryoftheAuthproject.

FROMjava:8-alpine

MAINTAINERHelpingHands<helpinghands@hh.com>

COPYtarget/helping-hands-auth-0.0.1-SNAPSHOT-standalone.jar/helping-hands/app.jar

COPYconfig/conf.edn/helping-hands/

EXPOSE8080

CMDexecjava-Dconf=/helping-hands/conf.edn-jar/helping-hands/app.jar

Next,createaDockerimageusingthedockerbuildcommand,asshowninthefollowingexample.ThedockerbuildcommandlooksforaDockerfileinthesamedirectorywhereitstartedfrom.IfDockerfileispresentatsomeotherlocation,explicitpathtotheDockerfilecanbespecifiedusingthe-fflag.Formoredetails

onthedockerbuildcommand,refertotheusageinstructions(https://docs.docker.com/engine/reference/builder/#usage).

#buildthedockerimage

%dockerbuild-thelping-hands/auth:0.0.1.

SendingbuildcontexttoDockerdaemon48.44MB

Step1/6:FROMjava:8-alpine

8-alpine:Pullingfromlibrary/java

709515475419:Pullcomplete

38a1c0aaa6fd:Pullcomplete

5b58c996e33e:Pullcomplete

Digest:sha256:d49bf8c44670834d3dade17f8b84d709e7db47f1887f671a0e098bafa9bae49f

Status:Downloadednewerimageforjava:8-alpine

--->3fd9dd82815c

Step2/6:MAINTAINERHelpingHands<helpinghands@hh.com>

--->Runningindd79676d69a4

--->359095b88f32

Removingintermediatecontainerdd79676d69a4

Step3/6:COPYtarget/helping-hands-auth-0.0.1-SNAPSHOT-standalone.jar/helping-

hands/app.jar

--->952111f1c330

Removingintermediatecontainer888323c4cc30

Step4/6:COPYconfig/conf.edn/helping-hands/

--->3c43dfd4af83

Removingintermediatecontainer028df1e03d58

Step5/6:EXPOSE8080

--->Runningin8cf6c15cab9f

--->e79d993e2c67

Removingintermediatecontainer8cf6c15cab9f

Step6/6:CMDexecjava-Dconf=/helping-hands/conf.edn-jar/helping-hands/app.jar

--->Runningin0b4549cf84f2

--->f8c9a7e746f3

Removingintermediatecontainer0b4549cf84f2

Successfullybuiltf8c9a7e746f3

#listtheimagestomakesureitisavailable

%dockerimages

REPOSITORYTAGIMAGEIDCREATEDSIZE

helping-hands/auth0.0.1f8c9a7e746f317secondsago174MB

hello-worldlatestf2a91732366c7weeksago1.85kB

java8-alpine3fd9dd82815c10monthsago145MB

Oncetheimageiscreatedandregisteredusingthespecifiednameandtag,newcontainerscanbecreatedfromthesameimage,asshownhere:

#createanewcontainerfromthetaggedimage

%dockerrun-d-p8080:8080--namehh_auth_01helping-hands/auth:0.0.1

286f21a088dd8b6b6d814f1fb5e4d27a59f46b6d8c474160628ffe72d3de2b56

#verifythatthecontainerisrunning

%dockerps-a

286f21a088ddhelping-hands/auth:0.0.1"/bin/sh-c'exec..."5secondsagoUp3

seconds0.0.0.0:8080->8080/tcphh_auth_01

happy_rosalind

fervent_engelbart

CheckthelogmessagesgeneratedbyDockertomakesurethatAuthserviceisupandrunning,asshownhere:

%dockerlogs286f21a088dd

{:conf#object[java.io.File0x47c40b56"/helping-hands/conf.edn"]}

TheAuthservicecannowbeaccesseddirectlyatthe8080portasshowninthefollowingexample.Theportismappedusingthedockerruncommandwith-pflag,asusedearlierwhilecreatingthecontainer.

%curl-i"http://localhost:8080/tokens?uid=hhuser&pwd=hhuser"

HTTP/1.1200OK

Authorization:Bearer

eyJlbmMiOiJBMTI4Q0JDLUhTMjU2IiwiYWxnIjoiQTEyOEtXIn0.1enLmASKP8uqPGvW_bOVcGS8-

0wtR3AS0xxGolaNixXCSXaY_7LKqw.RcXp4s0397a3M_EB-DyFAQ.B6b93-

1_grZa7HJee6nkcT4LM3gV7QxmR3CIHxX9ngzFqPyyJTcBWvo2N4TTlY4gJYgeNtIyaJsAmvVYCEi7YKyp47bF1wzgFbpjkfVen6y-

580kmf5JqaP2vXQmNpFiVRB6FGGqldnAaDKdBCCrv0HRgGbaxyg_F_05j4G9AktO26hUMfXvmd9woh61Id-

lV4xvRZOcn57X6aH-HL2JuA.hUWvDD6lQWmXaRGYCf3YOQ

Transfer-Encoding:chunked

Tostopthecontainer,usethedockerstopcommand,andtodeletethecontainer,usethedockerrmcommand,asshownhere:

%dockerstophh_auth_01

hh_alert_01

%dockerrmhh_auth_01

hh_alert_01

%dockerps-a

happy_rosalind

fervent_engelbart

FormoredetailsonhowtocreateeffectiveDockerfile,takealookatthebestpracticesforwritingDockerfiles(https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/).

IntroducingKubernetesContainerizingtheservicesoftheHelpingHandsapplicationallowsthemtobedeployedacrossmultiplemachinesfaster,butscalingthemrequiresmanualeffortandinvolvementoftheDevOpsteamtoscaleitupanddown.Monitoringalltherunningcontainersmayalsobecomeoverwhelmingovertime,astheservicesmayscaletohundredsofinstancesrunningcontainersacrosstheclusterofmachines.Althoughthefailureofanyserviceorcontainercanbealertedtotheteam,butrunningthemmanuallyisatedioustask.Moreover,itisexhaustiveandoftenerror-pronetoestimateandachieveeffectiveresourceutilizationandoptimallybalancethenumberofrunninginstancesofeachservicemanually.

Toavoidsuchmanualtasksandensurethattheconfigurednumberofservicesarealwaysrunningandeffectivelyutilizingtheavailableresources,containerorchestrationenginesarerequired.Kubernetesisonesuchopensourcecontainerorchestrationenginethatiswidelyusedforautomateddeployment,scaling,andmanagementofcontainerizedapplicationssuchastheservicesoftheHelpingHandsapplication.

InaKubernetesdeployment,therearetwokindsofmachines,MasterandNode(previouslyknownasMinions).MasterinstancesarethebrainofKubernetesenginethatmakeallthedecisionsrelatedtothedeploymentofcontainersandalsorespondtovariouseventsoffailuresandnewallocationrequests.Masterinstancerunskube-apiserver(https://kubernetes.io/docs/admin/kube-apiserver/),etcd(https://kuber

netes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/),kube-controller-manager(https://kubernetes.io/docs/admin/kube-controller-manager/),andkube-scheduler(https://kubernetes.io/docs/admin/kube-scheduler/).Theyalsorunkube-proxy(https://kubernetes.io/docs/admin/kube-proxy/)toworkwithinthesameoverlaynetworkasthatofnodes.ItisrecommendedtorunMasteronseparatemachinesthatarededicatedforclustermanagementtasksonly.

Nodes,ontheotherhand,areworkermachinesinaKubernetesclusterandrunsPods.APod(https://kubernetes.io/docs/concepts/workloads/pods/pod/)isasmallestunitofcomputingthatcanbecreatedandmanagedbyKubernetescluster.Itisagroupofoneormorecontainersthatsharenetwork,storage,andacommonsetofspecifications.EachNoderunsaDockerservice,kubelet(https://kubernetes.io/docs/admin/kubelet/),andkube-proxyandismanagedbythemastercomponents.ThekubeletagentrunsoneachNodeandmanagesthepodsthatareallocatedtotheNode.ItalsoreportsthestatusofthepodsbacktotheKubernetescluster.

FormoredetailsonKubernetesMasterandNodecomponents,takealookatKubernetesConceptsdocumentathttps://kubernetes.io/docs/concepts/overview/components/.

Kuberneteshasanin-builtsupportforDockercontainers.ServicesuchasAutomaticBinPacking(https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/)foreffectiveutilizationoftheresources,horizontalscaling(https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)toscaletheservicesupanddownandbasedonfactorssuchasCPUusageandSelf-Healing(https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller/#what-is-a-replicationcontroller)toautomaticallyrestartcontainersthatfailisprovidedout-of-the-boxbyKubernetes.

KubernetesalsosupportsServiceDiscoveryandLoadBalancingbyallocatingcontainerstheirownIPaddresses.ItalsoallocatesacommonDNSnameforasetofcontainersthatallowotherexternalservicestojustknowtheDNSnameandusethesametoreachtheservice.KubernetesinternallybalancestherequestsamongtheservicesrunninginthecontainersthatareregisteredwiththeDNSwiththespecifiedname.RollingUpgrades(https://kubernetes.io/docs/tutorials/kubernetes-basics/update-intro/)arealsoprovidedbyKubernetesbyincrementallyupgradingthecontainerswiththenewones.AllupdatesareversionedbyKubernetes,anditallowsrollbacktoanypreviousstableversion.

FormoredetailsonKubernetes,takealookatKubernetestutorialsthatcoverallthebasicfeaturesofKuberneteswithexamplesfromhttps://kubernetes.io/docs/tutorials/.

GettingstartedwithKubernetesThesimplestwaytogetstartedwithKubernetesandrunlocallyonasinglemachineistouseMinikube(https://github.com/kubernetes/minikube).TosetupMinikube,usethefollowinginstallationscript:

%curl-Lominikubehttps://storage.googleapis.com/minikube/releases/latest/minikube-

linux-amd64&&chmod+xminikube&&sudomvminikube/usr/local/bin/

TheprecedingcommanddownloadsthelatestreleaseofMinikubescriptandmakesitavailableonthepathbycopyingittothe/usr/local/bindirectory.Minikubealsorequireskube-ctltointeractwiththeKubernetescluster.Tosetupkube-ctl,usetheinstallationscriptofkube-ctl,asshownhere:

#downloadkube-ctlscript

%curl-LOhttps://storage.googleapis.com/kubernetes-release/release/$(curl-s

https://storage.googleapis.com/kubernetes-

release/release/stable.txt)/bin/linux/amd64/kubectl

#makethescriptexecutable

%chmod+x./kubectl

#makeitavailableonthepath

%sudomv./kubectl/usr/local/bin/kubectl

FormoredetailsonhowtousetheminikubecommandtocreateaKubernetesclusterandkube-ctltointeractwiththeMasteranddeploycontainers,takealookattheMinikubeprojectdocumentation(https://github.com/kubernetes/minikube).

SummaryInthischapter,youlearnedhowtopreparemicroservicesforproduction.WefocusedonsecurityandmonitoringofmicroservicesoftheHelpingHandsapplication.Wealsolearnedhowtodeployandscalemicroservicesusingcontainers.WealsodiscussedorchestrationenginessuchasKubernetesandhowtheyareusefultoorchestratecontainers.Withthischapter,youareallsettobuildyournextbestapplicationusingmicroservices-basedarchitectureanddeployiteffectivelyinproduction.Whatwillyoubuildnext?

OtherBooksYouMayEnjoyIfyouenjoyedthisbook,youmaybeinterestedintheseotherbooksbyPackt:

MasteringMicroserviceswithJava9-SecondEditionSourabhSharma

ISBN:978-1-78728-144-8

Usedomain-drivendesigntodesignandimplementmicroservicesSecuremicroservicesusingSpringSecurityLearntodevelopRESTservicedevelopmentDeployandtestmicroservicesTroubleshootanddebugtheissuesfacedduringdevelopmentLearningbestpracticesandcommonprincipalsaboutmicroservices

Spring5.0Microservices-SecondEditionRajeshRV

ISBN:978-1-78712-768-5

FamiliarizeyourselfwiththemicroservicesarchitectureanditsbenefitsFindouthowtoavoidcommonchallengesandpitfallswhiledevelopingmicroservicesUseSpringBootandSpringCloudtodevelopmicroservicesHandleloggingandmonitoringmicroservicesLeverageReactiveProgramminginSpring5.0tobuildmoderncloudnativeapplicationsManageinternet-scalemicroservicesusingDocker,Mesos,andMarathonGaininsightsintothelatestinclusionofReactiveStreamsinSpringandmakeapplicationsmoreresilientandscalable

Leaveareview-letotherreadersknowwhatyouthinkPleaseshareyourthoughtsonthisbookwithothersbyleavingareviewonthesitethatyouboughtitfrom.IfyoupurchasedthebookfromAmazon,pleaseleaveusanhonestreviewonthisbook'sAmazonpage.Thisisvitalsothatotherpotentialreaderscanseeanduseyourunbiasedopiniontomakepurchasingdecisions,wecanunderstandwhatourcustomersthinkaboutourproducts,andourauthorscanseeyourfeedbackonthetitlethattheyhaveworkedwithPackttocreate.Itwillonlytakeafewminutesofyourtime,butisvaluabletootherpotentialcustomers,ourauthors,andPackt.Thankyou!

microservices with clojure: develop event-driven, scalable ... · chapter 3, microservices for...

Documents

helping hands

helping hands, inc

helping hands - online.gffg.com

helping hands march newsletter

strategic planning report - helping hands...

helping hands client service handbook · helping hands’...

helping hands - june 19

plan for helping hands

helping hands of ny

helping hands caring hearts,

helping hands - 4h.tennessee.edu

the helping hands -

kalavrezou helping hands

helping hands newsletter february

helping hands. on demand

helping hands 2013

helping hands application form - wellchild · web version 1...

helping hands - marie

helping hands instruct manual helping hands

helping hands newsletter