continuous integration, delivery, and deployment
TRANSCRIPT
ContinuousIntegration,Delivery,andDeployment
Reliableandfastersoftwarereleaseswithautomatingbuilds,tests,anddeployment
SanderRossel
BIRMINGHAM-MUMBAI
ContinuousIntegration,Delivery,andDeployment
Copyright©2017PacktPublishing
Allrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem,ortransmittedinanyformorbyanymeans,withoutthepriorwrittenpermissionofthepublisher,exceptinthecaseofbriefquotationsembeddedincriticalarticlesorreviews.
Everyefforthasbeenmadeinthepreparationofthisbooktoensuretheaccuracyoftheinformationpresented.However,theinformationcontainedinthisbookissoldwithoutwarranty,eitherexpressorimplied.Neithertheauthor,norPacktPublishing,anditsdealersanddistributorswillbeheldliableforanydamagescausedorallegedtobecauseddirectlyorindirectlybythisbook.
PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthecompaniesandproductsmentionedinthisbookbytheappropriateuseofcapitals.However,PacktPublishingcannotguaranteetheaccuracyofthisinformation.
Firstpublished:October2017
Productionreference:1271017
PublishedbyPacktPublishingLtd.LiveryPlace35LiveryStreetBirminghamB32PB,UK.
ISBN978-1-78728-661-0
www.packtpub.com
Credits
Author
SanderRossel
CopyEditor
AkshataLobo
Reviewers
WenGu
OleksandrTkachuk
RuiVilão
OlgaFilipova
ProjectCoordinator
DevanshiDoshi
CommissioningEditor
AshwinNair
Proofreader
SafisEditing
AcquisitionEditor
ShwetaPant
Indexer
PratikShirodkar
ContentDevelopmentEditor
RoshanKumar
Graphics
JasonMonteiro
TechnicalEditors
BharatPatil
SachinSunilkumar
HarshalKadam
ProductionCoordinator
ShraddhaFalebhai
AbouttheAuthorSanderRosselisaprofessionaldeveloperwithworkingexperiencein.NET(VBandC#,WinForms,MVC,WebAPI,andEntityFramework),JavaScript,Git,Jenkins,Oracle,andSQLServer.Hehasaninterestinvarioustechnologiesincluding,butnotlimitedto,functionalprogramming,NoSQL,ContinuousIntegration(andmoregenerally,softwarequality),andsoftwaredesign.
Hehaswrittentwoe-bookssofar:Object-OrientedProgramminginC#SuccinctlyandSQLServerforC#DevelopersSuccinctly,whichyoucandownloadfromSyncfusionforfree.HeseekstoeducateothersthroughhisarticlesonhisCodeProjectprofile,andthroughhisbookwriting.
Iwouldliketothankmyparentsfortheirsupportandadvice.Also,ofcourse,IwouldliketothankthepeopleatPackt,andespeciallyShwetaPant,JohannBarrettoandRoshanKumar,formakingthisbookpossible.
AbouttheReviewersWenGuhasworkedatseveralindustryleadingtechnologycompanies.Heledtheefforttocreatelarge-scaleContinuousIntegrationandContinuousDeliveryplatformsandsolutionstodrivetheadoptionofContinuousIntegrationandContinuousDelivery.
Iwouldliketothanktomywife,Annie,andmydaughter,Tiffany,fortheirinspirationandlove.Iwouldalsoliketothankmycolleaguesatworkfortheirencouragementandadvice.
OleksandrTkachukhasmorethan15yearsofexperience.Hehasbeenworkingasasoftwareengineerandisresponsibleforfulllifecycledevelopmentofnext-generationsoftware,frominitialrequirementgathering,planning,andanalysistodesign,coding,testing,documentation,andimplementation.
HegraduatedfromLvivPolytechnicNationalUniversity,Ukraine,withamaster'sdegreeincomputersystemsandnetworks.
HeworkedasaSeniorDeveloper,TeamLeader,SolutionArchitect,SolutionArchitect,andalsoworkedinUkraine,UK,andGermanyITcompanies.
Thisishisfirstreviewedbook.
RuiVilãoandOlgaFilipovaareahappycoupleofsoftwareengineersandtravelerscurrentlylivinginBerlin.RuiisoriginallyfromCoimbra,Portugal,andOlgaisfromKiev,Ukraine.
BothRuiandOlgaaretechnicalcofoundersofanon-profitonlineeducationprojectcalledEdErabasedinUkraine.Besidesthat,OlgaisaleadsoftwareengineeratafintechcompanycalledOptioPaybasedinBerlin,andRuiisaleadsoftwareengineeratanonlinefitnesscompanycalledGymondobasedinBerlin.
TheyliveinBerlinsince2014;beforethat,theylivedinPortugal,wherebothgraduatedasmastersincomputerscienceandworkedfor5yearsatFeedzai--themostsuccessfulPortuguesestartupthatpreventsfraudallovertheworld.
OlgaisauthorofLearningVue.js2andWebdevelopmentwithVue.js,Bootstrap,andFirebase,andRuiunofficiallyreviewedboththesebooks.
www.PacktPub.comForsupportfilesanddownloadsrelatedtoyourbook,pleasevisitwww.PacktPub.com.DidyouknowthatPacktofferseBookversionsofeverybookpublished,withPDFandePubfilesavailable?YoucanupgradetotheeBookversionatwww.PacktPub.comandasaprintbookcustomer,youareentitledtoadiscountontheeBookcopy.Getintouchwithusatservice@packtpub.comformoredetails.Atwww.PacktPub.com,youcanalsoreadacollectionoffreetechnicalarticles,signupforarangeoffreenewslettersandreceiveexclusivediscountsandoffersonPacktbooksandeBooks.
https://www.packtpub.com/mapt
Getthemostin-demandsoftwareskillswithMapt.MaptgivesyoufullaccesstoallPacktbooksandvideocourses,aswellasindustry-leadingtoolstohelpyouplanyourpersonaldevelopmentandadvanceyourcareer.
Whysubscribe?FullysearchableacrosseverybookpublishedbyPacktCopyandpaste,print,andbookmarkcontentOndemandandaccessibleviaawebbrowser
CustomerFeedbackThanksforpurchasingthisPacktbook.AtPackt,qualityisattheheartofoureditorialprocess.Tohelpusimprove,pleaseleaveusanhonestreviewonthisbook'sAmazonpageathttps://www.amazon.com/dp/1787286614.Ifyou'dliketojoinourteamofregularreviewers,youcane-mailusatcustomerreviews@packtpub.com.WeawardourregularreviewerswithfreeeBooksandvideosinexchangefortheirvaluablefeedback.Helpusberelentlessinimprovingourproducts!
TableofContentsPreface
Whatthisbookcovers
Whatyouneedforthisbook
Whothisbookisfor
Conventions
Readerfeedback
Customersupport
Downloadingtheexamplecode
Downloadingthecolorimagesofthisbook
Errata
Piracy
Questions
1. ContinuousIntegration,Delivery,andDeploymentFoundationsContinuousIntegration
Sourcecontrol
CIserver
Softwarequality
Unittests
Integrationtests
Bigbangtesting
Incrementaltesting
Acceptancetests
Smoketests
Otherqualitygates
Automation
Teamwork
ContinuousDelivery
ContinuousDeployment
Summary
2. SettingUpaCIEnvironmentInstallingaVirtualMachine
InstallingUbuntu
InstallingGit
InstallingGitLab
ConfiguringGitLab
UsingGit
InstallingJenkins
InstallingJenkinsonUbuntu
InstallingJenkinsonWindows
ConfiguringJenkins
InstallingPostgreSQL
InstallingPostgreSQLonUbuntu
InstallingPostgreSQLonWindows
InstallingpgAdmin
InstallingSonarQube
ConfiguringPostgreSQL
InstallingSonarQubeonUbuntu
InstallingSonarQubeonWindows
TriggerSonarQubefromJenkins
Summary
3. VersionControlwithGitThebasics
CentralizedSourceControlManagement
DistributedSourceControlManagement
Theworkingdirectory
Thestagingarea
Committingandpushing
Reviewingcommits
Pullingandstashing
Branching
Merging
Cherrypicking
Rebasing
Revertingchanges
Thebranchingmodel
Tagging
Summary
4. CreatingaSimpleJavaScriptAppThewebshopspecs
InstallingNode.jsandnpm
Creatingtheproject
CreatingtheHomepage
CreatingtheProductpage
CreatingtheSearchpage
CreatingtheShoppingcartpage
Summary
5. TestingYourJavaScriptUnittestingwithJasmine
TheYeomanshortcut
WhyJasmine?
Testingthewebshop
RunningtestswithKarma
Installation
Karmaplugins
Browserlaunchers
Codecoverage
JUnitreporter
RunningMochaandChaiwithKarma
End-To-EndtestingwithSelenium
RunningSeleniumtestswithProtractor
Testingourwebsite
Customizingreporters
TestingwithMocha
Headlessbrowsertesting
PhantomJS
Summary
6. AutomationwithGulpGulpbasics
TheGulpAPI
Gulpplugins
Minification
Cleaningyourbuild
Checkingfilesizes
LintingwithJSHint
RunningyourKarmatests
Gettingoursiteproductionready
MinifyingHTML
MinifyingCSS
BundlingJavaScriptwithBrowserify
KarmaandBrowserify
ConcatenatingCSS
ReplacingHTMLreferences
Puttingitalltogether
Summary
7. AutomationwithJenkinsInstallingNode.jsandnpm
CreatingaJenkinsproject
ExecutingGulpinJenkins
Publishingtestresults
JUnitreport
Coberturareport
HTMLreport
Buildtriggers
Buildperiodically
PollSCM
Oncommit
Settingupemailnotifications
SettingupSonarQube
Settingupemail
HTMLandCSSanalysis
Includingcodecoverage
Leakperiods
Artifacts
RunningonWindowswithJenkinsSlaves
Runningourtests
Triggeringaprojectpipeline
Summary
8. ANodeJSandMongoDBWebAppInstallingMongoDB
InstallingMongoDBonUbuntu
InstallingMongoDBonWindows
CreatingtheNode.jsBack-end
Express
EJS
TheLoginPage
ConnectingwithAJAX
SavingtoMongoDB
MovingourProductstoMongoDB
PuttingtheShoppingCartinMongoDB
MovingtheShoppingCartModule
Gulp
Jenkins
PM2
Summary
9. AC#.NETCoreandPostgreSQLWebAppInstalling.NETCoreandVisualStudioCode
Creatingtheviews
Runningtasks
Addingthedatabase
EntityFrameworkCore
Selectingthedata
Fixingthelogin
Addingtothecart
Testingthedatabase
InstallingpgTap
TestingourC#code
Reporting
AddingSeleniumtests
Jenkins
Buildingtheproject
Testingtheproject
Testingthedatabase
Summary
10. AdditionalJenkinsPluginsViews
Cleanupworkspaces
Conditionalbuildsteps
Pipelines
Promotedbuilds
Parameterizedbuilds
Triggeringbuildsremotely
Blueocean
Security
JenkinsonHTTPS
HTTPSonLinux
HTTPSonWindows
Jenkinssecurity
Role-basedauthorizationstrategy
Summary
11. JenkinsPipelinesGroovy
Pipelineprojects
Declarativepipelinesyntax
Scriptedpipelinesyntax
Pipelinestages
Thesnippetgenerator
Buildingthewebshop
TheJenkinsfile
Multibranchpipeline
Branchscripts
Completingthescript
Node.jstests
SonarQube
Seleniumtests
Archivingartifacts
Buildfailure
Summary
12. TestingaWebAPIBuildingaRESTservice
Postman
Writingtests
TestingXML
Collections
Environments
Newman
Summary
13. ContinuousDeliveryBranching
Manualdeployment
InstallingNGINX
Node.jswebshop
PM2
MongoDB
Run
E2Etesting
C#.NETCorewebshop
Run
PostgreSQL
Summary
14. ContinuousDeploymentJavaScriptDeploymentusingSSH
E2Etesting
C#.NETCoreDeploymentusingSSH
E2Etesting
Database
Summary
PrefaceAproblemthatalotofdevelopersfaceisthatsoftwareiscomplexandbecomesonlymorecomplexovertime.Asinglechangetothesoftwarecanleadtonumerousunexpectedbugsthatmaynotbediscoveredintime.UsingContinuousIntegration,wecanautomaticallytestsoftwarebeforeitisreleased.Usingothertools,suchasSonarQube,wecanensurethatourcodeadherestothelateststandards.Unfortunately,gettingstartedwithtestingandautomationrequiresvarioustoolsandallofthosetoolstaketimeandefforttolearn.
Inthisbook,wewillstartaprojectfromscratchanduseContinuousIntegrationtechniquestoguaranteeacertainsoftwarequality.ToolssuchasGit,Jasmine,Karma,Selenium,Protractor,Gulp,Jenkins,SonarQube,andPostmanareintroducedandusedtoensurethatoursoftwareisuptopar.
Finally,tofurtherreducethechancesofhumanerror,wewillautomaticallydeployoursoftwaretoanotherenvironmentsothatwecangofromGitcommittoproductiondeploymentfullyautomatedandstillsleepeasyatnight.
WhatthisbookcoversChapter1,ContinuousIntegration,Delivery,andDeploymentFoundations,startsbyexploringsomeofthetheorycoveringContinuousIntegration,Delivery,andDeployment,aswellasthedifferencesbetweenthethree.
Chapter2,SettingUpaCIEnvironment,teachesushowtoinstallourenvironment.WewillsetupaLinuxVirtualMachineandinstallsomeofthetoolswewillusethroughoutthebook,suchasJenkins,PostreSQL,andSonarQube.
Chapter3,VersionControlwithGit,sourcecontrolisanecessityforanysoftwareprojectandisalsoaprerequisiteforCI,sothisexploresGitandhowtoworkwithit,bothfromacommandlineaswellasfromagraphicaltool.
Chapter4,CreatingaSimpleJavaScriptApp,saysthatbeforewecancontinue,weneedaproject,sowewillcreateasimplewebshopusingonlyfrontendtechnologies.Throughouttherestofthebook,wewillusethisappandexpandonit.
Chapter5,TestingYourJavaScript,informsusthatwecanstartwritingtestsforourprojectusingtheprojectfromChapter4,CreatingaSimpleJavaScriptApp.Forourunittests,wewilluseJasmineandKarma,andforourEnd-To-End(E2E)tests,wewilluseSeleniumandProtractor.
Chapter6,AutomationwithGulp,beginswithautomatingourtestsandaddingothertasks,suchaslintingandotherfrontendwork,toourautomatedbuildusingGulp.
Chapter7,AutomationwithJenkins,takesourautomationastepfurtherwithJenkins.WithJenkins,wecanruntasksautomaticallyonanyGitcommitsothatnocodeisleftuntested.
Chapter8,ANodeJSandMongoDBWebApp,informsusthatmovingourappforward,wewilladdaNode.jsandMongoDBbackendforourwebshop.Thispresentsuswithsomenewchallengesthatarebestautomated.
Chapter9,AC#.NETCoreandPostgreSQLWebApp,repeatsChapter8,ANodeJSAndMongoDBWebApp,butwithC#.NETCoreandPostgreSQL.Additionally,wewilladdSQLdatabasetests.
Chapter10,AdditionalJenkinsPlugins,exploresJenkinsfurthernowthatourappsareprettymuchdone.Wewillseepluginsthatwillmakeyourlifeeasierandyouroptionsgreater.
Chapter11,JenkinsPipelines,divesdeeperintoJenkinstoexploreJenkinspipelines,whicharebasicallyJenkinsconfigurationincode.ThisopensupnewpossibilitiesinJenkins.
Chapter12,TestingaWebAPI,isalittleunrelatedtotherestofthebook,butnotunimportant;here,wewilltakealookatPostmanandAPItesting.
Chapter13,ContinuousDelivery,takesusneartotheendofthebook,andallthatisleftistodeployourapps.First,wewilldoamanualdeployment,andthenautomatethistaskusingJenkins.
Chapter14,ContinuousDeployment,concludeswithcompletelyautomatingourentiredeploymentprocess.ThegoalofthischapteristogetaGitcommittoproductionwithoutanymanualintervention.
WhatyouneedforthisbookForthisbook,wewillneedvarioussoftware,buteverythingisexplainedinthechaptersofthisbook.Acomputerwithatleast16GBmemoryisadvised.
WhothisbookisforTheaudiencewilltypicallybethedeveloper.Developerstendtolosethemselvesintheircodewhilemissingthebiggerpicture.ContinuousIntegration,formanydevelopers,issomethingthatmustbedone,butisnottrulyunderstoodbymany.Thisbookwillgetdevelopers,buthopefullyalsoteamleads,intoContinuousIntegrationandContinuousDeployment,andshowthemtheaddedvalueitbrings.
BasicknowledgeofatleastJavaScriptandHTML/CSSisrequired.KnowledgeofC#andSQLcomesinhandy.Mostprogrammersthathaveprogrammedina(compiled)C-likelanguageshouldbeabletofollowalong.
ConventionsInthisbook,youwillfindanumberoftextstylesthatdistinguishbetweendifferentkindsofinformation.Herearesomeexamplesofthesestylesandanexplanationoftheirmeaning.
Codewordsintext,databasetablenames,foldernames,filenames,fileextensions,pathnames,dummyURLs,userinput,andTwitterhandlesareshownasfollows:"Athousand-linefunctionwithmultiplenestedifandwhileloops(andI'veseenplenty)isprettymuchuntestable."Ablockofcodeissetasfollows:
publicstaticclassMyMath
{
publicstaticintAdd(inta,intb)
{
returna+b;
}
}
Whenwewishtodrawyourattentiontoaparticularpartofacodeblock,therelevantlinesoritemsaresetinbold:
<!DOCTYPEhtml>
<html>
<head>
[...]
</head>
<bodyng-app="shopApp">
<%-includenavbar.ejs%>
<divclass="container"ng-controller="productController">
[...]
</div>
</body>
</html>
Anycommand-lineinputoroutputiswrittenasfollows:
sudoapt-getupdate
sudoapt-getinstallubuntu-desktop
Newtermsandimportantwordsareshowninbold.Wordsthatyouseeonthescreen,forexample,inmenusordialogboxes,appearinthetextlikethis:"Gitwillrewindyourworkuntilitfindsthemaster,whichisaftertheAddedweaponscommit."
Warningsorimportantnotesappearlikethis.
Tipsandtricksappearlikethis.
ReaderfeedbackFeedbackfromourreadersisalwayswelcome.Letusknowwhatyouthinkaboutthisbook-whatyoulikedordisliked.Readerfeedbackisimportantforusasithelpsusdeveloptitlesthatyouwillreallygetthemostoutof.Tosendusgeneralfeedback,[email protected],andmentionthebook'stitleinthesubjectofyourmessage.Ifthereisatopicthatyouhaveexpertiseinandyouareinterestedineitherwritingorcontributingtoabook,seeourauthorguideatwww.packtpub.com/authors.
CustomersupportNowthatyouaretheproudownerofaPacktbook,wehaveanumberofthingstohelpyoutogetthemostfromyourpurchase.
DownloadingtheexamplecodeYoucandownloadtheexamplecodefilesforthisbookfromyouraccountathttp://www.packtpub.com.Ifyoupurchasedthisbookelsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilese-maileddirectlytoyou.Youcandownloadthecodefilesbyfollowingthesesteps:
1. Loginorregistertoourwebsiteusingyouremailaddressandpassword.2. HoverthemousepointerontheSUPPORTtabatthetop.3. ClickonCodeDownloads&Errata.4. EnterthenameofthebookintheSearchbox.5. Selectthebookforwhichyou'relookingtodownloadthecodefiles.6. Choosefromthedrop-downmenuwhereyoupurchasedthisbookfrom.7. ClickonCodeDownload.
Oncethefileisdownloaded,pleasemakesurethatyouunziporextractthefolderusingthelatestversionof:
WinRAR/7-ZipforWindowsZipeg/iZip/UnRarXforMac7-Zip/PeaZipforLinux
ThecodebundleforthebookisalsohostedonGitHubathttps://github.com/PacktPublishing/Continuous-Integration-Delivery-and-Deployment.Wealsohaveothercodebundlesfromourrichcatalogofbooksandvideosavailableathttps://github.com/PacktPublishing/.Checkthemout!
DownloadingthecolorimagesofthisbookWealsoprovideyouwithaPDFfilethathascolorimagesofthescreenshots/diagramsusedinthisbook.Thecolorimageswillhelpyoubetterunderstandthechangesintheoutput.Youcandownloadthisfilefromhttps://www.packtpub.com/sites/default/files/downloads/ContinuousIntegrationDeliveryandDeployment_Col
orImages.pdf.
ErrataAlthoughwehavetakeneverycaretoensuretheaccuracyofourcontent,mistakesdohappen.Ifyoufindamistakeinoneofourbooks-maybeamistakeinthetextorthecode-wewouldbegratefulifyoucouldreportthistous.Bydoingso,youcansaveotherreadersfromfrustrationandhelpusimprovesubsequentversionsofthisbook.Ifyoufindanyerrata,pleasereportthembyvisitinghttp://www.packtpub.com/submit-errata,selectingyourbook,clickingontheErrataSubmissionFormlink,andenteringthedetailsofyourerrata.Onceyourerrataareverified,yoursubmissionwillbeacceptedandtheerratawillbeuploadedtoourwebsiteoraddedtoanylistofexistingerrataundertheErratasectionofthattitle.Toviewthepreviouslysubmittederrata,gotohttps://www.packtpub.com/books/content/supportandenterthenameofthebookinthesearchfield.TherequiredinformationwillappearundertheErratasection.
PiracyPiracyofcopyrightedmaterialontheInternetisanongoingproblemacrossallmedia.AtPackt,wetaketheprotectionofourcopyrightandlicensesveryseriously.IfyoucomeacrossanyillegalcopiesofourworksinanyformontheInternet,pleaseprovideuswiththelocationaddressorwebsitenameimmediatelysothatwecanpursuearemedy.Pleasecontactusatcopyright@packtpub.comwithalinktothesuspectedpiratedmaterial.Weappreciateyourhelpinprotectingourauthorsandourabilitytobringyouvaluablecontent.
QuestionsIfyouhaveaproblemwithanyaspectofthisbook,[email protected],andwewilldoourbesttoaddresstheproblem.
ContinuousIntegration,Delivery,andDeploymentFoundationsContinuousIntegration,Delivery,andDeploymentarerelativelynewdevelopmentpracticesthathavegainedalotofpopularityinthepastfewyears.ContinuousIntegrationisallaboutvalidatingsoftwareassoonasit'scheckedintosourcecontrol,moreorlessguaranteeingthatsoftwareworksandcontinuestoworkafternewcodehasbeenwritten.ContinuousDeliverysucceedsContinuousIntegrationandmakessoftwarejustaclickawayfromdeployment.ContinuousDeploymentthensucceedsContinuousDeliveryandautomatestheentireprocessofdeployingsoftwaretoyourcustomers(oryourownservers).
IfContinuousIntegration,Delivery,andDeploymentcouldbesummarizedwithoneword,itwouldbeAutomation.Allthreepracticesareaboutautomatingtheprocessoftestinganddeploying,minimizing(orcompletelyeliminating)theneedforhumanintervention,minimizingtheriskoferrors,andmakingbuildinganddeployingsoftwareeasieruptothepointwhereeverydeveloperintheteamcandoit(soyoucanstillreleaseyoursoftwarewhenthatonedeveloperisonvacationorcrashesintoatree).Automation,automation,automation,automation...SteveBallmer,wouldsay,whilestompinghisfeetonthegroundandsweatinglikeapig.
TheproblemwithContinuousIntegration,Delivery,andDeploymentisthatit'snotatalleasytosetupandtakesalotoftime,especiallywhenyou'veneverdoneitbeforeorwanttointegrateanexistingproject.However,whendoneright,itwillpayitselfbackbyreducingbugs,makingiteasiertofixthebugsyoufindandproducingbetterqualitysoftware(whichshouldleadtomoresatisfiedcustomers).
ThetermsContinuousIntegration,ContinuousDelivery,andContinuousDeploymentareoftenusedincorrectlyorinterchangeably(andthenI'vealsoseenthetermContinuousRelease).PeoplesayContinuousIntegrationwhentheymeanContinuousDeployment,ortheysayContinuousDeploymentwhen
theymeanDelivery,andsoon.Tomakemattersmorecomplex,somepeopleusethewordDevOpswhentheymeananyoftheContinuousflavors.DevOps,however,ismorethanjustContinuousIntegration,Delivery,and/orDeployment.Whentalkingtopeopleaboutanyofthesesubjects,don'tmakeassumptionsandmakesureyou'reusingthesamedefinitions.DevOpsisoutsidethescopeofthisbook.
ContinuousIntegrationThefirststeptodeliveringconsistentandhigh-qualitysoftwareisContinuousIntegration(CI).CIisallaboutensuringyoursoftwareisinadeployablestateatalltimes.Thatis,thecodecompilesandthequalityofthecodecanbeassumedtobeofreasonablygoodquality.
SourcecontrolCIstartswithsomesharedrepository,typicallyasourcecontrolsystem,suchasSubversion(SVN)orGit.Sourcecontrolsystemsmakesureallcodeiskeptinasingleplace.It'seasyfordeveloperstocheckoutthesource,makechanges,andcheckinthosechanges.Otherdeveloperscanthencheckoutthosechanges.
Inmodernsourcecontrolsystems,suchasGit,youcanhavemultiplebranchesofthesamesoftware.Thisallowsyoutoworkondifferentstagesofthesoftwarewithouttroubling,orevenhalting,otherstagesofthesoftware.Forexample,itispossibletohaveadevelopmentbranch,atestbranch,andaproductionbranch.Allnewcodegetscommittedondevelopment;whenitistestedandapproved,itcanmoveontothetestbranchand,whenyourcustomerhasgivenyouapproval,youcanmoveitintodevelopment.Anotherpossibilityistohaveasinglemainbranchandcreateanew(frozen)branchforeveryrelease.Youcouldstillapplybugfixestoreleasebranches,butpreferablynotnewfeatures.
Don'tunderestimatethevalueofsourcecontrol.Itmakesitpossiblefordeveloperstoworkonthesameprojectandeventhesamefileswithouthavingtoworrytoomuchaboutoverwritingothers'codeorbeingoverwrittenbyothers.
Nexttocode,youshouldkeepeverythingthat'snecessaryforyourprojectinyourrepository.Thatincludesrequirements,testscripts,buildscripts,configurations,databasescripts,andsoon.
Eachcheckintothisrepositoryshouldbevalidatedbyyourautomatedbuildserver.Assuch,it'simportanttokeepcheck-inssmall.Ifyouwriteanewfeatureandchangetoomanyfilesatonce,itbecomeshardertofindanybugsthatarise.
CIserverYourbuildsareautomatedusingsomesortofCIserver.PopularCIserversoftwareincludesJenkins(formerlyHudson),TeamFoundationServer(TFS),CruiseControl,andBamboo.EachCIserverhasitsownprosandcons.TFS,forexample,istheMicrosoftCIserverandworkswellwith.NET(C#,VB.NET,andF#)andintegrateswithVisualStudio.Thefreeversiononlyhaslimitedfeaturesforonlysmallteams.BambooistheAtlassianCIserverand,thus,workswellwithJIRAandBitBucket.LikeTFS,Bambooisnotfree.Jenkinsisopensourceandfreetouse.ItworkswellforJava,inwhichJenkinsitselfwasbuilt,andworkswithplugins.TherearealotofotherCIservers,allwiththeirownprosandcons,butthethingtheyallhaveincommonisthattheyautomatesoftwarebuilds.Forthisbook,wewilluseJenkinsastheCIserverofchoice.
YourCIservermonitorsyourrepositoryandstartsabuildoneverycheckin.Asinglebuildcancompileyourcode,rununittests,calculatecodecoverage,checkstyleguidelines,lintyourcode,minifyyourcode,andmuchmore.Wheneverabuildfails,forexample,becauseaprogrammerforgotasemi-colonandcheckedininvalidcodeorbecauseaunittestfails,theteamshouldbenotified.TheCIservermaysendanemailtotheprogrammerwhocommittedtheoffendingcode,totheentireteam,oryoucoulddonothing(whichisnotbestpractice)andjustcheckthestatusofyourbuildeveryonceinawhile.Theconditionsforfailurearecompletelyuptothedeveloper(ortheteam).Obviously,whenyourcodedoesnotcompilecorrectlybecauseit'smissingasemicolon,that'safail.Likewise,afailingunittestisanobviousfail.Lessobviousisthatabuildcanfailwhenacertainprojectdoesnothaveatleasta90%testcodecoverageoryourtechnicaldebt,thatis,thetimeittakestorewritequickanddirtysolutionstomoreelegantsolutionsgrowstomorethan40hours.
TheCIservershouldbuildyoursoftware,notifyaboutfailuresandsuccesses,andultimatelycreateanartifact.Thisartifact,anexecutableofthesoftware,shouldbeeasilyavailabletoeveryoneontheteam.Sincethebuildpassedalloftheteams,criteriaforpassingabuild,thisartifactisreadyfordeliverytothecustomer.
SoftwarequalityThatbringsustothepointofsoftwarequality.IfabuildonyourCIserversucceeds,itshouldguaranteeacertainlevelofsoftwarequality.I'mnottalkingperfectsoftwarethatisbug-freeallofthetime,butsoftwarethat'swelltestedandcheckedforbestpractices.Numeroustypesoftestsexists,butwewillonlylookatafewoftheminthisbook.
UnittestsOneofthemostimportantthingsyoucandotoguaranteethatcertainpartsofyoursoftwareproducecorrectresultsisbywritingunittests.Aunittestissimplyapieceofcodethatcallsamethod(themethodtobetested)withapredefinedinputandcheckswhethertheresultiswhatyouexpectittobe.Iftheresultiscorrect,itreportssuccess,otherwiseitreportsfailure.Theunittest,asthenameimplies,testssmallandisolatedunitsofcode.
Let'ssayyouwriteafunctionintAdd(inta,intb)inC#(I'mprettysureeveryprogrammercanfollowthough):
publicstaticclassMyMath
{
publicstaticintAdd(inta,intb)
{
returna+b;
}
}
ThefirstthingyouwanttotestiswhetherAddindeedreturnsa+bandnota+a,orb+b,orevensomethingrandom.Thatmaysoundeasierthanitis.IfyoutestwhetherAdd(1,1)returns2andthetestsucceeds,someonemightstillhaveimplementeditasa+aorb+b.Soattheveryleast,youshouldtestitusingtwounequalintegers,suchasAdd(1,2).NowwhathappenswhenyoucallAdd(2147483647,1)?Doesitoverfloworthrowanexceptionandisthatindeedtheoutcomeyoususpected?Likewise,youshouldtestforanunderflow(whileadding!?).-2147483647+-1willnotreturnwhatyou'dexpect.That'sthreeunittestsforsuchasimplefunction!Arguably,youcouldtestfor+/-,-/+,and-/-(-3+-3equals-6andnot0),butyou'dhavetotryreallyhardtobreakthatkindoffunctionality,sothosetestswouldprobablynotgiveyouanextrausefultest.Yourfinalunittestsmaylooksomethinglikethefollowing:
[TestClass]
publicclassMathTests
{
[TestMethod]
publicvoidTestAPlusB()
{
intexpected=3;
intactual=MyMath.Add(1,2);
Assert.AreEqual(expected,actual,"Somehow,1+2didnotequal3.");
}
[TestMethod]
[ExpectedException(typeof(OverflowException))]
publicvoidTestOverflowException()
{
//MyMath.Addcurrentlyoverflows,sothistestwillfail.
MyMath.Add(int.MaxValue,1);
}
[TestMethod]
[ExpectedException(typeof(OverflowException))]
publicvoidTestOverflowException()
{
//MyMath.Addcurrentlyunderflows,sothistestwillfail.
MyMath.Add(int.MinValue,-1);
}
}
Ofcourse,ifyouwriteasingleunittestanditsucceeds,itisnoguaranteethatyoursoftwareactuallyworks.Infact,asinglefunctionusuallyhasmorethanoneunittestalone.Likewise,ifyouhavewrittenathousandunittests,butalltheydoischeckthattrueindeedequalstrue,it'salsonotanyindicationofthequalityofyoursoftware.Laterinthisbook,wewillwritesomeunittestsforoursoftware.Fornow,itsufficestosayyourtestsshouldcoveralargeportionofyourcodeand,atleast,themostlikelyscenarios.Iwouldsayqualityoverquantity,butinthecaseofunittesting,quantityisalsoprettyimportant.Youshouldactuallykeeptrackofyourcodecoverage.Therearetoolsthatdothisforyou,althoughtheycannotcheckwhetheryourtestsactuallymakeanysense.
Itisimportanttonotethatunittestsshouldnotdependuponothersystems,suchasadatabase,thefilesystem,or(third-party)services.Theinputandoutputofourtestsneedtobepredefinedandpredictable.Also,weshouldalwaysbeabletorunourunittests,evenwhenthenetworkisdownandwecan'treachthedatabaseorthird-partyservice.Italsohelpsinkeepingtestsfast,whichisamust,asyou'regoingtohavehundredsoreventhousandsofteststhatyouwanttorunasfastaspossible.Instantfeedbackisimportant.Luckily,wecanmock(orfake)suchexternalcomponents,aswewillseelaterinthisbook.
Justwritingsomeunittestsisnotgoingtocutit.Wheneverabuildpasses,youshouldhavereasonableconfidencethatyoursoftwareiscorrect.Also,youdonotwantunitteststofaileverytimeyoumakeeventheslightestchange.Furthermore,specificationschangeandsodounittests.Assuch,unittestsshouldbeunderstandableandmaintainable,justliketherestofyourcode.Andwritingunittestsshouldbeapartofyourdaytodayjob.Writesomecode,thenwritesomeunittests(orturnthataroundifyouwanttodoTest-Driven
Development).Thismeanstestingisnotsomethingonlytestersdo,butthedevelopersaswell.
Inordertowriteunittests,yourcodeshouldbetestableaswell.Eachifstatementmakesyourcodehardertotest.Eachfunctionthatdoesmorethanonethingmakesyourcodehardertotest.Athousand-linefunctionwithmultiplenestedifandwhileloops(andI'veseenplenty)isprettymuchuntestable.Sowhenwritingunittestsforyourcode,youareprobablyalreadyrefactoringandmakingyourcodeprettierandeasiertoread.Anotheraddedbenefitofwritingunittestsisthatyouhavetothinkcarefullyaboutpossibleinputsanddesirableoutputsearly,whichhelpsinfindingedgecasesinyoursoftwareandpreventingbugsthatmaycomefromthem.
IntegrationtestsCheckingwhetheranAddfunctionreallyaddsaandbisnice,butdoesnotreallygiveyouanindicationthatthesystemasawholeworksaswell.Assaid,unittestsonlytestsmallandisolatedunitsofcodeandshouldnotinteractwithexternalcomponents(externalcomponentsaremocked).Thatiswhyyouwillwantintegrationtestsaswell.Integrationteststestwhetherthesystemasawholeoperatesasexpected.Weneedtoknowwhetherarecordcanindeedbesavedinandretrievedfromadatabase,thatwecanrequestsomedatafromanexternalservice,andthatwecanlogtosomefileonthefilesystem.Or,morepracticallywecancheckwhetherthefrontendthatwascreatedbythefrontendteamactuallyfitsthebackendthatwascreatedbythebackendteam.Ifthesetwoteamshavehadanyproblemsorconfusionincommunication,theintegrationtestswill,hopefully,sortthatout.
Lastyear,wecreatedaserviceforathirdpartywhowantedtointerfacewithasystemwewrote.Theservicedidnotdoalotbasicallyittookthereceivedmessageandforwardedittoanotherservicethatweusedinternally(andwasn'tavailableoutsideofthenetwork).Theinternalservicehadallofthebusinessrulesandcouldreadfrom,andwriteto,adatabase.Furthermore,itwould,insomecases,createadditionaljobsthatwouldbeputona(asynchronous)queue,whichisyetanotherservice.Last,afourthservicewouldpickupanymessagesfromthequeueandprocessthem.Inordertoprocessasinglerequest,wepotentiallyneededfivecomponents(externalservice,internalservice,database,queue,andqueueprocessor).Theinternalservicewasthoroughlyunittested,sothebusinessruleswerecovered.However,thatstillleavesalotofroomforerrorsandexceptionswhenoneofthecomponentsisnotavailableorhasanincompatibleinterface.
BigbangtestingTherearetwoapproachestointegrationtesting:bigbangtestingandincrementaltesting.Withbigbangtesting,yousimplywaituntilallthecomponentsofasystemarereadyandthenstarttesting.Inthecaseofmyservice,thatmeantdevelopingandinstallingeverything,thenpostingsomerequestsandcheckingwhethertheexternalservicecouldcalltheinternalservice,andwhethertheinternalservicecouldaccessthedatabaseandthequeueand,notunimportant,givefeedbacktotheexternalservice.Furthermore,ofcourse,Ihadtotestwhetherthequeuetriggeredtheprocessingserviceandwhethertheprocessingserviceprocessedthemessagecorrectlytoo.
Inreality,theprocessingalsousedthedatabase;itputnewmessagesonthequeueandsentemailsincaseoferrors.Additionally,allthecomponentshadtoaccesstheharddriveforloggingtoafile(anddonotassumethefilesystemisalwaysavailable;thefirsttimeonproductionIactuallyranintoanUnauthorizedExceptionandnothingwaslogged).Sothatmeansevenmoreintegrationtesting.
IncrementaltestingWithincrementaltesting,youtestcomponentsassoonastheyareavailableandyoucreatestubsordrivers(somesortofplaceholder)forcomponentsthatarenotyetavailable.Therearetwoapproacheshere:
Top-downtesting:Usingtop-downtestingwouldmeanIwould'vecheckedwhethertheexternalservicecouldmakeacalltotheinternalserviceand,iftheinternalservicewasnotavailableyet,createastubthatpretendstobetheinternalservice.
Bottom-uptesting:Bottom-upistestingtheotherwayaround,soI'dstarttestingtheinternalserviceandcreateadriverthatmimicstheexternalservice.
Incrementaltestinghastheadvantagethatyoucanstartdefiningtestsearlybeforeallthecomponentsarecomplete.Afterthat,itbecomesamatteroffillinginthegaps.
AcceptancetestsAfterhavingunittestedourcodeandcheckedwhetherthesystemasawholeworks,wecannowassumeoursoftwareworksandisofdecentquality(atleast,thequalityweexpect).However,thatdoesnotmeanthatoursoftwareactuallydoeswhatwasrequested.ItoftenhappensthatthecustomerrequestsfeatureA,theprojectmanagercommunicatesB,andtheprogrammerbuildsC.Thereisareallyfunnycomicaboutitwithaswing(doaGoogleimagesearchforhowprojectsreallywork).Luckily,wehaveacceptancetests.
Anacceptancetesttestswhetherspecificfunctionality,asdescribedinthespecification,worksasexpected.Forexample,theexternalservicewebuiltmadeitpossibleforthethirdpartytomakeacallusingaspecificloginmethod,createauser,updatetheuser,andfinally,deactivatethatuser.Thespecificsoftheupdatesweredescribedinthespecificationsdocument.Somefieldswerespecifiedbythethirdpartyandsomefieldswerecalculatedbytheservice.Keepinmindthattheactualcalculationshadbeenunittestedandthatweknewallthepartsworkedtogetheraswehaddonesomeintegrationtesting.Thistestwasallabouttestingwhetherthethirdparty,usingtheirJavatechnology(ourservicewaswritteninC#,butcommunicationwasXML),couldindeedcreateandupdateauser.Iprobablytestedthatmanuallyonceortwice.Theproblemwithtestingthismanuallywasthatitwasawebservice;theinputandoutputwasXMLwhichisnotthateasytoreadandwrite.Theserviceonlyreturnedwhetherornottheuserwassuccessfullycreated(andifnot,why)soinordertotestwhethereverythinghadgonewell,Ineededtolookuptheuserrecordinthedatabase,alongwithallotherrecordsthatshouldhavebeencreated.Iknewhowtodothatatthetime,butifIneededtodoitagainnow,I'dbeprettyfrustrated.AndifIdonotknowhowtoproperlytestit,thenhowwillmycoworkerswhoneedtomakechangestotheserviceknow?Needlesstosay,Icreatedsomethinglike30automatedteststhatcheckwhetherspecificusecasesworkasintended.
Anotheroneofourapplications,awebsite,worksprettymuchthesame.AusercancreatearecordonpageA,lookituponpageB,andupdateit.Obviously,XMLisnotgoingtocutithere;thisisnotawebservice.Inthiscase,weusedGUItests(thatis,GraphicalUserInterfacetests).Ourbuildserverisjustgoing
toruntheapplicationandclickonthebuttonsthatwetoldittoclick.Ifthebuttonisnotavailable,we'vegotourselvesanerror.Ifthebuttonisavailable,butdoesnottakeustotherequestedpage,we'vegotanerror.Ifthepageiscorrectlyloaded,buttherecordisnotvisible(forwhateverreason),we'vegotanerror.Theimportantthinghereisthatthetestsdomoreorlessexactlywhatouruserswilldoaswell.
Thereissomeconfusiononthedifferencebetweenintegrationtestsandacceptancetests.Bothtesttheentiresystem,butthedifferenceisthatintegrationtestsarewrittenfromatechnicalperspectivewhileacceptancetestsarewrittenfromtheperspectiveoftheproductownerorbusinessusers.
SmoketestsOfcourse,evenwhenallofyourtestssucceed,aproductcanstillbreakinproduction.Thedatabasemaybedownormaybeyouhaveawebsiteandthewebserverisdown.Itisalwaysimportanttoalsotestwhetheryoursoftwareisactuallyworkinginaproductionenvironment,sobesuretoalwaysdoanautomatedsmoketestafterdeploymentthatgivesyoufastanddetailedfeedbackwhensomethinggoeswrong.Asmoketestshouldtestwhetherthemostimportantpartsofyoursystemwork.Amanualsmoketestisfine(andI'dalwaysmanuallycheckwhetheryoursoftware,atleast,runsafterarelease),butrememberit'sanotherhumanactionthatmaybeforgottenordonepoorly.
Somepeoplerunsmoketestsbeforedoingintegrationandacceptancetests.Integrationandacceptanceteststestanentiresystemand,assuch,maytakeabitoftime.Asmoketest,however,testsonlybasicfunctionality,suchasdoesthepageload?Whenasmoketestfails,youcanskiptherestofyourtests,savingyousometimeandgivingyoufasterfeedback.
Therearemanytypesoftestsavailableoutthere.Unittests,smoketests,integrationtests,systemtests,acceptancetests,databasetests,functionaltests,regressiontests,securitytests,loadtests,UItests...itneverends!I'mprettysureyoucouldspendanentireyeardoingnothingbutwritingtests.Trysellingthattoyourcustomer;youcan'tdeliveranyworkingsoftware,butthesoftwareyoucan'tdeliverisreallyverywelltested.Personally,I'mmorepragmatic.Atestshouldsupportyourdevelopmentprocess,butatestshouldneverbeagoalonitsown.Whenyouthinkyouneedatest,writeatest.Whenyoudon't,don't.Unfortunately,Ihaveseenteststhatdidabsolutelynothing(exceptgiveyouafalsesenseofsecurity),butI'mguessingsomeonejustwantedtoseetests,anytests,reallybad.
OtherqualitygatesNexttotests,youwantothermeasurementsofcodequality.Forexample,codethathasmanynestedifstatementsishardtotestandunderstand.Writinganifstatementwithoutcurlybraces(forsinglestatements)willincreasethechancesofbugsinthefuture.Notclosingdatabaseconnectionsorfilehandlesmaylockupyoursystemandcauseotherprocessestofail.Failingtounsubscribefrom(static)eventsmaycausememoryleaks.Sucherrorsmayeasilypassunittests,butwilleventuallyfailinproduction.Thesesortoferrorscanbeverydifficulttofindaswell.Forexample,amemoryleakmaycauseyourapplicationtorunslowlyorevencrashafteradayortwo.Goodluckfindingbugsthatonlyhappentosomeusers,sometimes,becausetheyhaven'tclosedtheapplicationintwodays.Luckily,therearetoolsthatfindexactlythesekindsofissues.SonarQubeisonesuchtool.Itwillshowyouwhereyoucanimproveyourcode,howimportantitisthatyoufixthiscode,thetimeitwillprobablytaketofixit,andatrendinggraphofyourtechnicaldebt.
Itisimportanttonoteherethattheseissues,unlikeunittests,mayormaynotbeactualbugs.Forexample,thefollowingcodeiscompletelyvalid,butmayintroducebugsthatarenoteasytospot:
if(valid)
DoSomething();
Nowthespecificationschangeandyou,oracoworker,havetochangethiscodesosomethingelseisalsoexecutedwhenvalid.Youchangethecodeasfollows:
if(valid)
DoSomething();
DoSomethingElseIfValid();//Thisisabugasit'salwaysexecuted.
ToolssuchasSonarQube,willrecognizethispatternandtheywillwarnyouthatthecodeisnotbestpracticeincludinganexplanationonwhat'swrongwithitandhowtochangeit.Inthiscase,theoriginalcodeshouldbechanged,soit'sclearwhathappenswhenvalid:
if(valid)
{
DoSomething();
}
WewillhavealookatSonarQubelaterinthisbookandseebothC#andJavaScriptissuesthatmayormaynotbebugs.
AutomationDependingonwhatyou'reusedto,I'vegotsomebadnewsforyou.WhendoingCI,thecommandlineisyourbestfriend.Personally,Iseetheneedforacommandline,butIdon'tlikeitonebit.Itrequireswaytoomuchtypingandmemorizationformytaste.Anyway,LinuxusersrejoiceandspoiledWindowsusersgetreadyforatripbacktothe80swhenuserinterfaceshadyettobeinvented.However,we'regoingtoautomatealot,andthatwillbethecomputer'sjob.Computersdon'tuseuserinterfaces.So,whileyouhitF5inVisualStudioandcompileyourcode,yourbuildserverneedstoknowitshouldrunMSBuildwithsomeparameters,suchasthelocationofyoursolutionorthemsbuildfile.
Luckily,mosttoolshavesomeformofcommand-lineinterface.Whetheryouareworkingwith.NET,JavaScript,Java,SQLServer,Oracle,oranylanguageortool,youcanalwaysrunitusingacommandline.Throughoutthisbook,wewillusevarioustoolsandIdonotthinkwewilluseanyofthemwithoutusingthecommandlineaswell.Infact,thecommandlineseemstobeback(although,wasiteverreallygone?).Varioustools,suchasNodeJS,npm,andMongoDB,areusedthroughthecommandline.Furthermore,wewillseetools,suchasMSBuild,MSTest,andNuGet,thatallworkfromthecommandline(orfromasingleclickinyourIDE).
TeamworkImaginedoingallthislocallyonyourowncomputer.Forsimplicity,let'ssayyou'vegotsomecodethathastocompileandsomeunitteststhathavetorun.Easyenough,everybodyshouldbeabletodothat.Exceptyourmanager,whodoesn'thavethedevelopersoftwareinstalledatall.Ortheintern,whoforgottokickofftheunittests.Orthedeveloper,whoworksonadifferentOSmakingsometests,thataren'timportanttohim,fail(forexample,wehaveanapplicationdevelopedonandforWindows,butacomplimentaryappforiOSdevelopedonaMac).Suddenly,gettingaworkingandtestedexecutablebecomesahassleforeveryonewhoisn'tworkingonthisprojectonadailybasis.Besides,thepeoplewhocangetaworkingexecutablemayforgettoruntests,creatingariskthattheexecutableiscompiling,butnotactuallyworking.Asyoucansee,alotcangowrongandthereareonlytwosteps.I'veintentionallyleftoutalltheothertestsandqualitygateswemighthave.Andthat'sthebiggestbenefittoCI.Thesoftwareiscompiledandfullytestedautomatically,reducingthechanceofhumanerrorsandmakingitconsiderablyeasiertogetaworkingexecutablethatismoreorlessguaranteedtowork.Bytestingonaserverthatcloselyorcompletelyresemblestheproductionenvironment,youcanfurthereliminatehardtofindbugs.
Asyoumighthaveguessed,CIisnotsomethingyoujustdo.It'sateameffort.Ifyou'rewritingunitteststomakesureeverythingworksasbestasitcan,butyourteammemberscommitlargechunksofcode,neverwritetestsandignorethebuildstatus,yourbuildbecomesuntrustworthyandquiteuseless.Inanycase,itwillnotleadtothe(increasein)softwarequalityyouwerehopingfor.
Havingsaidalloftheabove,it'scrucialthatyou,andyourteam,takeyourautomatedbuildenvironmentveryseriously.Keepbuildtimesshort,sothatyougetnear-instantfeedbackwhenabuildfails.Whensomeonechecksincodethatmakesthebuildfail,itshouldbecomeatopprioritytofixthebuild.Maybeit'sthatmissingsemi-colon,maybeatestfails,ormaybemoretestshavetobeadded.Thebottomlineis,whenthebuildfails,itbecomesimpossibletogetanexecutablewiththelatestfeaturesthat'sguaranteedtopassyourtestsandotherqualitycriteria.
Whenyourbuildpasses,itguaranteesthatthesoftwarepassesyourtestsandotherqualitygatesforgoodsoftware,whichshouldindicatethatit'sunlikelythatthesoftwarewillbreakor,worse,produceerroneousresultsinthatpartofthesystem.However,ifyourtestsareoflowquality,thesoftwaremaystillbreakeventhoughyourtestspass.Partsofthesystemthatarenottestedmaystillbreak.Eventestedpartscanstillproducebugs.Assuch,ContinuousIntegrationisnotsomemagicalpracticethatwillguaranteethatyourcodeisawesomeandfreeofbugs.However,notpracticingitwillalmostcertainlyguaranteesomethingsomewheresometimewillgowrong.
ContinuousDeliveryThenextsteptowardssuccessfulsoftwaredeploymentisContinuousDelivery(whichdoesn'thaveanabbreviation,asitwouldbethesameasthatofContinuousDeploymentandthedifferencebetweenthetwoisalreadyconfusingenough).WithContinuousDelivery,theartifactsthatareproducedbyyourCIserveraredeployedtotheproductionserverwithasinglebuttonclick.ContinuousIntegrationisaprerequisiteforsuccessfulContinuousDelivery.
Let'sfirsttakeaquicklookatwhatyouareprobablydoingnoworhavedoneinthepast.Youimplementthefeature,compile,runtests,andwhenitallworks,decidetoreleasethenewversionofthesoftwaretoyourcustomer.Youneedtocopy/pastesomefilestotheirenvironmentmanually.Youneedtocheckwhetherthespecificconfigurationiscorrect(soyourcustomerwillnotbetargetingyourlocaldatabase).Bytheway,didyoumakeabackupofthecurrentversionoftheirsoftwareincaseyourfixbreakssomethingelse?Whenadatabaseupdateisinvolved,youprobablyneedtostopsomeservicesthatreadand/orwritetothedatabase.Oops,youforgottoturnonthemaintenancepageforyourcustomers,website.Now,they'llseethiswonderfulThissitecan'tbereachedERR_CONNECTION_REFUSEDpage.Ahwell,updatethedatabase,copythosefiles,andgettingitbackupassoonaspossible.Andnow,youhavetotestifeverythingworksasexpected.Also,don'tforgettorestartthatserviceyouhadtostopinordertoupdatethedatabase.Thosearequitealotofstepsforasingledeploymentandeachofthosestepscangowrong.Andevenifyoudoitright,willyourcoworkersknowwhattodoaswell?Willyoustillknownextmonthwhenyouneedtodoanotherrelease?So,you'vejustfinishedthisreleaseandnowthecustomercallsagain,looksgood,butcanyoumakethatbuttongreen?Yes,youcanmakethatbuttongreen.That'slikethreesecondsofworkandthenanotherthirtyminutessweatingandswearingwhileyoureleaseittoproduction.Andwhowilldoallofthiswhenyou'reonvacation?
Butwehavedocumentedourentiredeliveryprocess,Ihearyousay.Bethatasitmay,peoplecouldstillskipasteporexecuteitincorrectly.Documentationneedstobewrittenandupdated,whichis,again,atime-consuminganderrorpronetask.Sure,ithelps,butisbynomeansfail-safe.
Thebenefitsofhavinganautomateddeploymentsoonbecomevisible.Lessobvious,butmostuseful,isthatautomateddeploymentmakesitsomucheasiertodeploythatyoucan(andwill)deploymorefrequentlyaswell.Whenareleasetakesanhourofyourtimeandhasconsiderablerisksoffailure,nottomentionfrustration,everytimeyoudoit,youtendtopostponeareleaseaslongaspossible.Thatalsomeansthatthereleasesyoumakeprobablyhavemanychanges,whichincreasestheriskthatanythingbreaksthesoftwareandwhichmakesithardertofindthebug.Makingthereleaseprocesseasier,byautomatingit,isimportant,sinceyouwillnowdeploysmallerchangesmoreoften.Thatmeansthatitiseasiertorollbackanydeploymentsincaseoffailure,butitwillalsoreducetheriskthatanyfailuresoccurinthefirstplace.Afterall,thechangesrelativetotheoldversionofthesoftwareremainsmall.
Youshouldbeabletodeployyoursoftwaretoaproduction(like)environmentatanytime.Toachievethis,yoursoftwaremustalwaysbeinadeliverablestate,meaningyourbuildsucceeds,thecodecompiles,andyourtestssucceed.Theadvantagesshouldbeobvious;yourcustomercallsandasksforanewfeature,ormaybehehasfoundabuginthesoftwareandwantsyoutofixit.Youcannowsimplyimplementthefeatureorfixthebugandhaveitonproductionjustminuteslater.YourCIserverbuildsthesoftwareandyouknowitisprobablyalrightbecauseitcompilesandyourtestssucceeded.
Unfortunately,ContinuousDeliveryisnotalways(completely)possible.Forexample,whenyourcustomerhasadatabaseadministrator(thedreadedDBA)thatpreventsyoudirectaccesstothedatabase,youcouldstillautomatethedeliveryofthesoftware,butnotthedatabase.Inonecase,Ievenhadacustomerwhereonlythesystemadministratorhadinternetaccess.Allothercomputersinthecompany(oratleast,atthatspecificsite)werenotconnectedtotheinternet.Asaresult,eachsoftwareupdatewasdonemanuallyandonsite(aone-hourdrivesingletrip,nomatterhowsmalltheupdate).Eventhen,themoreyoucanautomate,thebetter,sogetthatCIserverupandrunning,getthattestedartifact,drivetothecustomer,anddeploythatartifact(usingalocalscriptifpossible).Ifyoudoitright,whichwedidn'tatthetime,youcouldemailittothesystemadministratorandtellhimjustrunthatscript,savesyouatwohourdrive!
Noteveryoneiskeenonhavingtheirdeploymentsautomated,especiallynotcustomers.When,atonetime,mymanagermentionedthatwewerelookingatautomatingdeploymentsasitwouldbefaster,easier,andlesslikelytogowrong,
thecustomeractuallyrespondedthattheydidnotwantthataswe,andthey,couldnotcheckon,orcontrol,automateddeployments.That'saprettyabsurdstatementasifanythingiscontrollable,it'sascript,andifanythingisn't,it'sapersonwhocanignoreprotocolormakehonestmistakes.Still,peoplearen'tknowntoberationalaboutstufftheydon'tunderstand.So,ifyouarelookingtoimplementanyoftheaboveexpectsomeinitialresistance.Ontheflipside,customersareneveraroundwhenyoudeployanyway,sotheywillnotevennoticewhenyouautomateit(youdidnotgetthatfromme).
ContinuousDeploymentThefinalstageofautomatingyoursoftwaredevelopmentprocessisContinuousDeployment.WhenpracticingContinuousDeployment,everycheckintoyoursourcecontrolisdeployedtoaproduction(like)environmentonasuccessfulbuild.Therationalebehindthisisthatyouaregoingtodeploythesoftwaretoproductionsoonerorlateranyway.Thesooneryoudothis,thebetterthechanceyou'llbeabletofixbugsfaster.It'seasiertorememberwhatyoudidyesterdaythatmighthavecausedthebugthanitistorememberwhatyoudidtwomonthsagothatmighthavecausedthebug.Imaginecheckingsomecodeintosourcecontrolandgeterrormessagesfromyourproductionenvironmentfiveminuteslater.You'llbeabletofindandfixthebugimmediatelyand,fiveminuteslater,theproductionsoftwareisupandrunningwithoutbugsagain.Unfortunately,mostmanagersandsoftwareownersIknowgetprettynervousatthethoughtofautomateddeployment,letaloneautomateddeploymentoneverycheckin.
Again,aswithContinuousDelivery,ContinuousDeploymentisnotalwayspossible.AlltheissueswithContinuousDeliverystillapply,exceptnowwhenaDBAdoesn'tgiveyouaccesstothedatabase,ContinuousDeploymentisprettymuchoutofthequestion.Afterall,youcan'tautomaticallydeploysoftwaremultipletimesadaywhileyourdatabaseisonlyupdatedwhensomeDBAhastime.Currently,I'mworkingonawebsiteforacustomerwhointurnhasacustomerwhoneedsthreedays'noticebeforeanychangestothewebsitecanbemade.It'sacontractualobligationandwhetheritmakessenseornot,itiswhatitis.Sointhatparticularcase,ContinuousDeploymentisobviouslyano-go.Still,weusethistechniqueonourowntestenvironmentandautomateasmuchaspossiblewhilestillgivingthreedays,notice.
ThedifferencebetweenContinuousIntegration,ContinuousDelivery,andContinuousDeploymentmaystillbeabitvague.Considerthefollowingimage(IapologizeformypoorMSPaintskills)whichindicateswherethethreetypesstartandstop:
Doesallofthismeanwehavenomoremanualtasks?Notatall.Forexample,theonlywaythatyou'regoingtoknowifwhatyoubuildisactuallywhatthecustomerwantedisbyhavingthecustomerseeitand,ideally,useit.Sothecustomershouldvalidateanychangesmanually.Evenifit'sjustabugfix,yourcustomerprobablywantstoseethatit'sfixedwithhisowneyesbeforeyoucanreleasetoproduction.Likewise,exploratorytestingisatypicalmanualtask.Othertasks,suchasmakingchangestoyourfirewall,(web)server,ordatabasemay(orevenmust)bedonemanually,althoughpreferablynot.
SummaryContinuousDeploymenthelpsingettingsoftwareouttoyourcustomerassoonasitiswritten.ContinuousDeliveryisagoodalternativeifyouneedmorecontroloveryourdeployments.Tominimizetheriskofdeployingbugs,yoursoftwareshouldbethoroughlytestedusingContinuousIntegration.ContinuousIntegrationisallaboutmakingsureyoursoftwareistestedanddeployable.Inthenextchapter,wearegoingtosetupanenvironmentwithsometoolsthatarenecessaryforContinuousIntegration.
SettingUpaCIEnvironmentIntheremainderofthisbook,we'regoingtoimplementContinuousIntegration,Delivery,andDeployment.However,beforewestart,wemustchoosesomesoftwaretoworkwith.AsIhavementioned,wehavemultiplechoicesforoursourcecontrolandforourCIserverand,ofcourse,wecanuseatonofprogramminglanguagesanddatabases.Additionally,weneedtocreatesomeprojecttoworkwith.Thischapterwilllayoutthetechnologiesthatareusedintheremainderofthebook,aswellasahigh-leveloverviewofhowtheyallworktogether.Forthetestproject,we'llcreateasimpleto-dolistwebapp.We'llimplementitinNode.jswithaMongoDBdatabaseandinC#CorewithaPostgreSQLdatabase.Thatway,we'llseeCIinactioninbothfrontendandbackenddevelopment,aswellasthepopularJavaScriptlanguage,thecompiledC#.NETCorelanguage,SQL,andNoSQL.
We'llstartbybuildingafrontendusingJavaScriptandthenpmpackagemanager,creatingunittestsandUItestsinSeleniumandJasmine,andautomatingtheseusingKarmaandGulp.Afterthat,we'llhookituptoaJavaScript-drivenNode.jsandMongoDBbackend.
Withthesamefrontend,wecanbuildaC#.NETCoreandPostgreSQLdatabasebackend.We'lluseNuGetandMicrosoft'sunittestingframeworkaswellasMSBuildandMSTest.
Chapter1,ContinuousIntegration,Delivery,andDeploymentFoundations,startswithsourcecontrol.Forthisbook,IhavechosenGit,asitisawidelyusedsourcecontrolsystem.AsfortheCIserver,wearegoingtouseJenkins.BothGitandJenkinsarefreeandcanbedownloadedandusedprivatelyandprofessionallyatnocost.Inareal-worldscenario,youwouldinstallGitandJenkinsononeortwoserversthatyourentireteamcanaccess.Likewise,yourdatabasemaygetaseparateserveraswell.However,sinceyouandI,aspoorprogrammers,probablydonothaveanidleserver(ortwo)layingaround,wewillmakeuseofVirtualMachines(whichislikeacomputeronyourcomputer).Lateron,wewillalsoneedanenvironmenttodeployto,sothatcouldbeasecondorthirdVirtualMachine(VM).ForourVMs,wearegoingtouse
OracleVMVirtualBox.Thissoftwareisalsofreetouse.Ofcourse,anyVMisgoingtoneedanoperatingsystem,justlikeanormalcomputer.Imayhavegivenitawayearlier,butIamaWindowsuser.Unfortunately,Idon'thavesomespareWindowslicenseslayingaroundandIamguessingyouhaven'teither.Luckily,thereisanotherpopularoperatingsystemthatisfreetouseandcanrunGit,Jenkins,JavaScript,andanythingwearegoingtousethroughoutthisbook.YouhaveprobablyguessedthattheoperatingsystemisLinux.AsfortheLinuxdistribution,IamusingUbuntuServer,asitisoneofthemostusedLinuxdistributionsoutthere(ifnotthemostused).Donotworry,IexpectnopriorknowledgeofanyofthetoolsImentioned,soIwillbetakingyouthroughtheinstallationsstepbystep.
IshouldmentionthatrunningaVMisprettyheavyworkforyourcomputer,letalonerunningtwo.Afterall,yourcomputerwillberunningmultipleoperatingsystemsandalltheprogramsinit.Ifyouhavelessthan8GBRAMmemory,IrecommendnotinstallingeverythingonyourVM(itispossible,butitwillnotrunverysmoothly).Thereisonetoolinthisbook,GitLabforGit,thatreallyneedsLinuxtorunon.Youwillalsoneedsomespaceonyourharddisk.TheminimumrecommendedsizeforanewVMis8GB,butthatisnotenoughforourenvironment.Iwoulduse30GBjusttobesafe(especiallyifyouplanoninstallingauserinterface).Wearegoingtoinstallsomeothersoftwareaswell,soreserveatleast40GBtobeonthesafeside(fortwoVMs).
MyownsystemrunsWindows10andhas16GBRAMmemory,soalloftheexamplesinthisbookareguaranteedtoworkonWindows10(probably8andeven7aswell).TheexamplesmayworkonLinuxandevenMac,butIhavenottestedthem.SomeoftheexplanationsareWindows-specific,butmostofitisprettygeneric.
YoumayhavenoticedthatwearegoingtouseC#.NETCore.C#.NETtraditionallyonlyworksonWindows(unlessyouuseMono),but,withtherecentlyreleased.NETCore,thatisallinthepast.With.NETCore,anofficiallightversionof.NET,youcancodeC#andruniteverywhere.Ofcourse,itwouldbeniceifwecouldalsodevelopthatapplicationeverywhere.TheVisualStudioflagshipeditorfromMicrosoftstillonlyworksonWindows.Luckily,MicrosoftrealizedthattooandcreatedVisualStudioCode,alightweighteditorforallyour.NETcode.
InstallingaVirtualMachineAsmentioned,wewillneedaVirtualMachine.WearegoingtouseOracleVMVirtualBoxtohostourVMs,whichyoucandownloadat:https://www.virtualbox.org(IhavedownloadedtheVirtualBox5.1.12platformpackageforWindows).Downloadtheversionthatisapplicabletoyouonthedownloadspageandinstallitonyourcomputer.I'veleftallthedefaultsastheywere,butyoucanchangethemasyouseefit(atyourownrisk).Ifallgoeswell,youshouldsoonseetheOracleVMVirtualBoxManager,asfollows:
TocreateanewVM,clicktheNewbutton.Youthengettopickaname(IhavecalledmyVMCIserver),atype(Linux),andaversion(Ubuntu(64-bit),whichisthedefault).Thenextwindowletsyouspecifytheamountofmemory.Thedefaultis1GB,butIrecommendmakingthat4GB(4096MB),unlessyouareonlygoingtorunGitLabonit,inwhichcaseyoucandowithless.Afterthat,youhavetoactuallycreatethevirtualHD.YoucanpickCreateavirtualharddisknow(whichisselectedbydefault).Thenexttwowindowsletyouspecifythetypeoffileyouwishtocreateandwhetheryouwouldliketodynamicallyallocatespaceorhaveitfixed.Irecommendleavingthedefaults.Afterthat,yougetasummaryandyoucancreateyourVM.
TheVMisnowaddedtothelistofVMsintheVirtualBoxManager.Youcanstartitbyeitherdouble-clickingorbyselectingitandclickingStart(don'tdothisjustyet).
InstallingUbuntuWhenyoustartyourVMforthefirsttime,VirtualBoxwillaskforastartupdisk.Thestartupdiskshouldinstallouroperatingsystem,butwedon'thaveoneyet.Headovertohttps://www.ubuntu.com/andfindtheserverdownload.Atthetimeofwritingthis,IdownloadedUbuntuServer16.04.1LTS(LongTermSupport).Thedefaultdownloadshouldbeanisofile.
Technicallyspeaking,thereisnosuchthingasUbuntuServer.ItisalljustUbuntu.TheonlydifferencebetweenUbuntuandUbuntuServeristheinstallationprocedureandthepreinstalledpackagesthatcomewithit.
OnceyouhavedownloadedtheUbuntuisofile,starttheVM.Whenitasksforthestartupdisk,youcanbrowseyourcomputerandselecttheUbuntuisofile.Onceitisselected,clickNextandtheinstallationwillbegin.
IfyoueverloseyourmousecursorintheVMandyoucannotdeselect,minimizeorcloseyourVM,usetheright-CtrlkeytogetyourVMoutoffocus.If,forsomereason,youdidnotfollowmytutorialandbackedoutoftheinstallation,youmayfindthatyourVMgivesyoutheerrormessageFATAL:Nobootablemediumfound!Systemhalted.Ifthisisthecase,youcancloseyourVM(usingVirtualBox,likeusingtheoffbuttononyourphysicalcomputer)andthengotothesettingsofyourVMtoselectabootablemedium,whichisyourUbuntuisofile:
WhenyouselecttheisofileandrestartyourVM,itwillnowtakeyoutotheinstalleragainandyoucanfollowthestepsasexplained.
First,youhavetochoosethelanguage(youcannavigateusingtheupanddownarrowkeysandselectusingtheEnter/returnkey;usetheleftandrightarrowkeystoselect<Goback>).IhavechosenEnglishandIsuggestyoudotoo,soyoucanfollowalongwiththetutorial(andtherestofthebook)withouttranslatingeverything.Inthemenuthatfollows,pickthetopoption,InstallUbuntuServer.Next,youhavetopickalanguageforyourinstallation.Again,IpickedEnglish.Thenextstepistopickyourlocation,whichwill,amongotherthings,determineyourtimezone.I'veactuallypickedTheNetherlandshereandIsuggestyoulookforyourownlocationaswell.Ifyouarelucky,youwillbetakentothekeyboardconfiguration.However,if,likeme,youpickedalanguagethatisnotspokeninyourlocation(andwedon'tspeakEnglishintheNetherlands)oryourlocaleisnotpreinstalled,youwillbepromptedtopickalocalefirst.IhavepickedUnitedStates(en_US.UTF-8).
So,nextupisthekeyboardlayout.Irecommendfollowingtheautomatickeyboarddetection,unlessyouknowyourkeyboardlayout(thereareloadsofthem).
Afteryouhavedeterminedyourkeyboardlayout,youwillseesomeprogressbars.Afterthat,youneedtospecifyahostname.Again,Ipickedciserver(nospacesorcapitalsthistime).Next,youarepromptedforyourrealname(SanderRossel),andafterthat,youcanmakeupausername(sander).Afterthat,pickapassword(1234orwhatever;youwillonlyusethisonyourlocalcomputeranyway).Re-enterthepassword.Ifyourpasswordisaweakpassword(like1234),thenextquestionwillbeifyouaresurethatyouwanttousethisweakpassword.Next,donotencryptyourhomedirectory.Youcanjustacceptthechosentimezone.Foryourdiskpartition,choosethedefault,Guided-useentirediskandsetupLVM.AnLVMisaLogicalVolumeandallowsyoutodynamicallycreate,resize,ordeletepartitions.Afterthat,justchooseYesandContinueuntilafewmoreprogressbarsappear.LeavetheHTTPproxyemptyandcontinue.Also,donotinstallautomaticupdates.Afterthat,theinstallationwillpromptyoutopicksomesoftwaretoinstall.Leavethestandardsystemutilitiesselected,butdonotselectanyoftheothersoftware.Wecanalwaysinstallothersoftwaremanuallylater.Afterthat,choosetoinstalltheGRUBbootloader.
Hooray,youhavenowsuccessfullyinstalledUbuntuServer.TheVMwillnowrestartandyoucanloginwiththecredentialsyouchoseduringinstallation(forme,thatistheusernamesanderandthepassword1234):
YouwoulddowelltomakefrequentbackupsofyourVM.Itisaseasyascopy/pastingsomefiles.VirtualBoxstorestheVMfilesinthefolderspecifiedunderthePreferences|General|defaultmachinefolder.Simplycopy/pastethefolderoftheVMyourwanttobackup.RestoringisaseasyasreplacingtheVMfileswithyourbackupfiles.DoingthisallowsyoutoplayaroundandmessupyourVMwithouthavingtoworryaboutgoingthroughtheentireinstallationagain.Backupaftereachofthesubsequentsteps,soyouneverlosework.
YouwillnoticethatUbuntugivesyounothingmorethanacommandline.Wehavejustinstalledaserver,andserversdonotreallyneedfancyuserinterfaces.Userinterfacesdocomeinhandy,especiallywhenyouarenotusedtodoingeverythingthroughacommand.Luckily,Ubuntudoesactuallyhaveauserinterface(multipleactually);itisjustnotinstalled.Inthisbook,IamnotgoingtousetheUbuntudesktop,butifyouwant,youcaninstallitbysimplyrunningthefollowingcommands:
sudoapt-getupdate
sudoapt-getinstallubuntu-desktop
Whentheinstallationisdone,youcanrestarteitherbyusingtherebootcommandorbyusing,poweroffcommandandthenrestartingfromVirtualBoxagain.
IfyouarenotfamiliarwithLinuxterminology,thesetermsaresuperuserdo(sudo)(soyouarerunningthecommandsasanadministrator)andAdvancedPackagingTool(apt).apt-getupdatemakessureyourpackagessourceisuptodate(itdoesnotupdatepackages!).Youshouldrunthisbeforeinstallingapackagetomakesureyouinstallthemostrecentversion.apt-getinstallsome-packageinstallsapackage.
ThenextthingwewillneedtodoistomakesurewecanaccessourUbuntuserverrunningintheVMfromourhost.Todothis,wefirstneedtocloseourVM.Afterthat,gotothesettingsofthatVMandgototheNetworktab.SelectAdapter2andselecttheEnableNetworkAdapterbox.Afterthat,pickHost-onlyAdapterintheAttachedtodropdown.SaveyourchangesandstartuptheVMagain.Itgetsalittletrickyfromhere.LogintoyourVMandexecutethecommandls/sys/class/net.Thiswilllistyouravailablenetworkdevices.Youshouldbeseeingsomethinglikeenp0s1enp0s2lo(thenumbersofenp0s#mayvary;Iactuallyhad3and8).Thenextthingweneedtodoisaddoneofthoseenp0s#'stoyournetworksettings.Firstthough,openyourVirtualBoxpreferences(undertheFilemenu)andgototheNetworksettings.There,selecttheHost-onlyNetworkstabandselectthenetworkyouusedforyourVM'ssecondadapter(thereshouldbeonlyone).NowcheckouttheDHCPServertab,specificallyLowerAddressBound.YouwillneedthisIPaddress(oranyIPaddressbetweenthelowerandupperbound).Minewas192.168.56.101,soIsuspectitwillbethesameforyou.Now,edityournetworksettingsinUbuntu.Todothis,openthe/etc/network/interfacesfileinvi;youcandothisbyexecutingthecommandsudovi/etc/network/interfaces.Now,pressItoeditandaddthefollowinglinestothefile:
autoenp0s8
ifaceenp0s8inetstatic
address192.168.56.101
netmask255.255.255.0
Onceyouhavemadethechanges,hitEsctostopeditingandrun:wqtoexitandsave(or:q!toexitwithoutsaving).RestartyourVMagain(usingpowerofforreboot),logintoUbuntu,andusetheifconfigcommand.Ifeverythingwentwell,
youshouldnowseeyourthreenetworkdeviceslisted.
WhilethedefaultNetworkAddressTranslation(NAT)networkadaptergivesyourVMaccesstotheInternetthroughthehost,thehost-onlyadapterisamethodthatactuallymakesyourVMvisibletothehost.Thesettingsforthisnetworkadapterroughlytranslateasfollows:
autoenp0s8:Automaticallybringsuptheenp0s8devicewhenUbuntuboots
ifaceenp0s8inetstatic:Givestheenp0s8networkinterfaceastatic(asopposedtodynamic,ordhcp)IPv4address(inet6forIPv6)
address:TheactualIPaddressnetmask:Thenetmask
UsingthisVM,wearetryingtomimicareal-worldserver.Intherealworld,securityisofutmostimportance.ItispossibletorunallthetoolswearegoingtoinstallonSSL(HTTPS)andevenrestrictthelogintospecificIPaddresses.However,sincethisisnottherealworld,theVMwillonlybeavailablefromthehost,andadditionalsecurityismoreworkbeforewecanactuallygettowritingcodeandusingtools,Iwillleavesuchsecuritytoyouforpractice.RememberingyourIPaddresseverytimeyouneedtoaccessyourserver(forapplicationssuchasGit,Jenkinsandothers)israthertiresome.Unfortunately,itisnoteasytomapyourIPaddresstoahuman-readablehostname,suchasciserver.Luckily,inWindows,wedohavealittleworkaround.FindtheC:\Windows\System32\drivers\etc\hostsfileandopenitinNotepad(asadministrator).Addthefollowinglineatthebottomofthefile:192.168.56.101ciserver.BesuretoreplacetheIPaddresswithyourown.Youcannowaccessyourserverbynavigatingtociserver[:port].
InstallingGitThefirstthingwehavetodoisinstallGit.AsIsaidatthestartofChapter1,ContinuousIntegration,Delivery,andDeploymentFoundations,CIstartswithasharedrepositoryandthisisit.GitistheimmenselypopularSourceControlManagement(SCM)toolfromthecreatorofLinux.Itshowsthateverythingisdonethroughthecommandline,andtheonlyofficialuserinterfaceissobadyoumightaswellusethecommandline.Luckily,therearesomethird-partytoolsavailable.Asidefromthetooling,Gitisareallygoodsourcecontrolsystemthathassomebenefitsoveritscompetitors.AsImentionedearlier,wearegoingtouseGitLab(https://gitlab.com)onUbuntu,whichgivesyouaniceGitHub-likeportal.
AgoodalternativeforhostingyourownGitserveristolookforanonlinehost.Themostpopular,byfar,isGitHub(https://github.com/).Personally,IuseGitHubforallmypersonalprojects.Itisfreebuteverythingisopensource,meaningeveryonemaybrowseanddownloadyourcode.Therearepricedplansforprivateprojectsthough.AnotherpopularalternativeisAtlassian'sBitBucket(https://bitbucket.org/),whichI(amforcedto)useprofessionally.LikeGitHub,BitBuckethasfreeandpricedplans.AnotherbenefittousingBitBucketisthatitworkswellwithotherpopularAtlassianproducts,suchasSourceTree,JIRA,andBamboo.Youmayskipthissectionaltogetheranduseoneoftheonlineprovidersifyouwish.Ifyoudo,besuretofollowtheirtutorials,asIamnotgoingtodiscussthemfurtherhere.
InstallingGitLabInstallingandsettingupGit(withSecureShell(SSH))authenticationinUbuntuisapainwhenyouarenotaLinuxveteran.Onceyouhaveinstalledeverything,youareleftwithaplain,bare,command-lineGit.Luckily,therearethird-partyprovidersthatdoalltheheavyliftingandgiveyouaneatportalwithallyourprojectsandcommitsasanaddedbonus(makingitlooklikeGitHub)!TheonewearegoingtouseisGitLab.Again,thedocsareprettyexplicitonhowtoinstallGitLab(https://about.gitlab.com/downloads/#ubuntu1604).
Thefirstthingwehavetodoisinstallsomenecessarydependencies:
sudoapt-getupdate
sudoapt-getinstallcurlopenssh-serverca-certificatespostfix
Theinstallationofpostfixwillgiveyouaninstallwindow(kindoflikewhenyouwereinstallingUbuntu)thatwillletyouchooseadefaultconfiguration.Justpickthedefault(InternetSite)andhitEnter.Afterthat,itwillaskforthesystemmailname.Thedefaultisyourservername,sojustgowithit.Afterthat,theinstallationwillcontinue.
Afterthat,weneedtoaddtheGitLabpackagesowecaninstallit:
curl-sShttps://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh|sudobash
sudoapt-getinstallgitlab-ce
Thecurlprogramisawell-knownprogram(alsoavailableinWindows)totransferdatafromcommandlinesorscripts.Accordingtothecurlwebsite(https://curl.haxx.se/),itisusedinyourcar,television,andpracticallyeverythingyouown.The-sswitchisthesilentmode,meaningitwillnotoutputerrorsorprogress.TheS(from-S)makessureanyerrormessagesarestillprinted(despitethesilentmode).Theresultingscript(whichyoucanalsoviewonyourbrowserbysimplybrowsingtotheURL)isgivenasaparametertoBash.Bashisashellprogram,oracommand-lineprogram,thatexecutescommands.Kindoflikewhatweweredoingtheentiretime,butwithdifferentcommands.Afterthescriptruns,wecaninstallGitLab.
Ihavementioneditbefore,butalotofprogramsrunonport8080.GitLabuses
port8080,whichisaproblemforus,sinceJenkinsalsorunsonport8080.YoucanpickeithertochangetheJenkinsport(andreboot)orchangetheGitLabport.YoucanchangetheJenkinsportbychangingthe/etc/default/jenkinsfile(simplyfindtheportandchangeit)oryoucanchangetheGitLabportbyaddingalinetothe/etc/gitlab/gitlab.rbfile.IwouldrecommendchangingtheGitLabport,sinceweareinstallingandconfiguringitrightnowanyway.Justareminder,sudovi/etc/gitlab/gitlab.rbopensthefile,Ilet'syouedit,Escgetsyououtofeditmode,:wqsavesandquits,and:q!quitswithoutsaving.Underthelineexternal_url'http://ciserver',addthelineunicorn['port']='8081'.
Now,youneedtoreconfigureGitLab(whichyoualwaysneedtodoafterchangingthegitlab.rbfile):
sudogitlab-ctlreconfigure
Thiscantakeafewminutes.GitLabmayalsoneedaminuteortwotoboot,sodonotbealarmedwhenitdoesnotshowrightafterbootingyourVM.Now,inyourhost,browsetociserver(or192.168.56.101,theIPofyourVM)andyoushouldgettheGitLabloginpage:
ConfiguringGitLabConfiguringGitLabisabreeze.Thefirstthingyouneedtodoisprovideanadministratorpassword.Thiscanbedoneonthefirstpageyougetafterstartingitforthefirsttime.Afterthat,yougetaRegisterorSignInbox,justlikewhenyouwerevisitinganactualwebsite.Justcreateanaccount(theaccountwillbestoredonyourVM,notongitlab.com).Youwillnotreceiveaconfirmationemail;apparently,thisisaknownissueatGitLab.Fromhereon,youcancreateprojects,collaboratewithyourteam,anddoeverythingGitwasdesignedfor.
Now,let'stakeherforatestdrive!InGitLab,createanewproject(itshouldbeonthepagerightafteryoucreateanaccount).Pickaprojectname,suchastest,andhitCreateproject.NoticethatyoucanmakeyourprojectsPrivate(default),Internal,orPublic.Wehavejustcreatedafreeprivateproject.
Creatingfreeprivateprojectsisawesome,butyouprobablywantyourprojecttobealittlelessprivate.Inthesettingsofyourproject(inthetop-rightcorner),youcangrantaccesstoothermembersandgroupsandevengivethemaccesstoseparateactions,suchasreadandwrite.
Speakingofgroups,logoutandcreateanewaccountforJohnorAliceorwhatever.Logoutandloginwithyourfirstaccountagain.Youcannowcreateagroup(fromthemenuatthetopleft)andinviteyourseconduser.Thistime,youwillgetanemailaboutgettingaccesstoagroup(itmightgostraighttospam).Createanewprojectinthegroupandnowyouandyourseconduserhaveaccesstothisrepository(andanyrepositorythatiscreatedinthisgroupinthefuture).
Inthefollowingsection,wearegoingtoinstallGitonWindowssothatwecantestifwecanactuallyputsomethinginourrepository.
UsingGitInstallingGitonWindowsiseasyenough.Simplyheadovertotheirwebsite(https://git-scm.com/)anddownloadtheWindowsinstaller.Runitandleaveallthedefaults.ThiswillalsoinstalltheGitGUIandtheGitBash.
OpenupaCommandPrompt.First,weneedtoidentifyourselvestoGit.Githassettingsonthreedifferentlevels,system,global,andlocal.Thesystemsettingsaresystem-wideandapplytoallGitrepositoriesonthecomputer.Theglobalsettingsapplytoalltherepositoriesforthecurrentlyloggedinuser.ThelocalsettingsapplytoasingleGitrepository.Morespecificsettingsoverridelessspecificsettings,sosystemsettingscanbeoverriddenbyglobalandlocalsettings,andglobalsettingscanbeoverriddenbylocalsettings.Toidentifyourselves,wearegoingtosettheglobalsettingsuser.nameanduser.email.Afterthat,wecancloneourGitrepository(meaningwearecopyingittoourcomputer):
gitconfig--globaluser.name"YourName"
gitconfig--globaluser.email"[email protected]"
gitclonehttp://ciserver/user/test.gitdesktop\myrepo
ThelinktoyourGitprojectcanbefoundontheprojectpageofGitLab.Thedesktop/myrepopartisoptionalandspecifiesacustomfolderforyourproject.ThedefaultfolderisthenameoftheGitrepositoryinyourcurrentfolder.AddafiletothefolderGitjustcreated.Asimpletextfilewilldo.Nowmovetoyourrepositorywithcddesktop\myrepo(orwhateverfolderyouclonedto).Thenextthingweneedtodoisaddthenewtextfiletotherepository;youcandothiswithgitadd.(the.meanswearejustaddingallthefiles).Youcannowcheckoutthestatusofyourrepositoryusinggitstatus.Youwillseethatthetextfileneedstobecommitted.Wecancommitourchangesusinggitcommit-m"Somecommitmessage".BecauseGitcreatesalocalrepository,westillneedtopushtocommittotheserver.Wecandothisusinggitpush:
cddesktop\myrepo
gitadd.
gitstatus
gitcommit-m"Addedatextfile."
gitpush
Hereisoutputoftheprecedingcommands:
Now,ifyougobacktoGitLab,youshouldseeyourcommit.Ifeverythingworked,youhavesuccessfullyinstalledGitandGitLab!
Workingfromthecommandlineisdoable,butnotverypractical.Especiallywhenyouhavegotabigprojectwithmanyfiles.WhenyouuseGitHub,youcanusetheirGitHubDesktop.WhenyoudecidedtogoforBitBucket,youareprobablyusingSourceTree.Well,GithastheGitGui.WithGitGui,itisalot
easiertoseewhichfilesneedtobeadded,arechanged,willbecommitted,andsoon.Whenyouopenit,youcancreate,clone,oropenarepository.GoforCloneExistingRepository,entertherepositoryURLandanon-existentfolder,andtherepositorywillbeclonedtoyourcomputer:
Thenextwindowwillshowyouyourchanges.Whenyoumakesomechangestoyourtextfile,addanothertextfile,andhitRescan,youwillseethefilesintheupper-leftcorner.Youcanclicktheiconstoaddorstagethefiles,afterwhichyoucancommitandpush:
TheGitGUI,whilefunctional,doesnotgiveyouallthatmuch.TheGitHubDesktopandSourceTreeclientsgiveyousomuchmanyoptionsandarealotprettiertolookat.AnotheronethatisworthcheckingoutisGitKraken(https://www.gitkraken.com/).GitKrakeniscompletelymultiplatform(itworksonWindows,Mac,andLinux)andintegrateswithbothGitHubandBitBucket.Unfortunately,GitKrakenisonlyfreeforopensource,education,non-commercial,andprivateprojects.
InstallingJenkinsInthissection,wearegoingtoinstallJenkinsandconfigureitfromthehostmachine.WedonotneedtoinstallJenkinsonaserver,soifyouarelowonRAM,considerinstallingitonyourlocalmachine(seetheInstallingJenkinsonWindowssection).Ofcourse,sinceyouwillbeusingCImostlyinateam,installingJenkinsonyourlocalmachinedoesnotmakemuchsense.However,ifyourcompanyisrunningWindowsservers,ordevelopingusingMicrosofttechnology,youwillwanttoreadtheWindowssectionanyway.UnlikeGit,Jenkinsdoesnotneedaclientapplication.
InstallingJenkinsonUbuntuUnfortunately,Jenkinsisnotasstraightforwardasdoingapt-getinstall.IfyoutrytoinstallJenkinsthroughsudoapt-getinstalljenkins,youwillgetanerrorsayingJenkinswasnotfoundintherepository.Luckily,theJenkinsWikididagoodjobofdescribinghowtoinstallitonUbuntu(orDebian-baseddistributions).See:https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+Ubuntu,formoreinformation:
wget-q-O-https://pkg.jenkins.io/debian/jenkins-ci.org.key|sudoapt-keyadd-
sudosh-c'echodebhttp://pkg.jenkins.io/debian-stablebinary/>/etc/apt/sources.list.d/jenkins.list'
sudoapt-getupdate
sudoapt-getinstalljenkins
So,whatdoesthisdoexactly?Thewget(orwebget)commanddownloadsanythingfromtheweb.Thatcouldbeafileorawebpage,orevenanentirefolder.Inthiscase,wearegoingtodownload:https://pkg.jenkins.io/debian/jenkins-ci.org.key(youcandownloaditusingabrowseraswell).The-qswitchmeansthereisnologging(qisforquiet).The-Oswitchisalittlemorecomplexandensuresthatalldownloadedcontentisconcatenatedandwrittentoasinglefile.Since-isusedasthenameofthefile,thedocumentwillbewrittentothestandardoutput,insteadofanactualfile.Next,theresultingoutputispassedasinput(withthepipe|character)tothesudoapt-keyadd-command.apt-keyaddsimplyaddsakeytotheapt-getrepository(soweshouldupdatethat!)andthe-attheendmeansthekeyiswrittenfromthestandardoutput(whereweputthekeydownload)insteadofafile.Simplysaid,wearedownloadingapt-getkeyandaddingittoapt-getrepository.
Thenextlinestartswithsudosh,meaningthatwearegoingtoexecuteashellcommandastherootuser.The-cisactuallyaparametertotheshandmeansthefollowingistheshelltoexecute.Withintheshell,wearegoingtowritethecontentsofthedebURLtothestandardoutput,whichiswhatechodoes.ThedebURLgetsaDebiansoftwarefilefromtheURL.Wearethenredirectingtheoutputtothefile/etc/apt/sources.list.d/jenkins.listusingthe>character.ThisaddstheJenkinspackagetotheapt-getsources.
sudoapt-getupdateandsudoapt-getinstalljenkinsshouldbefamiliar.
AfteryouhaveinstalledJenkins,youwillnoticeyoucannotusethepoweroffandrebootcommandsanymore.UbuntuwilltellyouthattheJenkinsuserisstillrunningandthatyoucanusesystemctlpoweroff/reboot-iinstead.Easierstill,youcansimplyusesudopoweroff/reboot.
Ifallofthatworked,youhavejustinstalledJenkins!Nowgobacktoyourhost,openupabrowser,andbrowseto:http://ciserver:8080(orhttp://192.168.56.101:8080/(theIPaddressyouaddedtotheinterfacesfileearlieronport8080,whichisthedefaultJenkinsport)).YoushouldseethefollowingpagewithinstructionsonhowtounlockJenkins:
Youknowwhattodo:runsudovi/var/lib/jenkins/secrets/initialAdminPasswordinyourVM,enterthecodeinthatfileonyourbrowser,andhitContinue.Fromhereon,theinstallationisthesameastheWindowsinstallation,soskiptheInstallingJenkinsonWindowssectionandheadovertotheConfiguretheJenkinsadminsection.
InstallingJenkinsonWindowsIfyouchosenottouseaVM,youcaneasilyjustinstallJenkinsonWindows.Headovertohttps://jenkins.ioandfindthedownloadspage.Atthetimeofwritingthis,thereisabigredbuttoninthecenterofthepagethatsaysDownloadJenkins.Evenifitisnotthere,Iamprettysureyouwillfindit.IdownloadedJenkins2.32.1forWindows;itisazipfilecontaininganmsifile.Simplyrunthemsifileandtheinstallationwillstart.Uponcompletion,Jenkinswillbestartedonyourbrowserunderlocalhost:8080:
Thedirectionsonthepageareprettyclear.OpenC:\ProgramFiles(x86)\Jenkins\secrets\initialAdminPassword,copythepassword,pasteitintotheAdministratorpasswordfield,andhitContinue.
Jenkinsinstallstoport8080bydefault,justlikealotofotherapplications.IfJenkinsdoesnotstartorlocalhost:8080doesnotshowJenkins,chancesareport8080isinusebyanotherapplication.TochangethedefaultportusedbyJenkins,gototheinstallationfolder(C:\ProgramFiles(x86)\Jenkins)andlookfor
jenkins.xml.OpenthefileasadministratorinNotepadandlookfor--httpPort=8080.Change8080tosomethingelse,suchas8081,andsaveit.Headovertoyourservices(underAdministrativeToolsunderControlPanel)andlookforJenkins.Ifitisnotrunning,startit;otherwiserestartit.Openupyourbrowserandbrowsetolocalhost:8081(orwhateverportyouchose)andyoushouldseeJenkins.YoucanactuallydownloadaJenkinswarfile(WebapplicationARchive)andrunitlocally.YouwillgetalltheJenkinsgoodnesswithouthavingtoinstallit.Yoursettingsandbuildsaresaved,soyoucanmanuallystartitandstillhaveallyourstuffpersisted.Bydoingthis,youcaninstallJenkinsfromJenkins(andyouwillnotloseanydata).YouwillneedtoinstallJavaandaddittoyourWindowspath.Youcanthenstartthewarfilefromthecommandlineusingjava-jarsomewhere/jenkins.war--httpPort=8081(httpPort,whichisoptional;thedefaultis8080).
ConfiguringJenkinsThenextpagewillaskyouifyouwanttoinstallsuggestedplugins,butyoudonotwantthat.Infact,youdonotwantanythingatthismoment,sojustclosethepopup(ontheJenkinswebsite,notthebrowser).JenkinswillinformyouthatithascreatedanadminuserwiththepasswordyouusedearlierandthentakeyoutotheJenkinshomepage(loggedinasadmin,asseenatthetop-right).OnthehomepagefindManageJenkins(intheleftsidemenu)and,fromManageJenkins,gotoManageUsers.Youshouldseeyouradminuser,soyoucaneitherselectitandthengotoConfigureontheleft-handsidemenu,oryoucanclickonthecogontheright-handsideoftheadminusertobetakentotheconfigurationscreendirectly.Ontheconfigurationscreen,youcanenter(andre-enter)apasswordyoucanactuallyremember(like1234)andsaveyoursettings.Tryloggingout(inthetoprightofthescreen)andloggingbackinwithoutyournewpassword.
ThenextthingwearegoingtodoistryandgetourGitprojectpulledintoJenkins.Jenkinsalonedoesnotdoalot.Luckily,thereisaJenkinspluginforpracticallyeverything.GotoManageJenkinsandfromtheretoManagePlugins.Manypluginsrequireotherpluginstobeinstalledtoo,butJenkinswillhandlethatforyou.Youcanbrowsethroughallthedifferentpluginsatyourleisure,butfornowweareinterestedinjustoneplugin,whichistheGitplugin.YoucanfilterforGitintheupper-rightcorner(NOTtheSearchbar,quiteconfusing).YouwillfindGitHub,GitLab(whichwewilllookatlater),Gitclient,Gitserver...justawholelotofGitreally.WewantthevanillaGitPlugin.SocheckitandhitInstallwithoutrestartatthebottom.ThiswillinstalltheGitPluginandthirteenotherpluginsthatareneededtoruntheGitplugin.
Now,wecancreateajob.GobacktotheJenkinshomepageandeitherclickNewitemontheleft-handsidemenuorclickonPleasecreatenewjobstogetstartedrightinthemiddleofyourscreen.ChooseFreestyleprojectandanamesuchasTestorwhatever.Thenextscreenisthejobconfigurationscreen.Whatyouseeheredependsonyourkindofjobandonthepluginsyouhaveinstalled.TheonlythingwewilldorightnowisgetoursourcefromGitandintoJenkins.UnderSourceCodeManagement,selectGit.YoucanfindtheRepositoryURL
inyourGitLabproject;mineishttp://ciserver/sander/test.git.Then,choosetoaddacredential.WewanttheUsernamewithpasswordkindandyourusernameandpasswordaretheonesyouusetologintoGitLab.Also,giveitanIDsoyoucanrecognizeitlater.Thatshoulddothetrick.Saveyourconfigurationandyouwillbetakentotheprojectpage.Ontheleft-handsidemenu,youcanviewtheworkspace(whichincludesthefilesthatarecurrentlyinyourjob).Youdonothaveaworkspaceyet,butthatwillchangeassoonasyoubuildthejob.SoclickBuildNowandyouwillnoticethatabuildisaddedtothebuildhistoryandthat,whenyourefreshyourworkspace,youwillseeyourtextfiles.
Ifabuildfails(fornowbecauseyoumisspelledyourGitrepositoryoryouusedthewrongcredentials),youwillseethatthebuildballinthebuildhistoryturnsred,indicatingfailure.Thatiswhatweexpected.However,whenabuildsucceeds,theballturnsbluewhenyouprobablyexpectedgreen.IthastodowiththeJenkinscreator,KohsukeKawaguchi,beingJapanese.InJapan,redmeansstopandbluemeansgo.Apparently,itisanissueforalotofpeoplesotheGreenBallsPluginisprettypopular(asistheChuckNorrisPlugin!).YoucanreadabouttheblueballphenomenoninthisJenkinsblog:https://jenkins.io/blog/2012/03/13/why-does-jenkins-have-blue-balls/.
Congratulations,youhavesuccessfullyinstalledandconfiguredJenkins!ThatisallwearegoingtodowithJenkinsfornow,butwewillbeusingJenkinsalotthroughoutthisbook.
InstallingPostgreSQLPostgreSQL(https://www.postgresql.org/)isapowerfulandpopularopensourceSQLdatabase.EventhoughithasnothingtodowithContinuousIntegration,wewillneedittoinstallthenexttool,SonarQube.WewillmakeuseofPostgreSQLinthelaterchaptersofthebook,sowearenotgoingthroughallthistroublejustforSonarQube(buttrustme,SonarQubealoneisalreadyworthit!).
InstallingPostgreSQLonUbuntuFirst,weneedtoinstallPostgreSQLandsomepopularextensions.UbuntuhassomeprettygooddocumentationonhowtoinstallPostgreSQL(https://help.ubuntu.com/community/PostgreSQL):
sudoapt-getupdate
sudoapt-getinstallpostgresqlpostgresql-contrib
Theinstallationcreatesadefaultpostgresuserwithoutapassword.However,wearegoingtocreateourownuserwithadminrights:
sudo-upostgrescreateuser--superusersa
sudo-upostgrespsql
\passwordsa
\password
\password
\q
Thesudo-upostgrescommandmeanswearegoingtoexecutethenextcommandasthepostgresuser.Thecommand--superusersacommandcreatesausernamedsa(forsystemadmin).ThepsqlcommandputsusinthePostgreSQLterminal.Thismeansthatpasswordsa,whichsetsthepasswordfortheusersa,isexecutedbyPostgreSQL.Youcanthensetthepasswordforthesysadmin.Youhavetoenterthepasswordtwice;thesecondtimeisaconfirmation.Thefinalcommand,q,getsusoutofthePostgreSQLterminal.
Intheory,thisshouldbeenough.Unfortunately,ormaybeitisbetterthisway,PostgreSQLhassomeprettystrictdefaultsettingswhenitcomestoconnecting.So,wewillneedtochangesomesettingsbeforewecanconnectfromourhostcomputer.Thefirstfileweneedtochangeispg_hba.conf.Thefollowingfilepathhasaversioninit.Myversionis9.5(.5),butifyouhaveanotherversion(whichyoucouldseeinthePostgreSQLterminal),changethefilepathaccordingly.Sochangethefileusingsudovi/etc/postgresql/9.5/main/pg_hba.conf.Youmighthavetoscrollabit,butyoushouldfindalinethatreadshostallall127.0.0.1/32md5.Inthisline,replace127.0.0.1with0.0.0.0/0.Thenextfilewearegoingtochangeispostgresql.conf.So,openthefileusingsudovi/etc/postgresql/9.5/main/postgresql.confandmindtheversionnumber.Thisfileisprettybig,butyoucanscrolltothe
bottomprettyquicklyusingthepagedownkey.Atthebottom,addthefollowinglinetothefilelisten_addresses='*'.
Last,butnotleast,wemustrestartPostgreSQLforthechangestotakeeffect:
sudosystemctlrestartpostgresql
InstallingPostgreSQLonWindowsInstallingPostgreSQLonWindowsisaseasyasdownloadingandinstallingfromthePostgreSQLwebsite(https://www.postgresql.org/download/windows/),runningit,andclickingNext,Next,Next.Besuretorememberthepasswordyouuseforthepostgresqluser.Also,attheendoftheinstallation,besuretodeselecttheStackBuilderinstallation.
InstallingpgAdminNow,inourWindowshostwewillneedtoinstallsomemanagementtools.ThereareplentyofmanagementsystemsforPostgreSQLthatrunonvarioussystems,butIhavechosenthepopularpgAdmin.YoucandownloadthelatestversionofpgAdmin(pgAdmin4atthetimeofwriting)fromthePostgreSQLwebsite(https://www.postgresql.org/ftp/pgadmin3/pgadmin4/v1.1/windows/).Simplydownloadtheexefileandrunit.Thisinstallationis,again,amatterofclickingnextandleavingallthedefaults.
OnceyouhaveinstalledpgAdmin,openit,right-clickonServersinthetopleftcorner,andgotoCreate|Server....EnteranameforyourserverintheGeneraltabandenteryourcredentialsintheConnectionstab.IfyoufollowedmyUbuntututorial,yourcredentialswillbesaandyourchosenpassword.IfyoudidtheWindowsinstallation,yourusernamewillbepostgresqlwiththepasswordyouchose.Also,ifyoudidtheWindowsinstallation,yourHostname/addressshouldbelocalhost.pgAdminmightfindyourlocaldatabaseautomatically.Inthatcase,justenteryourpasswordwhenopeningtheservernode:
OnceyouhitSave,pgAdminshouldconnectandyouwillknowwhetheryoudideverythingcorrectlyduringtheinstallationofPostgreSQL.
InstallingSonarQubeSonarQube(https://www.sonarqube.org/)isatoolthatscansyourcodeanddoesaqualitycheck.Asetofrulesareappliedtoyourcodeandeverytimeyoubreakarule,SonarQubewillreportitandaddittothetechnicaldebt.Arulecanbesimple,suchasamissingsemi-colonattheendofaJavaScriptline.Thatshouldbeafewsecondsfix.Anotherrulecanbemoredifficult,suchasthatthecomplexityofafunction(nestedloopandifstatementsandthelinesofcodeaddtothecomplexity)shouldnotbegreaterthanacertainvalue.SonarQubehasadefaultsetofrules,butyoucanrolloutyourown.Inthisbook,wearegoingtoseeSonarQubewithHTML,CSS,JavaScript,andC#,butSonarQubesupportsmanylanguages,suchasJava,VB.NET,SQL,Haskell,PHP,andmanymore.
ConfiguringPostgreSQLThefirstthingweneedtodoiscreateauseranddatabasethatSonarQubecanuse.Justfortherecord,SonarQubecanworkwithSQLServer,MySQL,andOracletoo,butPostgreSQLisalwaysagood(andfree)choice.
OnUbuntu,openthePostgreSQLterminalandrunthescripttocreateasonaruserandthenthescripttocreateadatabase,makingthesonarusertheowner:
sudo-upostgrespsql
\createusersonarwithpassword'sonar';
\createdatabasesonarwithownersonarencoding'UTF8';
\q
TheUTF8encodingisNOToptional,sobesuretoincludeit.Also,don'tforgetthesemicolonsattheendofyourstatements.
OnWindows,youcanalsocreateauseranddatabaseusingthecommandprompt.ItonlydiffersfromUbuntuinthefirstline:
cd"C:\ProgramFiles\PostgreSQL\9.5\bin\psql"-Upostgres
[enterpassword]
\createusersonarwithpassword'sonar';
\createdatabasesonarwithownersonarencoding'UTF8';
\q
IfyouinstalledPostgreSQLinanotherfolder,youshouldchangethatinthecommand.Youmay,ofcourse,alsocreateauseranddatabaseusingpgAdmin.Justright-clickontheLogin/GroupRolesnodeandcreate.Sameforthenewdatabase.
InstallingSonarQubeonUbuntuToinstallSonarQubeonUbuntu,wemustfirstmakesureweareonthecorrectJavaversion.Youcancheckyourversionusingthejava-versioncommand.Itshouldbeon8.Ifyouhavefollowedthistutorial,yourJavaversionwillnotbewhatSonarQubeexpects.So,letusfirstinstallJava.Theadd-apt-repositorycommandisnewtous.ItaddsaPPA,orPersonalPackageArchive,totherepository,tellingUbuntuitshouldlookforupdatesfromthatspecificpackage:
sudoadd-apt-repositoryppa:webupd8team/java
sudoapt-getupdate
sudoapt-get-yinstalloracle-java8-installer
[Ok]
[Yes]
java-version
Now,wecandownloadtheSonarQubepackageandaddittotheapt-getsources.Youcanfindtheinstallationstepsinthedocumentation(http://docs.sonarqube.org/display/SONAR/Installing+the+Server).Afterthat,wecanupdateapt-getandinstallSonarQube.Whenprompted,choosetoinstallwithoutauthentication:
sudosh-c'echodebhttp://downloads.sourceforge.net/project/sonar-pkg/debbinary/>/etc/apt/sources.list.d/sonarqube.list'
sudoapt-getupdate
sudoapt-getinstallsonar
Now,weneedtotellSonarQubehowtoconnecttoourdatabase.Openthesonar.propertiesfilewithsudovi/opt/sonar/conf/sonar.properties.Nearthetopofthefile,youwillseesonar.jdbc.username=andsonar.jdbc.password=.Changethemtosonar.jdbc.username=sonarandsonar.jdbc.password=sonar.Afterthat,scrollfurtherdownthefileandyouwillfindthesettingsforPostgreSQL.Theonlythingyouneedtodoisuncommenttheline#sonar.jdbc.url=jdbc:postgresql://localhost/sonarbyremoving#.
Now,weonlyneedtostartSonarQube.Youcanstartandstopitasfollows,butdonotrunthesecommandsjustyet:
sudoservicesonarstart
sudoservicesonarstop
ManuallystartingtheSonarQubeservicewhenevertheserverstartsisnotanoption,soinstead,wearegoingtomakesureSonarQubestartsatserverstartup
andstopswhentheservershutsdown:
sudoupdate-rc.dsonardefaults
sudoreboot
Thiswillrunsudoservicesonarstartandstopcommandsautomaticallyatstartandshutdownrespectively(morespecifically,itwillrunasonarscriptwithstartorstopasaparameter).
Now,onyourhostmachine,openabrowserandbrowsetociserver:9000.SonarQubemaytakeaminuteortwotostart,soifitdoesnotshowimmediately,tryagaininaminute.
InstallingSonarQubeonWindowsToinstallSonarQubeonWindows,downloadthelatestversion,Igot6.2,fromthewebsite(https://www.sonarqube.org/downloads/).Unzipthecontentsofthezipfileandputthemsomewhere.Isimplyunzippedthesonarqube-6.2foldertoC:\.Now,weneedtohookupSonarQubetothedatabase.ThisisactuallyexactlythesameasinLinux.OpenC:\sonarqube-6.2\conf\sonar.properties(orwhereveryouhaveunzippedit)andchangesonar.jdbc.username=andsonar.jdbc.password=tosonar.jdbc.username=sonarandsonar.jdbc.password=sonar.Afterthat,scrollfurtherdownthefileandyouwillfindthesettingsforPostgreSQL.Theonlythingyouneedtodoisuncommenttheline#sonar.jdbc.url=jdbc:postgresql://localhost/sonarbyremoving#.
Now,ifyougotoC:\sonarqube-6.2\bin\windows-x86-64(replaceWindows-x86-x64withyourownOS),youwillfindsome.batfiles.TostartSonarQube,simplyrunStartSonar.bat;toinstallSonarQubeasaservice,runInstallNTService.bat(withadministrativeprivileges);andtostarttheservice,runStartNTService.bat(alsowithadministrativeprivileges).Usingabrowser,browsetolocalhost:9000andyoushouldseeSonarQube.
TriggerSonarQubefromJenkinsYoucanlogintoSonarQubewiththedefaultadministratoraccount;bothusernameandpasswordareadmin.Inareal-worldscenario,youshouldabsolutelychangethose,butwearegoingtousethemasis.Ifyoudodecidetochangethem,besuretochangethemintheconfigurationupaheadaswell:
LikeJenkins,SonarQubealonedoesnotdomuch.Itneedsarunnerthatdoesalloftheheavywork.Therearespecificrunnersforcertainplatforms,butthegenericrunnerwillsufficeinmostcases:
sudoapt-getupdate
sudoapt-getinstallunzip
wgethttps://sonarsource.bintray.com/Distribution/sonar-scanner-cli/sonar-scanner-2.8.zip
unzipsonar-scanner-2.8.zip
sudomvsonar-scanner-2.8//opt/
rmsonar-scanner-2.8.zip
Thesecommandsdownloadazipfilecontainingarunner;unzipit,movetheunzippedfoldertotheoptfolder,andremovetheoriginalzip.Weneedtomakesomeconfigurationchangesaswell.Inthe/opt/sonar-scanner-2.8/conf/sonar-scanner.propertiesfile,uncommentsonar.host.url,sonar.jdbc.username,sonar.jdbc.password,andsonar.jdbc.urlunderPostgreSQL.Also,addthetwolines
sonar.host.username=adminandsonar.host.password=admin.
ThosestepsareexactlythesameinWindows,exceptyoucandothemusingyourmouseinsteadofkeyboard.IhaveunzippedtheSonarScannerinC:\,justlikeSonarQube.
Now,weneedtoregisterSonarQubeandtheSonarScannerinJenkins.Todothat,wefirstneedtoinstalltheSonarQubeplugin.Onceyouhavedonethat,youcanregistertheSonarQubeserverunderManageJenkinsandthenConfigureSystem.FindSonarQubeserversandaddone.Theleastyouhavetodoisgiveitaname(suchasciserver).Rightnow,JenkinscannotconnecttoSonarQubeandyouwillnoticethattheusernameandpasswordfieldsaredisabled.LogintoSonarQubeandheadovertoAdministration(inthetoprightmenu).Now,gotoUsersintheSecuritydropdown.Now,clickonthelittleiconintheTokenscolumnoftheAdministratorrow.Createanewtoken,callitJenkins,andbesuretocopythattoken.YoucannowpastethattokeninJenkins,intheServerauthenticationtokenfield.Also,makesuretouncheckHelpmakeJenkinsbetterbysending...atthebottom,unlessyoureallywanttomakeJenkinsbetter.
Afterthat,weneedtoregistertheSonarScanner.GotoManageJenkinsandthengotoGlobalToolConfiguration.FindSonarQubeScannerandaddascanner.NameitLocal,disableInstallautomatically,andset/opt/sonar-scanner-2.8/asSONAR_RUNNER_HOME.
Ifeverythingwentwell,youhavenowsuccessfullyconfiguredJenkinstoworkwithSonarQube.Ofcourse,wewanttotestitandseeitwithourowneyes.First,wewillneedalittlecodefilethatSonarQubecananalyze.InyourGitproject,createanewfileandcallitsomething.js.PutthefollowingcodeinitandthenpushittoGit:
varsomething='something'
if(something)
console.log('something');
NowthatwehavesomecodethatSonarQubecananalyze,gobacktoourJenkinsproject.Ontheprojectpage,clickConfiguretochangetheconfigurationofyourbuild(whichwillbeactiveimmediatelyaftersaving).Addabuildstep,ExecuteSonarQubeScanner,and,inAnalysispropertiesputthefollowingconfiguration:
sonar.projectKey=test
sonar.projectName=Test
sonar.projectVersion=1.0
sonar.sources=.
sonar.exclusions=*.txt
ThisisbasicallysayingcreateaprojectwiththeIDtestanddisplaynameTestonversion1.0.Thesourcespropertyincludesallthefilesinthecurrentfolder,butweignoreallthefileswithatxtextension.RunthejobandheadovertoSonarQube(youwillnoticeaSonarQubelinkonyourprojectpage;itdoesnotwork).InSonarQube,viewyourprojectsontheupper-leftmenuandbesuretofilterall(thedefaultisyourfavorites,whichisnoneatthemoment).Clickontheprojecttogetadetailedreport.Youwillseeonevulnerabilityissue,bringingyoursecurityratingtoB.Ifyouclickonit,youwillgetdetailsonwhatiswrong,whyitiswrong,andhowyoushouldfixit;thatisprettyawesome.Inthiscase,theconsole.logstatementisasecurityrisk.
Unfortunately,wehadexpectedatleastonemoreissue.Thefirstlineofourcodeismissingasemicolonattheend.Thatmayprovedisastrousifwearegoingtominifythecode!Onthetopleftmenu,gotoQualityProfiles.Here,youcancreateandmodifythesetsofrules.OurprojectusedtheSonarwayprofile.ClickonthelittlebuttonbehindtheprofileandchooseActivateMoreRules.FindStatementshouldendwithsemicolonsandactivateit.Youcansetaseverityfortheissue.Imadeitacriticalissue.Onceitisactivated,youcanreturntoyourprojectandyouwillseeitnowhasacodesmellandoneminutetechnicaldebt:
ItispossibletomakeaJenkinsbuildfailwhenyourtechnicaldebtrisesorwhenyouwriteblockingissues(whichisevenworsethancritical).ThisiswhatCIisabout:checkincodeandgetinstantfeedback.
YoumayhavenoticedthatthetermsSonar,SonarQube,Sonar
Runner,andSonarScannerarebeingusedinterchangeablyontheinternetbyJenkinsandevenbySonarQube.ThereasonforthisisthatSonarQubeusedtobecalledSonarandSonarScannersusedtobecalledSonarRunners.Makenomistakethough,Sonar/SonarQubeisnotthesameasaSonarRunner/Scanner.
SummaryInthischapter,wehaveinstalledourCIenvironment.Wehavediscussedthevarioustoolswearegoingtousethroughoutthebook.WehavealsohadourfirstlookatGit,Jenkins,andSonarQubeandmadethemworktogether.Intheremainderofthebook,wearegoingtoinstallmanymoretoolsontheserverandlocally.Butbeforewegettothat,wewilltakeacloserlookatGitinthenextchapter,asthereisalotmoretoGitthanwehaveseensofar.
VersionControlwithGitBeforewecontinuewithanexampleinthenextchapter,wewanttotakesometimetotrulygettoknowGit.Gitisaninvaluabletoolinyourtoolbelt.Ithelpsyouandyourteamtosharecode,checkouteachotherscode,reverttopreviouscode,andkeepdifferentversionsofthesamecodesidebyside.Inthischapter,wearegoingtoexploretheseissuesandlearnhowtomakethemostofitusingJenkins.
Forsomereallyin-depthGitreading,IrecommendthebookProGit,whichyoucanreadforfreeattheGitwebsite,https://git-scm.com/book.Withalmost600pages,thisistheabsoluteGitbible.Forsomepracticaluse,justkeeponreading.
ThebasicsBeforeweheadovertothemoreadvancedstuff,let'sgooverthebasics.Youhavealreadydonemostofthisinthepreviouschapters,butgettingagoodgraspofwhatisgoingonisimportantifyouwanttouseGiteffectivelyandtothefullest.IrecommendcreatinganewGitrepositorysoyoucanexperimentandthrowoutyourrepositorywhenwearedone.
CentralizedSourceControlManagementIntraditionalSourceControlManagement(SCM)systems,suchasCVSandSubversion,whichhavelongbeentheindustrystandards,yourcodewassavedtoaserver.Thisserverkepttheentirehistoryofyourproject.Developersworkingontheprojectcouldcheckoutasnapshotoftheproject,maketheirchanges,andcommitbacktotheserver.Itenabledteamstoworktogetherandgetanideaofwhatotherpeopleontheteamwereupto.
However,sinceallcodewasstoredinasinglerepository,youhadaproblemifthisrepositorybecame(temporarily)unreachable(forexample,duetonetworkproblems)or,worse,whenyourrepositoryorserverbecamecorrupted.Storingtheentirehistoryofyourprojectsinasingleplacecomeswiththeriskoflosingeverything.Thismodelofversioncontroliscalledacentralizedversioncontrolsystem.
DistributedSourceControlManagementGitisalittledifferent.WithGit,youalwayscopytheentirerepository,includingtheentirehistory,toyourlocalmachine.Thatmeansyoucanmakelocalcommitswithoutactuallypushinganythingtoyourserver.Italsomeanswhenyourserverbecomesunreachable,youcanstillmakelocalcommitsandpushthemwhenyourserverbecomesavailableagain.And,ofcourse,itmeansthatwhenyourserverexplodesintotinybits,everyprogrammerontheteamstillhastheentireprojecthistorysittingontheircomputersreadytoberestoredtoanewserver.Thismodelisalsoknownasthedistributedversioncontrolsystem.WhileGitisthemostpopulardistributedversioncontrolsystembyfar,andactuallythemostpopularsourcecontrolsystemperiod,Mercurialisalsoareasonablypopulardistributedversioncontrolsystem.
TheworkingdirectoryAswehaveseeninthepreviouschapters,gettingaGitrepositoryonyourlocalcomputerisaseasyascloningarepository.ThefolderthatservesasyourGitrepositoryisaregularfolderlikeanyotherandisalsoknownastheworkingdirectory.Themagictrickisthehidden.gitfolderthathasallthedatathatisnecessaryforGittotrackyourfiles.Itallowsyoutoexecutecommands,suchasgitstatus,gitadd,andgitcommit.The.gitfilecontainsyourHEAD,whichisbasicallythecurrentstateofyourbranch.Wheneveryoumovecommits,branch,cherrypick,orwhatever,theHEADwillknowyourcurrentstateandwhatitoncewas.Knowingthis,resettingyourworkingdirectorybecomesaseasyasresettingyourHEAD,aswewillseelater.ItalsomeansthatyoucanmoveyourcurrentbranchtoanothercommitsimplybyeditingyourHEAD.Now,Idonotrecommendyougoaroundandeditfilesinthe.gitfolder,butthisisexactlywhatGitdoesforyouwhenyouexecutecommandthroughthecommandlineorthroughotherGitclients.
ThestagingareaOnceyoucommityourchanges,theygooutoftheworkingdirectoryandintoyourhistory.Thereisastagebetweenthetwothough,calledthestagingarea.Wheneveryouchange,add,ordeleteafileinyourworkingdirectoryandthencheckoutthestatususinggitstatus,itwilltellyouwhetheryouhavenew(untracked)files,changesthatarenotstagedforcommit,orchangestobecommitted:
Fromthatexplanation,itbecomesclearthatwhateverisnotstagedwillnotbecommitted.Thisallowsyoutochangemultiplefiles,butonlycommitafewofthem.Itisevenpossibletostageonlypartsoffiles.Itcancomeinhandywhenyouareworkingonsomebigchangesandthensomeoneasksyoutofixthewhatchamacallit.Youcanalteracoupleoflinesofcodeandstageandcommitonlythose.
Aswehaveseenbefore,puttingyourfilesintothestagingareaiseasilydoneusingthegitadd.command.Now,insteadofusingadot(forallfiles),wecanspecifyasinglefile.Forexample,gitaddrepository.js.
Wecanstageandcommitonlypartsoffilesthatwechanged.Suchapartiscalledahunk.First,createanewfile,callithunks.txt,andaddfourlinestoit.Youcanjustnumberthelines1,2,3,and4andcommititusinggitaddhunks.txt.Nowaddtwolines,onebetween1and2andonebetween3and4;justputin1.1
and3.1orwhateveryoulike.Yourfileshouldnowlookasfollows:
1
1.1
2
3
3.1
4
Nowyoucancreateapatch,orsomechangesthatyouwouldliketostage.Youcandothisusingthegitadd--patchfilenamecommand.Youcannowseethecontentsofyourfileandthehunksingreen.Youwillbeaskedifyouwanttostagethecurrenthunkandyouroptionsare[y,n,q,a,d,/,s,e,?].Simplytype?andEntertoseewhattheymean:
Inthiscase,thecommandlinehasbothourfilesinonehunk,sowewanttosplitthehunkbytypings.Afterthat,youseeonly1.1isgreenandwedowanttostagethat,sowepicky.Afterthat,wegettwomorehunks,butweonlyhaveonemoreaddedline.Wedonotwanttostage3.1,sopickn.ThelasthunkmightbeaWindowsthing,Iamnotsure.Anyway,donotstageit:
Nowwhenyouusegitstatusyouwillseethathunks.txtisbothinthestagedpartandintheunstagedpart.
OnemorethingIwouldliketosayaboutgitaddisthatthereisaninteractivecommandsoption.ThisisprobablytheclosestyougettoanactualGUIintheconsole.Youcangetitusingthegitadd-icommand.Here,youcaneasilyselectfilesforstaging,applypatches,removefilesfromstaging,andseethechangesyouhavemadetofiles.Youshouldcheckitout.
CommittingandpushingOnceyouhavestagedthechangesyouwanttocommit,youcanproceedwiththeactualcommit.Committingyourchangeswillwritethemtothehistoryofyourlocalproject.Yourchangesaremoreorlesscastinstone.Youcancommitasmuchasyoulike,butrememberthatacommitisonlylocal.Toactuallypushyourworktotheserversootherscangetittoo,youmustpushyourcommitsusingthegitpushcommand.Whenyoupushyourcommits,threethingscanhappen.First,yourchangesarepushedandeverythingisfine.Second,yourchangesarepushed,butothershavealsopushedchangestothesamefilesresultinginamergeconflictthatGitcanresolve.Third,amergeconflictthatGitcannotresolverequiresyoutomanuallychangeyourfilesandpickbetweenyourchangesorthoseofyourcoworker.Incaseofamerge,anextracommitwillbecreated(onyourname)thatcontainsthemerge.
Mergeconflictscanbearealpaininthebehind,sobesuretokeepcommitssmallandpullregularly.Wheneveryoudohaveamergeconflict,despiteallyourbestefforts,youmusteditthefilemanuallyandsimplystageitwhenyouaredone.Aconflictlooksasfollows:
<<<<<<<HEAD
Thesearemylocalchanges.
=======
Remotechanges.
>>>>>>>449d9120c205609132e0983230fa48f5629dc41c
Toclearthatup,IliterallytypedThesearemylocalchangesonthesamelinethatsomeoneelsetypedRemotechanges.Gitcannotdecidewhetherbothlinesshouldstay;ifso,inwhatorder;orifoneshouldoverwritetheother.Besidesmanuallyeditingyourconflictedfiles,youcanalsokeepyourownchangesorthechangesofthem:
gitcheckout--oursfilename
[or]
gitcheckout--theirsfilename
gitaddfilename
Stagingyourfilewillmarkitasresolved.Afteralltheconflictshavebeenresolved,youcancontinueyourpush.
ReviewingcommitsNodoubt,yousometimesneedtoreviewsomecommitsorthoseofothers.OnethingyoucandoislookthemupinGitLaborwhateverGitserveryouuse.YoucanalsoviewtheminGitGUIoranyGitclient.However,itisalsopossibletolookthemupusingthecommandline.Thegitlogcommandlistsallcommitsinyourcurrentbranch.Therearesomecaveatsthough.Sincethelistofcommitscanbeverylong,theywillneverfitinyourconsolewindow.WhatGitdoestomakethismanageableispagetheresults.YoucanusetheEnterkeytoshownewlines.Whenyouwanttoexitthelog,youhavetotypeq(Linuxstyle).UsuallyinWindows,youexitsuchoperationsusingCtrl+C,butthiswillnotexitthelogandwillprobablymessupyourcommandwindowandleaveyouconfusedandannoyed,andultimatelymakeyouhittheXbuttonandrestartyourconsole:
gitlog
commitf90cfa90227bf1cb21d8023b03273c991fc2f471
Author:SanderRossel<[email protected]>
Date:SunJan2921:40:162017+0100
Addedxlsxsupport.
commit475bb165299b3d85c142b03462be9f82b8fbefb1
Author:SanderRossel<[email protected]>
Date:SunJan2921:38:162017+0100
Addedreportingmodule.
commit51eadeab96e1950f5e61273fcb3bbfb24a75e901
Author:SanderRossel<[email protected]>
Date:SunJan2921:37:072017+0100
Addedreportingfiles.
[...]
:q
[Alternatively]
(END)q
Asyoucansee,thelatestcommitsareatthetopasthosearetheonesyoumostlikelywanttosee.Thereareacoupleofusefulswitchestologthatcanbeveryuseful.The--pretty=onelineor--onelineswitchesareprobablytheonesyouwouldusethemost.Itsimplyprintseverycommitonasingleline,givingyouamuchcleareroverviewofthevariouscommits.Theonlydifferencebetweenthetwois
that--pretty=onelineprintsthefullcommithashwhereas--onlineprintstheshortversion.Other--prettyformatsareshort,full,andfuller,butyouwillneedtousethefull--pretty=[format]syntaxforthose.The-pswitchlistsallthedifferencesinthecommit.Itisalsopossibletolimitthenumberofresultsusing-[somenumber],forexample-2:
gitlog--oneline-p-2
Otherusefulswitchesthatcanhelpyousorttheresultsare--skip,--since,--after,--until,--before,and--author.Therearedozensofswitchesforlog,soifyoureallyneedthiskindoffunctionality,IsuggestyoulookitupintheGitdocumentation.
PullingandstashingThatleavesuswiththefinalpartoftheGitbasics,pullingcodefromtheserver.Wheneveryourcoworkerspushcodetotheserver,youwanttogetitfromthereontoyourowncomputer.Youcanpullcodeusingthegitpullcommand.Justlikewhenyoupush,apullcanresultinamergeconflict.Wheneverthishappens,therearetwothingsyoucando.Youcaneithercommityourcurrentwork,pullthecodefromtheserver,andthenresolvetheconflicts.Oryoucanstashyourchanges,meaningyouputthemasideforthetimebeingandresetallthefilesastheywerewhenyoulastpulled,andthendothepullandapplyyourstashoverthenewlypulledcode.Aconflictbetweenyourcurrentcodeandthestashwillstilloccur,butyoucannowresolveitmanuallybeforecommittingyourcode.
Stashesareaneasywaytoputsomecodeaside,eitherbecauseyouwanttopullfromtheserverorbecauseyouwanttotrysomealternativesolutiontoaproblemwithoutlosingyourcurrentsolution.Itisgoodtonoticeherethatnewfilesinyourrepositorywillnotbestashed;theywillsimplycontinuetoexistinyourcurrentworkingdirectory.Youcancreateastashusingthegitstashcommand.Youcanoptionallygiveyourstashaname.Youcanthenapplyanystashyoulikeorapplyanddeleteyourlateststashoranyotherstashbyindex.Trycommittingafile,thenchangeandsaveitscontents,andseewhathappenstoitaftereachofthefollowingcommands:
gitstash
gitstashpop
gitstashsave"Myawesomestash"
gitlist
gitstashapply0
gitstashdrop0
Ifyouapplyastashtoyourworkingdirectoryandyougetamergeconflict,theirsisactuallyyourstash.Ioncelostquitealotofworkthatway!Imaginethis:youwanttopull,butgetaconflict.Youstash,pull,andthenapplythestash.Yourstashfeelslikeyourcode(orours).However,whenyoupulltheircode,itactuallybecomesyourcode.Nowwhenyouapplythestashtoyourcode,thestashis
theircode.Itmakessensewhenyouthinkaboutit,butusuallyastashdoesnotfeelliketheirs.Soremember,whenyouwanttoresolveandkeepyourstash,resolveusingtheirs.
BranchingAnothermajorfeatureofGitisbranching.Withbranches,youcancreateacopyofyourcurrentrepositorythatisisolatedfromyourmainbranch.Sofar,we'vejustcommittedeverythingtothedefaultmasterbranch,butyoucouldmakeanewbranchtodevelopcertainfeatures.ThinkofthemanydifferentversionsofLinux.Theyarebasicallyalldifferentbranchesofthesamemasterbranch.Somebranchesevengettheirownbranch.Ubuntu,forexample,isabranchofDebian.
WhenyoufirstcreateaGitrepository,itwillnothaveanybranchesbydefault.Youneedtoaddafileandcommitittoinitializethemasterbranch.Ifyouhavenothingtoaddtomaster,becauseyouwanttousefeaturebranches,justaddareadme,license,or.gitignorefile,whichcanallbeaddeddirectlyfromtheGitLabprojectpage.Onlyafteryouhavecreatedyourmasterbranchcanyoucreatenewbranches.Youcanrenameyourmasterbranchafteryouhavecreateditifyouwant,butIreallyseenoreasonforthat.
WhileLinuxdistributionsseparatefromeachotherandgrowindifferentdirections,thatprobablydoesnotmakemuchsenseforyourownprojects.Ultimately,youprobablywanttomergeyourworkfromonebranchwiththatofanother.Imagineworkingonsomeadministrativesoftwareandyourclientrequestedareportingfeatureandameanstosendemailsdirectlyfromthesoftware.Youaretaskedwithwritingthereportingmodulewhiletwoofyourcoworkersaretaskedwithwritingtheemailmodule.Therestoftheteamcontinuestoworkonotherstuff.
Youcannowcreatetwonewbranches,oneforthereportingandonefortheemailingmodule.Thisway,youandyourothercoworkerscansafelycommitanycodewithouthavingtoworryaboutbreakinganythingordeployingahalf-finishedreportingmodule.Whenyouarefinishedwiththereportingmodule,youcansimplymergeitintothemaindevelopmentbranch.
Youcancreateanewbranchusingthegitbranchbranch-namecommand.Yourbranchnamecannotcontainspacesanditisgenerallyconsideredabestpractice
tostartyourbranchnamewithaletter,uselowercaseonly,usehyphensbetweenwords,andkeepyourbranchnamessmall.Thegitbranchcommandwillshowalistofbranchesanditindicatesyourcurrentbranchwithanasterisk(*).Finally,usingthegitcheckoutcommand,youcanswitchtoanotherbranch:
gitbranchreporting
gitbranch
gitcheckoutreporting
HereisanexampleofsomebranchesinGitGUI,foundunderthemenuitemRepositoryandthenVisualizeAllBranchHistory:
Here,wehavethemasterbranch,thecodebranch,andthejarvisbranch.IncaseyouareunfamiliarwiththeAvengersmovies(orcomics),thesearetheguystryingtoassembleanIronMansuit.Apparently,Mr.IronManhimself,TonyStark,istakingcareofthemasterbranchwhileBlackWidowandTheHulkareworkingonlasersandJARVIS,IronMan'sownreallyverycoolSiri(heactuallydiditallhimselfasheisarichgenius,butfortheexample,let'ssaytheyareworkingtogether).GitGUIisnottheprettiesttooloutthere,sohereisanothervisualizationofthesamebranch,butthistimeinGitKraken:
Aftermergingthelasersandjarvisbranchesbacktomaster,wegetaprettyclearpictureofwhathashappened.Wewillgetbacktomerginginaminute:
Youcanalsocreatebranchesoffofbranches.Simplycheckoutthebranchyouwanttouseasabaseandcreateanewbranchthere:
gitbranchweapons
gitcheckoutweapons
gitbranchlasers
gitcheckoutlasers
Thingscouldnowlookasfollows:
Itispossibletoremovebranches.Ifabranchismergedintoanotherbranchandhasnouncommittedchanges,youcansimplydeletethatbranchusingthe-dswitch.Ifabranchisnotfullymergedorhasuncommittedchanges,youmustforcedeleteitwiththe-Dswitch:
gitbranch-dbranch-name
gitbranch-Dbranch-name
Ifyoumergedthebranchwithanother,stillexisting,branch,thedeletedbranchwillstillshowupinyourbranchoverview.Thatmakessenseasitispartofyourhistory,whetherthebranchexistsornot.Putdifferently,deletingthelasersandjarvisbrancheswillnotchangethepreviouspicture.
Onceyoucreateabranch,thatbranchonlyexistslocally.Thatmaybeenoughforyourneeds,forexample,ifyouwanttotryoutsomestuffonashort-livedbranch.Creatingabranch,makingchanges,committing,andthenpushingwillnotwork.Onceyoupush,youwillgetanerror.Next,wewillcreateanew
branch,code(thatisthebigglowingorbinIronMan'schest):
gitbranchreactor
gitcheckoutreactor
gitpush
fatal:Thecurrentbranchreactorhasnoupstreambranch.
Topushthecurrentbranchandsettheremoteasupstream,use
gitpush--set-upstreamoriginreactor
What--set-upstreamoriginreactorwilldoistellyourservertocreateanewbranchwhereallyourcommitsonthisbranchwillgoto.Theshortcutfor--set-upstreamis-u,sothatiswhatyouwilluse:
gitpush-uoriginreactor
WhenyougotoGitLabandcheckoutthebranchestabontheprojectpage,youcannowseeyouhavetwobranches.Youwillalsoseewhetherthebranchesaremerged;youcancomparebranches,deletebranches,andalsocreatebranchesfromhere.Keepinmindthatifyoudeleteabranchhere,itwillstillbepresentinyourlocalworkingdirectory,buttryingtopushorpullwillresultinanerror(unlessyoupushwiththe-uswitch,whichwillrecreatethebranchonyourserver).
Theadvantageofpushingyourbranchtotheserveristhatyoucanshareyourbranchwithothers.Youwilloftenhavedifferentbranches,suchasthedevelopment,test,andmaster(orproduction)branches,thateveryoneontheteamwillsoonerorlatercommitto.Creatingabranchtodevelopasinglefeatureisoftenreferredtoasafeaturebranch.Featurebranchescanalsobepushedtotheservertohavemultipledevelopersworkingonthesamefeature.Asanaddedbonus,ifsomethinghappenstoyourcomputer,yourbranchisstillontheserver.
TheoriginissimplyalocalaliastoaURLthatGitcreates.Inthiscase,theoriginpointstohttp://ciserver/the_avengers/the-suit.git.SowhatyouareactuallysayingispushmycommitstotheGitURLonthespecifiedbranch.EveryGitrepositoryhasanoriginbydefault.Youcancheckyourremotealiasesusingthegitremote-vcommandandyoucanrenameyourURLsusinggitremoterenameoriginsomethingelse.
MergingMergingabranchintoanotherbranchiseasyenough.Firstofall,makesureyouareinthebranchwhereyouwanttomergeto.So,ifyouwanttomergejarvisintomaster,makesureeverythingyouwanttomergeiscommittedandpushedandthenswitchtomasterusinggitcheckoutmaster.Now,simplyusemergebranch-nameandresolveanyconflictsthatoccur:
gitbranchmy-branch
gitcheckoutmy-branch
[Makesomechanges]
gitadd.
gitcommit-m"Somemessage"
[optional]gitpush-uoriginmy-branch
gitcheckoutmaster
gitmergemy-branch
Aniceoptionwhenmergingistomergebutnotcommitautomatically.Thisisthedefaultbehaviorwhenyouhavemergeconflicts.Thatway,youcaninspectandedityourchangesbeforecommitting.Todothis,youcanusethe--no-commitflagwhilemerging:
gitmergemy-branch--no-commit--no-ff
The--no-ffswitchstandsfornofast-forward.Afast-forwardhappenswhenyourcurrentbranchisanancestorofthebranchyouaretryingtomergeandthecurrentbranchhashadnochangessinceyoubranchedit.Inthatcase,yourbranchissimplyacontinuationofyourcurrentbranch.Gitwillnowmoveyourcurrentcommittothelastcommitofthebranchyouaretargeting.
Thedifferencebecomesclearwhenwedoamergeofeachscenario:
gitbranchmerge-test
gitcheckoutmerge-test
[Addafile]
gitadd.
gitcommit-m"NoFFcommit"
gitcheckoutmaster
gitmergemerge-test
gitlog--oneline-2
3d28b26NoFFcommit
8bff38eHULKSMASH!!!
gitcheckoutmerge-test
[Addafile]
gitadd.
gitcommit-m"FFcommit"
gitcheckoutmaster
[Addafile]
gitadd.
gitcommit-m"Thisdoesthetrick"
gitmergemerge-test
gitlog--oneline-5
5948c05Mergebranch'merge-test'
b21e3f3Thisdoesthetrick
944321fFFcommit
3d28b26NoFFcommit
8bff38eHULKSMASH!!!
Asyousee,thereisnoextramergecommitafterthefirstmerge,butthereisanextracommitmessageafterthesecondmerge.ThatisbecausetheThisdoesthetrickcommitisinthewayofFFcommit.Thisisalsowhathappenswhenyoutrytocommitwhiletherearestillchangesontheserverthatyoudonotyethavelocally.Gitwillpushyourchangesand,atthesametime,pullchangesfromtheserverandanextramergecommitwillbecreated.Asweknow,themergecommitwillnotbeautomaticallycommittedifthereareanymergeconflicts.
CherrypickingAnotherformofmergingischerrypicking.Withacherrypick,youtargetaspecificcommitononebranchandmergejustthatonecommit,asanewseparatecommit,toanotherbranch.Thatmeansyoudonothavetomergeanentirebranchallatonce.
Italsomeansthatwhenyoudocherrypickallthecommitsonabranch,thebranchstillwillnotbemarkedasmergedandsodeletingitisonlypossiblebyforcingit:
gitcherry-pick[commitid]
YoucangetthecommitIDusinggitlogandpreferablygitlog--oneline.Forexample,Ihaveaddedareactorbranchandcreatedtwocommits.Inthefirstcommit,Iaddedareactor.txtfileandinthesecondcommit,Ichangedsometextinthatfile.Now,onmymasterbranch,Ionlywantthereactor.txtfile,butnotthechange.Wecanusegitlogbranch-nametolookforacommitonaspecificbranch:
gitcheckoutmaster
gitlogreactor--oneline-2
6cc4c08Changedthereactor.
710a366Addedthereactorfile.
gitcherry-pick710a366
[masterbaefa8b]Addedthereactorfile.
Date:SunJan2914:51:232017+0100
1filechanged,0insertions(+),0deletions(-)
createmode100644reactor.txt
Noticehowthechangesofthecherrypickedcommitareappliedtoyourcurrentworkingdirectoryandcommittedwiththecherrypickedcommit'smessageimmediately.Itispossibletonotcommityourcherrypicksautomaticallyusingthe-nswitch:
gitcherry-pick-n710a366
Thisway,youcancherrypickmultiplecommitsfromdifferentbranches,makechanges,andcommiteverythingatonce.Whateveryoudo,yourcherrypickisaseparatecommitandnotlinkedtotheoriginalcommitinanyway(exceptmaybeyourcommitmessage).Wheneveryoudecidetomergethebranchlater,youmay
encountersomeconflicts.
CherrypickingbyIDisabitofahassle.TheIDsarenotexactlyeasytoremember,letalonetype.Youcanusevariouspatternstotargetaspecificcommitorcommits(plural).Forexample,thebranch-name~indexpatternselectsthecommitonaspecificbranchonaspecificindex(beginningat0forthelatestcommit).Thepreviouscherrypickwouldthenlookasfollows:
gitcherry-pick-nreactor~1
YoucancherrypickmultiplecommitsusingIDsorpatterns:
gitcherry-pick-n710a366reactor~0
Makesureyouapplythecommitsinthecorrectorder.Therearewaystocherrypickentirebranchesandmultiplebranches,butthereareeasierwaystodothisaswewillsee.
IfyouareusingGitGUI,oranyotherGitclient,makingacherrypickisaseasyasright-clickingonthecommityouwantandselectingCherrypickorsomethingalongthoselines.
RebasingAlltheseextramergecommitsyoucangetfrommergingandcherrypickingmakeamessofyourcommithistory.Theyareallsuperfluouscommitsthatonlyexistbecauseyouneedtomergesomechanges.Insteadofmerging,youcanrebaseyourbranches.Whenyourebaseabranch,youarebasicallysayingyourbranchwasnotbranchedfromcommitA,butfromcommitB.So,imagineyouareworkingonafeaturebranch,butyoudowanttokeepupwiththemasterbranchsoyourbranchisnotleftbehindonfeatures.Besides,ifyoudonotpullregularly,thechancesofconflictsincreaseanditiseasiertosolvethemastheycome.Insteadofmergingorcherrypickingeverythingyourcoworkersdo,youcansimplyrebaseyourbranch:
gitcheckoutsome-feature
gitrebasemaster
First,rewindingheadtoreplayyourworkontopofit...
Fast-forwardedsome-featuretomaster.
Yourbranchisnowanextensionofmaster,sowhenyoumergeafeatureintothemasterbranch,youdonotneedanextramergecommit.
Youcanalsousetheshortcut,soyoudonotneedtocheckoutyourbranchexplicitly:
gitrebasemastersome-feature
Amoreadvanceduseofrebasingisrebasingcommitsontospecificothercommits.Forexample,supposeyouhaveabranchthatisbranchedoffanotherbranch.InthecaseoftheIronMansuit,let'ssaywehavetheweaponsbranchandthelasersbranchonthatbranch.Now,ifwewouldrebaselaserstomaster,theentirehistoryoflasers,whichmeansincludingthepartoftheweaponsbranch,wouldberebasedtomaster.
So,herewehaveourstartingsituation,thelasersbranchwasbranchedofftheAddedweaponscommit:
Whenwerebaselaserstomaster,gitrebasemasterlasers,Gitwillrewindyourworkuntilitfindsmaster,whichisaftertheAddedweaponscommit.Next,Gitwillreplayyourcommitsonmaster,sotheAddedweaponscommitandtheAddedlaserscommitwillbereplayedonmaster.Thiswillleaveyouinthefollowingsituation:
However,wecanspecifyacommittorebase.Inthiscase,wewantthecommitAddedlaserstorebasetomaster.Wecanspecifyacommitusingthe--ontoswitch:
gitrebase--ontomasterweaponslasers
Inthisexample,wearetellingGittorebaselaserstomaster,butonlycommitaftertheweaponsbranch.Thiswillgiveusthefollowingsituation:
Asyoucansee,theAddedweaponscommitwasnotreplayedonmaster.
Using--onto,itisalsopossibletoremovecertaincommits:
gitrebase--ontosome-branch~2some-branch~1some-branch
Here,wearetellingGittorebasetothecommitatthetopofsome-branchtothecommitthatisonindex2(~2),whichisthethirdcommitfromthetopandwestartsearchingfromthesecondcommitfromthetop(index1,~1).Thismeansthefirstcommitisrebasedtothethirdcommitand,asasideeffect,thesecondcommitisremoved.
Anothercoolfeatureisthatofinteractiverebasing.Thisallowsyoutotakeyourcommitsandeditthembeforerebasing.Youcancombinecommits,changecommitmessages,andremovecommits.Thefollowingexampleshowswhathappenswhenyourebasemasterusingtherebase-imastercommand:
ThisexamplewilldropCommit3andmergeCommit2intoCommit1,creatingasingle(new)commit.
Ofcourse,asalways,rebasingcanendupwithconflictingfiles.Ifthishappens,youshouldfixtheconflicts,addthemtoyourstagingarea(gitaddsome-file),andthenusegitrebase--continue.Incaseyoudecidenottorebaseafterall,youcanusegitrebase--abort.
Unfortunately,thereisacaveattorebasing.Itwilldestroyyourcommits!What
Gitdoesisrecreatethecommitsonyournewbaseanddiscardtheoldones.Thismaybeclearinthecaseofinteractiverebasing,forexample,whendeletingorsquashingcommits.However,rebasingwillalwaysdothis.Thereisagoldenrulewhenrebasing:onlyrebaselocalbranches!Ifyoueverrebaseabranchthatyourcoworkersuseaswell,theywillsuddenlymisshistoriccommits.Gitwillnotknowwhattodowiththosecommits.Yourcoworkerswillcurseyouforitandyouwillbedoingalotofapologizing.
RevertingchangesSometimes,youjustwanttogetridofwhateveritisyoudid.Whetheryoujustwanttocleanyourworkingdirectoryoryouwanttoactuallyundosomeitemsyou(accidentally)committed,Gitmakesitpossible.
Thereareacoupleofscenarioswecanthinkofthatwewantreverted.Thefirstisquitesimple.Wehavestagedsomefilesandwesimplywanttounstageeverything.Thegitresetcommanddoesthis:
gitstatus
Onbranchmaster
Changestobecommitted:
(use"gitresetHEAD<file>..."tounstage)
newfile:accidentallyadded.txt
modified:kernel.txt
deleted:lasers.txt
gitreset
Unstagedchangesafterreset:
Mkernel.txt
Dlasers.txt
gitstatus
Onbranchmaster
Changesnotstagedforcommit:
(use"gitadd/rm<file>..."toupdatewhatwillbecommitted)
(use"gitcheckout--<file>..."todiscardchangesinworkingdirectory)
modified:kernel.txt
deleted:lasers.txt
Untrackedfiles:
(use"gitadd<file>..."toincludeinwhatwillbecommitted)
accidentallyadded.txt
Itoftenhappensthatyoudonotwanttounstageafile,butundoallyourchangescompletely.Maybeyoumessedeverythingup;maybeyourcoworkerjustcommittedafixthatyouwerealsoabouttocommit;ormaybeyoujusttriedsomethingandnowwanttorevertit.Thetrickisstillgitreset,butwiththe--hardswitch.Makesureyouhavecommittedatleasttwofilestosomebranch.Now,changeafile,deleteafile,andeditafile.Checkyourchangesusinggitstatus,reset,andcheckyourstatusagain:
gitstatus
gitreset--hard
gitstatus
Thiswillundoallthechangesyoumadetofilesaswellasrestorefilesyoudeleted.However,whatitwillnotdoisdeleteuntrackedfiles.Thatbehaviorisasexpected;Gitisreallycarefulwithdeletingyourworkafterall.OnceyouaddyouraddedfiletoGit,gitreset--hardwilldeleteitthough:
gitaddsome_untracked_file.txt
gitstatus
gitreset--hard
gitstatus
Incaseyouwishtoremoveallyouruntrackedfilesaswell,youcanissueacleancommand.Bydefault,gitcleanneedsaparameter,either-i(forinteractive);-n(beingadry-run,ittellsyouwhatitwouldclean,butdoesnotactuallydoanything);or-f(force).Anotherusefulparameteris-d,whichspecifiednotonlyuntrackedfiles,butalsountrackeddirectoriesshouldberemoved:
gitclean-n
WouldremoveNewTextDocument.txt
gitclean-nd
WouldremoveNewTextDocument.txt
WouldremoveNewfolder/
gitclean-f
RemovingNewTextDocument.txt
gitclean-fd
RemovingNewTextDocument.txt
RemovingNewfolder/
Asyousee,allofthembehaveslightlydifferently,butareincrediblyusefulforcleaningoutyourworkingdirectory.
Towrapitup,tocompletelyremoveeverythinginyourworkspace,youcandoareset,followedbyaclean:
gitreset--hard
gitclean-fd
Anotherfeaturethatissometimesusefulisthatofrevertingcommits.Youcanrevertacommit,keepthechanges,andtheneditthembeforecommittingagain,oryoucanjustcompletelydeleteacommit,thereforeremovingitfromhistory.Forthis,youusegitresetwiththe--softor--hardswitchandthecommit,relativetoHEAD,thatyouwanttorevert.Inthefollowingexample,Ihavemadefourcommits,doc1,doc2,doc3,anddoc4:
gitlog--oneline
6a4bfff4
083179b3
86049c92
04e7e801
26df795Add.gitignore
Wecannowrevertthelastcommit,butkeepthechangesusing--soft:
gitreset--softHEAD~1
gitstatus
Changestobecommitted:
(use"gitresetHEAD<file>..."tounstage)
newfile:doc4.txt
gitlog--oneline
083179b3
86049c92
04e7e801
26df795Add.gitignore
Wecanalsorevertthelasttwocommits.Wegetallthechangesthatwerecommittedinthosetwocommitssowecaneditthemandrecommitthemasonecommit(orasmanycommitsasyoulike):
gitreset--softHEAD~2
gitstatus
Changestobecommitted:
(use"gitresetHEAD<file>..."tounstage)
newfile:doc3.txt
newfile:doc4.txt
gitlog--oneline
86049c92
04e7e801
26df795Add.gitignore
Lastly,itispossibletojustcompletelydeletecommitsinthiswayusingthe--hardswitch:
gitreset--hardHEAD~2
HEADisnowat86049c92
gitstatus
nothingtocommit,workingtreeclean
gitlog--oneline
86049c92
04e7e801
26df795Add.gitignore
Justlikewithrebasing,beverycarefulnottorevertcommitsthathavealreadybeenpushedtotheserver!Youmaydoit,butanyonethatalreadyhasthat
commitpulledwillnowpossiblyalsopulltheirhairouttryingtofixalltheerrors.
ThebranchingmodelNowthatyouknowhowtouseGit,createbranches,mergethem,movecommits,push,pull,reset,clean,stash,andmore,itistimeyouputthisintopractice.Anyprojectshouldhaveatleasttwopermanentbranches,masteranddevelopment.Thedevelopmentbranchiswhereprogrammersputtheirwork.Whensomethingisreadyforrelease,itgoesontomaster,whichshouldthenbereleased.
Ifyoudoatleastthat,youareonyourway.Whenabugarisesinproduction,youcanbranchfrommaster;fixthebugonthatbranch,ahotfixbranchifyoulike;andthenmergethatbranchbackintomaster(andreleaseit)anddevelopment,fixingthebugforfuturereleases.
Developersshouldalwaysbranchfromthedevelopmentbranch.Anynewfeaturescanbedevelopedonso-calledfeaturebranches,sometimesalsocalledtopicbranches.Oncethefeatureisdone,theycanbemergedbackintodevelopment.
Youprobablyhavesome(acceptance)testenvironmentaswell.Icanrecommendcreatinganotherpermanentbranchforeachenvironmentyouhave.Themergeflowshouldthenbesomethinglikefromthedevelopmentbranchtoyourtestbranch,andfromyourtestbranchtoyourmasterorproductionbranch.Thatway,youcanreleasefeaturestoyourtestenvironment,independentfromthemaster.YoucanputfeaturesAandBontest,andifthecustomeronlyapprovesfeatureB,thenyoucanputonlythatintoproduction.
ThismodelisactuallydescribedreallywellinablogpostbyVincentDriessen.Itisfrom2010,butstillrelevanttoday.Theimagesheusestodescribethemodelmayalsohelpingraspingwhatisgoingon.Irecommendyoureaditasitisprettygood.Youcanfinditathttp://nvie.com/posts/a-successful-git-branching-model/.
TaggingOnelastissueIwouldliketodiscuss,whichreallycomesinhandywhenreleasing,istagging.Youcantagacommitforlaterreference.Taggingismostlyusedtogiveacommitaversiontag,soyoucanfinditlater.Youwillknowexactlywhatcommitrepresentsversion1.0or1.1ofyoursoftwareasthatcommitistaggedwiththatspecificversion.
Therearetwotypesofcommits,lightweightandannotated.Alightweighttagisjustthat,atag.Anannotatedtagkeepssomeextrainformation,suchasacommitmessage,creationdate,andtheauthorofthetag.Creatingatagisreallyeasy.Simplyusegittagtag-name.Forannotatedtags,usethe-aswitchandspecifyamessage.Thiswillcreateatagforyourcurrentcommit:
gittagv0.1
gitshowv0.1
commitcf5e5c6af16f90990b2fb439e65973a66ff717aa
[...]
gittag-av1.0-m"Tagforv1.0"
gitshowv1.0
tagv1.0
Tagger:TonyStark<[email protected]>
Date:WedFeb123:55:362017+0100
Tagforv1.0
commit6a4bfff9386a18f6122bc1e419d35d1e3625b0bd
[...]
Asyousee,usinggitshowtag-namerevealsthecommitthatwastaggedwiththespecifictag.Theannotatedtagalsoshowsinformationonwhenandwhocreatedthetag.Youcanlistallyourtagswiththegittagcommand.
Tagsarenotautomaticallypushedtotheserver.Whenyouwanttoshareatag,youneedtoexplicitlypushthetag:
gitpushoriginv1.0
*[newtag]v1.0->v1.0
gitpull
remote:Countingobjects:1,done.
remote:Total1(delta0),reused0(delta0)
Unpackingobjects:100%(1/1),done.
Fromhttp://ciserver/the_avengers/the-suit
*[newtag]v1.1->v1.1
Youcanalsocreatetagsforoldercommits.Simplyspecifythecommithashtothegittagcommand:
gitlog--oneline
083179bAddedevenmorefiles
86049c9Changedsomefiles
04e7e80Addedsomefiles
26df795Add.gitignore
gittagv0.504e7e80
Bytheway,whenyougointoGitLab,thereisaTagspage,whereyoucanviewandcreatetags.
SummaryInthischapter,wehavelookedatthemostusedfeaturesofGit.Youshouldnowbeabletoeffectivelycontrolyoursource.Therewillbetimewhenyoumessup,butatleastyoustillhavesourcecontrol.Gitclients,suchasGitHub,SourceTree,andGitKraken,allmakeuseofthefeaturesdiscussedinthischapter.Wheneveryouclickabutton,oneormoreofthecommandswehaveseeninthischapterwillbeperformedinthebackground.Personally,Ifinditfareasiertouseaclient,butIknowsomepeoplewhowouldratherusethecommandline(toeachhisown).Whateveryouchoose,thischaptershouldbeaprettygoodintroduction.Itisnotuntilthelaterchaptersinthisbook,whenwearegoingtouseJenkinsextensively,thatmanyoftheadvantagesofusingGitandbranchesbecomeapparent.Inthenextchapter,wewillstartwithwritingsomeJavaScriptthatwecansetupforCIusingNode.jsandnpm,bothlocallyonyourdevelopmentmachineandonyourCIserverforyourentireteamtosee.
CreatingaSimpleJavaScriptAppInthischapter,wearegoingtocreateasimplewebshop.WewillstartbywritingafrontendusingHTML,CSS,andJavaScript.Noticethelackofabackend.WhatIreallylikeaboutfrontenddevelopmentisthatitissoeasytogetstartedwith.WecancreateacompleteappusingonlyNotepad(++)oranyothertexteditor.Ofcourse,thelackofadatabasewillpreventusfromstoringourresults,butfornow,thatdoesnotmatter.ThefocusofthisbookisCI,notdatabasesorbackenddevelopment.ThefrontendalonewillbeenoughtoexploreCI.Afterall,wecandotestsandotherautomatedtasks,suchasminification.EvenContinuousDeliveryandDeploymentarepossibletoimplement(justcopyyourfilestosomeserverhostingyourwebsite),butwewilldelaythatuntilwehaveourbackendaswell.Whatisreallycoolthoughisthatwewillnowbuildsomelamefrontend,makethenecessarytests,andthenlaterwecanconnectittoabackend,runourtests,andbeconfidentthateverythingworks.Actually,ourfirstversionofthefrontendwillbesonaive,aproofofconceptratherthananythingelse,thatwewillneedtochangethefrontendlateraswellwithoutbreakingit.
Tomakeourlivesalittleeasier,wewilluseTwitterBootstrapforstylingourpages,andjQuery(becauseBootstrapneedsit)andGoogle'sAngular.jstobindourdatatoourpages.IamkeepingthecodesamplessimplebecausethefocusofthisbookisCIandnotfrontend(orbackend)development.IfyouhaveneverworkedwithBootstraporAngular.jsbefore,youshouldstillbeabletofollowalltheexamples.
Nowhereinthischapter,ortherestofthebook,willIgiveyouthecompletecodeinonelisting.AllcodecanbedownloadedfromtheGitHubpageforthisbook,https://github.com/PacktPublishing/Continuous-Integration-Delivery-and-Deployment.Youwillfindthecodeforthisspecificchapterinitsrespectivefolder,https://github.com/PacktPublishing/Continuous-Integration-Delivery-and-Deployment/tree/master/Chapter0
4.;Inthischapter,IwillexplainthemoreinterestingorimportantpartsofthecodefromGitHub,oftenbeforelistingtheactualcode,soexpectthefollowingformat:
Wecanseefeaturexandyimplementedinthecode:
[Actualcodesample]
Sometimesadditionalinformationonthepreviouscodeorcommentsonthenextcodesample.
Thenode_modulesfolderisnotincludedinGitHub,soyouwillhavetodoannpminstalltogetthose(explainedinCreatingtheproject).Thisistruefortherestofthebook.
ThewebshopspecsBeforewegetstarted,wewillhavetoknowwhatwearegoingtobuild.Sincewearesuperagile,weonlyneedabasicoutline(Iusedagileasanexcusenottowritespecs):
Homepage:First,ofcourse,wewillneedahomepage.Thehomepageisverybasicandshowsuswhatotherpeoplehavebeenbuying.Thesearetilesthatshowanimage,name,andprice.Searchpage:Whenyousearchforaproduct,yougettothesearchpagewhereyoucanfilterbycategoryandsortyourresults.Thepagewillshowusimages,names,categories,prices,andashortdescriptionofalltheproductsthatmatchyourcriteria.Fromhere,youcanplaceanyproductinyourcartifyouareloggedin.Ausercansearchforproductsonanypage.Productpage:Whenyouclickonaproduct,eitherfromthehomepageorthesearchpage,youwillbetakentotheproductpage,whichisalsoverybasic.Itshowsanimage,thename,category,price,anddescriptionoftheproduct.Ifauserhasloggedin,theycanplaceaproductintheircartfromhere.Shoppingcartpage:Finally,yourcartshowstheproductsyouwanttobuy,howmanyofeach,andthepriceandtotalsofeachproductandofyourcompleteorder.Itwillhaveacheckoutbuttonthatsimplyshowsamessagethatsaysyourorderisbeingprocessedandyoucartwillempty.Whenauserisloggedin,thecarticonisshownoneverypageatthetopright.Loginpage:Anywebshopneedssomemethodofloggingin.Wecanloginatanypageatthetoprightofthescreen.Atthistime,itisnotpossibletologin,sinceweonlyhaveafrontendanditisnotpossibletokeepanysessionorstateonourpages.Fornow,let'sjustaddthebutton,butitwillnotdoanythingyet.Let'sassumeyouarealwaysloggedin.
InstallingNode.jsandnpmForthisproject,wearegoingtousethepackagesBootstrapandAngular.js.Wecaninstallthesethroughnpm.npmistheNode.jsPackageManager,soweneedtoinstallNode.jsandgetnpmforfree.Inthenextchapters,wearegoingtouseNode.jsaswellasmorenpm.
Node.jsisaJavaScriptruntime.JavaScript,traditionally,couldonlybeexecutedonyourbrowser.ThatposesaproblemwhenyouwanttoautomateyourJavaScripttests.Youreallydonotwantabrowsereverytimeyoutest,lint,orminify.
Node.jsmakesitpossibletorunJavaScriptoutsideyourbrowser.Itstartsupalocalserver(thatcanbeexposedtotheoutsideworld)andrunsJavaScript.Node.jsiscurrentlyapopularalternativeforApache,IIS,andNginx.InordertoautomateourJavaScripttests,wewillneedNode.js.Additionally,wewillcreateawebapplicationusingNode.js.
npmistheNodePackageManager.Kindoflikeapt-get,wecaninstall(oruninstall)packagesinourwebprojects.ItcomesbundledwithNode.js,sowedonotneedtodoanyadditionalworktoinstallthis.
Fornow,wewillneedNode.jsandnpmonourdevelopmentmachineonly;inthenextchapters,wewillneeditonourCIserveraswell,sowecanrunitinJenkins.
InstallingNode.jsandnpmisquiteeasy.Simplyheadovertohttps://nodejs.organddownloadthelatestLTSversion.Thisshouldgiveyouanmsifile.Simplyrunthemsifile,leaveallofthedefaults,andclickNext,Next,Next.Ifeverythingwentwell,youshouldnowbeabletoopenupacommandpromptandchecktheversionsofboththeprograms:
node-v
v6.9.4
npm-v
v3.10.10
CreatingtheprojectFirst,let'screateaGitrepositorythatwillholdourwebshopproject.MakesureyourVMisrunningandbrowsetoGitLab.Itdoesnotreallymatterifyoucreatetheprojectunderyourlocalaccountorinagroup;youcanalwayschangethatlater.Ihavecalledtherepositoryweb-shop,butyoucanreallynameitanythingyoulike.Ifyougoforanothername,makesuretochangethenameoftherepositoryinallupcomingcodesnippets.
Nowthatyouhavearepository,youcancloneittoyourdevelopmentmachine.Wehavedonethisbefore,sothatshouldnotbeaproblem.Ihavechosentoclonetherepositorytomydesktopforquickaccess,butIwouldrecommendcloningittoC:\RepositoriesoryourDocuments(\Repositories)folder,someplacewhereyoukeepyourGitrepositories:
cdyour-folder
gitclonehttp://ciserver/youruser/web-shop.git
cdweb-shop
Makesuretoreplaceyouruserwithyourusernameandweb-shopwiththenameofyourrepositoryifyoupickedanothername.IfyouareunsureabouttheURL,youcanalwayscopy-pasteitfromGitLab.
Afterthat,wewillwanttoinstallBootstrapandAngular.js.Wewillinstallthisusingnpm.Oneofthebenefitsofusingapackagemanageristhatitinstallsdependenciesforyouautomatically.Itcanalsokeeptrackofwhichpackagesyouhaveinstalledinaseparatefile.Thebenefittothatisthatyoudonotneedtocheckyourpackagesintosourcecontrolandyoucanretrievethosepackagesonyourbuildserver.Yourpackagesfoldercanbecomeprettybig,soitisusuallyfastertodownloadthemonyourCIserverthantocheckthemintoGit.Forexample,installingasomewhatbiggerpackage,suchasGulp(whichwewilluseinalaterchapter),willgiveyouastaggering1,000+files.Theyareonly4MBaltogether,butcheckingtheminandoutofGittakestime.Inoneofmyownprojects,whichisnoteventhatbig,Ihaveabout80MBin10,000+files,allnpmmodules.Thinkwhatyouwant,butthatistherealityofJavaScriptprogrammingintheyear2017.
Nexttothetimeitsavesbynothavingtocheckyourpackagesintosourcecontrol,anotherbenefitisthatwheneverapackagegetspulledfromGit,yourbuildwillfail(hopefully)andyoucanimmediatelytakeaction.Youcouldsavethemissinglocallyandusethat,butyouwillknowforsureitwillneverbeupdatedtothelateststandards(whichmayormaynotbeaproblem,dependingonthenatureofthepackage),unlessyouupdateityourself,ofcourse.
Adownsidetonotcheckinginyourpackagesisthatyourbuildisnowdependentontheactionsofthirdparties.Although,arguably,thatisalwaysthecasebecauseyouareusingthird-partysoftware.Actually,aboutayearago,thousandsofprogrammersworldwidesawtheirbuildsfailingbecausealeft-padpackagewith11linesofcodewaspulledfromnpm.Nooneknewtheydependedonthepackage,butotherpackagestheydependedondiddependonit.
Thatsaid,whenyourbuildsucceeds,youshouldsavethepackageswithyourbuild,whetheryoukeeptheminyourrepositoryornot,soyoucaneasilydeploythebuildanditwilljustwork.So,knowingtheprosandconstocheckingandnotcheckingyourpackagesintosourcecontrol,Ileaveyouwiththefinaldecisionwhetheryouwanttoincludetheminyourrepositoryornot.Inthisbook,wearenotgoingtodoit,butitisyourchoice.
Now,let'sactuallyinstallthepackages.Inthecommandprompt,browsetothefolderofyourweb-shopproject.Ifyouhavenotclosedthecommandafterthelastexample,youareprobablyalreadythere.Beforeweinstallanypackages,wewillneedapackages.jsonfile.Theeasiestwaytocreateapackages.jsonfileisusingthenpminitcommand.Thepackages.jsonfileholdsseveralpropertiesaboutyourproject,suchastheID,name,description,version,license,andwhichpackagesitdependsupon.Youcancreateonemanually,butusingnpminitletsyouspecifyvaluesforthesepropertiesandwillputtheminthefileforyouautomatically.Forthedefault,betweenparentheses,youcansimplyhitEnter:
cdyour-folder\web-shop
npminit
name:(web-shop)[Enter]
version:(1.0.0)[Enter]
description:AsimplewebshopfortheCIexample.[Enter]
entrypoint:(index.js)[Enter]
testcommand:[Enter]
gitrepository:(http://ciserver/sander/web-shop.git)[Enter]
keywords:[Enter]
author:SanderRossel(yournamehere)[Enter]
license:(ISC)[Enter]
{examplejson}
Isthisok?(yes)[Enter]
Now,checkoutthepackage.jsonfilethatwascreated.Theonlymandatoryfieldsherearenameandversion,astheyuniquelyidentifyyourownprojectshouldyoueverwanttopublishthemtonpm.Youcanfindwhatotherfieldscangointopackage.jsoninthenpmdocumentationathttps://docs.npmjs.com/.
Now,wecaninstallBootstrapandAngular.js:
cdyour-folder\web-shop
npminstalljquery--save
npminstallbootstrap--save
npminstallangular--save
Thereisashortcutforinstallingmultiplepackagesatonce;justlistthemoneaftertheother:
npminstalljquerybootstrapangular--save
The--saveswitchisoptional,butitisnecessaryfornpmtorestoreyourpackagefromthenpmrepositoryautomatically.Usingthe--saveswitchupdatesyourpackage.json.Itshouldnowcontainthefollowinglines:
"dependencies":{
"angular":"^1.6.1",
"bootstrap":"^3.3.7",
"jquery":"^3.1.1"
}
Youwillalsofindnpmhascreatedanode_modulesfolder,whichcontainstheBootstrap,jQuery,andAngular.jssources.Apparently,neitherofthemhaveanydependenciesbecausenootherpackageswereinstalled.Youcancheckoutwhathappenswhenyouinstallapackage,suchasGulp.Itwillinstalllotsofpackages,althoughitwillonlyaddonelinetoyourdependenciesinpackage.json.Goahead,tryit:
npminstallgulp--save
npmuninstallgulp--save
Theonlythingthatwillleavebehindisanempty.binfolderinnode_modules.
ThenextthingweneedtodoismakesureGitdoesnotaddallofourpackagestotherepository.CheckoutthestatusofyourGitrepositoryandyouwillseethe
node_modulesfolderandpackage.jsonhavebeenadded:
gitstatus
[...]
Untrackedfiles:
[...]
node_modules/
package.json
Youcancreateagitignorefile,whereyoucanspecifywhichfiles,folders,orpatternstoignore.Creatingthisfileisalittletrickyasthenamehastobe.gitignore,whichisnotavalidfilenameinWindows.Instead,simplycreateatextfileandnameit.gitignore.Windowswillautomaticallyremovethelastdotandyouaregoodtogo.Inthe.gitignorefile,putthetextnode_modules.ThiswilltellGittoignoreanythingcallednode_modules.Now,checkthestatusagain:
gitstatus
[...]
Untrackedfiles:
[...]
.gitignore
package.json
TimetocommitthesechangestoGit.Lessonnumberone:whenusingsourcecontrol,keepyourcommitssmall:
gitadd.
gitcommit-m"Initializedtheweb-shopproject."
gitpush
CheckitoutinGitLabandyoucanconfirmnode_modulesreallywerenotpushedtoGit.
Totestwhetherwecanreallyrestoreourpackagesfromnpm,clonetheprojectagain,intoadifferentfolder.Youcancheckthattherereallyreallyreallyarenonode_modules.Afterthat,usethenpminstallcommandtorestoreyourpackages.Afterthat,youcansimplydeletethisfolder:
cdyour-folder
gitclonehttp://ciserver/youruser/web-shop.gitweb-shop-test
cdweb-shop-test
npminstall
cd..
rmdir-s-qweb-shop-test
CreatingtheHomepageOntotheactualcode.ThefirstthingwewillneedaresomeHTMLpages.IamtryingtokeepthingsassimpleaspossiblesowewillnotuseanyHTMLrenderingengines.Asaresult,wewillhavetocopy/pastesomeHTMLintoeverypagewehave(fornow).Inyourprojectfolder,createthreenewfolderscalledcss,scripts,andviews.Also,createafilecalledindex.htmlandputthefollowingcodeinit:
<!DOCTYPEhtml>
<html>
<head>
<metacharset="UTF-8">
<title>CIWebShop</title>
<linkrel="stylesheet"type="text/css"
href="node_modules\bootstrap\dist\css\bootstrap.css">
<linkrel="stylesheet"type="text/css"href="css\layout.css">
<linkrel="stylesheet"type="text/css"href="css\utils.css">
<scriptsrc="node_modules\angular\angular.js"></script>
<scriptsrc="node_modules\jquery\dist\jquery.js"></script>
<script
src="node_modules\bootstrap\dist\js\bootstrap.js"></script>
<scriptsrc="scripts\utils.js"></script>
<scriptsrc="scripts\repository.js"></script>
<scriptsrc="scripts\index.js"></script>
</head>
<bodyng-app="shopApp">
...
</body>
</html>
Thisispartofthecodethatismoreorlessthesameforeverypage.Alltheotherpageswillgointheviewsfolder,sothereferencestothestylesheetsandscriptswillstartwith..\togetbackinthemainfolderfirst.ng-appisanAngular.jsnotationanddesignatestherootelementofyourapplicationtoAngular.js.Withinyourapparecontrollersthatcanneatlydefinethepropertiesandfunctionalityof(apartof)apageandsohelptokeepfunctionalityisolatedor"separateyourconcerns."Anappcanhavemultiplecontrollersandcontrollerscanbere-used.
Insidethebodytagsgoestherestofthepage.Thefirstpartisreservedforamenubarthatisthesameforeverypage.AfterthatcomestheAngular.jscontroller,
whichisuniqueforeachpage(separationofconcerns).Thefooter,whichisstillinthesamedivasthecontroller,isthesameonallpages:
<navclass="navbarnavbar-defaultnavbar-fixed-top">
<divclass="container">
...
</div>
</nav>
<divclass="container"ng-controller="homeController">
...
<footerclass="footer">
<p>Copyright©2017</p>
</footer>
</div>
Thereisquitealotgoingoninthesefewlines.Theclassesnavbar,navbar-default,andnavbar-fixed-topareBootstrapclasseswhichcreateanavigationbarthatispage-wide,fixedtothetop,andalwaysvisible.ThecontainerclassisthebaseofBootstrap'spowerandisnecessarytowrapsitecontentandalsohousesBootstrap'sgridsystem.Thegridsystemdividesyourpageinrowsandtwelvecolumns.Thereisarowclass,whichwewillseeinabit,andvariouscolumnclassesthatyoucanusetodivideyourcontentoveranumberofcolumns.
Theng-controllerbitisanotherAngular.jsnotationandindicatesyourbindingcontext.SinceweareusingthehomeControllerandthehomeControllerexposestopProducts,wecanloopthroughthoseproductsinourHTML.Eachloopedproducthasanamepropertythatwecanbindto,soourbindingcontextisalistofobjects(products)withanameproperty.Wewillseethisbindinginactioninthenextfewexamples.Noticethatnavbarhasnoexplicitcontroller,butwecanstillusebindingaswewillsee.
Thefooterkindofspeaksforitself,soIamnotgoingtoexpandonthat.
Beforewecontinue,hereisanimageofwhatthepageultimatelylookslike.Itshouldgiveyouabitofanideaofwhatisgoingon:
Let'stakeacloserlookatthenavigationbar.Itisnotexactlyeasy,butitlookscoolandBootstrapstillmakesitaloteasierthanwhenyouhadtodoityourself.So,inside<nav...><div...>goesthefollowingHTML:
<divclass="navbar-header">
<buttontype="button"class="navbar-togglecollapsed"data-toggle="collapse"data-
target="#menu">
<spanclass="icon-bar"></span>
<spanclass="icon-bar"></span>
<spanclass="icon-bar"></span>
</button>
<aclass="navbar-brand"href="index.html">CIWebShop</a>
</div>
<divclass="collapsenavbar-collapse"id="menu">
<ulclass="navnavbar-navnavbar-right">
<li><ahref="#"><spanclass="glyphiconglyphicon-user"></span>
Login</a></li>
<li><ahref="views\shopping-cart.html"><spanclass="glyphiconglyphicon-
shopping-cart"></span>Order</a></li>
</ul>
<divclass="navbar-formnavbar-right">
<divclass="form-group">
<inputclass="form-control"placeholder="Search"ng-model="query">
</div>
<ahref="views\search.html?q={{query}}"class="btnbtn-default">
<spanclass="glyphiconglyphicon-search"></span>
</a>
</div>
</div>
Therearetwocomponentshere,navbar-headerandnavbar-collapse.Thebuttonintheheaderistheso-calledhamburgermenu,whenyourpagegetstoosmalltodisplaythemenu(onyourphone,forexample).Thespanelementswiththeicon-barclassarejusttheimagesonthebutton.Theanchorwithnavbar-brandisthecompanylogoatthetopleftofthepage.
Inthenavbar-collapsepart,wefindtheactualmenu.Thereisanunorderedlistofmenuitems,beingourloginpage,andourshoppingcart.Thespanelementswiththeglyphiconclassesarejustimages.Thedivelementwiththenavbar-formclassisoursearchbar.WhatisinterestinghereisthattheinputelementhastheAngular.jsng-modelsettoquery.Thismeansthevalueoftheinputisboundtosomequerypropertyonourimplicitmodel.Sinceweneverexplicitlycreatedamodelwithaqueryproperty,Angular.jscreatesthemodelwithaquerypropertywhichcanthenbeusedonthepage.Thequerypropertyisthenusedintheanchortagwiththe{{query}}syntax.ThissyntaxtellsAngular.jstoplacethecontentsofthequerythereinsteadof{{query}}.Theupdateisdoneautomaticallywhenthevalueofthequerychanges.So,whenthepagesarecomplete,ifyoutypeFanintotheinputandthenclickthelink(whichtakestheformofabutton),itwilldirectyoutoview\search.html?q=fan.
Nowfortheuniquebodypartofourhomepage.ItstartsoffwiththisbigsquarethatsaysCIWebShopandWelcome....ItiscalledaJumbotronanditisactuallyprettyeasytoimplementusingBootstrap:
<divclass="container"ng-controller="homeController">
<divclass="jumbotron">
<h1>CIWebShop</h1>
<pclass="lead">WelcometotheCIWebShop!<br/>
Browseourwares,butremember:ifyoubreakityoubuyit!</p>
</div>
Thencomesthesearchbar.Weputthisinrowsonothingwillevergetplacednexttoit.ItispartoftheBootstrapgridsystem.Here,wealsoseethecol-lg-12class.Itmeansthecontentoftheelementshouldbespreadoutover12columns
whenthescreensizeislarge.Ifyouwantedtwoelementsnexttoeachother,youcouldspecifycol-lg-6twice.Whenthescreengetssmaller,youmightnotwanttodisplaytwoelementsnexttoeachother.Inthatcase,youcouldusetheclasscol-sm-12orspreadoutover12columnswhensmall.Puttingthetwoclassesononeelementistotallyvalidandwillchangehowtheelementsaredisplayedindifferentscreensizes.Thiskindofdesigniscalledresponsive,astheUIrespondstothesizeofthescreen,makingitreadableonallscreensizes(whendoneright,ofcourse).Again,youseetheng-modeland{{}}syntaxes.Thistime,wearebindingtothesearchTermpropertyofhomeController:
<divclass="jumbotron">
[...]
</div>
<divclass="row">
<divclass="col-lg-12">
<h2>Searchforproducts...</h2>
</div>
<divclass="col-lg-12">
<divclass="input-group">
<inputclass="form-control"placeholder="Searchfor..."ng-model="searchTerm"/>
<divclass="input-group-btn">
<ahref="views\search.html?q={{searchTerm}}"class="btnbtn-default">
<spanclass="glyphiconglyphicon-search"></span>
</a>
</div>
</div>
</div>
</div>
Thelastpartofourpageisthepopularproductssection.Here,weseeng-repeat,whichloopsthroughanarrayandappliesthebindingforeachiteminthatarray:
<divclass="row">
<divclass="col-lg-12">
<h2>Otherpeoplebought...</h2>
</div>
<divng-repeat="productintopProducts">
<divclass="col-lg-4">
<divclass="thumbnail">
<ahref="views\product.html?id={{product.id}}">
<imgsrc="http://placehold.it/150x150"alt="..."/>
</a>
<divclass="captionclearfix">
<ahref="views\product.html?id={{product.id}}">
<h3class="wrap"title="{{product.name}}">{{product.name}}</h3>
</a>
<p>{{'€'+product.price}}</p>
<pclass="clearfix">
<ahref="views\shopping-cart.html"class="btnbtn-primarypull-
right">Buy</a>
</p>
</div>
</div>
</div>
</div>
</div>
IntheBootstrapdepartment,weseetheclassesthumbnail,caption,clearfix,andpull-right.Itisallprettystraightforward.Youcancheckoutwhattheydoonyourbrowser.Theimagecomesfromplacehold.it,whichisawebservicethatservesimagesinanydimensionyouspecifyintheURL.Soplacehold.it/150x150returnsa150x150-pixelimage.Reallyveryhandy.
TheonlyclassyouseeherethatisnotBootstrapiswrap.Itresidesincss\utils.cssanditmakessurethetitlesofourproductsareshownonasinglelineandshowellipsiswhentheyaretoolong.TheonlyotherCSSfilewehaveislayout.cssanditaddssomepaddingonthetop,soourpagestartsbelowthetopmenuinsteadofbehindit.Herearethecontentsofbothutils.cssandlayout.css.Wecouldputitinasinglefile,butthetwofileshaveveryseparatefunctions,sowedon't:
body{
padding-top:70px;
}
.wrap{
overflow:hidden;
text-overflow:ellipsis;
white-space:nowrap;
}
ThatleavesuswiththeJavaScriptcode.Itisactuallyprettysimple.Inindex.js,wecreatetheAngular.jsappandcontroller.Inthebindingmodel,wespecifythetoppopularproductsandsearchTermforthesearchfieldthatisnotonthemenu:
angular.module('shopApp',[])
.controller('homeController',function($scope){
$scope.topProducts=repository.getTopProducts();
$scope.searchTerm='';
});
Andsomostofthecodegoesintotherepository.jsfile.Thisfilehasthedataweshowonthevariousscreens:
varrepository=(function(){
'usestrict';
varproducts=[{
id:1,
name:'FinalFantasyXV',
price:55.99,
description:'FinalFantasyfinallymakesacomeback!',
category:'Gaming'
},{
//...
}];
return{
getTopProducts:function(){
return[products[1],products[2],products[3]];
},
getProduct:function(id){
returnproducts.filter(p=>p.id===id)[0];
},
search:function(q){
if(q==null){
return[];
}else{
returnproducts.filter(p=>p.name.toLowerCase().indexOf(q.toLowerCase())>=0);
}
}
};
})();
Atthetopisourarrayofalltheproducts.Thefunctionsatthebottomareaninterfacetotheseproducts.Thefunctionsareprettyself-explanatory.Itisgoodtomentionthatthesearchfunctionreturnsanemptyarraywhenthesearchstringisemptyandthesearchisperformedonnameonlyandiscaseinsensitive.ThegetTopProductsfunctionsimplyreturnsthefirstthreeproducts,becausewecannotactuallycalculateamostorderedlist.
The=>notationisarelativelynewEcmaScript6notation.Itisbasicallytheshortcutforaregularfunction.Using"arrowfunctions,"astheyarecalled,wecanusep=>p.id===idinsteadofthemuchlongerfunction(p){returnp.id===id;}.Formoreinformationonarrowfunctions(andES6ingeneral),seehttps://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Arrow_func
tions.
CreatingtheProductpageNextupistheproductpage.Inyourviewsfolder,createanewfileandcallitproduct.html.Thisisbyfarthesimplestpage:
<divclass="row">
<divclass="col-lg-12text-center">
<h2>{{name}}</h2>
<p>
<imgsrc="http://placehold.it/300x300"alt="..."/>
</p>
<p>{{description}}</p>
<p>{{'€'+price}}</p>
<p>
<ahref="shopping-cart.html"class="btnbtn-primary">Buy</a>
</p>
</div>
</div>
Thatisallthereistoit.Thetext-centerclass,whilecalledtext-center,willactuallycentereverything.
Wecanputthescriptsforutils,repository,andtheproductpageitselfintheheader:
<scriptsrc="..\scripts\utils.js"></script>
<scriptsrc="..\scripts\repository.js"></script>
<scriptsrc="..\scripts\product.js"></script>
Theproduct.jsfileisactuallyprettystraightforward.Wecreatetheappandthecontroller,wegettheproduct,andwecopytheproduct'spropertiestoourownviewmodel:
angular.module('shopApp',[])
.controller('productController',function($scope){
varid=+utils.getQueryParams()['id'],
p=repository.getProduct(id);
$scope.name=p.name;
$scope.price=p.price;
$scope.description=p.description;
$scope.category=p.category;
});
Theinterestingpartistheutils.getQueryParams()['id']part.Itcomesfromtheutils.jsfileandlooksasfollows:
varutils=(function(){
return{
getQueryParams:function(){
varqs=document.location.search.split('+').join(''),
params={},
tokens,
regex=/[?&]?([^=]+)=([^&]*)/g;
while(tokens=regex.exec(qs)){
params[decodeURIComponent(tokens[1])]=decodeURIComponent(tokens[2]);
}
returnparams;
}
};
})();
Iadmit,Igotthecodesomewherefromtheinternet,butitiseasyenough.WegettheURLthatisonourbrowser,getthequeryparameters(someurl.com?param1=value+param2=value)usingaregularexpression,decodethevalues(astheyareURIencoded),andputtheparameternamewiththevalueinanobjectthatwereturn.So,inutils.getQueryParams()['id'],wegetthevalueoftheidparameterintheURL.Now,wecanactuallyshowproductswithid=2whenwebrowsetofile:///C:/Users/sander.rossel/Desktop/web-shop/views/product.html?id=2(orwhereveryousavedyourproject).
CreatingtheSearchpageThenextpagewearegoingtoimplementisthesearchpage.Thispageshouldhavefewsurprisesforyounow:
<divng-repeat="productinresults">
<divclass="thumbnail">
<divclass="row">
<divclass="col-lg-2">
<ahref="product.html?id={{product.id}}">
<imgsrc="http://placehold.it/150x150"alt="..."/>
</a>
</div>
<divclass="col-lg-9">
<divclass="caption">
<ahref="product.html?id={{product.id}}">
<h3class="wrap"title="{{product.name}}">{{product.name}}</h3>
</a>
<pclass="labellabel-default">{{product.category}}</p>
<pclass="wrap">{{product.description}}</p>
</div>
</div>
<divclass="col-lg-1clearfix">
<divclass="captionpull-right">
<p>{{'€'+product.price}}</p>
<ahref="shopping-cart.html"class="btnbtn-primarypull-right">Buy</a>
</div>
</div>
</div>
</div>
</div>
TheJavaScriptisprettymuchwhatyouwouldexpectaswell:
angular.module('shopApp',[])
.controller('searchController',function($scope){
varq=utils.getQueryParams()['q'];
$scope.results=repository.search(q);
});
CreatingtheShoppingcartpageLast,butnotleast,istheshoppingcart.Thispageisalittledifferentfromtheotherpages,asithassomelogictoit.Onthispage,wecanincrementacounteranddeleteitems.Ifwehadabackend,itwouldbepossibletoadditems,butthatissomethingforlaterchapters:
<divclass="row"ng-hide="lines.length">
<divclass="col-lg-12text-center">
<p>Therearenoitemsinyourshoppingcart...</p>
<p>Youshoulddosomeshopping!</p>
</div>
</div>
<divng-repeat="lineinlines">
<divclass="thumbnail">
<divclass="row">
<divclass="col-lg-2">
<ahref="product.html?id={{line.product.id}}">
<imgsrc="http://placehold.it/150x150"alt="..."/>
</a>
</div>
<divclass="col-lg-7">
<divclass="caption">
<ahref="product.html?id={{line.product.id}}">
<h3>{{line.product.name}}</h3>
</a>
<pclass="labellabel-default">{{line.product.category}}
</p>
</div>
</div>
<divclass="col-lg-1clearfix">
<divclass="captionpull-right">
<inputng-model="line.number"/>
<buttontype="button"ng-click="removeLine(line)">
<spanclass="glyphiconglyphicon-trash"></span>
Delete
</button>
</div>
</div>
<divclass="col-lg-2clearfix">
<divclass="captionpull-right">
<p>{{'€'+line.subTotal()}}</p>
</div>
</div>
</div>
</div>
</div>
<divclass="row">
<divclass="col-lg-12">
<pclass="pull-right">{{'€'+total()}}</p>
</div>
</div>
Thetoprowhasng-hide,whichmeansthiselementshouldnotbevisiblewhen,in
thiscase,lines.lengthevaluatestotruthy.So,whenlines.lengthreturns0,wedisplaythetextTherearenoitemsinyourshoppingcart....Youwillalsoseewebindtosomefunctions.ng-click,whichbindsafunctiontotheclickeventhandler,invokestheremoveLinefunction,andpassesinthecurrentlinevariable(fromng-repeat)asanargument.Otherthanthat,weseetheline.subTotal()andtotal()functionsbeingbound.NoticethatweneedtoinvokethefunctionorAngular.jswillbindtothestringrepresentationofthefunction:
angular.module('shopApp',[])
.controller('shoppingCartController',function($scope){
varLine=function(options){
this.product=options.product,
this.number=options.number
};
Line.prototype.subTotal=function(){
return(this.product.price*this.number).toFixed(2);
}
$scope.lines=[newLine({
product:repository.getProduct(1),
number:1
}),newLine({
product:repository.getProduct(3),
number:2
})];
$scope.total=function(){
varsum=0;
$scope.lines.forEach(function(l){
sum+=+l.subTotal();
});
returnsum.toFixed(2);
};
$scope.removeLine=function(line){
$scope.lines.splice($scope.lines.indexOf(line),1);
}
});
So,wearecreatingaLineclass,whichcontainsproductandacountproperty.Iamusingprototypebecauseithasbeenaroundforalongtime,allbrowserssupportit,andIthinkmostpeoplewillunderstandthisbetterthannewerES6classes.Anyway,thesubtotalofanorderlineisthepriceoftheproducttimesthenumberofproductsyouwant.Wecreatetwolinesusingrepository.getProduct.Thetotaloftheorderisthesumofsubtotalsofallthelines.ThetoFixed(2)functionwillrounddecimalstotwo(thisisafloatingpoint,sobeforeweknowit,weget.000000001kindofnumbers).toFixed()returnsastring,sowehavetocastsubTotalofalinetoanintegerbeforewecanaddthemintotal.Last,removeLineremovesalinefromthelinesarray.YouwillseeallthisbindsnicelyintheHTML.Youcanchangethenumberofaline,andboththesubtotalandtotalwillupdate
immediately.Keepinmindthatwearenotcheckingwhetherthenumberisactuallyanumber,sogivingitaninputofsomestringwillgiveitthesubtotalNaN(NotaNumber),andbecauseanythingplusNaNequalsNaN,thetotalwillalsobecomeNaN.
SummaryInthischapter,wehavecreatedthefirstversionofawebshopapplicationthatwearegoingtousethroughoutthebook.Tokeepitsimple,ithasnobackendyet,butinthefollowingchapterswewilladdabackendwithNode.js,MongoDB,C#Core,andPostgreSQL.Butfirst,let'saddvarioustestsandautomatethose.Inthenextchapter,wewilluseJasminetowriteourtests,Karmatoautomaticallyrunthosetests,andSeleniumwithProtractortowriteandrunend-to-endtests.
TestingYourJavaScriptInthischapter,wearegoingtotestthecodewehavewritteninthepreviouschapters.Wewillstartoutwithunittests.Wecanwriteunittestsin,andfor,JavaScriptusingJasmine.Afterthat,wewillwriteUItestsusingSelenium.Attheendofthechapter,wewillhavetestedourcompleteapplicationsofarandalittlebitmore.
Theobviousadvantagetotestingyourcodeisthatyouwillbeabletocatchbugsassoonastheyareintroducedinanypartoftheapplication.HowoftenhaveyoumadeachangetosomepageonlytofindoutanotherpagebrokebecauseyouchangedsomeJavaScriptthatwassharedbetweenthetwopages?Themoreimportantquestion:howoftendidyounotfindoutabouttheotherpagebreaking?Exactly!SothatiswhytestingyourJavaScriptcanreallygiveyouanedgeindeliveringhigh-qualitysoftware.
Alessobviousadvantagetotestingyourcodeisthatinordertomakeyourcodetestable,youneedtowriteitinacertainway.A1,000-line-longfunction(andyes,Ihaveseenthem)isnottestable.Suchafunctiondoessomuchthatyouwouldhavetowrite100testsjusttocovereverypossibleoutcome.However,sincenooneunderstandsa1,000-linefunction,noonewillunderstanditstestseither,letaloneexpandonthem.
Youarealsoforcedtothinkaboutseparationofconcernswithinyourapplication.Afunctionthatdirectlyinsertsintothedatabaseisnottestablebecausewecannotwriteanassertionthatcheckswhethertherecordwasinserted(well,wecould,butweshouldn't).So,somehow,weneedtocreateinterfacesthatwecanmock,andimplementittwice.Oneimplementationthatdoesadatabaseinsertforourproductionapplication,andoneimplementationthatuses"inserts"tomemoryforuseinourtests.Thishastheaddedadvantagethatyourcodebecomesmoremodular,makingthedifferentpartsproperlyisolatedandthuseasiertounderstand,easiertotest,andeasiertochange.
Again,Iwillnotgiveyouthecompletecodeinonelistinganywhereinthischapter.ThecodecanbedownloadedfromtheGitHubpageforthisbook,https:/
/github.com/PacktPublishing/Continuous-Integration-Delivery-and-Deployment.Torestorenode_modules,usenpminstall.
UnittestingwithJasmineFirstthingsfirst,wewillneedtoinstallJasmine.IcanrecommendgoingthroughthischaptertogetstartedwithJasmine.Afterthat,checkoutthedocumentationonthewebsite,https://jasmine.github.io/.Jasmineisprettyextensive,butthedocumentationisprettyspoton.Now,openupacommandpromptandbrowsetoyourprojectfolder.Oncethere,wecaninstallJasminethroughnpm.LikewithAngular.jsandBootstrap,wewanttosaveJasminetoourpackage.jsonfile,butthistime,asadeveloperdependency.AregulardependencysuchasAngular.jsandBootstrap,isnecessarytorunthecode.Adeveloperdependencyisonlyinterestingfordeveloperswhowanttomakeabuildofthesoftware.Inotherwords,softwarecannotrunwithoutthedependencies,butitcanrunperfectlyfinewithoutthedeveloperdependencies.Thatsaid,developerdependenciesarenotlessimportanttothedevelopmentprocess.OurwebsitedoesnotneedJasminetorun,butwedefinitelywanttorunourunittestsafterwehavemadesomechanges:
cdfolder-of-your-project
npminstalljasmine--save-dev
Thisaddsanewpropertyinyourpackage.jsonfile:
"devDependencies":{
"jasmine":"^2.5.3"
}
Developerdependenciesareinstalledonnpminstalljustlikeregulardependencies.However,npminstallhasanadditional--production(or--prod)flagthatcanpreventthedeveloperdependenciesfrominstalling.
Youcanalsolistyourinstalleddependenciesandfilterondevorprod.Inthenextexample,wewillseedifferentwaystoinstallandlistdependencies.The--depthflagindicateswhethertoshowdependenciesofyourpackages(andthedependenciesofyourdependenciesandsoforth).ThermdircommandisaWindowscommandandsimplyremovesthenode_modulesfolder(/Stoforcedeletionofanyfilesandsubdirectoriesand/Qtonotaskforconfirmation):
rmdir/S/Qnode_modules
npminstall
npmlist--dev--depth=0
npmlist--prod--depth=0
rmdir/S/Qnode_modules
npminstall--only=prod
npmlist--depth=0
rmdir/S/Qnode_modules
npminstall--only=dev
npmlist--depth=0
npmlist--dev--depth=0
npminstall
NowthatwehaveJasmineinstalled,wecanalmoststartusingit.JasmineisjustsomeJavaScriptlibrary,soitneedsabrowsertorun.Browsershavenobuilt-insupportforJasmine,soweneedtocreateapagetorunourtestsandshowtheresults.Luckily,Jasminehasalotofthat;weonlyneedtogluesomepiecestogetheronasimpleHTMLpage.
Inyourprojectfolder,createafoldercalledtest.Inthere,createafilenamedindex.html.Intheindex.htmlfile,putthefollowingHTMLcode:
<!doctypehtml>
<html>
<head>
<title>JasmineSpecRunner</title>
<linkrel="shortcuticon"type="image/png"href="../node_modules/jasmine-core/images/jasmine_favicon.png">
<linkrel="stylesheet"href="../node_modules/jasmine-core/lib/jasmine-core/jasmine.css">
</head>
<body>
<scriptsrc="../node_modules/jasmine-core/lib/jasmine-core/jasmine.js"></script>
<scriptsrc="../node_modules/jasmine-core/lib/jasmine-core/jasmine-html.js"></script>
<scriptsrc="../node_modules/jasmine-core/lib/jasmine-core/boot.js"></script>
<!--includesourcefileshere...-->
<!--includespecfileshere...-->
<scriptsrc="spec/test.js"></script>
</body>
</html>
Tobehonest,Ireallydonotknowwhatthisdoes.Obviously,weloadsomeJasminefilesandtheysomehowworktheirmagic.ThegoodthingiswereallydonotneedtoknowhowJasminedoesit;thatistheentirereasonweuseathird-partylibraryinsteadofwritingourown.
Theimportantpartinthisfileisthespec/test.jsscript,whichwehavenotcreatedyet.Sogoaheadandcreateafoldercalledspec(withinthetestfolder)andputatest.jsfileinit.Wecanputourtestsinthetest.jsfile:
(function(){
'usestrict';
describe('sampletests',function(){
describe('ourfirstJasminetests',function(){
it('shouldsucceed',function(){
expect(true).toBe(true);
});
});
});
})();
Thereisquiteabitgoingonthereandwewillgettoitinasecond.First,makesureyoudideverythingright.Openupyourindex.htmlfileonyourbrowser.Youshouldseesomethinglikethefollowing:
Theysayapicturesaysmorethan1,000wordsandIguessthatistrue.Lookingatthecodeandthescreenshot,itbecomesclearwhatthedescribeanditfunctionsdo.Thedescribefunctionsimplyaddsabitofcontexttoyourtests,whatyouaretesting,orhowyouaretestingit.Youcannestyourdescribefunctionstoaddevenmorecontext.OurfirstdescribemakesitclearthatwearetestingthewebshopwhiletheseconddescribemakesitclearthatwearerunningourfirstJasminetests.Theitfunctioncontainsyouractualtestandshouldcontainatleastasingleassertion.Anassertion,inunittesting,issimplycheckingwhetherthevalueyouexpectmatchestheactualvalueinyourcode.Inthisexample,trueindeedmatchestrue.Asyoucansee,thetestalmostreadsasplainlanguagewiththeexpectandtoBefunctions.
ExplainingwhatyouaregoingtotestinthismannerisalsocalledBehavior-DrivenDevelopment(BDD)forshort.Youdefineanddescribethebehavior
andthentestwhetheryourapplicationsuccessfullyimplementsthatbehavior.Thetest.jsfileisinafoldercalledspecbecauseyourtestsdescribethespecs,orspecifications,ofyourapplication.
Next,let'saddafailingtest.Inyourtest.jsfile,addthefollowingtestdirectlyundertheprevioustest(includedintheexampleforclarity):
it('shouldsucceed',function(){
expect(true).toBe(true);
});
it('shouldfail',function(){
expect(false).toBe(true);
});
RefreshingyourJasminepageshouldnowgiveyouanerrorwiththedetailsofwhatiswrong:
Yougetthenameofyourfailingtest,theexpectedvalue,theactualvalue,andacompletestacktraceofyourerror.Whenmultipletestsfail,ormultipleassertionswithinthesametestfail,theyareallshowninthisoverview.Noticethatyoucanstillswitchyourviewtothelistofallyourtests,whereyougetabetteroverviewoftheteststhatfailed:
UsefultomentionisthattoBeteststwoobjectsforequality.Quiteoften,youwillwanttocomparetwoobjectsorarraysthatareequalintermsofpropertiesandvalues,butnotintermsofreferenceinmemory.InlanguagessuchasC#,thisisoftenaproblem,asyoueitherneedtoloopthroughpropertiesandcomparethemusingreflectionormanually,bothofwhichareahassle.InJavaScript,thisisaloteasier.ThetoBefunctionisnotgoingtocutithere,butthelookalikefunctiontoEqualwilldothejobforyou:
it('shouldbethesameobject',function(){
varo1={
firstName:'Sander',
lastName:'Rossel'
};
varo2={
firstName:'Sander',
lastName:'Rossel'
};
expect(o1).toEqual(o2);
});
Obviously,o1ando2arenotactuallythesameobject,buttheydohavethesamepropertiesandvalues.BecauseweusetoEqualforcomparison,thistestwillnotfailunlessweadd,change,orremoveapropertyononeoftheobjects.YoumightbehappytoknowthattoEqualalsoworksonarrays:
it('shouldbethesamearray',function(){
vararr1=['Hello',{},1,true];
vararr2=['Hello',{},1,true];
expect(arr1).toEqual(arr2);
});
Otherfunctionsyoucanusetocreateyourassertsarenot,toBeDefined,toBeNull,toBeTruthy,toBeFalsy,toContain,toBeNaN,toThrow,andlotsmore:
expect(something).toBeDefined();
expect(something).not.toBeNaN();
expect(something).toContain('avalue');
expect(something).toBeGreaterThan(10);
expect(function(){
something();
}).toThrow();
Additionally,therearesetupandteardownfunctionsyoumayuseforeverytestinyourdescribe.ThefunctionsbeforeEachandafterEachrunbeforeandaftereachit,soyourtestsmayusethesamevariableandyoucanresetandreinitializeitbetweeneachtest.Alternatively,youcanusebeforeAllandafterAll,whichonlyrunsoncebeforeallyourtestsandafterallyourtestsarefinished:
describe('setupandteardowntests',function(){
varsomething;
beforeEach(function(){
something='Somevalue';
});
afterEach(function(){
something=null;
});
it('shoulddostuffwithsomething',function(){
expect(something).toBe('Somevalue');
});
});
Oneofthecoolestfeaturesisprobablythatofspies.Youcanactuallytrackwhetherthemethodsarecalled,howoften,andusingwhatarguments:
//Intentionallyoutsidethescopeofthetests...
varmath={
add:function(a,b){
returna+b;
}
};
(function(){
'usestrict';
//...
describe('sampletests',function(){
describe('spytests',function(){
beforeEach(function(){
spyOn(math,'add').and.callThrough();
});
it('shouldadd1and2andcallontheaddfunction',function
(){
expect(math.add(1,2)).toBe(3);
expect(math.add).toHaveBeenCalled();
});
});
});
})();
Youmaynotneedallofthisfunctionality.Forthesamplewebshop,wecertainlywon't.Butknowingwhatisavailabletoyounowwillsaveyoutimeandtroubleinthefuture.Again,theJasminedocumentationisprettyextensiveanddescribesallofthisinmoredetail.Itcontainssamplesonasserts,setupandteardown,spies,regularexpressions,mocking,andasynchronoustesting.
Onceyougetafewmoretests,itispossibletosingleoutasingletestoragroupoftestsintheway.Ithelpsinfocusingonfixingasingletest.Thisisveryhelpfulwhenyouhavedozensoffailingtests,butyouknowallfailuresarecausedbythefailingofsomecorefunctionality.Youcannowfocusonjustthetestsofthatcorefunctionality,soyouwillnotbedistractedbyalltheotherfailingtests.Youcansimplyclickonatestordescribetofilterthemout:
TheYeomanshortcutGettingJasmineupandrunningisquiteahassle,especiallyifyouhaveneverdoneitbefore.Thereisatoolthatwillsetupthissortofprojectforyou,calledYeoman(http://yeoman.io/).Yeomanhasallkindsofprojecttemplatesthatitcaninstallforyouautomatically(includingAngular.js).ThedownsidetousingYeomanwithJasmineisthatitusesBowerinsteadofnpm.Bowerisapackagemanagerthatcanbeinstalledthroughnpm(yes,youneedapackagemanagertoinstallapackagemanager).Bowerdoesacoupleofthingsbetterthannpm,mostnotablykeepingtrackofdifferentversionsofthesamepackagethatareusedbydifferentpackages.However,wehavechosennpmforthisprojectandwedonotwantyetanotherpackagemanager,anothertool,oranotherdependency.YoucanchoosetoinstallBower,butifyoudonot,Yeomanwillsimplysetuptheproject,butfailtoactuallyinstallJasmine,whichyoucantheninstallmanually.ItisbesttoinstallbothBowerandYeomangloballysoyoucanusethemdirectlyfromthecommandline.Usingthe-gswitchwithnpmwillinstallpackagesgloballyasopposedtoonlyinyourcurrentproject.Thedownsidetothisisthatitisimpossibletodonpminstallonglobalpackages:
[optional]npminstall-gbower
npminstall-gyo
npminstall-ggenerator-jasmine
yojasmine
IfyoudonothaveBowerinstalled,youwillonlygetthetestfolderwithanindex.htmlandaspecfolder.YoucaninstallJasminethroughnpmandchangethebower_componentsreferenceinindex.htmlwithnode_modulesmanually.Additionally,Yeomanwillcreate.yo-rc.json,whichyoucandeleteifyouarejustusingYeomanforthisone-timesetup.ThisisactuallywhatIdomyselfwhenIneedtosetupJasmine.
WhyJasmine?NowthatwehaveseensomeJavaScriptunittestsinaction,letustakeastepback.WhywouldweuseJasmineandarethereanyalternatives?JasmineworkswellwithKarma,atestrunnerwewilluselaterinthischapter.KarmawascreatedbytheAngular.jsteam,andthusAngular.js,Jasmine,andKarmaareaprettygoodmatch.
YoudoneedtorealizethatJasmineisnotyouronlychoice,butsinceweareworkingwithAngular.js,itisarecommendedchoice.OthertestinglibrariesincludeQUnit,usedbythejQueryteam,Unit.js,Intern,andMocha.Thereareaprettymuchagazillionunittestinglibrariesouttherethough,buttheseonesareprettypopularchoices.
Idonotwishtoconfuseyou,butyoushouldbeawarethatsomeoftheselibrarieshaveabstractedjustabouteverything.Forexample,whenyouuseMocha,youcanusedifferentassertionlibraries,suchasshould.js,expect.js,orchai.Likewise,mockingandspiesarenotapartofalltestinglibraries,requiringadditionallibraries,suchasSinonJS.Suchanapproachhasprosandcons.Obviously,knowingwhichlibrariestoinstallandhowtoconfigurethemcanbeprettyconfusing,especiallyifyouhaveneverusedatestingframeworkbefore.Ontheotherhand,youcanconfigureeverythingspecificallytoyourneedsandpreferences.AgreatexampleofwhatsomelibrariescanandcannotdoisonthehomepageofIntern,https://theintern.github.io/.ThehomepagehasaWhatcanInterndothatotherscan't?section.Youmaynoticenotevenalltestingframeworkscanperformunittests(suchasKarma).Soyouwillneedaunittestinglibraryforyourunittestinglibrary.WhichwillmakesenseoncewegettothepartaboutKarma.
AswithmostthingsJavaScript,pickingyourtestingframeworkcanendupininstallingmultiplelibrariesthatalldoaveryspecializedthingandsometimeshavesomeoverlap.Jasminejusthasitallandrequiresnoextralibraries.Ultimately,alltestinglibrariesdothesame:testyourJavaScript.
TestingthewebshopLet'scontinuetoaddsometestsforourwebshop.Thefirstthingweshouldaskourselvesiswhichfilesweshouldtestinthefirstplace.Fornow,themostobviousfileistheshoppingcartmodule.Whenwelookattheshoppingcartmodule,itseemslikeitisprettydifficulttotestthisfile.Anobvioustestwouldbetoaddaproductandcheckwhetherthepriceiscorrectlyupdated.However,theLineobjectisprivatetothecontrollerandthecontroller$scoperepresentstheentirecart.Wehavetwooptionshere:eitherwefindoutifandhowwecangetaccesstothe$scopeobjectinourunittestsorwecreateanadditionalfilethathastheshoppingcartobjectandthatwecanreuseinourcontroller(orevennon-Angular.jsprojects).Youcanalreadyseethatweareforcedtothinkaboutourcodeinacertainway,becauseweneedtouseourcodeintheapplicationwherewereallyneedit.Also,inourunittests,weneedtowritemoregenericcodethanwemightdootherwise.
Luckily,Angular.jshassomemeanstotestcontrollers.Wewillneedanadditionalpackagethough,angular-mocks:
npminstallangular-mocks--save-dev
Next,weneedtoaddAngular.js,angular-mocks,andourownshopping-cart.jsscripttothetestpageorourtestwillnotbeabletomakeuseofthem:
<!--includesourcefileshere...-->
<scriptsrc="../node_modules/angular/angular.js"></script>
<scriptsrc="../node_modules/angular-mocks/angular-mocks.js"></script>
<scriptsrc="../scripts/repository.js"></script>
<scriptsrc="../scripts/shopping-cart.js"></script>
<!--includespecfileshere...-->
<scriptsrc="spec/test.js"></script>
Afterthat,wecanwriteatestusingthecontroller:
describe('webapptests',function(){
describe('shoppingcartcontroller',function(){
beforeEach(module('shopApp'));
var$controller;
beforeEach(inject(function(_$controller_){
$controller=_$controller_;
}));
it('shouldupdatethelinesubtotalwhenaproductisadded',function(){
var$scope={};
varcontroller=$controller('shoppingCartController',{$scope:$scope});
$scope.lines[0].number=2;
expect($scope.lines[0].subTotal()).toBe('111.98');
});
});
});
Thereisquitealotgoingoninthosefewlinesofcode,solet'sbreakitdown.Therealtrickisinthemodule(shortforangular.mock.module)andinject(shortforangular.mock.inject)functionsthatcomefromangular-mock.Themodulefunctionsimplyregistersourmoduleconfigurationcode,soitcanbeusedbyAngular.jsand,specifically,theinjectfunction.Theinjectfunctionislessstraightforward.Itwrapsafunctionintoanotherfunction,whichcanbeinjected.Theobjecttobeinjectedisthenameoftheparameter,whichiskindofaproblemwhenyouareminifyingyourcode.
Anyway,wecannameourlocalvariable$controllerandstillpassin$controllertotheinjectfunctionbywrappingitinunderscores,becauseinjectignorestheunderscores.Thisbehaviorwasspecificallydesignedforthispurpose.Thinkofitwhatyouwant,thisishowitis.So,wecannowcallourcontrollerandpassour$scopeobjecttoit,whichwecanthenuseinourtests.
Wecannowcreateasecondtestusingthesamecontroller:
it('shouldupdatetheordertotalwhenaproductisadded',function(){
var$scope={};
controller('shoppingCartController',{$scope:$scope});
$scope.lines[0].number=2;
expect($scope.total()).toBe('131.96');
});
Inhindsight,Iamnotveryhappywiththechoicetoreturnstringsinsteadofnumbers.WeusedthetoFixed(2)functiontoroundtotwodecimalpoints,butthatfunctionreturnsastring.Inshopping-cart.js,IchangedthesubTotalfunction:
Line.prototype.subTotal=function(){
return+(this.product.price*this.number).toFixed(2);
}
Ifwesavethischange(+wasaddedtoconverttoanumber)andrunourtestsagain,wegetthefollowingerror:Expected111.98tobe'111.98'.Thatmakessense:itexpectedanumbertobeastring.Wecannowdotwothings,fixthecodesoit
returnsastringagain(afterall,wemayhavebrokensomeotheruntestedfunctionality)orchangeourtestbecauseweknowthisisnowthenewexpectedbehavior.Let'schangethetestbecausethisisreallywhatwewant:
expect($scope.lines[0].subTotal()).toBe(111.98);
Iamgoingtochangethetotalfunctioninthesameway.Wecanremovea+signandthenaddonetocasttheresult.Ofcourse,weneedtofixthetestaswell:
$scope.total=function(){
varsum=0;
$scope.lines.forEach(function(l){
//Removeaplusonthenextline.
sum+=l.subTotal();
});
return+sum.toFixed(2);
};
//Andinthetestcheckforanumber.
expect($scope.total()).toBe(131.96);
SpeakingoftoFixedandroundingerrors,itisriskytocheckafloatingpointnumeric(whichJavaScriptuses)toanexactnumber.JasminehasatoBeCloseTofunctiontocheckforapproximatevalues.Inthisparticularcase,wedowanttocheckforanexactnumber,becausewedonotwantanyroundingwhatsoever.ThisistheexpectedbehaviorofthesubTotalandtotalfunctionsandwehavenowtestedthattowork.
Let'saddathirdtest:
it('shouldupdatetheordertotalwhenaproductisremoved',function(){
var$scope={};
controller('shoppingCartController',{$scope:$scope});
$scope.removeLine($scope.lines[0]);
expect($scope.total()).toBe(19.98);
});
YoushouldnowgetawarmfeelinginsidethatiscausedbyknowingthatyourJavaScriptcodecanbemodifiedandanyerrorsarecaughtbyyourtests.Wearemissingacoupleofteststhough.Forexample,whathappenswhenweremovealinefrom$scope.linesdirectly?Whathappenswhenwechangetheproductofaline?Whathappenswhenwechangethepriceofaproduct?AndsincethisisJavaScript,whathappenswhenweoverwriteorremovesomepropertyorpassinavalueofthewrongtypesomewhere(likesettingline.numbertofoo)?
Onewordofadvice:thisisJavaScriptandpeoplecanmessupyourcodein
waysyoucannotevenimagine.Youcannotpossiblytestwhatwillhappenifpeopleoverwriteorremoveproperties,sodonotevenbothertrying.So,whatthenwhenpeoplesetnumbertofoooranegativenumber?Personally,Iwouldnottestsuchascenarioasitisimproperuseofthecode.Itisreallyhardtotestimpropercode,aspeoplecanreallyabuseit.Youcan,however,testwhatwillhappeninsomecasesthatdomakesomesense.Thinkabouttestingwithnumbersettoundefined,null,or2.Itisreallyuptoyouhowfaryouwanttogowiththis.Myadvice:donotgotoofar.Justdocumentandtestthecasesyouwishtosupport.Thatway,youknowyourcodewillworkwhenitisusedasintendedandotherpeoplecanreadhowyouintendedit.
Atthispoint,weareabletotestoursoftware,whichisawesome,butrunningourtestsrequiresustoopenupawebpagemanuallyandchecktheresults.Ifwewanttotestourcodeondifferentbrowsers,thiscanbequiteannoying.ThisiswhereKarmacomesin.
RunningtestswithKarmaKarma,asmentioned,isatestrunner(anditisspectacular,accordingtotheKarmateam).Soatestrunnerreallydoeswhatthenameimplies:itrunstests.So,insteadofhavingtorefreshyourtestpage,Karmajustrunsthemforyou.Karmacanbeeasilyconfiguredtorunyourtestsonmultiplebrowsers,suchasIE(orsomespecificversion),Edge,Firefox,Chrome,Safari,andOpera.Afterrunningyourtests,Karmacangeneratereportsonyourtestsandyourcodecoverage.Youcansetminimumcoveragethresholdsandmakeyourtestrunsfailwhenyourcodeisnotsufficientlycovered.Ontopofthat,Karmaisalayerofabstraction.Youcanconfigureyourtestframework,suchasJasmine,andthenswitchtosomethingelse,suchasMocha,andeverythingwillstillwork.Youonlyhavetochangeyourtests,ofcourse,andasinglelineinyourKarmaconfiguration.
InstallationFirstthough,let'sinstallKarma.WearegoingtoinstallKarmausingnpm.Karmabasicallyhastwocomponents:Karmaitself,whichusesNode.js,andtheKarmaCommandLineInterface(CLI).WearegoingtoinstallKarmainourprojectfolderandwecaninstalltheCLIgloballyforeasyaccessfromthecommandline:
npminstallkarma--save-dev
npminstallkarma-cli-g
Optionally,youcanuseYeoman,butwewillnotusethedefaultsfornow,sothatisprettyuselessatthispoint.However,ifyoureallywantto,youcan:
npminstallgenerator-karma-g
yokarma
npminstallkarma-cli-g
Karmaisallaboutconfiguration.NowthatwehavetheKarmaCLItoolinstalled,wecaneasilygenerateaconfigurationfile.Justusekarmainitandanswerthequestionsthatcomeup,prettymuchthesameasnpminit.Whereyouwanttoputthefileisuptoyou;youcanputitinyourtestfolder,butputtingitintherootfoldermakessensetoo.SinceYeomanputsitinatestfolder,Iamgoingtofollowthatexample:
cdtest
karmainit
Whichtestingframeworkdoyouwanttouse?
Presstabtolistpossibleoptions.Entertomovetothenextquestion.
>jasmine
DoyouwanttouseRequire.js?
ThiswilladdRequire.jsplugin.
Presstabtolistpossibleoptions.Entertomovetothenextquestion.
>no
Doyouwanttocaptureanybrowsersautomatically?
Presstabtolistpossibleoptions.Enteremptystringtomovetothenextquestion.
>Chrome
>
Whatisthelocationofyoursourceandtestfiles?
Youcanuseglobpatterns,eg."js/*.js"or"test/**/*Spec.js".
Enteremptystringtomovetothenextquestion.
>../node_modules/angular/angular.js
>../node_modules/angular-mocks/angular-mocks.js
>spec/*.js
>../scripts/*.js
>
Shouldanyofthefilesincludedbythepreviouspatternsbeexcluded?
Youcanuseglobpatterns,eg."**/*.swp".
Enteremptystringtomovetothenextquestion.
>
DoyouwantKarmatowatchallthefilesandrunthetestsonchange?
Presstabtolistpossibleoptions.
>yes
Configfilegeneratedat"C:\Users\sander.rossel\Desktop\ci-book\Chapter5\Code\test\karma.conf.js".
Now,ifyouopenthegeneratedkarma.conf.jsfile,whichisjustaJavaScriptfile,andremoveallthecommentsandwhitespace,itshouldlookasfollows:
module.exports=function(config){
config.set({
basePath:'',
frameworks:['jasmine'],
files:[
'../node_modules/angular/angular.js',
'../node_modules/angular-mocks/angular-mocks.js',
'spec/*.js',
'../scripts/*.js'
],
exclude:[],
preprocessors:{},
reporters:['progress'],
port:9876,
colors:true,
logLevel:config.LOG_INFO,
autoWatch:true,
browsers:['Chrome'],
singleRun:false,
concurrency:Infinity
})
}
Themodule.exportsisaNode.jsthingandmeansthatwhenthecurrentmoduleisloaded(withrequire,somethingwewillseeinlaterNode.jsmodules),module.exportsisexposedtoNode.js.Donotworryaboutmodule.exportsorrequirefornow;Ijustwantedtopointitoutincaseyouwerewonderingaboutit.Theconfigurationitselfisprettystraightforward,especiallywithallthecomments.Fornow,wewillleaveitasitisandrunourfirsttestswiththecurrentconfiguration.NotethatyouwillneedChrometorunthisexample(wewilluseotherbrowsersinaminute).Torunyourtests,simplyusethekarmastartcommand.ThiswillstartupaNode.jsserver,openupanewbrowserwindow,andrunyourtests.Karmawillpickupyourkarma.conf.jsfileautomatically.Alternatively,ifyouareinadifferentfolderthanyourKarmaconfigfileorifyouhavemultipleconfigs,youcanspecifyyourfileusingkarmastart
path_to_file/my_karma_conf.js:
Andthereyouhaveit.Oneofourtestsisstillfailing!Luckily,weplannedthis.So,nowisthetimetofixthisfailingtest.Fixithoweveryoulike;Ihaveoptedforexpect(false).toBe(false);.Now,herecomesthegoodpart.Watchwhathappenswhenyousavethefile:
Withoutdoinganythingbutsavingthefile,yourtestsarerunandyougetimmediatefeedback!Thatisawesome!Thesamehappenswhenyoueditasourcefile.Seewhathappenswhenyouremovethat+symbolfromthesubTotalfunction.YourtestsarerunimmediatelyandnotonlyisyourtestforsubTotalfailing,butalsothatfortotalandremove,becausetheydon'tgetstringinputdatafromsubTotalinsteadofthenumbertheywereexpecting.
Bytheway,ifyouwishtoquitKarmainyourconsolewithoutactuallyclosingyourconsole,youcanuseCtrl+C(onWindows)anditwillaskyouifyouwanttoterminatethebatchjob,whichyouwant.
KarmapluginsKarmaalonedoesnotdoallthatmuch.Youneedpluginstogetanythingdone.Infact,ifyoucheckyourdevdependencies,youwillseeKarmaalreadyhassomepluginsinstalled:
"devDependencies":{
"angular-mocks":"^1.6.1",
"jasmine":"^2.5.3",
"karma":"^1.4.1",
"karma-chrome-launcher":"^2.0.0",
"karma-jasmine":"^1.1.0"
}
TheChromelauncherandJasmineplugincomewithKarmabydefault.AtleasttheChromelaunchermakessense,asKarmawasdevelopedforAngular.js,whichwasdevelopedbyGoogle,whoalsomadeChrome.Googleownsyounow.Inthenextpart,wearegoingtoinstallandusesomeplugins.
BrowserlaunchersMaybeyoudonothaveChromeinstalledandyouweretryingtomakethisworkwithIEorFirefox.Unfortunately,whenyouconfiguredIEorFirefoxinsteadofChrome,youweremetwithanerroruponkarmastart.TheerroryouseeisCannotloadbrowser"Firefox":itisnotregistered!Perhapsyouaremissingsomeplugin?.Well,perhapsyouaremissingapluginindeed,solet'sinstallit:
npminstallkarma-firefox-launcher--save-dev
Now,youcanchangebrowsersinyourKarmaconfigurationtoFirefox.IfyourunKarmanow,youwillseeitusesFirefoxinsteadofChrome.Youmayneedtorestartyourconsolebeforethisworks.
YoucaninstalllaunchersforIE,Edge,andotherbrowsersaswell:
npminstallkarma-ie-launcher--save-dev
npminstallkarma-edge-launcher--save-dev
npminstallkarma-safari-launcher--save-dev
npminstallkarma-opera-launcher--save-dev
Ofcourse,youwillneverbeabletoruntheSafarilauncheronanythingotherthananApplecomputerandtheIEandEdgelaunchersonanythingotherthanaWindowscomputer.Iamprettysurethereareotherbrowserlaunchers,butyougettheidea.
WhenyouconfigureKarmatorunwithIE,somethingfunnyhappens.Yourtestsfail.Apparently,thereisasyntaxerrorinourrepository.jsfile.Weusedthelatest,greatestES2015arrowsyntaxandIEdoesnotsupportthat(C#hasuseditforalmost10years,butahwell...also,IE).Thisisexactlythereasonwhyyoushouldtestonmultiplebrowsers:youneverknowwhichbrowserssupportwhichfunctionality.Youcaneasilytestmultiplebrowsers.
Youmayhavenoticedthatthebrowseroptionstakeanarray,soyoucaneasilyconfiguremultiplebrowsers:
browsers:['Chrome','Firefox','Edge'],
BacktoIEforamoment.Personally,IthinkIEreallyneedstogo.Itisahorriblebrowser,impossibletoworkwith,anditdoeseverythingjustalittlebitdifferentthanotherbrowsers.Especiallytheoldversionsareaproblem.We,developers,knowweshouldnotuseIEforanythingotherthandownloadinganotherbrowser,butunfortunately,ourcustomersjustcannotgetenoughofIEand,preferably,IE8orevenworse.Luckilyforus,IEhasabuilt-intimemachinethatletsyoutestagainstaspecificversionofIE(atleastitdoesthatright).InKarma,youcancreateacustombrowserthatletsyouprovidesomeflagsforyourinstalledbrowsers,allowingyoutotestagainstaspecificversionofIE.
Unfortunately,runningagainstIE8isabitofahasslewithnewerversionsofKarma.SomethingtodowithsocketsnotbeingsupportedbyIE8andlower.IE9stillworksoutoftheboxthough,andIE8wouldworkthesameifyougotyourIEsettingsright.InyourKarmaconfiguration,addcustomLauncherandaddittothebrowsers:
browsers:['IE','IE9'],
customLaunchers:{
IE9:{
base:'IE',
'x-ua-compatible':'IE=EmulateIE9'
}
},
Ourtestsarestillfailingthough,butthatcanbefixedbyremovingtheES2015codeandreplacingitwithsomegoodoldJavaScript.Theonlyfilethatisaffectedisrepository.js.YoucanchangegetProductandsearchtolookasfollows:
getProduct:function(id){
returnproducts.filter(function(p){
returnp.id===id;
})[0];
},
search:function(q){
if(q==null){
return[];
}else{
returnproducts.filter(function(p){
returnp.name.toLowerCase().indexOf(q.toLowerCase())>=0;
});
}
}
Onceyousavethefile,yourtestsshouldberunautomaticallyandyoushouldseetheyarepassingnow.
CodecoverageKarmacanalsogiveyoucodecoveragereports.Codecoverageindicateshowmuchofyourcodeistested.Forexample,ifyouhavetwofunctions,each10lineslong,andyoutestone,youshouldhaveacodecoverageof50%.Codecoverageisalittlemoresophisticatedthanthat,aswewillseeinamoment.
Beforewestart,wewillneedtoinstallthekarma-coverageplugin:
npminstallkarma-coverage--save-dev
Afterthat,weneedtochangeourconfiguration,soKarmawillrunthecoverageplugin.Wewillneedtoindicatethecoveragepreprocessor,thecoverageReporter,andthetypeofreporter:
preprocessors:{
'../scripts/*.js':['coverage']
},
reporters:['progress','coverage'],
coverageReporter:{
reporters:[
{type:'html',subdir:'html'}
],
dir:'coverage/',
},
So,allofourscriptsinthescriptsfolderarecheckedforcoverage.Wealreadyhadtheprogressreporter,butweaddedthecoveragereporter.Wecanthencustomizethecoveragereporter.Wewantthehtmlreporter,whichshouldsaveitsresultstothehtmlsubdirectory.Allcoveragereportsarewrittentothecoveragedirectory(relativetotheconfigfile).RunKarmaagainandyouwillnowfindtheHTMLreportintest/coverage/html.Youcanopenindex.htmlanditshouldshowyouthecoverageofyourfiles:
Asyoucansee,thecoverageofyourstatements,branches,functions,andlinesaretrackedseparately.Ourcoverageisnotallthatgood,butthatmakessense,asweonlyreallytestedonefile,whichhas100%coverage.Sincewecalltheangular.moduleandcontrollerfunctionsinourtests,theyareconsideredtestedinallourfiles(thatiswhyallfileshaveatleastsomecoverage).Whatisreallyniceisthatyoucanclickonafileandyoucanseeeverythingthatwasnotcoveredbyyourtests:
Asyoucansee,thereturnstatementwasexecutedonceduringourtests.Youcanevenseehowoftenaparticularlinewasexecuted.Thelinereturnproducts.filter...wasexecutedsixtimesandthecallbackthirtytimes.Executingafunctionmoreoftendoesnotincreaseyourcoveragerating.Yourreportisoverwrittenoneachtestrun.
Itispossibletoenforceaminimumcoverage.Wecanaddthisinourconfiguration.Whenthecoveragefallsbelowacertainpercentage,yourtestswillfail.YoucandothisinyourcoverageReporter:
coverageReporter:{
reporters:[
{type:'html',subdir:'html'}
],
dir:'coverage/',
check:{
global:{
statements:50,
branches:50,
functions:50,
lines:50,
excludes:['../scripts/repository.js']
}
}
},
Withthecheckobject,youcansetyourglobalthreshold.Allfilestogethershouldhaveatleast50%coverage.Thiswillsucceedifonefilehas100%coverage
whileanotherhas0%.Weareexcludingrepository.js,becauseincludingitwouldmakeourtestsfail,asitsetsourbranchcoverageto0%.
Additionally,youcansetthresholdsonaperfilebasis.Forexample,wewantatleast50%ofourcodetested,butatleast10%ofeachfileshouldbetestedaswell.Ifwehadafilewith100%coverageandafilewith0%coverage,ourtestswouldnowfail:eveniftheglobalthresholdof50%ismet,the10%perfileminimumisnot.Youcanevensetthresholdsforspecificfiles.Forexample,ourshopping-cart.jsfileisfullytestedandwewanttokeepitthatway:
check:{
global:{
statements:50,
branches:50,
functions:50,
lines:50,
excludes:['../scripts/repository.js']
},
each:{
lines:10,
excludes:[],
overrides:{
'../scripts/shopping-cart.js':{
statements:100
}
}
}
}
Ultimately,wewanttobeabletopublishthesecoveragereports.TheHTMLreportisawesomeforpeopletoread,butbuildsystems,suchasJenkins,willhaveahardtimeparsingit.Jenkins,specifically,usesacoveragereportintheso-calledCoberturaformat(whichisreallyjustaspecificXMLfile).WhatissoawesomeaboutthiscoveragepluginisthattheCoberturareportformatisbuilt-in.Wecansimplyaddittoourreporters:
reporters:[
{type:'html',subdir:'html'},
{type:'cobertura',subdir:'cobertura'}
],
Itreallyisthatsimple.Othersupportedformatsarelcov(LinuxCOVerage:alsoincludesHTML),lcov-only,text,text-summary,teamcity,json,json-summary,in-memory,andnone.
Youcanfindthisandmoreonthekarma-coverageGitHubrepository(https://github.com/karma-runner/karma-coverage)inthedocsfolder.Itisreallywelldocumented,
soIsuggestyoucheckitout.
Bytheway,sincethecoveragereportsaregeneratedeverytimeyourunyourtests,youprobablydonotwanttoincludetheminyourGitrepository.Besuretoaddalineinyour.gitignorefileandignoretheentirecoveragefolder:
**/coverage/**
Awordofcaution:codecoveragetellsyouhowmuchofyourcodeisexecutedbytests.Itdoesnottellyouwhetheryourtestsareofhighquality.Intheory,youcouldhaveacodecoverageof100%thatdoesnottestanything!Forexample,Ionceworkedonawebapplicationthatwasthoroughlytested,orsoIthought.Uponcloserinspectionofthecode,Ilearnedthatanimportantpartofthesoftwarecouldnotbemockedandalwaysreturnednullduringtests.Theresultwasthateverytestthrewanullreferenceexception.Thetestsallsucceeded,becausealltheydidwastestwhetheranullreferenceexceptionwasthrown...Idiedalittleinsidethatday.Evenifyourtestsmakesense,asinglefunctionshouldoftenhavemorethanonetesttotestdifferentscenariosandedgecases.Codecoveragedoesnotexceed100%,butwell-testedcodeshouldhaveatleast400%.Still,ifyourtestsmakesense,codecoverageisagoodindicationofhowwellyourcodeistested.
JUnitreporterAsmallbutveryusefulpluginiskarma-junit-reporter.WhenwearegoingtotestourcodeusingJenkins,wewanttoknowwhichtestsfailedandwhy.Ourcurrentprogressreporter(whichwritestheresultstotheconsole)isnotgoingtocutit.JenkinsmakesuseoftheJUnitstyleXMLtopublishtestresults(nottobeconfusedwithcoverageresults).Wecansimplyinstallthepluginandthenaddittoourreporters:
npminstallkarma-junit-reporter--save-dev
Andinourconfigurationfile,wecanaddittoreporters:
reporters:['progress','junit','coverage'],
Ifyourunyourtestsnow,youshouldgetareportforeachbrowserandOS,forexample,TESTS-IE_11.0.0_(Windows_10_0.0.0).xml.Ofcourse,theJUnitreportercanbeconfiguredaswell;simplyaddthefollowingconfigurationsomewhereinyourKarmaconfigfile:
junitReporter:{
outputDir:'junit',
suite:'WebShop',
useBrowserName:true
},
YourJUnitreportsarenowsavedinthesubdirectoryjunitandsomewhereinthefile,itwillsaypackage="WebShop".
Again,thedocumentationisprettyspoton,soyoushouldcheckitout(https://github.com/karma-runner/karma-junit-reporter).
Likethecoveragereports,youdonotwantthisreportinGit.Wecansimplyregeneratethereportanytimewewantbyrunningourtests.Addthefollowinglinetoyour.gitignorefile:
**/junit/**
RunningMochaandChaiwithKarmaAsIsaidearlier,Karmaisanabstraction,whichmakesitpossibletorundifferenttestsfromdifferenttestframeworks.WhileIdonotwanttodwellonthisfortoolong,learningasingletestframeworkisenoughafterall,Idowishtoquicklyshowyouhowthisisdone.
So,let'sinstallMocha(https://mochajs.org/)andthepopularChai(http://chaijs.com/)assertionlibrary:
npminstallmocha--save-dev
npminstallchai--save-dev
Now,createafilecalledmocha-tests.jsinsideyourtestfolder.Wearegoingtotestrepository.searchinmocha-tests.js:
describe('webapptests',function(){
describe('repository',function(){
it('shouldsearchproductswith"fantas"',function(){
varproducts=repository.search('fantas');
products.should.have.lengthOf(3);
//Alternatively,Chaisupportsthefollowingsyntax:
//chai.expect(products).to.have.lengthOf(3);
//chai.assert.lengthOf(products,3);
});
});
});
LikeJasmine,Mochacanbesetuptorunonyourbrowser.However,sincewearefocusingonKarmaandnotonMocha,weareonlygoingtorunMochainKarma.YoucanreadhowtosetupyourMochaforyourbrowserontheMochahomepage.NowthatwehaveMochaandChai,wecaninstalltheKarmaplugins:
npminstallkarma-mocha--save-dev
npminstallkarma-chai--save-dev
TheKarmaconfigurationissosimple,IamnotsureifIevenneedtoshowyou:
frameworks:['jasmine','mocha','chai'],
files:[
'mocha-tests.js',
'../node_modules/angular/angular.js',
'../node_modules/angular-mocks/angular-mocks.js',
'spec/*.js',
'../scripts/*.js'
],
Ihaveaddedmochaandchaitotheframeworksandmocha-tests.jstothefiles.ThatisallyouneedtodoandKarmawillnowalsorunyourMochatests.Youcanverifybycheckingyourcodecoveragereport.Youwillseethatrepository.searchisnowtested.
Youcandownloadotherframeworks,suchasQUnit,andprettymuchconfigureitlikewehaveconfiguredJasmineandMocha.Basically,ifyouhavesometestframeworkyouwanttousewithKarma,simplyGoogleforkarmayour-test-frameworkandyouwillprobablyfindapluginyoucanuse.Asthisexamplealsoshows,youcanusemultipleframeworkssidebyside.Thatmaycomeinhandywhenyoudecidetoswitchframeworks(maybeyourcurrentframeworkisnotactivelysupportedanymore)orwhenanotherframeworkcantestsomepartofyourcodebetterthanyourcurrentframework.
End-To-EndtestingwithSeleniumThenextthingwearegoingtodoisEnd-To-End(E2E)testingusingourbrowser-ouractualbrowser,asifitwasoperatedbyahuman,butfullyautomated.ThepopularwebUItestframeworkSelenium(http://www.seleniumhq.org/)willdothisforyou.Asyoucanimagine,thisisnoeasytask.Seleniumneedstointerfacewithdifferentbrowsers,differentlanguages,anddifferentframeworksanditisallsetupsofuturebrowsers,languages,andframeworkscanbeimplemented.Assuch,itcanbeabitofapaintosetup.Thereareafewmovingpartsyouneedtoinstall,eitheronyourcomputerorinyourproject,andtomakethingsmorecomplicated,thosepartshavedifferentversionswithdifferentnames.Don'tpanicthough:throughouttheremainderofthischapter,allwillberevealed.
Seleniumhasitsownlanguage,Selenese,inwhichyoucanwritetestsdirectlyusingtheSeleniumIDE,anadd-onforFirefox.We,however,arenotgoingtodothat.YoumaydownloadtheIDE(http://www.seleniumhq.org/projects/ide/)andplayaroundwithit,butourfocuswillbeonwritingandautomatingSeleniumtestsinJavaScript.SeleniumcurrentlyhasclientAPIsforJava,C#,Ruby,Python,andJavaScipt.TheclientAPIscommunicatewiththeSeleniumWebDriver,thecentralcomponentinSelenium.TheWebDrivercancommunicatewithbrowsersdirectly.YoumaycomeacrossSeleniumRemoteControl,orSeleniumRC,whichisadeprecatedtechnologythatisreplacedbySeleniumWebDriver.
Let'sstartwiththemostbasicSeleniumexamplethatisfairlyeasytoreproduce.First,wemustinstalltheSeleniumWebDriverAPIforJavaScript.Ofcourse,wecandothisusingnpm:
npminstallselenium-webdriver--save-dev
Next,wecanwriteafilethatwilldosomeautomatedworkforus.Let'skeepitsimpleanddoaGooglesearch.Putyourfileinthetestfolderandnameitselenium-tests.js.Putthefollowingcodeinsideit:
varwebdriver=require('selenium-webdriver');
varby=webdriver.By;
varuntil=webdriver.until;
vardriver=newwebdriver.Builder()
.forBrowser('firefox')
.build();
driver.get('http://www.google.com/ncr');
driver.findElement(by.name('q')).sendKeys('selenium');
driver.findElement(by.name('btnG')).click();
driver.wait(until.titleIs('selenium-GoogleSearch'),5000);
driver.quit();
Hereiswhereweuserequireourselves.Inthiscase,requirechecksthenode_modulesfolderforafoldercalledselenium-webdriver.Whenitfindsthatfolder,itwilllookforindex.jsbydefault.Theindex.jsfileshouldhavemodule.exports,likeourkarma.conf.jsfile,orjustexports,anobjectwhichisloadedbyrequire.Otherthanthat,thetestreadsasifyouwerereadingregularEnglish,soIsupposeIdonothavetoexplainwhatthisdoes.IwillgetbacktothesyntaxandAPIinabitthough.Thegoogle.com/ncr,linkdoesnotredirecttoyourowncountrypage(ncrmeansnocountryredirect),sothetitleisfairlycertaintobeselenium-GoogleSearch.Now,gotoyourconsoleandrunthefileusingNode.js.WehavenotusedNode.jsbefore,butwehaveitinstalled.RunningafileinNode.jsisaseasyasnodeyourfile.js:
cdtest
nodeselenium-tests.js
Error:Thegeckodriver.exeexecutablecouldnotbefoundonthecurrentPATH.Pleasedownloadthelatestversionfromhttps://github.com/mozilla/geckodriver/releases/WebDriverandensureitcanbefoundonyourPATH.
Unfortunately,wegetanerrormessagesayingthatwearemissingthegeckodriver.ThegeckodriveristhedriverthatrunsFirefox.SeleniumWebDriverneedsawebdrivertocommunicatewithyourbrowser.Firefox,Chrome,IE,Edge,andSafariallhavetheirowndrivers.Youcanfindlinkstothepageswiththedriverdownloadsontheselenium-webdrivernpmpage,https://www.npmjs.com/package/selenium-webdriver.Foryourconvenience,IhavealsoincludedthewebdriversintheWebDriversfolderintherootfolderoftheci-bookGitrepository.Putthewebdriverssomewhereonyourcomputer,forexample,C:\WebDrivers,andaddareferencetothatfolderinyourPATHvariable,soSeleniumknowswhereitcanfindthedrivers.OnWindows,youcandothisbygoingtoControlPanel,thenSystemandSecurity,ifyouhavetheViewbyCategoryoptionon,andthenSystem.InSystem,gotoAdvancedSystemSettingsand,fromthere,gotoEnvironmentVariables…YoucannoweditthePATHvariableinyouruserorsystemvariables.AddanewpathandsetittoC:\WebDriversorwhereveryouputyourdrivers.Younowhavetorestartyourconsoleandtrynodeselenium-tests.jsagain.
Youwillnowseeyourbrowserstarting,searchingforselenium,andthenclosing.Oneproblemyouwillfindwithbrowsertestingisthatyouwillhavetowaitforpagestoloadandyouwillneverknowhowlongthingsaregoingtotake.Inthelinedriver.wait(until.titleIs('selenium-GoogleSearch'),5000);,weallowGoogletotakeupto5,000milliseconds(5seconds)tochangethetitle.Itusuallyloadsthenewpageinasecond,sometimes2,so5secondsisquitegenerous(but1secondissometimestooslow).Now,watchwhathappenswhenyouchangethetitleinyourscripttosomethingelse,suchassomethingelse.After5seconds,yougetanerror:
TimeoutError:Waitingfortitletobe"somethingelse"
Waittimedoutafter5001ms
Apparently,ourtestfailed.Seleniumnowleavesthebrowseropen(whichmakessense,sinceourcodeneverhitsdriver.quit();).ThereareafewthingstotakeintoaccountwhenworkingwithSeleniuminJavaScript,butIwillcomebacktothatinabit.
First,let'srunourexampleondifferentbrowsers.ToruntheexampleonChrome,simplygettheChromewebdriverandchange'firefox'to'chrome'intheselenium-tests.jsfile.IE,ofcourse,isbeingdifficultasalways.InordertorunIE,youneedtosetProtectedModeonoroffforallzones.GotoIEsettingsandthentheSecuritytab.TherearethezonesInternet,LocalIntranet,Trustedsites,andRestrictedsites.Oneachzone,thereisacheckboxEnableProtectedMode(requiresrestartingInternetExplorer).Eitherenableordisablethemall(Irecommendenablingthemall)andcloseIE.Also,thereseemstobeabug(orweirdintendedbehavior?)withthex64IEdriver,sobesuretousetheWin32(x86)one.Inyourscript,youcanuseforBrowser('ie');.TorunEdge,youneedtogetthecorrectEdgewebdriverandthenuse'MicrosoftEdge'inforBrowser.
Wearenowabletorunourlittleexampleonallmajorbrowsers(exceptSafari).Thenextstepistocreateactualtests.Hereiswherethetroublestarts.Karmastartsupabrowserandrunsyourteststhere.However,SeleniumneedstobeexecutedinNode.js.Thisshouldbepossible,sinceKarmaitselfisalsoexecutedinNode.js.However,Karmaisnottherecommendedtoolforthejob.TheAngular.jsteambuiltProtractor(http://www.protractortest.org)specificallyforE2EtestsusingSelenium.
RunningSeleniumtestswithProtractorSoyes,weneedyetanothertool.ItmakessensetousesomethingelseforE2Eteststhough.E2Etestsarenotunittests.Whileunittestsmustbeexecutedfasttogiveyouinstantresultsoneverychangeyoumake,E2Etestsareusuallyalotslowerandmoreintrusive(startingbrowsersandall).So,wewillbeusingProtractorinadditiontoKarma.Thegoodnewsisthatwecanautomateboth,sowedonotactuallyhavetoremembertorunbothtomakeabuild,butwewillgettothatinthenextchapter.
ProtractormakesuseofSeleniumServer,alsocalledSeleniumGrid.Youcandownloadtheserveryourself(http://www.seleniumhq.org/download/),butProtractorcanactuallydothisforus.SeleniumServermakesitpossibletocallbrowsersonremotemachines.ItalsohastheaddedadvantagethatyourSeleniumtestscanbeconfiguredinacentralplace,allowingyoutoruntestssimultaneouslyandondifferentbrowsers.ToinstallProtractor,wecanusenpm.Thiswillalsoinstallautilitytool,webdriver-manager:
npminstallprotractor--save-dev
npminstallprotractor-g
webdriver-managerupdate
webdriver-managerstart
[...]
21:13:08.723INFO-SeleniumServerisupandrunning
Thewebdriver-managerupdatecommanddownloadsSeleniumServeralongwiththeFirefoxandChromewebdriversusingcurl.Youcanreadexactlywhatitdoesinyourcommandwindow.Thewebdriver-managerstartcommandactuallystartstheserver.Ofcourse,assoonasyouterminatethebatchorclosethisconsolewindow,yourserverisclosedaswell.So,fortheremainderofthechapter,wewillneedasecondconsolewindowandletthisonebe.
Wecannowchangeourselenium-tests.jsscript,soitcontainsanactualJasminetest.SinceProtractorisdevelopedbytheAngular.jsteamandtherecommendedtestingframeworkforAngular.jsisJasmine,ProtractorusesJasminebydefault.ProtractoralsohasitsownAPI,soourtestisgoingtolookalittledifferentfrom
theonewehadbefore:
describe('Seleniumtests',function(){
varEC=protractor.ExpectedConditions;
it('shouldgoogle"selenium"',function(done){
browser.ignoreSynchronization=true;
browser.get('http://www.google.com/ncr');
element(by.name('q')).sendKeys('selenium');
element(by.name('btnG')).click().then(function(){
browser.wait(EC.titleIs('selenium-GoogleSearch',2500));
expect(browser.getTitle()).toBe('selenium-GoogleSearch');
done();
});
});
});
Youshouldimmediatelynoticeafewthings.Firstofall,youdonotneedtoinitializethewebdriveryourself.Wealsodonotspecifyabrowsertouse.Then,thereisthatweirdbrowser.ignoreSynchronization=true;line.ProtractorwasbuiltforAngular.jsand,bydefault,itwaitsforapagetoloadAngular.js.Sincegoogle.com,thewebsitewearerequesting,doesnotuseAngular.js,Protractorkeepswaitingforeverandeventuallytimesout.BysettingignoreSynchronizationtotrue,wetellProtractornottowaitforAngular.js.Last,theclickfunctionnowtakesacallbackasargument.BecauseJavaScriptexecutesyourbrowsercommandsasynchronously,weeitherneedtowaitforacertaincondition,likeinthepreviousexamplewithdriver.wait(until...)(replacedwithEC.titleIs(...)),ormakeuseofcallbacksorboth,likeinthisexample.Changingyourcodetoclickandthencheckingthetitlewillprobablybeafewsecondstooslowandmakeadifferencebetweenafailingandapassingtest:
element(by.name('btnG')).click();
//Thismayfailastheclickmaynotyetbeperformed.
expect(browser.getTitle()).toBe('selenium-GoogleSearch');
Simplywaitingforthetitletochangewouldhavesufficedinthisexample.However,Iwantedtoshowyoualittletrickwhenworkingwithcallbacks.Sincethebuttonclickisperformedasynchronously,thecodesimplycontinueswhilethebrowserisworking.Assuch,yourunittestwouldsimplyexitandthebuttonclickwouldnothaveanyeffectonyourtestassuch.Theclickfunctionreturnspromise,whichhasthethenfunction.Thethenfunctiontakesacallbackfunctionthatwillbeexecutedoncethebuttonisclicked.Butweshouldalsostopourtestfromexiting(andfailingbecausenoexpectorassertwasdone).Wecandothisbypassingthedoneargumenttoouritfunctionandcallingdonewheneverourtestisreallyfinished(inwhateverasynchronouscode):
it('shouldgoogle"selenium"',function(done){
//Somecodehere.
somethingAsync().then(function(){
//Morecodehere.
done();
});
//Willnotexituntildoneiscalled.
});
Hereisthethingthough:allofProtractor'sfunctionsreturnpromise,whichcanbefollowedupbythen.Forthesimplerscenarios,expectwillresolvepromiseforus.So,getTitle()returnsapromiseandnotastring,buttoBe('title')willstillevaluatetotrueorfalse.
Thereisonemorethingweneedtodobeforewecanrunourtest.Inyourtestfolder,createanewfileandnameitprotractor.conf.js.Putthefollowingconfigurationinthatfile:
exports.config={
framework:'jasmine',
seleniumAddress:'http://localhost:4444/wd/hub',
specs:['selenium-tests.js']
};
Wecannowfinallyrunourtest!MakesureyouhaveSeleniumServerrunningandthenusetheprotractoryour-config-file.jscommand:
cdtest
protractorprotractor.conf.js
Youwillnow,again,seeabrowseropeninganddoingwhatyouscriptedinyourtest.Additionally,youcanseetheconsolewindowthatisrunningyourserverloggingexactlywhatitdoes.ThecommandwindowrunningProtractorwillgiveyouyourtestresults:
ThedefaultbrowserforProtractorisChrome,butyoucaneasilyconfigureanotherbrowserinyourProtractorconfigurationfile.Simplyaddacapabilitiesobject:
capabilities:{
browserName:'internetexplorer'
},
YoucanalsosetupmultiplebrowsersusingthemultiCapabilitiesobject.YoucansetbothcapabilitiesandmultiCapabilities,butonewilloverwritetheother:
multiCapabilities:[{
browserName:'chrome'
},{
browserName:'MicrosoftEdge'
}],
Unfortunately,thereissomethingstrangegoingonwithFirefoxsupportatthemoment(monthsbeforeyouarereadingthis).SeleniumServerthrowsanerrorwhentryingtorunit.Thereisaworkaround,whichisrunningwithadirectconnection,bypassingSeleniumServer.Unfortunately,thisoptionisonlysupportedforChromeandFirefox.IfyouusedirectConnect,youshouldremoveseleniumAddress:
//seleniumAddress:'http://localhost:4444/wd/hub',
multiCapabilities:[{
browserName:'chrome'
},{
browserName:'firefox'
}],
directConnect:true,
SpeakingoftheseleniumAddresssetting,IfinditannoyingthatIhavetostartaSeleniumservereverytimeIwanttorunmytests.Runningaseparateserverisgreatwhensomethinggoeswrongandyouneedadditionalloggingbut,hopefully,youdonotneedthatallthatoften.Protractorcanrunourownserver,butyouneedaJARfilecontainingtheserver.YoucandownloadtheJARserverfileontheSeleniumwebsite(http://www.seleniumhq.org/download/).Now,youcansimplyputitanywhereonyourcomputerandreferenceitinyourconfigurationfile.Forthis,youneedtheseleniumServerJarandtheseleniumServerPortsetting.BesuretoremoveseleniumAddress:
seleniumServerJar:'selenium-server-standalone-3.1.0.jar',
seleniumServerPort:4444,
//seleniumAddress:'http://localhost:4444/wd/hub',
Inthatexample,IhaveputtheSeleniumserverJARfileinthesamefolderasmytests,butwecanplaceitanywhere,forexample,onthedesktop:
seleniumServerJar:'C:\\Users\\sander.rossel\\Desktop\\selenium-server-standalone-3.1.0.jar',
YoucannowsimplystartupProtractoranditwilltakecareoftheserverforyou.
TestingourwebsiteTimetoaddsometestsforourwebshop.ThisalsogivesusthechancetoseeabitmorefromtheProtractorAPI.Icannotexplainitallinthischapter,butwewillseeacoupleoffunctionsonelementsandlistsofelements.JustkeepinmindthateveryfunctioninProtractorreturnsapromiseandthatJasminewillresolveitforusautomatically.
Personally,IamnotafanofHTMLandCSS.Itjustdoesnotwork,sowhenitdoes,Iwouldliketotestthatthingsstayastheyare.HowaboutweaddatestthatcheckswhetherthetitlesoftheOtherpeoplebought...productsonourhomepageareproperlycutofftofitthethumbnails?ItimmediatelybecomesclearwhyIdonotlikeHTMLandCSS.Thereisnowaytocheckthis.Yourtextcanoverflowwhilethewidthofyourh3elementstaysthesame.Thereisalsonowaytogettheactualwidthofyourtext,atleastnotaneasyone.Itwouldhelpifyoucouldgettheactualtext,whichwouldbeTheGood,TheBadandThe...,butyoucanonlygetthecompletetext(withUglyattheend).So,theonlythingwecantestisactuallycheckingwhethertheCSSwedefinedisapplied.Notabadtest,mindyou,becauseCSSiseasilyoverwrittenorremoved,especiallywhenusingsomeglobalCSS,ifyouoryourcoworkersarenotcareful:
describe('homepage',function(){
it('shouldcutofflongtitles',function(){
browser.get('file:///C:/Users/sander.rossel/Desktop/ci-book/Chapter%205/Code/index.html');
browser.wait(EC.presenceOf($('h3'),1000));
varelems=element.all(by.css('h3'));
expect(elems.count()).toBe(3);
varelem=elems.get(0);
expect(elem.getCssValue('overflow')).toBe('hidden');
expect(elem.getCssValue('text-overflow')).toBe('ellipsis');
expect(elem.getCssValue('white-space')).toBe('nowrap');
});
});
Thefirstthingwedo,ofcourse,isbrowsetoourwebsite,which,inthiscase,isjustafileonourlocalsystem.Afterthat,wewaituntilthepageisactuallyloadedanddisplaystheh3elements.Usingelement.all(by.css('h3')),wecangetalltheelementsthatmeettheCSSselector'scriteria.Weexpecttogetthreeh3elements,sinceweshowthreeOtherpeoplebought...products.Onceweknow
wehavethreeelements,wecanchecktheCSSpropertiesforeitheroneofthem.Wecouldcheckallofthem,butcheckingoneofthemwillprobablybeenough.
Onceyoutrytorunthistest,youwillfindthatyourbrowsernavigatestodata:text/html,<html></html>,afterwhichyourtestfails.Protractorusesdata:text/html,<html></html>asaresetURL,whichisusedwhenProtractorloadsapage.BecausewecannotnavigatefromthedataURIschemetothefileprotocol,weneedtomakethisexplicit.WecandothisbyaddingtheonPreparefunctioninournavigationandchangingresetUrl:
onPrepare:function(){
browser.resetUrl='file://';
},
Whenwerunourtestsnow,westillgetanerror!ProtractorwaitsforAngular.jstoload,butcannotdothisonceweswitchtothefileprotocol.Wecandotwothings:eitheraddbrowser.ignoreSynchronization=true;toourtest(eitherinthetestitselforusingJasmine'sbeforeEachfunction)orwecanadditintheconfig,whichsoundslikeagoodplanbecauseallourtestswilleitherneeditornot.SinceonPrepareiscalledbeforeanytestsarerun,butafterJasminewasloaded,wecansimplyaddignoreSynchronizationinonPrepare:
onPrepare:function(){
browser.resetUrl='file://';
browser.ignoreSynchronization=true;
},
WecouldhaveaddedbothtoourtestorinthebeforeEach,,buthavingthesesortofsettingsintheconfigfilemakessense,astheyarenoweasilyturnedonoroffforallyourtestsatonce.InternetExplorerwillstopworkingnow,butwillstartworkingagainafterweaddabackend.
Upuntilthispoint,ourGoogletestandournewtestranfine.However,wearenowgoingtosetapropertyintheconfigthatwillmakeourGoogletestfail,sobesuretoremoveit.Havingthatlongpathtoyourpageisabitofapain.Whatisevenmoretroublesomeisthat,later,wearegoingtorunourwebshopusinganactualserverandwewanttobeabletoswitchfromoneURLtoanotherURLwithouthavingtochangeallourtests.TheconfigfilesupportsabaseUrlpropertythatwecanuse.Inthiscase,baseUrlissimplythepathtoourindex.htmlfile:
baseUrl:'file://C:/Users/sander.rossel/Desktop/ci-book/Chapter5/Code/',
SwitchingfromfiletoHTTPwillprobablystillrequireustochangeallourtests,becausewearegoingtoaddsomeroutingontheserver,butafterthat,wecaneasilychangebaseUrlfromlocalhosttoourtestserver.Ofcourse,thisalsomeansthatyourtestisnowgoingtogetarelativepage,soweneedtochangethebrowser.get();line:
browser.get('index.html');
Let'saddafewmoreteststoourhomepage.ThenexttestnavigatestoaproductbyclickingtheOtherpeoplebought...producttitle:
it('shouldnavigatetotheclickedproduct',function(){
browser.get('index.html');
browser.wait(EC.presenceOf($('[ng-controller=homeController]'),1000));
varproduct=element.all(by.css('h3')).get(1);
product.click();
browser.wait(EC.presenceOf($('[ng-controller=productController]'),2500));
vartitle=element(by.css('h2'));
expect(title.getText()).toBe('TheGood,TheBadandTheUgly');
});
Nothingwehavenotseenalready.Insteadofwaitingfortheh3tobecomevisiblewearewaitingforhomeControllertobecomevisible,whichisalittlesafer,asitreallyisuniqueforthehomepage.ThesamegoesfortheproductController:
Andlet'stestthesearchfunctionality:
it('shouldsearchforallproductscontaining"fanta"',function(){
browser.get('index.html');
browser.wait(EC.presenceOf($('[ng-controller=homeController]'),1000));
varinput=element(by.css('[ng-model=query]'));
input.sendKeys('fanta');
varsearchBtn=element.all(by.css('a.btn.btn-default')).get(0);
searchBtn.click();
browser.wait(EC.presenceOf($('[ng-controller=searchController]'),2500));
varresults=element.all(by.css('.thumbnail'));
expect(results.count()).toBe(3);
});
Again,itisallprettystraightforward.Asyoucansee,wearerelyingonthefirsta.btn.btn-defaultelementtobetheelementweneedinordertofindproductscontainingfanta.Ifsomeoneaddedanotheranchor/buttonlikethattothepage,ourtestmightfailwhilenothingreallybroke.WecouldaddaclassorIDtotheelement,whichwouldmakeeverythingalittlesafer:
<aid="search-btn"href="views\search.html?q={{query}}"class="btnbtn-default">
<spanclass="glyphiconglyphicon-search"></span>
</a>
Thetestwouldnowselectforsearch-btnofwhichonlyoneexistsonthepage:
varsearchBtn=element(by.css('#search-btn'));
Thetestexpectsthreeresultstobefound.Wecouldbealittlemorespecific.Wecouldtestthatoneormoreofthetitlesreallyarethetitleswewouldexpect,forexample.Wecansearchforelementswithinelementsbyusingtheelementfunctiononanotherelement:
varresults=element.all(by.css('.thumbnail'));
expect(results.count()).toBe(3);
varfirstElem=results.get(0);
varfirstCaption=firstElem.element(by.css('h3')).getText();
expect(firstCaption).toBe('FinalFantasyXV');
Evenbetter,weshouldloopthroughtheresultsandcheckthateachoftheresultshasfantainthetitle.Thisisalittletricky,becauseweneedtoextractallvaluesfromallpromisesandthendoourassertionsbeforereturning.Protractorhasacoupleofarray-likefunctions,suchasmap,filter,andeach.Inthiscase,wewillusemap,thentransformourelementstoabunchofgetText(),whichwillberesolvedtostringsinthethenfunction.Wecanthensimplyloopthroughourregulararrayofstringsandexpecteachofthemtocontainfanta:
varresults=element.all(by.css('.thumbnail'));
expect(results.count()).toBeLessThan(20);
results.map(function(elem){
returnelem.element(by.css('h3')).getText();
}).then(function(captions){
captions.forEach(function(caption){
expect(caption.toLowerCase().indexOf('fanta')).toBeGreaterThan(-1);
});
done();
});
Ihavechangedthecountassertiontobelessthan20.Ifwecheckallourelementsindividually,itdoesnotmatterhowmanyresultswegetback,butlet'spretendwehaveapagesizeof20.Thingsgetalittletrickywithallthesepromises,especiallysinceyourtestwillnotfailifnoassertionsweremade.Irecommendmakingyourtestsfailatleastonce,soyouknowitisactuallyexecutingyourexpect()functions.Also,pleasebeawarethatthisfunctionwillfail20timesifsomethingbrokethesearchfunctionality.Alternatively,youcansimplykeeptrackofwhethereverycaptionsofarhasfantainitandthenexpect
hasFantatobetrue:
varhasFanta=true;
results.map(function(elem){
returnelem.element(by.css('h3')).getText();
}).then(function(captions){
captions.forEach(function(caption){
hasFanta=hasFanta&&caption.toLowerCase().indexOf('bla')>-1;
});
expect(hasFanta).toBeTruthy();
done();
});
Wecansimplifythisfurtherusingreduce,whichallowsyoutocreateavalue,oraccumulator,basedonalltheelementsonthelist:
varallHaveFanta=results.reduce(function(acc,elem){
returnelem.element(by.css('h3')).getText().then(function(caption){
returnacc&&caption.toLowerCase().indexOf('fanta')>-1;
});
},true);
expect(allHaveFanta).toBeTruthy();
Thecallbackwepasstoreducereturnspromise(returnedbythen),whichresolvestoabooleanspecifyingwhetheracc(theaccumulator)istrueandthecurrentcaptioncontainsfanta.Ifanycaptiondoesnotcontainfanta,theaccumulatorwillbesettofalseandeverysubsequentcallwillalsoreturnfalse.Thesecondparametertoreduceistheinitialvalueofacc.Usingthismethod,wecanevengetridofthedone()call!
Youmaythinkthistestcoverseverything,butitonlyteststhattheresultswegetbackmeetoursearchcriteria;itdoesnotcheckwhetherweactuallygetalltheresultsfromtheserver.Perhaps,weareactuallyreceivingpage2,insteadofpage1.Thatisnotsomethingwecantestrightnowthough,soallinall,thistestisprettysolid.
Youmaybeconfusedaboutfunctionssuchasmap,reduce,orgetText.YoumayalsobewonderingwhatotherfunctionsyoucanuseinProtractor.Luckily,theProtractorAPIisproperlydocumented:http://www.protractortest.org/#/api.
Asalasttest,beforewecontinue,Iwanttotestwhetheraclickonthedeletebuttonintheshoppingcartreallydeletestheappropriate,andindeedany,itemfromthecart:
it('shouldremoveanitemfromtheshoppingcart',function(done){
browser.get('views\\shopping-cart.html');
browser.wait(EC.presenceOf($('[ng-controller=shoppingCartController]'),1000));
varitems=element.all(by.css('.thumbnail'));
varfirstItem=items.get(0);
varfirstCaption=firstItem.element(by.css('h3')).getText();
vardeleteBtn=firstItem.element(by.css('button'));
items.count().then(function(count){
deleteBtn.click().then(function(){
varnewItems=element.all(by.css('.thumbnail'));
varnewFirstCaption=newItems.get(0).element(by.css('h3')).getText();
expect(firstCaption).not.toBe(newFirstCaption);
expect(newItems.count()).toBe(count-1);
done();
});
});
});
So,inthisexample,wegettheitemsinthecartandthenwegetthecaptionofthefirstitemandthedeletebuttonofthefirstitem.Afterthat,weneedthecurrenttotalcountofitemsinthecart.Wewantthecountasanumber,andnotasapromise,becausewewanttocheckthatclickingthedeletebuttonremovesexactlyoneitem.Afterthat,weclickthedeletebutton.Oncethebuttonisclicked,wegettheitemsagain.Nowitisjustamatterofcheckingwhetherthecaptionofthefirstitemhaschanged,thatis,theitemwedeleted,andwhetherthecountofalltheitemsisexactlyonelessthanbefore.
Wecouldwriteadozenmoretests,butatthispoint,thatisreallynotallthatexciting.Wewillleaveitatthisfornow.
CustomizingreportersRunningyourtestsfromtheconsolegivesyouaprettycrappyoverviewofwhattestsfailedandsucceeded.Itworks,butIpreferalessverboseoverview.Later,wewillalsowanttopublishourtestresultsinadifferentformat,suchasJUnit.Luckily,thisisnotveryhard.First,wewillneedtoinstallthejasmine-reportspackage:
npminstalljasmine-reporters--save-dev
Now,weneedtoaddareportertoJasmineintheProtractorconfiguration.Let'sstartwithJUnit.WecanusetheonPreparefunctionforthistask:
onPrepare:function(){
varjasmineReporters=require('jasmine-reporters');
varjunitReporter=newjasmineReporters.JUnitXmlReporter({
savePath:'selenium-junit/',
consolidateAll:false
});
jasmine.getEnv().addReporter(junitReporter);
browser.resetUrl='file://';
browser.ignoreSynchronization=true;
},
YoucannowsimplyrunyourtestsandProtractorwillcreateareport.Thereisstillatinyproblemwiththissetupthough.Whenyoutestusingmultiplebrowsers,onlyonefileisgenerated,whichcontainstheresultsofonlyonetestrun.Ofcourse,thereisasolutiontothisproblem.Wecansetthereporterperbrowserconfigusingbrowser.getProcessedConfig():
onPrepare:function(){
browser.resetUrl='file://';
browser.ignoreSynchronization=true;
varjasmineReporters=require('jasmine-reporters');
returnbrowser.getProcessedConfig().then(function(config){
varbrowserName=config.capabilities.browserName;
varjunitReporter=newjasmineReporters.JUnitXmlReporter({
savePath:'selenium-junit/',
consolidateAll:false,
filePrefix:browserName+'-',
modifySuiteName:function(generatedSuiteName,suite){
returnbrowserName+'-'+generatedSuiteName;
}
});
jasmine.getEnv().addReporter(junitReporter);
});
},
PleasenotethatIhavemovedresetUrlandignoreSynchronizationupsothatwecanreturnpromisethatsetsthebrowserconfiguration.YouneedmodifySuitNamesoyoucankeepyourChromeresultsapartfromyourFirefoxresults.Whenyourunyourtestsagain,usingdifferentbrowsers,yougetfilessuchaschrome-Seleniumtests.xmlandfirefox-Seleniumtests.xml.
Now,addingtheTerminalreporterisapieceofcake:
jasmine.getEnv().addReporter(junitReporter);
jasmine.getEnv().addReporter(newjasmineReporters.TerminalReporter({
verbosity:3,
color:true,
showStack:true
}));
Thislooksawholelotbetter:
Asusual,wedonotwanttoincludeourgeneratedreportsinGit.So,addthefollowinglineinthe.gitignorefile:
**/selenium-junit/**
TestingwithMochaProtractorallowsyoutouseadifferenttestingframework.Jasmineisthesupporteddefault,butMochahaslimiteddefaultsupportaswell.Youcansettheframeworkpropertyintheconfigfiletojasmine,mocha,orcustom.Whenyousetittocustom,youneedtospecifytherelativepathtotheframework,forexample,Cucumber,usingtheframeworkPathproperty.
Unfortunately,wecannotrunmultipleframeworksinthesametestrunlikewithKarma.Copyyourconfigfile,nameitprotractor.mocha.conf.js,andchangethefollowingproperties:
onPrepare:function(){
browser.resetUrl='file://';
browser.ignoreSynchronization=true;
},
framework:'mocha',
specs:['selenium-mocha-tests.js'],
WealsoneedtoinstallMochagloballyforthistowork:
npminstallmocha-g
Nowweneedtocreatetheselenium-mocha-tests.jsfiletoo.Becausetheonlythingthatreallychangesistheassertionframework(mostofitisjustProtractor),ourtestscanstayprettymuchthesame.Chaiisalittlemoreexplicitinthatitishandlingasynchronouscode:
describe('SeleniumMochatests',function(){
varEC=protractor.ExpectedConditions;
varchai=require('chai');
varchaiAsPromised=require('chai-as-promised');
chai.use(chaiAsPromised);
varexpect=chai.expect;
describe('homepage',function(){
it('shouldcutofflongtitles',function(){
browser.get('index.html');
browser.wait(EC.presenceOf($('h3'),1000));
varelems=element.all(by.css('h3'));
expect(elems.count()).to.eventually.equal(3);
varelem=elems.get(0);
expect(elem.getCssValue('overflow')).to.eventually.equal('hidden');
expect(elem.getCssValue('text-overflow')).to.eventually.equal('ellipsis');
expect(elem.getCssValue('white-space')).to.eventually.equal('nowrap');
});
});
});
Wearealmostthere.Asyoucansee,weareusingChaiasPromised(becauseitdealswithpromises,getit?)insteadofjustplainChai.So,weneedtoinstallthatfirst:
npminstallchai-as-promised--save-dev
protractortest\protractor.mocha.conf.js
WecancustomizethebehaviorofMocha,forexample,togetanotherreporter,althoughthedefaultreporteralreadylooksprettygood.WesimplyaddthemochaOptsnodeintheconfigfile.YoucanreadaboutthevariousreportersontheMochawebsite,https://mochajs.org/#reporters:
mochaOpts:{
reporter:"spec",
},
Mochaisprettycrazy,itevenhasaNYANcatreporter(code"nyan"):
Selenium,Protractor,andthevarioustestingframeworkshavelotsofotherfeaturesandpluginsthatyoucanusetofurthertestyourcode.Forexample,itispossibletousetwobrowsersinasingletestincasetwobrowserwindowsneedtointeractwitheachother,thinkchatapplicationsor,moregeneral,sockettechnology.However,wehaveseensomebasicsanditisenoughtounderstandhowbrowsertestingworksandhowitcancontributetobetter-testedcodethatleadstofewerbugs.
ProtractorisnotyouronlychoicewhenyouwanttoworkwithSelenium.OtherpopularoptionsincludeNightwatch.js(http://nightwatchjs.org/),whichhasabuilt-intestingframework,soitjustworksoutofthebox;andWebDriverIO(http://webdriver.io/),whichrequiresyoutoinstallanexternaltestingframework.Likeeverything,bothhavetheirprosandcons,butbothareworthcheckingout.
HeadlessbrowsertestingThereisjustoneproblemwehavenottackledyet.First,itisprettyannoyingthatyouneedtohaveallthosebrowserwindowsopentotestyourcode.Second,ourserverdoesnotevenhaveauserinterface,sohowisitgoingtoopenanybrowser?Iwillgetbacktotheserverissuelater:evenifwehadagraphicalinterface,wewouldstillhaveproblemswithIE,Edge,andSafari,soweneedtotackleallofthat.However,Iamgoingtotacklethatfirstissuerighthere.
Whileitisnotnecessarilyaproblemthatwehavemultiplebrowserwindowsrunning,theycomewithatrade-off.First,ofcourse,theytakeupspaceonyourscreen.Youhavetoworkaroundthem,makesureyoudonotaccidentallycloseanyofthem,andyoushouldalsonotgettoodistractedbyalltheflashytestruns.AnotherissuewithbrowsersisthatittakestimetorenderyourHTMLandapplytheCSS.Enterheadlessbrowsers.
Aheadlessbrowserisreallyjustabrowserthathasnointerface.Itrunssilentlyinthebackgroundandpretendsitisanactualbrowser.Whileitisnotareplacementforanactualbrowser,itisagoodfirstlineofdefense.YoucantestyourwebsiteusingaheadlessbrowserlocallyandthentestonotherbrowsersonyourCIserver.Becauseaheadlessbrowserdoesnotrenderanygraphics,itcanbealotfasterthanregularbrowsers.EspeciallywhenyougetmoreE2Etests,headlessbrowserscanbetwiceasfast.Asanaddedbonus,youdonotseethemonyourdesktop,sotheycannotannoyyouandyoucannotclosethembyaccident.
PhantomJSApopularheadlessbrowserisPhantomJS(http://phantomjs.org/).Sinceitisprettypopularandeasytosetup,Iwanttodiscussithere,butIshouldmentionupfrontthatitdoesnotworkwellwithProtractor.TheProtractorteamactuallyrecommendsthatyoudonotusePhantomJSwithProtractor.Iconcur,Inevergotittoworktogethereither.However,itworksfineforKarma,solet'ssetthatup.
InstallationofPhantomJSiseasy,itisjustaregularnpminstallation.Ofcourse,youwillalsoneedtoinstalltheappropriateKarmalauncher:
npminstallphantomjs--save-dev
npminstallkarma-phantomjs-launcher--save-dev
YoucannowusePhantomJSasabrowserinyourKarmaconfiguration:
browsers:['PhantomJS'],
IfyounowstartKarma,youwillnoticenotasinglebrowserisstarted,butyourtestsarestillexecuted.YoucanchangeatesttomakeitfailandyouwillseethatKarmaupdatesalmostinstantlyinthecommandwindow.YoumightnoticethatPhantomJSgivesyouabigrederrorwhenitstarts.Donotworryaboutit;itissomePhantomJSthing,butwillnotpreventitfromstartingandexecutingyourtests.Itisannoyingatworst.
Intheory,PhantomJSshouldworklikeanyotherbrowser,butunfortunately,thisisnotquitethecaseinpractice.Assaid,forKarma,everythingworksasexpected,butwhenitcomestoSeleniumandProtractor,PhantomJSjustdoesnotworkthatwell.Itmayworkforyouonyourexactsystemconfiguration,butyouwilllikelyrunintoalotofissues.Forexample,personally,IcouldnevergetittoworkonaglobalPhantomJSinstallation.IcanmakeSeleniumdosomeworkonalocalPhantomJSinstallation,butitwillloopindefinitelyontryingtoselectanelement.ThefunnythingisthattheJasmineconfigurationworksdifferentlythantheMochaconfiguration,butneitherwork.
JustincasetheterribletrioSelenium,Protractor,andPhantomJSevergettheir
stufftogether,thisishowyourconfigurationissupposedtolook:
seleniumAddress:'http://localhost:4444/wd/hub',
multiCapabilities:[{
browserName:'phantomjs',
'phantomjs.binary.path':require('phantomjs').path
}]
//directConnect:true
Aswithmostbrowsers,directConnectisnotsupportedatall.
PhantomJScandomorethanjusttestyourJavaScriptcode.Itisafull-fledgedbrowserthatyoucancontrol.However,forthepurposeofCI,itisnotveryinterestingbeyondheadlesstesting.
Unfortunately,thereisnogoodheadlessalternativeforSeleniumandPhantomJSonWindows.YoucanrunbrowsersinheadlessstateonLinuxusingXvirtualframebuffer(Xvfb),butIhavenottrieditbefore,soyouareonyourownthere.
SummaryInthischapter,wehavetestedvariouspartsofourwebshop.WithJasmineandKarma,itispossibletocreateunitteststhatrunwheneveryouchangesomething.Dependingonthequalityandcoverageofyourtests,errorscanbefoundassoonastheyareintroduced.Goingastepfurther,wecantestourapplicationasthoughitwastestedbyanactualhumanusingSeleniumandProtractor.Inthenextchapter,wearegoingtoautomateourbuild,includingtesting,linting,andminifying,usingGulp,aJavaScripttaskrunner.
AutomationwithGulpInthepreviouschapters,wehavesetupGit,asampleapplication,andsometests.However,whileGit,orsourcecontrolingeneral,andtestsareveryimportantforsuccessfulCI,perhapsthemostimportantpartisautomation.Youcanwritehundredsoftests,butifaprogrammerforgetstorunthem(orknowinglyrefusestodoso),thosetestsbecomeprettyuseless.Thesamegoesforothertasks,suchaslintingandminifyingyourcode.Later,whenyouneedtoreleaseyourapplicationtoaliveenvironment,thehumanfactorisyournumberoneconcern;onewrongmoveandtheentireapplicationgoesdown!Forthisreason,itisimportanttoautomateallthethings.
WhenitcomestoautomatingyourHTML,CSS,andJavaScripttasks,therearetwopopularcontendersforthejob:Grunt(https://gruntjs.com/)andGulp(http://gulpjs.com/).BotharelabelledasJavaScripttaskrunners.Grunthasbeenaroundalittlewhilelongerandiswidelyused.However,ithassomeissuesthatGulpissupposedtoaddress.Unfortunately,GulpalsointroducessomenewissuesthatGruntdoesnothave.Overall,thechoicebetweenGruntorGulpisprettymuchapersonalpreference.However,inthischapter,wearegoingtoworkwithGulp.
GulpbasicsInabstractterms,ataskrunnertakessomeinput,workswiththat,andproducesanoutput.Thatoutputcouldthenbeusedforfurtherprocessing.Forexample,aJavaScriptfilecouldbetheinput,aminifyingjobcouldbetheprocess(thatismakingyourJavaScriptunreadable,butverycompact),andtheminifiedJavaScriptwouldbetheoutput.Now,theminifiedJavaScriptcouldbeinputtoanewtestprocess,whichwouldhavesomereportasitsoutput.Gulpdoesexactlythis.Gulpisalittledifferentfromothertaskrunnersinawaythatitkeepsintermediateresultsinmemoryinsteadofwritingthemtodisk.
InstallingGulpisaseasyasdoingnpminstall.WealsowanttheGulpCLIforeasyuse:
npminstallgulp--save-dev
npminstallgulp-cli-g
Thenextthingweneedisaso-calledgulpfile.Intherootofyourproject,Chapter06,inthebook'sGitHubrepository,createanewfileandnameitgulpfile.js.GulpsimplystartsaNode.jsprocess,sowecandowhateverNode.jscandointhegulpfile.WestartbyrequiringGulpandsettingadefaulttask:
vargulp=require('gulp');
gulp.task('default',function(){
console.log("We'rejustrunningtasks...");
});
Youcannowrunthisfromthecommandlinebysimplyrunningthegulpcommand.Makesureyourcommandlineisinyourprojectfolder.Youcan,ofcourse,alsomanuallytargetyourgulpfile:
gulp
[alternatively]
gulp--gulpfilepath_to_your_gulpfile\gulpfile.js
TheoutputshouldbeWe'rejustrunningtasks....Ifitwas,youknowyoudiditrightandGulpisworking.
TheGulpAPIGulphasfourmethodsthatyouwillbeusingalot.Thosearesrc(),dest(),task(),andwatch().Additionally,youwillbeusingNode'spipe()methodtopasstheoutputfromonefunctiontotheother.Thefollowingexampletakesthecssfolderandsimplycopiesittoanewcss_copyfolder:
vargulp=require('gulp');
gulp.task('copycss',function(){
gulp.src('css*.css')
.pipe(gulp.dest('css_copy'));
});
gulp.task('default',['copycss']);
Asyoucansee,wedefinedtwotasks,'copycss'and'default'.The'default'taskisstartedbyGulpby,well,default.Inthiscase,thedefaulttasksimplystartsthetasksdefinedinthearray,socopycss.Inthecopycsstask,wetakealltheCSSfilesfromthecssfolder.Asyoucansee,wildcardsareallowed,whichreturnsaNode.jsStreamobject,andpipesthecontentsofthatStreamobjecttothegulp.destfunction,whichwritesthecontentstothecss_copyfolder.IfyourunthisinGulp,youwillfindyourCSSfilescopied.YoucanalsorunthecopycsstaskdirectlybypassingthetasknametotheGulpprogramonthecommandline:
Youcanwatchfilesandautomaticallyrunaseriesoftaskswhenawatchedfilechanges.Forexample,let'ssaywewanttocopyallourCSSfilesifonechanges:
gulp.watch('css*.css',['copycss']);
Weusegulp.watch,specifythefilestowatch,andthenspecifyanarrayoftasksthatneedtobeexecutedwhenawatchedfilechanges.Whenyourunthegulpcommandnow,itwillnotendafterithasexecutedthedefaulttask,butitwillwaitforchanges.TrychangingaCSSfileandconfirmthatGulpimmediatelycopiesitonsave.
Andthereyouhaveit,theentireGulpAPI.Wellalright,wemayhavemissedsomefunctionsoroverloads,butthisiswhatitallboilsdownto.
GulppluginsYoumayhaveguessedit,butGulponitsowndoesnotdoalot(Iamhavingdejavuhere).Youneedpluginstogetanyseriousworkdone.Luckily,Gulphasquitealotofthose.TheinitialideawasthatanyGulpplugintakessomeinputandoutputstheresultasaNode.jsStreamobject.However,inpractice,thisisoftennotthecase.Somepluginsreturnnothing,somewritetheiroutputtodiskandreturntheoriginalinput,whilesomewatchfilesandneverreturnatall.Thatsoundsbad,butitdoesnothavetobe.YouhavetotakeintoaccountthatalotofpluginsaresimplywrappersaroundotherprogramsthatwerenotcreatedtoworkwithGulp(oranytaskrunner)atall.Justbesureyoureadtheplugin'sdocumentationanduseitasitwasintended.Luckily,Gulpcanworkaroundtheseissuesquitewell.
MinificationThefirstpluginwearegoingtolookatistheminifyplugin(https://www.npmjs.com/package/gulp-minify).ThispluginletsyouminifyyourJavaScriptsource.Wecaninstallitusingnpm:
npminstallgulp-minify--save-dev
Nowwecansimplygetsomesource,pipeittotheminifyplugin,andoutputtheresults:
vargulp=require('gulp');
varminify=require('gulp-minify');
gulp.task('minify',function(){
gulp.src('scripts/*.js')
.pipe(minify({
ext:{
src:'.debug.js',
min:'.js'
}
}))
.pipe(gulp.dest('prod'));
});
gulp.task('default',['minify']);
Now,whenyourunGulp,youwillseeaprodfoldercontainingallthescriptstwice,a.debug.jsanda.jsvariant.Theextobjectletsyouspecifytheinputandoutputsuffixes.Itisalsopossibletotransformyourfilenamesusingregularexpressions,butyoucanfigurethatoutonyourown.Theminifiedindex.jsfileshouldlooksomethinglikethefollowing:
angular.module("shopApp",[]).controller("homeController",function(o){o.topProducts=repository.getTopProducts(),o.searchTerm=""});
Thatisnotveryreadable,butverycompact.Itisperfectforproductionsystemswherethosepreciousbytesmatter.
Wecantweaktheminifypluginevenfurther.Inourcase,havingthesourcefilesisnotnecessaryatall,sowecanexcludethem:
.pipe(minify({
ext:{
min:'.js'
},
noSource:true
}))
Otherthanthat,youcanignorefilesandfolders,preservecomments(astheyareremovedbydefault),andkeeptheoriginalvariablenames.Havingallofthoseoptionsinyourgulpfileisfine,butsometimesitgetsalittlebigforjustonefile.Itiseasytomovetheoptionsintoaseparateconfigurationfile.Createanewfileandnameitminify.conf.js.Inside,itisjusttheoptionsobjectyouwouldliketopasstotheminifyplugin,but,likeyourProtractorscript,exposedthroughthemodule.exportsobject:
module.exports={
ext:{
min:'.js'
},
noSource:true
};
Youcannowsimplyrequiretheconfigurationfileinyourgulpfile:
gulp.src('scripts/*.js')
.pipe(minify(require('./minify.conf.js')))
.pipe(gulp.dest('prod'));
Thatway,itbecomeseasytoshareconfigurationsbetweenmultiplegulpfilesortochangeconfigurations,butnotyourgulpfile.
CleaningyourbuildIfyourantheprevioussample,youmighthavenoticedthatwhenyouranitwithoutthesourcefiles,yourminifiedscriptswereupdated,butyourcopiedsourcefileswerenotremovedfromtheprodfolder.Itisbestpracticetocleanupanybuildfoldersbeforeyoucreateanewbuild.Thisassuresthatyourbuildworksfromscratchandthatyoudonothaveanycachedfiles.Thereusedtobeapluginforthis,butthereisaNode.jsmodulethatalreadydoeswhatweneed.AsIsaidearlier,Gulppluginsshouldworkoninputandproducesomeoutput.Deletingfilesandfoldersdoesnotfitthatdescriptionthough,sothisisnotreallyaGulpplugintaskanyway.So,wearegoingtousetheNode.jsdelmodule(https://www.npmjs.com/package/del):
npminstalldel--save-dev
YoucannowjustusedelinaGulptask:
vargulp=require('gulp');
vardel=require('del');
gulp.task('clean',function(){
del(['prod/*']);
});
Easyasthat.Also,donotforgettoconfigureyourminifyjob,soitwillrunthenewcleantaskbeforeitdoesanyminifying.Youcanpassinanarrayoftasknamesthathavetobeexecuted(asynchronously)beforethecurrenttaskexecutes:
gulp.task('minify',['clean'],function(){
Ifyounowrungulporgulpminify,youwillfindthatitfirstdeletedtheprodfolderandthenputyournewlyminifiedscriptsinit.
CheckingfilesizesAsyouaredeletingandminifyingfiles,youmaybeinterestedinseeingexactlyhowmanybytesyouaresaving.Thereisapluginforthat,calledgulp-size:
npminstallgulp-size--save-dev
Usageissimple:
vargulp=require('gulp');
varminify=require('gulp-minify');
varsize=require('gulp-size');
gulp.task('minify',['clean'],function(){
gulp.src('scripts/*.js')
.pipe(size())
.pipe(minify(require('./minify.conf.js')))
.pipe(size())
.pipe(gulp.dest('prod'));
});
Thisoutputsthesizeofallthefilesbeforeandafterminificationtothecommandwindow.Youcanalsopassinsomeadditionaloptionstosize:
.pipe(size({
title:'Beforeminification',
showFiles:true,
showTotal:true,
pretty:false
}))
Thetitlecanbeusedtokeepaseparateinstanceofsizecallsapart.TheshowFilesoptionindicatesitshouldshowthesizeofeachfileseparately.Likewise,showTotalindicateswhetherthecombinedsizeshouldbedisplayed.Last,prettyindicateswhetherthefilesizeshouldbedisplayedinaprettyformat,suchasKB,orinbytes.
LintingwithJSHintThenextthingwewanttodoislintourJavaScriptfiles.Lintingistheprocessofcheckingcodeforpotentialerrorsorcodesmells.Youcantryitmanuallyathttp://www.jslint.com/.Becarefulthough,thatthingismerciless!IdonotthinkIhaveeverhadaperfectscoreonthatthing.JSHintisbasedonthesamelinter,butalotmoreforgiving.Youcanconfigureallkindsofrulestomakeiteasierorharderonyourself.Italsohasawebsite,http://jshint.com/,whereyoucanlintyourJavaScriptcodeonthefly.Whenyouaregoingtousethislinter,besuretocheckoutthedocumentationaswell,asithasalotofhintsandusecases.
TouseJSHintinGulp,youneedJSHintinadditiontotheGulpplugin,gulp-jshint:
npminstalljshint--save-dev
npminstallgulp-jshint--save-dev
Wecannowcreateaseparatelinttaskandhaveitexecutebeforetheminifytask,butatthesametimeasthecleantask:
vargulp=require('gulp');
varjshint=require('gulp-jshint');
gulp.task('lint',function(){
gulp.src('scripts/*.js')
.pipe(jshint())
.pipe(jshint.reporter('default'));
});
gulp.task('minify',['clean','lint'],function(done){
Asyoucansee,welintthesourceandpipetheoutputtoareporter(thatwillprinttotheconsole).Thereisonecaveatthough.Gulprunsallofitstasksasynchronously,sothejshintfunctionwillexecuteduringtheminifyfunction.Ifyourunthisnow,youroutputwillhavefilesizesandlintingerrorsallmixedup.WecanreturnthestreamandGulpwillwaitforittofinishthough:
gulp.task('lint',function(done){
returngulp.src('scripts/*.js')
.pipe(jshint())
.pipe(jshint.reporter('default'));
});
Youwillnowfindtheoutputtobesequentialandprettyreadable.Luckily,IhavemessedupourJavaScriptprettybad,soweactuallyhavesomethingtofixnow:
Asyoucansee,evenwhenwehavelintingerrors,Gulpstillcontinues.Maybewedonotwantthat.Afterall,amissingsemicoloncanreallymessupourminifiedcode!So,weneedtofailourbuildonlintingerrors.JSHinthassomeadditionalreportersandoneofthemisthefailreporter:
returngulp.src('scripts/*.js')
.pipe(jshint())
.pipe(jshint.reporter('default'))
.pipe(jshint.reporter('fail'));
Withthisreporter,Gulpwillthrowanerror,Message:JSHintfailedfor:file1.js,file2.js,andstopexecution.Afterthis,itdoesnotmakesensetocleanourprodfolderfirst,soweshouldchangethatsoitlintsfirstandcleansonsuccess:
gulp.task('clean',['lint'],function(){
[...]
gulp.task('minify',['clean'],function(){
[...]
Bewarewiththesekindoftoolsthough,whetheryouareusingJavaScriptoranyotherlanguage.Sometimes,yourlinterisjustwrong.Take,forexample,thewarninginrepository.js:Use===tocomparewithnull(inthesearchfunction).The
lineif(q==null)willevaluatetotrueifqisnullorundefined,changing==to===hastheeffectthattheexpressionwillnowevaluatetofalseforundefined,possiblybreakingyourcode!Sowhateveryoudo,keepusingyourhead!
Luckily,youcantellJSHinttoshutupandignorevariousissues:
search:function(q){
/*jshintignore:start*/
if(q==null){
return[];
}else{
returnproducts.filter(function(p){
returnp.name.toLowerCase().indexOf(q.toLowerCase())>=0;
});
}
/*jshintignore:end*/
}
Thiswillmaketheerrorgoaway,butnowtheentirefunctionisnotlinted.Wecannotputignore:endonthefollowingline,becausethenJSHintwillthinkthelineif(q==null)doesnotexistatallanditwillstartcomplainingaboutelsewithoutif(...)andreturn[];thatmakesothercodeunreachable.However,whatwecandoisspecifyexactlywhatissuestoignore:
/*jshinteqnull:true*/
[...]
/*jshintignore:end*/
Andifyoudoitlikethat,youcanevenendtheignoreonthenextline.So,ifyouputanothersomething==nullinthatfunction,JSHintwillpickitupagain:
/*jshinteqnull:true*/
if(q==null){
/*jshintignore:end*/
return[];
}else{
[...]
Ifyouneedtogloballyignorecertainissues,youcanpassinanoptionsobjecttothejshintfunction:
.pipe(jshint({
eqnull:true
}))
Alternatively,youcancreateaJSONfileanddefineyoursettingsthere.Createafilecalledjshint.conf.jsoninyourprojectfolderandputthefollowingJSONinit:
{
"eqnull":true
}
Nowyoucanpassthefileintothejshintfunction:
.pipe(jshint('jshint.conf.json'))
Therearequitealotofoptions;youcanfindtheminthedocumentation,http://jshint.com/docs/options/.
YoucanfixalloftheJavaScripterrors;JSHintgivesyoutheexactcauseandlineoftheerror.Inoneinstance,wehaveacommainsteadofasemicolonattheendoftheline.Also,insteadofgetQueryParams()['id'],wecansimplyusegetQueryParams().id.Theonlyissuethatisalittlebithardertofixistheoneinutils.js.TheJSHintapprovedcodeshouldnotdoanassignmentinawhileloop:
tokens=regex.exec(qs);
while(tokens){
params[decodeURIComponent(tokens[1])]=decodeURIComponent(tokens[2]);
tokens=regex.exec(qs);
}
Personally,Iamabigfanoflinting.Notonlydoesitscanyourcodeforpossibleerrors,italsomakesyouthinkaboutyourcodeinawaythatyoumaylearnfromit.Bytheway,wehaveseenlintingbeforeinChapter2,SettingUpaCIEnvironment,butwithSonarQube.WewillgetbacktoSonarQubelateranduseitwithJavaScriptandC#.
Bytheway,becausewemadelintingaseparatejob,wecannowalsorunitwithouthavingtocleanorminify;simplyrunthegulplintcommand(thatis,gulp[taskname]).
Nexttothebuilt-indefaultandfailreporters,youcangetsomeexternalreportersaswell.Icanrecommendthestylishreporter(https://github.com/sindresorhus/jshint-stylish).Installitusingnpminstalljshint-stylish--save-devandreplacethedefaultreporterwith'jshint-stylish'.
RunningyourKarmatestsNext,wewanttorunourtests.Ofcourse,wealreadycreatedsometestsandranthemwithKarma.Noworries,wearenotgoingtoundoallthat.Instead,wearegoingtorunKarmaautomaticallyusingGulp.Thatway,welint,test,andminifyourcodewithasinglegulpcommand.ThereusedtobeaKarmapluginforGulp,butnowitisrecommendedtojustrunKarmadirectlyfromGulp.Thisisactuallysurprisinglyeasy:
vargulp=require('gulp');
varkarma=require('karma').Server;
gulp.task('test',function(){
newkarma({
configFile:__dirname+'/test/karma.conf.js'
}).start();
});
The__dirnamevariableisaglobalNode.jsvariableandcontainsthecurrentfilepath.So,wesetupanewKarmaserverfromcode,passitourconfigfile,andstartitup.TheconfigfilestillhasPhantomJSsetupandwatchesfilesastheychange.Thisisabitofaproblem,asourGulpfilecanwatchfilesaswell,butcurrentlydoesnot.So,Gulpneverreturnsbecauseofourtests,butwheneverafilechanges,itisonlytested,butnotlintedoranythingelse.So,let'stweaktheKarmaconfigurationabitsoitplaysnicelywithGulp.WearegoingtosetsingleRuntotrue.Nowwecanputanyfilewatchersinourgulpfile.Luckily,wedonotneedtochangeourconfigurationfile;wecanoverrideourconfigurationfromGulpdirectly:
newkarma({
configFile:__dirname+'/test/karma.conf.js',
singleRun:true
}).start();
Also,wewanttotestourcodeafterlinting,butbeforeminification:
gulp.task('lint',function(){
[...]
gulp.task('test',['lint'],function(){
[...]
gulp.task('clean',['test'],function(){
[...]
gulp.task('minify',['clean'],function(){
[...]
Irealizethatthedependenciesarebecomingabitconfusing,butfornow,thisistheonlywaytoguaranteetasksareruninserieswithoutusingaplugin.StartinginGulp4,wegetgulp.series('firstTask','secondTask'),but,unfortunately,ithasnotyetbeenofficiallyreleasedatthetimeofwriting.
Tomakethingsworse,thetesttaskisrunasynchronouslyandwedonothaveanystreamtoreturn.Luckily,Gulphasonemoretrickupitssleeve.Wecanpassinacallbacktosignalwhenourtaskiscompleted,muchlikewedidwithProtractorandJasmineinthepreviouschapter:
gulp.task('test',['lint'],function(done){
newkarma({
configFile:__dirname+'/test/karma.conf.js',
singleRun:true
},done).start();
});
WhenwerunGulpnow,wecanseethatourcodeislintedandthentested.Onsuccess,ourprodfolderiscleanedandfinallyoursourceisminified,inthatorder.
Thereisjustonemoreissuethough.Itismoreofanannoyancethananactualissue,butIwouldliketofixitnonetheless.Wheneveratestfails,GulpgivesusthishugeerrormessagewithsomeGulpstacktracethatisofnoimportancetous.OtherGulpplugins,suchasJSHint,havethiscovered,butKarmaisnotaGulppluginanddoesnotevenusestreams.So,asyoumayhaveguessed,wewillneedanotherplugin,gulp-util(https://www.npmjs.com/package/gulp-util):
npminstallgulp-util--save-dev
WenowhavetowriteourownfunctionfortheKarmaservercallback.Wegetanerrornumberinthecallback;whenitisgreaterthan0,wecanpassinanerrortothedonecallback.Notanyerrorthough,thiswouldgiveustheuselessstacktracewegotearlier.Instead,wewillusethePluginErrorfunctionfromgulp-utilsandpassittothedonecallback:
vargutil=require('gulp-util');
[...]
newkarma({
configFile:__dirname+'/test/karma.conf.js',
singleRun:true
},function(err){
if(err>0){
returndone(newgutil.PluginError('karma','Karmatests
failed.'));
}
returndone();
}).start();
Gulpwillgiveusaneatlittleerrormessageinsteadofthegarbageitthrowsatusnow:
WecannowclearlyreadthattheKarmapluginfailedandwecanalsostillseeexactlywhereithasgonewrong.
Next,wearegoingtorunourKarmatests.Again.Itisgreatthatwecanlintandtestandminify,butweshoulddefinitelytestourminifiedcodeaswell.Wewanttotestouroriginalcodeonatleastonebrowser.PhantomJSwilldo,sowhensomethinggoeswrong,weimmediatelyseetheerror,file,andlinenumber.Wealsoneedthisforourcodecoverage.However,ourcodegetsprettymessedupduringminification,soitcannothurttotestagain.Atleastmosterrorsthatmayoccurhavebeeneliminatedbyourlintingprocess;wedonothavemissingsemicolonsoranythinglikethat.Fortestingtheminifiedfiles,wecanuseacompletelydifferentKarmaconfiguration.Ofcourse,youcanspecifyitcompletelyinGulporoverwritesomeoptionsfromouroriginalconfiguration
file,butIliketokeepacompletelyseparateKarmaconfigurationformyminifiedtests.So,createanewfileinyourtestfolderandnameitkarma.min.conf.js.Makesureyouremovethecoveragereporterandsettingscompletely,wedonotneedthemforthesetests.Also,makesureyoureferenceyourminifiedscriptsandtestonasmanybrowsersasyouwishtosupport:
module.exports=function(config){
config.set({
files:[
[...]
'../prod/*.js'
],
reporters:['progress','junit'],
browsers:['IE','IE9','Edge','Chrome','Firefox'],
singleRun:true,
[...]
})
};
WecannowcreateanothertaskinGulp,whichisalmostanexactcopyofourregulartesttask:
gulp.task('test-min',['minify'],function(done){
newkarma({
configFile:__dirname+'/test/karma.min.conf.js'
},function(err){
if(err>0){
returndone(newgutil.PluginError('karma','Karmatestsfailed.'));
}
returndone();
}).start();
});
Donotforgettochangeyourdefaulttaskaswell:
gulp.task('default',['test-min']);
Youwillimmediatelyseewhytestingyourminifiedcodeissoimportant.RunGulpandyouwillseethatacoupleoftestsfailonallyourbrowsers!Theyworkedfinebeforeminification,buttheyaretotallybrokenafter.Yourconsolewindowwillgiveyoualotofgarbage(acompletestacktraceforeveryfailingtestforeverybrowser),butaftersomedigging,wefindthefollowingerror:[$injector:unpr]Unknownprovider:tProvider<-t<-shoppingCartController.Angular.jsusesdependencyinjectionanddoessobasedonvariablenames.Thesamevariablesthatgetcompletelymessedupbyourminificationprocess.Luckily,thefixissimple.Angular.jsletsyouannotateyourvariables:
angular.module('shopApp',[])
.controller('shoppingCartController',['$scope',function($scope){
[...]
}]);
Afterthis,fixyourtestsandtheywillrunagain.Rememberthatweonlyunittestedtheshoppingcartpagethough,soallyourotherpageswillstillbreakatthispoint.
Youhaveanotheroptionthough.Wecanchangethesettingsofourminificationprocess,soitwillkeepvariablenamesintact.Notideal,butbetterthancodethatdoesnotrunatall.Inyourminify.conf.jsfile,addthefollowingproperty:
mangle:false,
Youcannowkeepyourshoppingcartfileasitwasandthetestswillstillrunasexpected.Iamallformanglingourvariablenames(tosavesomeextrabits),soIamgoingtoannotateourAngular.jsvariablesineveryfile.Youcandowhateversuitsyoubestthough.
Speakingoftestsandlinting,youcanlintyourtestfilestomakesureyourtestsareuptopar.Youdonotwantatesttosucceed(orfail)becauseyouused==insteadof===.Itmightmakeadifference,forexample,whenweexpect1.23,butweget"1.23"(noticethedoublequotes),likeintheshoppingcarttests.Ofcourse,usingJasmine(orChai)assertionssuchastoBeandtoEqualminimizestherisks,butyoumaystillwanttolintyourtestsanyway.Minifyingyourtestsisratheruselessthough.
GettingoursiteproductionreadyNowthatwehaveminifiedandtestedourJavaScriptandgotsomeworkdoneusingGulp,wearegoingtodoafastforwardandminifyourHTMLandCSSandbundleourscriptsandreplacetheonesinourHTMLfileswiththeminifiedones.
MinifyingHTMLFirst,let'sstartwithminifyingourHTML.Therearevariouspluginsyoucanuse,butwewilluseHTMLMinifier(https://www.npmjs.com/package/html-minifier).WewillneedtheGulpplugin,whichwillalsoinstallHTMLMinifier(https://github.com/jonschlinkert/gulp-htmlmin):
npminstallgulp-htmlmin--save-dev
Theusageisquitesimple:
varhtmlmin=require('gulp-htmlmin');
gulp.task('minify-html',function(){
returngulp.src(['index.html','views/*.html'])
.pipe(htmlmin({
collapseWhitespace:true
}))
.pipe(gulp.dest('prod'));
});
ThepossibleoptionscanbefoundintheHTMLMinifierdocumentation.Bewaresomeoptions:theymaybreakyourpage.Forexample,somelibraries,suchasKnockout.js,candependoncomments,butHTMLMinifierhasanoptiontoremovecomments.Again,(automated)testingmayhelpyoufindsuchissues.Ourfolderstructureismessedupnow:wehaveindex.htmltogetherwithalltheotherHTMLfilesandJavaScriptfiles.Thisisonlyaproblemnow,sinceweareusingthefileprotocolandwilldependuponrelativepathfiles.Itmaybeagoodideatooutputtheviewsandscriptstotheirownfoldersthough.Andsincewearenowminifyingmultipleresources,wecanrenameour'minify'tasktosomethinglike'minify-js':
gulp.task('minify-js',['clean'],function(){
[...]
.pipe(gulp.dest('prod/scripts'));
});
gulp.task('minify-html',function(){
[...]
.pipe(gulp.dest('prod/views'));
});
Wewillworryaboutthedependencieslater.First,let'sjustmakesurewehaveallthetaskswewanttorun.Youcanrunasingletaskusingthegulp[task-name]
command,orinthiscase,gulpminify-html.
Theresultshouldlooksomethinglikethefollowing:
<!DOCTYPEhtml><html><head><metacharset="UTF-8"><title>CIWebShop</title><link...
AllyourHTMLisnowonasingleline,attributeswithdefaultvaluesareremoved,commentsmayhavebeenremoveddependingonyouroptions,andmoreoptimizationshavebeenapplied.
MinifyingCSSNext,wearegoingtogiveourCSSthesamekindoftreatment.Ithinkyouprettymuchgetthepointbynow,soIwillbeshortaboutthis.First,installthegulp-clean-csspackage(https://github.com/scniro/gulp-clean-css):
npminstallgulp-clean-css--save-dev
TheGulpcleanCSSpluginmakesuseofcleanCSS,soforanyoptions,checktheirdocumentation,https://github.com/jakubpawlowicz/clean-css.Therearealotofoptions.Iwillnotgooverthem,butIamsureyoucanfigurethemoutifyouneedthem.
Andthensimplycreateataskthatdoesthework:
gulp.task('minify-css',function(){
returngulp.src('css/*.css')
.pipe(cleancss({
compatibility:'ie9'
}))
.pipe(gulp.dest('prod/css'));
});
Runthegulpminify-csscommandandbeholdthefruitsofourlabor:
.wrap{overflow:hidden;text-overflow:ellipsis;white-space:nowrap}
Thatistheentireutils.cssfileonasingleline(althoughitwasnotverybigtobeginwith).Also,noticethatallthespacesandthesemicolonafterwhite-space:nowraphavebeenremoved.
BundlingJavaScriptwithBrowserifyNowthatwehaveminifiedallofourcode,wewantourHTMLfilestoreferencethenewlycreatedfiles.Theeasiestsolutionistokeepthecompletefolderstructureandfilenamesthesameasbeforeminification.Ifyoumoveindex.jstotheviewsfolderandfixallthereferencestootherfiles,youareprettymuchdone.However,ifyoudidnotdothat,forwhateverreasonyouhave,oryousuffixedyourminifiedfileswith.min.ext,youwillneedtodosomeadditionalwork.Youprobablywanttodosomeadditionalworkanyway,aswehavenotyetbundledourJavaScriptandCSSyet.
BundlingJavaScriptisdifficult.Thepainisinmanagingyourdependencies.Let'ssaywehavescriptsA,B,andC.CdependsonBandBdependsonA.Thisiseasywhenyouhavethreescripts,butwhenyougettothirty,thingsstarttogetmessy.Onemethodtokeeptrackofdependenciesisdoingthisbyhand,whichiswhatwecurrentlydoandwhichisdifficultanderrorprone.Thenextdeveloperwillpullouthishairtryingtofigureoutallthedependencies.AnothermethodisbyusingtherequirefunctionwehavebeenusinginNode.js.Node.jsusesCommonJSformoduleloadingthroughrequire.Therequirefunctionloadsascriptonthefly,sowhenyouneedscriptC,youcanjustreferencescriptCusingrequire('C').OncescriptCloads,itwillloadscriptBandscriptBwillloadscriptA,allthroughrequire.Thismethod,however,dependsonsomemodule.exportsvariableinyourscript.Wedonothavethat.Addingittoourscriptsnowisnotmuchofaproblem,butitissomethingtothinkaboutbeforeyoustartwritingthosethirtyormorescripts.IfweusedCommonJS,ourscriptswouldlookasfollows:
//a.js
varb=require('./b.js');
console.log('a');
b.log();
//b.js
module.exports={
log:function(){
console.log('b');
}
};
Unfortunately,browsersdonotsupporttherequirefunctionlikeNode.jsdoes.Anothermethod,designedforthebrowser,isbyusingAsynchronousModuleDefinition(AMD).UsingAMDonthebrowserstillrequirestheuseofthird-partylibraries,suchasrequire.js.SincewecurrentlyhavetomakeachoicebetweenCommonJSandAMD(althoughwecansupportboth),wearegoingforCommonJS.UsingCommonJS,wegetNode.jssupportoutoftheboxandtherearetoolsthatcanconvertCommonJStosomethingwecanuseonthebrowser,mostnotablyBrowserify(http://browserify.org/).
Sofirstofall,wemustchangeourscripts.Theonlyscriptsthatneedtoexportanythingarerepository.jsandutils.js,becausethosearethescriptsthatwereuseinotherscripts.Becauseofthewaywesetupourscriptsinthefirstplace,changingthemisaseasyasreplacingthefirstlineofthescriptstothefollowing:
//varrepository=(function(){
//varutils=(function(){
module.exports=(function(){
Next,weneedtouserequireinourotherscripts.Dependingonthescript,weneedtorequirerepositoryand/orutils.Simplyaddthefollowinglinestoyourscripts(atthetoporinsideyourcontroller;itdoesnotreallymatter):
varrepository=require('./repository.js');
varutils=require('./utils.js');
Sofar,wehavemadeourscriptsBrowserify-readyandwehavebrokenourentirewebsite.Afterall,thebrowserdoesnotunderstandrequire.So,thenextthingweneedtodoisinstallBrowserifyandrunitonourscripts:
npminstallbrowserify-g
npminstallbrowserify--save-dev
Wecannowbundleupourscriptsusingthecommandline.Forexample,let'smakeabundleofourproduct.jsscript.Thebundlewillcontainrepository.js,utils.js,product.js,andabitofoverheadfromBrowserify.Pleasenotethattheoverheadisrelativelybig,butthatisonlybecauseourfilesarerathersmall.Wedogettheaddedbenefitthatthebrowsernowonlyhastodownloadonefileinsteadofthree.Theadvantagesinloadingspeedbecomesbiggeronceyougetmorefiles:
browserifyscripts/product.js-oscripts/bundles/product.bundle.js
Wenowhavetoeditourproduct.htmlfile,soitreferencesournewproduct.bundle.jsfileratherthanrepository.js,utils.js,andproduct.js(intherightorder):
<!--scriptsrc="..scripts\utils.js"></script>
<scriptsrc="..scripts\repository.js"></script>
<scriptsrc="..scripts\product.js"></script-->
<scriptsrc="..scripts\bundles\product.bundle.js"></script>
Ifyounowbrowsetofile://filelocation/product.html?id=3,youwillseeeverythingworksasexpected.However,allyourscriptsarenowonescriptandithassomeBrowserifygibberish,makingdebuggingmoredifficult.Luckily,Browserifyhasaswitchthatlet'syouspecifythisisadebugbuildandaddstheso-calledsourcemapstoyourfile,makingitappearasthethreeoriginalfilesonyourbrowser:
browserifyscripts/product.js-oscripts/bundles/product.bundle.js--debug
Youcangenerateallyourbundledscriptsthiswayandneverhavetoworryaboutdependencies.Theonlythingleftforustodoisautomatethisprocess.Unfortunately,BrowserifydoesnothaveanactivelymaintainedGulpplugin.Instead,wecandirectlycallBrowserifyfromGulp.
UsingBrowserifyisabitofahassle,soIamgoingtotakeyouthroughitstepbystep.Browserifycancreateonebundleatatime.Creatingasinglebundleisaprettystraightforwardtask.Unfortunately,thereturnvalueofBrowserifyisnotcompatiblewithotherGulpstreamfunctions.Luckily,wecanusevinyl-source-stream(https://www.npmjs.com/package/vinyl-source-stream)tohandlethisforus:
npminstallvinyl-source-stream--save-dev
Oncewehavevinyl-source-stream,wecanrunBrowserifyonafile,converttheresult,andsaveitlikewenormallywould:
varbrowserify=require('browserify');
varsource=require('vinyl-source-stream');
gulp.task('browserify',function(){
browserify({
entries:['scripts/product.js'],
debug:true
})
.bundle()
.pipe(source('product.bundle.js'))
.pipe(gulp.dest('prod/scripts'));
});
Doingthisforallourscriptsseemsrathercumbersome.Browserifycantake
multipleentries,butthatwillstilljustcreateasinglescript.So,weneedamechanismtogetthefilesweneed,loopthroughthem,andBrowserifythemonebyone.Theglobmodule(https://www.npmjs.com/package/glob)cantakecareofthefirststep.Itfetchesthenamesofthefilesasanarrayusingwildcards.Withanarrayofthefilenames,wecanloopandcreateaneventstreamforeachfilename.Onceallthefileshavebeenprocessed,wecansignalGulpthatthetaskhascompleted.Inpsuedo-code,itwouldlookasfollows:
gulp.task('browserify',function(done){
glob('./scripts/*.js',function(err,files){
varstreams=files.map(function(file){
//CreateBrowserifystream.
});
streams.execute().on('end',done);
});
});
Theglobfunctionsgetsallofourscriptsasynchronously,hencethecallback.Inthecallback,wemapthefilenamestoBrowserifystreams,thatistosaythatforeachfilename,afunctionliketheonewehadbeforethisisexecuted.Oncewehaveexecutedallthestreams,wesignaldonetoGulp.Thispseudo-codeisactuallyprettyclosetowhatweneedtodoalready.Wecanusetheevent-streammodule(https://www.npmjs.com/package/event-stream)towaitforallthestreamstocomplete:
npminstallglob--save-dev
npminstallevent-stream--save-dev
Wecanusethemodulesasfollows:
varglob=require('glob');
varbrowserify=require('browserify');
varsource=require('vinyl-source-stream');
vares=require('event-stream');
gulp.task('browserify',function(done){
glob('./scripts/*.js',function(err,files){
if(err){
done(err);
}
varstreams=files.map(function(file){
returnbrowserify({
entries:[file],
debug:true
})
.bundle()
.pipe(source(file))
.pipe(gulp.dest('prod'));
});
es.merge(streams).on('end',done);
});
});
Thisintroducesjustonemoretinyproblem.Intheprevioussample,weBrowserifiedjustonefile,sointhesourcefunctionwecouldrenameit.Wenowgetavariablenumberoffileswithdifferentnames,sowedonotknowwhatnametopasstothesourcefunctionexceptfortheoriginal.Wecouldkeeptheoriginalname,butIwouldliketorenameitto.bundle.jsanyway.Wecanusethegulp-renamemodule(https://www.npmjs.com/package/gulp-rename)forthistask.Asanaddedbonus,wecanremovethefolderstructure,soinsteadofwritingto'prod',wemustnowwriteto'prod/scripts',whichgivesusalittlemoreflexibility:
npminstallgulp-rename--save-dev
Andnowwejustneedtoaddanadditionalpipedfunctiontoourpipechain:
varrename=require('gulp-rename');
[...]
.bundle()
.pipe(source(file))
.pipe(rename({
extname:'.bundle.js',
dirname:''
}))
.pipe(gulp.dest('prod/scripts'));
Andthereyouhaveit,allyourJavaScriptfilesnicelybundled!Donoticethatwenowalsobundleutils.jsandrepository.jseventhoughweneverusethem.Youstillwanttowriteyourbundlestoyourscriptsbundlesdirectoryfordebuggingaswell.Youcansimplyaddanothergulp.dest:
.pipe(gulp.dest('prod/scripts'))
.pipe(gulp.dest('scripts/bundles'));
Unfortunately,Idonotdaresayit,thisprettymuchmessedupourKarmatests.
KarmaandBrowserifyAndso,wearedrawndeeperanddeeperintothewebofJavaScriptframeworksandmodules.OurKarmatestsstillusetheregularJavaScriptfilesthatnowusetherequirefunction,whichisnotfound.Andso,weneedtoinstallanotherplugin:
npminstallwatchify--save-dev
npminstallkarma-browserify--save-dev
npminstallbrfs--save-dev
WecannowchangeourKarmaconfiguration(s):
frameworks:['browserify','jasmine','mocha','chai'],
preprocessors:{
'../scripts/*.js':['browserify','coverage'],
'mocha-tests.js':['browserify']
},
browserify:{
debug:true,
transform:['brfs']
},
Wemustaddthe'browserify'frameworkandpreprocesssomefiles.The'scripts/*.js'filesuserequire,sotheymustbebrowserified.mocha-tests.jsdoesnotuserequireyet,butitteststherepositorydirectly,whichisnowbrowserified.So,wedoneedtochangethemocha-tests.jsfileaswell.Simplyaddthefollowingline:
varrepository=require('../scripts/repository.js');
AllofourAngular.jstestsmakeuseoftheAngular.jsappsandcontrollers,whichdorequire,butdonotexposeit.So,withthesefewchanges,ourKarmatestsshouldrunagain.Donotforgettochangetheconfigurationforourminifiedfilesaswell.
OurProtractortestsareunaffected,becausetheyusetheoriginalsourcefilesthatalreadymakeuseofthebrowserifiedscripts.
Unfortunately,Browserifybreaksourcoverage.Thecoveragepluginworksontheoriginalsourcefiles,butBrowserifycreatesnewsources.WhatweneedtodoistaketheBrowserifyscriptsandruncoverageonthosefiles.Forthis,we
canuseyetanotherplugin,Istanbul.Istanbul(https://istanbul.js.org/)isapopularJavaScriptcodecoveragetool.Andthen,weneedapluginforourplugin,browserify-istanbul(https://www.npmjs.com/package/browserify-istanbul):
npminstallistanbul--save-dev
npminstallbrowserify-istanbul--save-dev
Luckily,theimplementationisverysimple:
module.exports=function(config){
varistanbul=require('browserify-istanbul');
config.set({
[...]
preprocessors:{
'../scripts/*.js':['browserify'],
'mocha-tests.js':['browserify']
},
browserify:{
debug:true,
configure:function(bundle){
bundle.on('prebundle',function(){
bundle
.transform('brfs')
.transform(istanbul({
defaultIgnore:true
}));
});
}
},
[...]
});
};
First,makesureyouremovethe'coverage'preprocessor.Itisnotgoingtowork.Second,intheBrowserifyoptions,wenowspecifyaconfigurefunction,whichgetsabundleobject.Onbundle,wecallthetransformfunctionanduseittotransformourscriptstotheIstanbulcoveragereports.Otherthanthat,nothingchanges.Westillneedthecoveragereporteranditssettingsdonotneedtochange.
ConcatenatingCSSThenextstepisasimpleone.SinceweonlyhavetwoCSSfiles,let'sbundlethem.ItisnotascomplicatedasourJavaScriptbundles.Justconcatenatethetwofilesandsavetheminasinglefile:
npminstallgulp-concat--save-dev
Theconcatstepisjustasinglestepanddoesexactlywhatyouexpectittodo:
varconcat=require('gulp-concat');
gulp.task('css-concat',function(){
returngulp.src('css/*.css')
.pipe(concat('all.css'))
.pipe(gulp.dest('prod/css'));
});
Theresultisafilecalledall.cssandcontainsthecontentsofalltheCSSfilesinthecssfolder.
ReplacingHTMLreferencesWenowhaveonlyoneproblemleft:ourdebugHTMLisreferencingscripts/bundles/some-bundle.jswhileourproductionbuildisreferencingscripts/some-bundle.js.Additionally,wewouldliketoreplacesomeCSSfileswiththeall.cssfilewecreated.WecanreplacepatternsinourHTMLusingthegulp-html-replacemodule(https://www.npmjs.com/package/gulp-html-replace).Formoregenericstringreplacement,checkoutgulp-replace(https://www.npmjs.com/package/gulp-replace):
npminstallgulp-replace--save-dev
npminstallgulp-html-replace--save-dev
Allwehavetodonowiseditourviewsandindicatethepartswewanttoreplace.Thisisfromtheindex.htmlfile:
<head>
<metacharset="UTF-8">
<title>CIWebShop</title>
<!--build:css-->
<linkrel="stylesheet"type="text/css"href="node_modules\bootstrap\dist\css\bootstrap.css">
<linkrel="stylesheet"type="text/css"href="css\layout.css">
<linkrel="stylesheet"type="text/css"href="css\utils.css">
<!--endbuild-->
<!--build:js-->
<scriptsrc="node_modules\angular\angular.js"></script>
<scriptsrc="node_modules\jquery\dist\jquery.js"></script>
<scriptsrc="node_modules\bootstrap\dist\js\bootstrap.js"></script>
<scriptsrc="scripts\bundles\index.bundle.js"></script>
<!--endbuild-->
</head>
Nowthatwehaveacssbuildstepandajsbuildstep,wecancreateataskthatreplacesitscontents.Luckyforus,wheneverwereplaceaCSSorJavaScriptfile,thepluginwilltakecareoftheHTMLelement,soweonlyneedtospecifyfilenames:
varhtmlreplace=require('gulp-html-replace');
gulp.task('html-replace',function(){
returngulp.src('index.html')
.pipe(htmlreplace({
css:[
'node_modules/bootstrap/dist/css/bootstrap.min.css',
'css/all.css'
],
js:[
'node_modules/angular/angular.min.js',
'node_modules/jquery/dist/jquery.min.js',
'node_modules/bootstrap/dist/js/bootstrap.min.js',
'scripts/index.bundle.js'
]
}))
.pipe(gulp.dest('prod/views'));
});
Asyousee,wecannowreplaceallnode_modulestogivethemthecorrectminifiedsources.Notethatthisallscriptshavethesamepathasourcurrentfiles,becausewhenwearereleasing,wearejustgoingtoreplaceoursourcefileswiththeminifiedproductionfiles.WecanreplaceourscriptsandCSSfileswiththeproductionversionsaswell.Thisisabitofahasslethough,doingthisforallourHTMLfiles.Wecanmakethiseasierbyusingpatterns:
.pipe(htmlreplace({
[...]
node_modules:[
'node_modules/angular/angular.min.js',
'node_modules/jquery/dist/jquery.min.js',
'node_modules/bootstrap/dist/js/bootstrap.min.js'
],
js:{
src:'scripts',
tpl:'<scriptsrc="%s/%f.bundle.js"></script>'
}
}))
Asyoucansee,Ihaveaddedanextranode_modulesbuildstep.Wecouldrequiretheminourscripts,andweprobablyshould,butIalsowantedtoshowyouhowtosimplyreplacesomefileswiththeirminifiedversions.Inthejsbuildstep,Iamreplacing%s,whichreturnsthecurrentfoldername,withscripts.The%fvariablereturnsthenameofthecurrent(HTML)filebeingprocessedwithoutextension(use%etoincludeextensions).Wearenotoverwritingit,soitreturns'index'.SincewenameourJavaScriptfilesthesameasourHTMLfiles,wecanjustappend.bundle.jstoitandwewillhavescripts/index.bundle.js.ThiswillworkforallourHTMLpages.
Onelastthing:ouranchortagsinindexreferenceotherviewsintheviewsfolderandtheviewsintheviewfolderreferencetheindexpagebygoingapageup.Oncewehaveminifiedeverything,theyareallinthesamefolder,soyourlinkswillbreak.Thereisasimplesolutionusinggulp-concat:
varreplace=require('gulp-replace');
[...]
returngulp.src(['index.html','views/*.html'])
.pipe(replace('href="views','href="'))
.pipe(replace('href=".."','href="'))
.pipe(htmlreplace({
[...]
Thiswillreplaceallinstancesof'href="views'(secondtoescapethebackslash)and'href="..'with'href="'.This,effectively,fixesallofourlinks.
PuttingitalltogetherBynow,youareprobablywonderingifitisallworthit.Thereisjustsomuchtodo,withsomanypluginsandsomanytasksanddependencies.Well,fearnot.Wecancombinealotoftasksinonetasktomakeiteasier.
WearegoingtodoallofourJavaScript-relatedtasksinasingletask.Wenowhavethetaskslint,browserify,minify-js,test,andtest-min.Unfortunately,thelintandteststepsneedtoremainseparate,butwecanBrowserifyandminifyinthesamestep.TorunadditionalpluginsafterBrowserify,weneedtobufferthevinylstreamsusingthevinyl-buffermodule:
npminstallvinyl-buffer--save-dev
Afterthat,wecanjustpipebrowserify,source,buffer,size,minify,anddestinasingletask:
returnbrowserify({
entries:[file],
debug:true
})
.bundle()
.pipe(source(file))
.pipe(buffer())
.pipe(rename({
extname:'.bundle.js',
dirname:''
}))
.pipe(gulp.dest('scripts/bundles'))
.pipe(size({
title:'Beforeminification',
showFiles:true
}))
.pipe(minify(minifyOpts))
.pipe(size({
title:'Afterminification',
showFiles:true
}))
.pipe(gulp.dest('prod/scripts'));
Itisquiteabitofcode,buthandlesmostofyourJavaScriptissuesinonego!
WecangiveourCSSandHTMLtasksthesametreatment:
gulp.task('css',['clean'],function(){
returngulp.src('css/*.css')
.pipe(concat('all.css'))
.pipe(cleancss({
compatibility:'ie9'
}))
.pipe(gulp.dest('prod/css'));
})
.task('html',['clean'],function(){
returngulp.src(['index.html','views/*.html'])
.pipe(replace('href="views','href="'))
.pipe(replace('href=".."','href="'))
.pipe(htmlreplace({
[...]
}))
.pipe(htmlmin({
collapseWhitespace:true
}))
.pipe(gulp.dest('prod/views'));
});
Theycanallrunsimultaneously,butalldependonclean.Thecleantaskshouldbechangedaswell.Cleanneedsnodependenciesandshouldcleanthescripts/bundlesfolderaswell:
gulp.task('clean',function(){
del(['prod/*','scripts/bundles/*']);
});
gulp.task('build',['js','css','html']);
Wecannowrungulpcleantocleaneverythingandthengulpjs,gulpcss,orgulphtmltobuildaspecificpartoftheapplication.Runninggulpbuildwillsimplycleanourbuildfilesandminifyandreplaceeverythingagain.Youcouldcreateseparatecleantasksforjs,css,andhtmlaswell,butIdonotreallyseeaneedforit.
Thatleavesonlythelintingandtesting.Ourtestsare,obviously,dependentonourbuild,becauseweneedourbundledandminifiedscripts.Thelinting,however,isnotdependingonanything.Otherthanthat,thosetaskscanstayastheyare:
gulp.task('lint',function(){
[...]
gulp.task('test',['build'],function(done){
[...]
gulp.task('test-min',['test'],function(done){
[...]
Now,wecanchangeourdefaulttasktofireupthelint,test,andtest-mintasks.Noticethatthetest-mintaskisdependingonthetesttask.Sinceboththetaskscleanandbuild,onecleanmightundotheother'sbuild,solet'sjustnotgothere
andrunthemoneaftertheother:
gulp.task('default',['lint','test-min']);
Bytheway,inthesourcefile,youwillseethatIhavechainedthetaskfunctioncallsforaslightlylessbloatedgulpfile:
gulp.task([...]
})
.task([...]
})
.task('build',['js','css','html'])
.task('default',['lint','test','test-min']);
Thelintingnowrunsduringthebuildand,whenitfails,thetestswillnotbefired(unlesstheyarealreadyrunning).DuetotheasynchronousnatureofGulp,thelintingcanbeabitweird.Ofcourse,youcantweakitifyouwant.
Asicingonthecake,wearegoingtoautomateeverythingwejustcreated.AssoonasyouchangeaCSS,JavaScript,orHTMLfile,wewantanewbuild,andwhenwechangeaJavaScriptfile,wealsowantourlinterandteststorun:
gulp.watch(['css/*.css','views/*.html','index.html'],['build']);
gulp.watch('scripts/*.js',['default']);
Andthatisit.Everythingisnowautomated!Onyourlocalmachine,thatis.
Checkwhethereverythingstillworksby(manually)runningyourProtractortests.
Also,sinceyourentirebundlesandprodfolderaregeneratedfromyourbuild,youmaywanttoaddittoyour.gitignorefile.Runningasinglebuildwillprobablymarkallyourgeneratedfilesaschangedevenwhenyouonlyeditedasinglesourcefile:
**/prod/**
**/bundles/**
Ihaveshownyouquiteafewtasksyoucandowhenworkingwithfrontendtechnologies.Itseemslikealot(althoughsomeofitisbecausewearestillusingthefileprotocol),butitisreallygoodpracticetominify,lint,andtestyourfrontendfilesassoonaspossible.Optionally,youcanleavetheminifiedtestsandrunthemmanuallywhenyouaredonefixingsomecode.TheBrowserifypart
isjustabitofapain,butitisaone-timesetup.Afterthat,youhaveallthegoodnessofrequireonyourbrowser,soIthinkitisdefinitelyworthit.Bytheway,wehavenotyetlintedourCSSandHTMLfiles,soyoucanaddthoseyourselfifyoulike.
SummaryInthischapter,wecametothecoreofCI,automation.UsingGulp,weautomatedourbuild,linting,andtesting.Thatisallverynice,butfornow,itonlyworkslocally.YoucanstillforgettorunGulp,orignoreit,andcommitabrokenbuild.Inthenextchapter,wearegoingtotakeacloserlookatJenkins.Jenkins,likeGulp,isallaboutautomation.WhileGulpandJenkinsarenothingalike,besidesthefactthattheybothassistinautomation,themostimportantdifferenceisthatJenkinsrunsonaserverandwillcheckyourcommitevenifyoudidnot.
AutomationwithJenkinsSofar,wehaveprettymuchautomatedourentirebuild,includingtesting,onourlocalcomputer.Unfortunately,westillneedtomanuallystarttheautomationprocessandthatisnotsomethingwecanenforcebeforeacommit.Luckily,wecankickoffthebuildprocessonacommitfromourserver.ThisiswhereJenkinscomesintoplay.WithJenkins,wecanpollforchangesonourGitrepositoryandrunthebuildprocessautomatically.Whenabuildfails,Jenkinscansendanemailtotheentireteamtoletthemknowsomeonebrokethebuildandthatitshouldbefixed.Inthischapter,wearegoingtoexploreJenkinsinmoredepthtoautomateourbuilduponeverycommit.
Jenkinshasatonofsettings,options,andpluginsandsomepluginshaveanothertonofsettingsandoptions.NottomentionthatJenkinsandpluginskeepchangingwitheachnewupdate.Itisimpossibleforme(oranyone)tocoverthemall.However,Jenkinsanditspluginsareprettywelldocumented.Alotoffieldshaveaniconwithaquestionmarknexttothemthatshowadditionalinformationofthefieldwhenclicked.Additionally,youshouldcheckthewikipageforeachplugin;theycanhelpyouingettingstartedwithaplugin.Inthischapter,Iamgoingtoguideyouthroughsomecommonusecases.YoushouldgetfamiliarwithJenkins,soyoucanfigureoutwhatappliestoyoupersonallyinyourdailyjobyourself.
Incaseyouforgot,youcanaccessJenkinsbystartingupyourVMandbrowsingtociserver:8080.
InstallingNode.jsandnpmFirstthingsfirst,forourbuild,weneedNode.jsandnpm(again)attheveryleast.LikeonWindows,wecaninstallNode.jsandgetnpmasabonus.WemustinstallthemonourCIserver.UnlikeJenkins,Node.jshasaninstallpackageinapt-get.Unfortunately,thisisanoldversionandwewanttousethelatestLTSversion.Soagain,wearegoingtorunsomearcaneLinuxcommands:
curl-sLhttps://deb.nodesource.com/setup_6.x|sudo-Ebash-
sudoapt-getinstall-ynodejs
TheLswitch(from-L)incurltellsittoredotherequestiftheresponsereturnsthattherequestedpagehasmoved.Weknowthepipecharacter;itgivestheoutputoftheleftsideofthepipe'sinputtotherightsideofthepipe.sudo-Ebash-willrunthebashcommandastherootuser(superuser).-Emeansthatanyenvironmentvariableswillbekept.Thelast-meansthatthestandardoutputisgivenasacommandtobash.Simplysaid,weareusingcurltodownloadsomescriptthatwepassasaparametertothebashprogram.IncaseyouwerewonderinghowIgotsosmart,Ididnot.ThisissimplypastedfromtheNode.jsdocumentation(https://nodejs.org/en/download/package-manager/#debian-and-ubuntu-based-linux-distributions).
Last,butnotleast,wecaninstallnodejs.The-yswitchinsudoapt-getinstall-ynodejsisoptionalandtellsapt-gettoassumeyasanyinputtoprompts(suchasAreyousureyouwishtoinstallthispackage(Y/n).
Tocheckwhethereverythingwassuccessfullyinstalled,youcanchecktheversionsofNode.jsandnpm:
nodejs-v
v6.9.4
npm-v
v3.10.10
CreatingaJenkinsprojectOurnextstepwillbetocreateaJenkinsproject.MakesurethewebshopprojectiscompletelypushedtoGit(yourlocalGitLabinstallation).Forclarity,everythinginsidetheChapter06codefolderofthecodesamples(intheGitHubrepositoryforthisbook)shouldbeinyourGitLabrepository.Personally,Ihaveputitintheweb-shoprepository.WehavecreatedthisrepositoryinChapter4,CreatingASimpleJavaScriptApp.WehavealreadycreatedaJenkinsprojectinChapter2,SettingUpaCIEnvironmentintheConfiguringJenkinssection,butIwilltakeyouthroughsomeofthestepsagain(youshouldalsoreallyfollowthestepsinChapter2,SettingUpaCIEnvironment,astheytakeyouthroughtheinstallationofsomepluginsandcredentials).Speakingofplugins,bemindfulthatalotofpluginsarecreatedandmaintainedbypeoplelikeyouandme,peoplewhohavetakenupprogrammingasahobby.Thesepeoplearenotpaidtocreateormaintaintheseplugins.Thesepeoplearebusyandmaynotfindthetime,need,ordrivetoupdatetheirplugin,leavingyouwithanoutdatedpluginthatdoesnotsupportJenkinsoryourlanguage'slatestfeatures.Itisalwaysgoodtocheckwhetherapluginisstillactivelymaintainedbeforeyouinstallit(thatisnottosayeverypluginthatisnotmaintainedisbad;somejustdowhattheymustanddoitwell).
SologintoJenkinsandclickNewItemontheleft-handsidemenu.Enteranitemname,eitherWebShop,Chapter7,orMyLittlePony,whateveryoufancy,althoughIrecommendanamethatsayssomethingabouttheprojectyouareworkingon.IhavenameditChapter7.Youwillnowbetakentotheconfigurationscreenofyournewproject.
Inournewproject,wewillwanttocheckouttheGitrepository.So,underSourceCodeManagement,selectGitandentertherepositoryURL.ThisistheURLthatisalsodisplayedontheGitLabprojectpage.YoushouldstillhaveyourcredentialsfromChapter2,SettingupaCIEnvironment,soselectthoseaswell.Ifyourcredentialschanged(becausewehavebeenplayingaroundwithGitandGitLababit),youcanaddnewcredentialsorchangeyourexistingone,whichyoucanfindundertheleft-handsidemenuoptionCredentialsonthemainpage.Aswithaproject,youcanselectacredentialandthenupdateordeleteit
fromtheleft-handsidemenu.
Onceyouhaveselectedyourcredentials,savetheprojectandbuilditfromtheprojectpage:
Youshouldseeablueballforsuccess.Whetheryouhaveablueballoraredball,youcanclicktheballnexttoyourbuildnumberandyouwillseeexactlywhatcommandsJenkinshasexecuted.Forexample,whenyouenterincorrectcredentials,theoutputwillbesomethingasfollows:
Startedbyuseradmin
[...]
>gitfetch--tags--progresshttp://ciserver/sander/web-shop.git+refs/heads/*:refs/remotes/origin/*
ERROR:Errorcloningremoterepo'origin'
hudson.plugins.git.GitException:Command"gitfetch--tags--progresshttp://ciserver/sander/web-shop.git+refs/heads/*:refs/remotes/origin/*"returnedstatuscode128:
stdout:
stderr:remote:HTTPBasic:Accessdenied
fatal:Authenticationfailedfor'http://ciserver/sander/web-shop.git/'
[...]
ERROR:null
Finished:FAILURE
YoucanalsoviewthisinformationbyclickingonthebuildnumberandthenclickingConsoleOutputfromtheleft-handsidemenu.Funfact,youcanviewitasitisexecuting.Youwillhavetobereallyquickforthistwosecondbuildto
seesomeaction,butonceyougetbiggerprojects,buildscantakeuptominutes(hoursreally,dependingonhowcrazyyouget).
Iamassumingeverythingwentwell.ThenextthingwewanttodoisassurethatJenkinshasreallypulledyourcodefromGit.Ontheprojectpage,thereisamenuitemlabeledWorkspace.YoucanclickittobrowsethefilesinsidetheJenkinsproject.Thisisalsowhereyoucanclearoutyourworkspace,whichmeansJenkinswillsimplydeleteallthefilesinsidetheproject.Thisissometimesusefulwhenyouaredealingwithsomebadcachingissues.Ofcourse,clearingyourworkspaceisnoproblematall,sinceyoucansimplyruntheprojectagainandJenkinswillpullyourcodefromGitoncemore.
YouwillnoticethatJenkinshaspulledallyoursourcefilesfromGit.Thenextthingwewanttodoisrestoreournpmpackages.OnceyourealizeJenkinsisjustrunningaseriesofconsolecommands,thingsgetprettyeasy.GotoyourprojectconfigurationandaddabuildstepExecuteShell.Putnpminstallinthecommandinput:
Nowbuildtheprojectagainandyouwillnoticethatitdownloadsallofyournpmmodules.Ifyoucheckouttheworkspace,youwillseethenode_modulesfolderwithallofthemodules.
ExecutingGulpinJenkinsRunningyourbuildshouldnowbeeasy,becauseitisjustanothershellcommand.TheonlyproblemisthatwedonothaveGulp,Karma,oranyothertoolinstalledgloballyonourVM,soagulporkarmastartcommandwillfail.Anycommand-linetoolinstalledthroughNode.jswillhavetheirexecutablesinthenode_modules.binfolder(whichiswhyweinstalledallthosetoolsinourprojectinadditiontogloballyonourdevelopmentmachine).Sowecannowsimplyrunnode_modules/.bin/gulp.Theshellstilloperatesfromtherootofourprojectandsowillalsouseourlocalgulpfile.
Unfortunately,alotwillgowrong.Yourtestswillfail,thebrowserswillnotstart,andyourjobwillneverfinish.Youmayhavealreadyguessedwhy.First,wedonothaveanybrowsersinstalledonourUbuntumachine.Wedonotevenhaveauserinterface!So,ofcourse,JenkinscannotstartIE,Edge,Chrome,andFirefox.Second,wehaveafilewatchinourgulpfile,whichpreventsGulpfromexitingandsoitpreventsourbuildfromeverfinishing.
Fornow,let'sjustgetittowork.Wehavetwooptions,changeourfiles,pushthemtoGit,runourprojectagain,andhopeitworks,orchangeourfilesmanuallyinourworkspace.ChangingthemmanuallyismuchfasterandwedonotgetatonoftrytofixJenkinsbuildcommitsinGit(whichwecould,ofcourse,putonapersistedbranch).WecouldalsotrytorunourbuildonourVMdirectlyintheshellandcheckouttheerrorlogs.Whilethatissometimesreallyuseful,asyougetdirectfeedback,itcanalsobeahassle,asyouneedtoreplicateyourentireJenkinsprojectmanually,whichisnotalwayseasyorpossible.
Solet'schangeourfilesdirectly.OnyourVM,browsetoyourJenkinsworkspaceusingcd"var/lib/jenkins/workspace/Chapter7".Wecannowchangethegulpfiletodisablethefilewatcherusingsudovigulpfile.js.Entereditmodebypressingi.Now,simplycommentoutthetwogulp.watchlinesattheendandsaveusingescapeandthen:wq:
cd"var/lib/jenkins/workspace/Chapter7"
sudovigulpfile.js
i
[..]
//gulp.watch(...);
//gulp.watch(...);
esc
:wq
Likewise,changeyourtest/karma.conf.jsandtest/karma.min.conf.jsfilesandsetthebrowserspropertyto['PhantomJS'].Remember,PhantomJSisaheadlessbrowser,iseasilyinstalledusingnpm,anddoesnotrequireauserinterface.Unlike,IE,Edge,Firefox,andChrome,itwillrunjustfineonourcurrentsystem.
Now,gobacktoyourJenkinsprojectandchangeSourceCodeManagementtoNone(sowedonotoverwriteourlocalchanges).Runtheprojectagainanditshouldsucceedthistime.Unfortunately,thereisnotaneasywaytotroubleshootissuesinJenkins.Sometimes,itisjustabitoftrialanderror(butsometimes,thatiswhatprogrammingisallabout).
Nowthatweknowtheproblem,wecanchangeourfilesonourdevelopmentmachineandpushthemtoGit.However,ifwechangeourgulpfile,itwillbreakourdevelopmentusageofGulp,andwedowantthat.Wecouldcopyourgulpfile,removethewatch,andusethenewoneinJenkins,butthatwouldmeanwehavetwogulpfilestomaintain.WhatwereallyneedisawaytospecifyourenvironmenttoGulp.Luckily,thisiseasyenoughusingthegulp-utilplugin,whichwealreadyhave.Ofcourse,westillneedtochangeourgulpfile.YoucanpassanyvariabletoGulpusing--something=valuefromthecommandline:
gulp--env=prod
Inourgulpfile,wecanreadanyenvironmentvariableusinggutil.env.something:
vargulp=require('gulp'),
[...]
gutil.log(gutil.colors.cyan('Environment:'),gutil.colors.blue(gutil.env.env));
gulp.task('clean',function(){
[...]
if(gutil.env.env!=='prod'){
gulp.watch(['css/*.css','views/*.html','index.html'],['build']);
gulp.watch('scripts/*.js',['default']);
}
Atthetopofourgulpfile,wecanlogthevalueofenv,usingcolorsforclarity.Atthebottomofthefile,wecanthenskipgulp.watchifwearerunningaprodbuild.Youmayuseanyvariableandvalueyoulike.Ihavecalledthevariableenvand
givenitthevalueofprod,butifyouwanttocallitmonkeywiththevalueastronaut,youcan.
Wehavethesameissuewithourbrowsers.Fornow,wewanttousedifferentbrowsersforaJenkinsbuildthanalocalbuild.Ourdevelopmentmachinehasallthebrowsersinstalled,whileUbuntuonlyhasPhantomJSinstalledthroughnpm.Withthenewenvvariable,wecaneasilyfixthisinthetestandtest-mintasks:
newkarma({
configFile:__dirname+'/test/karma.min.conf.js',
browsers:gutil.env.env==='prod'
?['PhantomJS']
:undefined
},[...]
Ifwearenotbuildingin'prod',weusethebrowsersasdefinedintheconfigurationfile,orelseweoverwriteitwithPhantomJS.CommitthesechangestoyourGitrepository,sowecanpullitinJenkins.
Now,intheJenkinsproject,wemustchangethejobtorunGulpwiththeprodenvironment:
npminstall
nodenode_modules/.bin/gulp--env=prod
IfyouhaveremovedtheGitsettingsunderSourceCodeManagement,besuretosetyourGitrepositorysoyoucangetthelatestversionofthegulpfile.Ifyousaveyourconfigurationandrunthebuilt,youshouldseethatitwillnowfinishwithinaminute.
Thereisonethingwearenotdoingyet,whichiscleaningourworkspaceforeverynewbuild.ThatmeansthatwedonothavetopullourentireprojectfromGiteverytimewestartabuildandthatwedonothavetoinstalleverynode_moduleoneverybuild.npminstallalonecostsabouttwominutesforoursmallproject;addtothatthetimeitcoststocheckoutlargeprojectsandskippingthesestepsmaysaveyouaboutfivepreciousminutes.However,itisgoodpracticetoalwaysstartacleanbuild.Forexample,npmpackagesmayberemovedfromnpm(ithappens)oryoumayremovefilesfromtheGitrepositoryyourself.Whenyoucleantheworkspacebeforeeachbuild,thesefileswouldbemissingonyournextbuild.Ifyoudonotremovethem,however,Jenkinswillusethecachedfilesandyouwillnotknowsomefilesaremissinguntilitistoolate.
UndertheGitsettings,youcanaddadditionalbehaviors;besuretoaddCleanbeforecheckout.Ifyousaveandrunyourjobagain,itwilltakeconsiderablylongerthanthelasttimeyoubuiltit,butJenkinswillcleartheentireworkspacebeforebuilding.
IfyouarerunningJenkinsonWindows,youcanusetheExecuteWindowsbatchcommandstepinsteadoftheExecuteShellstep.Otherthanthat,everythinglooksthesame.Ofcourse,especiallywhenyourJenkinsmachineisthesameasyourdevelopmentmachine,JenkinsshouldbeabletorunIE,Edge,Firefox,andChrome.ThisistrueforFirefoxandChrome,butIEandEdgecannotrununderthedefaultLocalSystemaccountunderwhichJenkinsisrunning.Wewilllookintotheseproblemslater;fornow,followwiththePhantomJSbrowserinstead.
PublishingtestresultsNowthatwehaverunsometestsusingKarmawewillalsogetourtestreports.Wehadthree;theJUnitreport,whichshowsusthetestresults;theCoberturareports,showingushowmuchofourcodeiscoveredbytests;andtheHTMLcoveragereport,showingexactlywhatlinesaretestedandnottested.
JUnitreportTheeasiestonetoimplementistheJUnitreportasitcomesstraightoutofthebox.SimplyconfigurethePublishJUnittestresultreportpost-buildaction.Thereportcan,ofcourse,befoundinthetest/junit/folder(afterbuilding).WeneedanyXMLfiles,sothecompletepatternforthereportsfieldistest/junit/*.xml:
Afterthat,youcanmakeanotherbuildandthetestresultswillbepublishedtoJenkins.Youcanfindtheresultsinyourbuildontheleft-handsidemenuunderTestResult.Alternatively,thereisadrop-downmenuinthebuildoverviewontheleft-handside(theonewiththeredandblueballs).ThedropdownhasaTestResultmenuitem,whichtakesyoutothetestresultspageofthebuild.Now,whenyoumessupyourcode,oratest,andatestfails,Jenkinswillfailthebuildandyouwillknowsomethingisnotasitshouldbe.Thetestreportshouldgiveyouaprettydetailedoverviewofwhatwentwrong:
Additionally,onyourprojectpage,youshouldseeatrendgraphthatshowshowmanytestssucceeded(andfailed)inyourpreviousbuilds(uptoafewbuildsback).
TheJUnitreporterpluginhastheveryunfortunatehabitofnotpublishingoldreports.Iamnotsurewhatthedefinitionofoldis,butIbelieveitisafewseconds.IonceranintoanissuewhereIwouldgeneratetheJUnitreports,thendosomeotherstuffthattookafewminutes,andthenpublishthereports.ImaginemysurprisewhenIwasgreetedbythemessageERROR:Step'PublishJUnittestresultreport'failed:Testreportswerefoundbutnoneofthemarenew.
Didtestsrun?.TheleastyoucandotomakesuresuchanerrordoesnotoccurtoooftenistocleanyourJUnitreportsbeforeeverybuild.Ifyoueverrunintotheissuebecauseyourbuildistakingtoolong,goodluck.Inevergotitproperlyfixed.
CoberturareportNextupistheCoberturareport.WewillneedanadditionalJenkinspluginforthisone.SogototheJenkinsmanagerand,fromthere,tothepluginmanager.SearchforCoberturaplugin(https://wiki.jenkins-ci.org/display/JENKINS/Cobertura+Plugin)andinstallitwithoutrestarting(thiswillalsoinstalltheMavenIntegrationplugin).Gobacktoyourprojectconfigurationandaddanewpost-buildaction,PublishCoberturaCoverageReport,whichwasnottherebefore.Usetest/coverage/cobertura/*.xmlasthereportpattern.Now,ifyousaveandruntheproject,youwillseethatyourbuildfailsevenwhenallyourtestssucceed.ThatisbecausetheCoberturamakesyourbuildfailifyoudonothaveaminimumcoveragepercentage.WehaveseenthisinKarmaaswell.
Inyourpost-buildaction,clickontheAdvancedbuttoninthelower-rightcorner.Younowgetacoupleofadditionaloptions,butthemostinterestingareCoverageMetricTargets.Youcansetaminimumthresholdforastable,failing,orunstablebuild.Unfortunately,theoptionsarenotasadvancedasthoseinKarma.Wecanonlysetoverallthresholdsandwecannotexcludefiles.Sinceourcoverageisnotallthatgreat,youcaneithersetthetargetsveryloworremovethemetricscompletely.YourKarmarunwillstillfailyourbuildifyourthresholddoesnotmeetyourKarmasettings.Speakingoffailingbuilds,theCoberturareporterhasanadvancedoptionConsideronlystablebuilds,whichmakessensetoenable.Whenyourbuildoratestfails,thereisagoodchancethatitaffectsyourcoverage.Ifanexceptionisraised,forexample,furthercodewillnotbeexecutedandsoyourcoverageislowerthanwhenyourbuildwouldhavepassed.However,inthiscase,italsomeansthatwhenKarmafailsyourbuildbecauseyourcoverageistoolow,yourcoverageisnotpublished.Forthatreason,Iamkeepingtheoptiondisabled,butIthoughtyoushouldstillknowaboutit.
Whenyourunyourbuildanditsucceeds,youcanseesomenicetrendgraphsonyourcodecoverage.Ideally,thegraphshouldgoupovertime.Thegraphshouldatleasthaveastraightlineastimepasses.Ifthelinegoesdown,youknowyourcoverageisgoinginthewrongdirection.ThereisalsoaCoberturaCoverageReportoptionontheleft-handsidemenunow.Whenyouclickit,youwillbe
takentoareportthatisnotunliketheHTMLreportKarmagenerates.
Inthisreport,youcanseewhichfilesandlinesweretestedandwhichwerenot:
HTMLreportKarmaalsogeneratesanHTMLreportthatshowsexactlywhatlinesweretested.TheJenkinsreportalreadyshowsprettymuchthesame,butIprefertheKarmareport.Besides,IwanttoshowyouhowtopublishHTMLreports.YoucanpublishprettymuchanyHTMLpage.First,weneedtoinstallanadditionalplugin.Gotothepluginmanager,findtheHTMLPublisherplugin(https://wiki.jenkins-ci.org/display/JENKINS/HTML+Publisher+Plugin),andinstallit.Afterthat,gobacktoyourprojectconfiguration.Youcannowaddanotherpost-buildaction,PublishHTMLReports.Youwillgetaboxwhereyoucanaddreports,soyoucanonlyaddtheactiononce,butthenaddasmanyreportsasyoulike.So,addoneandspecifytest/coverage/htmlastheHTMLdirectorytoarchive.
Thedefaultindexpage,index.html,isalreadysetandwecankeepthedefault.Youcangiveyourreportatitle,suchasCoverageHTML.Makeanotherbuildofyourprojectandyoushouldseealinktoyourreportontheleft-handsidemenu,rightunderneathCoverageReportfromtheCoberturaplugin.Youwillfindyourreportpublishedthere.
TheHTMLPublisherpluginhassomeadditionalsettings,whichcanbesetthroughthePublishingoptions...buttonontheright-handsideunderyourHTMLreportsettings.Theyarewelldocumentedunderthequestionmarkicon,soIwillleaveittoyoutotrythemallout.
BuildtriggersWheneverwepushcodechangestoGit,wewantourJenkinsprojecttostartassoonaspossible.Afterall,thesoonerweknowsomethingisbroke,theeasieritisforustofixit.YouhaveprobablyalreadyseentheBuildTriggerssectionintheconfigurationofyourJenkinsproject.Therearefourbuildtriggerscurrentlyavailabletous.
First,wecanbuildtheprojectremotely,whichwewillnotdo.Second,wecanbuildafterotherprojects,whichishandywhenyouhavemultipleprojectsthatdependoneachother.Nextistheperiodicbuild,whichshouldspeakforitself.Andlast,wehavethepollSCMoption,whichpollsyourSCM-inourcaseGit-forchangesandtriggersabuildwhensomethingchanged.
Fornow,wewilllookattheperiodicbuildandthepollSCMtriggers.Bothusecronsyntax(croncomesfromtheGreekwordfortime,chronos),whichIfindratherdifficulttograsp.Luckily,eachofthetriggeroptionshasaquestionmarkbehindit,whichgivesprettydetailedinformationaboutthespecifictrigger.Cronisalsoexplainedinthosehelptexts,soIsuggestyoureadthem.Ifyouarestuckoncron,donotworry.ManypeopleareandGooglewillhelpyouingivingyouthe(almost)exactcronexpressionyouneed.
BuildperiodicallyThefirstbuildtriggerwearegoingtouseistheperiodicbuilttrigger.Thistriggerisprettyhandywhenyouhavesometasksthatcanbeexecutedeveryfewhours,onceaday,andsoon.Forourbuild,ithasadownside.Cronisflexibleenoughtoexpressaschedulelike-everyhourbetween9AMand5PM(or9:00and17:00),soyoucandoperiodicbuildsduringofficehours.Ofcourse,youcanalsostilltriggerabuildmanuallyincaseyouareworkingovertime.Thedownsidetothistrigger,forus,isthatwemayhavenocommitsforafewhours,butwewillhavebuilds.Orworse,wehavemultiplecommitsinonehoursandtheywillallbetestedatthesametime.
Whenourbuildfails,wewillnotknowwhatcommitmadeitfailandfindingtheproblemwillbeharder.Anyway,let'sjustbuildeveryminute,sowecanseethetriggerinaction.SimplycheckBuildperiodicallyandputinthecronscheduleforeveryminute,whichis*****(everypossiblevalueforminute,hour,dayofmonth,month,anddayofweek).JenkinsisprettysmartandevengivesawarningDoyoureallymean"everyminute"....Yes!Wedo.Saveyourconfigurationandsimplywaitafewminutes.Youwillseeabuildbeingtriggeredeveryminute:
PollSCMThenexttriggerwearegoingtotryistheSCMpolling.Thisisagoodoptionwhenyouwanttoperiodicallycheckfornewcommitstoyourrepository.Thedownsidetothisoptionisthatyourcommitsarenotbeingbuildinstantlyandthatallcommitssincethelastbuildwillbebuildallatonce.However,whenyouhavelotsofcommitsinshortperiodsoftimethisoptionmayalsosaveyoutimeandresources.Noticethatthistriggerisnotmutuallyexclusivewiththeperiodicbuild(youcanpollyourSCMandbuildeveryfewhoursregardlessofcommits).TocheckouttheSCMpolling,unchecktheperiodicbuild,andchecktheSCMpollingoption.Putinthescheduleforeveryminuteagain.Thistime,youwillnotseeanybuildsbeingtriggered;atleast,noteveryminute.Youmayhaveguessed,butweneedtomakesomechangetoourcodeandcommitittoGit.Yourchangecanbeanything,justaddaspaceornewlinesomewhereandcommitittoyourrepository.Onceyouhavecommittedyourchange,youwillhavetowaitanotherminute(atmost)andyoushouldseeabuildbeingstarted.Waitforafewmoreminutesandyoushouldnotseeanyadditionalbuildsbeingstarteduntilyoumakeanothercommit(maybetoreverseyourchange).Now,ifyoucommitanycodethatbreaksyourbuild,youshouldgetanautomaticbuildthatfails.This,inturn,shouldtriggeryoutofixthatbuild!Personally,Ifindthataperiodofabout15minutesisgoodenoughifyouhavetopollyourSCM.Speakingofchangesinyourcode,Jenkinstellsyouwhatcommitsarenewinanyspecificbuild.Justgotothespecificbuildpageanditwillshowyouthechangesfromthepreviousbuildatthetop.Again,thereisalsoabuttononthemenuthattakesyoutothesamepage.
Youhaveprobablygotquiteafewbuildsbynow.Ifyouhavenotfounditalready,atthetopofyourprojectconfigurationisaDiscardoldbuildscheckbox.Youprobablywanttocheckitandkeeparoundfivetotenbuildsatatime.Youmayalsospecifytokeepbuildsforanumberofdays.Youcanalsodobothofcourse,clearyourbuildsafteraweek,butkeepnomorethan10buildsatatime.
OncommitGettingaJenkinsbuildtostartoncommitisalittletricky,butwellworthit.Unlikeapoll,thisoptioncanbuildanycommitinstantlysoyougetdirectfeedbackoneverysinglecommit.WeneedtheGitLabplugin,soinstallitand,thistime,restartaftersuccess(checkRestartJenkinswheninstallationiscompleteandnojobsarerunning).AfteryouhaveinstalledthepluginandJenkinshasrestarted,headovertoGitLab.InGitLab,gototheprofilesettingsofyouraccount.Now,gotoAccessTokens.CreateanaccesstokenbyenteringanameandselecttheAPIscope.Makesureyoucopythetoken,asthiswillbetheonlytimeyougettoseeit.Ifyouloseit,itisreallygone.Don'tworry,youcancreateasmanyaccesstokensasyoulike.Now,withthefreshlycreatedaccesstoken,gotoJenkinsManagementandConfigureSystem.TherewillbeanewsectioninConfigureSystemthatwasnottherebefore,GitLab.Filloutthefieldsasyouseefit.DisabletheEnableauthenticationfor'/project'end-pointcheckbox.Givetheconnectionaname,suchasLocalGitLab.ThehostURLisnothttp://ciserver,buthttp://localhostorhttp://127.0.0.1(sinceJenkinsandGitLabarerunningonthesamemachineandciserverisjustanalias,ifyoufollowedmyexamples).Youwillneedtocreateanewcredential.ChoosetheGitLabAPItokenkindanduseyouraccesstoken.GiveitanIDsuchasGitLabToken,soyouknowwhatthisisfor.
Testyourconnectionandifitsucceeds,JenkinsshouldnowbeabletoreceivemessagesfromGitLab:
Nowthatthisisdone,wecanconfigureourprojecttotriggeronGitLabhooks.Simplygotoyourprojectconfiguration,disableallthecurrenttriggers,andselectthenewtrigger,BuildwhenachangeispushedtoGitLab.GitLabCIServiceURL:http://ciserver:8080/project/Chapter%207(whereChapter%207isyourprojectname).Herecomesthetrickypartthatcostmeagoodhourwhileconfiguringthis.Yourbrowserwillshowthatyourcurrentpageishttp://ciserver:8080/job/name,buttheGitLabpluginlistensathttp://ciserver:8080/project/name(andagain,replaceciserverwithlocalhostor127.0.0.1).Thisisimportantinthenextstep.ConfiguringaGitLabwebhook.Fornow,wedonothavetochangeanyconfigurationonthistrigger.Anoptional,butveryniceaddition,isthepost-buildactionPublishbuildstatustoGitLabcommit(GitLab8.1+required).Addthispost-buildactionandwatchthemagichappeninthefinalstep.
Next,weneedtotellGitLabtogiveJenkinsalittlepushwhenacommitispushed.So,gotoyourGitLabprojectandfindtheWebhookssettings.IntheURLfield,youneedtoputtheURLofyourJenkinsproject(project,notjob),http://127.0.0.1:8080/project/Chapter%207.However,thiswillnotworkinourcase.Jenkinswillneedtologin,butwehavenotspecifiedanyusernameandpassword.Inalaterchapter,wewillenableSSH,butfornow,weneedtoaddtheusernameandpasswordtotheURL.ThenewURListhenhttp://username:[email protected]:8080/project/Chapter%207.MakesureyouuncheckEnableSSLverification.
Otherthanthat,theonlyhookweneedisPushevents.Now,createthewebhook.Youcantestitandifallgoeswell,youwillgetamessageHookexecutedsuccessfully:HTTP200.Toverifythateverythingreallyworks,gobacktoJenkinsandverifythatabuildwastriggered(ifyoureplaceprojectwithjobintheURL,youwillgetthesamemessage,butnobuildisactuallytriggered):
Makeachangetothesamefile,addsomewhitespace,commitit,andpushittoyourGitrepository.Andnow,themomentwehaveallbeenwaitingfor,checkwhetheryourJenkinstriggersabuildautomatically.Ifitdoes,congratulations!
ThisisactuallyaprettycoolmilestoneinourCIprocess.
Thereisjustonemorething,ifyouhaveaddedthepost-buildaction,headovertoyourGitLabprojectpage.Ifyoucheckoutyourcommits,youwillfindthattheyhavealittleiconnexttothemindicatingthestatusofyourcommit.Itnowbecomesprettyeasytorecognizewhichcommitsbrokeyourbuild:
Anothersurprise,ifyouhaveconfiguredyouractualemailaddress,thereisagoodchanceyouhavegotanemailfromGitLabtellingyouthatyourbuildpassed.Ifyoudonothaveit,makesuretoalsocheckyourspamfolder.YoucanchangeyouremailsettingsinGitLabunderyouraccount'snotificationsettings.
SettingupemailnotificationsNowthatwehaveconfiguredaprojectandknowitcanfailwhenwemessupourcode,weneedtosetupsomesortofnotificationsoweknowwhenourbuildbreaksorgetsfixed.YoucouldhaveyourJenkinsprojectopenalldayandcheckitaftereverycommit,butweneedtomakesureeveryoneisnotifiedwhenabuildfails.Sowearegoingtosetupemailnotifications.IamgoingtoshowyouthisexampleusingaGmailaccount(Googlemail),butthiswillworkwithanyemailprovideraslongasyouhaveanSMTPserverandlogincredentials.GototheJenkinssystemconfigurationandfindtheemailnotification(nearthebottom).Google'sSMTPserverissmtp.gmail.com,sobesuretoputthatinSMTPserverunderAdvanced.CheckUseSMTPAuthenticationanduseyourGmailusernameandpassword(thatis,thecredentialsyouusetologintoGMail).Also,checkUseSSLandsettheportto465,whichisthedefaultforSMTPS(SMTPwithSSL).ChecktheTestconfigurationbysendingteste-mailboxandentertheemailaddressyouwanttosendyourtestemailto.Ifallwentwell,youshouldgetanemailsayingThisistestemail#1sentfromJenkins.
Youwillfindthattheemailwassentbyaddressnotconfiguredyet.GobacktoyoursystemconfigurationandfindtheJenkinsLocationsection.SetSystemAdmine-mailaddresstosomethinglikeJenkinsadmin<[email protected]>.SendatestemailagainandyoushouldseethesenderisJenkinsadmin.
Wecannowsetupemailnotificationsinourproject.Addthepost-buildactionE-mailNotificationtoyourprojectandaddtherecipients,forexample,[email protected],probably,yourpersonalemail.Makethebuildfail(forexample,byreturning-1inthebatchscriptstep)andcheckyourinbox.YoushouldgetanemailwiththesubjectBuildfailedinJenkins:Chapter7#69andalinktothebuildanddetailedconsoleoutputinthebody.Now,fixyourbuildandyoushouldgetanotheremailwiththesubjectJenkinsbuildisbacktonormal:Chapter7#70.Ifyoumakeanotherbuild,youwillnotgetanotheremail,asyouwillonlygetemailsforbrokenbuildsorwhenthebuildgoesfromfailedstatustosuccess.
SettingupSonarQubeNowthatwecanrunourGulptasksandtriggerabuildautomaticallyoncommit,itistimetoaddthenextsteptowardsqualitycode,SonarQube.Wehavealreadyinstalled,configured,andusedSonarQubeinChapter2,SettingUpaCIEnvironment,soIassumeyouhaveitreadyforuse.Ifthingsdonotwork,besuretoreviewChapter2,SettingUpaCIEnvironment,andthepartonSonarQubeinparticular.Hereisalittlereminder:SonarQubeisaccessibleonciserver:9000.So,gotoyourprojectconfigurationandaddtheExecuteSonarQubeScannerbuildstep.PutthefollowingconfigurationintheAnalysisPropertiesfield:
sonar.projectKey=chapter7
sonar.projectName=Chapter7
sonar.projectVersion=1.0
sonar.sources=.
sonar.exclusions=node_modules/**,prod/**,scripts/bundles/**,test/**
Istronglysuggestyouexcludenode_modules(becausethesearenotyourfiles)andprodandbundles(becausetheyaregenerated).Ihaveexcludedtestsaswell,althoughyoucouldaddthemtomakesureyourtestsareproperlywritten.Now,runtheprojectinJenkinsandchecktheprojectinSonarQube.BecausetheSonarQubedefaultqualityprofilesarequiteforgivingandwealreadytookcareofourissuesusingJSHintinGulp,ourprojectisdoingprettygood!Notasingleissueinfact.However,weshouldnotusethedefaultSonarQubequalityprofile.Creatingyourownisahellofajob;SonarQubehashundredsofrulesthatyoucanactivateatdifferentlevelsofseverity.Luckily,youcancopythedefaultqualityprofilesandgofromthere.InSonarQube,gotoQualityProfilesandcopytheJavaScriptSonarwayprofile(fromthedrop-downmenu).Callitwhateveryoulike,IcalledmineSanderway.Youcannowactivatemorerules.Forexample,enableSourcefilesshouldhaveasufficientdensityofcommentlines(atthedefault25%).Becausewehavenotwrittenasinglecomment,thiswillbesuretoaddsometechnicaldebttoyourproject.Youcansetthenewqualityprofileasyourdefaultprofileoryoucanassignittoyourproject.GotoyourqualityprofilepageandChangeProjects.Now,runyourprojectagaininJenkins.Youshouldseesometechnicaldebtbeingadded.
Next,wearegoingtochangethequalitygate.Thequalitygatedefineswhetheryourproject'scodefailsorsucceedstomeetcertainqualitycriteria.Thecriteriaforqualityisdefinedinthequalityprofile.Forexample,ourcriteriaforqualityisthatallthefilesshouldhaveaminimumamountofcomments.Failingtodosoresultsinacodesmell.Ourqualitygatedefineshowmanycodesmellsourcodemayhavebeforewefailourproject.Let'smakeourqualitygatefail.
Wehaveeightfilesthatarenotproperlycommentedaccordingtoournewqualityprofile.So,gotothequalitygates,copytheSonarQubeway(again,IhavenamedthenewoneSanderway),andaddanewcondition.AddtheCodeSmellsconditionandmakeiterroratgreaterthanfive.Assignthenewqualitygatetoourproject.Now,runthebuildagaininJenkins.YoushouldnowseethatSonarQubereportsthatourprojectfailstomeetitsqualitygate'scriteria.YoucanalsoseethisontheJenkinsprojectpage.Unfortunately,ourJenkinsbuildstillsucceeds.Wehaveafewoptions,installtheQualityGatespluginorsendanemailfromSonarQube(orboth).
GotothePluginManagerandinstalltheQualityGatesplugin(https://wiki.jenkins-ci.org/display/JENKINS/Quality+Gates+Plugin).Thepluginisabitbuggyifyoudonotconfigureitcorrectly,butitisnotveryhardtoconfigure.Now,gototheJenkinsSystemConfigurationandfindtheconfigurationfortheQualityGatesplugin.Unfortunately,weneedtoaddourSonarQubeinstanceagain.Theplugindoesnotsupporttokenauthentication,sowearestuckwithausernameandpasswordauthentication.So,givetheinstanceaname,specifytheURL(youcanleaveitemptyforthedefault,whichitisifyoufollowedmyexamples)andyourusernameandpassword.Now,gotoyourprojectandaddthequalitygatespost-buildstep.SpecifyChapter7asprojectkey.RunyourbuildagainandyoushouldseethebuildfailbecausetheSonarQubeanalysisfails.Now,ifyouremovethequalitygatefromtheSonarQubeprojectandrunanotherbuild,youwillseethatbothSonarQubeandJenkinssucceedagain:
Sofar,wehaveputtheSonarQubeconfigurationinourJenkinsjob.ItispossibletoputitinsideatextfileandhaveitinGit.ThishasthebenefitthatyourSonarQubeconfigurationisalsoinsourcecontrol.Createafileintherootfolder
ofyourprojectandnameitsonar-project.properties.PuttheSonaranalysispropertiesthatyouhaveinyourJenkinsprojectconfigurationinthefileandcleartheanalysispropertiesinJenkins.PushthefiletoGitandseehowJenkinsexecutesaSonarQubeanalysisasifnothingchanged.YouhavenowmovedtheconfigurationtoyourGitrepository.Ifyoudecidetoputyoursonar-project.propertiesanywhereelse,youcanconfigurethisintheSonarQubestepinJenkins.
SettingupemailBecausetheQualityGatepluginisalittlequirkyandrequiresyoutouseausernameandpasswordinsteadofatoken,youmaynotwanttouseit.Also,ifyouaredealingwithlegacyprojectsorprojectsthatonlyjustuseSonarQube,youmaynotfinditaproblemfornowthattheSonarQubequalitygatefails.WhileyoudonotwanttofailyourJenkinsbuild,youdowanttokeepaneyeonSonarQube.Noworries,youcanconfigureemailsfromSonarQube.First,youneedtosettheemailaddressofyouruser.Ifyouarestillloggedinasadmin,gotoAdministration,thenUsers(underSecurity),andedittheadminusertosettheemailaddress.Next,gotoMyAccount(upperrightcorner)andchangeyournotificationsettingstoreceiveanemailonanewqualitygatestatus(andanyotheremailyouwanttoreceive).Youcansetsomeglobalemailsettingsorspecifyemailsettingsperproject.
Then,youneedtosetyourSMTPsettings.GotoAdministrationandthenfindtheemailsettingsunderGeneralSettings.Again,IamgoingtoshowyouanexampleusingGmail,butthiswouldworkwithanyotheremailprovideraswell,aslongasyouhaveanSMTPserver.Tofollowalong,youeitherneedaGmailaccountorfigureoutyourown(provider's)SMTPsettings.SettheSMTPHosttosmtp.gmail.comandthesecureconnectiontostarttls.TheSMTPpasswordandusernameareyouremailaddressandthepasswordyouusetologin.Now,trytosendatestemailandyoushouldreceiveanemailthatsaysThisisatestmessagefromSonarQube.
Changethequalitygateofyourproject(eithersetorunsetthequalitygatewejustcreated)andrunyourbuildinJenkins.Youshouldgetanemaillikethefollowing:
Project:Chapter7
Qualitygatestatus:Green(wasRed)
SeeitinSonarQube:http://localhost:9000/dashboard/index/chapter7
HTMLandCSSanalysisYoumayhavenoticedthatwhenwerunaSonarQubeanalysisonly,ourJavaScriptfilesarebeinganalyzed.WewouldsurelyliketoanalyzeourHTMLandCSSfilesaswell.NoncompliantHTMLorCSSmaybreakyourpagejustasmuchasnoncompliantJavaScript.InSonarQube,gotoAdministrationandthenUpdateCenterundertheSystemsubmenu.ThisiswhereyoucanupdateandinstallSonarQubeplugins.Themostimportantpluginsaretheanalyzersandweneedtwoofthem,oneforHTMLandoneforCSS.Otherthanthat,youcanfindanalyzersforprettymuchanything.
Java,C#,andJavaScriptcomepreinstalled,butthereareanalyzersforPHP,Python,VisualBasic(.NET),XML,andevenCOBOL.GotoavailablepluginsandsearchforHTML.YoushouldfindtheWebplugin,theCodeanalyzerforWeb(HTML,JSP,JSF,...).InstallitandyoushouldgetamessageSonarQubeneedstoberestartedinordertoinstall1plugins.Beforewerestart,searchtheavailablepluginsforCSS.YoushouldnowfindtheCSS/SCSS/LESSanalyzerthatEnablesanalysisofCSSandLessfiles.SCSSandLESSarebothCSSpre-processors,incaseyouwerewondering.Installitandtherestartmessagenowupdatestotwoplugins.YoucannowrestartSonarQubefromthemessagebox.
OnceSonarQubeisrestarted,runanotherbuildinJenkins.Unfortunately,thiswillfailthequalitygate,sincewehavefourbugsinourHTML.Thebugsareabitsillythough;weneedtoaddfavicondeclarationsintheheadertags.Wealsogetagooddealofcodeduplication(becauseweduplicatedtheheaderineachHTMLfile)andsomeextracodesmellsandtechnicaldebt.Youcannowdecidewhethertheseissuesarelegitimateissuesthatneedtobeaddressedorwhethertheserulesshouldbedeactivatedinthequalityprofile.Atleast,allourfilesarebeinganalyzed.Ihavechosentofixthebugsthough,simplyaddafaviconlink,rightunderthetitleelement,ineachHTMLfile.Yeah,IhaveusedthePackticon:
<linkrel="shortcuticon"href="https://www.packtpub.com/favicon.ico">
IncludingcodecoverageWecanalsoincludeourcodecoverageresultsinSonarQubeandsetrulesinourqualitygates.Forexample,wecansettheminimumamountofcoverageortheminimumamountofcoveragefornewcode.
Accordingtothedocumentation,SonarQubesupportstheCoberturaformatforcodecoverageoutofthebox.Unfortunately,Iwasneverabletogetthattowork.Luckily,wecangetittoworkwiththeLCOVformat.GettinganLCOVreportisaseasyasaddingittothecoveragereportersinyourKarmaconfiguration:
[...]
coverageReporter:{
reporters:[
{type:'html',subdir:'html'},
{type:'cobertura',subdir:'cobertura'},
{type:'lcov',subdir:'lcov'}
],
[...]
Now,addingcodecoveragetoSonarQubeisaseasyasaddingthepathtoyourreportfiletotheanalysispropertiesintheJenkinsconfiguration:
sonar.javascript.lcov.reportPaths=test/coverage/lcov/lcov.info
Ifyourunabuild,youshouldseeacoverageblockbeingaddedtoyourSonarQubeprojectpage:
ItshouldbepossibletogetyourJUnittestresultsinSonarQube,butthisfunctionalitychangedacoupleoftimesinthelastfewyears.ItcurrentlyneedssomespecificJUnitformatthatisslightlydifferentfromtheJUnitformatthatKarmauses.Togetittowork,wemustchangeourtestsandthewayKarmaoutputsthefile.Personally,Ihaveneverbothered.WehaveunittestreportinginJenkinsandtheonlymetricweneedis100%success.BothSonarQubeandtheKarmaJUnitReporterpluginhavesomedocumentationonthesubjectshouldyoubewillingtotry:https://docs.sonarqube.org/display/PLUG/Code+Coverage+by+Unit+Tests+for+Java+Project
andhttps://github.com/karma-runner/karma-junit-reporter#produce-test-result-with-schema-acceptable-in-sonar.
LeakperiodsThelastthingIwishtodiscussrelatingtoSonarQubeistheso-calledleakperiod.Theleakperiodisbasicallyatimeframeinwhichyoumonitornewcode.Let'ssaytheleakperiodisaweek.Withinthatweek,thecodesmells,bugs,andtechnicaldebtaremeasuredandaccumulated.Afteraweek,youstartfresh,butkeeptrackofhowmuchsmells,bugs,anddebtwasaddedsincethelastleakperiod.Yourissuesdonotgoaway,buttherewerenoissuesaddedsincethelastleakperiod.Thisstrategyisespeciallyusefulforlegacyprojects;projectsthatdidnotuseSonarQubebeforeandprojectsthatjusthavebeenignoredinSonarQube.Forsuchprojects,especiallyiftheyarebig,youmayhaveayearworthoftechnicaldebt.Youarenotinterestedinayearlytechnicaldebt.However,ifyoupushtoGitandgetanemailthatyouhavejustadded10minutesoftechnicaldebt,youareinclinedtolookandfixit:
Bydefault,yourleakperiodisanewprojectversion.So,SonarQubewillkeeptrackofnewdebtsincethepreviousversionofyourproject.Youcanchangetheversioninyoursonar-project.propertiesfile.Ihavechangedtheversionto2.0andremovedthefaviconfromafile(weknowthatcausesabug).Asyoucansee,therearetwobugsinmycode,butonlyonewasintroducedsinceversion1.0.Youcanbenotifiedbyemailbygoingtoyouraccountandchangingthenotifications.YouwillwanttocheckMynewissuesspecifically.Also,Changesinissuesassignedtomeisprettyniceincaseyou,orsomeoneelse,accidentallyfixesanissueforyou.
Youcanchangetheleakperiodforyourprojects.Youcansetagloballeakperiodforallyourprojectsandoverwriteitataprojectlevel.EithergotoAdministrationandthenGeneralSettingsorgotoyourprojectand,fromthere,toAdministration(nottheoneatthetop,butbelowit),andthenGeneralSettings.FindtheDifferentialViewssettings.Asyoucansee,onthepagethere
areafewoptionsforsettingleakperiods.Youroptionsareanumberofdays,aspecificdate,thepreviousanalysis,thepreviousversion,oraspecificversion.Youwillalsoseeperiods2and3ongloballevelandlevels4and5onprojectlevel.TheseperiodsaredeprecatedandwillberemovedfromSonarQube,sodonotpayattentiontothem.
ArtifactsNowthatwehaveacompletebuildandweknowourcodeisprettywelltestedandprobablyworksasitshould,weprobablywanttodeliverourfiles.Wearenotyetreadyforautomateddeployment,butattheveryleast,wewantonlythosefilesweneedtomanuallycopyandpastesomewhere.Gotoyourjobconfigurationandaddthepost-buildactionArchivetheartifacts.Youcannowspecifythefilesyouwanttoarchive.Attheveryleast,wewanttoarchivetheprodfolder,butwealsoneedtoarchivesomenodemodules.Thenodemodulescouldhavebeenbetter;wecouldhavecopiedthemtotheprodfoldersothatwasallweneededtoworryabout,butwedidnot.Ifyouwanttodoit,youknowhow.Fornow,wearegoingtospecifythefilesonebyone.So,youwanttoincludethefollowingfiles,seperatedbyacomma:prod/,node_modules/bootstrap/dist/css/bootstrap.min.css,node_modules/angular/angular.min.js,node_modules/bootstrap/dist/js/bootstrap.min.js,node_modules/jquery/dist/jquery.min.js.Youmayalsowanttotakealookattheadvancedsettings.Personally,IonlyneedartifactswhenthebuildsucceedsandIliketouncheckthecasesensitivitysetting.Now,buildyourprojectandyoushouldseeyourartifactsontheprojectpage:
ItlookslikeJenkinsmessedupyourfolderstructure,butitdidnot.WhenyouclickonLastSuccessfulArtifacts,youcanbrowseyourartifactfilesintheiroriginalfolderstructure.ItalsoletsyoudownloadallyourartifactsinaZIPfile,soallyouneedtodoisunpackthezipinyourproductionenvironmentandyouareprettymuchdone.
Wewillneedourartifactslaterastherearemultiplethingswecandowiththem.
RunningonWindowswithJenkinsSlavesWehavecomeprettyfarinwhatwecandousingJenkins,allautomated.However,thereisstilltheissuethatwearerunningonUbuntu.WestillneedtotestourcodeonInternetExplorerandEdgeand,intherealworld,probablySafari.Unfortunately,wearestuckonUbuntu,orsoitseems.Luckily,JenkinshasaneatfeaturethatenablesustorunJenkinsremotelyondifferentcomputers,slaves,ornodes.
GotoyourJenkinsmanagementandfindManageNodes.Onthemenu,clickNewNode.Pickanodename,forexample,WindowsSlave,andmakeitPermanentAgent(atthispointyouprobablyhavenootherchoice).Inthenextform,choosearemoterootdirectory,somethinglikeC:\Jenkins(thisisgoingtorunonWindows!).Also,givethisnodethewindowslabelandchooseOnlybuildjobswithlabelexpressionmatchingthisnodeforusage.Optionally,youcansetthe#ofexecutorstosomethingotherthan1.Thisvaluespecifiedhowmanybuildsmayrunonthismachineatthesametime.Ifyouhaveaveryfastmachinewithlotsofmemory,youmaybeabletobuildfiveprojectsatatime,butifyourserverisalittleslower,oneortwomightbeagoodoption.Ifyouhavemorebuildsthanavailableexecutors,somebuildswillbequeueduntilanexecutorbecomesavailableagain.Nowcomesthepartwherethingsgetalittletricky.JenkinsneedstolaunchtheJenkinsslaveservice.Ifthisserviceisnotavailableontheremotemachine,Jenkinscannotcommunicatewiththemachine(assumingtheJenkinsandremotemachinescanproperlycommunicateotherwise).JenkinscanlaunchtheslaveserviceviaacommandonthemasteroritcantakecontroloftheremotehostandinstallaWindowsservice.Thatlastoptionsoundstempting,butunfortunately,JenkinsusesDCOManditalreadywarnsyouforthesubtleproblemsyoumightencounter.Luckily,Jenkinsalsoproposesanalternative,JavaWebStart.However,JavaWebStartisnotanoptionrightnow.JustsavethisnodeandwewillenableJavaWebStart.GotoJenkinsmanagementandthentoGlobalSecurity.SetTCPportforJNLPagentsfromDisabletoRandomandsave.Now,gobacktoyournodeconfigurationandyoushouldbeabletopicktheLaunchagentviaJavaWebStartlaunchmethod.Onceyousaveyour
node,youshouldgetadownload.Ifyouarenotgettingadownload,gotothenodepageandclicktheJavaLaunchbutton:
OnceyouhavedownloadedtheJenkinsslaveagent,itshouldrunautomatically.Ifitdoesnotrun,youcanstartitmanually.Onceitisrunning,Jenkinsshouldreporttheslaveasconnected.However,runningsomefrontendprogramallthetimeisnotreallyaviablesolution.Jenkinsslaveagentcaninstallitselfasaservicethough.Justasingleclickanditisdone.Youmaygetanerrorwhenyoutrytoinstalltheslaveasaservice.Ifthatisthecase,tryrunningitasanadministratorandtryagain:
Wecannowchangeourprojectsothatitusesournewslave.GototheprojectandfindRestrictwherethisprojectcanberunintheGeneralsectionofyourconfiguration.Youcanrestrictyourprojectstoonlyrunoncertainslavesandslavescanbeidentifiedbylabels.Wegaveourslavethewindowslabel,sousethatasthelabelexpression.YoushouldseethemessageLabelwindowsisservicedby1nodeunderneaththeinputfield.Whileyouareatit,giveyourmasternodethelinuxlabel.
Agoodwaytolabelyourslaveisbythetoolsitprovides.Forexample,wecouldhavegivenourslavethelabelsieandedge.Bydoingthis,youcancreatemultipleslaveswiththeselabelsandJenkinscanpickeitheroneofthoseslavestobuildyourproject.Whenyouhave10buildsrunningconsecutively,itmaycomeinhandytohavemultipleslavestohandlethemall.Youcancombinelabelsusinglogicalandorsymbols,forexample,edge&&ieorwindows||linux.Jenkinswillgiveyouawarningwhenyouhaveselectedmultiplelabelsthatarenotconfiguredonasinglemachine,suchasedge&&safari.
NextweneedtospecifywheretofindSonarQube.OurUbuntuserveruseslocalhost:9000forSonarQube;WindowsisnotgoingtofindthatSonarQubeinstallation.GototheJenkinsmanagementandthentoGlobalToolConfiguration.FindtheSonarQubeScannersectionandclickthebuttontoshowyourSonarQubeScannerinstallations.YoushouldhaveoneSonarQubeScanner,theLocalone.TheSONAR_RUNNER_HOMEvariableissetto/opt/sonar-scanner-2.8orsomethinglikethat.LettheLocalinstallationbe,butaddanewinstallation.Giveitaname,suchasSlaveScanner,andcheckInstallautomatically(shouldbeonbydefault).Youcannowaddoneormoreinstallers,butoneshouldbeaddedautomatically,InstallfromMavenCentral.LeavetheMavenCentralinstallationand,fromthedrop-downmenu,pickSonarQubeScanner[your/latestversion].ThiswilldownloadyourscannerautomaticallyusingMaven(aJavasoftwareprojectmanagementandcomprehensiontool).So,thisinstallationshouldworkjustlikethat.Westillneedtotellourprojecttouseitthough.GobacktoyourprojectconfigurationandfindExecuteSonarQubeScannerbuildstep.Thisshouldnowhaveanewfield,SonarQubeScanner.Fromthedropdown,youcannowpickLocalandSlaveScanner.PickSlaveScannerandsavetheconfiguration.
RunningourtestsIfweweretorunourprojectnow,itwouldfailmiserably.WestillneedtochangeourprojectsoitrunsonWindows.RemovetheshellstepandaddtheExecutewindowsbatchcommandstep.Youwouldthinkthebatchcommanditselfwouldbethesame,butthatisslightlydifferenttoo:
npminstall&&node_modules\.bin\gulp.cmd--env=prod
Unfortunately,thebatchscriptdoesnotexecutemultiplelines,soweneedtouse&&toexecutemultiplestatements.Otherthanthat,westillrunnpminstallandtheGulpscript.NoticethatIhavechangedthegulpscripttogulp.cmd.
NowthatwearerunningonWindows,weprobablywantsomerealbrowserstotestwith.Wecanmakethatasanoptionsjustliketheenvvariable.Wecannotpassarraysfromthecommandlinethough,butwecaneasilyparseit.Putthefollowingcodeatthetopofyourgulpfile(underneathalltherequirestatements):
varbrowsers=gutil.env.browsers?gutil.env.browsers.split(','):undefined;
gutil.log(gutil.colors.cyan('Environment:'),gutil.colors.blue(gutil.env.env));
gutil.log(gutil.colors.cyan('Environment:'),gutil.colors.blue(browsers));
Ifwehavebrowsersdefined,weusethoseand,otherwise,weuseundefined(meaningweusetheonesfromtheconfigurationfile).Now,inthetest-minKarmatask,replacethebrowsersoptionwiththebrowsersvariable(onlyinthetest-mintask,justpretestinginPhantomJSisfine):
[...]
newkarma({
configFile:__dirname+'/test/karma.min.conf.js',
browsers:browsers
},function(err){
[...]
WecannowchangetheJenkinscommand,soitalsorunsthespecifiedbrowsers(case-sensitive):
npminstall&&node_modules\.bin\gulp.cmd--env=prod--browsers=Chrome,IE,IE9,Firefox
Thiswillworkforallyourbrowsersexcept(canyouguessit?)IEandEdge.The
problemwithIEisthatitcannotrununderyourLocalSystemaccount,butthatisexactlywhatyourJenkinsslaveisrunningon.TheeasiestwaytogetthistoworkistodownloadasmallutilityprogramcalledPsExecfromhttps://technet.microsoft.com/en-us/sysinternals/bb897553.DownloadthePsToolspackageandunzipPsTools.OpenupaCommandPrompt(asAdministrator)andrunpath_to_psexec\PsExec-s-i"%programfiles%\InternetExplorer\iexplore".ThiswillopenIEundertheLocalSystemaccount(the-sswitch).Idonotknowwhy,butinthissession,gotoyourIEoptionsandthentotheSecuritytab.
ClicktheResetallzonestodefaultlevelbuttonandthenunchecktheEnableProtectedModeoptionforallfoursecurityzones.ThenrestartIE.NowthatyourLocalSystemIEiscompletelyvulnerable,itwillworkinJenkins.WeirdIEstuff.Evennow,itisstillkindofsketchy,changingtheorderofbrowsersmaybreakIEagain.Unfortunately,IhavenotbeenabletofindanythingonEdge;itjustdoesnotwork.ProbablysomesecurityissuelikeIE,butIwasnotabletofindit.However,ifeverythingworksonIE,IE9,Chrome,andFirefox,itisaprettysafebettoassumeeverythingworksonEdgeaswell.Inanycase,Edgeismaturingasabrowserandafixorworkaroundmaybeavailablebythetimeyouarereadingthis.
Thenextstep,nowthatwearerunningonWindowswithallofourbrowsers,istoalsorunProtractor.Unfortunately,therearealoadofissueswithJenkinsandProtractor.AndProtractorandNode.jsandProtractorandnpmandevenProtractorandyourbrowsers...Theshortstoryisthatwhennpminstallspackagesglobally,itonlydoessoforthecurrentuser.YourJenkinsslaveisrunningundertheLocalUseraccountandnottheaccountyouusedforinstallingallthosepackagesglobally.Thatiswhyweusethecmdfilesinthenode_modules\.binfolderratherthancallinggulporprotractordirectly.However,itseemstheprotractor.cmdfileisnotquitethesameasexecutingprotractordirectly.EvenwhenyouruntheJenkinsslaveunderyourownaccount,itseemsProtractorhangswhenrunninginJenkins.Longstoryshort,justuseChrome.Youhavetestedyourfrontendonallthebrowsers(exceptEdge)usingKarma;itisusuallynotnecessarytorunyourE2Etestsonallthebrowsersaswell.So,makesureyourProtractorconfigurationusesChromeonly.Hopefully,theseissuesgetfixedinthefuture.Itisagoodideatoalsorunthewebdriver-managerupdatecommandbeforerunningProtractor.ThismakessureyoualwayshavethelatestChromedriver.Thismakesthefinalbatchcommandasfollows:
npminstall&&node_modules\.bin\gulp.cmd--env=prod--browsers=Chrome,IE,IE9,Firefox&&node_modules\.bin\webdriver-manager.cmdupdate&&node_modules\.bin\protractor.cmdtest\protractor.conf.js
Youarewelcometoputitinmultiplebatchcommandbuildsteps.Andjusttomakesure,hereisthepartoftheProtractorconfigurationthatmatterstogetthistowork:
[...]
seleniumServerJar:'selenium-server-standalone-3.1.0.jar',
seleniumServerPort:4444,
//seleniumAddress:'http://localhost:4444/wd/hub',
multiCapabilities:[{
browserName:'chrome'
}]
//directConnect:true
};
NowthatourProtractortestswork,weshouldalsopublishtheJUnitreportsitoutputs.So,makesuretoaddittotheJUnitpost-action,test/junit/*.xml,selenium-junit/*.xml.Seleniumputstheoutputinthecurrentfolder,notthefolderoftheconfigfile.
Trytobuildyourprojectandifeverythingwentwell,thebuildshouldsucceed!
WhenthingsgoawryinGulp,Karma,orProtractor,yourJenkinsjobmayhangindefinitely.Thispreventsotherprojectsfrombuildingandhogsupresources.Wedonotwanthangingbuilds.YoucaninstalltheBuild-timeoutplugin(https://wiki.jenkins-ci.org/display/JENKINS/Build-timeout+Plugin).Itcannotstopallthehangingbuilds,butitisbetterthannothing.
TriggeringaprojectpipelineAtthispoint,wecanrunourbuildontheUbuntumasternodeorontheWindowsslavenode.Now,wewanttobuildourcodeonUbuntuandthen,whenthebuildsucceeds,testitonWindows.Wefaceafewproblemsthough:wedonotfeellikerepeatingtheentirebuildjusttorunsomeadditionaltests.Passingallthenecessaryfilestothenextjobisquitealot,butwedoneedprettymuchallofournodemodules.Youmaynotlikeit,buttotestourcodeusingdifferentbrowsers,wemayaswelljustruntheentirebuildagain.
Let'screatetwonewprojects,oneprojecttobuildoursourceonthemasterandanotherprojectto(buildand)testontheslave.OntheJenkinshomepage,createanewitem.Enteranamefortheitem,forexample,BuildChapter7orBuildWebShop,andthenchoosetocopyfromanotherproject.Youcanenterthenameoftheprojecttocopy(thisiscase-sensitive,sobesuretomatchthecasing).Wearegoingtocopytheprojectwehavecreatedinthischapter;InameditChapter7,butyoushouldenterwhateveryoucalledit.Youwillnowbetakentotheconfigurationforyournewproject.Giveitthelinuxlabelexpression,sowearesureitonlyrunsonourLinuxVM.RemovetheProtractorJUnitreport,aswearenotgoingtorunProtractorinthisproject.WealsoneedtoswitchtheWindowsbatchcommandwiththeUbuntushellscriptandchangethecommandsaccordingly:
npminstall
node_module/.bin/gulp--env=prod--browsers=PhantomJS
Otherthanthat,wecanleavethisprojectasitis.
CreateanothernewitemandnameitTestChapter7orTestWebShopand,again,copyitfromtheprojectwehavecreatedinthischapter.ConfigureittorunonWindowsonly.Also,removetheSonarQubeRunnerbuildactionandthequalitygatespost-buildaction.
Atthispoint,wehaveabitofaproblem.OurprojectistriggeredonapushtoourGitrepository,justlikethebuildproject.However,wewouldliketorunthebuildjobfirstandonlytestfurtherwhenitsucceeds.SelecttheBuildafterother
projectsarebuilttriggerandsetBuildChapter7(orBuildWebShop)asProjectstowatch.Now,wehaveanotherproblem;whenthisprojectistriggered,itwillcheckoutourentireGitrepository.However,itispossiblethatanewcommitwasmadewhilethepreviousprojectwasrunning.Thisprojectwouldalsotestthatnewcommitwhilethepreviousjobisstillbuildingthatcommit.Ourcommitstateswouldbeamess.Whatweneedistocopytheentireworkspacefromthepreviousbuildtothisonesowearesureweareworkingonthesamefilesandthesamecommit.Sharingaworkspacebetweentwoprojectsiseasywhenyouareworkingonthesamemachine.Intheadvancedsettingsofthegeneralsectionofyourjobconfiguration,thereisanoptiontoUsecustomworkspace.Simplyenteracustomworkspaceandmakesurebothprojectstargetthesameworkspace.However,sinceweareworkingondifferentmachines,weareinabitofabind.JenkinsdoesnotsupportsharingworkspacesbetweenslavesoutoftheboxandIhaveyettofindagoodpluginthatenablesthiscompletely.Fornow,thereisnothingwecandobuttocheckouttheentireGitrepositoryandhopeforthebest.
Thatleavesuswithanotherproblem,wecouldpublishthestatustotheGitLabcommits,butitmaybeoverwritten.Imaginethefollowingscenario:commit1ispushedandourbuildprojectbuildsit.Whileitisbuilding,commit2ispushed.Whenthebuildprojectfinishes,thetestprojectisstartedandpullscommit1and2andteststhemboth.Meanwhile,thebuildprojectisgoingtobuildcommit2.Commit2doesnotpassthetestsandbothcommitsgetafailedstatus(whilecommit1wasactuallyfine!).Afterthat,thebuildprojectfinishesandgivescommit2apassedstatus.Younowhavetwocommitsthathaveawrongstatus.SoIamgoingtoleavethestatusupdatingtothebuildprojectandremovetheupdatingfromthetestproject.Wemaybeabletofixthisinalaterchapter.
Last,butnotleast,weneedtocreateanewwebhookinGitLab,soitnotifiestheBuildWebShopprojectthatitshouldrunwheneverachangeispushedtotherepository.Youmayalsodisablethefirstprojectwecreated,soitwillnotrunatthesametimeasournewproject.
Now,themomentoftruth,pushachangetoyourGitrepositoryandwatchJenkinsworkitsmagic.Whenallgoeswell,youhaveautomatedtheentirebuildandtestflowofourproject.
SummaryInthischapter,wehavetakenabetterlookatJenkins.WehavelearnedhowtoconfigureJenkins,howtoinstallanduseplugins,howtocreateaprojectandexpandonitasourneedsforqualitygrow,andhowtoinstallJenkinsslavesandrunmultipleprojectsconsecutively.Inaddition,wehaveplayedaroundwithSonarQubeanditsqualitygatesandprofiles.Allinall,itisalotofworktosetupeverything,butitisworthitinthelongrun.Yourcodeisautomaticallyreviewedandtestedonmultiplebrowsers,whichensuressomebaselinequalityforyoursoftware.Inthenextchapters,wearegoingtoexpandonbackendtechnologies,startingwithNode.jsandMongoDB,andlearnhowtofitthoseintoJenkins.
ANodeJSandMongoDBWebAppSofar,wehavetestedandautomatedallofourHTML,CSS,andJavaScriptjobs.However,allofitwasfrontendcode.Inthischapter,wearegoingtoaddsomebackend.WewilluseNode.jsandMongoDB,soitwillstillbeJavaScript.However,itwillstillbequiteanundertaking.InourcurrentHTMLcode,wehavecopiedandpastedtheentireheaderandsearchbar,butwhenwehaveabackend,wecangenerateHTMLusingatemplateengine.SinceweareusingNode.js,wecanreuseourshopping-cart.jsfilewithsomeslightmodifications.Also,wearecurrentlyminimizingallofourcodeusingGulp,andhence,inusingabackend,wemaybeabletominimizeourcodeontheflyandhaveitcached.
Afterwehavehauledoverourwebsite,soitusesaproperbackend,wecanrunourteststoseeifeverythingstillworks.However,wemayhavetochangeourtestsabitaswell.Protractoriscurrentlyrunningonthefileprotocol,butwithabackend,wewillhavetochangethattotheHTMLprotocol.
Wewillalsouseacompletenewpieceoftechnology,thedatabase.Oncewehaveadatabaseinplace,wemaywanttotestifitallworksasitshould.Likeprogramminglanguages,databaseshavetestingoptions.
Onceeverythingisinplace,wewillchangeourJenkinsprojectsaccordingly.
Iwanttoemphasizethatthisbookisnotaboutmanyofthetechnologiesdiscussedinthischapter,butContinuousIntegration,Delivery,andDeployment.ContinuouslyDeliveringdatabases,includingNoSQL,andotherserver-sidesoftwarearejustapartofthat.IamnotassuminganyknowledgeonMongoDBorNode.js,soifyoufollowmylead,Iamprettysureyouwillgetitallworkingjustfine.Iamnotgoingtogivedetailedexplanationsonthecodesamplesthough,buttheyshouldnotbetoocomplex.Forthesakeofsimplicity,Iamignoringbestpracticesandsecurityissues.Whatyouseehereisnotproduction-readycode,buttheprocesstogettingthereandhowCIhelpsinputtingthe
piecestogether.
InstallingMongoDBInthischapter,wearegoingtouseMongoDB,oneofthemostpopularNoSQLdatabasesatthetimeofwriting.ItisevenprettypopularwhencomparedtoSQLdatabases.AccordingtoDB-EnginesRanking(https://db-engines.com/en/ranking),MongoDBisthefifthmostpopulardatabaserightafterallthemajorSQLdatabases.IncaseyouhavenoexperiencewithNoSQL,itmeansNot-only-SQL(andnotNo-SQL-whatsoever).MongoDBisadocument-orienteddatabase,meaningitstoresdocument-orientedorsemi-structureddata.ItisnotverydifferentfromSQLandsoisaperfectintroductiontotheworldofNoSQL.Additionally,itisquiteeasytogetstartedwith.AndsoIchosetouseitforthischapter.
IjustwanttoquicklymentionthebiggestdifferenceswithSQLdatabases,suchasSQLServer,MySQL,andOracle.Firstofall,MongoDBstoresitsdataasBinaryJSON(BSON).Inpractice,thismeanseverythingcanbequeriedusingJavaScript.WecanputJavaScriptobjectsinitandwegetJavaScriptobjectsback(butyoucanuseMongoDBinanymajorprogramminglanguage).LikeJavaScript,andcompletelyunlikeSQL,JavaScriptobjectshavenosetstructure.Propertiesandevenfunctionsareaddedonthefly.Assuch,itispossibletohaveaSalesOrderobjectinyourPersontable(orcollection,astablesarecalledinMongoDB).Evencollectionsarecreatedonthefly.Becausecollectionsandobjectsdonothaveasetstructure,MongoDBiscalledaschemalessdatabase.Perhapsthebiggestdifference,andtheonethatalotofprogrammersandDBAsshun,isthatthereisnosuchthingasaforeignkeyconstraint.Thatmeansthatyouareprobablygoingtostoreredundantdataandthatdataisnotnecessarilyconsistent.
InstallingMongoDBonUbuntuLet'sstartwithinstallingMongoDB,sowecanbedonewiththat.Luckily,thedocumentationoninstallingMongoDBonUbuntuisprettyspoton(https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/):
sudoapt-keyadv--keyserverhkp://keyserver.ubuntu.com:80--recv0C49F3730359A14518585931BC711F9BA15703C6
echo"debhttp://repo.mongodb.org/apt/ubuntuxenial/mongodb-org/3.4multiverse"|sudotee/etc/apt/sources.list.d/mongodb-org-3.4.list
sudoapt-getupdate
sudoapt-getinstall-ymongodb-org
Thefirstcommand,sudoapt-key,isusedtomanagepackagekeys.Keysauthenticatepackagesandpackagesthatareauthenticatedwithakeyareconsideredtrusted.Theadvswitchallowsforadvancedoptions.With--recv,wecandownloadkeysfromaserverandputthemdirectlyintoatrustedlistofkeys.Weknowthedebcommand;itgetsaDebianpackagefilefromthespecifiedURL.Themultiverseswitchindicatesthepackageisrestrictedbycopyrightorlegalissues.Ourusecaseistraining,sowearegoodtogo.Thesudoteecommandreallyjustsavesthefiletothespecifieddirectory.Next,weupdateoursudo-aptrepositoryandinstallMongoDB.
Next,weneedtosetupourfirstdatabaseanduserandmaketheMongoDBinstancevisiblefromoutsidetheVM.Firstwearegoingtocreatethedatabaseanduser.ThisispossiblebecauseauthorizationisoffbydefaultinMongoDB.TypemongotoentertheMongoDBshell.MongoDBusesJavaScript,sothecommandsshouldlookprettyfamiliar.Weneedtocreateadatabaseandtheninsertauserandgiveitreadandwriteaccesstoourdatabase.Toinsertauser,wesimplypassanobjecttothedb.createUserfunction.Writingeverythingonasinglelinecanbemessy(Ifailedtoproperlycloseanobject/array/object/functiontwicethatway),butyoucanenteranewlineusingShift+Enter:
mongo
usewebshop
db.createUser({
user:'your_user',
pwd:'your_password',
roles:[{role:'readWrite',db:'webshop'}]
})
exit
Theusewebshopcommandautomaticallycreatesanewdatabaseifitdoesnotyetexist.ThedbvariablenowpointstothewebshopdatabaseandcreateUseraddstheusertothatdatabase:
TomakeourMongoDBdatabasevisibletoourclient,wemusteditthe/etc/mongod.conffile.Usevi(oranyothereditor)toeditthefile.CommentoutthebindIp:127.0.0.1line(byputting#infrontofit)anduncommitthe#security:lineandputauthorization:'enabled'underit.Whenyouaredoneediting,theconfigurationfileshouldlookasfollows:
sudovi/etc/mongod.conf
i
[...]
#networkinterfaces
net:
port:27017
#bindIp:127.0.0.1
#processManagement:
security:
authorization:'enabled'
[...]
esc
:wq
sudoservicemongodrestart
Pleasenotethatwecannotcreatenewusersanymoreafterthispointbecausewearenowrequiredtologinwithauserthathassufficientprivilegesandwedon't
havesuchauser.
MongoDBshouldautomaticallystartasaserviceonstartup,sowedonotneedtoworryaboutthat.Besuretorestarttheserviceafterhavingmadetheconfigurationchanges.
InstallingMongoDBonWindowsHeadovertotheMongoDBwebsiteanddownloadtheinstallerfromthedownloadcenter(https://www.mongodb.com/download-center).TheversionyouwantisprobablythemostrecentCommunityServerEditionforWindowsServer2008R264-bitandlater,withSSLSupportx64.Simplydownloaditandruntheinstaller.Chooseacompleteinstallationandleavethedefaults.ThefollowinginformationisallintheMongoDBdocumentation,butIhavesummarizeditforyou.
Now,headovertoyourC:\driveandcreateafolderandnameitdata.Insideyournewfolder,createtwonewfoldersandnamethemloganddb.Ofcourse,youcannamethemwhateveryouwantandplacethemwhereveryouwant,butIhaveputitinC:\,sothatisthepathIwilluseintheupcomingexamples.ThisisalsothedefaultpathMongoDBwilluse(onthedrivewhereyouinstalledit).YoucannowrunMongoDBfromyourCommandPrompt.Simplyrunmongod.exe,whichislocatedintheinstallfolder.YourdefaultinstallfolderisC:\ProgramFiles\MongoDB\Server\3.4:
cdC:\ProgramFiles\MongoDB\Server\3.4\bin
mongod.exe
Youcanspecifyadifferentdatafolderbypassingthe--dbpathparametertothemongodprogram:
mongod.exe--dbpathc:\something\db
BesurethefolderexistsorMongoDBwillnotrun.Whenitrunssuccessfully,itwillcreateanumberoffilesinthespecifiedfolder.
Youcanalsoputdbpath,alongwithotherconfiguration,inacustomconfigurationfile.Forexample,putafilecalledmongod.cfginC:\dataandputthefollowingconfigurationinit:
systemLog:
destination:file
path:c:\data\log\mongod.log
storage:
dbPath:c:\data\db
YoucannowrunMongoDBusingthisconfiguration:mongod.exe--configc:\data\mongod.cfg
Last,wewanttoinstallMongoDBasaservice.Youcansimplyusetheextra--installparametertomongod.exeanditwillinstallMongoDBasaservice.Wealsoneedtheconfigurationbecauseweneedtoconfigurealogfile.BesuretoexecutetheinstallationcommandfromanelevatedCommandPrompt:
mongod.exe--config"c:\ProgramFiles\MongoDB\Server\3.4\mongod.cfg"--install
netstartmongodb
IfnothinghappensandyoucannotfindMongoDBinyourservices,seetheconfiguredlogfilefordetails.Ifeverythingisalright,youshouldbeabletofindtheMongoDBserviceinyourservices.YoucannowaddauserintheexactsamewayasonUbuntu.InyourCommandPrompt,typemongotoopentheMongoDBshellanduseShift+Entertogotoanewlinewithoutexecutingyourcommand:
mongo
usewebshop
db.createUser({
[...]
YoucanenableauthorizationinyourconfigurationfilelikewedidonUbuntu:security:
authorization:'enabled'
Next,weneedtoconnectwithMongoDB.Thereareafewclientsavailable.WearegoingtouseRobomongo(https://robomongo.org/).Simplyheadovertotheirwebsiteanddownloadtheinstallerortheportableversion.OnceyoustartRobomongo,youwillbetakentotheconnectionmanager.CreateanewconnectionandnameitCIServer(orwhateveryoulike).Theaddressisciserver(oryourVMsIPaddressorlocalhostifyouinstalleditonthesamemachine)andtheportis27017(unlessyouchangeditintheconfiguration).GototheauthenticationtabandcheckPerformauthentication.PutwebshopforDatabaseandyourusernameandpasswordintheirrespectivefields.Testyourconnectiontovalidatethateverythingworks.Youcannowconnecttothedatabaseandcreatecollections(theSQLtablevariant)andfunctionsandread,insert,update,anddeletedata:
CreatingtheNode.jsBack-endNowthatthedatabaseisinplaceandweknowhowtobrowsecollectionsanddatainthedatabase,wecanstartcreatingabackendpoweredbyNode.js.TogetyourwebsitetorunonNode.js,allyouneedtodoiscreateaJavaScriptfileandloaditusingNode.Nodeprovidessomepackagesthatareatyourdisposalbyusingrequireaswehavedonebefore.TheNode.jswebsitehadsomeHello,Node.js!scriptontheirwebsite,butitwasremovedwhentheplatformgotmorematureandtheuserbasegrew.Luckily,IstillhaveitlayingaroundsomewhereandsowewillcreateourfirstNode.jswebsite.PutthefollowingJavaScriptcodeinanewfileandnameitindex.js:
varhttp=require('http');
varserver=http.createServer(function(req,res){
res.writeHead(200,{'Content-Type':'text/plain'});
res.end('Hello,Node.js!');
});
server.listen(80,'127.0.0.1');
console.log('Serverrunningathttp://127.0.0.1:80/');
YoucanrunthefileinNodebyopeningupaCommandPromptandpassingfileasaparametertonode:
nodepath_to_your_file\index.js
Now,browsetolocalhostandyoushouldseethetextHello,Node.js!.Iftheserverdidnotstartcorrectly,thereisabigchancesomethingisalreadyrunningonport80.ItisthedefaultHTTPportandmanyservicesmakeuseofit.Simplychangethescriptsoitusesanotherportinsteadof80.Youcanuseanyavailableport,theonlyadvantageofusingport80isthatwedonothavetospecifythatportinthebrowser.Ifyouuseanotherport,suchas1337,youneedtobrowsetolocalhost:1337.WecanchangethereturnedresponsesothatitreturnsHTMLratherthanplaintext:
varserver=http.createServer(function(req,res){
res.writeHead(200,{'Content-Type':'text/html'});
res.end('<h1>Header</h1><p>Hello,Node.js!</p>');
});
RestartyourNodeserverfromthecommand(Ctrl+Ctoclosethecurrentserver)andrefreshyourlocalhostpage.Youshouldnowgetsomepre-styled
headerandparagraph.
Closingandre-openingtheNodeserverisabitdistractingandtedious,especiallywhenyouarestilldevelopingandneedtorefreshalot.TherearesometoolsthatopenaserverforyouandrestartitwheneveryourJavaScriptfilechanges.TheeasiestIhavefoundisnodemon(https://nodemon.io/).Nodemonisjustannpmpackage,soyoucaninstallitusingnpm.Now,insteadofopeningaNodeserverusingthenodecommandyoucanusenodemonandmakechangeswithoutrestarting:
npminstallnodemon-g
nodemonpath_to_your_file\index.js
Browsetolocalhosttoseeifitworked.Now,changeyourindex.jsfile(justmakeitreturnsomedifferenttext)andrefreshyourbrowser.YoushouldseethenewtextwithouthavingrestartedyourNodeserver:
ExpressTomakethingseasierforourselvesinthelongrun,wearegoingtouseaframeworkforNode.js.Nodeisprettybarebones,whichmeansitcandoeverything,butitprobablyneedssomework.Thatworkisdoneforusindifferentframeworksthatallworkalittledifferently.Express(https://expressjs.com/)isoneofthemorepopularframeworks(amongHapi.jsandSails.js),solet'sgowiththat.SimplyinstallExpressusingnpm.PleasenotethatweneedExpresstorunthesoftware,soweuse--saveinsteadof--save-dev:
npminstallexpress--save
WecannowchangeourfilesoitusesExpress.WedonotneedtorequirethehttpmodulebecauseExpresshasthatallwrappedforus.Otherthanthat,thecodelooksprettymuchthesame.WithExpress,wecanmapmultiplepathstothesameroutehandlerthough,solocalhostandlocalhost/index.htmlwillnowreturnthesamepage.The*wildcardhandlesallotherrequestsand,inourcase,returnsa404:
varexpress=require('express'),
app=express();
app.get(['/','/index.html'],function(req,res){
res.writeHead(200,{'Content-Type':'text/html'});
res.end('<h1>Header</h1><p>Hello,Node.js!</p>');
});
app.get('*',function(req,res){
res.writeHead(404,{'Content-Type':'text/html'});
res.end('<h1>404-Pagenotfound!</h1><p>Whatwereyoueventryingtodo...?</p>');
});
varserver=app.listen(80,'127.0.0.1');
console.log('Serverrunningathttp://127.0.0.1:80/');
Next,wewanttoserveourHTMLfilesinsteadoftheHTMLstringwesendnow.Thatiseasyenough,butthereisacatch:
app.get(['/','/index.html'],function(req,res){
res.sendFile(path.join(__dirname,'index.html'));
});
Asyoucansee,wecansimplyreturnanHTMLfileusingres.sendFile.Expresswillsortoutourresponseheadersbasedonthefileformat.However,ifyou
refreshyourwebsite(usingCtrl+F5toclearthecache),itlookscrazy.Inyourbrowser'sconsole,youcanfindplentyof404s.WhathappensisthatyourbrowserreceivedtheHTMLfileandthenrequestsalltheCSSandJavaScriptfiles.ThebrowsertriestoGETfilessuchas:http://localhost/css/layout.cssandhttp://localhost/scripts/bundles/index.bundle.js.
Unfortunately,wehaveaGETrouterfor/and.index.htmlandalltheotherGETrequestsendupinthewildcard(*)route,whichreturnsa404.
Expresshasamethodforreturningstaticfiles.Itisreallyquiteeasy,enterafolderorfilename,andExpresswillservethefilesinthefolderorthespecifiedfile.Thefilesareavailableonthepathrelativetothespecifiedpath.Theeasiestmethodofjustmakingeverythingavailableisbyspecifying__dirname:
varexpress=require('express'),
path=require('path'),
app=express();
app.use(express.static(__dirname));
[...]
Byaddingtheapp.use(express.static(...))statementtoyourNodefile,youmakeallthefilesinyourwebfolderavailable.Formorefine-grainedcontrol,whichisrecommendedfromasecuritypointofview,specifyonlythefilesandfoldersyouwanttomakepublic.Wecanaddthedebugfilesandminifiedfiles.Ifyouwant,youcanaddaBooleanswitchandonlygetthedebugfilesforadebugrunandonlytheminifiedfilesforaproductionrun:
app.use('/node_modules/bootstrap/dist/fonts',express.static('node_modules/bootstrap/dist/fonts'));
app.use('/node_modules/bootstrap/dist/css/bootstrap.css',express.static('node_modules/bootstrap/dist/css/bootstrap.css'));
app.use('/node_modules/bootstrap/dist/css/bootstrap.min.css',express.static('node_modules/bootstrap/dist/css/bootstrap.min.css'));
app.use('/css',express.static('css'));
app.use('/node_modules/angular/angular.js',express.static('node_modules/angular/angular.js'));
app.use('/node_modules/angular/angular.min.js',express.static('node_modules/angular/angular.min.js'));
app.use('/node_modules/jquery/dist/jquery.js',express.static('node_modules/jquery/dist/jquery.js'));
app.use('/node_modules/jquery/dist/jquery.min.js',express.static('node_modules/jquery/dist/jquery.min.js'));
app.use('/node_modules/bootstrap/dist/js/bootstrap.min.js',express.static('node_modules/bootstrap/dist/js/bootstrap.js'));
app.use('/scripts',express.static('scripts'));
app.use('/scripts/bundles',express.static('scripts/bundles'));
BecauseExpressfindsthefilesrelativetothestaticdirectory,weneedtoexplicitlyspecifythepathforthegivenresource.Toillustratethis,app.use(express.static('css'));serveslocalhost/layout.cssandlocalhost/utils.css,whileapp.use('/css',express.static('css'));serveslocalhost/css/layout.cssand
localhost/css/utils.css.
Nowthatwehavetakencareofthat,wecanaddtherestoftheroutes:
app.get(['/','/index.html'],function(req,res){
res.sendFile(path.join(__dirname,'index.html'));
});
app.get('/views/product.html',function(req,res){
res.sendFile(path.join(__dirname,'views/product.html'));
});
app.get('/views/search.html',function(req,res){
res.sendFile(path.join(__dirname,'views/search.html'));
});
app.get('/views/shopping-cart.html',function(req,res){
res.sendFile(path.join(__dirname,'views/shopping-cart.html'));
});
Thewebsitedoesthesameasitdidbefore.Loandbehold,wecanverifythiswithrelativeease!GotoyourProtractorconfigurationfileandsetbaseUrltohttp://localhostandremovebrowser.resetUrl.WenowworkwiththeHTTPprotocolinsteadofthefileprotocol:
exports.config={
baseUrl:'http://localhost',
onPrepare:function(){
//browser.resetUrl='file://';
browser.ignoreSynchronization=true;
[...]
},
[...]
};
NowthatweuseHTTP,wecanalsorunFirefoxwithoutadirectconnection,meaningwecanrunallthebrowsersinasingleconfiguration.Anyway,runyourProtractortestsandverifythatalltestssucceed.
EJSNowthatwecanservefilesandHTMLfromourbackend,wecanusesomesortofHTMLtemplateengine.Basically,IamprettysatisfiedwiththeHTMLfileswehavesofar,exceptforthefactthatthenavigationbaratthetopisduplicatedineachfile.Wecandobetterthanthat.Expresshasbuilt-insupportforseveraltemplatingengines,mostnotablyJade.Ontopofthat,itisfairlyeasytocreateyourowntemplatingengine.WhileJadeisprettyawesome,ithasverycleanandconcisesyntax,wearegoingtouseEJS,orEmbeddedJavaScript(http://www.embeddedjs.com/).Allwewanttodoiscreatesomesortofpartialview(thenavigationbar)andinsertthatintoourpages.EJSsupportsjustthat(andmore):
npminstallejs--save
Now,inyourviewsfolder,createanewfileandnameitnavbar.ejs.Simplyputtheentirenavelementfromanyofyourviewsintothefile.Wecanstartwiththeproductpagebecauseitisaverysimplepage:
<navclass="navbarnavbar-defaultnavbar-fixed-top">
<divclass="container">
<divclass="navbar-header">
[...]
</div>
</div>
</nav>
Renametheproduct.htmlfiletoproduct.ejsandreplacethenavelementwithanEJSdirective:
<!DOCTYPEhtml>
<html>
<head>
[...]
</head>
<bodyng-app="shopApp">
<%-includenavbar.ejs%>
<divclass="container"ng-controller="productController">
[...]
</div>
</body>
</html>
IntheNodefile,wehavetomaketwochanges.First,wemustsetEJSastheviewengineforExpress.Second,weneedtouseittorendertheproductpage
ratherthanservesomestaticHTML:
varexpress=require('express'),
path=require('path'),
app=express();
app.set('viewengine','ejs');
[...]
app.get('/views/product.html',function(req,res){
res.render('product');
});
[...]
EJSwilllookfortheproduct.ejsfileintheviewsfolderbydefault.YoucannowmakethischangeforallyourHTMLfilesandplaceindex.htmltotheviewsfolder(afterall,wealreadyhaveindex.jsintheroot).Youshouldalsochangethesourceinyourhtmltaskinthegulpfile:
.task('html',['clean'],function(){
returngulp.src(['views/*.ejs'])
.pipe(replace('href="views\\','href="'))
[...]
Also,changeyourgulpfile,soitwatchesviews/*.ejsinsteadofviews/*.html.Thereisjustonemoreissueweneedtofix.Currently,allourpages,exceptindex,areroutingto/views/page.html.Notethatthisiscompletelyrandom.ItdoesnotnecessarilyserveHTMLpagesandthepagesdonothavetobeintheviewsfolder.Wecouldroutetomonkeyandreturntheproductpage.So,let'smakethatroutingabitmorefriendlyontheeyeandremovethe/viewsbitfromtheURL.Wecankeepthe.htmlpartbecauseitisprettystandardandwedostillreturnHTML(althoughitiscompletelyoptional):
app.get('/product.html',function(req,res){
[...]
app.get('/search.html',function(req,res){
[...]
app.get('/shopping-cart.html',function(req,res){
Youshouldnowfixyournav.ejsandindex.ejsfilessotheanchorsdonotlinkto../index.htmlandviews/...html,butdirectlytothecorrectpage.
TryrunningyourProtractortests.Onetestfailsbecausethetestistryingtoopen'views\\shopping-cart.html',whichobviouslyisnotgoingtowork.So,justchangethatto'shopping-cart.html'andtryagain.Yourtestsmightstillfailifyouhavenot
copiedthenavelementinnavbar.ejsfromtheindex.htmlpage.Wegavetheaelementforthesearchbuttonanid,soourtestscaneasilylocatethatelement,butweonlydidthatontheindex.htmlpage.Thatiswhatyougetforduplicatingcode.Addid="search-btn"totheanchorandrunyourtestsagain.Theyshouldsucceednow.
Atthispoint,Iwouldliketopointouthoweventheseminimaltestsgiveustheconfidencetocompletelyrebuildsomefrontend-onlypagestobackendNodewithEJS.
However,Iwouldalsoliketopointoutthattwotestsfailedbecausetheteststhemselveswerenotquitecorrect.Thesearchfunctionalitywouldhaveworkedevenwithoutid,whichwasonlythereforthetesttobeginwith.Havingtestsisgreat,buttheycomeatthecostofsomedebuggingandmaintenanceonceinawhile.
TheLoginPageOurnextstepistocreatesomemethodofloggingintogetherwithsavingwhatisinourshoppingcart.Wearenotgoingtocreateafullfledgedwebshop,soforthesakeoflearning,wewillmanuallyinserttwousersintothedatabase,sowecanloginandrememberwhatisineachoftheirshoppingcarts.BeforewecansaveanythingtoMongoDB,wemustfirstcreatealoginpageforourwebsite.Createaviewandnameitlogin.ejs.Thecontentsareprettymuchthesameasthatonotherpages.Samestylesheets,scripts,andfooter.Thepartthatisdifferentgoesincontainerdiv.IamusingaformelementsoAngulardoestherequiredvalidationforus.Otherthanthat,itisnotmuchdifferentfromdoingaregularAJAXPOSTlateron:
[...]
<scriptsrc="..\scripts\bundles\login.bundle.js"></script>
<!--endbuild-->
</head>
<bodyng-app="shopApp">
<%-includenavbar.ejs%>
<divclass="container"ng-controller="loginController">
<divclass="row">
<formng-submit="login()">
<divclass="col-lg-3centertext-center">
<h2>Login</h2>
<pclass="bg-danger"ng-show="notFound">Userorpasswordwereincorrect.</p>
<divclass="input-groupcenter">
<pclass="clearfix">
<inputclass="form-control"placeholder="Username..."ng-model="username"required/>
</p>
<pclass="clearfix">
<inputtype="password"class="form-control"placeholder="Password..."
</p>
</div>
<pclass="clearfix">
<inputtype="submit"class="btnbtn-success"value="Login"/>
</p>
</div>
</form>
</div>
<footerclass="footer">
[...]
Tomakethiswork,weneedtoaddsomeCSStotheutils.cssfile:
.center{
float:none;
margin:0auto;
}
Andyouneedalogin.jsfilethatdoesnotdoanythingyet:
angular.module('shopApp',[])
.controller('loginController',['$scope',function($scope){
$scope.username=null;
$scope.password=null;
$scope.notFound=false;
$scope.login=function(){
console.log($scope.username+'-'+$scope.password);
};
}]);
And,ofcourse,weneedtorouteitinourNodefile:
app.get('/login.html',function(req,res){
res.render('login');
});
Last,butnotleast,wemustchangethenavbar.ejsfilesotheloginbuttonactuallybringsustotheloginpage:
<li><ahref="login.html"><spanclass="glyphiconglyphicon-user"></span>Login</a></li>
BesuretorunGulp,soournewscriptsarebundled.YourKarmatestswillfailbecauseourcodecoveragejustwentbelowthe50%threshold,butdonotworryaboutthatfornow.
ConnectingwithAJAXNowthatwehavetheloginpageinplace,wecanactuallyconnecttoourbackend,somethingwehavenotdoneyet!Basically,whatwewantistologinandredirecttotheshoppingcart(whichkindofservesasourpersonalpageforthisproject).Inthefrontend,thisiseasyenough.Weneedtoinjectthe$httpmoduletoourAngularcontrollerthatwecanusetomakeanAJAXcall.Afterthat,wecancallourbackendandredirectonsuccessorshowanerrorwhentheuserwasnotfound:
$scope.login=function(){
$scope.notFound=false;
$http.post('/login',{
username:$scope.username,
password:$scope.password
}).then(function(response){
if(response.data.success){
window.location.replace("/shopping-cart.html");
}else{
$scope.notFound=true;
}
},function(){
alert('Anerrorhasoccurred.');
});
};
So,Iammakingadistinctionbetweena500errorcode(whichshouldnotoccur,ideally)andanonsuccessfuluserlookup.Whentheuserisfound,wecontinuetotheshoppingcartpage.
Onthebackend,thingsarealittledifferent.First,weneedtheExpressbodyparsertoparseJSONtoregularJavaScript.ThisusedtobeincludedinExpress,buttheymadeitaseparateplugin.So,weneedtoinstallitusingnpm:
npminstallbody-parser--save
Usingitissimpleenough.AttachthemiddlewaretoExpressandthatisthat:
varbodyParser=require('body-parser');
app.use(bodyParser.json());
WecannowinspectthebodypropertyonourrequestvariableonourloginPOSTcall:
app.post('/login',function(req,res){
if(req.body.username==='username'&&req.body.password==='password'){
res.json({success:true});
}else{
res.json({success:false});
}
});
Wewillalsoneedasession,sowecanremembertheloggedinuser.Likethebodyparser,Expresshassomemiddlewarepluginforsessionmanagement,express-session(https://www.npmjs.com/package/express-session):
npminstallexpress-session--save
Thedefaultsessionstoreisin-memory.However,therearevariouspluginsforothersessionstores,suchasMongoDB,Redis,Oracle,MySQL,andmanymore.Usageisnotparticularlydifficult,however,itisnotproduction-ready(youwillneedanothersessionstoreforthat).Thesessionmakesuseofacookie,sowhenyouloginandthenclearyourcookies,youwillbeloggedoutagain:
varsession=require('express-session');
app.use(session({
secret:'somesecret',
resave:false,
saveUninitialized:true,
cookie:{}
}));
[...]
app.post('/login',function(req,res){
if(req.body.username==='username'&&req.body.password==='password'){
req.session.authenticated=true;
req.session.username=req.body.username;
res.json({success:true});
}else{
res.json({success:false});
}
});
And,ofcourse,wewillneedalogoutmethodaswell:
app.get('/logout.html',function(req,res){
if(req.session){
req.session.destroy();
}
res.redirect('/');
});
Wecannowshowaloginoralogoutbuttondependingonwhetheryouareloggedinornot.WithEJS,wecaneasilyaddthis.However,itwouldbecumbersometoaddsomeauthenticatedvariabletoeachview:
res.render('some-view',{authenticated:req.session.authenticated});
Instead,wecanaddsomemiddlewaretoExpress.Itisimportantthatthisisaddedafteryouaddthesessionmiddleware:
app.use(session([...]));
app.use(function(req,res,next){
res.locals.authenticated=req.session.authenticated;
next();
});
res.localsisavailableinyourtemplates,soaddingthistoyourNodescriptwillmakeauthenticatedavailabletoallyourviews.Andwewillneeditinallviews.Luckily,weonlyhavetochangethisinonefile,beingnavbar.ejs:
[...]
<ulclass="navnavbar-navnavbar-right">
<%if(authenticated){%>
<li><ahref="logout.html"><spanclass="glyphiconglyphicon-off"></span>Logout<%=username%></a></li>
<%}else{%>
<li><ahref="login.html"><spanclass="glyphiconglyphicon-user"></span>Login</a></li>
<%}%>
<li><ahref="shopping-cart.html"><spanclass="glyphiconglyphicon-shopping-cart"></span>Order</a></li>
</ul>
[...]
YoucannowloginwiththeusernameusernameandthepasswordpasswordandyoushouldseethelogouticonappearoneverypageuntilyoulogoutorrestartNode.jsoryourbrowser.
SavingtoMongoDBThenextstepisgettingourusersfromthedatabase.Aswesaid,wearenotgoingtocreatearegistrationmethod,sowewillneedtoinsertatleastoneusermanually.YoucanlogintothewebshopdatabaseusingRobomongo.Right-clickonCollectionsandchooseCreatecollection....Nameyournewcollectionuser.Now,youcanright-clickontheusercollectionandpickInsertDocument....ThedocumentyouaregoingtoinsertisjustaJavaScriptobject,soentersomeobjectwiththeusernameandpasswordproperties:
Youcannowbrowsethenewlyinserteddocument(s):
WecannowquerythedatabasefromNodeandcheckforauserwiththegivenusernameandpassword.ToconnectwithMongoDB,wearegoingtousetheNode.jsMongoDBAPIMongoose(http://mongoosejs.com/).Asalways,wecaninstallMongoosewithnpm:
npminstallmongoose--save
Afterthat,usingMongooseissurprisinglyeasy.NowonderweareusingJavaScripttoinsertJavaScriptinadatabasethatusesJavaScript.TheonlythingthatmakesworkingwithMongoosealittlemessyistheasynchronousAPI.ItkindofworkslikeweknowfromProtractor:
varMongoClient=require('mongodb').MongoClient,
mongoUrl='mongodb://your_user:your_password@ciserver:27017/webshop';
[...]
app.post('/login',function(req,res){
if(req.body.username&&req.body.password){
MongoClient.connect(mongoUrl,function(err,db){
if(err){
console.log(err);
res.status(500).send(err);
}else{
db.collection('user').findOne({
username:req.body.username,
password:req.body.password
},function(err,user){
if(err){
console.log(err);
res.status(500).send(err);
}
if(user){
req.session.authenticated=true;
req.session.username=req.body.username;
res.json({success:true});
}else{
req.session.destroy();
res.json({success:false});
}
});
}
});
}else{
res.json({success:false});
}
});
Thatisquiteabitofcode,butIthinkitisnotthathardtofollow.WeuseMongoClient.connect(mongoUrl)togetaconnectiontothedatabase.Thecallbackfunctiongetsanerrorobjectandareferencetothedatabase.Whenanerrorisreturned,welogtheerrorandreturnaninternalservererror.Ifwedonotgetanerror,weusethedatabasetofindtheusercollectionandfindadocumentwiththespecifiedusernameandpassword.callbackgivesusanerrorandafounddocument.Theerrorhandlingisthesameasbefore.Now,ifwefindauser,wecreatethesessionvariablesauthenticatedandusernameandreturnsuccesstrue.Ifnouserwasfound,wedestroythesession(incaseyouwerealreadyloggedinassomeoneelse)andreturnsuccessfalse.Thisshoulddothetrick.So,trylogginginastheuseryouenteredinMongoDB.Also,tryinsertingaseconduserandlogginginasbothoftheminseparatebrowsers(orinprivatewindowsofthesamebrowser).
MovingourProductstoMongoDBWestillhavethatweirdrepository.jsfilethathassomehardcodedproductsinit.HavingproductslikethatinaJavaScriptfileisnotgoingtowork,solet'smovethemtothedatabaseaswell.Again,wearegoingtoinsertthemmanually,butthistime,wecancreatealittlescriptthatinsertsalloftheproductsatonce.InRobomongo,right-clickonthedatabaseandpickOpenShell....Thisopensanewtabwithasinglelineforyoutotypein.Donotworry,youcanenterasmanynewlinesasyouwant.
YourcommandwillsimplybesomeJavaScriptand,sincewealreadyhaveallourproductswrittendowninJavaScript,thisisbasicallyacopyandpasteaction:
db.getCollection("product").insert([{
id:1,
name:'FinalFantasyXV',
price:55.99,
description:'FinalFantasyfinallymakesacomeback!',
category:'Gaming'
},{
[...]
}]);
YoucanrunthequeryusingF5ortheplaybutton(thegreentriangle)inthemenu.RobomongotellsyouitInserted1record(s)in13ms,butitactuallyinsertedtheentirearray:
Next,weneedtorewritetheotherfunctionsintherepositoryfilesothattheymakeAJAXcallstotheserverandreturntheproductsfromthedatabase.Thefirstfunctionwearegoingtorewriteissearch.WearegoingtocreateasearchProductsPOSTmethodinourNodefileandsearchfortheproductstherebasedonaqparameter.Thefunctionalityshouldremainthesameasthefrontendfunctionthough:
functionhandleErr(err,res){
if(err){
console.log(err);
res.status(500).send(err);
}
}
app.post('/searchProducts',function(req,res){
req.body.q='fanta';
if(req.body.q){
MongoClient.connect(mongoUrl,function(err,db){
if(err){
handleErr(err,res);
}else{
db.collection('product').find({
name:newRegExp(req.body.q,'i')
}).toArray(function(err,products){
if(err){
handleErr(err,res);
}else{
console.log(products);
res.json(products);
}
});
}
});
}else{
res.json([]);
}
});
Again,itisabitofcodewithacoupleofcallbacks,butnothingtoobad.Asyoucansee,wecanfindrecordswithsomevaluelikeanothervalue(inthiscase,[Name]LIKE'%'+q+'%'inSQL)usingregularexpressions.Iamnotafanofregex,buttheydothejobandthisoneisnotparticularlycomplicated.Theioptionisjusttheretomakethelookupcase-insensitive.Wenowneedtocallthisfunctionfromtherepositoryfileinthefrontendandprocesstheresults:
search:function(q,$http,callback){
var$http=angular.injector(["ng"]).get("$http");
$http.post('/searchProducts',{
q:q
}).then(function(response){
callback(response.data);
},function(){
alert('Anerrorhasoccurred.');
});
}
Inthisexample,wearegettingtheAngular$httpvariableusingdependencyinjection.Afterthat,wesimplyexecuteanAJAXPOSTandpasstheresulttocallback:
varq=utils.getQueryParams().q;
repository.search(q,function(results){
$scope.$apply(function(){
$scope.results=results;
});
});
Thesearch.jsfilesimplycallstherepository.searchfunction.Inthecallback,weneedtouse$scope.$applybecauseweareworkingonanotherthread.Doingthingsthiswayhastheleastimpactonourexistingfunctionality.Indeed,whenyouruntheProtractortests,youwillseeeverythingworkedasbefore.Notethatusing$scope.$applyisnotthebestpractice(itisoneofafewdodgyworkarounds),butwewillfixthatinabit.However,whenyourunyourunittests,youwillnoticeoneofthemfails.Wewroteaunittestfortherepository.searchfunctionusingtheMochaframework.Sincethisfunctioncannotbeproperlyunittestedanymore,itisadataretrievalactionontheserver,wecanremovethistestcompletely(andwiththatIamremovingMochacompletely).However,theProtractortestsstillcoverthispagesowearegood.
Thisisreallytheimportantthinghere.Nomatterhowyouwouldhavesolvedthis(wegetthepageandthenthedata,butitwouldhavebeenperfectlyvalidto
getthepageandthedatainonegoaswell),yourtestsensurethateverythingkeepsworkingasitshould.Whenyourtestsfailyoushouldstartthinkingwhetheryouintroducedabugorifthespecschangedandyoushouldfixyourtests.
Ihavefixedtheothertworepositoryfunctions,getTopProductsandgetProduct,inthesameway.Itisnotveryinterestingtorepeatithere.
PuttingtheShoppingCartinMongoDBTheonlythinglefttodobeforewearecompletelybackenddrivenistoaddfunctionalityforaddingandremovingproductsfromtheshoppingcartandrememberingthem.First,somerules.Whenyouaretryingtobuysomethingwhileyouarenotloggedin,youwillbetakentotheloginpage.Whenyougototheshoppingcartpage,youwillalsobetakentotheloginpageand,whenyoulogin,youwillbetakentotheshoppingcartpage.Whenyoubuysomething,itwillbeaddedtoyourshoppingcartanditwillberememberedacrosssessions.Thatisreallyhoweasyitis.
First,let'screateafunctionthatgetsustheloggedintheuser'sshoppingcartfromthedatabase.Iamgoingtoshowtherelevantbackendcode.Onthefrontend,wearejustgoingtocreateanotherrepositorymethodanduseitlikeweuseanyoftheothermethods:
db.collection('user').findOne({
username:req.session.username
},{shoppingCart:[]},function(err,user){
if(err){
handleErr(err,res);
}else{
res.json({
authenticated:true,
shoppingCart:user.shoppingCart||[]
});
}
});
ThesecondparametertofindOneisaprojector,whichtellsMongoDBtoonlygettheshoppingCartfieldofthefounduser.Intheresponse,wejustreturnshoppingCart.SinceMongoDB,likeJavaScript,isschemaless,theshoppingCartfieldisnotguaranteedtoexist.Itisnotevenguaranteedtobeanarray,butwearegoingtoassumethatanyway.Wearesettingauthenticatedtotrue(asopposedtofalse)justtoletthefrontendknoweverythingwentwell.
TheaddProductToCartfunctionusestheMongoDBfindOneAndUpdatemethod.The$addToSetupdateoperatortellsMongoDBtoinserttheitemonlyiftheitemisnot
yetpresentinthearray(asopposedto$push,whichjustaddstheitem).Wearenotupdatingthenumberfield(ever)formultipleitems.Itwouldmessup$addToSet,butmaybe,wewilladdthisfeatureinthefuture:
db.collection('user').findOneAndUpdate({
username:req.session.username
},{
$addToSet:{
shoppingCart:{
product:req.body.product,
number:1
}
}
},function(err){
if(err){
handleErr(err,res);
}else{
res.json({authenticated:true});
}
});
Ofcourse,weneedtochangeallofourbuybuttonsforthisone.Thefollowingexampleisfromtheindex.ejsfile.Otherbuybuttonsareontheproductandsearchpages:
<buttonng-click="addToCart(product)"class="btnbtn-primarypull-right">Buy</button>
Last,weneedsomemethodofremovingaproductfromtheshoppingcart:
db.collection('user').findOneAndUpdate({
username:req.session.username
},{
$pull:{
shoppingCart:{
product:req.body.product
}
}
},function(err){
if(err){
handleErr(err,res);
}else{
res.json({authenticated:true});
}
});
The$pullupdateoperatorwillremoveanyobjectsthatmeetthequeryfromanarray.Inourcase,itwillremoveanyobjectsthathavethespecifiedproduct.
MovingtheShoppingCartModuleIwanttodoonelastthing.Theshoppingcartcontrollerismightyhandy.NodoubtwewanttoreusethisfunctionalityinNode.jstocreateinvoices.Luckily,wehavekindoffutureproofedourfrontendscriptswithBrowserifyalready,sowecansimplycreateamoduleanduseitonboththefrontendandbackend.
Createanewfileinyourscriptsfolderandnameitorder.js.Initgoesmostofthecodefromtheshopping-cart.jsfile:
module.exports=(function(){
varOrder=function(){
this.lines=[];
};
Order.prototype.removeLine=function(line){
this.lines.splice(this.lines.indexOf(line),1);
};
[...]
return{
Order:Order,
Line:Line
};
})();
Sincewearenotworkingonthe$scopeobjectanymore,weneedsomenewobjecttocontaintheLineobjects.So,wecreateanOrderconstructorthathasalinesarray.ThetotalfunctionandtheremoveLinefunctionareputonOrderprototype.PleasenotethatremoveLinenolongermakesanAJAXcalltothebackend.ThatkindoffunctionalitywouldnotmakeitusableinNode.js.TheLineobjectisexactlythesameasitwas.
Now,intheshopping-cart.jsfile,weneedtochangeathingortwoaswell:
varorder=require('./order.js');
$scope.order=neworder.Order();
repository.getCart(function(data){
if(data.authenticated){
$scope.$apply(function(){
$scope.order.lines=data.shoppingCart.map(function(l){
returnneworder.Line({
[...]
$scope.removeLine=function(line){
repository.removeFromCart(line.product,function(){
$scope.$apply(function(){
$scope.order.removeLine(line);
[...]
First,weneedtorequiretheorder.jsfileandcreateanewOrderobject.Afterthat,weneedtoassigntheorder.Lineobjectsto$scope.order.lines.TheremoveLinefunctionstays,butonlybecausewestillneedtoremovethelineinthebackend.Westillneedtocall$scope.order.removeLine(line)toactuallyremovetheproductonscreen.
Intheshoppingcartview,theejsfile,wealsoneedafewchanges.Basically,weneedtoreplaceanyreferencetolinestoorder.linesandweneedtoreplacetotaltoorder.total.OnlytheremoveLinereferencestaysput:
[...]
<divclass="row"ng-hide="order.lines.length">
[...]
<divng-repeat="lineinorder.lines">
[...]
<pclass="pull-right">{{'€'+order.total()}}</p>
[...]
Thatwrapsupourcodingsofar.WehavenowsuccessfullyaddedaMongoDBdatabaseandaNode.jsbackendtoourwebsite.Itisprobablynotthebestwebshopyouhaveeverseen,butithasallthefunctionalityweneedtocontinuewithourGulp,Karma,Protractor,andJenkinsexamples.
SpeakingofKarmaandProtractor,ourtestsbrokesomewherealongtheway.Thatmakesperfectsenseasweaddedtheloginfunctionality.However,wearenowunabletotestifourwebsiteworksasitshould,whichisunfortunate.Timetofixourtests.
OurKarmatestshavebeenfailingforawhilenow.Unittestsshouldnotdependuponexternalsystems.Theyshouldideallytestsmallandisolatedpiecesofcode.WhenrevisitingourKarmatests,weseethatwetestshoppingCartController.However,wechangedtherepositorysothatitgetsourdatafromthebackendmakingourunittestsfail.Ourunittestsshouldnotdependuponexternalsystems,suchasNode.jsandMongoDB.So,thischangeprettymuchmakesshoppingCartControllerimpossibletotest.
Afterthat,weremovedtheLineobjectcompletely,createdanewOrderobject
instead,andputtheminaseparatefile.ThisisgoodnewsbecausewehavenowabstractedthecalculationsawayfromthecontrollerandtheAJAXcalls.Thatmeanswecantestitagain.However,becauseweusedthemodule.exportsobject,weneedtobeabletoloadthescriptusingrequire.Ifwedonot,wehavetodirectlyusemodule.exports,whichwillbeoverwrittenoncewestarttoloadmorepackageslikethis.WealreadyknowaboutBrowserifyandCommonJS.So,oneoptionwehaveistojustBrowserifyourspecfiletoo.WecanedittheKarmaconfiguration.
KeepinmindthatIhaveremovedMochaandChaifromtheconfigurationaswell.So,whatchangesaretheframeworks,files,andpreprocessors:
[...]
frameworks:['browserify','jasmine'],
files:[
'../node_modules/angular/angular.js',
'../node_modules/angular-mocks/angular-mocks.js',
'spec/*.js',
'../scripts/*.js'
],
preprocessors:{
'../scripts/*.js':['browserify'],
'spec/*.js':['browserify']
},
[...]
Andnowwehavetofixthetests.Ofcourse,loadingsomecontrollerisnotgoingtocutitanymore.That,unfortunately,meansthatwehavetorewriteourtestsprettymuchcompletely.WecanusetheJasminebeforeEachfunctiontosetupsomeinitialorderandthenuseourteststoupdatetheorderandcheckthetotals:
describe('webapptests',function(){
describe('orderobject',function(){
varo=require('../../scripts/order.js');
varorder;
beforeEach(function(){
order=newo.Order();
order.lines.push(newo.Line({
product:{
price:1.23
},
number:1
}));
order.lines.push(newo.Line({
product:{
price:5
},
number:1
}));
});
it('shouldcorrectlycalculatethetotal',function(){
expect(order.total()).toBe(6.23);
});
});
});
Andweshouldaddsometeststocheckwhetheraddingandremovingproductsalsocorrectlyupdatesthetotals:
it('shouldupdatethelinesubtotalwhenthenumberofproductsischanges',function(){
varline=order.lines[0];
line.number=2;
expect(line.subTotal()).toBe(2.46);
});
it('shouldupdatethetotalwhenaproductisadded',function(){
order.lines.push(newo.Line({
product:{
price:2
},
number:1
}));
expect(order.total()).toBe(8.23);
});
it('shouldupdatethetotalwhenaproductisremoved',function(){
order.removeLine(order.lines[0]);
expect(order.total()).toBe(5);
});
Andthatisprettymuchit.WehavenowfixedourKarmatests.However,westillhavesomewhatofaproblem.WecannowrunourtestsfromtheCommandPrompt,butwealsohadahandypagethatshowedusallourtestsinHTMLandenabledustorunanddebugtests.Unfortunately,thatpagebroke.Inordertofixit,weshouldbundlethespecfilejustlikewebundleallourotherJavaScriptfilesorweshouldremovetherequirefromourtestsomehow.Wewillgetbacktothetestpageshortly.
Wehavenowcomeatapointthatwereallywanttobeabletotestourcontrollersaswell.Afterall,theymaycontainpage-specificlogicthatwewanttotestwithoutallthehassleofcreatinganewfilethatwehavetoBrowserifyandallthat.Thecontrollers,however,arecallingourbackendandwereallydonotwantthat.Luckily,weareusingAngular,whichhasbuilt-independencyinjection.So,let'smakeuseofthatfeature.
AddingdependencyinjectiontoAngularisnotallthatdifficult,whichisprobablyoneofthereasonsitissopopular.Yousimplyhavetochangeyourrepository.jsfilesothatitaddstherepositoryasaservicetotheshopapp:
angular.module('shopApp',[])
.service('repositoryService',['$http',function($http){
'usestrict';
return{
[...]
};
}]);
BecausewearenowcreatinganAngularservice,wecanusedependencyinjectiontoinjectthe$httpmoduleaswell.Let'sfixtheindexpagefirst.homeControllerintheindex.jsfilecannowinjectrepositoryServiceratherthanrequireit:
angular.module('shopApp')
.controller('homeController',['$scope','repositoryService',function($scope,repository){
repository.getTopProducts(function(results){
$scope.topProducts=results;
});
$scope.searchTerm='';
$scope.addToCart=function(product){
repository.addToCart(product);
};
}]);
Noticethefeweffectsthishas.First,weretrievethemoduleratherthaninstantiateit.Thesecondparametertoangular.modulewasremoved.Next,sincewearenowallAngular,theneedfor$scope.$applyisgone!ThisisreallyAngular'srecommendedwayofworking.
Last,wewillneedtofixourHTMLbecausethebundledJavaScriptisnotgoingtoworkanymore.Simplyreplacethe<!--build:js-->blockinyourindex.ejsfilewiththeoriginalscriptsfornow.Thiswillbreakthepagesthatstilluserequire,butwewillfixthatinabit:
<scriptsrc="scripts\repository.js"></script>
<scriptsrc="scripts\index.js"></script>
And,ofcourse,forthistowork,weneedtotellNode.jsthatitmayservefilesfromthisfolder:
app.use('/scripts',express.static('scripts'));
Wecandothisforeverypage.Removethesecondparametertoangular.module,injecttherepositoryService,remove$scope.$apply,andfixtheEJSfiles.Weshoulddothesameforutils.jsandmakeitAngularutilsService.Unfortunately,wearenowgettingstuckagainwithdependenciesandwhoshouldcreatetheAngular
module.So,hereisadifferentapproach.Wearestillgoingtorequireallofourfiles,includingtherepository,butwearealsogoingtoinjectthemasaservice.Thisgivesusthebestofbothworldsanditwillmakeourservicesinterchangeablefortestingpurposes.So,onceagain,changetherepository.jsfile,soitexportsafunctionthatwillserveasaserviceforAngular:
module.exports=function($http){
'usestrict';
return{
[...]
};
};
WecannowcreatetheshopAppmodule,createrepositoryService,andinjectitintothecontroller:
angular.module('shopApp',[])
.service('repositoryService',['$http',require('./repository.js')])
.controller('homeController',['$scope','repositoryService',function($scope,repository){
[...]
Also,changeindex.ejs,soitlookslikeitdidbefore.Andnowwecanchangeutils.jsjustabit,soitreturnsafunctioninsteadofanobjectandwecanuseitasaservice:
module.exports=function(){
'usestrict';
return{
[...]
};
};
productController,forexample,shouldnowlookasfollows:
angular.module('shopApp',[])
.service('utilsService',require('./utils.js'))
.service('repositoryService',['$http',require('./repository.js')])
.controller('productController',['$scope','utilsService','repositoryService',function($scope,utils,repository){
[...]
Nowthatwehavedependencyinjectioninplace,wecanchangeourtestsandinjectanalternativerepository:
describe('webapptests',function(){
describe('shoppingcartcontroller',function(){
//modulehasanamecollisionwithBrowserify,
//sousethefullangular.mock.module.
beforeEach(angular.mock.module('shopApp'));
varcontroller;
var$scope={};
varrepositoryService={
getCart:function(callback){
callback({
authenticated:true,
shoppingCart:[{
product:{
price:1.23
},
number:1
},{
product:{
price:5
},
number:1
}]
});
},
removeLine:function(){
//Donothing.
}
};
beforeEach(inject(function($controller){
controller=$controller;
controller('shoppingCartController',{
$scope:$scope,
repositoryService:repositoryService
});
}));
it('shouldloadthecart',function(){
expect($scope.order.total()).toBe(6.23);
});
});
});
Keepinmindthatyouonlyhavetomockthefunctionsthataregoingtobecalledinyourtests.Inthiscase,weknowgetCartisgoingtobeexecutedinthecontrollersothatisthefunctionwemock.Servicesthatyoudonotneedtomock,suchastheutilsService,cansimplybeomittedfromthecontrolleroptionsobject.Wearenowtestingtheorder.jsfileusingrequireandtheshoppingCartControllerusingAngulardependencyinjection,allinonespecfile!
Thenextthingwewanttodoisunittestourorder.jsfileinNode.js.Wearealreadytestingitinourbrowsers,buttestingitonthebackendisequallyimportant,asJavaScriptinyourbackendcanbehavedifferentlyfromyourJavaScriptinthefrontend.Inourcase,wearenotusingorder.jsinthebackend,butforthesakeoflearning,wearegoingtotestitanyway.
Again,wecanuseJasmine,Mocha,orothers.SinceweareusingJasmine,let'sstickwiththat.Weneedtoinstallaseparatepackagethough,jasmine-node(https://
github.com/mhevery/jasmine-node):
npminstalljasmine-node-g
npminstalljasmine-node--save-dev
Createanewfileinyourspecfolderandnameitnode-spec.js.Jasmine-nodeneedsyourspecstobeinthespecfolderandyourfilestohaveanamesuchas*spec.js,*spec.coffee,or*spec.litcoffee(forCoffeeScript).So,createanewfileinthespecfolderandnameitnode-spec.js.JustmakesureyourNodetestsdonotmixwithyourbrowsertests.Fornow,wearesafeasourbrowsertestsareinafilenamedtest.jsandsotheyarenotpickedupbyjasmine-node.
Now,becausethisisstilljustJasmineandwewanttotestourorder.jsfile,again,wecanjustcopyandpasteourbrowserteststothisfile:
(function(){
'usestrict';
varo=require('../../scripts/order.js');
describe('webapptests',function(){
describe('orderobject',function(){
varorder;
beforeEach(function(){
[...]
});
it('shouldcorrectlycalculatethetotal',function(){
expect(order.total()).toBe(6.23);
});
[...]
});
});
})();
Now,removethedescribe('orderobject',...);sectionfromthetest.jsfile.OurKarmaconfigincludesalltheJavaScriptfilesinthespecfoldersothenode-spec.jsfileisalreadyincluded.TryrunningKarmaoncetoseeifthetestsareindeedexecutedandpass.Asforthenode-spec.jsfile,thattitledoesnotcoveritanymore,renameittoorder-spec.js.
Now,torunthetestsinjasmine-node,openupaCommandPromptinthetestfolderandsimplyrunjasmine-nodespec:
jasmine-nodespec
LikewithKarma,itispossibletowatchfilesandautomaticallyrunthetests
whenanyfileschanges.Towatchthetestfiles(allfilesinspec),simplyaddthe--autotestflag.ToalsowatchforchangesintheapplicationJavaScriptfiles,addthe--watchoptionwiththefoldersyouwanttowatchforchanges:
jasmine-nodespec--autotest--watch..\scripts
Nowthatwehaveremovedtherequirecallfromthetestscript,ourindex.htmltestpageworksagain,butonlyforthecontrollertests.Justmakesureyouaddtheorder.jsandtheshopping-cart.bundle.jsfiletoyourpage:
<scriptsrc="../scripts/order.js"></script>
<scriptsrc="../scripts/bundles/shopping-cart.bundle.js"></script>
WestillcannotaddtheorderteststoourtestHTMLpage,butatleastwehavegotsomething.
And,ofcourse,wewantsomereporting.Thejasmine-nodecommandcanoutputJUnitstylereportsbyspecifyinganoutputfolderandaddingthe--junitreportflag:
jasmine-nodespec--autotest--watch..\scripts--junitreport--outputnode-junit
Ofcourse,wealsowantcodecoverageforourNodetests.Forthis,wecanuseIstanbul(https://github.com/gotwarlost/istanbul).Wealreadyhaveitinstalledinnode_modules,becausewealsouseittogeneratecoverageinourKarmatests.Usagewithjasmine-nodeisabitdifferentandsometimesunexpected.Forexample,weneedtorunthecommandfromtherootofourproject(whereournode_modulesfolderandpackage.jsonare).Wemustalsousethelocallyinstalledjasmine-node,butnotinthe.binfolder,butthejasmine-nodeinstallationfolder:
node_modules\.bin\istanbul.cmdcovernode_modules\jasmine-node\bin\jasmine-nodetest\spec
Thatisquitealotoftyping!Luckily,wehaveashortcut.Upuntilnow,wehavebeenusingnpmasapackagemanageronly,butitcanrunscriptsaswell.Inpackage.json,add(orreplace)thefollowingnode:
"scripts":{
"test":"node_modules/.bin/istanbul.cmdcovernode_modules/jasmine-node/bin/jasmine-node----junitreport--outputnode-junittest/spec"
},
Thedoubledash(--)separatestheIstanbulflagsfromtheJasmine-nodeflags(otherwise,the--outputflagcausestrouble).Andnow,wecansimplyrunnpmtest
fromthecommand.Youcanaddmorescriptslikethattoyourpackage.jsonfile,suchasprepublish,preinstall,install,version,andpostversion.Isuggestyoureadthedocumentation(https://docs.npmjs.com/misc/scripts).
So,wecannowtestourscriptsinNode.jsandgeneratecoverage.TheonlythingweneedisareportintheCoberturaformatforJenkinstoprocess.ThedefaultreportisLCOV,whichwealsowantforSonarQube.Istanbulallowsustoaddvariousreportstothecommand:
"scripts":{
"test":"node_modules/.bin/istanbul.cmdcover--reportcobertura--reportlcovnode_modules/jasmine-node/bin/jasmine-node----junitreport--outputnode-junittest/spec"
},
AndthataddsallreportstoourNode.jstests.
Next,wearegoingtofixourProtractortests.Whenyourunyourtests,youwillseethattwotestsstillsucceedandtwotestsfail.Wecaneasilyfixoneofthefailingtests.Weusedtoreturnproducts1,2,and3astopproducts,butwithMongoDBinplace,wereturn0,1,and2.Sotheproductsshiftedabitandwenowgetaproductinaplacewherewedidnotexpectit.Ifwehadsomeactuallogicinplacetoselectthetopthreeproducts,thiscouldhavebeenaproblem,butnow,wecanchoosetoreturn1,2,and3orwecanchoosetofixthetestsoitexpectsproduct1insteadof2.Let'sgowiththelatter:
it('shouldnavigatetotheclickedproduct',function(){
[...]
//expect(title.getText()).toBe('TheGood,TheBadandTheUgly');
expect(title.getText()).toBe('CaptainAmerica:CivilWar');
});
Thesecondfailingtestisalittlemoredifficulttofix.Thetestcheckswhetherremovinganitemfromtheshoppingcartsucceeds.However,ittimesoutafterabout30seconds.Thatisbecausewearenottakentotheshoppingcartpageuntilwelogin.So,wewillneedtochangethetestsothatitlogsinfirst.Itisalsonicetorequesttheshoppingcartpage,butexpecttheloginpagetocomeup:
it('shouldremoveanitemfromtheshoppingcart',function(done){
browser.get('shopping-cart.html');
//redirect
browser.wait(EC.presenceOf($('[ng-controller=loginController]'),2000));
varusername=$('input[ng-model=username]');
varpassword=$('input[type=password]');
username.sendKeys('some_user');
password.sendKeys('some_password');
varsubmit=$('input[type=submit]');
submit.click()
//anotherredirect
browser.wait(EC.presenceOf($('[ng-controller=shoppingCartController]'),2000));
[...]
});
Whenweaddthistothetest,wefirstloginandproceedtotheshoppingcartpage.However,thereisstillaproblem.Thecontentsofourshoppingcartarerememberedfromourlastsession.Earlier,whenwedidnothaveabackendyet,wealwayshadthesameitemsinourcartuponopeningthepage.Now,wemayormaynothaveitemsinthecart.Sohereiswhatwewilldo.Wewillloadtheindexpage,addtwooutofthreetopproductstoourcart,gobacktothecart,andproceedtotestwhetherwecanremoveanitemfromthecart.Ifwesucceed,wewillrefreshthepage,checkwhethertheproductisstillgone,andlogout.Thatisquiteatest,butitwilltesttheentireshoppingcartmechanism.Asyoucanimagine,itwillbeprettyannoyingtohaveyourtestsmessupyourowntestuser,sobesuretocreateauserspecificallyforyourtests.So,thenextstepistoaddtwoproductstoourcart:
browser.get('index.html');
browser.wait(EC.presenceOf($('.btn.btn-primary'),2000));
varalert;
varbuttons=$$('.btn.btn-primary');
buttons.get(0).click();
browser.wait(EC.alertIsPresent(),2000);
alert=browser.switchTo().alert();
alert.accept();
buttons.get(1).click();
browser.wait(EC.alertIsPresent(),2000);
varalert=browser.switchTo().alert();
alert.accept();
Asyoucansee,weclickthefirstbutton,waitforthealerttopopup,acceptit,anddothesamewiththesecondbutton.Thatshouldhaveaddedtwoproductstoourshoppingcart.Next,wecanrequesttheshoppingcartpage,gettheproducts(thumbnails),deletethefirstproduct,andcheckthatthenumberofthumbnailschangedaswellasthecaptionofthefirstproduct.Thecodethatwasalreadytheredoesthat,sowedonotneedtochangeitmuch.Ofcourse,wedoneedtoaddsomecodetorefreshthepageandcheckwhethertheproductisstillgone:
browser.get('shopping-cart.html');
browser.wait(EC.presenceOf($('[ng-controller=shoppingCartController]'),2000));
browser.wait(EC.presenceOf($('.thumbnail'),2000));
varitems=$$('.thumbnail');
items.count().then(function(count){
varfirst=items.get(0);
first.$('h3').getText().then(function(caption){
first.$('button').click().then(function(){
//wedon'treallyhaveanythingtowaiton...
//waitingfortheelementnottobetheremaycause
//aninfiniteloop,sojustsleepforasecondandcontinue.
browser.sleep(1000);
varnewItems=$$('.thumbnail');
expect(newItems.count()).toBe(count-1);
expect(newItems.get(0).$('h3').getText()).not.toBe(caption);
//Refreshandcheckifthedeleteditemisstillgone.
browser.refresh();
browser.wait(EC.presenceOf($('.thumbnail'),2000));
varafterRefresh=$$('.thumbnail');
expect(afterRefresh.count()).toBe(count-1);
done();
});
});
});
Itisquiteabitofcode,someevenduplicated,butIwillleavetherefactoringuptoyou.Fornow,wejusthaveatestthatcountsover50linesofcode.So,thislasttestalmostdoeswhatitdidbefore,exceptitlogsinfirst,addssomeproductstothecart,andthenrefreshesthepagetocheckagain.Thatmeansitdoesnotreallymatterhowmanyitemswereintheshoppingcarttobeginwith;therewillalwaysbeatleasttwoitemsintheshoppingcart(unlessthetestfailsforwhateverreason).Itisprobablyagoodideatocreateaseparatefunctionthathandlesthelogin,soyoucanreuseitforotherteststhatrequireyoutobeloggedin.
YouseeIalsomadearemarkaboutaninfiniteloop.Thisonecostmehourstodebug,soIwillgiveyouthistiphereandnow.Waitingforelementsthatarenotonthepage(anymore)toberemovedfromthepageisnotagoodidea.YouwouldhopeSelenium(orProtractor)wouldnotfindtheelementonthepageandcontinuerightaway,butunfortunatelythatisnothowitworks.Sobeware!Whenyoufindyourtestsfailing,itisagoodideatostartuptheSeleniumserverinaseparatecommandwindowandcheckthelogs(starttheserverusingwebdriver-managerstart).
GulpYoumayhavenoticed,butourGulpfilefailstorun.Basically,everythingiswelluptothetestsoftheminifiedJavaScriptfiles.Luckily,wecanfixthisreallyeasy!Inthekarma.min.conf.jsfile,simplymakesureyoubrowserifyeverythinginthespecfolder:
[...]
preprocessors:{
'../prod/scripts/*.js':['browserify'],
'spec/*.js':['browserify']
},
[...]
ThatshouldfixyourGulprun.Everythingshouldworknow,testing,linting,minifying,andsoon.Anotherthingthough,wehaveaddedtheNode.jstests,whicharenotyetincludedinourgulpfile.Wecansimplycreateanextrataskandrunjasmine-nodetests.Ofcourse,weneedanextraKarmaplugin,thekarma-jasmine-nodeplugin(https://www.npmjs.com/package/gulp-jasmine-node):
npminstallkarma-jasmine-node--save-dev
Wecannowaddittoourgulpfile.Itisaverysmalltask,theonlythingworthmentioningisthatwehavetospecifyourfileexplicitly,becausewecannotjusttakeeveryJavaScriptfilebecausetest.jsdoesnotruninNode.js:
varjasmineNode=require('gulp-jasmine-node');
gulp.task('test-node',['build'],function(){
returngulp.src('test/spec/order-spec.js')
.pipe(jasmineNode());
});
Unfortunately,wenowmissourJUnitresultreportandtheIstanbulcodecoverage.AddingtheJUnitreportiseasyenough:
.pipe(jasmineNode({
reporter:[
newjasmine.JUnitXmlReporter(__dirname+'/node-junit',true,true),
newjasmine.TerminalVerboseReporter({
color:true,
includeStackTrace:true
})
]
}));
However,forourIstanbulcodecoverage,weneedanadditionalplugin:
npminstallgulp-istanbul--save-dev
WecannowcallIstanbulbeforerunningourtestsandthengenerateourreportsafterthetest.Istanbulneedsalittlesetupaswell.Itneedstointerceptrequiredfilesinordertotrackcoverage:
varistanbul=require('gulp-istanbul');
varjasmineNode=require('gulp-jasmine-node');
gulp.task('istanbul-setup',['build'],function(){
returngulp.src('scripts/order.js')
.pipe(istanbul())
.pipe(istanbul.hookRequire());
})
.task('test-node',['istanbul-setup'],function(){
returngulp.src('test/spec/order-spec.js')
.pipe(jasmineNode({...}))
.pipe(istanbul.writeReports({
reporters:['cobertura','lcov']
}));
});
Theistanbul-setuptaskpicksupallthefilesthatshouldbecovered;inourcasethatisonlytheorder.jsfile.ItthencallshookRequire(),whichoverwritesrequire()soitcantrackcoverage.WecanthencallwriteReportsandspecifythereportstogenerate.Let'saddanHTMLreport.Bydefault,alltheHTMLfilesarewrittentothecoveragefolder,whichmakesitkindofamess.Itispossibletoaddreportspecificoptions:
.pipe(istanbul.writeReports({
reporters:['cobertura','lcov','html'],
reportOpts:{
html:{dir:'coverage/html'}
}
}));
WehavenowaddedIstanbulcodecoveragetoourjasmine-nodetestingthroughGulp.Wecannowsimplyexecutegulptest-nodeinsteadofnpmtest.Wenowonlyneedtoaddthetest-nodetasktoourdefaulttasksoitrunseverytimeweexecutegulp:
.task('default',['lint','test-min','test-node']);
WeshouldalsoaddthenewreportfilestoourSonarQubeexclusions,sobesuretoaddthetwofolderstothesonar.exclusionspropertyinthesonar-project.propertiesfiles,coverage/**,node-junit/**.Thesamegoesforyour.gitignorefile.
Onelastthing,wewanttoBrowserifyourspecfilesowecanuseitinthetestHTMLpage.WealreadyhaveataskthatBrowserifiesallourJavaScriptfiles;weonlyneedtoaddthespecfile(s)toit.Weuseglobtofindthefilesand,withglob,wecanindicatemultiplepatternslikethis,{pattern1,pattern2,...}:
glob('{./scripts/*.js,./test/spec/order-spec.js}',function(err,files){
Thatfetchestheorder-spec.jsfile,butitplacesitinscripts/bundles.Personally,Iwouldratherhaveitintest/spec/bundles,whereitbelongs.Thisisalittlemorecomplex,butstillnotdifficult.Weareprocessingourfilesonebyone,whichmeanswehavethenameofthefilewhileweareprocessingit.Wesimplyneedtogetthedirectoryofthefileandappendthebundlesfoldertoit.Togetthedirectory,wecanusethepathmodule.Noneedforextraplugins,wealreadyhaveit:
varpath=require('path');
[...]
.task('js',['clean'],function(done){
[...]
glob('{./scripts/*.js,./test/spec/order-spec.js}',function(err,files){
[...]
vartasks=files.map(function(file){
returnbrowserify({
[...]
.pipe(gulp.dest(path.dirname(file)+'/bundles'))
[...]
Thatisreallyallweneedtochangetoselectfilesfromdifferentdirectoriesandoutputthosefilesto[originaldirectory]/bundles.So,youshouldnowfindyourbundlesfileintest/spec/bundles.Thatmeanswecanchangeourtestpage:
<!--includespecfileshere...-->
<scriptsrc="spec/test.js"></script>
<scriptsrc="spec/bundles/order-spec.bundle.js"></script>
Andyouwillnowhaveallthetestsinyourtestpageoverview,includingsourcemappingforeasydebugging.
JenkinsIfyoucommittedyourworktoJenkins,youmayhavenoticedtheJenkinsbuildbrokesomewhere.Wefixedmostofit;wefixedourGulpbuildafterall.However,someissuesremain.ThebuildprojectfailsbecauseofsomeSonarQubeissues.Youcandothreethings,fixyourcode(youwouldhavetoremovesomealert()andconsole.log()statements),disabletherulesinSonarQube,orconfigureyourJenkinsproject,soitwillnotfailbecauseofSonarQube.Thatlastoptionisthequickest;simplyremovetheQualityGatespost-buildaction.Inproductioncode,wewouldnotusealert()andconsole.log(),butfornow,Idonotfindthataproblem.
Withthebuildprojectrunningagain,weshouldpublishtheextrareportswearegeneratingforourNode.jstests.WehaveanextraCoberturareport,sotheXMLreportpatternshouldnowbetest/coverage/cobertura/*.xml,coverage/*.xml.WeshouldalsoaddanextraHTMLreport.TheHTMLdirectoryiscoverage/htmlandthetitleissomethinglikeNodeJSCoverageHTML.Last,thereistheextraJUnittestresult;thereportXMLsarenowtest/junit/*.xml,node-junit/*.xmlortest/junit/*.xml,selenium-junit/*.xml,node-junit/*.xmlforyourtestproject.Makesureyouaddthenewreportstobothyourbuildandtestprojects.
Otherthanthat,thetestprojectfails(afterquitealongtime).Readingtheoutput,wecanfindthateverythinggoeswell,rightuptotheProtractortests.AlltheProtractorteststimeout,whichmakessense,sinceweaddedabackendthatisnotcurrentlyrunning.Beforethischapter,weonlyhadfrontendfiles,sowedidnotneedtostartupanything.Youmaythinkfixingthisissueisaseasyasaddinganodeindex.jscommandtothebuild,butthatwillonlymakeyourbuildhangindefinitely.Instead,weneedsomethingtogetourNode.jsserverupandrunningwithoutblockingtheconsole.Wecould,ofcourse,permanentlyhostourwebsite,butthatwouldbequiteahassle.Infact,gettingyourscriptsdeployedandrunningfullyautomatedisthetopicofalaterchapter.Fornow,wejustneedtostartit,runourtests,andstopit.Enterpm2.
PM2TorunandmanageNode.jsapplications,youwillneedsomethingmoresophisticatedthannodemon.ThetoolwearegoingtouseisPM2,anadvanced,productionprocessmanagerforNode.js(http://pm2.keymetrics.io/).WearegoingtoinstallitlocallyforJenkinsandgloballyforourconvenience:
npminstallpm2--save-dev
npminstallpm2-g
WithPM2,wecanstartandstopNode.jsscripts:
pm2startindex.js
pm2list
pm2stopindex.js
pm2deleteindex.js
pm2kill
Inthecommanduserinterfacewecanseethatourwebsiteisactuallyrunning.
InJenkins,wehaveabitofaproblemthough,wecanstartupPM2andrunourtests,butwhenourtestsfailthenextstatement,whichwouldbestoppingPM2,isneverexecuted.Anynextbuildstepswillbeignoredaswell.Wecouldjustletthescriptrun,soitwillbeavailableonlocalhost,butthatwouldpreventusfrom
startingitlocallyasourJenkinsslaveandourdevelopmentmachinearethesame(whichisnotreallyanidealsituationanyway).So,basicallyweneedtobeabletoruntheservertwiceonthesamemachine.Timetointroducesomecommand-lineargumentstoourNode.jsscript,sowecanusedifferentports.Wecandothisusingcommand-line-args(https://www.npmjs.com/package/command-line-args):
npminstallcommand-line-args--save
Thepackageitselfworksprettyeasy:
varcommandLineArgs=require('command-line-args');
varoptions=commandLineArgs({name:'port',defaultValue:80});
varserver=app.listen(options.port,'127.0.0.1');
console.log('Serverrunningathttp://127.0.0.1:'+options.port);
Wecannowstarttheserverusingnodeindex.js--port=1234.Thisworksexactlythesameinnodemon.Notspecifyingaportwillmaketheserverrunonport80.UsingPM2,thislooksslightlydifferent,pm2startindex.js----port=1234;youneedthedoubledash.WecannowrunourscriptonadifferentportinJenkins.Wewoulddowelltonameourprocessaswell,sowecanreferenceitbynameratherthanscriptnameorrandomID.Beforewestartascript,wemustmakesureitisnotalreadyrunning.Ifthescriptisalreadyrunning,thenPM2willreturnanerror,stoppingtheexecutionofourtests.Unfortunately,whenProtractorfails,anysubsequentcommandswillnotbeexecutedeither,sowecannotdeletethewebsiteafterourtests.So,wereallyonlyhaveoneoption:deletethewebsitebeforewetrytostartit.However,PM2alsoreturnsanerrorwhenyoutrytodeleteawebsitethatdoesnotexist.Weshouldaddthe-s,orsilent,flagtothedeletecommand,soanyerrorsarenotreturnedandthescriptcontinuesexecution:
npminstall&&node_modules\.bin\gulp.cmd--env=prod--browsers=Chrome,IE,IE9,Firefox&&node_modules\.bin\webdriver-manager.cmdupdate&&
ThefinalpieceofthepuzzleistohaveProtractortestthewebsiterunningatport1234,nottheonewemayormaynothaverunningonport80.Wecaneasilyoverwriteanyconfigurationsettingbyaddinganextracommand-lineargument:
node_modules\.bin\protractor.cmd--baseUrl='http://localhost:1234'test\protractor.conf.js
Atthispoint,wehaveprettymucheverythingready,backend,tests,andJenkins,everythingexceptdeployment.Wewillgettodeploymentinalaterchapterand,
asyoumighthaveguessed,wewillneedPM2again.
SummaryInthischapter,wehaveaddedaNode.jsandMongoDBbackendforourwebsite.Asaresult,wehadtoaddsomeadditionalNode.jstests,changeourProtractortests,andspawnaNode.jsserverinJenkins.Inthenextchapter,wearegoingtodothesame,butusingC#andPostgreSQL.Inthenextchapter,wewillseesomedifferenttechnologiesandthetechniquesthatcomewiththem.
AC#.NETCoreandPostgreSQLWebAppInthepreviouschapter,wecompletedourwebshopexample.Afterthefrontendwecreated,built,andtestedinthepreviouschapters,weaddedthebackendandadjustedourtests.AllweusedsofarwasJavaScript.However,languages,suchasJavaandC#,requireothermethodsoftestingandbuilding.Someofthemajordifferencesherearethatthesearetypedandcompiledlanguages.Inthischapter,wearegoingtosetupthewebsiteagainbutthistime,usingC#.Asabackend,wewillusePostgreSQL,apopularSQLdatabase.AlongthewaywewillseehowbuildingandtestingdiffersfromthatofJavaScriptandhowtheVisualStudioenvironmenttoolinghelpsustodothethingsweneed.
TobuildourC#website,wearegoingtouseVisualStudioCode,thelittlebrotherofMicrosoft'sflagshipIDEVisualStudio.UnlikeVisualStudio,VisualStudioCodeisamultiplatformeditorthatrunsonWindows,Linux,andMac.AllexamplesinthischapteraretestedonWindowsonly,butshouldworkonLinux.Otherthanthat,weareusing.NETCore,therelativelynewmultiplatformversionof.NET.
Installing.NETCoreandVisualStudioCodeThefirstthingweneedtodobeforewegettoworkisinstallthe.NETCoreSDK.Headovertohttps://www.microsoft.com/net/download/coretodownloadthelatestversion.Ihaveusedversion2.0,butIamassumingMicrosoftiskeepingthingsbackwardcompatible.Assaid,IamusingthisonWindowssoIamassumingyouarealsousingWindows,butitshouldworkonLinuxjustaswell.Installationofthe.NETCoreSDKisstraightforward.
Onceyouhavefollowedtheinstallationwizardandinstalled.NETCore,itistimetoinstallVisualStudioCode(VSCode).Headovertohttps://code.visualstudio.com/toinstallthelatestversionofVSCodeforyourplatform.Again,installationisstraightforward.Bytheway,youarefreetousethefullVisualStudio(CommunityEdition)ifyoulike.Themainpointofthischapteristhecode,nottheeditor.However,sincewewillneedtocompileourC#code,Ireallyrecommendusingeitheroneoftheseastheycomewithabuilt-incompilerthatcompilesyourcodeonasinglekeypress.Inthischapter,IwillnotbecoveringhowtomanuallycompileyourC#code(well,wewillhavetoinJenkins,butfornow,Iwillnot).
Afterinstallation,startVSCode.Itmaysurpriseyou,butVSCodeissolightweightitdoesnotcomewithstandardC#tooling.ToinstalltheseheadovertoExtensions,whichisthebuttonatthebottomoftheleft-handsidemenu.YoushouldseetheC#extensionalongwithotherextensions,suchasPython,C/C++,PowerShell,andGo.SimplyinstalltheC#extension,whichshouldonlytakeafewseconds:
Next,gototheExplorer,whichisthetopbuttonontheleft-handsidemenu.Itwillsaytherearenofoldersselected,butinsteadofselectingourweb-shopfolder,wewillcreateanewfolder.SinceC#issoverydifferentfromNode.js,wewillprobablynotbereusingalotofcode,savefortheHTML,CSS,andJavaScript,butitiseasiertocopythoselater.SinceIhavetwofolders,oneforweb-shopandoneforthebookasawhole,Ihavedeletedtheweb-shopfolderandrecreateditasanemptyfolder.So,forthisexample,Iwillstillusetheweb-shopfolder.Onceyournewfolderiscreated,openuptheVSCodecommandwindoworaseparatecommandwindow.YoucanopenthecommandinVSCodeusingCtrl+`(thatistheback-quotecharacter)orfromtheViewmenuatthetop,whichlistsitasIntegratedTerminal.Insidetheterminalwindow,enterdotnetnewmvc.ThiswillscaffoldanewASP.NETCoreMVCproject,themultiplatformMicrosoftwebplatform:
Beforewecontinue,Icanrecommendcreatinganothernewprojectandinitializingitusingjustdotnetnewconsole.ThiswillcreateaHelloWorldprogramthatyoucanusetoplayaroundwith.VSCodewillwarnyouthatrequiredassetstobuildanddebugcodearemissingandwillaskifyouwanttoaddthem.Iguessyouknowtheanswertothatone:yes,wewanttoaddthem.And,therearealsounresolveddependencies,soyoucaneitherrestorerightawayorexecutethedotnetrestorecommandfromtheterminal.YoucanrunanddebugthecodeusingF5orfromtheDebugmenu.Youcanaddbreakpointstoyourcodebyclickingonthecolumnbeforetherownumbers.VSCodewillbreakonthesebreakpointsduringdebuggingsoyoucaninspectvariables:
SothatwasareallyquicktourofVSCode.Now,let'sreturntoourweb-shopproject.
CreatingtheviewsLet'sreturntotheweb-shopproject.Ifyouhavenotyetaddedtherequiredassetsfordebuggingandresolvedyourdependencies,youshoulddothis.Now,alittleinformationonASP.NETMVC.MVCisshortforModel-View-Controller.Basically,wehaveacontroller,whichissittingonourbackendwaitingforrequests.Wheneverarequestissenttothecontroller,itwillserveupanHTMLpage,aso-calledview.Thecontrollertypicallypassesamodel,someC#classwithpropertiesandmethods,totheviewwhichtheviewcanusetorenderinformationonthepage.MVCisacommonpatternusedinmanylanguages(itiscertainlynotC#-or.NET-specific).
Inyourfileexplorer,youcanseethat.NETalreadycreatedaHomeControllerandHomeview.MVCisconvention-based,meaningthatwhenyoubrowseforaHomepage,inthisexamplehttp://mywebsite.com/Home,MVCwilllookforHomeController,whichwilllookforaHomeviewtoserve.ThefollowingcodeistheIndex()methodonHomeController.TheIndex()methodiscalledbydefaultwhenyoubrowsetoapage.NoticehowitsimplyreturnsView(),butstillknowswhatviewtoreturn:
publicIActionResultIndex()
{
returnView();
}
ThiscodewilllookfortheIndex.cshtml(C#HTML)fileintheViews\Homefolder.ThesamegoesfortheAbout()andContact()methodsonHomeController;theywilllookforViews\Home\About.cshtmlandViews\Home\Contract.cshtml,respectively.Youcanchangetheserules,astheyaresimplyconfiguredincode.YoucanfindthisinStartup.cs:
app.UseMvc(routes=>
{
routes.MapRoute(
name:"default",
template:"{controller=Home}/{action=Index}/{id?}");
});
Asforthecshtmlfiles,theyaretransformedby.NETtoproperHTMLpages,likeEJStransformedourHTMLinNode.js.ThisiscalledtheRazorengine.By
default,MVCcreatesa_LayoutpageintheSharedfolder.This_Layoutpageisusedasabaselayoutforallyourscripts.Ifyoulookatityouwillfinda@RenderBody()methodcallinthe_LayoutpagewhichrenderstheHTMLinthepagethatiscurrentlybeingrequested.So,putdifferently,theView()functionlooksfortheappropriateviewandwrapsitinthe_Layout.cshtmlview.Atthebottomofthelayoutpageisalso@RenderSection("Scripts",required:false),whichmeansanypagecanimportadditionalscripts,whichareplacedatthebottomofthepage.Thelastimportantdetailisinthetitleofthepage,<title>@ViewData["Title"]-web_shop</title>.TheViewDataisadictionary-likeobjectthatissharedbetweencontrollersandviewsandcancontainanyvalueyoulike.Youcanseethetitlesbeingsetintheseparateviews,ViewData["Title"]="HomePage";.Furthermore,itisimportanttoknowthatthecshtmlfilesarerenderedinthebackendandsoitispossibletorunanyC#codewithinbyprefixingwith@,like@RenderBody()does.ForablockofC#code,likewhensettingthetitles,youcanusecurlybraces@{...}.YoucanfindtheJavaScriptandCSSfilesinthewwwrootfolder.SimplyhitF5andyourwebsitewillbehostedlocallyandyoucanbrowsetolocalhost:5000,localhost:5000/Home/About,andallothercontrollersandmethodsyouhave.TheprebuiltstandardwebsiteisactuallysomeinformationonASP.NETCore,VisualStudio,Azure,andotherMicrosoftproducts.
So,let'sstartbychangingthethe_Layout.cshtmlfile.Wearebasicallygoingtoinvertourpreviousviews.WithEJS,wehadapieceofcommonHTMLthatweinjectedintoeachview.ButwithRazor,wearegoingtoinjecttheviewintothecommonHTML:
<!DOCTYPEhtml>
<html>
<head>
<metacharset="UTF-8">
<title>CIWebShop</title>
<linkrel="shortcuticon"href="https://www.packtpub.com/favicon.ico">
<linkrel="stylesheet"type="text/css"href="~/lib/bootstrap/dist/css/bootstrap.css"/>
<linkrel="stylesheet"type="text/css"href="~/css/layout.css">
<linkrel="stylesheet"type="text/css"href="~/css/utils.css">
<scriptsrc="~/lib/jquery/dist/jquery.js"></script>
<scriptsrc="~/lib/bootstrap/dist/js/bootstrap.js"></script>
<scriptsrc="~/lib/angular/angular.js"></script>
@RenderSection("Scripts",required:false)
</head>
<bodyng-app="shopApp">
<navclass="navbarnavbar-defaultnavbar-fixed-top">
<divclass="container">
<divclass="navbar-header">
[...]
<aclass="navbar-brand"href="@Url.Action("Index","Home")">CIWebShop</a>
</div>
[...]
</div>
</nav>
<divclass="container"ng-controller="@ViewData["ngController"]">
@RenderBody()
<footerclass="footer">
<p>Copyright©2017</p>
</footer>
</div>
</body>
</html>
So,thatisalotofcodeandIevenleftthebiggestpartout.Itshouldlookfamiliarthough.Intheheadelement,youcanseehowwereferencethestylesheetsandscripts.~(tildesymbol)denotesthewwwrootfolder.Thatway,nomatterwhereyourviewislocated,youalwaysstartfromthesamerootfolder.Youcansimplyputthelayout.cssandutils.cssfilesinthewwwroot/cssfolder--justcopythemfromapreviouschapter.YoucanalsodeletetheCSSfilesgeneratedby.NET.Whileyouareatit,youcandeletethecompletewwwroot\imagesfolder.NextaretheJavaScriptscripts.Theyareaddedinprettymuchthesameway,exceptthatwehavenotyetaddedanyoftheselibraries.Wewilldothatinaminute.Nextisthe@RenderSectionfunction,whichcanbeusedtorenderanyotherscriptsrequiredforthepagethatwearerequesting.Afterthatsimplycomestheentirenavbar;again,youcancopyitfromapreviouschapter(beforethebackendchanges,sowithout<%if(authenticated){%>...).Then,noticethe@ViewData["ngController"]bit.ng-appisthesameforourentireapplication(althoughitdoesnothavetobe),butng-controllerisdifferentoneachpage.Becausethefooteristhesameeverywhereanditiscontainedintheng-controllerelement,wearegoingtoinjectthecontrollerthroughViewData.AfterthatcomesRenderBodyandthefooter.
OnechangewearegoingtomakeisthatourURLsarenotgoingtoendwith.html.ItissimplynotthedefaultinMVC,sowearegoingtoremovethemfromourURLsaswell(whichwecouldhavedonewithNode.js,butneverdid).And,insteadofindex,ourmainpageiscalledhome(asitcallsHomeController).OnesafeandeasywaytocreateURLsinRazorisbyusingtheUrlHelper.Actionfunction,shortenedas@Url.Action.Thefirstparameteristhenameoftheactionyouwanttocall,sothatisIndex,andthesecondparameteristhenameofthecontrollerwithoutthe"Controller"suffix,soHome.Personally,IamnotafanofRazor,butthisisoneofthoseutilitiesthatyoujusthavetohave.Otherthanthat,wearenotactuallyusinganyRazor.
Nowweneedtoinstallthescripts.TheASP.NETCoreMVCtemplateusesBower(https://bower.io/)asthedefaultpackagemanagerforfrontendlibraries(itusesNuGetforbackendpackages--moreonthatlater).However,weneedtoinstallBowerbeforewecanactuallymakeuseofit.OpenuptheterminalinsideVSCodeorjustacommandwindowinWindowsandinstallBowerusing*pausefordramaticeffect*npm.Yes,weneedapackagemanagertoinstallapackagemanager:
npminstallbower-g
Likenpm,BowerusesaJSONfiletokeeptrackofwhatpackagesandversionsareinstalled.Youcanfinditinyourweb-shopfolderalready.Itlooksalittlelikethepackages.jsonnpmuses.Anyway,youcanseethatMVCCorecomespreinstalledwithbootstrap,jquery,jquery-validation,andjquery-validation-unobtrusive.Unfortunately,thelibrariesinstalledbyMVCarenotuptodateandwedonotneedhalfofthem.So,wearegoingtodeletethemallandtheninstalltheonesweneedagain:
boweruninstalljquery-validation-unobtrusive--save
boweruninstalljquery-validation--save
boweruninstalljquery--save
boweruninstallbootstrap--save
bowerinstallbootstrap--save
bowerinstallangular--save
InstallingBootstrapwillautomaticallyinstalljQuery,whichistheonlyreasonweneedit.Finally,wecaninstallAngular.Thattakescareof_Layout.WecannowchangetheViews/Home/Index.cshtmlfile,sowegettheactualindexpageofourwebshop:
@{
ViewData["ngController"]="homeController";
}
@sectionScripts{
<scriptsrc="~/js/repository.js"></script>
<scriptsrc="~/js/index.js"></script>
}
<divclass="jumbotron">
<h1>CIWebShop</h1>
<pclass="lead">WelcometotheCIWebShop!<br/>
Browseourwares,butremember:ifyoubreakityoubuyit!</p>
</div>
<divclass="row">
[...]
</div>
Ihaveleftmostofitout,butitisexactlythesameasinthepreviouschapters.It
isgoodtonoticethattheHTMLstartsattheBootstrapJumbotroncomponent,becausethatisprettymuchwherethenavbarendsandtheindexpagebegins.ThisisalsowherewespecifyngControllerinViewDataandspecifytheScriptssection.Idonotthinkthereisanyrocketsciencegoingonhere.Wedostillneedtocreatesomeindexandrepositoryscriptsthough.Fornow,copythescriptsfromChapter4,CreatingASimpleJavaScriptApp,andputtheminwwwroot/js.
Now,hitF5andyoushouldseetheindexpageasitoncewas.
Wecannowaddtheotherpagesandscripts.Justrememberthateachpage,ideally,getsitsowncontrollerand,thus,itsownfolderinviews.Thishelpstoseparateconcerns.WherewehadonehugeJavaScriptfileinNode.js,wearegettingalotofsmallfilesinC#.YoucanalsoremovetheAboutandContactfunctionsfromtheHomeControllerandthecshtmlfilesthatgowithitandthesite.jsandsite.min.jsfiles.LeavetheErrorfunction.So,let'scontinuewiththeProductpage:
usingMicrosoft.AspNetCore.Mvc;
namespaceweb_shop.Controllers
{
publicclassProductController:Controller
{
publicIActionResultIndex()
{
returnView();
}
}
}
Thatishowallyourcontrollersaregoingtolookfornow.TheProduct/Index.cshtmlpagelookslikeitdidinChapter4,CreatingASimpleJavaScriptApp:
@{
ViewData["ngController"]="productController";
}
@sectionScripts{
<scriptsrc="~/js/repository.js"></script>
<scriptsrc="~/js/utils.js"></script>
<scriptsrc="~/js/product.js"></script>
}
<divclass="col-lg-12text-center">
<h2>{{name}}</h2>
<p>
<imgsrc="http://placehold.it/300x300"alt="..."/>
</p>
<p>{{description}}</p>
<p>{{'€'+price}}</p>
<p>
<buttonclass="btnbtn-primary">Buy</button>
</p>
</div>
Whenyoucontinueonlikethis,youwillendupwithprettymuchwhatwehadinChapter4,CreatingASimpleJavaScriptApp,butwithabackend.ThecontrollerfortheshoppingcartisnamedShoppingCartControllerbecauseshopping-cartisnotavalidnameforaclassinC#.TheJavaScriptfilecanstillbenamedshopping-cart,ofcourse.
RunningtasksJustbecausewearenowworkinginC#itdoesnotmeanwedonothaveanyfrontendworriesanymore.Westillneedtominify,bundle,Browserify,andmanymore.Wecanhandleminificationandbundlingin.NET.ForBrowserify,wewillneedGulpagain.First,takealookatthebundleconfig.jsonfile.Itcontainssomebundlingandminificationinformationthatishandledby.NET:
//Configurebundlingandminificationfortheproject.
//Moreinfoathttps://go.microsoft.com/fwlink/?LinkId=808241
[
{
"outputFileName":"wwwroot/css/site.min.css",
//Anarrayofrelativeinputfilepaths.Globbingpatternssupported
"inputFiles":[
"wwwroot/css/site.css"
]
},
[...]
]
Togetthistowork,wefirstneedtoinstalltheBundlerMinifier.Coretool.YoucandothisbymanuallyeditingyourcsprojfileandaddingareferencetotheBundlerMinifier.Coretool:
<ItemGroup>
<PackageReferenceInclude="BundlerMinifier.Core"Version="2.4.337"/>
[...]
<PackageReferenceInclude="Npgsql.EntityFrameworkCore.PostgreSQL"Version="1.1.0"/>
<DotNetCliToolReferenceInclude="BundlerMinifier.Core"Version="2.2.281"/>
</ItemGroup>
Donotforgettorestoreittoo;VSCodewillpromptyouforitoryoucanrunthecommanddotnetrestoremanually.Afterthat,youcansimplyexecutedotnetbundleandyourscriptsarebundledaccordingtobundleconfig.json:
dotnetrestore
dotnetbundle
Ofcourse,thebundletoolwillgivesomewarningsthatitcannotfindyourfiles(ifyouhavedeletedthem).Fornow,wecanjustminifyandbundleourCSSfiles,aswearegoingtobundleourJavaScriptusingBrowserifyanyway.bundleconfigshouldlookasfollows:
[
{
"outputFileName":"wwwroot/css/site.min.css",
"inputFiles":[
"wwwroot/css/layout.css",
"wwwroot/css/utils.css"
]
}
}
YoucannowsimplyreplacealltheCSSfileswiththisonefileinyourviews.
Thenextstepistocreateatask.Inthe.vscodefolderisatask.jsonfile.ItdefinessometasksthatcanbeexecutedusingVSCode.Youwillfindthebuildtaskalreadydefined.Wearegoingtoaddabundletask:
{
"version":"0.1.0",
"command":"dotnet",
"isShellCommand":true,
"args":[],
"showOutput":"always",
"tasks":[
{
"taskName":"build",
[...]
},
{
"taskName":"bundle",
"isBuildCommand":true
}
]
}
taskNameisusedascommandtodotnet,sowedonothavetospecifyanyargs.YoucanrunanytaskfromtheCommandPalette(foundundertheViewmenuitemortheshortcutCtrl+Shift+P).SimplytypeTaskandchooseTasks:RunTask.Youcanthenchoosefromyourdefinedtasks:
Asyoucanseefromthescreenshot,thereisalsoaTasks:RunBuildTask(Ctrl+
Shift+B)command.Runningthisrunsyourdefaulttask,whichisthefirsttaskthathasisBuildCommandsettotrue.
Next,weneedtosetupBrowserify.WecandothisusingGulp.Thisisbasicallyacopyandpastefromourpreviouschapterwithafewedits.So,creategulpfile.jsandputinthejstaskfromthepreviouschapter,butrenameittobrowserify:
vargulp=require('gulp'),
glob=require('glob'),
browserify=require('browserify'),
source=require('vinyl-source-stream'),
path=require('path'),
rename=require('gulp-rename'),
es=require('event-stream'),
buffer=require('vinyl-buffer');
gulp.task('browserify',function(done){
glob('wwwroot/js/*.js',function(err,files){
if(err){
done(err);
}
vartasks=files.map(function(file){
returnbrowserify({
entries:[file],
debug:true
})
.bundle()
.pipe(source(file))
.pipe(buffer())
.pipe(rename({
extname:'.bundle.js',
dirname:''
}))
.pipe(gulp.dest(path.dirname(file)+'/bundles'));
});
es.merge(tasks).on('end',done);
});
})
.task('default',['browserify']);
Ofcourse,wealsoneedtoinstallthenecessarynpmpackages,suchasgulp,glob,browserify,andmanymore.Basically,everythingthatisrequiredinthegulpfile.Youknowhowtodoit,butdonotforgettocreateapackage.jsonfileeithermanuallyorusingnpminit.Ofcourse,itiseasiesttojustcopyandpastepackage.jsonfromthepreviouschapter.Whateveryoudo,dependenciesanddevDependenciesshouldlookasfollows:
"dependencies":{
},
"devDependencies":{
"browserify":"^14.1.0",
"event-stream":"^3.3.4",
"glob":"^7.1.1",
"gulp":"^3.9.1",
"gulp-rename":"^1.2.2",
"vinyl-buffer":"^1.0.0",
"vinyl-source-stream":"^1.1.0"
}
YoucannowsimplyrunnpminstallsoallpackageswillbeinstalledandthenyoucanrungulpandyourfileswillbeBrowserified.WecannowsetupatasksoyoucanrunGulptasksfromVSCode.YoucansetupGulpasyourdefaulttaskrunnerandVSCodewillpickupanytasksyoudefineautomatically,butsinceyoucanonlyhaveonetaskrunner,thatwouldpreventusfromrunningourcurrentcommand-linetasks.Ofcourse,wecanstillsetupataskthatjustrunsGulpfromthecommandline.Weneedtodoabitofreworkinginourtask.jsonfilethough:
{
"version":"0.1.0",
"tasks":[
{
"taskName":"build",
"command":"dotnet",
"args":["build"],
"isBuildCommand":true,
"problemMatcher":"$msCompile"
},
{
"taskName":"bundle",
"command":"dotnet",
"args":["bundle"]
},
{
"taskName":"gulp",
"command":"gulp",
"isShellCommand":true
}
]
}
Becauseeachtasknowhasalocalcommand,thesuppressTaskNameoptionistruebydefaultandsothetasknameisnotaddedtothecommandanymore,hencetheargsparameters.YoucannowrunthegulptaskfromVSCode.Optionally,youcanaddataskspecificallyforBrowserify--justadd"args":["browserify"]tothegulptask(ortoanewtaskthatiscopiedfromgulp).
So,youarenowprobablythinkingthisisawesomeandyouwanttomakealltasksbuildcommands,sotheyallgetexecutedwhenyoubuildyoursoftware.Unfortunately,andIpersonallythinkthismakestheentiretasksystemprettyuseless,youcanonlyrunonetaskatatime.Inyour.vscodefolder,thereisa
launch.jsonfile.IthasapreLaunchTaskproperty.Itcanonlyhandleonetaskfromthetasks.jsonfile:
{
"version":"0.2.0",
"configurations":[
{
"name":".NETCoreLaunch(web)",
"type":"coreclr",
"request":"launch",
"preLaunchTask":"build",
[...]
}
So,youseehowdefiningeverythinginGulpstillmakessenseevenwiththisbrandnewshinyVSCodetasksystem.However,wearestillgoingtousethistasksystemjustforfun.Afterall,wehaveseenplentyofGulpinthepreviouschapter.Wesimplydefineanewtask,descriptivelynameitdoItAll,andmakeitdoeverything:
{
"taskName":"doItAll",
"command":"cmd",
"isShellCommand":true,
"args":[
"/C\"dotnetbuild&&dotnetbundle&&gulp\""
],
"isBuildCommand":true
}
Makesuretoremove"isBuildCommand":truefromthebuildtask.Now,setpreLaunchTaskto"doItAll"inlaunch.json.Ofcourse,wecanseethisbecomesamesswhenourprojectgetsbigger,butitwillnot.Bytheway,itmaybeagoodideatothrownpminstallintheresomewhere.
NowthatwehaveBrowserifyinplaceandrunningonabuild,weneedtoactuallyuseitinourviews.Andwemightaswellimplementminificationwhileweareatit.First,thebadnews:thisisnojobforbundleconfig.jsonbecauseBrowserifyisalreadyabundler.So,wearegoingtouseGulp.Thenthegoodnews:itisnotasdifficultasyouwouldthink.First,weneedtwomorenpmpackages,gulp-size(optional)andgulp-minify.Afterthat,itisjustamatterofputtingitinthebrowserifytask:
returnbrowserify({
entries:[file],
debug:true
})
.bundle()
.pipe(source(file))
.pipe(buffer())
.pipe(rename({
extname:'.bundle.debug.js',
dirname:''
}))
.pipe(gulp.dest(path.dirname(file)+'/bundles'))
.pipe(size({
title:'Beforeminification',
showFiles:true
}))
.pipe(minify({
ext:{
min:'.js'
},
noSource:true
}))
.pipe(size({
title:'Afterminification',
showFiles:true
}))
.pipe(rename(function(file){
file.basename=file.basename.replace('.debug','');
}))
.pipe(gulp.dest(path.dirname(file)+'/bundles'));
Firstofall,noticethatIamnamingtheoriginalbundledfiles.bundle.debug.js.Afterthat,weareminifyingourbundlesandrenamingthefilestoget.debugoutagain.Boththeoriginalbundlesandtheminifiedbundlesarewrittentowwwroot/js/bundles.Wecannoweasilyreferenceoneormorespecificscriptsbasedonourbuildconfiguration:
@sectionScripts{
<environmentnames="Development">
<scriptsrc="~/js/bundles/index.bundle.debug.js"></script>
</environment>
<environmentnames="Staging,Production">
<scriptsrc="~/js/bundles/index.bundle.js"></script>
</environment>
}
Youcanfindtheconfigurationinthelaunch.jsonfile:
[...]
"env":{
"ASPNETCORE_ENVIRONMENT":"Production"
},
[...]
WecannowchangeallofourscriptssothattheyuseAngular.jsdependencyinjectionagain:
//repository.js
module.exports=function($http){
[...]
};
//index.js
angular.module('shopApp',[])
.service('repositoryService',['$http',require('./repository.js')])
.controller('homeController',['$scope','repositoryService',function($scope,repository){
[...]
}]);
//etc.
Thatsetsupourtasksandfrontendworkfornow.Wearesettogetourdatafromthedatabase.Afterthat,wecanaddtestingandlintingandbuildourwebsiteinJenkins.
AddingthedatabaseNowthatwehavewhatweprettymuchstartedwithinthepreviouschapters,thistime,inC#,wecancontinuebyaddingadatabaseconnection.YoucanfindallSQLscriptsinthischapterintheGitHubrepositoryinthesqlfolderinthechapterfolder.WeinstalledPostgreSQLandpgAdmininChapter2,SettingUpACIEnvironment,becausewealsoneededittorunSonarQube.So,openuppgAdminandfindtheconnectionyoumadeinChapter2,SettingUpACIEnvironment.Ifyoudonothaveitanymore,youcanreadhowtogetitinChapter2,SettingUpACIEnvironment,soIwillnotrepeatthathere.NowthatyouareconnectedtotheserverinpgAdmin,wecancreatethewebshopdatabase.Eitherright-clickontheDatabasesnodeandselectCreateandthenDatabase;justgivethedatabasethenamewebshopandsave.Oryoucanconnecttothedefaultpostgresdatabase,right-clickonit,andthenopenQueryTool.PutthefollowingSQLscriptinsidethequerytoolandexecuteit(usingthebuttonwiththethundericonorbypressingF5).Youmayneedtochangetheownerifyourusernameisnotsa,likemine.Beforeyoucanseeyournewdatabase,youhavetorefreshthelistbyright-clickingontheDatabasesnodeandthenclickingRefresh:
CREATEDATABASEwebshop
WITH
OWNER=sa
ENCODING='UTF8'
CONNECTIONLIMIT=-1;
Theoutputisasfollows:
Itisagoodideatokeepascriptofeverything.Infact,itisabsolutelynecessaryifyouwanttocontinuouslydeliveryourdatabase(oratleastdoasomewhatsmoothmanualdeployment).
Thenextthingweneedisacoupleoftables.UnlikeMongoDB,PostgreSQLisaSQLdatabaseandhasafixedschema.Allofourtables(collectionsinMongoDB)areknownupfrontandhaveaknownsetofcolumns.Ourfirsttableistheproducttable.Followingthepreviouschapter,aproducthasanID,name,price,description,andcategory.Theyareallprettystraightforward,exceptmaybecategory.Wehavetwooptionshere:makeitaregulartextcolumnorspecifyasetofcategoriesandcreateaforeignkeyrelation.Let'sgowiththeeasyoptionandjustmakeitatextcolumn.Right-clickonthewebshopdatabaseandopenQueryTool.YoucannowexecutethefollowingSQLstatementtocreatetheproducttable:
CREATETABLEpublic.product
(
idserialNOTNULL,
nametextNOTNULL,
pricemoneyNOTNULL,
descriptiontext,
categorytext,
PRIMARYKEY(id)
)
WITH(
OIDS=FALSE
)
TABLESPACEpg_default;
ALTERTABLEpublic.product
OWNERtosa;
Thenexttableistheusertable.Theusertableisprettyeasy--justusernameandpassword:
CREATETABLEpublic."user"
(
idserialNOTNULL,
usernametextNOTNULL,
passwordtext,
PRIMARYKEY(id)
)
WITH(
OIDS=FALSE
)
TABLESPACEpg_default;
ALTERTABLEpublic."user"
OWNERtosa;
InMongoDB,thiswasprettymuchit.Anyproductswerestoredinanarrayintheusertable.However,thatisnothowSQLworks.Wewillneedathirdtablethatservesasashoppingcartandconnectsourusertooneormoreproducts:
CREATETABLEpublic.shopping_cart
(
idbigserialNOTNULL,
user_idintegerNOTNULL,
product_idintegerNOTNULL,
"number"integerNOTNULLDEFAULT1,
PRIMARYKEY(id),
CONSTRAINTfk_shopping_cart_userFOREIGNKEY(user_id)
REFERENCESpublic."user"(id)MATCHSIMPLE
ONUPDATENOACTION
ONDELETENOACTION,
CONSTRAINTfk_shopping_cart_productFOREIGNKEY(product_id)
REFERENCESpublic.product(id)MATCHSIMPLE
ONUPDATENOACTION
ONDELETENOACTION
)
WITH(
OIDS=FALSE
)
TABLESPACEpg_default;
ALTERTABLEpublic.shopping_cart
OWNERtosa;
ThefollowingdatabasediagramwascreatedusingSchemaSpy(http://schemaspy.sourceforge.net/).ItvisualizesyourPostgreSQLdatabasequitewellforfree.Prettyawesome.Anyway,yourdatabaseshouldnowlookasfollows:
Ourdatabaseisstillprettyempty,solet'sinsertsomedata.Firsttheproducts,thenouruser:
INSERTINTOpublic.product(
name,price,description,category)
VALUES
('FinalFantasyXV',55.99,'FinalFantasyfinallymakesacomeback!','Gaming'),
('CaptainAmerica:CivilWar',19.99,'EvenmoreAvengers!','Movies'),
('TheGood,TheBadandTheUgly',9.99,'Thistimelessclassicneedsnodescription.','Movies'),
('J.K.Rowling-FantasticBeastsandWheretoFindThem',19.99,'NotHarryPotter.','Books'),
('FantasticFour',11.99,'Supposedly,averybadmovie.','Movies');
INSERTINTOpublic.user(
username,password)
VALUES
('user','password');
YoucancheckthecontentsofyourtablesusingaSELECTstatement(onebyone):
SELECT*FROMpublic.product;
SELECT*FROMpublic.user;
WehavenowsetupthedatabaseandwearereadytouseitfromC#.
EntityFrameworkCoreForourdatabaseoperationswearegoingtousetheEntityFramework,Microsoft'sObjectRelationalMapper(ORM).TheEntityFramework(EF)hasa.NETCoreversionandPostgreSQLhasanEFCoreProvider.IcouldwriteabookonEFalone,butfornow,itsufficestoknowtherearebasicallythreewaysofworkingwithEF:databasefirst,codefirst,oramixofthetwo.Thedatabasefirstapproachassumesyouhaveadatabasealreadyinplaceandgenerates,orletsyoumanuallywrite,classesbasedonthat.Incode,firstyoubasicallydescribeyourdatabaseincode,afterwhichEFgeneratesthescriptsnecessaryto(automatically)updateyourdatabasetothelatestschema.Thethirdapproachassumesyoualreadyhaveadatabase,butalsoallowsyoutoworkcodefirstfromthereon.Codefirstisidealfromacontinuousintegrationperspectiveasallyourdatabasescriptsarealreadyautomaticallygeneratedandexecuted.Unfortunately,thisisnotreallyanoptionformostmatureprojects,ashuge(legacy)databasesaresometimeshardtoscriptlikethat.AnevengreaterlimitingfactorisaDBAorprojectmanager,whostrictlyforbidsallautomaticschemaalterationonadatabasebecauseitisunreliableorposesarisk(likepeopleandmanualscriptsarenot).Anyway,codefirstisnotalwaysanoption.Forthisproject,wewillbeusingdatabasefirst,asitintroducessomenicechallengeslateronwhenwearegoingtocontinuouslydeliveroursoftware.
First,weneedtoinstallthePostgreSQLEFCoredriver,whichcanbedoneusingtheVSCodeterminal.YoumayneedtorestartVSCodebecausetheintellisensedoesnotalwaysreloadautomaticallyandVSCodemaynotrecognizethepackageintheeditor.
Usually,wewouldusedotnetaddpackageNpgsql.EntityFrameworkCore.PostgreSQLfromtheterminal,butsomehow,thereisabugthatpreventsusfrominstallingpackagesafterwehaveinstalledatoolandhaveinstalledtheBundlerMinifier.So,wecandotwothings:removeBundlerMinifierfromthecsprojfileandaddthelatestsessionpackagethroughdotnetaddpackageorwecanmanuallypastethepackageintocsproj.Bythetimeyouarereadingthis,chancesarethatthisbughasbeenresolvedandyoucanjustusedotnetaddpackage:
[DisabletheBundlerMinifiertool]
<!--DotNetCliToolReferenceInclude="BundlerMinifier.Core"Version="2.2.281"/-->
[InstallthePostgreSQLpackageusingtheterminal]
dotnetaddpackageNpgsql.EntityFrameworkCore.PostgreSQL
[Re-enableBundlerMinifiertool]
<DotNetCliToolReferenceInclude="BundlerMinifier.Core"Version="2.2.281"/>
[Addtocsprojmanually]
<PackageReferenceInclude="Npgsql.EntityFrameworkCore.PostgreSQL"Version="2.0.0"/>
[Inanycase]
dotnetrestore
Afterthat,wemustwriteourdatabasecontextclassandourPlainOldCLRObject(POCO)classes(abigwordforjustaclass).CreateanewfolderandnameitDatabaseorsomesuch.Init,createanewfilenamedWebShopContext.cs.TheWebShopContextclassshouldlookasfollows:
usingMicrosoft.EntityFrameworkCore;
namespaceweb_shop.Database
{
publicclassWebShopContext:DbContext
{
publicDbSet<Product>Products{get;set;}
publicDbSet<User>Users{get;set;}
protectedoverridevoidOnConfiguring(DbContextOptionsBuilderoptionsBuilder)
{
optionsBuilder.UseNpgsql("Host=ciserver;Database=webshop;Username=sa;Password=sa");
}
}
}
WebShopContextisinheritingfromDbContext,anEFbaseclass.Init,wefindourDbSet<T>;basically,acollectionobjectthatrepresentsaquerytothedatabase.Intheconfiguration,wecanspecifyourconnectionstringtoPostgreSQL.BesuretochangeUsernameandPasswordifyouneedto.
Normally,youwouldnotcreateyourdatabasecontextandclassesinthesameprojectasyourcontrollers,views,andscripts.Inanyseriousproject,thereshouldbeaclearseparationofconcerns,andafrontendlayershouldnotbehandlingdatabaserequests.However,VSCodecannothandlemultipleprojectsatonce(useregularVisualStudioforthat)andsowearestuckwithwritingeverythinginthesameproject(thereisnorealprojectnotioninVSCode--itisreallyjustasinglefolder).Furtherdowntheroad,wearegoingtoplaceSQLqueriesincontrollers--suchpracticeswouldbethestuffofnightmares!Forourexampleitisfinethough.
Now,createtwonewfiles,alsointhedatabasefolder,namedProduct.csandUser.cs.TheProductclasslooksasfollows:
usingSystem.ComponentModel.DataAnnotations.Schema;
namespaceweb_shop.Database
{
[Table("product")]
publicclassProduct
{
[Column("id")]
publicintId{get;set;}
[Column("name")]
publicstringName{get;set;}
[Column("price")]
publicdecimalPrice{get;set;}
[Column("description")]
publicstringDescription{get;set;}
[Column("category")]
publicstringCategory{get;set;}
}
}
Unfortunately,boththetableandcolumnnamesarecase-sensitive.Ilikestickingtoacertaintechnology'spreferrednamingstyle,camelCaseinJavaScript,PascalCasein.NET,andall_lower_caseinPostgreSQL.ThatmeanswehavetomakesomesortofmappingbetweennaminginPostgreSQLand.NET.Asyoucansee,thiscanbedoneusingtheTableandColumnattributes.Ifyoudonotfindnamingconventionsworththetrouble,IcannotblameyouandyoumayjustwanttousePascalCasinginyourdatabaseorall_lower_casein.NET(althoughIrecommendtheformer).TheUserclasslooksprettymuchthesame.
WewillalsoneedtoaddaShoppingCartclass.Usually,itwouldmakesensetohavesomeheader-detailrelationship,asinShoppingCartandShoppingCartDetails.However,inourexample,theuserservesasthemasterrecordandshopping_cartisalreadyadetailtable.Ausercanonlyhaveoneshoppingcartever.Itcanbechangedoremptied,butnevercanauserhavemorethanoneshoppingcart(butitcanhavemorethanoneshopping_cartrowinit,astheyarereallyjustdetails).InEF,wecanindicatethatapropertyisaforeignkeyproperty,so-callednavigationproperties,soEFcanloadtheseentitiesforusautomatically.ForShoppingCart,wewanttocreateforeignkeypropertiesforUserandProductsowecaneasilyqueryforthemlater:
usingSystem.ComponentModel.DataAnnotations.Schema;
namespaceweb_shop.Database
{
[Table("shopping_cart")]
publicclassShoppingCart
{
[Column("id")]
publicintId{get;set;}
[Column("user_id")]
publicintUserId{get;set;}
[Column("product_id")]
publicintProductId{get;set;}
[Column("number")]
publicintNumber{get;set;}
[ForeignKey("UserId")]
publicUserUser{get;set;}
[ForeignKey("ProductId")]
publicProductProduct{get;set;}
}
}
WiththeWebShopContext,User,Product,andShoppingCartclasses,wehavemodeledoutourdatabaseinC#.
SelectingthedataWearenowreadytogettoworktoreplaceourrepositorymethodstouseAJAXrequeststofetchthedatafromthebackend.Let'sstartwiththetopthreeproductsonthehomepage.First,wemustdecidewheretoputourGetTopProductsfunction.Sincetheproductsarebeingshownonthehomepage,HomeControllerseemslikeagoodspot.Thefunctionitselfisprettyeasy,especiallycomparedtotheJavaScriptMongoDBcallbacks:
publicIActionResultGetTopProducts()
{
using(varcontext=newWebShopContext())
{
varproducts=context.Products.Take(3).ToList();
returnJson(products);
}
}
Thatisreallyallthereistoit.Thesethreelinesofcodeopenupaconnectiontothedatabase,generateaSQLquerythatisroughlyequivalenttoSELECTTOP3[allfields]FROMproduct,mapthemtoyourclass,andreturnthemasaJSONresulttoyourbrowser.Notonlythat,thebrowserreceivesallthiscamelCasedlikeweareusedtoinJavaScript!Youcanseetheresultwhenyoubrowsetolocalhost:5000/Home/GetTopProducts.However,wewerekindofexpectingGetTopProductstobeaPOSTcall,notaGETcall.WecaneasilychangethisbehaviorbydecoratingthefunctionwithHttpPostAttribute:
[HttpPost]
publicIActionResultGetTopProducts()
[...]
Onelastthing--EFsupportsnavigationproperties,likewesawintheShoppingCartclass.Thiscanleadtoinfinitereferences:aShoppingCartentityhasaProductentity,iscontainedinShoppingCartentities,hasaProductentity,andmanymore.Forthisreason,itisneveragoodideatosendanentitydirectlytoyourfrontend;chancesarethat,oneday,you'llendupwithanendlessloopinyourobjectgraphandyouwillneedtodosomeadditionalworktoproperlyserializeyourobjects(whichyouwillnotfindoutatdesigntime!).So,toeliminatethechancesofserializationexceptions,wearegoingtomaptheentitiestoacustom(anonymous)modelthathastheaddedbenefitofonlysendingthefieldsweneed
tothefrontend:
varproducts=context.Products
.Select(p=>new
{
Id=p.Id,
Name=p.Name,
Price=p.Price
}).Take(3)
.ToList();
returnJson(products);
Wecannowfixthefrontendlikewedidbefore.Inrepository.js,wejustquicklyinjectthe$httpmoduleandchangegetTopProducts:
return{
getTopProducts:function(callback){
$http.post('/Home/GetTopProducts')
.then(function(response){
callback(response.data);
},function(){
alert('Anerrorhasoccurred.');
});
},
[...]
};
And,intheindex.jsfile,wecanchangethecontroller:
repository.getTopProducts(function(results){
$scope.topProducts=results;
});
Next,wecanimplementGetProductinProductController:
[HttpPost]
publicIActionResultGetProduct([FromBody]ValueModel<int>model)
{
using(varcontext=newWebShopContext())
{
varproduct=context.Products
.Select(p=>new
{
Id=p.Id,
Name=p.Name,
Price=p.Price,
Description=p.Description,
Category=p.Category
})
.SingleOrDefault(p=>p.Id==model.Value);
returnJson(product);
}
}
BecausethisfunctiontakesPOSTrequests,weneedtogetthedatafromthe
bodyoftherequest(insteadoftheURLparameters).FromBodyAttributeontheinputparametermakesthathappen.Thebodywillmaptosomeobject.Ihavecreatedasmallgenericclassforsingle-valuefunctions.Wecouldsimplyputthevalueinthebody,butthatwillcausesomeproblemswhenputtingstringsinyourbody(isitastringorinvalidJSON?).IhaveputthemodelinanewfoldernamedModels:
publicclassValueModel<T>
{
publicTValue{get;set;}
}
Otherthanthat,itprettymuchlookslikeGetTopProducts,exceptweareusingSingleOrDefaultinsteadofTake(1).Onthefrontend,wecanimplementthisprettymuchlikebeforewithjustaminordifference--thePOSTdatawhichnowhasvalueinsteadofid:
getProduct:function(id,callback){
$http.post('/Product/GetProduct',{
value:id
})
.then(function(response){
callback(response.data);
},function(){
alert('Anerrorhasoccurred.');
});
},
And,ofcourse,donotforgettoeditboththeProduct/Index.cshtmlfileandtheproduct.jsfile.IdonotthinkIneedtoexplainSearchController.SearchProductsandShoppingCartController.GetCart.TheyworkprettymuchthesameasGetTopProductsandGetProduct.Youcanalsoremoveproductsfromtherepository.jsfilenow.
FixingtheloginNext,wearegoingtofixthelogin.Afterthat,ourwebsiteisprettymuchcompleteagainandwecangettotesting.Firstthingsfirst,ourloginmethodmakesuseofsessions.Gettingsessionstatein.NETCoreisprettywelldocumentedathttps://docs.microsoft.com/en-us/aspnet/core/fundamentals/app-state.Toenablesessionsin.NETCore,wemustinstallapackage.
Again,youmayneedtodisabletheBundlerMinifiertoolfirstandyoumayneedtorestartVSCodetogetittorecognizetheaddedtoolintheintellisense:
[Installusingtheterminal]
dotnetaddpackageMicrosoft.AspNetCore.Session
[Addtocsprojmanually]
<PackageReferenceInclude="Microsoft.AspNetCore.Session"Version="2.0.0"/>
[Inanycase]
dotnetrestore
Togetthesessionworking,weneedtoaddsomeinitializationinStartup.cs.TheConfigureServicesmethodneedstoaddthesessionfirst:
publicvoidConfigureServices(IServiceCollectionservices)
{
services.AddMvc();
//Addsadefaultin-memoryimplementationofIDistributedCache.
services.AddDistributedMemoryCache();
services.AddSession(options=>
{
options.Cookie.HttpOnly=true;
});
}
And,inthesameStartupfile,weneedtoexplicitlycallapp.UseSession()intheConfiguremethod:
publicvoidConfigure(IApplicationBuilderapp,IHostingEnvironmentenv,ILoggerFactoryloggerFactory)
{
loggerFactory.AddConsole(Configuration.GetSection("Logging"));
loggerFactory.AddDebug();
app.UseSession();
[...]
Andthatisallthatweneedtodotoenablesessions.Inthecontrollers,wecan
accessoursessionprettyeasilyaswell.InLoginController,wecancreateaLoginmethod:
usingMicrosoft.AspNetCore.Http;
[...]
[HttpPost]
publicIActionResultLogin([FromBody]LoginModelmodel)
{
using(varcontext=newWebShopContext())
{
boolsuccess=false;
if(context.Users.Any(u=>u.Username==model.Username&&u.Password==model.Password))
{
HttpContext.Session.SetString("Username",model.Username);
HttpContext.Session.SetString("IsAuthenticated",bool.TrueString);
success=true;
}
returnJson(new
{
Success=success
});
}
}
TheSessionobjectcanbeaccessedthroughtheHttpContextproperty(inheritedfromController).FortheSetString(extension)method,weneedtoaddareferencetoMicrosoft.AspNetCore.Http.So,wecansimplycheckwhethertheuserexistsinthedatabaseand,ifitdoes,wesetthesessionvariables.
Wecannowchange_Layout.cshtmlsoitshowstheloginorthelogoutbuttondependingonyoursession:
@usingMicrosoft.AspNetCore.Http;
[...]
@if(Context.Session.GetString("IsAuthenticated")==bool.TrueString)
{
<li><ahref="@Url.Action("Logout","Login")"><spanclass="glyphiconglyphicon-off"></span>Logout@(Context.Session.GetString("Username"))</a></li>
}
else
{
<li><ahref="@Url.Action("Index","Login")"><spanclass="glyphiconglyphicon-user"></span>Login</a></li>
}
TheSessionobjectcanbeaccessedthroughtheContextproperty,whichisaccessibleinallyourviews.WeusetheGetStringfunctiontocheckwhethertheuserisauthenticated.Iftheuserisauthenticated,weshowthelogoutbutton.ThelogoutbuttonlinkstotheLogoutactiononLoginController:
publicIActionResultLogout()
{
HttpContext.Session.Remove("Username");
HttpContext.Session.Remove("IsAuthenticated");
returnRedirectToActionPermanent(nameof(Index));
}
InourJavaScript,everythingstaysprettymuchasitwasinChapter8,ANodeJSAndMongoDBWebApp.Therearejustsomeminordetails.
AddingtothecartFortheinsertingofproductsintothedatabase,Iwanttodothingsalittledifferent.Basically,whenitcomestoSQL,Ihaveknowntwotypesofpeople.Thepeoplewhothinkdatabasesareanecessaryevilandshouldbeusedfordatastorageonlyandthepeoplewholovedatabasesandwhothinkalllogicshouldbeinthedatabase'sstoredproceduresandfunctions.Personally,Iamsomewhereinthemiddle.Thedatabasecertainlyhasitsplacewhenitcomestobusinesslogic,butIprefertokeepittoaminimum.Anyway,fortheinsertionofproducts,Idowanttouseastoredprocedure.WearegoingtosendauserIDandaproductIDtothedatabaseandletthedatabasefigureoutwhethertoinserttheproductfortheuserortoupdatethenumberfield.So,openupthequerytoolandexecutethefollowingSQLscript:
CREATEORREPLACEFUNCTIONadd_product_to_cart
(
p_usernameTEXT,
p_product_idINT
)
RETURNSvoidAS$$
DECLAREv_user_idINT;
BEGIN
v_user_id:=(SELECTidFROMpublic.userWHEREusername=p_username);
IFEXISTS(SELECT*FROMshopping_cart
WHEREuser_id=v_user_id
ANDproduct_id=p_product_id)THEN
UPDATEshopping_cart
SETnumber=number+1
WHEREuser_id=v_user_id
ANDproduct_id=p_product_id;
ELSE
INSERTINTOshopping_cart
(
user_id,
product_id,
number
)
VALUES
(
v_user_id,
p_product_id,
1
);
ENDIF;
END;
$$LANGUAGEplpgsql;
Youcantryitoutbyrunningthestoredprocedureinthequerytoolandchecking
theshopping_cardtable:
SELECTadd_product_to_cart('username',1);--MakesuretheIDsarevalid!
SELECT*FROMshopping_cart;
WecanaddthefunctioncallinWebShopContextbysimplyexecutingaqueryandpassingtheusernameandproductIDsasparameters:
publicvirtualvoidAddProductToCart(stringusername,intproductId)
{
this.Database.ExecuteSqlCommand("SELECTadd_product_to_cart(@username,@product_id);",
newNpgsqlParameter("@username",username),
newNpgsqlParameter("@product_id",productId));
}
InShoppingCartController,wecansimplycallthismethod.ItisreallynotallthatinterestingsoIamleavingthatpartout.TheonlypartthatmaybeinterestingtomentionistheRemoveProductFromCartmethod,thatmakesuseoftheUsernavigationpropertyforqueryinganddeletesarecordbeforecallingSaveChanges()whichsavesanyedits,adds,andremovalsinthecontext.TheGetCartmethodseemsabitbloated(anditkindofis),butmostofitisthemappingoftheshoppingcart,usingtheProductnavigationproperty,toacustomobject.
TestingthedatabaseNowthatwehavethisprocedureweprobablywanttotestit,justlikewetestourothercode.NexttotestingwhetheryourSQLisinorder,youcanalsotestwhetheryourschemaisinorder(doestableproductexistanddoesithaveapricecolumn,forexample).ThisissomethingthatiscompletelydifferentfromMongoDBasitdidnothaveaschema.Testingdatabasescanbeabitdifficult.Youneedtestdata,maybe,insertedwithsomeseedfunction,maybe,alreadypresent.Youdonotwanttomessupyourtestdatabymakingchangesinyourapplication.Youprobablyneedtosetupdatabaseconnectionsandtransactions.Itcanbeahassle.However,therearesometoolsthatmakeitabiteasier.pgTapisoneofthosetools.UsingpgTap,youcanwriteatestscriptinSQL,runitinaconsole,andoutputyourdatainTAPformat(TestAnythingProtocol,https://testanything.org/).
InstallingpgTapTheinstallationguideforLinuxisonhttp://pgtap.org/,butIwilltakeyouthroughithereaswellasIhadsomeissueswiththedocumentation.ThebadnewsaboutpgTapisthatitworksbestonLinux.ItcanworkonWindows,butitisabitofahassleandyouneedPerl,soIamnotgoingtoguideyouthroughit.So,wearegoingtoinstallpgTapontheserverthatrunsPostgreSQL,whichisourLinuxserver.First,weneedtodownloadthelatestversion(0.97.0atthetimeofwriting).Afterthat,weneedtounzipthedownloadedfileandbuilditusingmake(asdescribedinthedocs).Afterthat,wecanruntheinstallationscriptsoPostgreSQLinstallsallthenecessaryscriptsfortestingourcode:
sudoapt-getupdate
sudoapt-getinstallmake
wgethttp://api.pgxn.org/dist/pgtap/0.97.0/pgtap-0.97.0.zip
unzippgtap-0.97.0.zip
cdpgtap-0.97.0
sudomake
sudomakecheckinstall
sudomakeinstall
sudocpanTAP::Parser::SourceHandler::pgTAP
cdsql
psql[username]-h127.0.0.1-dwebshop<pgtap.sql
[enterpassword]
Thiswillinstalladazzling916functionsinyourdatabase'spublicschema.Foranyseriousapplications,youwoulddowelltocreatecustomschema'sforyourownfunctions.ThecpancommandisfordownloadingandbuildingPerlmodules.Inthiscase,itinstallspg_prove(http://pgtap.org/pg_prove.html)whichwewilluselater.Anyway,wecannowcreateatestscript.Basically,weneedatestuserandforhimtoaddaproducttohiscart,checkwhetheritwasadded,addthesameproductagain,checkwhetherthenumberwentup,andthenaddadifferentproductandcheckwhetherthatwasalsoaddedcorrectly.So,createatestuserandtwotestproducts.Then,putthefollowingscriptinafilenamedtest.sqlundertest/sql:
\setQUIET1
--Turnoffechoandkeepthingsquiet.
--FormattheoutputforniceTAP.
\psetformatunaligned
\psettuples_onlytrue
\psetpager
--Revertallchangesonfailure.
\setON_ERROR_ROLLBACK1
\setON_ERROR_STOPtrue
\setQUIET1
BEGIN;
SELECTplan(1);
DO$$
BEGIN
PERFORMadd_product_to_cart('test_username',1);
PERFORMadd_product_to_cart('test_username',1);
PERFORMadd_product_to_cart('test_username',2);
END$$;
SELECTresults_eq(
'SELECTuser_id,product_id,numberFROMshopping_cartWHEREuser_id=1',
$$VALUES(1,1,2),(1,2,1)$$,
'Theproductsshouldbeinsertedintotheshoppingcart.'
);
SELECT*FROMfinish();
ROLLBACK;
END
ThestuffatthetopissomestandardpgTapstuff,takenrightfromthetutorial.Itiscommented,soyoucanreadwhatitdoes.AfterthatcomestherealpgTaptest.Usingtheplanfunction,youcanspecifyhowmanytestsyouwanttorun.Theonlytestwehaveisthatwearegoingtoselectsomedataandcheckwhetheritisthedataweexpected,sowehaveonetest.Afterthat,wesimplyinserttheproductsintotheuser'sshoppingcart.WedothisinaPL/pgSQLblock,becausewedonotneedtheresult(usingSELECTwilloutputanemptylineinourtestresults).Withresults_eq,wecanconfirmthatthecorrectnumberofproductswasindeedinsertedintotheshopping_carttable.Thisisalsoourtest,so1inplan(1)ifyouwill.Withfinish(),wetellpgTapwearedoneanditcanoutputtheresults.Finally,werollbackthetransactionsoourshoppingcartisemptiedagainandwecanrunthetestagain.
YoucanrunyourtestsinWindowsusingthecommandprompt.Simplyusethepsqltool(inyourPostgreSQLinstallationfolder)andrunthescript:
cd"C:\ProgramFiles\PostgreSQL\9.5\bin"
psql-Usa-hciserver-dwebshop-fC:\Users\sander.rossel\Desktop\web-shop\test\sql\test.sql
Passwordforusersa:[enterpassword]
Theresultshouldlookasfollows:
Weknowitworksbecause,ifwechangethefunctionsothatitinsertstwoproductsinsteadofone,wegetanerrormessage:
Let'squicklyaddthreeotherteststoensure(partof)ourschemaiscorrect:
SELECTplan(4);
SELECThas_table('shopping_cart');
SELECThas_column('shopping_cart','number');
SELECTcol_type_is('shopping_cart','number','integer');
[...]
Thehas_table,has_column,andcol_type_isfunctionsdonotreallyneedadditionalexplanation--theirnamessayitall.Youcan,optionally,addaschemaasthefirstparameter,SELECThas_table('my_schema','my_table');,butweareusingthedefaultpublicschemasothereisnoneedforthat.Noticethatwenowplanforfourtests(plan(4))insteadofone.Eachfunctionisanadditionaltest.Youcanfindfunctions,suchasplan,results_eq,andhas_tableinthedocumentation(http://pgtap.org/documentation.html).
TestingourC#codeNowthatwehaveourSQLcovered,let'sgobacktoourC#codeforabit.Thereisnotreallyanythingtounittest,aswehavelittletonobusinesslogic.Itisalljustselectingdataandreturningittothefrontend.GreatforE2Etests;notsogreatforunittesting.So,justasinNode.js,let'screateanOrder.csfileandpretendweneeditforinvoicinglater.IhavecreatedanewfolderintheprojectrootandnameditInvoicing.Init,Ihaveplacedtwofiles,Order.csandOrderLine.cs.Bothareprettystraightforward.
TheOrderLineclassisprettymuchthesameasinJavaScript,exceptwecanworkwiththedecimaldatatypewhichdoesnothaveroundingerrorsandsowedonotneedtoround.ThemethodsGetSubTotalandGetTotalcouldbeimplantedaspropertieswithjustagetter,butforsomereason,ourcodecoveragewillnotcoverthemlateron.Iamguessingthisissome.NETCoreincompatibility(Iknowthisworksinfull.NETandthisissueactuallycostmehourstofind!):
publicclassOrderLine
{
publicProductProduct{get;set;}
publicintNumber{get;set;}
publicdecimalGetSubTotal()
{
returnProduct.Price*Number;
}
}
AndthesamegoesfortheOrderclass:
publicclassOrder
{
publicList<OrderLine>Lines{get;set;}
publicdecimalGetTotal()
{
returnLines.Sum(l=>l.GetSubTotal());
}
}
Andnowwewanttotestthem.WecanusethesametestsaswehadinJavaScript,butwritteninC#.
Unfortunately,yourC#testscannotbeinthesameprojectasyourcode(well,theyprobablycan,butjustbecauseyoucould,doesnotmeanyoushould).The
reasonforthisisthatwhenyoutrytostartaprojectthathasbothexecutablecodeandunittests,VSCodewillfindtwoprogramentrypointsandwillnotknowwhichonetostart(yourprogramoryourtests).So,atthispoint,wehavetocreateanewproject.Inthesamefolderasyourweb-shopfolder,createanewfolderandnameitweb-shop-tests.OpenthefolderinVSCode.Wecannowinitializeanewprojectusingtheconsole.Fortheweb-shopproject,weusedthemvctemplatebutforthisproject,weneedthexunittemplate:
dotnetnewxunit
dotnetrestore
YounowgetacsprojfileandUnitTest1.cs.Wearegoingtousethecsfile.First,renameittoOrderTests.cs.Also,renametheclassinthefiletoOrderTests..NETCoregeneratedoneemptytest,whichyoucanrunnow.Itdoesnotdomuch,buttryrunningdotnettestintheterminal.Ifallgoeswell,youshouldseethetestpassing.Next,weneedtomakeareferencefromourweb-shopprojecttoourtestsproject:
dotnetaddreference../web-store/web-store.csproj
dotnetrestore
Alternatively,youcanaddittoyourcsprojfilemanually:
<ItemGroup>
<ProjectReferenceInclude="..\web-shop\web-shop.csproj"/>
</ItemGroup>
Wecannowusecodefromtheweb_shopprojectinthetestsproject.Let'schangetheOrderTestsclassandaddatest:
usingSystem;
usingSystem.Collections.Generic;
usingweb_shop.Database;
usingweb_shop.Invoicing;
usingXunit;
namespaceweb_shop_tests
{
publicclassOrderTests
{
privateOrderCreateOrder()
{
returnnewOrder
{
[...]
};
}
[Fact]
publicvoidCalculateTotal_Test()
{
varorder=CreateOrder();
Assert.Equal(6.23M,order.GetTotal());
}
}
Totestit,runthedotnettestintheterminal:
Tocheckwhetheritactuallyworks,tryandmakethetestfail.Simplychange6.23Mtosomethingelseandrunthetestagain.Thistime,thetestshouldfail:
Changetheexpectedresultbackagain.WecannowaddtheotherthreetestsweusedtohaveinJavaScript:
[Fact]
publicvoidUpdateLineTotalOnNumberChange_Test()
{
varorder=CreateOrder();
order.Lines[0].Number=2;
Assert.Equal(2.46M,order.Lines[0].GetSubTotal());
}
[Fact]
publicvoidUpdateTotalOnLineInsert_Test()
{
varorder=CreateOrder();
order.Lines.Add(newOrderLine
{
Product=newProduct
{
Price=2
},
Number=1
});
Assert.Equal(8.23M,order.GetTotal());
}
[Fact]
publicvoidUpdateTotalOnLineRemoval_Test()
{
varorder=CreateOrder();
order.Lines.RemoveAt(0);
Assert.Equal(5,order.GetTotal());
}
Runyourtestsagainandvalidatethattheysucceed.Now,gobacktoyourweb-shopproject.Youprobablywantaneasywaytotestyourcodefromhereaswell.Openupthetasks.jsonfileandaddanewtask:
{
"taskName":"test",
"command":"dotnet",
"isShellCommand":true,
"args":["test","../web-shop-tests/web-shop-tests.csproj"]
},
YoucanaddittoyourdoItAlltaskaswell,ifyouwant:
"args":[
"/C\"dotnetbuild&&dotnetbundle&&gulp&&dotnettest../web-shop-tests/web-shop-tests.csproj\""
]
Now,ifatestfails,youwillnotbenotifiedwhenyoutrytodebugtheapplicationandyoucanfixthetestfirst.Personally,Ithinkitisabitoverkilltorunallyourtestsateverybuild,especiallywhenyouhavehundredsandittakesafewsecondstorunthem,butit'syourchoice.
ReportingOfcourse,wewillneedsometestreportsthatwecanpublishtoJenkinslateron.Unfortunately,anditpainsmetosaythis,Microsoftisdoingareallybadjobatthis.Youcanforgetaboutanout-of-the-boxcodecoverageoptionunlessyouarerunningVisualStudioEnterpriseEdition(whichcostsyouanarmandalegandonlyrunsonWindows).EvenasimpleJUnitstylereportistoomuchtoaskforatthispoint;MicrosoftonlyoutputsTRXfiles(TestResultX...?),whichisjustanotherXMLfile.
First,let'sjustoutputtheTRXfile.Itisquiteeasyactually--justappend--loggertrx,or-ltrxforshort,toyourdotnettestcommand:
dotnettest-ltrx
ThefilewillbegeneratedinaTestResultsfolder.Wecanalsospecifyacustomfilename:
dotnettest-l"trx;LogFileName=result.trx"
So,nowwehaveaTRXfilethatwecanpublishtoJenkinslater.
Backtothecodecoverage.Thisisrathertrickyasthereisnobuilt-insupportforthis.OurbestbetistouseOpenCover(https://github.com/OpenCover/opencover),butthis(otherwisegreattool)doesnotofficiallysupport.NETCoreyet.Theproblemisthatitdependsonsomelibrarythatisnotyetfullyportedto.NETCore,sowearegoingtorunintosomeissues.Thegoodnewsisthatthereisquitealotofdemandforofficial.NETCoresupportforOpenCoverandpeopleareworkingonit.Meanwhile,peoplehavefiguredouthowtogetthecurrentversionworkingwith.NETCore.Ofcourse,without.NETCoresupport,itonlyworksonWindows.
ThefirstissueiswithactuallygettingOpenCovertowork.Normally,youwouldinstallOpenCoverasaNuGetpackage,whichwouldinstallOpenCoverinyoursolutionpackagesfolder.However,thisisnotthecasewith.NETCoreandVSCode.So,getthelatestmsifilefromGitHub,https://github.com/opencover/opencover/releases,andinstallit.Youprobablywanttogointotheadvancedoptionsand
installOpenCoverforalltheusersonthismachine(soJenkinswillbeabletorunittoo).
NowthatOpenCoverisinstalled,wecan,theoretically,simplyrunitfromtheconsoleandtargetourtestproject:
"C:\ProgramFiles(x86)\OpenCover\OpenCover.Console.exe"-target:"C:\ProgramFiles\dotnet\dotnet.exe"-targetargs:"test"-register:user-filter:"+[web-shop]*-[web-shop-tests]*"-output:"TestResults/OpenCoverCoverage.xml"
Thatisquiteacommand-lineprogram.WearerunningOpenCover.Console.exe(inthefolderwhereyouhaveinstalledit,ProgramFiles(x86)\OpenCoveristhedefault)andspecifydotnet.exe(inC:\ProgramFiles\dotnet)asthetargettorun.Theargumentgiventodotnet.exeissimplytest;thisisequivalenttorunningdotnettestfromtheterminalinVSCode.The-filterswitchtellsOpenCoverwhichfilestoincludeorexclude.Thedefaultincludefilteris+[*]*,oreverything.Therearealsosomedefaultexcludefilters,suchas[System]*,[System.]*,and[mscorlib]*.
Wewanttoincludeweb-shop*,butexcludeweb-shop-tests*.The-outputswitchjusttellsOpenCoverinwhatfiletoputtheresult.WeareputtingtheresultwiththerestofourtestresultsgeneratedbyxUnit.Lastbutnotleast,wejustwanttoaddthe-register:userswitchwhichisnecessarytogetitallworking.ItregisterstheprofilerOpenCoverusesfortrackingyourcoveredcode.Withoutit,youwillgetanerrormessage:
Noresults,thiscouldbeforanumberofreasons.Themostcommonreasonsare:
1)missingPDBsfortheassembliesthatmatchthefilterpleasereviewthe
outputfileandrefertotheUsageguide(Usage.rtf)aboutfilters.
2)theprofilermaynotberegisteredcorrectly,pleaserefertotheUsage
guideandthe-registerswitch.
So,runthecommandandyoustillgetthaterrormessage!Wecanopenthegeneratedfileandlookforthecause:
<ModuleskippedDueTo="MissingPdb"hash="B5-C4-7A-E0-72-26-30-32-27-17-25-28-B2-D0-1A-95-DD-94-E2-57">
<ModulePath>C:\Users\sander.rossel\Desktop\web-shop-tests\bin\Debug\netcoreapp2.0\web-shop.dll
<ModuleTime>2017-06-03T12:34:39.9752551Z</ModuleTime>
<ModuleName>web-shop</ModuleName>
<Classes/>
</Module>
TheMissingPdbreasonisabitmisleading.Whenyoulookinyourbinfolder,youwillfindaProgramDatabase(PDB)file,butitisnotinthecorrectformat.APDBfileisusedformatchingruntimecodetothelinesofcode,allowingyoutodebugyourcode.However,.NETCoreapparentlyusesadifferentformatthan
full.NET.Wecaneasilyfixthisthough.Headovertoyourweb-shopprojectfileandaddthefollowingPropertyGrouptoit:
<PropertyGroupCondition="'$(Configuration)|$(Platform)'=='Debug|AnyCPU'">
<DebugType>full</DebugType>
<DebugSymbols>True</DebugSymbols>
</PropertyGroup>
Thistells.NETtouseafullPDBfile.Thedefaultfor.NETCoreisportable.NowrunOpenCoveragainandfindthatitworks,exceptthatyougetzerocoverageoneverything:
Committing...
VisitedClasses0of17(0)
VisitedMethods0of72(0)
VisitedPoints0of193(0)
VisitedBranches0of84(0)
====AlternativeResults(includesallmethodsincludingthosewithoutcorrespondingsource)====
AlternativeVisitedClasses0of28(0)
AlternativeVisitedMethods0of133(0)
Thereisonemoretricktogettingthisright.OpenCoverdoesnotsupport.NETCorebecausetheProfilerAPIisnotyetconvertedto.NETCore.Weneedtoaddthe-oldStyleswitch,whichisprettyhackyandnotreallyintendedforthispurpose,butitdoesthetrick:
"C:\ProgramFiles(x86)\OpenCover\OpenCover.Console.exe"-target:"C:\ProgramFiles\dotnet\dotnet.exe"-targetargs:"test"-register:user-filter:"+[web-shop]*-[web-shop-tests]*"-output:"TestResults/OpenCoverCoverage.xml"
Youshouldnowgetanoutputthatlookslikethefollowing:
KeepinmindthatOpenCoveronlycovers.NETcode.AnyJavaScriptcode(orothernon.NETcode)isignored.So,whenyouhaveathousandlinesofJavaScriptandonelineofC#andyoutestthatoneline,youwillhavea100%coverage(accordingtoOpenCover).Keepinmindthatpropertiesarenotcovered(probablybecauseofsome.NETCoreincompatibility).Youmayalsowanttoworkwiththe-skipautopropsswitch,whichdoesnottakeautoproperties(propertieswithemptygettersandsetters,suchaspublicintSomeProperty{get;set;})intoaccountforcodecoverage.
Unfortunately,westillhaveaproblem.WehavethisOpenCovercoveragereportandthereisnothingwecandowithit.Jenkinscannotpublishit,wecannotreadit,andVSCodeisatalossaswell.Weneedtwoadditionalpackagestogetusefulreports.Thefirst,aratherlonganddescriptivename,istheOpenCoverToCoberturaConverter(https://github.com/danielpalme/OpenCoverToCoberturaConverter)andthesecondistheReportGenerator(https://github.com/danielpalme/ReportGenerator).Thefirstpackagedoeswhatitsnameimplies--itconvertsthereportcreatedbyOpenCovertoaCobertura-styledXML.TheReportGeneratorcangeneratevariousreports,suchasHTMLandtext,fromvariousinputsamongwhichareOpenCoverandCobertura.AswithOpenCover,wecannotreallyinstallthesetoolsusingVSCodeandrunthem.
Instead,weneedNuGettodownloadthem.Gotohttps://www.nuget.org/andinstallthelatestnuget.exe(notthevsix).Now,youcansimplyrunitfromthecommandandinstallOpenCoverToCoberturaConverter:
nuget.exeinstallOpenCoverToCoberturaConverter
Thiswillputthepackageinyourcurrentfolder.Putitsomewherewhereyoucaneasilyaccessit.Puttingitinyourweb-shop-testfolderisactuallyaprettygoodidea,soyouwillalwayshavetheconverterwithyourpackage.YoureallyonlyneedtheTools\OpenCoverToCoberturaConverter.exefilesoyoucanjustkeepthatanddiscardtheotherfiles.YoucandownloadReportGeneratorthesamewayoryoucangetitfromGitHubathttps://github.com/danielpalme/ReportGenerator/releases.IfyoudownloadedfromGitHub,makesureyouunpackthezipfile.Youdonotneedallthefiles,butwearegoingtocreateaseparatefolderforthistoolanyway,soyoumightaswellkeepthemall.
Usageofbothtoolsisprettystraightforwardnowandgivesustheresultsweneed.Thefollowingcommandsareexecutedfromtheweb-shop-testsfolderandassumethatOpenCoverToCoberturaConverter.exeisinthesamefolderandthatReportGeneratorisinafoldernamedReportGenerator:
OpenCoverToCoberturaConverter.exe-input:"TestResults\OpenCoverCoverage.xml"-output:TestResults\Cobertura.xml
ReportGenerator\ReportGenerator.exe-reports:"TestResults\OpenCoverCoverage.xml"-targetDir:TestResults\CoverageHTML
OpenCoverToCoberturaConverterwilloutputawarning,butwecanignoreitaswearenotgoingtomergefiles.WenowgetaniceCoberturafileandadetailedHTMLpageshowinguswhatlinesareandarenotcovered.Theoverviewpageshowsuscoverageperassembly,class,ornamespace(youcanslidethegroupingslideratthetop)andtotalcoveragestatistics:
Clickingonaclassshowsthecoverageperline:
ThisisprettymuchthesamereportwehadearlieranditisareportwecanactuallyuseinJenkinsandthatwecanreadourselves.Sothatsetsupourtestreportingfornow.
AddingSeleniumtestsNext,wearegoingtoaddSeleniumteststoourapplication.Youcould,ofcourse,usetheJavaScripttestswewroteinthepreviouschapters.Afterall,SeleniumtestsyourfrontendandthathasnothingtodowithourC#backend.OurroutingchangedabitsotheywillnotworkuntilwefixtheURLsinthetests,butapartfromtherouting,everythingshouldbeasexpected.However,SeleniumhasaC#implementation,solet'skeepthisprojectasmuchC#aspossibleandexploretheSeleniumC#API.
Tomakethiswork,weneedyetanotherproject.Strictlyspeaking,wedonotneedanewprojectandwecouldsimplyputourSeleniumtestsinourregulartestproject.However,yourtestprojectdependsonyourweb-shopprojectandneedstoaccessitinordertobuild.YourSeleniumtestsaregoingtoneedyourweb-shopprojecttorun.Andhereistheproblem:yourunittestprojectcannotbuildwhenyourweb-shopprojectisrunningbecausethefileswillbelocked.Inanycase,itisbesttocreateanewprojectanyway,asyouprobablydonotwantyourunittestsandSeleniumteststoalwaysrunatthesametime.Socreateanewfolder,nameitweb-shop-selenium(orsomethinglikethat)andinitializeitwithdotnetnewxunit.
Ishouldwarnyou,the.NETCoreSeleniumimplementationisstillinbetaphaseatthetimeofwriting.Notallthefeaturesaresupported,butwewillmanage.Asluckhasit,withthelatestreleaseof.NETCore,whichwasreleasedwhilewritingthisbook,wecansimplyaddtheSeleniumpackagestoourcsproj.Youwillgetanerrorthatthepackageswererestoredusing.NETFrameworkv4.6.1andthatthepackagemaynotbefullycompatiblewithyourproject.Asalways,youcanusedotnetaddpackageoraddthereferencetotheprojectfilemanually.YouwilleventuallygetanerrorthatSystem.Security.Permissionismissing,sobesuretoinstallthatonetoo.YoumayneedtorestartVSCode:
<PackageReferenceInclude="Selenium.Support"Version="3.5.2"/>
<PackageReferenceInclude="Selenium.WebDriver"Version="3.5.2"/>
<PackageReferenceInclude="System.Security.Permissions"Version="4.4.0"/>
dotnetaddpackageSelenium.Support
dotnetaddpackageSelenium.WebDriver
dotnetaddpackageSystem.Security.Permissions
dotnetrestore
RenamethetestfiletoSeleniumTests.cs.Init,wearegoingtoplaceourtests.Thisoneisgoingtobeabitdifferent.Weneedawebdriver,forexample,theChromedriver,toworkwith.Wecanusethesamedriveracrossalltestssowedonotneedtoopenupanewbrowserwindowforeverytest.So,thedriveriscreatedasafieldthatwillneedtobeproperlydisposed.Sincetheverybeginning,.NEThastheIDisposableinterface,whichmustbeimplementedinaspecificmannerforittoworkproperly.Luckily,VSCodehelpsyouwiththeexactcode(includingcomments)toimplementthis.Hereisthebarebonesversionofourtestclass,includingdispose,butexcludingthecomments:
usingXunit;
usingOpenQA.Selenium;
usingOpenQA.Selenium.Chrome;
usingSystem;
namespaceweb_shop_selenium
{
publicclassSeleniumTests:IDisposable
{
privateIWebDriverdriver=newChromeDriver();
#regionIDisposableSupport
privatebooldisposedValue=false;
protectedvirtualvoidDispose(booldisposing)
{
if(!disposedValue)
{
if(disposing)
{
driver.Dispose();
}
disposedValue=true;
}
}
voidIDisposable.Dispose()
{
Dispose(true);
}
#endregion
}
}
ChromeDriverishardcodedfornow,butitcanbeexchangedwithotherdriverssuchasFirefoxDriverandEdgeDriver.ThisworksbecausewehavethedriverforChrome,andotherwebdrivers,inourPATHvariable,asexplainedinChapter5,TestingyourJavaScript.Alternatively,youcanspecifythepathtoyourwebdriverintheChromeDriverconstructor:
privateIWebDriverdriver=newChromeDriver("C:\\WebDrivers");
Thisisthepartwhereweneedourweb-shopproject.Inyourcsprojfile,addaprojectreference:
<ItemGroup>
<ProjectReferenceInclude="..\web-shop\web-shop.csproj"/>
</ItemGroup>
Becausetheweb-shopneedsanappsettings.jsonfile,itneedstobecopiedtoanyoutputfolderthatisgoingtousethisproject.Now,inyourweb-shop.csproj(nottheSeleniumproject!),addthefollowingcodetoalwayscopytheappsettings.jsonfile:
<ItemGroup>
<ContentUpdate="appsettings.json">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
</ItemGroup>
WecannowimplementtheconstructorofourSeleniumTestsclass.ThisiswherewecreateareferencetoourserverandconstructaclientusingtheStartUpclassofourweb-shopproject:
privatereadonlyTestServerserver;
privatereadonlyHttpClientclient;
publicSeleniumTests()
{
server=newTestServer(newWebHostBuilder()
.UseStartup<Startup>());
client=server.CreateClient();
client.BaseAddress=newUri("http://ciserver:5000");
}
Wecannowwriteourfirsttest.Let'sstartwiththetestthatcheckswhethertheproducttitlesonthehomepageareproperlywrapped:
[Fact]
publicvoidShouldCutOffLongTitles_Test()
{
driver.Navigate().GoToUrl("http://localhost:5000");
driver.Manage().Timeouts().ImplicitWait=TimeSpan.FromSeconds(2);
varelems=driver.FindElements(By.CssSelector("h3"));
Assert.Equal(3,elems.Count);
varelem=elems[0];
Assert.Equal("hidden",elem.GetCssValue("overflow"));
Assert.Equal("ellipsis",elem.GetCssValue("text-overflow"));
Assert.Equal("nowrap",elem.GetCssValue("white-space"));
}
Theonlyweirdthinggoingoninthissampleisthedriver.Manage().Timeouts().ImplicitWaitthing.Thissetsthewaitperiodtofindelements.Weareexplicitlysettingitto2seconds,becausewewantthetopthreeproductstoloadwithintwoseconds(andactuallythatisprettylongalready).Ifittakeslonger,wewillgetatimeoutexception.Intheregularversionof.NET,wecouldusetheSelenium.Supportpackage,whichaddsaWebDriverWaitclassthatletsuswaitforcertainconditions,suchastheExpectedConditionsutilityinJavaScript.Unfortunately,wearestuckwiththeImplicitWaitproperty,whichsetsthewaittimeforalltheoperationsthedriverexecutes.IdoexpectSelenium.Supporttocometo.NETCoreaswellthough.
Thenexttwotestsarestillprettystraightforward:
[Fact]
publicvoidShouldNavigateToTheClickedProduct_Test()
{
driver.Navigate().GoToUrl("http://localhost:5000");
driver.Manage().Timeouts().ImplicitWait=TimeSpan.FromSeconds(2);
varproduct=driver.FindElements(By.CssSelector("h3"))[1];
product.Click();
vartitle=driver.FindElement(By.CssSelector("h2"));
Assert.Equal("CaptainAmerica:CivilWar",title.Text);
}
[Fact]
publicvoidShouldSearchForAllProductsContainingFanta_Test()
{
driver.Navigate().GoToUrl("http://localhost:5000");
driver.Manage().Timeouts().ImplicitWait=TimeSpan.FromSeconds(2);
varinput=driver.FindElement(By.CssSelector("[ng-model=query]"));
input.SendKeys("fanta");
varsearchBtn=driver.FindElement(By.CssSelector("#search-btn"));
searchBtn.Click();
varresults=driver.FindElements(By.CssSelector(".thumbnail"));
Assert.True(results.Count<20);
varallHaveFanta=results.All(elem=>
{
varcaption=elem.FindElement(By.CssSelector("h3")).Text;
returncaption.ToLowerInvariant().Contains("fanta");
});
Assert.True(allHaveFanta,"Notallfoundresultscontainthetext'fanta'.");
}
Asyoucansee,westillusetheSendKeysandClickmethodsandthefoundelements(ofthetypeIWebElement)canstillbesearchedfornestedelements.Onemajordifference,however,isthatthetextofanelementissimplyaproperty.Nodifficultasynchronouscallsandcallbacks.
ThelasttestislessdifficultthantheJavaScriptequivalent,butitisstillquitetricky:
[Fact]
publicvoidShouldRemoveAnItemFromTheShoppingCart_Test()
{
driver.Navigate().GoToUrl("http://localhost:5000/ShoppingCart");
driver.Manage().Timeouts().ImplicitWait=TimeSpan.FromSeconds(2);
//redirect
varusername=driver.FindElement(By.CssSelector("input[ng-model=username]"));
varpassword=driver.FindElement(By.CssSelector("input[type=password]"));
username.SendKeys("user");
password.SendKeys("password");
varsubmit=driver.FindElement(By.CssSelector("input[type=submit]"));
submit.Click();
//anotherredirect
driver.Navigate().GoToUrl("http://localhost:5000");
varbuttons=driver.FindElements(By.CssSelector(".btn.btn-primary"));
buttons[0].Click();
driver.SwitchTo().Alert().Accept();
buttons[1].Click();
driver.SwitchTo().Alert().Accept();
driver.Navigate().GoToUrl("http://localhost:5000/ShoppingCart");
varitems=driver.FindElements(By.CssSelector(".thumbnail"));
varcount=items.Count;
varcaption=items[0].FindElement(By.CssSelector("h3")).Text;
items[0].FindElement(By.CssSelector("button")).Click();
//LikewiththeJavaScripttests,justwaitforasecond...
Thread.Sleep(1000);
varnewItems=driver.FindElements(By.CssSelector(".thumbnail"));
Assert.Equal(count-1,newItems.Count);
Assert.NotEqual(caption,newItems[0].FindElement(By.CssSelector("h3")).Text);
driver.Navigate().Refresh();
varafterRefresh=driver.FindElements(By.CssSelector(".thumbnail"));
Assert.Equal(count-1,afterRefresh.Count);
}
Here,weseethattheSeleniumAPIisquitedifferentfromtheJavaScriptone.Again,nocallbacknesting.Also,weseetheSwitchTo().Alert()functionstocatchanalertwindow,andAccept()toacceptthealert.UsingNavigate(),wecanrefreshthebrowserwiththeRefresh()method.Otherthanthat,thismethoddoesthesameastheJavaScriptone.Wegototheshoppingcartpage,getredirectedtotheloginpage,login,getredirectedtotheshoppingcartpage,gotothehomepage,addtwoproductstothecart,gototheshoppingcartpage,removeanitem,and,finally,refreshthepageandcheckwhethertheitemisstillremoved.Quiteatest,butitisworthit.
JenkinsWenowhaveourcode,ourunittestswithcodecoverage,ourE2Etests,andourdatabasetest.TimetogetittoworkinJenkins.First,wemustcommiteverythingtoGit.WeneedthethreefoldersinthesameGitrepositorythough(strictly,wedonot,butitmakeseverythingsomucheasier).ItisagoodideatocreateanewGitrepositoryusingGitLab.Ihavenameditweb-shop-csharp.Clonetheweb-shop-csharprepositorytoyourmachineandputweb-shop,web-shop-tests,andweb-shop-seleniumintherepository.Youwillnowhaveover4,000filestocommit.Create(orcopy)a.gitignorefilesoweexcludesomegeneratedfiles.ThoseincludeBowerandnpmfiles,generatedJavaScript,CSSfiles,andtestresults.Youwillbeleftwithlessthan100files(69ifyoufollowedmyexactdirections):
**/bin/**
**/obj/**
**/TestResults/**
**/node_modules/**
**/bundles/**
**/wwwroot/lib/**
**/site.min.css
Thisisalsoagoodmomenttotestifoursoftwarebuildsatallifweremovethesefiles(youcancommitandpushthesechangesandthendoanothercleanrepositorycheckout).Spoiler--itdoesnot.Whenyoutrytorun,.NETCorewillcomplainthatitismissingsomefilesintheobjfolder.Tofixthis,weneedtododotnetrestore.Afterthat,Gulpisnotabletorun,soweneednpminstall.Lastbutnotleast,everythinggoesasplanned,exceptthewebsiteisnotworkingproperly.WearemissingjQuery,Bootstrap,andAngular,sowemustdobowerinstallaswell.Ofcourse,wewantallthattoautomaticallyhappenwhenwebuildthesoftware.AllwehavetodoischangethedoItAlltaskinthetasks.jsonfile:
"args":[
"/C\"dotnetrestore&&dotnetbuild&&dotnetbundle&&bowerinstall&&npminstall
],
SofixthebuildandpushthattoGitaswell.
BuildingtheprojectTimetostartupJenkinsagain.Jenkinsdoesnotsupport.NEToutofthebox.Ifyouarereallyseriousabout.NETandCI,youmaywanttolookatTeamFoundationServer(TFS),theCIplatformfromMicrosoft(https://www.visualstudio.com/tfs/).TFSisnotfree(unlessyouareusingthelimitedExpressversion)andworksonWindowsonly.SowearegoingtouseJenkinsagain.
So,createanewproject,nameitsomethingsuchasCSharpWebShop-Build,andpicktheFreestyleProjecttemplate.Theconfigurationshouldbeobviousbynow,butIwillwalkyouthroughitjustincase.IhavetickedtheDiscardoldbuildscheckboxandsetittokeep5builds.WearegoingtobuildthisonLinux,justbecausewewanttomakeuseofthemultiplatformcapabilitiesof.NETCore.So,itgoeswithoutsayingthatweshouldRestrictwherethisprojectcanberuntoLinux.SetthecorrectGitURLandcredentialsintheSourceCodeManagement.Youmaywanttosetabuildtrigger,butatthispoint,wejustwanttogetittorunmanuallyfirst.
Buildingourprojectjustforthesakeofbuildingitisquiteimportant.UnlikeJavaScript,C#isacompiledlanguage.Therearemanyreasonsforaprojectnottocompile.Aprogrammercouldhavecommittedinvalidsyntaxbymistakeorfilescanbemissing.Anyway,ifaprogramdoesnotcompile,evenifitisonlybecauseofamissingsemicolon,itcannotbeexecutedasawhole.Sobuild.
Forthebuildstep,wearegoingtoexecuteashellcommand:
cdweb-shop
dotnetrestore
dotnetbuild
dotnetbundle
npminstall
bowerinstall
gulp
ThatisprettymuchwhatourdoItAlltaskdoesaswell.Whenyourunthis,itwillfailrightaway.OurLinuxserverdoesnothave.NETCoreinstalledyet.Solet'sinstallitonourVM.Youcanfindhowtoinstall.NETCoreontheMicrosoft.NETCorewebsiteaswell(https://www.microsoft.com/net/core#linuxubuntu):
sudosh-c'echo"debhttps://apt-mo.trafficmanager.net/repos/dotnet-release/xenialmain">/etc/apt/sources.list.d/dotnetdev.list'
sudoapt-keyadv--keyserverhkp://keyserver.ubuntu.com:80--recv-keys417A0893
sudoapt-getupdate
sudoapt-getinstalldotnet-sdk-2.0.0
[agreewithinstallation]y
WhenyouruntheJenkinsjobagain,.NETCoreisgoingtosetupsometelemetrystuff(onetimeonly)andthenbuildyourproject.Andthen,thenextstepwillfailbecauseBowerisnotfound.Sofar,wehaveusedBowerasaglobalpackage,whichisfine,butwealsoknowitisdifficulttoinstallglobalpackagesastheyareinstalledperuser.So,theeasiestsolutionistoinstallBowerlocally,soitcanalwaysberunanywherewithouthavingtoinstallitforthatspecificuser.WearedoingthesamewithGulp:
npminstallbower--save-dev
ThatalsomeanswehavetoslightlychangeourJenkinsconfiguration:
npminstall
node_modules/.bin/bowerinstall
node_modules/.bin/gulp
YourJenkinsbuildshouldsucceednow,onLinux!ThenextthingwewanttodoistoaddSonarQubetothebuild.Createasonar-project.propertiesfileintherootofyourGitrepository.Wecannowsetthesourcespropertytoweb-shop,sincewedonotreallywanttoincludeourtestfilesintheSonarQubeanalysis.exclusionsare,unfortunately,notrelativetothesourcesproperty:
sonar.projectKey=CSharpWebShop
sonar.projectName=C#WebShop
sonar.projectVersion=1.0
sonar.sources=web-shop
sonar.exclusions=**/node_modules/**,**/wwwroot/lib/**,**/bundles/**,**/obj/**,**/bin/**,**/.vscode/**
Nowthefunpart:SonarQubewillnotanalyzeC#fileswhentheanalyzerdoesnotrunonWindows.So,weneedtomakeourprojectrunonWindowsafterall(changethelabelfromlinuxtowindows).Wewillneedtoreplaceourshellscriptwithacmdcommand.Thecmdcommandisthesameastheshellcommand,exceptweneedtoreplacethenewlineswith&&:
cdweb-shop&&dotnetrestore&&dotnetbuild&&dotnetbundle&&npminstall&&node_modules\.bin\bower.cmdinstall&&node_modules\.bin\gulp.cmd
Also,makesuretheSonarQuberunnerusestheWindowsSlaveinsteadofLocal.WhenyouruntheprojectandcheckoutSonarQube,youwillseeacoupleof
issues.Again,theusageofalertinJavaScriptcauseseightvulnerabilities.WhatwereallywanttoknowishowwellourC#codedoes.Itdoesreallyverywell(afterall,IhavebeencodingC#foracoupleofyears).Actually,theonlyC#issuewehaveisinsomecodethat.NETgeneratedforus,theProgram.csfile.Thefixiseasyenough--addthestatickeywordtotheclassdeclaration:
namespaceweb_shop
{
publicstaticclassProgram
{
publicstaticvoidMain(string[]args)
{
[...]
WhenyoupushthistoGitandruntheJenkinsprojectagain,youwillfindthatourprojectisprettyclean.Wealreadyknewaboutthealertissuesandtheexperimentaltext-overflow.WecanresolvethemasWon'tfixtogetridofthem.
TestingtheprojectNext,wearegoingtotestourproject.Wehavevarioustestsinplace,butwewillstartwiththeunittests.WewillneedtoinstallanewJenkinspluginbeforewecanpublishourTRXresultstoJenkins.So,gototheJenkinspluginmanagerandsearchformstest.Youshouldseetwoplugins,theMSTestpluginandtheMSTestRunnerplugin.TheMSTestRunnerisforrunningtestsusingMSTest.WeareusingxUnitthoughandwearerunningitthroughOpenCoverusingthecommandlineanyway.So,weneedtheMSTestplugin,whichcantransformTRXfilestotheJUnitformatandpublishthemtoJenkins.
Wehavetwochoicesnow.Wecancreateanewproject,likewedidwiththeNode.jswebsite,orwecanjustaddanewbuildsteptothecurrentproject.Let'sgowiththeformer,sowecancreateapipelineofprojects.Copythecurrentproject(thenameoftheprojecttocopyfromiscase-sensitive)andnamethenewoneCSharpWebShop-Test(byputtingCSharpWebShopinfrontofthetask,yourprojectsarenicelysortedtogether).Theonlythingyoureallyneedtodoischangethebatchcommandbuildstep.Also,setthebuildtriggertoBuildafterotherprojectsarebuiltandwatchthepreviousproject(onlywhenstable).Also,removetheSonarQubeanalysis.YoucouldpossiblykeeptheSonarQuberunnerinthisprojectandremoveitfromthebuildproject,soyoucanbuildthatoneonLinuxagain.
Totestourproject,weneedtodoafewthings:runthetestsusingOpenCoverandtransformthereports.Afterthat,wecanpublishthereports.Thereisalittleproblemwithdotnet,asitdoesnothaverightstocreatetheTestResultsdirectory,soweneedtodothatmanuallyaswell.Becauseofthesesamerightsissues,OpenCoveralsocannotregistertheProfilerdllwiththeuseraccount,sowecanomituser:
cdweb-shop-tests&&dotnetrestore&&(ifnotexistTestResultsmkdirTestResults)&&"C:\ProgramFiles(x86)\OpenCover\OpenCover.Console.exe"-target:"C:\ProgramFiles\dotnet\dotnet.exe"-targetargs:"test-l"trx;LogFileName=result.trx""-register-filter:"+[web-shop]*-[web-shop-tests]*"-output:"TestResults\OpenCoverCoverage.xml"-oldStyle&&OpenCoverToCoberturaConverter.exe-input:"TestResults\OpenCoverCoverage.xml"-output:TestResults\Cobertura.xml&&ReportGenerator\ReportGenerator.exe-reports:"TestResults\OpenCoverCoverage.xml"-targetDir:TestResults\CoverageHTML
Thatisaprettybigcommand,soyoumaywanttosplititupintomultiplecommands:
[CreatetheTestResultsfolder]
cdweb-shop-tests&&(ifnotexistTestResultsmkdirTestResults)
[RunOpenCover]
cdweb-shop-tests&&dotnetrestore&&"C:\ProgramFiles(x86)\OpenCover\OpenCover.Console.exe"-target:"C:\ProgramFiles\dotnet\dotnet.exe"-targetargs:"test-l"trx;LogFileName=result.trx""-register-filter:"+[web-shop]*-[web-shop-tests]*"-output:"TestResults\OpenCoverCoverage.xml"-oldStyle
[ConverttoCobertura]
cdweb-shop-tests&&OpenCoverToCoberturaConverter.exe-input:"TestResults\OpenCoverCoverage.xml"-output:TestResults\Cobertura.xml
[ConverttoHTML]
cdweb-shop-tests&&ReportGenerator\ReportGenerator.exe-reports:"TestResults\OpenCoverCoverage.xml"-targetDir:TestResults\CoverageHTML
Nothingwehavenotseenbefore,butitisalotinonecommand.Donotforgettopublishthegeneratedreports.TheCoberturareportcanbefoundinweb-shop-tests/TestResults/Cobertura.xml.TheHTMLreportisinweb-shop-tests/TestResults/CoverageHTMLandtheindexpageisindex.html.IntheCoberturaCoverageReport,itwillseemlikeyourclasseswithonlypropertieshavenocodeandyourotherclasseshavelesscode.Forexample,itseemslikeOrder.csonlyhasthreelinesofcode.Anyway,youknowwhyyouaregettingtheseresults:
YoumaybesurprisedtofindthatwhiletheCoberturareportdoesnotproperlycoverproperties,theHTMLreportdoes.
TheTRXreportisnew.WiththeMSTestplugin,yougotanewpost-buildactionnamedPublishMSTesttestresultreport:
Itisnotverydifficult,butyouhavetoknowitisthere.AnothersideeffectoftheMSTestpluginisthattheEmmapluginwasalsoinstalled.EmmaisanothercodecoverageformatthatisusedbyMicrosoft.Eventhoughwearenotusingit,there
isanewreportonthemenunamedCoverageTrend.Clickingitjustshowsabrokenimage.Wearenotgoingtouseiteither,soitjustsitstherebeinguseless.Youcanignoreit.
TestingthedatabaseSo,wenowhavethebuild,SonarQube,unittests,andcodecoverageinplaceanditistimetotestthedatabase.WearegoingtorunthisoneonLinuxagain.So,copythebuildprojectandnamethisoneCSharpWebShop-Database.Makesureyourlabelexpressionislinux,soitrunsontheVM.
Forthisone,wearegoingtousepg_Prove,acommand-lineutilityweinstalledearlierwiththecpanTAPcommandinLinux(http://pgtap.org/pg_prove.html).Withpg_Prove,wecanomitthetoppartofourSQLtestscriptbecausepg_provedoesthatforus:
--Thiscanberemoved.
\setQUIET1
[...]
--Fromhereiswhatweneed.
BEGIN;
Firstofall,pg_ProvewilltrytologintoPostgreSQLusingaJenkinsuser.Forsimplicity,wearegoingtocreatethatuserwithoutapasswordandmakeitasuperusersonothingwillkeepJenkinsfromdoingitsjob:
CREATEUSERjenkinsWITH
LOGIN
SUPERUSER
CREATEDB
CREATEROLE
INHERIT
NOREPLICATION
CONNECTIONLIMIT-1;
ThenextthingweneedtodoisinstallanadditionalplugininJenkins.GotothepluginmanagerandinstalltheTAPplugin.Thiswillgiveyouanewpost-buildaction.
Next,wecansettheshellcommandfortheproject:
pg_prove-dwebshop-vweb-shop/test/sql/*.sql|teetap.txt
The-vswitchoutputsalltheteststotheconsole.Thefilesthatwillbetestedareweb_shop/test/sql/*.sql.The|teeoutputsanyoutputtotheconsoleandtoafilecalledtap.txt.Onlythestderroutputisnotwrittentothefile.Wecannow
publishourtap.txtfiletoJenkins.AddthePublishTAPResultspost-buildactionandpublishtap.txt.Also,gointotheadvancedoptionsandmakesureFailthebuildifnotestresults(files)arefoundandFailedtestsmarkbuildasafailurearechecked.Notcheckingthemwillmakethebuildunstable,butwewantittofailexplicitly.Therearenowtwonewreports.TheTAPExtendedTestResultsandthepre-buildTAPTestResults:
Theper-buildTAPTestResultsshowsyoutheresultsperbuild,likeitsays:
IntheTAPTestResults,thereisalsoalinktoaHistoryreport,butitseemstobebroken.Tocheckwhetheritworks,trybreakingourdatabasefunctionandtriggerthebuild.Youwillseeitfailsandyoucanseetheresultsinthereportsasexpected.
TheonlythingleftforustodonowistorunourSeleniumtests.Sinceweneedoursoftwaretorunforthatanditisabithardertorunitinawaythatisnon-blockingforourconsole,wearegoingtohavealookatthatinlaterchapters.
SummaryInthischapter,wehaverepeatedtheentireprocessofbuildingandtestinganapplicationandthenautomatingit.WehaveseensomedifferencesbetweenJavaScriptandC#.C#needscompilationanddoesnotalwayshavethesamelevelofsupportthatJavaScripthas.CompletelynewwasthedatabasetestingwithTAP.
Inthenextchapter,wearegoingtohaveamorein-depthlookatJenkinssowecanfurtherenhanceourbuilds.
AdditionalJenkinsPluginsWehavecomeprettyfar.WehavebuiltaJavaScriptapplicationwithNode.jsandautomateditwithGulp.WethendidthesameinC#anddidalltheworkduringthebuild.Afterthat,weautomatedeverythinginJenkins,sonotasinglecommitisnottestedandbuild.However,asfarasJenkinsisconcerned,wedidthebareminimumtogetoursoftwaretobetestedandbuilt.
Unfortunately,inalimitedtestingenvironment,suchasours,wedonotrunintotheproblemsyouaregoingtofaceintherealworld.Forexample,wecurrentlyhavetwoprojects,JavaScriptandC#,whichtogethermakeupforsixprojects.Inmydailyjob,wehave,maybe,twohundredprojects.Personally,IprobablyneedaboutfiftyofthosebecausethosearefromtheprojectsIamworkingon.Jenkinshasallkindsofoptionsandpluginstomakesenseofitall.Inthischapter,wearegoingtodivedeeperintoJenkinsandexploresomeofthepluginsandoptionsyouhavetofurtherenhanceyourbuildsandoverallJenkinsexperience.
ViewsWhenyougetalotofprojectsformultipleprojectsformultiplecustomers,youmayloseoverview,whichmakesitdifficulttoproperlymaintainyourprojects.Jenkinshasviewstomanageyourprojects.Aviewisbasicallyacollectionofprojectsthatyouthinkshouldbegroupedtogether.Thesecanbealltheprojectsyouhaveaccessto,alltheprojectsforacertaincustomer,oralltheprojectsforacertainapplication.Inourcase,wehavetwoprojects,theJavaScriptwebshopandtheC#webshop(Ihavesomeadditionaltestprojects,butyoumayignorethem).
Atthetopofyourprojectslist,youfindtheglobalviews.Currently,thereisonlyoneview,All:
Clickontheplustabtocreateanewview.PicktheListViewtypeofviewandnameitJSWebShoporsomethingsimilar.Onthenextpage,youcanpicktheprojectstolistinyourview:
Here,youcanmanuallyselectprojectsorselectthembasedonaregularexpression.Youcanalsopickcolumnstoshowinyourview.HittheOKbuttonandyournewviewwillbecreated.YounowhavealistwiththeBuildWebShopandTestWebShopprojects.Forthenextview,CSharpWebShop,Iamgoingtousearegularexpression.Usingregularexpressionsisprettydifficult,butwhenyournamingconventionisright,itshouldnotbeaproblem.Asyoucansee,myCSharpWebShopprojectsareallnamedCSharpWebShop-(somebuildstep).Thatmakestheregularexpressionprettysimple,(CSharpWebShop)(.*).Theviewthatiscreatedlookslikeyouwouldexpect:
Theprotousingaregularexpressionisthatnewprojectsareautomaticallyaddedtotheview.Forexample,ifyoucreateanewprojectandnameitCSharpWebShop-Deployment,itisautomaticallyaddedtotheCSharpWebShopview.GoodnamingconventionswouldbeCustomer-Project-Buildstep(configuration),forexample,ACME-Website-Build(debug).Butitisalluptoyouofcourse.Worstcasescenario,youhavetohandpickallyourprojectsforaview.Itispossibletousearegularexpressionandhandpicksomeprojectsthatdonotfittheregularexpression.
AnotherviewtypeistheMyViewtype.Thisviewshowsalltheprojectsthatthecurrentlyloggedinuserhasaccessto.Alluserscurrentlyhaveaccesstoalltheprojectsbydefault,sothisviewtypeisnotveryusefultousnow.Youcanrestrictaccesstoprojectsonaperuserbasis,butyoumaylockyourselfoutwhileenablingit.Wearegoingtolookatsecuritylaterthough,sojusttakemywordforitnow.
Nexttoglobalviews,youcanhaveyourownpersonalviews.Fromthemainpage,youcanpicktheMyViewsmenuitemfromtheleft-handsidemenu.Anyviewscreatedherewillnotbevisibletootherusers.Whencreatingapersonalview,yougetanextraviewtype,Includeaglobalview.Thisdoesnothingmorethanlinktoaglobalview(unfortunately,viewingthisviewalsotakesyoutotheglobalviews).Itmaybehandywhenyouwanttofilteroutsomeglobalviewsandaddthemtoyourpersonalviews.Forexample,wehavecustomer-specificglobalviewsatwork,butIamnotinvolvedwithallofthosecustomers.So,Iaddedtheglobalcustomerviewsthatarerelevanttometomypersonalviews.Usingviewsthiswaycanreallysaveyoualotoftime,effort,andfrustrationwhenlookingforaproject.
CleanupworkspacesWhenwefirstworkedwithJenkins,Irecommendedyoutoalwayscleanyourworkspacebeforebuildingyoursoftware.Thatway,youcanalwaysbesureallthenecessarypackagescanbedownloadedfromnpmorNuGetandyoudonotaccidentallyusecachedfilesforyourbuild.However,doingallofthatcomesatacost.Cleaning,checkingyourentireGitrepository,anddownloadingyournpmorNuGetpackagesmaketakeafewminuteseverytimeyoustartabuild.Analternativeapproachistoonlycleanapartofyourworkspace.Toenablethisfeature,gotothepluginmanagerandinstalltheWorkspaceCleanupplugin(https://wiki.jenkins-ci.org/display/JENKINS/Workspace+Cleanup+Plugin).Thispluginaddsanadditionalcheckinthebuildenvironmentaswellasanadditionalpostbuildaction.
Wecanseethisplugininactionreallyeasy.CreateanewprojectandnameitCleanupTestorsomethingsimilar.MakeitrunonLinuxandaddashellbuildstep.Intheshell,wearegoingtocreatetwofiles,src/.srcandbin/.exec;twofilesthatdoabsolutelynothingexceptsitthereforustosee:
mkdir-pbin
touchbin/.exec
mkdir-psrc
touchsrc/.src
Runtheprojectandverifythatthefileswerecreatedasexpected.Wearenowgoingtocleanthebinfolderafterthebuild.AddthenewDeleteworkspacewhenbuildisdonepostbuildaction.ClickontheAdvanced...button.Everythingisprettyself-explanatoryandtherearehelpquestionmarks,butlet'sgothroughitrealquick.Wewanttokeepoursrcfolder,butcleanoutthebinfolder:
Wewanttoincludeallthefilesinthebinfolderinourdeletion.Alternatively,wecouldexcludeeverythinginsidethesrcfolderfromdeletion.WecanalsochecktheApplypatternalsoondirectoriescheckboxandsimplyusethebinpatterntosimplydeletetheentirebinfolder.Wealwayswanttocleantheworkspace,nomattertheresult.Butifacleanupfails,wedonotfailthebuild.TheExternalDeletionCommandusessomeexternalprogramtocleanyourworkspace.It'sbestyoureadthehelptextincaseyouwanttoeverusethis.
Now,runthebuildagainandcheckwhetheryourbinfolderisindeedempty(ordeleted,basedonyoursetup).
Youcanalsocleanupyourworkspacebeforethebuildstarts.Thisoptionisalittledifferentbecauseyoudonotknowthestatusofyourbuildyet.Whethertocleanornotisbasedonaparameterpassedtothebuild.Atthispoint,wecannotpassparameterstobuildyet,butwewillcheckthatoutlaterinthischapter:
ConditionalbuildstepsAnotherusefulJenkinspluginistheConditionalBuildStepplugin(https://wiki.jenkins-ci.org/display/JENKINS/Conditional+BuildStep+Plugin).Thispluginletsyouskiporincludebuildstepsbasedonsomecondition.Forexample,youcanbuildandunittestyoursourceoneverycommit,butonlyrunSeleniumtestseverythreehoursusingaperiodicbuildtrigger(soyourbuildisbeingtestedeventhoughtherearenocommits).
InstalltheConditionalBuildStepplugininthepluginmanager.Afterinstallation,yougettwonewbuildsteps:Conditionalstep(single)andConditionalsteps(multiple):
Therearequiteafewconditionstochoosefrom.TheBuildCausedetermines
howthebuildwastriggered(manual,SCM,ortimer...),butyoucanalsospecifyadayoftheweek,atimeoftheday,combineconditionswithlogicalANDandORclauses,andmuchmore.Themultiplestepsworkthesame,exceptyoucanaddmultiplebuildstepsthatwillallbeexecutedwhentheconditionismet:
Thebuildstepsarejustlikebefore,soyoucanstartusingthispluginrightaway.Theconditionsmayneedalittleexplanation,butoverall,theyareprettyobvious.Luckily,thequestionmarkexplanationisprettygood,sobesuretohavealookatit.
PipelinesWithmultipleprojectsthatallbuildafteroneanother,youcanquicklylosesightofdependenciesbetweenprojects.Wehaveabuildproject,unittests,E2Etests,databasetests,deployment,andmanymore.Whoknowswhatelse.Well,wehavegotapluginforthat.InstalltheBuildPipelineplugin(https://wiki.jenkins-ci.org/display/JENKINS/Build+Pipeline+Plugin)fromthepluginmanager.Afterthat,createanewviewandyouwillfindanewviewtype,BuildPipelineView.Pickthistype,giveitaname,forexample,CSharpWebShopPipeline,andpicktheinitialjob.Theinitialjobisthefirstprojectthatisstartedandwhichwilltriggerotherbuilds.
Forexample,Ihavesetupmybuildssothatdeployment(whichisjustanemptyproject)istriggeredaftertheE2Etests(whichwewillfixshortly),whichistriggeredafterdatabasetesting,whichistriggeredafterunittesting,whichistriggeredafterthebuild.Quiteapipeline.Bysettingupthebuildasmyinitialjobinthepipelineview,wegetaviewthatisnotquitewhatyouareusedtofromJenkins:
Asyoucansee,theviewshowswhatprojectsran,arerunning,andstillneedtorun.Itisalsopossibletorunmultipleprojectsinparallel,forexample,theunittests,thedatabasetests,andtheE2Etests.Theproblem,ofcourse,liesinwhenwewilldeploythewebshop.Wecannotdeployonallthree,butallthreemustpassbeforedeploymentcanbedone.So,forgetaboutthedeploymentjobfornowandlet'striggerallthetestprojectsafterthebuildproject:
Thebestpartisthatyouonlyneedtochangeyourprojectconfigurations(eithertheBuildotherprojectspostbuildactionortheBuildafterotherprojectsarebuiltbuildtrigger)andtheviewisupdatedautomatically.
So,howaboutweputthatdeploymentstepbackinthere?Tosupportthis,weneedanotherplugin,theJoinplugin(https://plugins.jenkins.io/join).Soinstallitandgototheprojectconfigurationofthebuildjob.TheJoinpluginrequiresyoutospecifyanydownstreamprojectsintheprojectthatstartsthem.So,inthiscase,weneedtoaddapostbuildactioninthebuildprojectthatstartsthetestjobs(unlessyoualreadyhaditconfiguredthatway).Makesureyourtestjobsarenottriggeredbythebuildjob.Now,youneedtospecifyanotherpostbuildactioninthebuildjob,JoinTrigger.IntheProjectstobuildonce,afteralldownstreamprojectshavefinishedfield,putthedeploymentproject.Yourbuildpipelineviewwilllooklikethedeploymentprojectistriggeredthreetimes,butinpractice,itisonlytriggeredoncebytheJoinplugin:
Inthenextchapter,wearegoingtolookatworkflows,whichgiveyouanevengreatercontroloverwhatrunswhenbyhavingyourcompleteJenkinsprojectincode(usingtheGroovyscriptlanguage).
PromotedbuildsAnotherverycoolpluginisthePromotedBuildsplugin(https://wiki.jenkins.io/display/JENKINS/Promoted+Builds+Plugin).Thispluginsolvessomeoftheproblemswehadearlier.ItalsomakestheJoinpluginobsolete.Theideabehindpromotedbuildsisthatabuildcanbepromotedbasedonvariouscriteria(includingpromotionofotherprojects).Uponpromotion,oneormoreadditionalbuildstepscanbeexecuted(includingtriggeredanotherproject).Wheneverabuildispromoted,itgetsapromotionsymbolonthebuildlist.So,installthePromotedBuildspluginfromthepluginmanager.
WejustusedtheJoinplugintowaitforourdatabasetests,browsertests,andE2Etestsbeforetriggeringthedeploymentproject.Instead,wecannowuseapromotion.GototheCSharpWebShop-BuildprojectconfigurationandfindthePromotebuildswhen...checkboxintheGeneralsection.Youmustnowenteranameforyourpromotion,forexample,TestsCompletedSuccessfully.Donotforgetthisoryouwillgetanerroronsavingyourconfigurationandyouwillhavetostartover.Youcanpickaniconorgowiththedefault.
Then,weneedtopickacriteriathatmustbemetforthebuildtobepromoted.PickWhenthefollowingdownstreamprojectsbuildsuccessfullyandthenentertheprojectsyouwanttowaitfor,CSharpWebShop-Database,CSharpWebShop-Test,CSharpWebShop-E2E.
Afterthat,wecanspecifyactionsthatwewanttoexecutewhenthecriteriaaremetandthebuildgetspromoted.PickBuildotherprojectsandchooseCSharpWebShop-Deploymentastheprojecttobuild.Unfortunately,thisdoesnotshowinthepipelineview,soyouwilllosethat.
RemovetheJoinpostbuildactionatthebottomofyourconfigurationandsaveit.Thatisreallyallthereistoit.Youcannowmanuallytriggerthebuildandseeitinaction.Wheneverythinggoeswell,yourbuildshouldgetpromotedandyoushouldseeapromotionstarinthebuildhistoryoftheproject:
Asyouhavealreadyseen,thereareothercriteriayoucanpick,suchaswaitingforupstreambuildstogetpromoted,havingtobemanuallyapproved,acustomGroovyscript,andmore.Andyoucanalsopickawholelotofactionstotriggerwhenthebuildgetspromoted.
ParameterizedbuildsItispossibletoparameterizeyourbuildsinJenkins.CreateanewjobandnameitParameterizedProjectorwhatever.Now,almostatthetopoftheconfigurationisanoptionThisprojectisparameterized.Checkitandaddaparameter.Thereareacoupleofparametersandsomepluginswilladdadditionalparameters.Mostofthemareprettystraightforwardandallofthemareexplainedinthehelptext.Let'sgowithastringparameterforsimplicity.Pickasinglewordname,suchasYourText,anoptionaldefaultvalue,andanoptionaldescription:
LetyourbuildrunonLinuxandaddashellscriptbuildstep.Thatiswherewecanuseanyparameters:
echo$YourText
Ifyougobacktotheprojectpage,youwillnoticethattheBuildbuttonchangedtoBuildwithParameters.Ifyoutrytorunthebuild,youwillbepromptedwiththeparametersthatyouhavetoselectfirst:
Andifyoubuild,youwill,ofcourse,seeyourtextinthebuildoutput.
Anotherplugin,whichyoushouldalreadyhaveinstalledthroughsomeotherplugin,makesitpossibletotriggerparameterizedprojectsfromotherprojects.CheckwhetheryouhavetheParameterizedTriggerplugin(https://wiki.jenkins.io/display/JENKINS/Parameterized+Trigger+Plugin)installedandifyouhavenotinstalledit.Createyetanotherprojectandtheonlythingyouneedtodoisaddapostbuildactionandtriggerparameterizedbuildonotherprojects.Specifytheparameterizedprojectwejustcreatedasaprojecttobuild.Youcanpicktousethecurrentparametersasinputtothenextproject,butwedonothaveanyparametersforthisproject.SoinsteadpickPredefinedparameters.YoucannowspecifyparametersasKey=Value,whereeachlineisakey-valuepair.So,forourproject,youcanspecifyYourText=I'mtriggeredfromanotherproject!.Now,buildthisprojectandyouwillseeittriggerstheparameterizedproject,whichwilloutputI'mtriggeredfromanotherproject!.
Itisalsopossibletouseenvironmentvariablesthatwecanalsouseinashellorcommandbuildstep.Forexample,YourText=I'mbuildfromproject$JOB_BASE_NAME!,whichwillprintI'mbuildfromproject[yourprojectname]!.
TriggeringbuildsremotelyInthepreviouschapters,wehavelookedatthevariousbuildtriggersthatareavailabletous.WeevenaddedtheGitLabtriggerwiththeGitLabplugin.Thereisonetriggerwehavenottriedyet,whichistheremotetrigger.So,createanewproject;IamsimplynamingitRemoteTest,andchecktheTriggerbuildsremotely(forexample,fromscripts)trigger.Youarenowpresentedwithanauthenticationtoken.Thisisyoursecrettokenthatallowsanyonewhohasittoremotelytriggerthebuild.So,youbestpicksomethingthatisnoteasytoguess(somethinglike47d6753c6f307c13edf010a2730f03a6).However,forourtest,wearegoingtopicksomethingthatisabitmorereadableandeasytotype,sojustputmy_tokeninthefield.
Normally,youcanbrowsetoaURLtotriggeraregularbuild,forexample,http://ciserver:8080/job/Remote%20Test/build(%20isaURLencodedspace).YouwillgetamessagethatyoushoulddoaPOSTrequestinsteadofaGETrequest,butyoucancontinueandthebuildwillstilltrigger.However,ifyoutrytodothesamefromanincognitowindow,youwillbetakentotheloginscreen.Instead,youneedanonymousaccess,whichisexactlywhattheauthorizationtokengivesyou(forthisonespecificprojectonly).So,openananonymouswindowandbrowsetohttp://ciserver:8080/job/Remote%20Test/build?token=my_tokeninstead.Ifyoukeeptheprojectpageandtheanonymouswindownexttoeachother,youcanseethebuildbeingtriggered.
Now,tryaddingaparametertoyourproject.IhaveaddedastringparameternamedMyString.Also,addabuildsteptoechotheparameter'svalue.Ifyoutriggertheprojectremotely,youwillbepromptedfortheparametervalues.However,onceyouhaveenteredavalue,youwillbetakentotheloginscreenagainandthebuildwillnotbetriggered.Instead,youneedtobrowsetobuildWithParameters,suchashttp://ciserver:8080/job/Remote%20Test/buildWithParameters?token=my_token.AnyparameterscomeattheendoftheURL,buildWithParameters?token=my_token&MyString=Value.Ifyouomitparameters,theywillsimplytaketheirdefaultvalue.
BlueoceanPerhapsthesinglemostawesomepluginforJenkinscomesfromtheJenkinsteamitselfandchangesyourJenkinsintosomethingcompletelydifferent.Jenkinsisnotthemostbeautifulandalsonotalwaysthemostintuitivetoolyouwillworkwith.Itisnonethelessagreattool,butitcouldbeevengreater.ThatiswhereBlueOceancomesin!BlueOceanisverynew.Itwentintoalphareleaselastyearandwentintoproductionwhilewritingthisbook.So,letthatbeawarningupfront;itprobablystillhassomeissues.ThegoodnewsisthatyoucanrunBlueOceanandtheclassicJenkinssidebyside.So,gotothepluginmanagerandinstallBlueOcean.Thiswillinstalllotsofstuff,suchastheGitHubplugin(whythough?),CommonAPIForBlueOcean,RESTAPIForBlueOcean,MetricsPlugin...Aboutthirtyofwhichmostare...ForBlueOcean.
Inthissection,wearegoingtogenerateascriptusingatool;youcanfindthefinalscriptintheGitHubrepositoryforthisbookunderChapter10/Code/Jenkinsfile(openwithanyothertexteditor).
Onceeverythingisinstalled,gobacktoyourmainpageandclickthenewOpenBlueOceanbuttonontheleft-handsidemenu.ThenewpagelooksnothinglikeJenkins:
Forsomethings,suchasadministrationand(classic)jobconfiguration,BlueOceanrevertstotheclassicJenkinslayout.Thisnewlookandfeelisallaboutpipelines,afeaturethatwearegoingtohaveanextensivelookatinthenextchapter.
Whatyouneedtoknowaboutpipelinesrightnowisthattheyletyoucodeyourbuildratherthanaddstepsthroughauserinterface.Theupsidetothisisthatyourbuildisjustcodethatischeckedintosourcecontrol.Thedownside,ofcourse,isthatitrequiresadditionalknowledgeonthesyntaxandfunctionsofyourpipelinefile.Wewilllookatthesyntaxofthesefilesinthenextchapter.
So,hereisthegreatnewthingaboutBlueOcean;itletsyoueditthesepipelinefilesdirectlyintoamoreintuitiveuserinterface,givingyouthebestofbothworlds.Thebadnewsisthatthisso-calledpipelineeditorisstillverymuchaworkinprogress.TheGitHubpageforthisproject(https://github.com/jenkinsci/blueocean-pipeline-editor-plugin)isquiteclearaboutit,Important!Thissoftwareisaworkinprogressandisnotcomplete(andherebedragons,thismaynotbeuptodate,andmanymore).However,becausethisfeatureissoawesome,Ireallywanttogiveyoualittlesneakpeakrighthere.WewillgetbacktopipelinesandtheBlueOceanplugininthenextchapteraswell.
So,thisnewestfeature,unfortunately,islargelyundocumentedand,sofar,itonlyworkswithGitHub.However,wecancreateadummypipelinethatstillgeneratesthecodeweneedforourpipeline.IfyouhaveaGitHubaccountwithsomerepository,youcantrythisoutwithGitHub.IfyoudonothaveaGitHubaccount,youcanskiptheremainderofthisparagraph.Youcanevenforktheci-bookrepositorytotryitoutonthechaptersinthisbook.Inthenewdashboard,youshouldseeaNewPipelinebuttonintheupper-rightcorner.ClickitandchooseGitHubasyourstore.YouthenneedtocreateatokenonGitHub;simplyfollowthelinkintheeditorandyouwillbetakentoGitHubwhereatokenisgeneratedforyou.BesuretocopythetokenandpasteitintothefieldinJenkins.Next,selectNewPipelinetocreateyourpipelinefile.Youcanthenselectarepositoryandcreateyourpipeline.JenkinswillsayTherearenoJenkinsfilesin[yourrepository]andyoucangoaheadandcreateonebyclickingthebutton:
IfyoudonothaveaGitHubrepository,youcanstillcreateapipeline.Youwillhavetocopyandpastethegeneratedcodefromtheeditorintoyourownfile.Togettothepipelineeditor,simplybrowsetothefollowingURLhttp://ciserver:8080/blue/organizations/jenkins/pipeline-editor/(ofcourse,youshouldreplaceciserver:8080withyourownJenkinsinstance).YoudonotneedaJenkinsprojectoranything;youwilljustbetakentotheeditor.
Nomatterwhatapproachyoufollowed,youwillbetakentoanewscreenwithastartnodeandaplusnodeandPipelineSettingsontheright-handside:
Apipelineconsistsofstages.Eachstagehasoneormorebuildsteps,liketheclassicJenkinsprojectconfigurationhasbuildsteps.Addinganodebyclickingonaplusnodeaddsanewstage.Eachstagehasitsownpipelinesettings.Thatmeanseachofyourbuildstepscanhavetheirownsettingsasopposedtosettingsthatapplytoallyourbuildsteps.Thataddsalotofflexibilitytoyourbuilds!Forexample,stageonecanrunonLinux,whilestagetwocanrunon
Windows,allinasingleproject!Withthenewpipelines,itisalsonowpossibletorunstagesinparallelandhaveanextstagewaituntilallparallelstageshavefinished.Basically,whattheJoinplugindoes,butmuchmoretransparent.JustclicksomeplusnodesandyouwillseewhatImean.
Unfortunately,thepipelinesaddedawholenewsetofproblemsaswell:backwardcompatibility.Alotofpluginsdonotworkwiththisnewfeature.ThereasonisthattheJenkinspipelineisaGroovyscript,sostepsaretriggeredfromascriptinsteadoffromtheJenkinsJavaenvironment(JenkinswasbuiltinJava).Notallpluginshaveacommand-lineinterfacethatcanbecalledfroma(Groovy)script.However,pluginsupportisincreasingandyoucanalwaysusethegoodoldLinuxshellorWindowscommandline.So,thismaynotbemuchofanissueinthefuture.
Now,let'screateourfirststage.AddthefirststageandgiveitanameinthePipelinesettings.Iamnotactuallyworkinginaprojectrightnow,soIamassumingwearegoingtobuildtheJavaScriptwebshop(withnowayoftestingourscript,fornow).Inourfirststage,wearesimplygoingtocheckoutourcodefromGitandinstallournpmpackages,solet'snamethefirststageCheckout.Next,wearegoingtoaddastep.Simplyclickthe+Addstepbuttoninthesettings.First,wewanttospecifyanodeonwhichtobuild,soaddtheAllocatenodestepandputLinuxinthelabel.Wecannowcreatechildsteps,whicharethestepsthatwillbeexecutedonthisnode.First,weneedtocheckoutourcodefromGit,soaddaGitstep.InthesettingsfortheGitstep,weneedtospecifyaURL,whichcanbefoundinyourGitLabrepository.Wealsoneedabranch,forwhichwecanpickmaster.Finally,youneedyourCredentialsId.YoucanfindtheIDofyourcredentialsontheclassicJenkinsleft-handsidemenu(inanewtab,becausewecannotsaveourcurrentpipeline).Inthemenu,gotoCredentialsandyoushouldseeyourGitLabcredentials(name/password)andyourGitLabAPItoken.GettheIDoftheGitLabusernameandpassword;itwillbeGitLabCredsifyoufollowedmyexample.GitLabCredsgointotheCredentialsIdfield.
Next,addaShellScriptstep.Intheshellscript,wewanttoinstallallnpmpackages.WewillwantaGulpbuildaswell,butwewilldothatatalaterstage:
npminstall
Thepipelinesettingswindowstillfeelsabitclunky.Itisimpossibletoswitch
stepsaroundandthedeletebuttonishiddenintheupper-rightmenuoneachstep.Hopefully,thiswindowwillgetabitmoreintuitiveinthenearfuture.Anyway,whenyouhavefoundallthesettings,youcanclickCtrl+Sandyoushouldseethegeneratedscript.Ifsomethingwentwrong,allyouseeisTherewerevalidationerrors,pleasechecktheeditortocorrectthem.Youcanclickthattextandanadditionalerrormightbeshown(itisreallytrialanderror).Anythingcouldbewrong.Yourstagesneedanameandatleastonebuildstep,butthismaynotbeimmediatelyclear.Ifeverythingwentwell,youshouldseethefollowingscriptbeinggenerated:
pipeline{
agentany
stages{
stage('Checkout'){
steps{
node(label:'linux'){
git(branch:'master',url:'http://ciserver/sander/web-shop.git',credentialsId:'GitLabCreds')
sh'npminstall'
}
}
}
}
}
ItlooksalittlebitlikeJavaScript,butitisGroovy(http://groovy-lang.org/).ThewindowyouareseeingwhenyoupressCtrl+Scanbeedited.Youcan,forexample,changeyourcurrentshellcommandoraddacompletenewshellstep(orwhateverstepifyouknowthesyntax).WhenyouclicktheUpdatebutton,thechangeswillbereflectedintheeditor.Whenyourscripthaserrors,youmaygetanerror(suchasillegalcolonafterargument...)ornothingmayhappen,inwhichcase,youcangoonalittlebughunt.Thelatterhappenswhen,forexample,youaddastepthatdoesnotexist;youcanchangeshtoshhh.Thewindowhasnocloseorcancelbutton,butEscdoesthetrick.
Next,let'saddateststage.WewanttomakeaGulpbuildandtestitinPhantomJSonLinux,whichshouldnotbeaproblem.Unfortunatelythough,itactuallyisabitofaproblem.AnewstagebasicallymeansanewprojectandsoJenkinscreatesanewworkspaceandstartsfromscratch.ThatmeanswehavetocheckoutourcodefromGitagainanddoannpminstalltogetallnecessarypackages.Thatisrathercumbersomeandkindofdefeatsthepurposeofthiswholeexercise.Luckily,thereisanalternative.Wecanallocateacustomworkspaceandusethatworkspaceagaininanextstage.Theworkspacewillstill
haveallofitsfiles,sowedonotneedtopullfromGitanddonpminstall.WefirstneedtochangeourBuildstagesothatitallocatesacustomworkspace.
Unfortunately,theAllocateworkspacestepmustbeachildoftheAllocatenodestepandallotherstepsmustbechildrenofAllocateworkspaceandtheeditorasitcurrentlyisdoesnotallowreorderingsteps.So,weneedtoremovetheGitandShellscriptstepstoinserttheAllocateworkspacestepandthenreaddthetwostepsaschildrenofAllocateworkspace.Abitofahassle.TheAllocateworkspacesteponlyneedsonevalue,whichisDir(ordirectory).Thisisthenameofyourworkspace.MakesureitdoesnotoverlapwiththeworkspacesofotherJenkinsprojects.Iamnamingmineweb-shop-pipeline:
Nowthatwehavethecustomworkspaceinplace,wecanuseitinournextstage,theBuildstage.Inthebuildstage,wearefirstgoingtoallocateourLinuxnode,thenweallocatetheweb-shop-pipelineworkspace,andlastweaddashellscripttobuildourprojectusingGulp:
node_modules/.bin/gulp--env=prod--browsers=PhantomJS
Ifallwentwell,yourpipelinescriptshouldnowlooksomethinglikethis:
pipeline{
agentany
stages{
stage('Checkout'){
steps{
node(label:'linux'){
ws(dir:'web-shop-pipeline'){
git(url:'http://ciserver/sander/web-shop.git',branch:'master',credentialsId:'GitLabCreds')
sh'npminstall'
}
}
}
}
stage('Build'){
steps{
node(label:'linux'){
ws(dir:'web-shop-pipeline'){
sh'node_modules/.bin/gulp--env=prod--browsers=PhantomJS'
}
}
}
}
}
}
SharingworkspaceslikethatwasalreadypossibleintheclassicJenkinsbytheway.IntheGeneralsectionoftheconfigurationpagearetheAdvancedoptions.OneoftheseadvancedoptionsisUsecustomworkspace.Whentwoprojectsonthesamenodehavethesamecustomworkspace,theywilleffectivelybesharingworkspaces.
ThenextstepforourpipelineistotestinIE,Chrome,andFirefoxonWindows.Asyoucanimagine,weprettymuchhavethesameproblemsasbefore.WeneedtocheckoutourGitrepositoryagain;weloseourcurrentGitcommitand,attheendofthebuild,wewillnotbeabletopushourstatusbacktoGitHub.Luckily,thisiswherethepipelineapproachhassometricksupitssleevethatwerenotpreviously(easily)available.Wecanstashfilesandsharethembetweenstages,nodes,andworkspaces(wecouldhaveusedthisinsteadofsharingworkspacestoo).
Ofcourse,wefirstneedtostashourfilesbeforewecanretrievethem.WecanstashthefilesafterwehavepulledthemfromGit,installedallnpmmodules,andranthebuild.So,addanadditionalstepintheBuildstage,Stashsomefilestobeusedlaterinthebuild.Youhavetonameyourstash;IhavenameditEverythingandyoumustspecifywhichfilestoincludeandexclude.Wearegoingtoexcludethenode_modulesfolder,becauseweneedtoinstallthemagainonWindows(fortheWindowscmdscripts),soputnode_modules/**intheexcludefield.Wewanteverythingelse,soput**/**intheincludefield(thatisallthefilesandfoldersfromallfolders).Youcould,optionally,excludesomereportfiles.Youcanalsoexcludethenode_modulesfolderandsimplydoannpminstallagainonWindows,butthatisyourchoice.
Next,wecancreateanewTestChromestage.Allocatethewindowsnodeand,asachildstep,addRestorefilespreviouslystashedandputintheEverythingstash.ThenextstepisaWindowsBatchScriptandthisisheavilybugged.Thisisthething,
yougetareallysmallfieldtoputinyourscript.Ourscriptcangetprettybig,butthatwillcausetheAllocatenodesidewindowtobecomeaswideasyourscript,meaningyourbackbuttonsandeverythingwillprobablybesofarleftyoucannotseethemonyourscreenanymore.
However,theTeststagesidewindowwillsimplywrapyourscriptandshowasyouexpect.So,whenyoucreatethescriptstepasachildstepofyourallocatenodestep,justputinasinglecharacter,thengobacktothestagesidewindow,openthescriptstepfromthere,andputintheentirescript.Youwillhavetodothiseverytimeyouwanttoopentheallocatenodestep.Whatisworse,weneedtoescapecertaincharactersinapipelinebatchstep.OneofthecharactersisthebackslashthatWindowsusesinitsfilesystem.So,insteadofC:\ProgramFiles,wegetC:\\ProgramFiles.However,whenyouopenandsavethescript(usingCtrl+S),theeditorwillun-escapethebackslashandturnC:\\P...intoC:\P...again,whichwillcauseanerrorwhenyoutrytoopenyourscriptagain.Hopefully,alltheseissueswillbesolvedbythetimeyougettousetheeditor,butIamgivingyouaheadsupjustincase.Anyway,thescriptswewanttoexecuteisthejustChrometests:
npminstall&&node_modules\\.bin\\gulp.cmdtest-min--browsers=Chrome
Ifyouhavedoneeverythingright,thepipelinescriptshouldnowlookasfollows:
pipeline{
agentany
stages{
[...]
}
stage('Build'){
steps{
node(label:'linux'){
ws(dir:'web-shop-pipeline'){
sh'node_modules/.bin/gulp--env=prod--browsers=PhantomJS'
stash(name:'Everything',includes:'**/**',excludes:'node_modules')
}
}
}
}
stage('TestChrome'){
steps{
node(label:'windows'){
unstash'Everything'
bat'npminstall&&node_modules\\.bin\\gulp.cmdtest-min--env=prod--browsers=Chrome'
}
}
}
}
}
Remember,whenyoupastethisintothescriptwindowandsaveit,youneedtomanuallychangetheWindowsbatchscriptsteptoescapethebackspaces.
HereisanothercoolfeatureofthepipelinewewanttotestinIEandFirefoxaswellandwecandosoinparallel.SimplyaddtwomorestagesandnamethemTestIEandTestFirefox.TheyarecompletelyidenticaltoTestChrome,exceptforthe--browsers=Chromepart,ofcourse.Atthispoint,yourpipelineshouldhaveaCheckoutstage,thenaBuildstage,andthenthreeteststages:
ThescriptnowhasaparallelnodeintheTeststage.Undertheparallelnodearethreesteps,TestChrome,TestIE,andTestFirefox:
[...]
stage('Test'){
steps{
parallel(
"TestChrome":{
node(label:'windows'){
unstash'Everything'
bat'npminstall&&node_modules\\.bin\\gulp.cmdtest-min--env=prod--browsers=Chrome'
}
},
"TestIE":{
node(label:'windows'){
unstash'Everything'
bat'npminstall&&node_modules\\.bin\\gulp.cmdtest-min--env=prod--browsers=IE'
}
},
"TestFirefox":{
node(label:'windows'){
unstash'Everything'
bat'npminstall&&node_modules\\.bin\\gulp.cmdtest-min--env=prod--browsers=Firefox'
}
}
)
}
[...]
Forsomereason,itseemsthattheeditorgivesthefirststepthesamenameasyourstage.OurstagewasinitiallynamedTestChrome,soIeditedthatinthescriptmanuallysothatthestageiscalledTestandtheindividualstepsincludethebrowsername.
Testingwithoutreportingisoflittleuse,sowewillneedtopublishourreportslikewedidbefore.TheJUnitreportisprettystraightforward;justaddaPublishJUnittestresultreportsteptoeachofthetestingstages.TheTestResults*fieldshouldbetest/junit/*.xml.Jenkinswillnowpublishfivereports,theChrome,IE,andFirefoxresults,obviously,andthePhantomJSresultsfromWindows(executedoneachGulpbuild)andthePhantomJSresultsfromLinux,whichwascopiedfromthestash.Thegeneratedcodeisprettysimple:
junit'test/junit/*.xml'
TheHTMLreportislessstraightforward.ThereisaPublishHTMLreportsbuildstep,butitonlyhasaverysmallinputfield.Itisnotclearwhatshouldgointhefield.Onceyouknowwhatshouldbeinthere,youwonderwhetherthisisjustabug.Also,itseemsthattheeditorcannotgenerateyourcodeforyounomatterwhatyouputinthere.So,anyway,youcanaddittothescriptmanually(andsavingitwillmessuptheeditorforboththebackslashesandtheHTMLstep).ThescriptfortheHTMLeditorcontainsprettymuchwhatisalsointheclassicconfiguration:
publishHTMLtarget:[
allowMissing:false,
alwaysLinkToLastBuild:false,
keepAll:false,
reportDir:'test/coverage/html',
reportFiles:'index.html',
reportName:'CoverageReport'
]
Wedonothavetopublishthecoverageofallthreebrowsers,sojustpickoneandplaceitthere.TheHTMLreportispublishedtoyourprojectpageandnotyourspecificbuildpage.
Idohavesomebadnewsforyounow.Wehavetwomorereports,thecoberturareportand,intheC#project,theTRXreport.Wecannotpublishthemusingthepipeline,editor,orotherwise.ThereisaworkaroundfortheTRXreport,butsofar,Ihavenotfoundaworkaroundforthecoberturareport.WecouldusethelcovreportanduseSonarQubeforourcoveragereportinginstead.FortheTRX
file,youcanrunanXMLTransformation(XSLT)toconvertTRXformattoJUnitformatandpublishthatinstead.Anyway,wearenotdoingeitherofthosenow.
WiththeHTMLpublishernotworkingandtheeditormessingupourbatchscripts,thisisbecomingadrag.However,thereisonemorefeatureIwouldliketodiscuss,theGitLabcommittriggerandupdatingthecommitstatus.ThereisaUpdatethecommitstatusinGitLabdependingonthebuildstatusbuildstep.Allitneedsisaname.Youcanaddthistoeverystageandputthestagenameasthenamefortheupdatestep.Yourbuildstepsgoinsideitaschildsteps:
[...]
stage('Build'){
steps{
node(label:'linux'){
ws(dir:'web-shop-pipeline'){
gitlabCommitStatus(name:'Build'){
sh'node_modules/.bin/gulp--env=prod--browsers=PhantomJS'
stash(name:'Everything',excludes:'node_modules/**',includes:'**/**')
}
}
}
}
}
ThiswillreportthestatusofeachstagetoGitLabandGitLabwillmarkthecommitassucceededorfailed.YoucanfindthestatusofeachstageinGitLabbygoingtothecommitandselectingtheBuildstab:
Thereisjustonemorethingthough.Forthistowork,youneedtosetsomethinginyourscriptmanually.Theeditorwillnotremovethesevalues,butfornow,theycannotbeeditedusingtheeditoreither:
options{
gitLabConnection('LocalGitLab')
}
triggers{
gitlab(triggerOnPush:true,branchFilterType:'All')
}
ThenamethatgoesinthegitLabConnectionoptionisthenameofaGitLabconnectionintheglobalJenkinssystemconfiguration.Thegitlabtriggertriggersthebuildonacommit,likewehadbeforeintheclassicconfiguration.Ofcourse,wewillneedtocreateawebhookinGitHuboncewearegoingtorunthisscriptinthenextchapter.YoucannowalsoremovetheGitbuildstepfromtheCheckoutstage.
BlueOceanisobviouslyaworkinprogress.Noteverythingworksasitshouldandsomethingsdonotworkatall.However,pipelinesarebecomingmorepopularandsoisconfigurationincode(andnotjustinJenkins).Inthenextchapter,wewilllookatpipelinesinmoredetail.Fornow,Ithink,wehaveseenenoughofBlueOceanandthepipelineeditor.
SecuritySecurityisoftenanafterthought,butitisquiteimportant.WhenyouarerunningJenkinsinyourlocalnetwork,youareprobablygoodtogo.However,whenyouneedeitheryourJenkinsservertobeaccessibleoutsideofyournetwork,youwillneedtodosomeadditionalworktomakesurenouninvitedguestscomesnoopingatyourdata.Also,especiallywhenlotsofpeopleareusingthesameJenkinsinstance,youmaywanttolimitwhoseeswhatbymanagingusersandgroups.
JenkinsonHTTPSYourfirstconcernisgettingJenkinstorunonHTTPSinsteadofHTTP.AlotofthisinformationisalreadyintheJenkinsdocumentation(https://wiki.jenkins.io/display/JENKINS/Starting+and+Accessing+Jenkins),butasalwaysIamgoingtowalkyouthroughitbecausethedocumentationisabithardtofollow.GettingthingstorunonHTTPSisabitdifficultbecauseweneedacertificateforourVMandlocalWindowsinstallation.Anyway,thefocushereisnotonobtainingacertificate,butingettingJenkinstouseit.
HTTPSonLinuxStartingwithHTTPSinLinux,wecancreateaself-signedcertificateusingtheopenssltool.Onceyougeneratethecertificate,youneedtoansweracoupleofquestionsaboutyouand/oryourorganization.Answerthequestions(orpickthedefaultorenteradot,.,toleaveitblank):
sudoopensslreq-x509-nodes-days365-newkeyrsa:2048-keyoutciserver.key-outciserver.crt
CountryName(2lettercode)[AU]:NL
StateorProvinceName(fullname)[Some-State]:.
LocalityName(eg,city)[]:
OrganizationName(eg,company)[InternetWidgitsPtyLtd]:Private
OrganizationalUnitName(eg,section)[]:
CommonName(e.g.serverFQDNorYOURname)[]:SanderRossel
EmailAddress[]:
So,wearerequestingan-x509structureinsteadofacertificaterequest;-nodespreventstheoutputkeyfrombeingdecrypted.Itisvalidforayear(-days365)withanew2048bitsRSAkeyandweareoutputtingthekeytociserver.keyandx509tociserver.crt.WenowhaveourprivatekeyandSSLcertificate.Theyarenotsignedbyanauthority,butthatisnotimportanthere.
ThenextthingweneedtodoisconvertourcertificatetoanintermediateformatnamedPublic-KeyCryptographyStandards#12(PKCS12).Again,wecanusetheopenssltool:
sudoopensslpkcs12-inkeyciserver.key-inciserver.crt-export-outkeys.pkcs12
EnterExportPassword:your_password
Verifying-EnterExportPassword:your_password
Ingotheciserver.keyandciserver.crtfileswejustcreated;outcomesakeys.pkcs12file.Makesureyourememberthepasswordyouentered.
ThenextstepiscreatingafileJenkinscanuse.WeneedaJavaKeystore(JKS),whichcanbegeneratedfromthePKCSusingtheJavaKeytool:
keytool-importkeystore-srckeystorekeys.pkcs12-srcstoretypepkcs12-destkeystore/var/lib/jenkins/jenkins.jks
Enterdestinationkeystorepassword:your_password
Entersourcekeystorepassword:your_password
Whenpromptedforthepassword,usethesamepasswordasbefore.NoticethatwearesavingtheoutputJKSfileasjenkins.jksinthe/var/lib/jenkinsfolder.
ThelastthingweneedtodoischangeourJenkinsconfigurationsothatitusesHTTPSinsteadofHTTP.Forthis,wemustchangethestartupJenkinsconfiguration,whichcanbefoundat/etc/default/jenkins.So,youcanedititusingvioryourownfavoritetexteditor.WeneedtochangeJENKINS_ARGSatthebottomofthefile.--webrootshouldbethereandyoucanaddalltherest:
sudovi/etc/default/jenkins
i
[...]
JENKINS_ARGS="--webroot=/var/cache/$NAME/war--httpPort=-1--httpsPort=$HTTP_PORT--httpsKeyStore=../../var/lib/jenkins/jenkins.jks--httpsKeyStorePassword=your_password"
Esc
:wq
Setting--httpPortto-1disablesHTTPentirely.For--httpsPort,weusethedefaultport,8080.--httpsKeyStorehasthepathtoyourjenkins.jksfile,sowegotwofoldersupandthenintovar/lib/jenkinswherethefileislocated.Last,-httpsKeyStorePasswordshouldhavethepasswordyouenteredwhengeneratingtheJKSfile.
Afterthat,youcanrestartJenkinsandcheckwhethereverythingwentalrightbycheckingtheservicestatus:
sudoservicejenkinsrestart
sudoservicejenkinsstatus
Afterthat,youcanopenyourbrowserandgotohttp://ciserver:8080andhttps://ciserver:8080andverifythathttpdoesnotworkandhttpsdoes.Youwillgetanerrorwhenbrowsingtohttps,butyoucanignoreit.Chrome,forexample,givestheerrorcodeNET::ERR_CERT_AUTHORITY_INVALID,whichisright,becauseourcertificateisnotbackedbyacertificateauthority.Firefoxgivesusthesameerror,Errorcode:SEC_ERROR_UNKNOWN_ISSUER,andIEgivesustheerrortoo(and,forsomereason,IstillcannotconnecttociserveronEdge).
Unfortunately,becauseweonlyswitchedtoHTTPSnow,ourGitLabwebhooksjustbecameinvalidbecausetheyconnecttohttp://...AnyotherlinkyoumayhavetoJenkins(maybe,someremotebuildtriggers)mustbechangedtotargethttps.However,sinceourcertificateisnotbackedbyacertificateauthority,itisaloteasier(andpractical)tojustturnitoffagain.
HTTPSonWindowsOnWindows,inourcase,thingsareabitmoreeasy.TheJenkinsdocumentationisprettyspotonforWindows(https://wiki.jenkins.io/display/JENKINS/Starting+and+Accessing+Jenkins).OpenaCommandPromptasanadministratorandrunthefollowingcommands:
cdC:\ProgramFiles(x86)\Jenkins\jre\bin
keytool-genkeypair-keysize2048-keyalgRSA-aliasjenkins-keystorekeystore
EnterKeystorePassword:your_password
Re-enterNewPassword:your_password
[Answerquestions]
IsCN=...correct?
[no]:yes
Enterkeypasswordfor<jenkins>
(RETURNifsameaskeystorepassword):[Enter]
ThiswillprettymuchdoallofthestepsnecessaryinLinuxatonce.Although,admittedly,wemayskipafewstepsbecausewearenowworkinglocalhost.Again,rememberyourpassword.
Youcanverifythatthekeystorewassuccessfullycreatedbyrunningthefollowingcommand:
keytool-list-keystorekeystore
Enterkeystorepassword:your_password
YounowneedtomoveyourkeystorefiletotheJenkinssecretsdirectory.ThekeystorefilewascreatedatC:\ProgramFiles(x86)\Jenkins\jre\binandneedstobecopiedtoC:\ProgramFiles(x86)\Jenkins\secrets.YoucandothismanuallyorusetheCommandPrompt,whateveryoulike.Wecannowchangethestartupconfiguration,whichisinC:\ProgramFiles(x86)\Jenkins\jenkins.xml.Youonlyneedtochangethelinestartingwith<arguments>:
<arguments>-Xrs-Xmx256m-Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle-jar"%BASE%\jenkins.war"--httpPort=-1--httpsPort=8080--httpsKeyStore="%BASE%\secrets\keystore"--httpsKeyStorePassword=your_password</arguments>
ThislooksprettymuchthesameastheLinuxarguments.Soagain,--httpPort=-1disablesloginthroughHTTP.Wewilluseourcurrentportfor--httpsPort.--httpsKeyStorepointstothekeystorefileinthesecretsfolder,andthe--httpsKeyStorePasswordisthepasswordyouchose.
YoucannowrestarttheJenkinsservicethroughtheWindowsServiceswindoworintheCommandPrompt:
netstopjenkins
netstartjenkins
Again,whentryingtoconnecttohttps://localhost:8080,youwillgetanerrorsayingthecertificateisinvalidbecausethecertificateauthorityisunknown.Weknow.Also,allyourURLspointingtoJenkins,suchasGitLabwebhooks,arenowinvalidandyouwillhavetofixtheURLsorregeneratethewebhooks.YoucanchangetheconfigurationbacktoHTTPifyouwant.
JenkinssecurityNowthatyournetworkcommunicationisencrypted,wecanworryaboutthenextlayerofsecurity.GotoJenkinsmanagementandthentoConfigureGlobalSecurity.Thehelp(questionmarks)onthispageisprettyhelpful,sobesuretoreaditwhenyouwanttoknowsomething.Beforedoinganythinghereisawarning,youcanshutyourselfout.AtthetopisanEnablesecuritysetting,whichison.Whenyoulockyourselfout,youcandisablethissettingintheJenkinsconfig.xml(invar/lib/jenkinsonLinuxorinC:\ProgramFiles(x86)\JenkinsonWindows)bychangingthevalueoftheuseSecurityelement.AfterrestartingJenkins,you(andeveryoneelse)canundowhateveryoudidandenablesecurityagaininJenkins.Alternatively,youcanmanuallychangetheauthorizationStrategyelementintheconfig.xml(ifyouknowwhatclassattributeitshouldhave).
So,anyway,wearecurrentlyinterestedinaccesscontrol.Thereareafewsecurityrealmsyoucanpick,forexample,LDAPorUnixuser/groupdatabase.Wearenotgoingtochangethesecurityrealmthoughbecausethatisalittledifficultwithoutaproperlyfunctioningnetworkwithusersandgroups.SotheJenkins'ownuserdatabaseisfine.Pluginsmayaddotherrealms.
So,whatwearegoingtolookathereistheauthorization.Thefirstoptionisthatanyonecandoanything.Thatisreallywhatitsays,onlyusefulinfullytrustedenvironments.LegacymodeisnotreallyinterestingtoussincewearenotrunningalegacyversionofJenkins.Thenextauthorizationmodeistheonewearecurrentlyusing;logged-inuserscandoanything.Peopleneedaccountsortheycannotperformanyactions.Optionally,anonymoususerscanstillreadJenkins,butnotperformanyactions.Thismodeisfinewhenyouareinatrustedenvironment,butyoudonotwanteverybodytohavefullaccesstoJenkins.Thismodealsokeepstrackofwhodoeswhat,soifsomeonemessesup,youknowwhodidit.Additionally,youcansetacheckboxthatdetermineswhetherpeoplecansignuptoyourJenkins.
Thenexttwooptionsaremoreinterestinganddangerous,becauseyoucanlockyourselfoutwiththese.Withmatrix-basedsecurity,youcanspecifyspecificprivilegesforusersandgroups.Withproject-basedmatrixauthorizationstrategy,
youcandothesame,butonaper-projectbasis,allowingforveryfine-grainedcontrol.Selectinganyofthesetwo,notdoinganythingandsaving,locksyouout.Therearenodefaultsettingsforadminusersoranything.AsIrecall,thereisevenabuginoneofthesemodesthatifyoudonotconfigureanypermissionsfor,atleast,oneuserJenkinsjustfailstoloadanypageandshowsanerrormessageinstead.
Let'stakealookatthematrix-basedsecurity.Enableitandimmediatelyaddyouradminusertothematrix.Makesureyouradminusergetsalltheprivileges(scrolltotherightfora(un)checkallbutton).Likewise,youcanaddotherusersandgivethemspecificpermissions.Certainpluginsmayaddnewpermissions:
Youcanalsoadd(LDAP/Unix)groupstothematrix,butsincewearenotusingthat,wecannotaddroles.Youcanaddnon-existingrolesandusers,butJenkinswillindicatethatthisuserorroledoesnotexist.Inorderforausertobeabletologin,theusershouldatleasthavetheOverallAdministerortheOverallReadpermission.Ifneitherofthesepermissionsareset,youwillgetanerrorwhentryingtologinsaying[user]ismissingtheOverall/Readpermission.
Theproject-basedmatrixauthorizationworksprettymuchinthesamewayasthematrix-basedsecurity,butwhenyouopenaprojectconfiguration,yougetanadditionalcheckbox,Enableproject-basedsecurity.Here,youcangiveuserspermissionsforthisspecificproject.Notenablingthiswillsimplytaketheuser'sglobalpermissionset,butenablingitallowsyoutogivestricterglobalpermissionsandthensetthesepermissionsonprojects.Forexample,givesomeuseronlytheOverallReadpermissiononagloballevelandthenpickanyprojectandcheckallpermissions(oracoupleofpermissions)intheprojectforthatuser.Now,ifyouloginasthatuser,theuserwillonlyseethatoneproject:
Optionally,youcanblocktheinheritanceoftheglobalauthorizationmatrix.Thatmeanstheglobalauthorizationisignored.Watchoutthough,becausethatmeansusersdonothaveanypermissionsforthisproject,evenwhentheydohavepermissionsintheglobalauthorization,soyoucanlockyourselfout.Ifyouenabletheinheritanceblockanddonotsetpermissionsforanyuser,yourprojectisgone(well,itisstillthere,butnoonemayvieworchangeit).Pluginscanaddadditionalpermissions,asyoucanalreadyseefromthePromotepermission.
Role-basedauthorizationstrategyHavingtoenableallthesepermissionsforindividualuserscanbeabittediousandgetsprettyannoyingafterawhile.Luckily,thereisaplugintosolvethis,theRole-basedAuthorizationStrategyplugin(https://wiki.jenkins.io/display/JENKINS/Role+Strategy+Plugin).Thisplugindoesquitealot.First,itaddsanewoptionforglobalsecurity,Role-basedStrategy.Onceyouhavechosenthistypeofauthorization,anewitemisaddedtotheJenkinsmanagementmenunamedManageandAssignRoles.Thisopensupanewmenuwiththreeitemsofwhichwewilldiscusstwo,ManageRolesandAssignRoles.
OntheManageRolesscreen,youcanaddrolesandassignpermissionsjustlikeinthematrix-basedsecurity.Therearethreekindsofroles:globalroles,projectroles,andslaveroles.Theglobaladminroleiscreatedbydefault.HereiswhatIwanttodo,IhaveaprojectteamthatisworkingontheC#WebShopandIwanteverynewmemberoftheteamtohaveonly(full)accesstothoseprojects.Usingthepreviousproject-basedmatrixsecurity,thiswouldhavebeenalittletime-consumingandIwouldhavetochangeallmyprojects.However,withtherole-basedsecurity,thistakesthreeminutestops.
GotoManageRolesandstartwithcreatinganewglobalroleandnamingituser.GivethisuseronlytheOverallReadpermission.Then,createaprojectroleandnameitCSharpWebShop(orsomething).Theawesome(andtricky)thinghereisthatthisjobrolehasaregularexpressionpatternthatisusedtomatchitemsforwhichtheroleisvalid.Thatalsomeansthatwhenaprojecttitlechanges,youmaynothavepermissiontoviewitanymore.So,inourcase,thepatternshouldbe(CSharpWebShop)(.*).Addtheroleandcheckeverypermission(no(un)checkallbuttonhere,veryannoying).So,wenowhaveaglobaluserroleandaprojectCSharpWebShoprole.
Toassigntheseroles,weneedtogototheAssignRolesscreen.First,addausertotheglobalrolesandassigntheuserrole.Then,addtheusertotheprojectrolesandassigntheCSharpWebShoprole:
>
Whenyousaveyourchangesandloginasthisuser,youwillnowfindthattheuseronlyhasaccesstoalltheprojectsstartingwithCSharpWebShop.Thatwasfastandeasyandaddingnewusersisfastandeasyaswell!Justgivethemtherole(s)youwantandyouaregoodtogo;noprojectconfigurationnecessary.
SummaryInthischapter,wehavetakenalookatvariousJenkinsfeaturesandplugins.Nexttovariousplugins,wehavediscussedsomesecurityconsiderations,bothonyournetworkandinJenkins.WiththeBlueOceanplugin,wehavetakenalookatthefutureofJenkins,bothvisuallyandfunctionallythroughcode.TheBlueOceanpluginisalsoourpreludetopipelines,whichwillbediscussedingreatdetailinthenextchapter.
JenkinsPipelinesSofar,wehaveworkedwiththeclassicJenkins.Wehavecreatedprojectsandpipelinesofprojects,createdviewstovisualizethedependencies,andusedvariouspluginstoaiduswiththistask.Inthepreviouschapter,wealsogotatasteoftheJenkinsprojectsincodeusingthenewBlueOceanpipelineeditor.WearenotgoingtouseBlueOceaninthischapter,butwearegoingtotakeabetterlookatthesepipelinesusingconfigurationascode.Justtobeclear,theBlueOceanpluginisnotrequiredtorunanysampleinthischapter.Youcanusepipelines,multibranchpipelines,andJenkinsfileswithoutBlueOcean.
Usingcodeforyourconfigurationhasacoupleofadvantages.Firstofall,youcanwriteyourowncodeaccordingtothelateststandardsandbestpractices,whichallowsforoptimization,parameterization,andreuse.Second,codeiseasytostoreinsourcecontrol,soyoucanapplyversioningtoyourJenkinsprojects.Codemoveseasy,socopyingaprojecttoanotherJenkinsserveriseasy.Thebestpartisthatyoucankeepyourcodewiththeprojectyouarebuilding,soyouhaveeverythinginoneplace(whichisprobablyGit).
Ofcourse,usingcodehassomedisadvantagesaswell.Forsomepeople,itmaybeeasiertoclickabitina(somewhatintuitive)userinterfacethanitistocode.YouwillnotgetanyintellisenseforyourJenkinscode,soitisnotalwaysclearwhatyoucanwrite.ThebiggestdownsidewiththeJenkinspipeline,forme,isthatnoteverypluginsupportsthisnewerstyleofconfiguringprojects.Andifaplugindoesnotsupportit,youcannotuseit.Luckily,thereisstilltheWindowsbatchorLinuxshellcommand,sobyfar,mostthingscanbeachievedusingthecodepipeline.Theworstcasescenarioisthatyoucreateaclassicprojectforthefewpluginsthatarenotsupportedandtriggeritfromyourpipelinecode.
Inthischapter,wearegoingtowritethesepipelinesandlearnhowtorunthemusingJenkins.
GroovyFirstthingsfirst,wehaveseenthepipelinesyntaxinthepreviouschapterwhenweusedtheBlueOceanpipelineeditor.ThelanguagethatisusedinthesepipelinesiscalledGroovy(http://groovy-lang.org/).GroovyisnotunlikeJavaScriptandJavaScriptwouldactuallybeamoreappropriatenamebecauseGroovyisascriptinglanguagefortheJavaplatform.YoucandownloadGroovyfromtheirwebsite,http://groovy-lang.org/download.html,incaseyouwanttopracticelocally.ForWindows,simplydownloadGroovy,unzipit,andaddanentrytothePATHvariableto[unpackfolder]\groovy-[your.version]\bin.YoucanaddaPATHvariablebygoingtothecomputer'sControlPanel,thenSystem,andthentheAdvancedSystemsettingsontheleft-handsidemenu.InthepopupisanEnvironmentVariables...button,whichwilltakeyoutoanotherpopupwhereyoucanedityourPATH.
Now,createafilesomewhereandputsomeHelloWorldcodeinit:
System.out.println("HelloJava");
println"HelloGroovy"
Asyoucansee,fromthefirstline,JavacodeisvalidinGroovy.ThesecondlineismoreGroovy-ish.Youcansavethefileasmyfile.groovy,.gvy,.gy,or.gsh,allarevalidGroovyfileextensions.Youcannowrunthescriptbysimplyrunninggroovymyfile.groovyinaCommandPrompt.IfyoudidnotaddthepathtothePATHvariableinWindows,youwillhavetoreferencegroovy-[version]\bin\groovy.batmanuallyeverytime.Hereisalittleshortcut:ifyourfilehasthegroovy,gvy,gy,orgshextension,thenyoudonothavetotypetheextension,justgroovymyfilewillsuffice.
Sonow,youhaveGroovyrunninglocally.IamnotgoingtodiscusstheentireGroovysyntaxhere.Instead,Iamgoingtodiscusslittlebitsthroughouttheexamplesintheremainderofthechapter.TheGroovywebsitehassomelearningmaterialsaswell.Restassured,youdonotneedtobeaGroovyaficionadotogetthisstuffworkinginJenkins.
PipelineprojectsSo,let'scontinueontoJenkins,wherewearegoingtocreateourfirstPipelineproject.GotoNewItemandpickthePipelineprojecttemplate.Thiswilltakeyoutotheconfigurationscreen,whichlooksalittledifferentfromwhatyouareusedto.Forexample,youaremissingsomeoptionsintheGeneralsectionoftheconfiguration.TheSourceCodeManagementsectionismissing.Butmostimportantofall,yougetabrandnewPipelinesection:
InthisPipelinesection,yougetachoicetopickyourpipelinedefinition,PipelinescriptorPipelinescriptfromSCM.Fornow,wearegoingtoleaveitonthePipelinescript,butlater,wewillseethattheotheroptionisactuallywaymorepractical.BelowthedefinitionfieldisalargeScriptfieldwhereyourpipelinegoes.RememberthatthisscriptwillruninsidetheJenkinsenvironment,soyougetvariousenvironmentvariablesandfunctionsthatwouldnototherwisebeavailableinGroovy.Ontheotherhand,becauseyouarerunninginJenkins,youractionsarelimited.Forexample,youdonothavesufficientpermissionstosaveafilesomewheretodisk.UndertheScriptfieldisacheckboxwhereyoucanenableordisabletheGroovySandbox.DisablingitgivesyoumoreprivilegesthanyouwouldnormallyhaveinJenkins,sofromasecurityperspective,itisbesttoleaveitenabled.
Wewillgettothescript,themostimportantpartofthisentireconfiguration,inabit.YoucanalreadyreadmoreaboutitontheJenkinswebsite(https://jenkins.io/doc/book/pipeline/syntax/).ItisgoingtolookabitdifferentfromthepipelinewegeneratedusingtheBlueOceanpipelineeditor.Thepipelineeditorgeneratedtheso-calledDeclarativePipelineSyntax,whichisamorerecentadditiontoJenkins,whilewearegoingtousethescriptedpipelinesyntax.Scriptedsyntaxisbasicallywhatyouwouldexpectfromascript;itjustrunscodefromtoptobottom.Beforewecontinue,let'sfocusonthedifferencesbetweenthetwoalittlebit.
DeclarativepipelinesyntaxDeclarativesyntax,whichissupportedsinceversion2.5ofthePipelineplugin,putsthefocusonwhatyouwantratherthanhowyouwantit.Ithasacertainstructure,whichisrequiredfortheBlueOceanpipelineeditor.Ithink,Icanbestexplainthiswithanexample.Takethefollowingdeclarativesyntaxscript:
pipeline{
agentany
stages{
stage('Greet'){
steps{
echoenv.greetings
}
}
}
environment{
greetings='Hello'
}
}
Eventhoughtheenvironmentnodeisbelowthestagesnode,theechoenv.greetingslineprintsHello.Obviously,theenvironmentnodewasinterpretedbeforethestageswereexecuted.Thisisbecausewearedescribingwhatwewantratherthanhowwewantit.Wewantanenvironmentvariablenamedgreetings.Wewantstagesandsteps.WedonotreallycarehowJenkinsorGroovyassignsthegreetingsvariableandreadsthevalueorhowitinjectsitintooursteps;wejustdescribethatwewantit.Adeclarativescriptalwaysstartswithapipelineblock,thatishowJenkins,andwe,knowthatitisdeclarative.
Sincewehavethepipelineeditorthatcangeneratethisforus,let'snotgodeeperintothisstyle.Alotofthesyntaxstillhasoverlapwiththescriptedpipelinesyntax.
ScriptedpipelinesyntaxThescriptedpipelinesyntaxisprobablyalittleeasiertograsp.Itisjustcodelikeyouareusedto.Assuch,itdoesnotreallyneedalotofexplanation.Thepreviousexamplecanberewritteninregularscriptedsyntax:
defgreetings='Hello'
stage('Greet'){
echogreetings
}
Groovyisanoptionallytypedlanguage,soyoumaybemorecomfortablewiththefollowingalternative:
Stringgreetings='Hello'
[...]
ThatisALOTshorterthanthesamecodeindeclarativestyle.Asyoucansee,westillusethestage('Greet'){...}block,butwedonotwrapitinastages{...}blockanditdoesnotcontainasteps{...}block.Infact,stagesandstepsarenotevenallowedinthiscontext.Jenkinswillnotrecognizethemandwillthrowanerror.
PipelinestagesAtthispoint,youmaybewonderingwhatthisstageblockisaboutanyway.PuteitherthedeclarativeorthescriptedpipelineexampleintheScriptfieldofyourpipelineproject,saveit,andrunit.Onyourprojectpage,youshouldnowseeyourfirstbuildandyourfirststage:
YoucanaddanotherstageandJenkinswillvisualizethatonetoo:
defgreetings='Hello'
stage('Greet'){
echogreetings
}
stage('Saygoodbye'){
echo'Goodbye'
}
Theoutputofthescriptlooksasfollows:
Asyoucansee,inthesecondpicture,build#1didnothavetheSaygoodbye
stageyet.Ifyouaddstagesattheendlikethat,Jenkinsjustleavesthemblankforbuildsthatdidnothavethatstageyet.However,ifyouchangeastagenamecompletely,oryouremoveastage,Jenkinscannotshowolderbuilds.Furthermore,ifastagefails,thiswillbeindicatedbyaredsquare:
Fornow,wehavenosourcecontrolinourpipelineyet,butwhenyoudo,theblockthatnowsaysNochangesshowsthenumberofnewcommitsincludedinthecurrentbuild.
Clickingonasquareenablesyoutoviewthelogsforthatstageandevenforseparatesteps,includinghowmuchtimeittooktocomplete.Thefollowingscreenshotcontainsthelogsoftheteststepswegeneratedinthepreviouschapter(wewillrunthemlaterinthischapter):
Ofcourse,youcanstillviewthecompleteconsoleoutput.Allinall,thesearesomeprettysweetfeaturesandwearegoingtoseethemliveinthischapter.
ThesnippetgeneratorAnotherusefulfeatureisthesnippetgenerator.Itisveryhardtoguesswhatsyntaxsomestepsexpectanddocumentationisoftenmissingorscattered.Youareprobablynotevensureifacertainpluginisinstalledornot.Thisiswherethesnippetgeneratorcomestotherescue.YoucanfindalinktothesnippetgeneratorundertheUseGroovySandboxcheckboxintheprojectconfigurationoratthebottomoftheleft-handsidemenuonapipelineprojectpage.Clickinganyofthoselinkstakesyoutoanewpagewhereyoucanselectastepyouwanttouse.Youcanthenspecifysomeoptionsyouwanttouse.Youcanthengenerateacodesnippetandcopyandpasteitintoyourowncode.Forexample,youcangenerateasnippetthatwillpollaGitrepositoryusingyourcredentials:
Additionalpluginscanaddnewstepsthatyoucaninspectinthesnippetgenerator.Becarefulthough,therearesomestepsthatfallunderthe-Advanced/Deprecated-category.Thosearetwoverydifferentthingsandyoucannottellthemapart.Theoneyoucanandprobablywanttouse,whiletheotherisbetternotused.Personally,Itrytoavoidstepsinthiscategory
altogether.Averyweirddesignchoice,butatleast,youhavebeenwarned.
BuildingthewebshopNowthatweknowhowtorunpipelines,wecantestthescriptthatwecreatedinthepreviouschapterusingtheBlueOceanpipelineeditor.Simplycopythescript(foundintheGitHubrepository)andcopyitintotheScriptfield.Alittlewarninghere,pipelinescancausedeadlocks!Apipelineneedsanodetorunon,butthenspawnsseparatebuildsfortheindividualstages.SowhenyourpipelineisrunningontheLinuxnodeandastagerequirestheLinuxnodeaswell,buttheLinuxnodeisconfiguredtoonlyrunonebuildatatime,thestagewillwaitindefinitelyfortheLinuxnodetobecomeavailableagain.Makesureyoualwayshaveatleastonemoreexecutoronyournodesthanyouhavepipelines.Unfortunately,itiscurrentlynot(yet?)possibletotriggerpipelinesonaspecificnode,soyoucanhaveanodespecificallyforbuildingpipelines.Thatway,youcanbuildstagesonotherslavesandyouwouldbesurestageswouldneverbewaitingontheirhostpipelinestofinish.
Wecannowrunthescriptfromthepreviouschapter.Youwillnoticethatthereareindeedtwobuildstriggered.Toobadyoucannotseewhichbuildisthepipelineandwhichbuildisthestage:
Funnything,thisscriptrunsthreeteststagesinparallel.Thescriptstillonlyhasoneteststagenodethough,sothestageviewwillstillonlyshowonestage,butthiswillshowascompletedassoonasoneofthetestscompletes.
YoucanstillseethethreedifferentteststagesrunningintheJenkinsbuildoverviewthough:
Thescriptshouldrunfine.SincewehavescriptedtheGitLabtrigger,youshouldnowalsoseethatconfiguredintheprojectconfigurationeventhoughyouonlypastedthescript.Forthistriggertoworkproperly,youstillneedtocreateawebhookinGitLabbytheway.Onceyoudothat,youwillevenseethatyourbuildstatusisupdatedinGitLab.IfyouscriptanythinglikeaGittriggerorpolling,youneedtorunthescriptatleastoncetoenableit:
AfunnythinghappenedwhileIwaswritingthischapter.Iwastestingsomescriptsandnoticedseveralpluginsneededupdating.Beingagoodcivilian,IupdatedmypluginsonlytofindmybuildsfailingwithsomeweirdJavaNullPointerException.TurnsouttherewasabuginthenewversionoftheGitLabplugin.Revertingtothepreviousversionfixedtheproblem.Ithappenswithnewversionsofpluginsthatintroducebugs(youthinktheyarepracticingCI?)orareincompatiblewithyourcurrentversionofsomeothercomponent.
TheJenkinsfilePuttingyourscriptintheJenkinsconfigurationisfine,butthereisanalternative,theJenkinsfile.TheJenkinsfileissimplyafileinyourGitrepositorythatcontainsthepipelinescriptyouwanttorun.So,copythepipelinescriptandplaceitinafilenamedJenkinsfile(noextension)intherootofyourrepositoryandcommitittoGit.Now,gototheconfigurationofyourpipelineprojectandchangethedefinitionfromPipelinescripttoPipelinescriptfromSCM.YoumustnowpickyourSCM,whichisGit,andthenconfiguretheconnectiontoyourrepository.Thisworksexactlythesameasinfreestyleprojects.Additionally,thereisaScriptPath,whichdefaultstoJenkinsfileandLightweightcheckout,whichyoucanleavechecked.Again,saveandruntheprojectand,basically,everythingwillbelikebefore,exceptyourscriptisnowinyourrepository.
ThatisreallyallthereistotheJenkinsfile.However,theJenkinsfileisaprerequisiteforthenextstepinourpipelineadventure,themultibranchpipeline.
MultibranchpipelineTheMultibranchpipelineisthenextstepfromtheregularpipeline.Itisprettymuchthesame,exceptitcanrunyourscriptoneverybranchthathasit.Anynewbranchesareautomaticallyscannedandbuild.AddanewiteminJenkinsandpicktheMultibranchPipelineprojecttype.TheconfigurationrequiresyoutospecifyyourGitrepository,butthistime,youcannotspecifyaspecificbranch.ThebuildconfigurationmodeissettobyJenkinsfilewhich,inthiscase,meansyourJenkinsfilefromsourcecontrol.Wecanleavealltheothersettingsfornow.SaveyourconfigurationandyouwillbetakentotheScanMultibranchPipelineLog.YoushouldseeJenkinscheckingyourrepositoryforbranchesandJenkinsfiles:
Gettingremotebranches...
Seenbranchinrepositoryorigin/master
Seen1remotebranch
Checkingbranchmaster
‘Jenkinsfile’found
Metcriteria
Scheduledbuildforbranch:master
Done.
Theprojectpageforthemultibranchpipelineprojectlooksverydifferentfromotherprojects.Thatisbecauseitdoesnotshowlatestbuildinformation,butbranchesthatcanbebuildusingtheJenkinsfile.TheprojectwillscanfornewbranchesregularlyandaddthemtotheprojectpageifaJenkinsfileisfound.Alternatively,youcantriggerascanmanuallybyclickingtheScanMultibranchPipelineNowontheleft-handsidemenu:
Ontheprojectpage,youcanseewhichbranchesarecurrentlypassingthebuild,whenbrancheswerelastbuilt,andhowlongthattook.Clickingonabranchtakesyoutothebranchpage,whichlooksmoreliketheprojectpagethatyouareusedto.Youcanconfigurebranchesandspecifybuildtriggers,howmanybuildstokeep,andsoon,butitisadvisedtokeepasmuchconfigurationinyourscriptaspossible.Forexample,youcankeepuptofivebuildsbycallingthepropertiesfunctioninyourscript:
properties([buildDiscarder(logRotator(numToKeepStr:'5',artifactNumToKeepStr:'5')),gitLabConnection('LocalGitLab'),pipelineTriggers([])])
Inthedeclarativesyntax,youcanaddthistotheoptionsnode:
options{
gitLabConnection('LocalGitLab')
buildDiscarder(logRotator(numToKeepStr:'5',artifactNumToKeepStr:'5'))
}
Formoreoptions,youcancheckoutthesnippetgeneratorforproperties:Setjobproperties.
Onethingthatiscurrentlyveryconfusingaboutmultibranchpipelineprojectsisthatyouhavenowayofseeingifanybranchfailswhenyouareonthemainpage:
Insteadofablueorredball,themultibranchpipelineshowsafolderwithnostatusinformation.Soyouhavenoideawhethereverythingbuildsordoesnot.Theweathericondoesshowthecumulativestatusofyourbranchbuildsthough,butevenwhenitiscloudy,yourbranchescanallcurrentlypassthebuild.Thelastbuildtimesarealsonotbuildtimesofyourbranches,butthelasttimeyourrepositorywasscannedforbranches.Finally,thebuildbuttontriggersabranchscanandnotabuildofyourbranches.TheBlueOceanplugindoesthisalotbetteralreadybecauseitshowswhetherallthebuildsarepassingorifoneormorebuildsarefailing.
Likewise,itshowswhenabuildfails:
Anotherverynicefeatureofthemultibranchpipelineisthereplayfunction.ItoftenhappensthatyouwanttotryoutsomescriptinJenkins,butyouarenotsureonthesyntaxorwhatworksbest.HavingtocommityourJenkinsfileforeachsmallchangeyoumakejusttotestwhetheritworksisrathertiresomeandreallypollutesyourcommitlog.Withreplay,youcansimplyclickonabuildandhitthereplaybutton.Youareallowedtochangethescriptbeforeyoureplaythethebuild,soyoucancorrectwhateveryouwantuntilitworks.Wheneverythingworks,youcancopythescriptfromyourreplaywindowandintoyourJenkinsfileandyouwillcommitaworkingscriptrightaway.Whenyouhavemultiplescripts,youcanediteachoneofthem.Replayonlyworkswhenthepreviousbuildhadvalidsyntaxthough.So,youcanreplayfailedbuildsunlesstherewasasyntaxerrorinyourscript.
BranchscriptsWehavetalkedaboutbranchingbefore,butitisnotuntilnowthatitsusebecomesapparent.Usingmultibranchpipelines,youcanhaveadifferentbuildstrategyperbranch.Forexample,youprobablywanttobuild,test,report,andSonarQubeyourdevelopmentbranch,butyouwanttobuild,minify,test,andreleaseyourproductionbranch.OneoptionistohaveadifferentJenkinsfileineachbranch.However,thatmakesitalittleharderforGittomergebranchesandyouwillalwayshaveanadditionalmergebranch.
AnothermethodischeckingoutwhatbranchyouareworkinginusingtheBRANCH_NAMEenvironmentvariableandactbasedonthat:
stage('TestScript'){
echoenv.BRANCH_NAME
if(env.BRANCH_NAME=='master'){
echo'Thisisthemasterbranch'
}else{
echo'Thisisnotthemasterbranch'
}
}
Havingacoupleofif-elsebranchesinyourJenkinsfilemaybefine,butwhenyourbuildlogicisalreadycomplicatedorcompletelydifferentforeachbranch,youmaybebetteroffbywritingaGroovyscriptperbranchandloadingthatintoyourJenkinsfile.Createtwonewfilesintherootofyourproject,jenkins.master.groovyandjenkins.Development.groovy.Wearegoingtocreateadifferenttestroutineforboththefiles.Inthejenkins.master.groovyfile,wearegoingtotestourminifiedscriptand,injenkins.Development.groovy,wearegoingtotestthesourcecodeandanalyzeitusingSonarQube(orforsimplicity,wearejustgoingtoechosomelinesoftext):
//jenkins.master.groovy
deftest(){
echo'Testingtheminifiedsoftware...'
echo'Minifiedsoftwaretested.'
}
returnthis
//jenkins.Development.groovy
deftest(){
echo'Testingthesourcecode...'
echo'Sourcecodetested.'
echo'RunningSonarQubeanalysis.'
}
returnthis
Inbothscripts,wearesimplydefiningafunctionandreturningthebody.ReturningthebodyisimportantifwewanttousethetestfunctioninourJenkinsfile.IntheJenkinsfile,wecanloadanyofthesescriptsbasedontheenv.BRANCH_NAMEvariable.WeneedtorunthisinsideanodeblockortheloadfunctionwillfailbecauseRequiredcontextclasshudson.FilePathismissing:
node('linux'){
stage('Build'){
//Samelogicforeverybranch.
checkoutscm
echo'Buildingthesoftware.'
}
stage('Test'){
//Logicisdifferentforeverybranch.
defscript=load'jenkins.'+env.BRANCH_NAME+'.groovy'
script.test()
}
}
Finally,createaDevelopmentbranchifyoudidnotalreadyhaveone;makesureeverythingiscommittedtoGitandscanthemultibranchpipelineforbranches.Youcannowbuildyourmasteranddevelopmentbranchesandyoushouldseetwoseparatebuilds.Ofcourse,youwillnowneedtoaddanadditionalscriptforeverynewbranchsowecanaddadefault:
deffileName='jenkins.'+env.BRANCH_NAME+'.groovy'
if(!fileExists(fileName)){
echo'Branchspecificscriptnotfound,usingDevelopmentdefault.'
fileName='jenkins.Development.groovy'
}
defscript=loadfileName
script.test()
Now,whenyouaddanewbranch,itwillbuildusingtheDevelopmentscriptbydefault.
Speakingofwhich,whenwecommittoabranch,wewantthatbranchtobuildandnotalltheothers.UsingtheGitLabtriggerwithawebhook,weshouldbesureourwebhookworks.Youcancreateawebhookfortheproject,suchas[...]/project/JS%20Web%20Shop%20-%20Pipeline,oryoucanspecifyabranch,suchas[...]/project/JS%20Web%20Shop%20-%20Pipeline/master.Testingthefirstwebhookwilltriggerallthebranches,butwillstillonlytriggerthecurrentbranchwhencommitting.
CompletingthescriptSofar,weknowhowtoutilizethemultibranchpipelineandwehavemostofaworkingscript,butitisnotcompletelywhatweconfiguredearlierinclassicJenkins.So,let'scompletethescriptsowehaveprettymucheverything.Wecannowalsoeasilydifferentiatebetweendifferentbranches.Basically,Iwanttogeteverythingupandrunningasthoughwewerebuildinganactualproduct.Wewillfillinthefinaldetailsonreleasingthesoftwareinthefinalchaptersofthisbook.
First,wearegoingtomakeafeweditstoourgulpfile.Itwasfineearlier,butnowthatwearegoingtotriggerthreebuildsatthesametime,itmakesnosensetousePhantomJSeverytime.Also,wewanttobeabletotriggerthetest-mintaskwithoutalsotriggeringtheregulartesttask.Thechangeisprettysimple.JustchangethebrowsersKarmaconfigurationinthegulpfile.YoucanfindthechangedgulpfileintheGitHubrepositoryforthisbookinthecodesamplesforthischapter:
.task('test',['build'],function(done){
newkarma({
configFile:__dirname+'/test/karma.conf.js',
singleRun:true,
browsers:browsers?browsers:['PhantomJS']
},[...]
})
.task('test-min',['build'],function(done){
newkarma({
configFile:__dirname+'/test/karma.min.conf.js',
browsers:browsers?browsers:['PhantomJS']
},[...]
})
WecannoweitherspecifythebrowserswhenstartingGulporwedonot,inwhichcaseKarmawillusePhantomJS.Youmayalsowanttochangeyourdefaulttasksothatittestsonlyyoursourcecodeandnottheminifiedcode:
.task('default',['lint','test','test-node']);
WecannowcontinuewiththeJenkinsfilethatwehadinthepreviouschapter.Wecanalsorewriteitusingthescriptedsyntax.Sincewehavethedeclarativescriptalready,let'sjustcontinuetousethat.Thechangewejustmadetoour
gulpfileactuallyplaysverynicewithourcurrentscript,sowedonothavetochangeanythingforthattowork.
OnethingthatbothersmeinthecurrentsetupisthatweneedtounstashEverythingandrunnpminstallforeverybrowsertest.Unfortunately,thereisnothingwecandoaboutthat.Jenkinsrunseverystageinitsownworkspaceand,whenaworkspaceisalreadyinuse,itcreatesworkspace@[email protected],insteadofsharingtheworkspace,parallelbuildsneedtorunintheirowncontextentirely.Wecanrunthebrowsertestsinsequence,butlet'sleaveitlikeitis.
Node.jstestsThefirstthingwecanaddtothescriptareourNode.jstests.WecanplacethisbetweentheBuildstageandtheTeststage.ItisfinetorunthesetestsonLinuxandtheyareprettylightweight.Andweshouldalsonotforgettopublishourreportsexcept,ofcourse,theCoberturareport,whichwecannotpublishusingthepipeline:
stage('TestNodeJS'){
steps{
node(label:'linux'){
ws(dir:'web-shop-pipeline'){
gitlabCommitStatus(name:'TestNodeJS'){
sh'node_modules/.bin/istanbulcovernode_modules/jasmine-node/bin/jasmine-node----junitreport--outputnode-junittest/spec'
junit'node-junit/*.xml'
publishHTMLtarget:[
allowMissing:false,
alwaysLinkToLastBuild:false,
keepAll:false,
reportDir:'coverage/lcov-report',
reportFiles:'index.html',
reportName:'NodeJSCoverageReport'
]
}
}
}
}
}
SonarQubeThenextthingwewillwanttodoisrunaSonarQubescanner.First,let'sfixoursonar-project.propertiesfile,whichstillhaschapter7forprojectKeyandprojectName:
sonar.projectKey=jswebshop
sonar.projectName=JSWebShop
sonar.projectVersion=1.0
[...]
ToinvoketheSonarQubescanner,wearegoingtouseascriptblock.Usingthescriptblock,wecandefinevariablesandassignthemvalues.Inthiscase,wearegoingtousethetoolcommandtogettheSonarQubelocation.Youcangeneratethetoolcommandusingthesnippetgenerator.Afterthat,wecaninvokeSonarQubeusingashellcommand.SinceitisthePhantomJSteststhatwillgeneratetheLCOVreport,wecanrunthisstageontheLinuxnodeagain:
stage('SonarQube'){
steps{
node(label:'linux'){
ws(dir:'web-shop-pipeline'){
gitlabCommitStatus(name:'SonarQube'){
script{
defsonar=toolname:'Local',type:'hudson.plugins.sonar.SonarRunnerInstallation'
shsonar+'/bin/sonar-scanner'
}
}
}
}
}
}
Weprobablyalsowanttofailthebuildifthequalitygatefails.Todothis,wemustrunourSonarQuberunnerinsideaSonarenvironmentand,afterthat,wecanwaitforSonarQubetoreportthestatusofthescanbacktoJenkins.WecanwaitforSonarQubebyusingthewaitForQualityGatefunction.YoushouldknowthatthesestepsdonotneedtobeinthesamestageandthatwaitForQualityGatepausesthecurrentbuildandwaitsforSonarQubetofinishthescan,whichmighttakeawhile.Agoodsetup,forexample,wouldbetorunyourSonarQubeanalysis,thendoyourtests,andfinally,reporttheSonarQubestatus.Thatway,youaremakingthemostoutoftheasynchronousnatureofSonarQube.Ourprojectsscanswithinsecondsthough,sotherereallyisnoneedforsuchasetup:
script{
defsonar=toolname:'Local',type:'hudson.plugins.sonar.SonarRunnerInstallation'
withSonarQubeEnv{
shsonar+'/bin/sonar-scanner'
}
defqg=waitForQualityGate()
if(qg.status!='OK'){
error"Pipelineabortedduetoqualitygatefailure:${qg.status}"
}
}
Thereisjustonemorethingweneedtodotogetthistowork.WeneedtosetupawebhookinSonarQube.SogotoSonarQube,thenAdministration,andthentheGeneralSettingsunderConfiguration.Thereisapageforwebhooks.NameyourwebhookJenkinsorsomethingsimilarandgiveittheURLhttp://127.0.0.1:8080/sonarqube-webhook/(or<yourJenkinsinstance>/sonarqube-webhook/).Thetrailingbackslashismandatory!Testheroutandseeifitworks.IfyourbuildhangsonsomethinglikeSonarQubetask'AV1NaMZ99HRZ_Yq5jAH6'statusis'PENDING',yourwebhookprobablyisnotconfiguredcorrectly.YoucouldwrapwaitForQualityGate()inatimeoutblockjusttobesureyourbuildwillnothangforever:
timeout(time:10,unit:'SECONDS'){
defqg=waitForQualityGate()
[...]
}
SeleniumtestsNext,aretheSeleniumtests.ThisisreallyjustabatchcommandontheWindowsslave.Wecanunstash'Everything'andrunnpminstallagain,butwecanalsojustsaveonefromourtests.Whilenotalltestscanusethesameworkspace,itisperfectlyfinetogivethemtheirownworkspaces.SincewearealreadyusingtheTestChromenodetopublishourreports,let'sbuildthatoneinaworkspacethatwecanalsouseforourSeleniumtests:
[...]
"TestChrome":{
node(label:'windows'){
ws(dir:'web-shop-pipeline'){
gitlabCommitStatus(name:'TestChrome'){
[...]
Afterthat,itisjustamatterofrunningthebatch.Wearegoingtorunthescriptalittledifferentthanbeforethough.First,forclarity,wewilljustrunfourbatchscriptsratherthanappendingthemwith&&.Second,becausewearerunninginascript,andthisisonlypossibleinascriptblock,wecanhaveatry-catch-finallyblock.Atry-catchblockcatchesanyerrors,whichwedonotreallyneed,butafinallyblockexecutesalways,evenwhenthereisanerror,whichisprettyhandy!Thatway,wecanalwaysdeleteourrunninginstancefromPM2,evenifProtractorfails:
stage('Selenium'){
steps{
node(label:'windows'){
ws(dir:'web-shop-pipeline'){
gitlabCommitStatus(name:'Selenium'){
script{
try{
bat'node_modules\\.bin\\webdriver-manager.cmdupdate'
bat'node_modules\\.bin\\pm2.cmdstart--name=jenkinsindex.js----port=1234'
bat'node_modules\\.bin\\protractor.cmd--baseUrlhttp://localhost:1234test\\protractor.conf.js'
}finally{
bat'node_modules\\.bin\\pm2.cmddelete-sjenkins'
}
}
}
}
}
}
}
ArchivingartifactsThenextstepisprettyeasy,wewanttoarchiveourartifactssowecanputtheminproductionlater.WearebuildingourwebshoponLinux,sothatiswherewearegoingtoarchiveourartifactsaswell:
stage('ArchivingArtifacts'){
steps{
node(label:'linux'){
ws(dir:'web-shop-pipeline'){
archiveArtifacts'prod/,node_modules/bootstrap/dist/css/bootstrap.min.css,node_modules/angular/angular.min.js,node_modules/bootstrap/dist/js/bootstrap.min.js,node_modules/jquery/dist/jquery.min.js'
}
}
}
}
WedonotneedtocommitthestatustoGitLabforthisone.Ifthisfails,Jenkinsfailsandobviouslynotourcommit,becausewetestedthosethoroughlyatthispoint.ThearchiveArtifactsstepcaneasilybecreatedusingthesnippetgeneratorbytheway.YouneedthearchiveArtifactsstepnearthetopofthedropdownandnotthearchivestepunderAdvanced/Deprecated(becausethatoneisdeprecated).
YoucanfindtheartifactsonyourbuildpageoronthebranchoverviewatthefrontofthebuildbarbetweenthetimeandGitcommitsandyourfirststage.
BuildfailureNomatterhowwellyouwriteyourcode,yourbuildwillfailonceinawhile(andprobablymoreoften),soitisagoodideatosendanemailwhenthishappens.Thedeclarativepipelinehasapostnodethatletsyouactonacertainbuildstatus,forexample,failure.Thepostblockcanbeplacedinthetopnodeofyourpipelineorinastageblock(thatway,wecouldhaveomittedthetry-finallyblockearlier).Thepostconditionsarealways,changed,failure,success,andunstable.Wearegoingtousefailureandsendanemail:
pipeline{
agentany
[...]
stages{
[...]
}
post{
failure{
mailbody:"""FAILED:Job'${env.JOB_NAME}[${env.BUILD_NUMBER}]':
Checkconsoleoutputat${env.BUILD_URL}""",
subject:"""FAILED:Job'${env.JOB_NAME}[${env.BUILD_NUMBER}]'""",
to:'[email protected]'
}
}
}
Youcanmakethebodywhateveryouwant,butusingsomeenvironmentvariablesisprettyhandy,soyoucanquicklygotothespecificbuildfromyouremail.Youmaywanttosendanemailonunstableaswell.Whatisalsoreallyhandyisanemailwhenyourbuildgoesfromwhateverstatetosuccess.Youareprobablynotinterestedineverysuccessfulbuild,butyoudowanttoknowwhenabuildrecoversfromapreviousfailure.Unfortunately,thisisnotcurrentlyeasilypossible;youcouldhaveavariable,buildFailed,andaddapostblockwithafailureblocktoeverystageandsetthebuildFailedvariable.Then,sendanemailinthechangedblockoftheglobalpostblock.Thisisquiteahassle;itisbettertojustsendanemaileverytimethebuildstagechanges:
pipeline{
agentany
[...]
stages{
[...]
}
post{
failure{
[...]
}
changed{
mailbody:"""STATECHANGED:Job'${env.JOB_NAME}[${env.BUILD_NUMBER}]':
Checkconsoleoutputat${env.BUILD_URL}""",
subject:"""STATECHANGED:Job'${env.JOB_NAME}[${env.BUILD_NUMBER}]'""",
to:'[email protected]'
}
}
}
}
ThatisitfortheJenkinsfilefornow.Inthenextchapters,wearegoingtoaddtotheJenkinsfilejustabitmore.Fornow,youcanfindthecompleteJenkinsfileforthischapterintheGitHubrepositoryforthisbook.
SummaryInthischapter,weexploredmultibranchpipelines.MultibranchpipelinesofferalotofflexibilitythroughscriptedanddeclarativestyleGroovyscripts.Unfortunately,writingascriptisnotevenhalfaseasyascreatingaprojectusingtheclassicprojects.Nexttoincreasedflexibility,thescriptoffersanadvantageinthatitisjustcodeandiscommittedtoGit,givingyoueasybackupsandversionmanagement.MultibranchpipelinesalsomakeiteasytobuildeverybranchinyourGitrepositoryautomatically,evenasyouaddthem.Inthenextchapter,wearemovingawayfromJenkinsagaintocreateandtestawebAPI.
TestingaWebAPIAtthispoint,youareprobablyhopingwearegettingtothedeliveryanddeploymentpart,right?Almost,sohangon.ThereisjustonemorethingIwouldliketodiscuss,whichisWebAPItesting.Thereareplentyofapplicationtypes,suchasdesktopapplications,embeddedapplications,andwebapplicationslikewecreatedinthisbook.And,ofcourse,allthoseapplicationscanbecreatedusingatonoflanguagesandatleastagazillionframeworks.Allofthemtakeadifferentapproachtotestingandhavetheirowntestingframeworks.Throughoutthisbook,wehaveusedJasmine,xUnit,andSelenium,tonamebutafew.Inthisdayandage,companieswantwebservices,though,betheyJSONservicesorXML/SOAPservices.AndthereisonethingIreallylikeaboutthosekindsofapplications;theyalloperateonHTTP,nomattertheirunderlyinglanguageorframework.Thatmeanstestingiscompletelylanguageagnostic!
Inthischapter,wearegoingtobuildasimplewebserviceusingNode.js.Noworries;itwillbereallysimpleandwedonotneedthewholeGulpcircusthistime.Iwillevenleaveoutthedatabasebecausewearereallynotinterestedinallthat.Afterwehavebuiltthissimpleservice,wearegoingtotestitusingPostman.Postmanisanapplicationformakingandtestingwebrequests.Imostlylikeitbecauseitiseasy,lightweight,andusesJavaScriptforscripting.
TherearetwoalternativesIwouldliketomentionreallyquicklyup-front:SoapUI,andApacheJMeter.BothSoapUIandJMeterareobviously(alittleuglyandclunky)Javaapplications.TheydonotreallyfitinwithyourotherWindowsapplications.Anyway,theyarequitepowerful,whichistheirstrengthandweakness.BothSoapUIandJMetersupportthetestingofHTTP,HTTPS,SOAP,REST,andevenmore.SoapUIevengeneratesendpointsandrequestsfromanyWSDLyouspecify,makingtestingreallyquickandeasy.Thelearningcurveforbothapplicationsisalittlesteep,though,becausetheycandojustaboutanything.
IhavealittlemoreexperiencewithSoapUIthanIhavewithJMeterandIknowSoapUIusesGroovyforcustomtests.Italsohasbuilt-infunctionalityforvarioustests,suchasschemacompliance,(not)SOAPfault,andHTTPstatus
codes.JMeter,ontheotherhand,isgreatforstresstestingandtiming.YoucaneasilysetmultiplecallstothesameAPIinaloop.Thebestpartisthatbotharefreetouse.
BuildingaRESTserviceThefirstthingwehavetodoiscreatealittleRESTservice.RepresentationalStateTransfer(REST)basicallymeansthataserviceisstatelessandmakesuseofthestandardHTTPverbslikeGET,PUT,POST,andDELETE.Thatmakesitabiteasierforustowrite.WecancreateaRESTfulserviceusingNode.jsandExpress,prettymuchlikewedidbefore.So,createanewfolderandnameitweb-apiorsomesuch.Next,weneedourpackage.jsonfile,sostartupacommandpromptandusethenpminitcommand.Youcanleaveallthedefaults,aswearenotreallygoingtodoanythingwiththemanyway.Next,wecaninstallExpress,thebody-parser,andthecommand-line-argspackage:
npminstallexpress--save
npminstallbody-parser--save
npminstallcommand-line-args--save
Wecannowsetupourbarebonesscriptthatallowsustoatleastruntheapplication:
varexpress=require('express'),
bodyParser=require('body-parser'),
commandLineArgs=require('command-line-args'),
app=express();
app.use(bodyParser.json());
varoptions=commandLineArgs({name:'port',defaultValue:80});
varserver=app.listen(options.port,'127.0.0.1');
console.log('Serverrunningathttp://127.0.0.1:'+options.port);
WecannowdecidewhatwewantwithourWebAPI.Howaboutwecreatealittleto-dolist?Wecaninsertitemsweneedtodo,updatethestatus,anddeleteitemswhentheynolongerneedtobedone.Let'screatealittlein-memorylist(assaid,wearenotgoingtouseadatabase)andreturnitonGET.SendinganidparameterintheURLisoptionaltoreturnasingleitemfromthelist:
vartodos=[{
id:1,
description:'Writeabook.',
status:'Inprogress'
},{
id:2,
description:'Celebratefinishingthebook.',
status:'New'
}];
app.get('/todo',function(req,res){
if(req.query.id){
res.json(todos.find(t=>t.id===+req.query.id));
}else{
res.json(todos);
}
});
Asyoucansee,wecanaccessURLparametersusingreq.query.parameterName.ThisisanExpressutilityandnotvanillaNode.js.Whenwehaveanid,wereturntheitemwiththatidfromthelistor,otherwise,wereturntheentirelist.Youcantestthisbyrunningtheapplication(usingnodemonindex.js,whichyoucaninstallusingnpminstallnodemon-gifyoudon'talreadyhaveit)andbrowsingtolocalhost/todoorlocalhost/todo?id=1usingyourfavoritebrowser.
Next,wewanttobeabletoinsertnewitems.InsertingisusuallydoneusingthePOSTverb.Wereallyonlyrequireonethinginthebody,whichisadescriptionoftheto-doitem.Wecan,optionally,allowastatustobeset,butuseadefaultofNew:
app.post('/todo',function(req,res){
if(req.body.description){
varnewId=Math.max.apply(null,todos.map(t=>t.id))+1;
varnewItem={
id:newId,
description:req.body.description,
status:req.body.status||'New'
};
todos.push(newItem);
res.json(newItem);
}else{
res.status(400).send('Descriptionfieldisrequired.');
}
});
Whenthedescriptionisnotset,wereturna400code,BadRequest,withthemessagethatthedescriptionfieldisrequired.Ifthedescriptionisset,wegetthemaxidinthecurrentto-dolistandadd1,whichwillbethenewid(itisalsonotthreadsafe,butthatdoesnotreallymatterforourexample).Wepushthenewitemintotheto-dolistandreturnthenewitemtotheclient.
WecurrentlyhavenowaytotestthismethodbecauseourbrowserdoesnotalloweasyPOSTing.WecouldwriteasmallJavaScriptscriptanddoanAJAXcall,butwearenotgoingtodothat.WearefirstgoingtowritethePUTandDELETEmethods,andthenwearegoingtotestthisthing.Then,wearegoingtorunactualE2Etestsandautomatethem!
WithaPUTrequest,weshouldbeabletoupdateitems.Basedontheid,wecansetthedescriptionandthestatusofanitem.Ifidisnotsetintherequest,wecanreturna400,likebefore,andwhenidisset,butnotfound,wecanreturna404.Whenalliswell,wereturnthe(updated)element:
app.put('/todo',function(req,res){
if(req.body.id){
varitem=todos.find(t=>t.id===+req.body.id);
if(item){
item.description=req.body.description||item.description;
item.status=req.body.status||item.status;
res.json(item);
}else{
res.status(404).send('Nosuchtodoitem.');
}
}else{
res.status(400).send('Idfieldisrequired.');
}
});
Asyoucansee,bothdescriptionandstatusareoptional.Ifneitherisset,wejustdonotupdateanythingandtherequestissortofequaltoaGETwithid.
Last,weneedaDELETEmethodtodeleteitemsfromthelist.LikewithPUT,weneedid.Whenidisnotset,wereturna400code.Whenidisset,butwecannotfindtheelement,wereturna404code.Whenanitemwithidisfound,wecanremoveitfromthearrayandreturnthedeletedelement:
app.delete('/todo',function(req,res){
if(req.body.id){
varitem=todos.find(t=>t.id===+req.body.id);
if(item){
todos.splice(todos.indexOf(item),1);
res.json(item);
}else{
res.status(404).send('Nosuchtodoitem.');
}
}else{
res.status(400).send('Idfieldisrequired.');
}
});
WenowhaveGET,POST,PUT,andDELETEmethodstoread,create,update,anddeleteourto-doitems.WehavenostateandeverythingisdoneusingregularHTTPverbs,sowearecompletelyRESTful.Intheremainderofthischapter,wearegoingtotesttheAPIusingPostman.
PostmanIhavealreadymentionedPostmanintheintroductionofthischapter.Postmancomesinthreeflavors:free,Pro,andEnterprise.Thefreeversionalreadygivesusmorethanweneedforthisprojectanditisreallygreatformostprojects,really.YoumayfindaPostmanChromeplugin,butthereisnoneedtobotherwithit.Itislimited,andlastIheard,theywilldeprecateitbytheendof2017.YoucandownloadPostmanfortheplatformyouareusingontheirwebsite,https://www.getpostman.com/.SimplyinstallPostman,nothingtoworryaboutthere,andstartit.WhenyoustartPostman,youwillseealoadingscreenwithatextlikeDistortingspace-timecontinuumorMovingsatellitesintoposition.
Bereallycarefulhere,oryouwillmessuprealityasweknowitforallofus!Justkidding;itisjustafunnymessageandwillnotactuallydoanything.WhenPostmanisstarted,itshouldlooksomethinglikethis:
Inhere,youcanenteraURL,verb,headers,cookies,payloaddata--prettymuchanything.MakesureyourAPIisrunning.NowtrytoGETthelocalhost/todoand
localhost/todo?id=1URLs.JustentertheURLsandhittheSendbutton.Youshouldgettheresultinthelargeresponsefieldatthebottom.Noticethereareacoupleoftabs.Youcan,forexample,viewallheadersthatwerereturnedintheresponse.
Ontheleft-handside,youcanfindyourcollectionsandhistory.Yourhistoryisautomaticallysavedandenablesyoutoquicklyredopreviouswebrequests.Youcansaverequestsandtestsincollections,whichwewilldolaterinthischapter.Ifyourhistoryandcollectionsarenotshowing,trymakingyourPostmanscreenalittlebigger.Youcanalsoshowitusingthebuttonatthetopleftofthescreen.
First,wearegoingtocreateanewto-doitemtocheckwhetheritallworksasexpected.YoucanleavetherequestURLonlocalhost/todo.SinceeverythingreliesontheHTTPverbs,wedonothaveURLs,suchaslocalhost/todo/createorlocalhost/createtodo.NowputtheverbonPOST.OnceyouputtheverbonPOST,theBodytabshouldbecomeavailable.OntheBodytab,youcanspecifythetypeandpayloadofthebody.Weneedtherawtypeand,fromthedropdownthatappears,youneedtoselectJSON(application/json).ThiswillactuallyaddaContent-TypeheaderontheHeaderstab.Wecannowspecifyourpayload,whichmustbeavalidJSON:
{
"description":"AddanewitemtotheTODOlist."
}
Ifyouhavedoneeverythingcorrectly,thisiswhatitlookslike:
Afterthat,wecanupdatetheitemwejustinserted.PuttheverbonPUTandsetidandanewdescriptionandstatusinyourbody:
{
"id":3,
"description":"AddanewitemtotheTODOlistusingPostman.",
"status":"Done"
}
Youcancheckwhethereverythingisasyouexpectittobebyre-runningyourGETrequestfromyourhistory.
Andlast,wearegoingtodeletetheitem,simplybyswitchingtheverbfromthePUTrequesttoDELETE.Youcanremovethedescriptionandstatusfromyourrequestbody,butleavingthemtherewillnothaveanyeffects,either.
ThatisreallyhoweasyPostmancanbe.ItissmallandintuitiveandthatiswhatIlikeaboutit.
WritingtestsNow,let'swriteacoupleoftestsforourAPI.AsImentionedintheintroduction,wecanwritetestsusingJavaScript.GetyourGETlocalhost/todo?id=1fromyourhistoryandselecttheTeststab(intherequest,nottheresponse).Youwillseeacoupleofsnippetsontheright-handsidethatyoucanclickafterwhichtheywillbeaddedtoyourscript.Forexample,wecanusetheResponsebody:Containsstringsnippetandchangeitalittlesowecanverifythatsomewhereinourresponsewegetid1:
tests["Bodycontainsid1"]=responseBody.has('"id":1,');
Weneedtotestfortherawresponse(whichhasnospacesornewlines)ratherthanthepretty,printedresponse(whichhasspacesandnewlines).Whenyousendtherequestnow,Postmanwillrunyourtestsaswell.YoucanseethestatusofyourtestsintheTeststabofyourresponsesection.
Wecanmakethistestbetter,though.WeshouldbeabletoparseourresponsetoJSONandthencheckwhethertheidpropertyisequalto1.Inthatscenario,wealsotestwhetherwearenotgetting,forexample,anarrayinourresponse.WecanusetheResponsebody:JSONvaluechecksnippet:
varjsonData=JSON.parse(responseBody);
tests["Idequals1"]=jsonData.id===1;
AndnowthatwehavetheJSONdataasaJavaScriptobject,wecaneasilytestallkindsofthings,suchasthestatus.
Wecanalsotimeourcall.Whilethatmaybegoodpractice,Ihavefounditveryhardtotest.Ideally,youwantyourrequesttobehandledasquicklyaspossible,butthatalsodependsonyourserverandhowbusyitis.Forexample,Iamgettingresponsetimesof500to600milliseconds,whichisactuallyabitslow(onlyonthefirstrequest,though).Ioncehadsuchatestonacrappytestserverandtheresponsetimesrangedfromlessthan200millisecondstooverasecond,dependingonwhatelsethatserverwasdoing(likerunningotherJenkinsjobs).However,wecantestthatitnevertakeslongerthanasecond.ThereisaResponsetimeislessthan200mssnippet:
tests["Responsetimeislessthanasecond"]=responseTime<1000;
AnotherusefultestistochecktheHTTPstatusthatisreturned.WecanusethesnippetStatuscode:Codeis200asis:
tests["Statuscodeis200"]=responseCode.code===200;
Endtherequestandcheckwhetheryourtestssucceed(Ihavemadeonefaildeliberately):
Asyoucansee,itisquiteeasytowritetestsusingPostman,asitisjustJavaScript.Thesnippetshelp,too.However,therecomesatimewhenyourtestsfailonsomeunexpectedvalue.Maybeaninvalidcastoranullreference.Atsuchtimes,itcanreallycomeinhandyifyouhavesomeextratoolstoinspectvaluesofobjects.Postmanhasjustthat.Inthetopmenu,gotoViewandthenShowPostmanConsole(shortcutkeysCmd+Alt+CorAlt+Ctrl+ConWindows).Thisopensuptheconsoleandshowsyourrequests,responses,andanythingyoulogtotheconsole:
varjsonData=JSON.parse(responseBody);
console.log(jsonData);
Theconsoleshouldshowyouwhat'sinthejsonDataobject:
TestingXMLPersonally,IamnotafanofXML.LikeLinusTorvalds,creatorofLinux,oncesaid:XMLiscrap.Really.Therearenoexcuses.XMLisnastytoparseforhumans,andit'sadisastertoparseevenforcomputers.There'sjustnoreasonforthathorriblecraptoexist.However,forsomereason,therearestillpeoplewhojustloveXML.Inmyexperience,itisespeciallyenterprisebusinesseswhorelyonXMLalot.So,whetheryoulikeitornot,youwillprobablyhavetoworkwithXMLatsomepointduringyourcareer.WhenworkingwithXML,theearliermentionedSoapUIisprobablyyourgo-toAPItestsolution.However,PostmancantestXMLjustfine.So,let'ssetupalittletestusingXML.WecaneasilyturnourGETmethodintoanXMLvariant.
First,weneedanXMLparser,likejs2xmlparser:
npminstalljs2xmlparser--save
Afterthat,wecancopy/pastetheGETmethodandsendXMLbackinsteadofJSON:
varjs2xmlparser=require('js2xmlparser');
app.get('/xml/todo',function(req,res){
if(req.query.id){
varitem=todos.find(t=>t.id===+req.query.id);
if(item){
varxml=js2xmlparser.parse('todo',item);
res.setHeader('content-type','text/xml');
res.end(xml);
}else{
res.status(500).send('Nosuchtodoitem.');
}
}else{
varxml=js2xmlparser.parse('todo',todos);
res.setHeader('content-type','text/xml');
res.end(xml);
}
});
So,weparsetheobjecttoXMLusingjs2xmlparser.parse.Afterthat,wesetthecontent-typeheadertotext/xmlandthenwesendtheXMLbackusingres.end.
ItisnotveryhardtotestXMLinPostman,butthereareafewgotchas.WecantestthismethodusingtheURLlocalhost/xml/todo?id=1.Wecanusethesametests
asbefore,sowecheckwhetheridreallyequals1.First,wemustparseourresponsetoJSONusingxml2Json.Afterthat,youshouldrememberthatXMLhasaparentnode,(inourcase,thatistodo),sowegetanobjectwithatodoproperty:
varjsonData=xml2Json(responseBody).todo;
Also,JSONhasstringsenclosedindoublequoteswhilenumericsarekeptastheyare.XMLhasnosuchthing,soeverythingbecomesastring.Becauseofthis,wemustcastouridpropertytoanumericbeforewecantestitforvalueandtypeequalitywith===:
varjsonData=xml2Json(responseBody).todo;
tests["Idequals1"]=+jsonData.id===1;
Otherthanthat,itisreallyprettymuchthesameaswithregularJSON.Ifyouarehavingproblemsfindingouthow,exactly,PostmanparsedyourXMLtoJSON,youcanalwaysusethePostmanConsole.
CollectionsNowthatwehavearequestwithsometests,weprobablywanttosaveitsomewhereandalsoaddotherrequestswiththeirowntests.Youcansavemultiplerequestsinacollection.SimplyhittheSavebuttonnexttotheSendbutton.Youwillhavetospecifyyourrequestname,whichdefaultstotheURL,butyoucannameitsomethinglikeGetTODO1.Youcanaddanoptionaldescription,suchasGetsTODOitem1andtestswhetherwegotthecorrectitemback.Andfinally,youhavetoeithersaveittoanexistingcollectionorcreateanewcollection.WearegoingtocreateanewcollectionnamedWebAPI.HitSaveandyourcollectionwillbesaved.YoucanviewitintheCollectionstabontheleft-handside(ifyourscreeniswideenough).CollectionsarenotfullysupportedonlyifyouhaveaPostmanaccountinthefreeversion,andtherearealsosomefeaturesthataresupported,suchasmockingandpublishingdocumentation.Whetheryouwantanaccounttoplaywiththesefeaturesisuptoyou.Forthischapter,youdonotneedanaccount.Otherthanthat,youcan,ofcourse,edityourcollectionandyourseparaterequestsandyoucanrunyourcollections.
ThereisalreadyapreinstalledcollectionnamedPostmanEchothatyoucanuseasanexample:
Whenyourunyourcollection,PostmanwillopenanewscreencalledaCollectionRunner.Inthisrunner,youcanspecifythecollectiontorun(whichisnowdefaultedtoWebAPI)howoftenyouwanttorunit,whatenvironmenttouse,andsomeotheroptions.Ontheright-handsideofthescreen,youcanfindahistoryoftestruns:
JusthittingStartRunwillrunyourrequestsandpresentyouwiththeoverallstatusofyourtests:
Now,let'saddsomeadditionalteststothecollection.First,let'saddthePOSTactionwithandwithoutadefaultstatus.Youknowhowtodotherequests,solet'sfocusonthetestscripts.Inthefirstrequest,wearenotgoingtogivethenewitemadefaultstatus,soweshouldcheckforthegivendescriptionandstatusNew:
varjsonData=JSON.parse(responseBody);
tests["Descriptionequalsrequestdescription"]=jsonData.description==="AddanewitemtotheTODOlist.";
tests["Statusequals'New'"]=jsonData.status==="New";
tests["Responsetimeislessthanasecond"]=responseTime<1000;
tests["Statuscodeis200"]=responseCode.code===200;
SavethisrequestandnameitCreateTODOdefaultstatus.Thistime,donotcreateanewcollection,butaddittoourexistingcollection.
ItwouldbecoolifwecoulddoanotherGETnext,tocheckwhethertheitemwasreallyaddedtothelist.Wecandothisusingeitherglobalvariablesorenvironmentvariables.Bothglobalandenvironmentworkprettymuchthesame,exceptthatwecurrentlydonothaveanenvironment.Solet'sgowithglobalfornow.SomewhereinyourPOSTtestscript,youcansetid:
postman.setGlobalVariable("createdId",jsonData.id);
Youcannowusethisvariableprettymucheverywhere.Touseit,usethe{{variable_name}}syntax.So,createanewGETrequestwiththeURL
localhost/goto?id={{createdId}}.Inthetestscript,youcannowalsoverifythatyoureallygetthecorrectIDfromtheserver:
varjsonData=JSON.parse(responseBody);
tests["Idequals"+globals.createdId]=jsonData.id===+globals.createdId;
tests["Descriptionequalsrequestdescription"]=jsonData.description==="AddanewitemtotheTODOlist.";
tests["Statusequals'New'"]=jsonData.status==="New";
tests["Responsetimeislessthanasecond"]=responseTime<1000;
tests["Statuscodeis200"]=responseCode.code===200;
Justkeepinmindthatallyourglobalvariablesarestoredasstrings.Intheupper-rightcornerisabuttonwithaneyesymbol.Clickingthisbuttonletsyouview,add,delete,andchangeglobalvariables.Beawarethatyourtestsarenowdependentuponeachother,though.YouneedtorunthePOSTbeforeyoucanruntheGET,becausePOSTsetsthevariablethatGETneeds.AddthenewGETrequestwithteststoyourcollectionaswell.Ifyourunyourcollectionnow,youshouldhavethreerequestswithatotaloftwelvetests.
YoucancreatesomethinglikethatforthePOSTwithadefaultstatusaswell.Youcaneasilycopyrequests,bytheway.Justclickthethree-dottedmenunexttoarequestinyourcollectionandchooseDuplicate.Therequestwillbeaddedtoyourcollectionrightaway.Hereisanothertip;donotforgettosave!Youmaybeusedtoeditorsauto-savingor,atleast,savingwhenyouhitrun/debug.Postmandoesnosuchthing.Youreallyhavetoexplicitlyhitthesavebutton.Annoying,butthatishowitworks.
WhenwehaveourtwoPOSTandtwocorrespondingGETrequests,wearemovingonwiththePUT,orupdate,request.Youcanusetheglobalvariableintherequestbody:
{
"id":{{createdId}},
"description":"AddanewitemtotheTODOlistusingPostman.",
"status":"Done"
}
Thetestscriptisalmostexactlythesameasbefore,saveforourupdates,ofcourse:
varjsonData=JSON.parse(responseBody);
tests["Idequals"+globals.createdId]=jsonData.id===+globals.createdId;
tests["Descriptionequalsrequestdescription"]=jsonData.description==="AddanewitemtotheTODOlistusingPostman.";
tests["Statusequals'New'"]=jsonData.status==="Done";
tests["Responsetimeislessthanasecond"]=responseTime<1000;
tests["Statuscodeis200"]=responseCode.code===200;
YoucanthencreateanotherGETrequesttomakesureeverythingwasreallyupdated.Afterthat,wecreateaDELETErequest.ThebodyisalmostthesameasthePUTrequest:
{
"id":{{createdId}}
}
Andthetestscriptisexactlythesame.
Lastly,wewanttotestifthedeleteditemwasreallydeleted.Thisisinteresting.Weexpect404,nosuchitemfound:
tests["Responsetimeislessthanasecond"]=responseTime<1000;
tests["Statuscodeis404"]=responseCode.code===404;
However,whenwenowrunourtests,thisonefails.Instead,wegetanemptyresponseandastatuscodeof200.So,weshouldreallychangeourcodeanditwillreturnwhatweexpect.Theproblemisthatwejusttrytofindtheitemandreturnwhateverisfound,evenifwedonotfindanything.So,weshouldchangethecodetoreturna404whenwehavenosuchitem:
app.get('/todo',function(req,res){
if(req.query.id){
varitem=todos.find(t=>t.id===+req.query.id);
if(item){
res.json(item);
}else{
res.status(404).send('Nosuchtodoitem.');
}
}else{
res.json(todos);
}
});
Icouldthinkofafewmoretests,suchasthe404withPUTandDELETEandatestfortheentirelist,butlet'skeepitatthis.Wereallyhavemorethanenoughteststomakeaninterestingexample.
WhenyouhaveanAPIwithmanyendpoints,youmaywanttoorganizeyourcollectionsabitbetter.Itispossibletocreatefoldersinyourcollectionsandthenfoldersinthosefolders.Usingfolders,itiseasiertokeepallyourteststogether.Forexample,youcan
haveatodofolderthattestsallyourcallstolocalhost/todoandyoucanhaveausersfolderthattestsallcallstolocalhost/users.
EnvironmentsIhavealreadymentionedenvironments,andnowwearegoingtotakeacloserlookatthem.Environmentsareaninvaluabletoolwhentesting.Forexample,allourURLsarenowtargetinglocalhost,butwhenwearegoingtorunourAPIonaserver(test,production,whatever),weprobablywanttorunourtestsonthatserver,too.ManuallychangingallURLsisnotreallyanoption,especiallywhenyouhavelotsoftests.Thisiswhereenvironmentscomein.Anenvironmentisbasicallyasetofvariableslikeglobalvariables,exceptenvironmentscanbeeasilyinterchanged,whileglobalvariablescannot.YoucanmanageyourenvironmentbyclickingthegearbuttoninthetoprightcornerofPostmanandclickingtheManageenvironmentsbutton.
Intheenvironmentsmanagement,simplyclicktheAddbuttontoaddanenvironmentandnameitLocal.Inyourenvironment,addakeybaseUrlandavaluelocalhost.YoucannowchangeyourURLsto{{baseUrl}}/todo.Postmanwillprobablystillgiveyouawarningthatthereisnosuchvariableinthecurrentenvironment.SomewhereintheupperrightcornerisadropdownthatsaysNoEnvironment,butyoucannowselectyourLocalenvironmentfromthedropdownandeverythingwillworkagain.
Youcannoweasilycreateanewenvironment(forexample,Test)andalsospecifythekeybaseUrl,butwiththevaluetest.mycompany.com.Nowallyouhavetodototargetyourcompany'stestserverischangetheenvironmentintheupperrightcorner.WecannowalsouseourenvironmentforthecreatedIdvariableinsteadoftheglobals.InyourPOSTtestscripts,changethesetGlobalVariablefunctiontosetEnvironmentVariable:
postman.setEnvironmentVariable("createdId",jsonData.id);
Andinyourothertestsscripts,youreplaceglobals.createdIdwithenvironment.createdId:
tests["Idequals"+environment.createdId]=jsonData.id===+environment.createdId;
Again,bothglobalandenvironmentvariablesarestoredasstrings,sobesuretouseJSON.stringifyandJSON.parseonarraysandobjectsifyouwanttostorethose.
NewmanNowthatwehaveourPostmantests,completeinacollectionwithenvironments,weprobablywanttoautomateourtests.InJenkins,thisworksprettymuchthesameasourSeleniumtests;makesureyourwebserviceisrunningsomewhereandrunthetestsusingthecommandline.TheproblemisthatPostmanisadesktopapplicationandwecannotrunitfromthecommandline.NotwithoutNewman,thatis(https://github.com/postmanlabs/newman).YoucaninstallNewmanusingnpm:
npminstallnewman--save-dev
npminstallnewman-g
BeforewecanuseNewman,wemustdotwothings:exportourcollectionandexportourenvironment(s).Wecanstartwiththecollection.GotoyourcollectionsinPostmanandclickthebuttonwiththreedots.Fromthemenu,chooseExport.Youwillnowgetapopupaskingifyouwanttosaveasv1orv2;choosev2(asrecommendedbythepopup).Ihavesavedthefileintheweb-api\testsfolderandIhavechosenthenamesuggestedbyPostman,WebAPI.postman_collection.json.
Next,wecanexportourenvironment.InPostman,gotoyourenvironments.Everyenvironmenthasfouroptions:Share(aProfeature),Duplicate,Export,andRemove.So,chooseExportandsavetheenvironmentwithyourcollection.Again,IhavekeptthenamesuggestedbyPostman,Local.postman_environment.json.
ThecollectionandenvironmentfilesarejustJSONfiles,soyoucanviewandedittheminanytexteditor.Italsoworksgreatwithyoursourcecontrol,becauseyoucaneasilyseewhochangedwhatandwhen.
Youcanalsoimportyourcollectionsandenvironment.ThecollectionimportbuttonisatthetopleftinPostman,thebigbuttonthatsaysImport(nexttoRunner).Youcanimportenvironmentsthroughtheenvironmentmanagementwindow.Thatway,youcanimport,edit,andexportyourcollectionsandenvironmentondifferentcomputers(orinateam).
NowthatwehaveourcollectionandenvironmentinJSONfiles,wecanrunthemusingNewman:
cdweb-api\tests
newmanrun"WebAPI.postman_collection.json"-eLocal.postman_environment.json
The-evariableisshortfor--environment,ofcourse.Itreallyisthateasy:
Createanerrorinyourcodeandverifythatitworks:
>
Oneofthemostimportantthings,especiallywhenyouaregoingtousePostmanandNewmaninJenkins,isyourreporting.Andasweknow,JenkinsprefersJUnitformatreports.Luckilyforus,Newmanhasthisfunctionalitybuiltin.Itisjustamatterofaddingthe--reportersoptionstothecommandline:
newmanrun"WebAPI.postman_collection.json"-eLocal.postman_environment.json--reporters"cli,junit"--reporter-junit-exportPostmanResults.xml
Weneedtospecifythe--reporter-junit-exportoptionsforthejunitreporter.Theclireporterisjustthedefaultconsoleoutput.AddingthiswillcreateaJUnitstyleXMLfilenamedPostmanResults.xmlinyourcurrentfolder.YoucanusethisreporttopublishinJenkins.
Therearesomeadditionalparametersyoumaywanttouse.Youcanspecifyafilewithyourglobalvariables.ThisisalsoaJSONfile.YoucanexportitfromPostman(butforsomereason,youcannotimportit,onlycopyandpasteit).Youcanusethe-gor--globalparametersforyourglobalfile.
Anotherusefulparameteris-n,or--iteration-count,forwhenyouwanttorun
somestresstests.
Becausesomeplatformsdonotdisplaycolorsproperly,youcanuse--no-color.Tonothaveanyoutputatall,use--silent.Tonotgetblacklistedbyyourserver(orjustplainDDoSit),youcansetadelaybetweenrequests(inmilliseconds)using--delay-requestandtonotwaittoolongfortheresult,youcansetatimeoutwith--timeout-request.
YouknowhowtogetthisworkinginJenkins.RuntheAPIusingPM2andthensimplypostthecommandinaWindowsbatchscriptoraLinuxshellscript.BecausewealreadyhaveanentireNode.jswebsite,wearenotgoingtoreproducethosesteps,andwearealsonotgoingtodoanythingelsewiththisexample.So,thiswasitforbothPostmanandNewman.
SummaryPostmanandNewmanareinvaluabletoolsintestingyourwebAPIs.Iprobablydonothavetomentionthis,buttheycanbeusedtotestyourregularwebapplicationsaswell.Inthelasttwochapters,wearegoingtoreleaseourwebshopandalso,finally,runourSeleniumtestsfortheC#application.
ContinuousDeliveryWearefinallygettingintothenextportionofthisbook,ContinuousDeliveryandDeployment.But,beforewecontinue,let'shaveabriefrecapofwhatwehavelearnedsofar.Whenwestartworkingonanewproject,thefirstthingweneedissomesourcecontrolrepository.Inthisbook,wehaveusedGit.UsingGit,wecancreatemultiplebranchestokeepdifferentenvironmentsandfeaturesapart.Inthischapter,wearegoingtoseewhythisissoimportant.Whilewewritethesoftware,wewanttocreateautomatedtests,suchasunittests,E2Etests,anddatabasetests.Themoreusefultestsyouhave,themorereliableyoursoftwarebecomes.UponeachGitcommit,wewanttoautomaticallytestoursoftwareusingcontinuousintegrationsoftware,inourcase,Jenkins.Whenourtestsand,optionally,staticcodeanalyzerssuchasJSLintandSonarQubepass,weknowoursoftwareisprobablyinprettygoodshape.Sogoodashape,infact,wecanreleaseoursoftwaretothecustomer.
ThewayIseeit,webasicallyhavethreetypesofdeployment:manualorautomatedand,whenautomated,thereisdeploymentthatdependsonhumaninteractionandthereisfullautomaticdeploymentthatrequiresnofurtherhumanintervention.TheautomateddeploymentthatstillneedssomehumaninteractioniswhatwecallContinuousDelivery.Basically,whenyouhaveyourreleasefilesreadytobedeployedatanytime,youarepracticingContinuousDelivery.Inthischapter,wearegoingtosetupanenvironmentusingContinuousDelivery.Thenextstep,continuousautomateddeployment,orsimplyContinuousDeployment,willbediscussedinthenext,andfinal,chapter.
Let'stakeanotherlookatthepicturefromChapter1,ContinuousIntegration,Delivery,andDeploymentFoundations,butslightlydifferent,tovisualizethedifferencebetweenContinuousDeliveryandDeployment:
AsyoucanseeContinuousDeliveryandContinuousDeploymentendwiththesoftwarebeingdeployed,themaindifferenceisthatthedeploymentisstillmanuallytriggered(andsometimesmanuallyexecutedaswell)inContinuousDelivery.Thetakeawayisthis,withcontinuousdeliveryweshouldalwayshaveareleasepackagethatisreadytobedeployedatanytime,butthedeploymentstepitselfisstillmanualormanuallytriggered.Whetherthatdeploymentstepiscopyingsomefiles,clickingabuttonorclickingafewbuttons(well,nottoomuchIguess)doesn'tmatterallthatmuch.ContinuousDeliveryalwaysrequiresamanualtrigger.Thedifferencewith"traditional"deployment,however,isthatthetriggercantakeplaceatanytimebecausewehavethatreleasepackagereadyandupdatingshouldbeeasy.Oftenaseasyasclickingabutton!Inthischapterwearegoingtostartwithafullymanualdeploymentandmakesurewecaneasilyreleaseoursoftwareatanytimeafterthatfirstsetup.Inthenextchapterwearegoingtocompletelyautomatethisprocessandmakethesteptocontinuousdeployment.Attheendofthischapterwe'llbeleftwithareleasepackagethatcanbemanuallydeployedatanytime,butafterthenextchapteritshouldn'tbetoodifficulttoimplementasinglebuttonclickstrategyforyourcontinuousdeliveryprocess.
BranchingWehavediscussedbranchingusingGitinChapter3,VersionControlwithGit.Wehavediscussedhowyourcodeshouldgofromacommitonadevelopmentenvironmenttoatestenvironment,beitautomaticallytestedandmanuallytestedifnecessary;thenanacceptanceenvironmentwherethecustomercanhavealookatitandfinally,aproductionenvironmentwherecustomerscanusethesoftwareprettymuchfullyautomated.So,youwillhavetosetupacompleteDTAPstreet(Development,Test,Acceptance,andProduction),butstillbeabletodifferentiatebetweenthemall.
Perhapsyoumayhavemoreorfewerenvironments,butyoustillwanttoknowwhatcommitsareonwhatenvironment.Wearegoingtosetuponeofthoseenvironmentsinthischapter.First,wearegoingtolookattheprocessofbranchingyoursoftwaresoitcanbedistributedtothedifferent(fictional)environments.
Wewantourcodetogofromourdevelopmentenvironmenttoourtestenvironment.Anycommitcanbedeployedtotestassoonasitisavailable.Afterall,testiswherewetestifoursoftwareisreadyfortherealworld.So,wecanhavethisfullyautomated.WecancreatenewGitbranchesnow,AcceptanceandProduction.Ultimately,wewantanenvironmentthatlookssomethinglikethefollowingscreenshot:
Inthisbranchingmodel,wehaveourlocalmasterbranch,whichweusefordevelopment.Everythingfrommasterisdeployeddirectlytoourtestserverandtested.Whenthetestssucceed,wecanmanuallymergemasterintoAcceptance(inthisexample,Iusedafast-forward).AcommittotheAcceptancebranchwilldeployeverythingtotheacceptanceenvironment,wherethecustomercantestwhethertheyaresatisfiedwiththechanges.Whenthecustomerissatisfiedwith
thechanges,wecanmergetheAcceptancebranchintotheProductionbranchand,again,acommittoProductionwilldeploythesoftwaretotheproductionenvironment.Ifeverythinggoeswell,everything,exceptthecommitsandmerges,happensinafully-automatedwayandthereislittletonodowntime.
Unfortunately,nomatterhowwelleverythingistested,therewillbebugsonproduction.Itcanbeatechnicalbug,butitcanalsobesomefunctionalbug--somethingeveryoneoverlooked.
Whateverthecasemaybe,thebugneedstobefixed.ItisnoweasytobranchfromtheProductionbranch,fixthebug,andmergethefixintomaster,Acceptance,andProduction(inwhateverorderisappropriategiventhesituation).
Theexampleisabithardtoread,butyoucanclearlyseethattheBugfixbranchwasbranchedfromProduction,thenmergedintomaster,thenAcceptanceand,aftercustomerconfirmation(implied),mergedintotheProductionbranch.Meanwhile,Productionisstillonversion1.0.0andthefixdidnothavetowaitforversion1.1.0,orworse1.2.0,tobeacceptedanddeployed.SincetheBugfixwasalsomergedtoAcceptanceandmaster,itisfixedinallthecurrentversionsandwecandeletetheBugfixbranch.
Wearenowgoingtocreateourownbranchtofinallyputthistopractice.WeneedtopickeithertheJavaScriptprojectortheC#.NETCoreproject.Youmaydoboth,ofcourse.SinceourJavaScriptprojectalreadyhasacompleteJenkinsfiles,let'sgowiththatone.ItiscustomtokeepyourmasterbranchforyourproductioncodeandcreateaseparateDevelopmentbranch.However,Ihavefoundthatpeopleoftencheckoutthemasterbranchandstartworkingonthat,soyoumaywanttokeepthemasterbranchasyourDevelopmentbranchandcreateaseparateProductionbranch.Whateveryoudo,itisallaboutagreeingwithyour
teamonwhatbranchesyoucancommitchanges.Nowthatwehavetwobranches,wearegoingtocreateseparateJenkinsfilesforeachbranch.WehavealreadyexploredthepossibilitiesforthisinChapter11,JenkinsPipelines.Iamgoingtohaveonebranchbuildandtestthesoftware,andtheothermakeadeployablepackage.Youshoulddecidewhetheryouwanttotestthesoftwareonyourproductionbranchagain.Intheory,itisnotnecessarybecauseithasbeenalreadytested.So,twobranches,twoseparateJenkinsfiles.
TheoneonDevelopmentdoesthebuildingandtesting:
pipeline{
agentany
options{
gitLabConnection('LocalGitLab')
buildDiscarder(logRotator(numToKeepStr:'5',artifactNumToKeepStr:'5'))
}
triggers{
gitlab(triggerOnPush:true,branchFilterType:'All')
}
stages{
stage('Checkout'){
//...
}
stage('Build'){
//...
}
stage('TestNodeJS'){
//...
}
stage('Test'){
//...
}
stage('SonarQube'){
//...
}
}
post{
//...
}
}
AndthemasterJenkinsfilecreatesthedeployablepackage:
pipeline{
agentany
options{
gitLabConnection('LocalGitLab')
buildDiscarder(logRotator(numToKeepStr:'5',artifactNumToKeepStr:'5'))
}
triggers{
gitlab(triggerOnPush:true,branchFilterType:'All')
}
stages{
stage('Checkout'){
//...
}
stage('Build'){
//...
}
stage('ArchivingArtifacts'){
//Weonlyneedproductionpackageshere.
sh'rm-rfnode_modules'
sh'npminstall--production'
archiveArtifacts'index.js,config.*.js,prod/,node_modules/'
}
}
post{
//...
}
}
Withmultiplebranches,youwillquicklyfindthatJenkinsrunsoutofavailableexecutorsandthatyournodeshavetowaitforeachotherandthatyouhavegotyourselfaJenkinsdeadlock.IhavementionedthisearlierinChapter11,JenkinsPipelines,buttheonlywaywecanreallyfixthisisbyhavingalightweightmasternodethatonlyrunsthescripts,butdoesalltheactualworkonseparatenodes.Becausesettingupsuchanenvironmentisalotofworkforusnow,wecanminimizethedamagebydisablingconcurrentbuildsintheJenkinsfile,whichisgoodpracticeanyway:
options{
[...]
disableConcurrentBuilds()
}
Also,keepinmindthatweusedacustomworkspaceforourbuild.Theseparatebranchesnowusethesamefolder,whichcanhaveunexpectedsideeffectssuchasfilesbeinglockedordeletedbytheotherbuild.Ofcourse,wecanfixthisaswell:remember,weknowwhatbranchwearecurrentlybuildingon.Simplycreateanenvironmentvariableanduseitinallinstancesofws.Youcouldappendenv.BRANCH_NAMEinthewsnodesdirectly,butbyusinganenvironmentvariable,youkeepyourcodealittlecleaner.Notethatthe${something}syntax(calledstringinterpolation)onlyworkswhenyouusedoublequotes(ifyouusesinglequotes,yourworkspacewillliterallybenamed...${env.BRANCH_NAME}):
pipeline{
[...]
environment{
ws="web-shop-pipeline-${env.BRANCH_NAME}"
}
[...]
ws(dir:env.ws){
YoucanfindthefullJenkinsfilesintheGitrepositoryforthebookonthesamebranch.IhavenamedthemJenkinsfile(master)andJenkinsfile(Development).
Wheneverythingisstableandyourwebsiteisrunning,trycommittingsomethingtoyourDevelopmentbranchthatwillbreakthebuild.Forexample,achangeinsomeHTMLfile,suchasindex.ejs.Breakthebuildbyomittingaclosingbracket(indiv)andchangethetextbyaddingAnyproduct!,soyouknowwhetherthechangewasdeployed:
<divclass="col-lg-12"
<h2>Searchforproducts...Anyproduct!</h2>
</div>
Whenyoucommitthischange,youwillfindtheDevelopmentbranchinJenkinsstartsbuildingandfails.Wecanstillmergethistomaster(whichwillalsofail,butonlybecausewedidnotbreakanactualunittest),butwewouldhavetobestupidtomergeabrokenbuild.Whatisimportanthereisthatourmasterbranchisstillcorrectandreadyfordeploymentatanytime.Jenkinswillsendanemailtous(andourteam)tosaythatsomethingisnotwellintheDevelopmentbranchandwecanfixourerror.Onceitisfixedandthebuildsucceeds,wecanmanuallymerge(orcherry-pick)tomasterandJenkinswilldeploythenewsoftware.
ManualdeploymentBeforewecanautomaticallydeployoursoftware,wemustfirstmanuallydeployoursoftware.Or,atleast,makesureeverythingisinplacebeforewecanautomateit.Deployingisabithardbecausewedonotreallyhaveanythingtodeployto,butatleastwecandeploytoeitherourLinuxVMorourWindowshost.
Inthischapterwearegoingtoworkwithsomedependenciesthatarerequiredtorunoursoftware,likeNGINXandourdatabasesoftware.Theinstallationandconfigurationofsuchtoolscanbeautomated,butthisisoutsidethescopeofthisbook.Youwillalsooftenfindotherteams,suchasnetworkadministrators,areresponsiblefordeliveringnecessary(virtual)serverswiththenecessaryinstalledsoftware.
InstallingNGINXThefirstthingweneedtodobeforewecandoanythingelseisinstallNGINX(https://www.nginx.com/).NGINXisanHTTPserver,reverseproxy,andIMAP/POP3proxyserver.WithNGINX,wecanhostourNode.jsapplicationusingPM2anduseNGINXasareverseproxytomakeitavailabletoourhostWindowssystem.Wecanalsorunour.NETCoreapplicationusingNGINX.
So,onyourLinuxVM,installNGINX:
sudoapt-getupdate
sudoapt-getinstallnginx
sudosystemctlrestartnginx
sudosystemctlenablenginx
Then,installNGINXandmakesurethatitgetsstartedonstartup.Iamgoingtodoafast-forwardhere;wearegoingtorunPM2onport8889internallyandmakeitavailableonport8888externally.Wearegoingtodothesameforour.NETapplicationusingports9998and9999.So,whatwehavetodoisconfigureNGINXsoitreroutes8889to8888and9998to9999.Wecanaddthatinthefile/etc/nginx/sites-available/default:
cd/etc/nginx/sites-available
sudovidefault
Now,inthedefaultfile,addthefollowingnodesatthebottom:
server{
listen8888;
server_name_;
location/{
proxy_passhttp://127.0.0.1:8889;
proxy_http_version1.1;
proxy_set_headerUpgrade$http_upgrade;
proxy_set_headerConnection'upgrade';
proxy_set_headerHost$host;
proxy_cache_bypass$http_upgrade;
}
}
server{
listen9999;
server_name_;
location/{
proxy_passhttp://127.0.0.1:9998;
proxy_http_version1.1;
proxy_set_headerUpgrade$http_upgrade;
proxy_set_headerConnection'upgrade';
proxy_set_headerHost$host;
proxy_cache_bypass$http_upgrade;
}
}
Then,restartNGINX.Youmayhavetorestartyourserverbeforethechangescantakeeffect(becausewejustinstalledit):
sudosystemctlrestartnginx
sudoreboot
ThissetsupNGINX.Now,ifyoubrowsetoyourserverports8888and9999,youshouldseeaNGINX502BadGatewaypage.
Node.jswebshopNext,wewanttorunourJavaScriptwebshop.WewillneedatleastPM2,whichwealreadyusedtorunourSeleniumtestsinJenkins.Thistime,wewillneedtoinstallPM2globally:
sudonpminstallpm2-g
Theeasiestwaytotestifeverything,includingNGINX,worksisbygoingdirectlyintoourJenkinsworkspaceandstartingourindex.jsfilefromthere.IamassumingyouhavetheentirepipelineexampleonyourLinuxVM:
cd/var/lib/jenkins/web-shop-pipeline
pm2startindex.js--name=webshop-js----port=8889
pm2ls
Trybrowsingtociserver:8888inyourWindowshostagainandyoushouldnowseeyourwebshop!
Ofcourse,wedonotwanttorunourapplicationfromaJenkinsfolder.Instead,weshouldgetourfilesfromJenkinstosomewhereelseonourmachine.Luckily,wehaveourartifactsinJenkins.Simplydownloadthegeneratedzipfile;itshouldhaveallthefilesweneed(andquiteafewunnecessarynode_modules--youcanremovethemandusenpminstall--productionifyouwantto).WenowneedtogetthefilesfromourhosttoourVM.WecandothisusingWinSCP(https://winscp.net/eng/index.php).Justheadovertothedownloadspageandgettheinstallationpackageofthelatestversion.Installitandgoforthedefaultoptions.
WhenyouopenWinSCP,youwillbepresentedwiththeloginscreen.SimplyloginwiththeSCPprotocol(SecureCopyProtocol,https://en.wikipedia.org/wiki/Secure_copy),whichusesSSH(SecureSHell,https://en.wikipedia.org/wiki/Secure_Shell)tocopyfiles.Youcansaveyoursessionsoyouhaveashortcutnexttimeyouneedtoconnect.
Onceyouareloggedin,youwillseeasplitscreenshowingyourlocalcomputerontheleft-handsideandyourremoteLinuxVMontheright-handside.UsingWinSCP,browseto/var/wwwonyourVMandcreatetwofolders,webshop-jsandwebshop-net.Ontheright-handside,browsetoyourunpackedJenkinsarchive.
Youmaybewonderingabouttheconfigsontheleft-handside--Iwillgettothatinaminute.Fornow,itwillsufficetodragallyourfilesfromtheleft-handside
intothewebshop-jsfolderontheright-handside.Unfortunately,youwillbepresentedwithaPermissionDeniedexception.Youwillneedtogiveyourusersufficientprivilegestowritetothefolder.
GointoyourVM(theactualone,notWinSCP)andmakeyourusertheownerof/var/www/webshop-js:
susander
sudochown-Rsander:sander/var/www/webshop-js
ls-l/var/www
NoticethatIamusingmyownadminusertochangetheownerofthefoldertotheadminuser.Now,trycopyingyourarchiveagainusingWinSCP.Ifyouhavenotremovedyourunnecessarynode_modules,thiswilltakeawhile.Evenwithouttheunnecessarynode_modules,itwilltakeaboutaminute.
If,forwhateverreason,youfailtogetyourfilesfromyourhosttoyourVM,youcanmanuallycopytheminyourVMfromyourJenkinsworkspaceto/var/www/webshop-js:
cd/var/lib/jenkins/web-shop-pipeline-master/artifacts
sudocp-rf{index.js,config.*.js,node_modules,prod/css,prod/scripts,prod/views}/var/www/webshop-js
PM2ThelastthingweneedtodoismakesurethatPM2,andourwebshop,startsautomaticallywhenyourVMstarts.PM2hasabuilt-incommandforjustthat,startup.Runningthiscommandwillgenerateacommandthatyouhavetorun.Justcopyitandrunit:
pm2startupsystemd
[PM2]Youhavetorunthiscommandasroot.Executethefollowingcommand:
sudoenvPATH=$PATH:/usr/bin/usr/lib/node_modules/pm2/bin/pm2startupsystemd-u[username]--hp/home/[username]
sudoenvPATH=$PATH:/usr/bin/usr/lib/node_modules/pm2/bin/pm2startupsystemd-u[username]--hp/home/[username]
systemdisforUbuntu16andhigherandCentOS,Arch,andDebian.Thereareothersupportedsystems,suchasDarwin,MacOSx,Gentoo,andFreeBSD.Itisprettywelldocumentedactually(http://pm2.keymetrics.io/docs/usage/startup/).Alternatively,youcanomitthesystemvariableandletPM2figureitout.
WecannowrunsomeprocessesinPM2andsavethem,sotheywillautomaticallybestartedwhentheVMboots.Makesureyouruntheindex.jsfilefrom/var/www/webshop-jsandnotfromyourJenkinsworkspace:
pm2start/var/www/webshop-js/index.js--name=webshop-js----port=8889
pm2save
[todisablestartup]
pm2unstartupsystemd
Whatthiscommandbasicallydoesiscreates,modifies,ordeletesthefile/etc/systemd/system/pm2-[username].serviceandaddsthePM2processsomewhere.
MongoDBSo,youshouldhaveyourwebshoprunningnow,butwearecurrentlyrunningonthedatabasethatweusedtodevelopoursoftware.Weneedaseparateinstanceforourdevelopmentenvironmentandourcurrentenvironment.So,firstlet'screateanewdatabaseandgiveourcurrentuserrightstothatdatabase.Wewillneedtocreateanewdatabaseanduserthatwecanuseforournewenvironment.
Sincewehavenotyetcreatedanadminuser,thiswasnotreallynecessaryuntilnow.Wekindoflockedourselvesoutofthedatabase.Notasmartthingtodo,buteasytofix.First,weneedtodisableauthenticationintheconfigfileandrestartMongoDB:
sudovi/etc/mongod.conf
[...]
#security:
#authorization:'enabled'
[...]
sudoservicemongodrestart
Afterthat,wecanentertheMongoDBshellandcreateouradminuser:
mongo
useadmin
db.createUser({
user:'admin',
pwd:'admin',
roles:[{role:'root',db:'admin'}]
})
exit
Therolerootgivesauserallprivilegesonadatabase.Bygivingtheusertheroleontheadmindatabase,theusereffectivelybecomesasuperuser.Youcannowre-enableauthorizationandrestartMongoDBagain.
Wecanalsocreateanewdatabaseanduserthisway:
mongo
useadmin
db.auth('admin','admin')
usewebshop-prod
db.createUser({user:'prod',pwd:'prod',roles:[{role:'readWrite',db:'webshop-prod'}]})
exit
Andnow,wecanalsologintothisdatabasewiththenewuserusingRobomongo.DoingallthisusingRobomongodidnotworkforme,probablybecauseMongoDBchangesitsAPIandRobomongoisnotuptothelatestversionyet.
Wealsoneedtocopythedatafromonedatabasetotheother.Fornow,itisaloteasiertojustcopythedatabasealtogether:
useadmin
db.auth('admin','admin')
usewebshop-prod
db.dropDatabase()
db.copyDatabase('webshop','webshop-prod')
Sonowthatwehavethenewdatabaseandthedata,wecanconfigurethepasswordinoursoftware.Createtwofilesnamedconfig.development.jsandconfig.production.js.WearegoingtoputourMongoURLsinthesefiles:
//config.development.js
module.exports={
dbConnection:'mongodb://username:password@ciserver:27017/webshop'
};
//config.production.js
module.exports={
dbConnection:'mongodb://prod:prod@ciserver:27017/webshop-prod'
};
Wenowhavetochangeourindex.jsfile,soitreadsthecorrectconfigfile:
varconfig=require('./config.'+(process.env.NODE_ENV||'development')+'.js'),
mongoUrl=config.dbConnection;
Weareusingthedevelopmentconfigurationfileasadefault.Tousetheproductionfile,oranyotherconfigurationyoumighthave,youcansettheNODE_ENVvariablefromyourconsole:
//Windows
setNODE_ENV=production
nodemonindex.js
//Linux
NODE_ENV=productionnodemonindex.js
DonotforgettoaddtheconfigurationyouwanttoyourJenkinsartifact.
RunNowthatwehaveallourfilesbuiltandready,alongwithaproductiondatabaseandanenvironment-specificconfigurationfileandourarchivecopied,wearefinallyreadytolaunchtheapplication:
cd/var/www/webshop-js
NODE_ENV=productionpm2startindex.js--name=webshop-js----port=8889
pm2save
Youarenowrunningalltheminifiedcodeontheproductiondatabaseautomaticallyonstartup.Onceagain,browsetociserver:8888andverifythateverythingworks.Trychangingsomethinginyourproductiondatabaseandseethatitreflectsonthewebsite.
E2EtestingAtthispoint,youcanaddanadditionalJenkinsprojectthattestswhetheryourproductionwebsiteisrunningproperly.Sincewedomanualdeploymentsatthispoint,thereisnowayforJenkinstoknowwhenyouaredonedeploying,andtheSeleniumtestscouldbetriggeredmanually.
Ofcourse,youshouldbecarefulwithsuchtests.Forexample,youneedtomakesureyouareusingatestaccountandthatyoudonotsendanyemailstoactualcustomers(wouldnotbethefirsttimesomeoneaccidentallysendstestdatatorealusers!).However,doingsomeautomatedsmoketestingonyourproductionenvironmentismorethanworthit!Mostthingsarecoveredbyunittests,butifsomethingobviousisamiss,suchasthedatabase,thisisthewaytonoticeitbeforeyourcustomersdo.Itisworthittodeployaversionofthesoftwaretoalocalserveraswell.Itgivesyoutheabilitytotestaproductiondeploymentwithoutanyriskofmessinguprealdata.
C#.NETCorewebshopAndnow,wecandothesameforour.NETCoreapplication.Ofcourse,weareusingdifferenttoolstodeploy.NETCoreapplications.For.NETCore,wecannotusePM2,whichisaNode.jstool.Instead,wewillcreateaservice(scriptthatexecutesonstartup)thatstartsthecross-platformwebserverKestrel,whichisusedtoruntheASP.NETCoresoftware.ItisthesamewebserverthatisusedwhenyourunyourwebshopfromVisualStudioCode.Again,wewillneedNGINXasareverseproxy,butwealreadysetthatupwhenweinstalledNGINX,sothatpartiscovered.
TocreatetheservicethatwillstartKestrelandrunour.NETCoreapplication,wecansimplycreateanewfileunder/etc/systemd/systemnamedkestrel-webshop-net.service:
sudovi/etc/systemd/system/kestrel-webshop-net.service
Intheservicefile,placethefollowingscript:
[Unit]
Description=A.NETCoreWebshop
[Service]
WorkingDirectory=/var/www/webshop-net
ExecStart=/usr/bin/dotnet/var/www/webshop-net/web-shop.dll
Restart=always
RestartSec=10#Restartserviceafter10secondsifdotnetservicecrashes
SyslogIdentifier=webshop-net
User=www-data
Environment=ASPNETCORE_ENVIRONMENT=ProductionASPNETCORE_URLS=http://localhost:9998
[Install]
WantedBy=multi-user.target
Wenowneedtogettheweb-shop.dllintothe/var/www/webshop-netfolder,justlikewedidwiththeJavaScriptwebshop.
Foraquicksee-if-it-workssolution,youcansimplygotoyourJenkinsworkspacefolder.WeshouldstillhaveaclassicJenkinsproject.Builditandrunitfromthere:
cd/var/lib/jenkins/workspace/CSharp\Web\Shop\-\Build
sudodotnetpublish-o../prodweb-shop/web-shop.csproj
cdprod
ASPNETCORE_URLS="http://*:9998"web-shop.dll
[Browsetociserver:9999inyourhost]
[Ctrl+Ctoclose]
Asyoucansee,weneedtopublishour.NETCoreapplicationbeforewecanrunit.The-oparameterspecifiestheoutputofthebuiltfiles.Thevalueoftheoutputparameterisrelativetothecsprojfileyouarebuilding,nottoyourcurrentlocation,hence,../.Wecanalsocopyeverythingtoourwwwfolderandcheckwhethertheservicecanruncorrectly:
cd/var/lib/jenkins/workspace/CSharp\Web\Shop\-\Build
sudocp-rfprod//var/www/webshop-net
Wecannowenableandstarttheserviceandmakesureitisrunningcorrectly:
sudosystemctlenablekestrel-webshop-net
sudosystemctlstartkestrel-webshop-net
sudosystemctlstatuskestrel-webshop-net
Now,onyourhost,browsetociserver:9999andyoushouldseeyour.NETCorewebshoprunning.
InJenkins,weshouldfixourprojectandmakesureitcreatesabuildaswellasanartifact.Perhaps,thisisagoodtimetocreateaJenkinsfilefortheC#.NETCoreprojectaswell.WewillstartwiththeDevelopmentJenkinsfile.AlotisjustcopiedandpastedfromtheJavaScriptJenkinsfile,soIamjustshowingthedifferenceshere.
Firstofall,thecheckoutstageisslightlydifferentbecausewealsohavetodoaBowerinstall:
stage('Checkout'){
steps{
node(label:'linux'){
ws(dir:env.ws){
checkoutscm
sh'''cdweb-shop
npminstall
node_modules/.bin/bowerinstall'''
}
}
}
}
Thetriplequotesareusedwithshwhenitismultiline.
Inthebuildphase,weneedtobuildtheprojectusingdotnetaswellasrungulp:
sh'''cdweb-shop
dotnetrestore
dotnetbuild
dotnetbundle
node_modules/.bin/gulp'''
stash(name:'Everything',excludes:'node_modules/**',includes:'**/**')
Again,westasheverythingsothatwecanuseitinthetestphaseonourWindowsslave.Thisoneisabittricky,especiallywiththeverylongandcomplexOpenCovercommand.ItwasexplainedinChapter9,AC#.NETCoreandPostgreSQLWebApp,though,soitshouldbefamiliar:
unstash'Everything'
bat'cdweb-shop-tests&&dotnetrestore&&(ifnotexistTestResultsmkdirTestResults)&&"C:\\ProgramFiles(x86)\\OpenCover\\OpenCover.Console.exe"-target:"C:\\ProgramFiles\\dotnet\\dotnet.exe"-targetargs:"test-l"trx;LogFileName=result.trx""-register:user-filter:"+[web-shop]*-[web-shop-tests]*"-output:"TestResults\\OpenCoverCoverage.xml"-oldStyle&&OpenCoverToCoberturaConverter.exe-input:"TestResults\\OpenCoverCoverage.xml"-output:TestResults\\Cobertura.xml&&ReportGenerator\\ReportGenerator.exe-reports:"TestResults\\OpenCoverCoverage.xml"-targetDir:TestResults\\CoverageHTML'
bat'msxsl-oJUnitResults.xmlweb-shop-tests\\TestResults\\result.trxtrx-to-junit.xsl'
junit'JUnitResults.xml'
publishHTMLtarget:[
allowMissing:false,
alwaysLinkToLastBuild:false,
keepAll:false,
reportDir:'web-shop-tests/TestResults/CoverageHTML',
reportFiles:'index.htm',
reportName:'CoverageReport'
]
step([$class:'CoberturaPublisher',
autoUpdateHealth:false,
autoUpdateStability:false,
coberturaReportFile:'web-shop-tests/TestResults/Cobertura.xml',
failUnhealthy:false,
failUnstable:false,
maxNumberOfBuilds:0,
onlyStable:false,
sourceEncoding:'ASCII',
zoomCoverageChart:false
])
So,afewthingsaregoingonhere.Firstistheweirdbatchscriptwiththemsxslcommand.Unfortunately,itisnotpossibletopublishTRXfilestoJenkinsusingthepipelinescript,soweneedtogetcreative.OnewaytogetaroundthislimitationisbytransformingtheTRX,whichisjustXML,totheJUnitformat,whichisjustslightlydifferentXML.YoucandownloadthemsxsltoolfromMicrosoftathttps://www.microsoft.com/en-us/download/details.aspx?id=21714.DownloaditandputitintherootofyourC#.NETCoreproject.TheXSLfiledescribesthetranslationfromoneformattoanother.ItisquitebigsoIwillnotpostithere,butitisinGitHub.Iusedthisapproachforthefull.NETtestframeworkaswell.IgotthecurrentxsltfromaGitHubGist(https://gist.github.com/cdroulers/510d2ecd6ff92002bb39469821a3a1b5)anditseemstoworkwellenough.
Then,thereisaCoberturaPublisherstep.PipelinesupportfortheCoberturapluginwasactuallyaddedwhilewritingthisbook,soyouguysareinluck!Youcansimplygenerateitfromthepipelinesnippetgeneratorbyselecting"step:GeneralBuildStep"in"SampleStep"andthen"PublishCoberturaCoverageReport"in"BuildStep".
Lastisthedatabasetest.YoucangeneratethecodefortheTAPPublishstepthesamewayyoucangeneratethecodefortheCoberturareport:
unstash'Everything'
sh'pg_prove-dwebshop-vweb-shop/test/sql/*.sql|teetap.txt'
step([$class:'TapPublisher',
testResults:'tap.txt'
])
Next,wecancreateaJenkinsfileforthemasterbranchtoo.Ofcourse,thisisalsonotverydifferentfromwhatwealreadyhad.Theonlythingweneedtoaddisbasicallyadotnetpublishcommand:
sh'rm-rfnode_modules'
sh'npminstall--only=production'
sh'dotnetpublish-o../prodweb-shop/web-shop.csproj'
archiveArtifacts'prod/'
ToruntheJenkinsfile,youneedtocreateamultibranchpipelineinJenkins.YoucancopytheexistingmultibranchpipelineforourJavaScriptwebshopandsimplychangetheGitURL.Last,besuretocreateawebhookinGitLab,soyourbranchesareautomaticallytriggeredwhenyoumakeacommittoabranch.
RunAgain,wecancopythefilesfromourJenkinsartifactusingWinSCP,butweneedtomaketherootusertheownerofthe/var/www/webshop-netfolder,sowehavesufficientpermissionstocopythefiles:
sudochown-Rjenkins:jenkins/var/www/webshop-net
Now,trycopyingtheartifactsusingWinSCPandbrowsetociserver:9999toseethateverythingworksasitshould.YoucanrunyourSeleniumtestsagainstciserver:9999toseeifeverythingstillworks!
PostgreSQLThelastthingthatremainsisourproductiondatabase.Wecanquiteeasilysetupournewdatabaseaswestillhaveallthescriptsweneedtosetitup.DonotworryaboutallthepgTapscripts.Wearenotgoingtotestthisdatabase;wealreadydidthatonourdevelopmentdatabaseafterall.SimplyruntheCREATEDATABASEscriptinpgAdmin,butbesuretorenamethedatabasetosomethingsuchas"webshop-prod"(quotesnecessary):
CREATEDATABASE"webshop-prod"
[...]
Nowopenthequerytoolonthenewdatabaseandexecutetherestofthescripts.
Toconfigureourdatabaseconnectionstringin.NETCore,wewillneedtodoalittlerefactoring(anddothingsthecorrectway).Wecanaddtheconnectionstringstotheappsettings.jsonfile.Eachenvironmenthasitsownappsettings.jsonfile,forexample,appsettings.Development.jsonandappsettings.Production.json.Defaultvaluescanbesetintheappsettings.jsonfile,whilespecificvaluescanbeoverwrittenperenvironmentfile.Youshouldhaveappsettings.jsonandappsettings.Development.jsonalready.Addappsettings.Production.jsonaswell.
AddaConnectionStringsnodetoappsettings.jsonandtoappsettings.Production.json:
//appsettings.json
{
"Logging":{
[...]
},
"ConnectionStrings":{
"WebShopDatabase":"Host=ciserver;Database=webshop;Username=sa;Password=sa"
}
}
//appsettings.Production.json
{
"ConnectionStrings":{
"WebShopDatabase":"Host=ciserver;Database=webshop-prod;Username=sa;Password=sa"
}
}
Thenextpartisabittricky.Wecanreadtheconfigurationatstartup,passittoDbContext,andinjectittoourclassesusingconstructorinjection.The
configurationhasalreadybeenreadandstoredintheConfigurationpropertyoftheStartupobject.WecanaddDbContexttoIServiceCollection,soitisinjectedintoourclasses.IServiceCollectionpassesDbContextOptionsBuilderthatwecanusetoinitializeourcontext,prettymuchlikewealreadydid.So,addtheserves.AddDbContextlineunderservices.AddMvc();inStartup.cs:
services.AddMvc();
//Addthisline.
services.AddDbContext<WebShopContext>(
optionsBuilder=>optionsBuilder.UseNpgsql(
Configuration.GetConnectionString("WebShopDatabase")
)
);
Tomakethiswork,weneedtomodifyWebShopContextaswell.AddaconstructorthattakesDbContextOptions<TContext>asaparameterandpassittothebaseconstructor.Also,removetheOnConfigurationmethod:
publicclassWebShopContext:DbContext
{
publicWebShopContext(DbContextOptions<WebShopContext>context)
:base(context)
{}
[...]
//Removethismethod.
//protectedoverridevoidOnConfiguring(DbContextOptionsBuilderoptionsBuilder)
//{
//optionsBuilder.UseNpgsql("Host=ciserver;Database=webshop;Username=sa;Password=sa");
//}
}
Lastbutnotleast,wecaninjectourcontexttoourcontrollersandusethatinsteadofcreatinganewcontexteverytimeweneedit.HereisanexampleforHomeController.NoticethatIremovedtheusing(varcontext=newWebShopContext())code:
publicclassHomeController:Controller
{
privatereadonlyWebShopContextcontext;
publicHomeController(WebShopContextcontext)
{
this.context=context;
}
>
[...]
publicIActionResultGetTopProducts()
{
varproducts=context.Products
[...]
returnJson(products);
}
[...]
}
Onceyouhaverefactoredallthecontrollers,youshouldbereadytogo.Youcanrunyoursoftwareandseeittaketheconnectionstringfromappsettings.json.WhenyourunintheProductionmode,youwillseetheconnectionstringfromappsettings.Production.json.YoucanrunintheProductionmodebychangingtheASPNETCORE_ENVIRONMENTvariableinlaunch.json.
Ofcourse,wewillneedtoreleaseoursoftwareagainforthistotakeeffect.However,sinceweseteverythingupinJenkinstoautomaticallycreateanewbuildthatwecansimplycopyandpasteusingWinSCP,thisshouldnotbeaproblem.Afterthenextchapteritwon'tbedifficulttoimplementasinglebuttonclickstrategyforyourcontinuousdeliveryprocess.Anychangestothedatabasewillhavetobemanuallydeployedthough!YoucanrunyourSeleniumtestsagaintoseeifthenewdatabaseworksasexpected.
Settingupallthesetoolsanddatabasesisarealdrag.YoucouldprobablyautomateitusingJenkins,butthatisprobablyprettydifficultandtime-consuming.However,thereareplentyofsituationswhereyouneedmultiplesofthesekindsofservers.Maybeyouwanttoputaloadbalancerbetweenyourcustomerandyourserverormaybeyoujusthavealotofwebsitesrunning.Ofcourse,therearetoolsthatgreatlyspeedupthisprocess,eitherthroughvirtualizationorcontainerization.Usingtools,suchasPuppet(https://puppet.com/)orChef(https://www.chef.io/chef/),theso-calledInfrastructure-as-Codesolutions,youcandeploynew(virtual)serversonthefly.Containerizationisalittledifferentinthatitrunsanisolatedruntimeenvironmentonyourhostmachine.Youcaneasilyspinupacontainerthatcontainsallthenecessarysoftwareandrunsomeprograminit.Docker(https://www.docker.com/)isthemostpopularalternativebyfar.Dockerisalsoaverypopulartoolfortestingpurposes.Thesetoolsareoutsidethescopeofthisbook.TheybelongtotheOpsinDevOps,whilethisbookhasafocusontheDevinDevOps.
SummaryInthischapter,wehavedeployedbothourJavaScriptwebshopandourC#.NETCorewebshop.Whilebothapplicationsrequireadifferentsetup,theprocessismoreorlessthesame.WecouldrelativelyeasilyinitiateadeploymentusingtheartifactsthatJenkinsalreadyarchivedforus.Afinalsmallstepawaitsinthenextandlastchapterofthisbook--gofromacommittoaproductionupdatethatiscompletelyautomated.
ContinuousDeploymentInthislastchapterofthebook,wewilllookatContinuousDeploymentanditsdifferencefromContinuousDelivery.WehavealreadysetupContinuousDeliveryinthelastchapter.Inthischapter,wewilluseournewbranchtobuildandtestoursoftwareandchangethemasterbranchtoactuallydeployandtestthesoftwarefullyautomated.Asexplainedinthelastchapter,simplymergingthosebranches,orcherry-pickingspecificcommits,canleadtoamore-or-lessautomateddeployment.WithContinuousDeployment,wewanteverythingtobefullyautomated.Inbothcases,itisveryimportantthatyoursoftwareisalwaysdeployable,butcontinuousdeploymenttakesthatstatementastepfurtherbyactuallydeployingthesoftwaretoaproductionenvironment.
Asmentionedbefore,andIwanttostressthisagainandagain,continuousdeploymentisnotalwaysanoption.Especiallydatabasesmakethisfullyautomatedprocessariskformanycustomers.However,youcanstillautomaticallydeployaversionoftheirproducttoyourlocalenvironmentfortestingpurposes.
JavaScriptDeploymentusingSSHThefirstthingweneedtodoisgetoutartifactsfromourJenkinsbuildtowherewewantthemfullyautomated.Usually,thiswillbesomeremoteserver,butwedonothavethat.However,whatwewilldoistopretendthatourlocalLinuxmachineissomeotherremoteLinuxmachineandtransferthefilesusingSSH.
First,inourmasterJenkinsfile,wewillneedtounpackourartifactssothatwecancopythematall.Jenkinswillmentionthatunarchiveisdeprecatedandreplacedformostpurposesbystashandunstash.AsfarasIknow,stashandunstasharetemporaryforthedurationofthebuild,whilearchivekeepsyourartifactsevenwhenthebuildiscomplete.Weneedthearchivebecausewealwayswanttoknowwhatiscurrentlyrunningonproductionandmaybebecausewewanttobeabletomanuallydeployfilesaswell.Theonlywaytogetyourfilesbackusingthepipelineisusingunarchive,soIamnotsurewhyitisdeprecatedorwhatmostpurposesmeans,butitseemsperfectlyfinetouseithere:
unarchivemapping:['**/**':'./artifacts']
Thismeansweareunarchivingeverything(**/**)andplacingitinourcurrentworkspaceintheartifactsfolder(whichwillbecreated).Alternatively,wecanjustcopythefilesdirectly,withoutunarchivingfirst.Itisabitmorework,butfasterandcertainlynotdeprecated.
WecannowcopythesefilesusingSSH.SincewewilluseSSHusingJenkins,andwewanttoconnectusingJenkins,weneedtogivetheJenkinsaccount,thatis,theuseronwhichJenkinsisrunning,remoteaccesstoourserver.YoucanlogintotheJenkinsuserusingthesucommand(switchuser).However,theuserneedsapasswordfirst:
sudopasswdjenkins
EnternewUNIXpassword:[yourpassword]
RetypenewUNIXpassword:[yourpassword]
passwd:passwordupdatedsuccessfully
sujenkins
Password:[yourpassword]
NowthatweareloggedinasJenkins,wecangenerateakeytobeusedbySSHandaddourkeytotheknownhosts:
ssh-keygen-trsa
[Enter]
[Nopassphrase,soenteragain]
[Again,nopassphrase,enter]
ssh-keyscan-trsaciserver>>~/.ssh/known_hosts
ssh-copy-idjenkins@ciserver
[Jenkinspassword]
Yourkeyfileswillbeplacedinthe~/.sshhiddenfolder:
cd~/.ssh
ls
WecannowuseSSHtoconnecttotheserver,butwedonothavewritingpermissionsto/var/www/webshop-jsyet.Insteadofgivingpermissions,wewillmakeJenkinstheownerofthefolder.First,switchbacktoyourrootusersothatwecanswitchtheuser:
sudochown-Rjenkins:jenkins/var/www/webshop-js
KeepinmindthatwesettheownerofthisfoldertoyourrootuserinChapter13,ContinuousDelivery.Nowthattherootuserisnottheowneranymore,theirpermissionshavebeenrevokedandlogginginusingWinSCPisnolongerpossiblewiththatuser.Youcan,however,loginusingthejenkinsusernow.
Finally,wecancopythefilesusingthescpcommand,whichusesSSHtocopyfiles.Beforewedoso,wecan,optionally,clearoutthewebshop-jsfoldersothatwedonotgetstuckwitholdfilesandweknowforsurethatourbuilddoesnotrelyonsomeoldfiles:
unarchivemapping:['**/**':'./artifacts']
sh'rm-rf/var/www/webshop-js/*'
sh'scp-rartifacts/*jenkins@ciserver:/var/www/webshop-js'
ThecompletecodefortheArchivingArtifactsandDeploymentstagesintheJenkinsfilelooksasfollows:
stage('ArchivingArtifacts'){
steps{
node(label:'linux'){
ws(dir:env.ws){
sh'rm-rfnode_modules'
sh'npminstall--only=production'
archiveArtifacts'index.js,config.*.js,prod/,node_modules/'
}
}
}
}
stage('Deployment'){
steps{
node(label:'linux'){
ws(dir:env.ws){
unarchivemapping:['**/**':'./artifacts']
sh'rm-rf/var/www/webshop-js/*'
sh'scp-rartifacts/*jenkins@ciserver:/var/www/webshop-js'
}
}
}
}
Ifyouarenotcleaningoutyourworkspacebeforeorafterbuilding,ensurethatyouaddtheartifactsfoldertothesonar.exclusionsinthesonar-project.propertiesfile:
sonar.exclusions=[...],artifacts/**
IfyourunyourJenkinspipelinenow,yourfilesshouldbeautomaticallycopiedto/var/www/webshop-js.TrymakingsometextchangeinanHTMLfile,commitit,andwaitforafewminutes;youshouldseeyourchangeappearonthatpage.
E2EtestingWecannowaddanewstepinourmasterJenkinsfile.Whenthewebshopisautomaticallydeployed,wecanrunsomeSeleniumteststocheckwhethereverythingworks.Wehaveabugproductionifthisdoesnotwork,andweshouldfixitasap!Itishighlyunlikelythatwegetissuesatthisstagethough,especiallysincetheDevelopmentbranchalsohasaSeleniumstep:
stage('Selenium'){
steps{
node(label:'windows'){
ws(dir:env.ws){
gitlabCommitStatus(name:'Selenium'){
script{
unstash'Everything'
bat'npminstall'
bat'node_modules\\.bin\\webdriver-manager.cmdupdate'
bat'node_modules\\.bin\\protractor.cmd--baseUrlhttp://ciserver:8888test\\protractor.conf.js'
}
}
}
}
}
}
C#.NETCoreDeploymentusingSSHNextupistheC#webapplication.Wehavealreadydonemostofthework;nowsetupSSH.
FixingourJenkinsfilesothatitcopiesourarchivedfilesautomaticallyshouldnotbeaproblem.WecandothisagainusingSSH.First,weneedtomakeJenkinstheownerofthewebshop-netfolder:
sudochown-Rjenkins:jenkins/var/www/webshop-net
Again,thismeansthatyourrootuserlosesprivileges,andifyouwanttocopyusingWinSCP,youshouldusethejenkinsuser.
Afterthat,itisprettymuchthesameasfortheJavaScriptwebshop:
sh'rm-rf/var/www/webshop-net/*'
sh'scp-rprod/*jenkins@ciserver:/var/www/webshop-net'
ThecompletecodefortheArchiveArtifactsandDeploymentstepslooksasfollows:
stage('ArchivingArtifacts'){
steps{
node(label:'linux'){
ws(dir:env.ws){
sh'rm-rfnode_modules'
sh'npminstall--only=production'
sh'dotnetpublish-o../prodweb-shop/web-shop.csproj'
archiveArtifacts'prod/'
}
}
}
}
stage('Deployment'){
steps{
node(label:'linux'){
ws(dir:env.ws){
unarchivemapping:['**/**':'./artifacts']
sh'rm-rf/var/www/webshop-net/*'
sh'scp-rartifacts/prod/*jenkins@ciserver:/var/www/webshop-net'
}
}
}
}
Again,trychangingsometextinanHTMLfile,commitit,andseethatitisautomaticallydeployed.
E2EtestingNowthatwehaveour.NETCoreapplicationrunning,wecanaddourSeleniumteststoourJenkinsbuild.Ensurethatyouaddthemafteryouhavedeployedthesoftware.Speakingofwhich,itmaybeworthittodeployaversionofyoursoftwarebeforeeverythingisminified,uglified,andbundledtosomelocalcompanyserver,andrunyourtestsonthatenvironmentaswell.Itwillbealoteasiertodebuganyproblems,andthesetupcanmorecloselymatchyourultimateproductionenvironmentthanyourlocaldevelopmentmachinedoes.
TofixourSeleniumtests,wefirsthavetomakesomeminorcodechanges.Unfortunately,.NETCoreunittestprojectsarealittlemorelimitedthanMVCCoreprojects,sowedonothaveanyappsettingsorotherconfigurationfiles.Itisprobablypossiblesomehow,butsinceitisnotsuppliedoutoftheboxandwedonotreallyneedit,wewillhardcodeourURLintothecode:
publicSeleniumTests()
{
server=newTestServer(newWebHostBuilder()
.UseStartup<Startup>());
client=server.CreateClient();
client.BaseAddress=newUri("http://ciserver:9999");
}
[...]
//ChangetheGoToUrltousethebaseaddress.
driver.Navigate().GoToUrl($"{client.BaseAddress}");
[...]
//NoslashbetweenBaseAddressandpage.
driver.Navigate().GoToUrl($"{client.BaseAddress}ShoppingCart");
Youcanalreadyrunthemlocally,usingdotnettest,andseewhethereverythingstillworks.Ifyouneedtotestlocally,youcanjustmanuallychange9999to5000,butthatshouldnotbenecessaryallthatoften.Keepinmindthatyouarecurrentlytestingtheminifiedsource,sowhenyoudothisonatestserverfirst,youcanbeprettysurethatyourcodeworksonproductionaswell.
ChangingyourJenkinsbuildisnowprettyeasy;whetheryouareusingtheclassicJenkinsprojectsortheJenkinsfile,justexecutethedotnettestcommand.Ofcourse,wewantsomereportingtoo.Wecan,again,usethemsxsltoolinthe
Jenkinsfile:
stage('Selenium'){
steps{
node(label:'windows'){
ws(dir:env.ws){
gitlabCommitStatus(name:'Selenium'){
script{
unstash'Everything'
bat'dotnettest-l"trx;LogFileName=result.trx"web-shop-selenium\\web-shop-selenium.csproj'
bat'msxsl-oJUnitResults.xmlweb-shop-selenium\\TestResults\\result.trxtrx-to-junit.xsl'
junit'JUnitResults.xml'
}
}
}
}
}
}
DatabaseDatabasesarereallydifficulttoupdateautomatically.AsIhavementionedsomewhereearlierinthisbook,youwilloftenfacecustomersordatabaseadministratorswhoflatoutforbidyoutodoanyupdatesonadatabase,letalonedosoautomatically.Itisnotuncommonthatdevelopersdelivertheirscriptstoaperson,typicallytheDBA,whothenmanuallychecksthemandrunsthemonthedatabase.Forgoodreason,thedatabasestoreswhatabusinessisallaboutorwhatitneedstorunproperly--data.Losingordamagingitcanputacompanyoutofbusiness(but,ofcourse,youhavebackups).
Lesssevere,butpotentiallydamagingtothebusiness,arescriptsthatlocktables,updatelivedata,orchangebusinessrules.Youcanimaginethatsomescripts,suchasupdatinga1,000GBtable,canpotentiallylockanentiresystem.Suchupdatesshouldhappenoutsideofbusinesshours(ifpossible)andincontrolledenvironments.
Itisprobablyagoodbettonotdocontinuousdeploymentonadatabase,butitmaystillbeavalidoptionwhenyouareincontrolofthesoftwareanddatabaseorwhenthedatabasehasnon-criticaldata.
Whenyouarecertainyoucandodatabaseupdatesautomaticallywithoutproblems,therearevarioustoolsthatcanhelpyouwiththistask.OnesuchtoolistheEntityFrameworkwhenyouaredoing.NET(Core)development.Wehavechosenadatabase-firstapproach,butyoumightaswellgoforacode-firstapproachfornewprojectsorswitchfromdatabase-firsttocode-firsthalfway.Acode-firstapproach,usingMigrations(https://docs.microsoft.com/en-us/aspnet/core/data/ef-mvc/migrations),meansyoucancodeyourclassesandyourobjectcontext,and.NETwillgenerateorupdateyourdatabasewhenyounextrunyourapplication.Thismeansthatanynewversionofyoursoftwarewillcheckyourdatabaseschemaandupdateit,ifnecessary,whenitisexecuted.Onecoolfeatureisthatyoucanalsodowngradeyourdatabasetoapreviousversion.Well,thisiscoolintheory,butpersonally,Ihaveneverdowngradedaproductionapplication.Migrationsaddalittlestartuptimetoyourapplication,butitispotentiallyworthit.Thereareother.NETtoolsthatcandomoreorlessthe
same,suchasFluentMigrator(https://github.com/fluentmigrator/fluentmigrator),whichisbasedonRubyonRailsMigrations.IhavewrittenaboutEntityFrameworkMigrationsinapreviousbookofmine--SQLServerForC#DevelopersSuccinctly(https://www.syncfusion.com/resources/techportal/details/ebooks/sql_server_for_c_sharp_developers_succinctly)--thatyoucandownloadforfree.Anyway,itisnotinthescopeofthisbook,soIamnotdiscussingitanyfurtherhere.
Ifyouareworkingwithscripts,therearesomepossibilitiesaswell.Flyway(https://flywaydb.org/)isonesuchpossibility;itsupportsmanySQLdatabases,suchasSQLServer,MySQL,Oracleand,ofcourse,PostgreSQL.Flywayisincrediblyeasytouse.Wecanstartbyusingthecommand-linetoolfromWindowsjusttoseewhatitisanddoes.So,downloadthelatestcommand-linetoolfromtheFlywaywebsite(https://flywaydb.org/getstarted/download).Simplyunzipthetoolfoldertosomewhereonyourcomputer;Ihaveplaceditonmydesktopforeasyaccess.Next,simplycreateanewdatabaseandnameitwhateveryouwant,suchasflyway_demo:
CREATEDATABASEflyway_demo
WITH
OWNER=sa
ENCODING='UTF8'
CONNECTIONLIMIT=-1;
Now,let'screateanewfolderinthesamefolderthatyouputtheFlywayfolderin;inmycase,itisthedesktop.Itdoesnotreallymatterwhatyounameit,butInameditFlywaydemo.Now,createanewfolderwithinthatfolderandnameitMigrations.IntheMigrationsfolder,wewillputourfirstmigrationscript,whichisjustaCREATETABLEstatement.So,createafileandnameitV1_0__Createtable.sql(withadoubleunderscorebetween0andCreate).Inthefile,putthefollowingSQLstatement:
CREATETABLEpublic.some_table
(
idintegerNOTNULL,
nametextCOLLATEpg_catalog."default"NOTNULL,
CONSTRAINTsome_table_pkeyPRIMARYKEY(id)
)
WITH(
OIDS=FALSE
)
TABLESPACEpg_default;
ALTERTABLEpublic.some_table
OWNERtosa;
WenowneedtotellFlywaywhereitcanfinditsmigrations,howitcanrecognizethem,onwhatserveranddatabasetoexecutethem,andhowtologin.Wecanuseaconfigurationfileforthis.Putafilenamedflyway.confinthesamefolderastheMigrationsfolder.Putthefollowingcontentsinthefile:
flyway.driver=org.postgresql.Driver
flyway.url=jdbc:postgresql://ciserver:5432/flyway_demo
flyway.user=sa
flyway.password=sa
flyway.locations=filesystem:Migrations
flyway.sqlMigrationPrefix=V
flyway.sqlMigrationSeparator=__
flyway.sqlMigrationSuffix=.sql
flyway.validateOnMigrate=true
Itisreallyquiteself-explanatory.Wespecifythedriver,theURLtothedatabase,ourcredentials(obviouslysa/saisnotverysecure!),thelocationsparameterspecifieswhereFlywaycanfinditsSQLscripts,thesqlMigrationPrefix,sqlMigrationSeparatorandsqlMigrationSuffixtellFlywayhowtorecognizemigrationfilesand,lastly,validateOnMigrateaddsanadditionalvalidation.Wecannowopenupacommandprompt,browsetothefoldercontainingourflyway.conffileandMigrationsfolderandrunflywaymigrate.Flywaywillpickuptheconfigurationfileandusethoseoptions:
..\flyway-4.2.0\flywaymigrate
Optionally,youcanaddadditionalparametersintheCommandPromptoroverwriteparametersfromtheconfiguration:
..\flyway-4.2.0\flyway-url=jdbc:postgresql://ciserver:5432/another_dbmigrate
Theoutputisasfollows:
Whenyoucheckoutthedatabasenow,ithastwonewtables:thesome_table,asperourmigration,andtheschema_versiontablethatFlywayaddedinordertokeeptrackofmigrations.Donotchangemigrationsthathavebeenmigrated;createanothermigrationinstead:
Ifyourunflywaymigrateagain,youwillfindthatnothinghappens;youwillgetaNomigrationnecessarymessage.Youcanaddnewmigrations,andtheywillallbeexecutedonasingleflywaymigrate.TryitbycopyingourfirstmigrationandnamingitV1_1__Anothertable,andchangesome_tabletoother_tableinthescript.
Wecannowtrytouseflywaymigrateonourwebshopdatabase.Unfortunately,youwillgettheERROR:Foundnonemptyschema(s)!message.Flywayissayingthatthereisnoschema_version,butthedatabaseisnotemptyeither.Wecanaddabaselinebyaddingonetoourconfigurationfileandrunningflywaybaseline:
[...]
flyway.baselineVersion=0.9
flyway.baselineDescription=BaseMigration
Now,runflywaybaseline,followedbyflywaymigrate:
..\flyway-4.2.0\flywaybaseline
..\flyway-4.2.0\flywaymigrate
Thebaselineversionisquiteimportant.Wehavetwoscripts:V1_0__*andV1_1__*.
Ifweputourbaselineonversion1.0or2.0,thentheV1_0and,possibly,theV1_1scriptswillnotbeexecuted.
So,nowyoucanputFlywayonyourWindowsorUbuntumachine,orcommititwithyourproject,anduseitinyourJenkinsbuildtoupdateyourdatabases.Justensurethatyouproperlyversionandwriteyourmigrations.
Next,toautomatetoolsandkeeptrackofdatabasescripts,youcanoptforathirdoption--databasecomparetools.Thereareplentyoftoolsouttherethatcancomparedatabaseschemasandgenerateupdatescriptsthatyoucanrunasyouseefit.
AsMongoDBisschemaless,updatesarealoteasier.Justupdatethesoftwarewithanewtableorfield,andMongoDBwilljustinsertitorreturnadefaultvalueforthealreadyexistingrecordsthatdonothavethenewfieldyet.
SummaryInthefinalchapterofthebook,wewentfromcommittingsomecodetoseeingitliveonanotherenvironmentwithoutanyfurtherhumanintervention.WecopiedfilesusingJenkinsandSSH,andwealsolookedatdatabasedeploymentwithFlyway.
Thisconcludesthebook.Wesawvariousmethodstoguaranteeacertainlevelofcodequality,rangingfromlintingtounittests.Otherthanthat,wehaveusedvarioustools,suchasSonarQube,Karma,Gulp,andJenkins.However,thereismoretosee.IhavementionedtoolssuchasPuppet,Chef,andDocker,butwedidnotdiscussthematlength.CIishotandthefieldisevergoingforward,sokeepyourselfupdated.Withthisbook,youshouldhaveasolidtheoreticalknowledgeandthenecessarypracticalskillstoputanynewtooltogooduse.