wso2con eu 2016: integrate apim to third-party tools: creating an agent for elk

WSO2APIManagersta1s1cs:integra1onwiththeELKstack

DominiqueDebailleuxSeniorArchitectEMOXA

Agenda

•  APIManagementtools:whatisitabout?•  APIU1liza1onmetrics…whatfor?•  WSO2APIManagerinanutshell•  Sta1s1cs:WSO2DASapproach•  YetAnotherSolu1on?•  Drawmean•  PuJngitalltogether•  Onestepbeyond•  FAQ•  Almostforgot!•  WSO2APIM:opensource…andopenminded

APIManagementtools:whatisitabout?

APIManagementtooling:whatisitabout?

•  Ali%lereminder,justincaseyoulivedinacaveintheNunavutduringthelast3years…•  APImanagement(APIM)

–  Publishing,promo1ngandoverseeingAPIs–  Inasecure,scalableenvironment.–  Alsoincludesthecrea1onofendusersupportresourcesthatdefineand

documenttheAPI.•  APIMtoolsmainfocus:

–  Design&deployAPIs–  Exposethemtointernalandexternaldevelopers–  Ensurethattheyaresecuredandfunc1onal–  Getra1ng&u1liza1onfeedbacks

APIManagementtooling:whatisitabout?

•  APIMtoolsbasics:–  Developerportal:engagementw/partnersordevelopers–  SLA&trafficmanagement–  Highavailability&scalability–  Consistencyacrossimplementa1onsandversions–  Cachingmechanism–  Security&authoriza1on–  Easytodeployandtest–  APIDocumenta1on–  Sta1s1cs&Analy1cs

APIU1liza1onmetrics…whatfor?

APIU1liza1onmetrics…whatfor?

•  Helptodiagnos1chowpeopleuseAPIsandgetinsightsforyourbusiness•  Alloworganiza1ons

–  tounderstandAPIu1liza1on–  tomakesuretheneedsofdevelopersandapplica1onsusingtheAPIare

beingmet–  toplanaccordinglyforfutureenhancementsorcapaci1es

•  Willalsobeusedtohelpbothmonitorandmone1zetheAPIsexposed

APIU1liza1onmetrics…whatfor?

•  APIMtoolscollectandstorerun1meinforma1onaboutAPIsandtheirconsumerssuchas:–  howo\enareAPIsbeingused(average,peakrequests,response1me,…)–  whichAPIsarebeingused,–  whoisregistreredandisusingtheAPIs,–  SLAcomplianceorexpecta1on,–  …–  well,anydatathatismeaningfulltotheorganiza1ons

WSO2APIManagerinanutshell

WSO2APIManagerinanutshell

•  h^p://wso2.com/products/api-manager/

•  FreeandopensourceAPIMsolu1on–  Designs&publishesAPIsinminutes–  Includesadeveloperportalw/selfregistra1on(Applica1onStore)–  GovernsAPIs(lifecycle,visibility,versions)–  APIcrea1onandUsersregistra1onvalida1onthroughworkflows(WSO2

BPS)–  ProvidessecuritythroughAPIsaccesscontrol(OAuth2)–  …

WSO2APIManagerinanutshell

•  FreeandopensourceAPIMsolu1on(cont’d)–  …–  On-premise&Saasversion–  HA&Scalable–  Limitedoverhead–  Highlyextensible&customizable–  ProvidesAnaly5cs(WSO2DAS)–  LeveragesonWSO2ESBandWSO2IS

WSO2APIManagerinanutshell

•  Fourcomponentswhichcanbesplitondifferentservers

•  Publisher(managementwebapp)–  APIdesignandconfigura1on–  Publica1ontowardtheStoreandtheGateway

•  Store(portalwebapp)–  APIscatalogue–  Subscrip1ons–  Accesskeygenera1on(Oauthtoken+ClientID&secret)–  APIstest(sandbox&produc1on)

WSO2APIManagerinanutshell

•  Gateway(run1mecomponent)–  APIProxy–  Appliespolicies(security,thro^ling)andcollectssta1s1cs–  Passesthewebservicecalltothebackend–  LeveragesonWSO2ESB

•  KeyManager(securitycomponent)–  Tokenmanagement–  Authoriza1onmanagement

14

Backend

APIGatewayExtract

accesstokenValidate

accesstokenApplyQoSpolicies

Publishsta1s1cs

Customextensions

APIStore

KeyManager

APIPublisher

Client

Publisher

1

1’

2

2

3

4 5

6 7 8

9

Sta1s1cs:WSO2DASapproach

16

Backend

APIGatewayExtract

accesstokenValidate

accesstokenApplyQoSpolicies

Customextensions

Publishsta1s1cs

Client

Publisher

5

1

2 3

4

APIPublisherAPIStore

APIMsta1s1cstables

request responsefault

othereventstreams

2

32’

3’

2’’

3’’

2’’

3’’

5

Streamdatatable

Summarydatatable

Eventflow

17

Backen

d APIMsta1s1cstables

BAMmediator

Workflowexecutor

In

APIMgtUsageHandler

Out

APIMgtResponseHandler

Fault

APIMgtFaultHandler

APIMgtUsageDataBridgeDataPublisher

mediate(mc)

mediate(mc)

mediate(mc)

publishEvent(DTO)

APIGateway(ESBinside)

Publishsta1s1cs

Usagepublisher

RequestPublisherDTO

Streamdatatable

Summarydatatable

SPAR

K

Analy1cs

Eventreceivers

Eventstore

Processedeventstore

APIMgtUsageDataPublisher

+publishEvent(RequestPublisherDTO)+publishEvent(ResponsePublisherDTO)+publishEvent(FaultPublisherDTO)

APIMgtUsageDataBridgeDataPublisher

RequestPublisherDTO

Requeststreamdefini=on

ResponsePublisherDTO

Responsestreamdefini=on

FaultPublisherDTO

Faultstreamdefini=on

APIMgtFaulHandler

+mediate(MessageContext)

APIMgtUsageHandler

+mediate(MessageContext)

APIMgtResponseHandler

+mediate(MessageContext)

DASPublisherimplementa1on

Handlers&Mediators

StreamDTOs

YetAnotherSolu1on?

YetAnotherSolu1on?

•  DASisagreatso\warebutwhatifonealreadyhasaBAMandSAMsolu1oninplace?

•  Opera1vesarenoteagertoaddasecond(third,…)tool•  Someonlyneedplainsta1s1cswithfirstlevelhistorycapabili1es

(summupsandsimpleaggrega1ons)thatmaynotrequirehighcompu1nganalysismodel

•  …notnecessarily->userstory

APIMgtUsageDataPublisher

+publishEvent(RequestPublisherDTO)+publishEvent(ResponsePublisherDTO)+publishEvent(FaultPublisherDTO)

YourAPIMgtUsagePublisher

RequestPublisherDTO

Requeststreamdefini=on

ResponsePublisherDTO

Responsestreamdefini=on

FaultPublisherDTO

Faultstreamdefini=on

APIMgtFaulHandler

+mediate(MessageContext)

APIMgtUsageHandler

+handleRequest(MessageContext)

APIMgtResponseHandler

+mediate(MessageContext)

YOURPublisherimplementa1on

YoukeepthesameStreamDTOs

22

Backen

d

BAMmediator

Workflowexecutor

In

APIMgtUsageHandler

Out

APIMgtResponseHandler

Fault

APIMgtFaultHandler

YourAPIMgtUsageDataPublisher

mediate(mc)

mediate(mc)

mediate(mc)

publishEvent(DTO)

APIGateway(ESBinside)

Publishsta1s1cs

Usagepublisher

RequestPublisherDTO

Yourpreferredanaly1cssolu1on

goesthere

Withsomeadjustments

here

Drawmean

•  ELK–  PipelineofthreeopensourceprojectsElas1csearch,Logstashand

Kibana–  Takedatafromanysource,anyformat–  Search,analyze,–  Visualizeitinreal1me

Drawmean

•  Logstash:3stagespipeline–  Inputs-Responsibleforcollectthedatafromvarioussourcessuch

aslogs,syslogfiles,shippers,beats,redis,…–  Filters–makescondi1onaltransforma1ons(grok,drop,clone,

1me,geoip,…)like•  parsing–usingregularexpressions–,•  addingfieldsortags,•  formaJngstructureslikeJSON,…

–  Outputs-sendsthedatatovariousdes1na1ons,likeaElas1cSearchcluster,filesortoolslikegraphiteornagios

Drawmean

•  Elas1csearch:RESTfuldataindexerandsearchengine–  Solu1ontomakesearchesandanalysisonasetofdataviaaquery

language–  Addressesstructured,unstrucureddataand1meseries–  Distributed,reliableandscalable–  Easymanagement(elas1csearch-headtools)–  LeveragesonLucene

Drawmean

•  Kibana:web-basedlightandeasy-to-usedashboardtool–  QueriesElas1csearchindexesanddiscoversdata–  Letyouinteractwithdataviagraphics(selec1ons,filters,1me

zooming–  Histograms,piecharts,1meseriesandevengeomapscreatedand

populatedonthefly–  Visualsthatcanbecombinedintodashboards–  SecuredaccessviaShieldplugin

Drawmean

Drawmean

Beats,tcp,h^p,udp,jmx,puppet…

Jdbc,Couchdb,Sqlite,Redis,ES…

Kaua,Rabbitmq,Pipe,…

File,Log4j,…

55outputplugins…

50inputplugins…

PuJngitalltogether

APIMgtUsageDataPublisher

+publishEvent(RequestPublisherDTO)+publishEvent(ResponsePublisherDTO)+publishEvent(FaultPublisherDTO)

APIMgtUsageLogPublisher

+publishEvent(RequestPublisherDTO){log.info(datafromRequestPublisherDTO);}…

RequestPublisherDTO

Requeststreamdefini=on

ResponsePublisherDTO

Responsestreamdefini=on

FaultPublisherDTO

Faultstreamdefini=on

APIMgtFaulHandler

+mediate(MessageContext)

APIMgtUsageHandler

+handleRequest(MessageContext)

APIMgtResponseHandler

+mediate(MessageContext)

LOGPublisherimplementa1on

31

Backen

d

BAMmediator

Workflowexecutor

In

APIMgtUsageHandler

Out

APIMgtResponseHandler

Fault

APIMgtFaultHandler

APIMgtUsageLogPublisher

mediate(mc)

mediate(mc)

mediate(mc)

publishEvent(DTO)

APIGateway(ESBinside)

Publishsta1s1cs

Usagepublisher

RequestPublisherDTO

statslogfiles

PuJngitalltogether

Logstashoutputw/sta1s1cs

PuJngitalltogether

PuJngitalltogether

PuJngitalltogether

PuJngitalltogether

PuJngitalltogether

•  Easyuh?–  Oneclasstocreate–  Oneconfigura1onlinetomodify–  Somefunwith–  That’sit

•  Let’strysomethingmorechallenging

Onestepbeyond

Onestepbeyond

•  WhatifIwanttoenrichtheprovidedsta1s1cswithmyownones?–  Nocorrela1onbetweenrequestandresponse–  Nospaceforyoubeloveddata

•  Well,let’srestartfromtheclassdiagram

APIMgtUsageExtendedLogPublisher

+publishEvent(ExtendedRequestPublisherDTO)+publishEvent(ExtendedResponsePublisherDTO)+publishEvent(ExtendedFaultPublisherDTO)

ExtendedRequestPublisherDTO

Requeststreamdefini=on

ExtendedResponsePublisherDTO

Responsestreamdefini=on

ExtendedFaultPublisherDTO

Faultstreamdefini=on

APIMgtFaulLogHandler

+mediate(MessageContext)

APIMgtUsageLogHandler

+handleRequest(MessageContext)

APIMgtResponseLogHandler

+mediate(MessageContext)

ExtendedLogPublisher

RequestPublisherDTO

Requeststreamdefini=on

ResponsePublisherDTO

Responsestreamdefini=on

FaultPublisherDTO

Faultstreamdefini=on

Extrafieldstolog•  Correla1onId•  BusinessData

+getextravaluesfromthe

mc

Exis1ngsta1s1csdata

41

Backen

d

BAMmediator

Workflowexecutor

In

APIMgtUsageLogHandler

Out

APIMgtResponseLogHandler

Fault

APIMgtFaultLogHandler

APIMgtUsageExtendedLogPublisher

mediate(mc)

mediate(mc)

mediate(mc)

publishEvent(DTO)

APIGateway(ESBinside)

Publishsta1s1cs

Usagepublisher

ExtendedRequestPublisherDTO

WSO2APIMlogfiles

APImedia1onisimpacted

Onestepbeyond Request

Response

Logstashoutputw/businessdataand

correla1onid

Logstashoutputw/sta1s1cs

Onestepbeyond

FAQ

FAQ

•  WhatifIwanttopopulatetheWSO2APIMsta1s1cstablesanyway?–  DASapproach,–  DIYapproach

•  WhatifIdon't(wantto)usetheELKstack?•  Whataboutthe1.xversions?•  WhatifIrepublishtheAPI(“onestepbeyond”case)?•  Whatabouttheoverhead?

Almostforgot!

Almostforgot

•  WSO2PreferredPartner•  WSO2TrainingFrance•  UsingWSO2solu1onsSince2013•  BarcelonaWSO2Con2014speaker•  SOAArchitects

co-authorsofthebook“SOA–aguideforarchitects”

WSO2APIM:opensource…andopenminded

WSO2APIM:opensource…andopenminded

•  WSO2APIMistherighttooltomanageyourAPIs.•  MostAPIToolshavetoconnecttoanexis1ngecosysteminplace•  WSO2APIMhasprovenitscapabili1estoembracereallifecases

andtoadapt.•  Extensioncapabili1esofWSO2APIMisunchallengedand"part

ofitsstrength”•  An“elas1c”solu1on…inaway

ThankYou!

#WSO2ConEU

Shareyourfeedbackforthissessionwso2con.com/app