gather those events : instrumenting everything for analysis
DESCRIPTION
About WSO2CEP and WSO2BAM on WSO2Con 2014 Europe BarcelonaTRANSCRIPT
Gather those Events Instrumenting Everything for Analysis
Sriskandarajah Suhothayan (Suho)Technical Lead
WSO2 Inc.
Connected Business !
Connected Business !
Connected Business !
● Services ● Systems ● Data ● APIs ● Cloud ● Mobile ● Workflows ● ….
Connected Business !
● on-premise / cloud ● From WSO2/ 3rd party
geographical locations ● using various protocols &
message formats
Issues
Issues
Issues
Issues
Issues
Issues
Issues
Monitor ..!
Present
!
Analyse ..!
Present
!
Analyse !!
!
Gather Info ..!
Present
!
Analyse !!
!
Gather !!
Gather Info ..!
Present ! !! ! Gather
!!
Having proper data @ proper place
@ proper time !
Gather Info ..!
Present ! !! ! Gather
!!
Having proper data @ proper place
@ proper time ! !
What is Proper ?
Do we know what proper is ?
Do we know what proper is ?
YES !
We captor them direct them for analysis & write queries to produce alerts using !
&
NO & Maybe !
?
Lost !
http://aroundtheworldineightywaves.com/wp-content/uploads/lost.jpg
NO & Maybe !
?
How we can solve ?
http://aroundtheworldineightywaves.com/wp-content/uploads/lost.jpg
NO & Maybe !
?
How we can solve ?
http://aroundtheworldineightywaves.com/wp-content/uploads/lost.jpg
How we gather ?
Gather !!
Analysis
Real time Post
Real time √ √
Batch x √
How We Gather ?
Gather !!
Strategy ...
From Where We Start ?
• There is always some sensitive parts of infrastructure Which we have to monitor !
• We can enrich them…. !
• When we have an infrastructure to analyse data, Add some more data
From where we gather?
From where we gather?
From where we gather?
From where we gather?
From where we gather?
From where we gather?
From where we gather?
From where we gather?
From where we gather?
From where we gather?
From where we gather?
From where we gather?
From where we gather?
From where we gather?
!! Is it only from WSO2 Servers ?
From where we gather?
!
From where we gather?
!
From where we gather?
!
From where we gather?
!
From where we gather?
!
From where we gather?
!
From where we gather?
!
From where we gather?
!
From where we gather?
!
From where we gather?
!
From where we gather?
!All
From where we gather?
!All
All these from Custom Data Publishers !
They can do … !
• Asynchronous • Auto retry • Auto recovery • Load-balancing • Fail-over
How good the Data Publishers are ?
!! Only from Thrift ?
How we gather?
!!!!!
How we gather?
!!!!!
How we gather?
They have many forms
• XML • JSON • Text • Map • WSO2 Event
How we gather?
!!!!!
How we gather?
!!!!!
!
How we gather?
!
How we gather?
!
Performance
!!
• 1.8M Events/Sec on same JVM • About 250K over network for single node !!!
• 30K over Network to 3 node Casandra cluster
How we manage ?
•Data is in different formats •Converted to a common format •Virtualised as Streams of Events •Events Stream has
Name : phone.retail.shop Version : 1.0.0 ! & also they have a Definition!
Stream Definition
{ 'name':'phone.retail.shop', 'version':'1.0.0', 'nickName': 'Phone_Retail_Shop', 'description': 'Phone Sales', 'metaData':[ {'name':'clientType','type':'STRING'} ], 'correlaitonData':[ {'name':’transactionID’,'type':'STRING'} ], 'payloadData':[ {'name':'brand','type':'STRING'}, {'name':'quantity','type':'INT'}, {'name':'total','type':'INT'}, {'name':'user','type':'STRING'} ] } !!
The Big Data View
HA / Scalability of
HA / Scalability of
HA / Scalability of
HA / Scalability of
HA of
!
HA of
!
What gets measured, gets managed. !
~ Peter Drucker ~
Thank You