operational intelligence with wso2 bam
DESCRIPTION
TRANSCRIPT
Gaining Opera+onal Intelligence �with WSO2
BAM
Director, Research, WSO2 Inc. Visi6ng Faculty, University of Moratuwa Member, Apache SoEware Founda6on
Research Scien6st, Lanka SoEware Founda6on
Srinath Perera
About WSO2 ๏ Global enterprise, founded in 2005
by acknowledged leaders in XML, web services technologies, standards and open source
๏ Provides only open source plaHorm-‐as-‐a-‐service for private, public and hybrid cloud deployments
๏ All WSO2 products are 100% open source and released under the Apache License Version 2.0.
๏ Is an Ac+ve Member of OASIS, Cloud Security Alliance, OSGi Alliance, AMQP Working Group, OpenID Founda+on and W3C.
2
๏ Driven by Innova+on
๏ Launched first open source API Management solu+on in 2012
๏ Launched App Factory in 2Q 2013
๏ Launched Enterprise Store and first open source Mobile solu+on in 4Q 2013
3
What WSO2 delivers
Business Model
4
More Informa+on !
๏ Include links to product downloads, white paper downloads , etc.
5
Outline § Connected Business and Big data analy+cs § Big Data Technologies from WSO2
§ BAM – Batch analy+cs § CEP – Real +me analy+cs § Lambda Architecture to combine
§ From your business to insights § Iden+fy KPIs § Calculate KPIs § Build Dashboards § Add Drilldown § Alerts and Ac+ons
About Connected Business webinar series
§ March 05:Introduction to the Connected Business
§ March 12: Enterprise Integration made easy with WSO2 ESB
§ .. § April 30: Gaining
Operational Intelligence with WSO2 BAM
§ May 7: The WSO2 Advantage for a Connected Business
Adap+ve Connected Business
Connected Business
Be Adap+ve ๏ Capture business ac+vity (iden+fied by messages, transac+on execu+on, and data state changes) and store data points for future analy+cs
๏ Deliver automated no+fica+ons to stakeholders and systems based on business ac+vity, stakeholder accountability, and authority.
๏ Automa+cally adapt business process execu+on based on events and current condi+ons
Big Picture
Collec+ng Data ๏ Data collected at sensors and sent to big data system via events or flat files
๏ Event Streams: we name the events by its content/ originator
• Get data through – Point to Point – Event Bus
• E.g. Data bridge – a thrib based transport we did that do about 400k events/ sec
Making Sense of Data ๏ To know (what happened?)
๏ Basic analy+cs + visualiza+ons (min, max, average, histogram, distribu+ons … )
๏ Interac+ve drill down
๏ To explain (why)
๏ Data mining, classifica+ons, building models, clustering
๏ To forecast
๏ Neural networks, decision models
Dashboards and last Mile § Presenting information
o To end user o To decision takers o To scientist
§ Interactive exploration § Sending alerts
http://www.flickr.com/photos/stevefaeembra/3604686097/
Big Data Architecture
Data Collec+on
• Can receive events via SOAP, HTTP, JMS, ..
• WSO2 Events is highly op+mized version (400K events TPS)
• Default Agents and you can write custom
agents.
Agent agent = new Agent(agentConfiguration); publisher = new AsyncDataPublisher(
"tcp://localhost:7612", .. ); StreamDefinition definition =
new StreamDefinition(STREAM_NAME, VERSION);
definition.addPayloadData("sid", STRING); ... publisher.addStreamDefinition(definition); ... Event event = new Event(); event.setPayloadData(eventData); publisher.publish(STREAM_NAME, VERSION, event);
Business Ac+vity Monitor
BAM Hive Query Find how much +me spent in each cell.
CREATE EXTERNAL TABLE IF NOT EXISTS PlayStream …
select sid,
ceiling((y+33000)*7/10000 + x/10000) as cell, count(sid)
from PlayStream
GROUP BY sid, ceiling((y+33000)*7/10000 + x/10000);
Complex Event Processor
CEP Query define partition sidPrt by PlayStream.sid, LocBySecStream.sid
from PlayStream#window.timeBatch(1sec)
select sid, avg(x) as xMean, avg(y) as yMean, avg(z) as zMean
insert into LocBySecStream partition by sidPrt
from every e1 = LocBySecStream ->
e2 = LocBySecStream [e1.yMean + 10000 > yMean
or yMean + 10000 > e1.yMean]
within 2sec select e1.sid
insert into LongAdvStream partition by sidPrt ;
Calculate the mean location of each player
every second
Detect more than 10m run
Lambda Architecture
Extrac+ng Opera+onal Intelligence from your
Business
Key Performance Indicators (KPIs) ๏ To monitor, you need to measure
๏ KPIs are the bojom line
๏ For a organiza+on it may be profit, or revenue
๏ For marke+ng, it may be LEADs generated
๏ They are oben domain specifics
๏ For each KPI, we need to see history, find rela+onships, and some+me possible future trends.
Many Stake Holders, Many Views
Decision Process
Dashboard
๏ Idea is to get the “Overall idea” in a glance
๏ Like your car dashboard
๏ Support for personaliza+on, you can build your own dashboard.
๏ This is the entry point for Drill down and ac+vity monitoring.
Drill Down
๏ Find the problem, and find what caused it.
๏ It is like “Finding CPU bojleneck”
๏ Idea is to recursively find categories which are responsible for the problem
๏ E.g. See year’s sales, then drill down to a par+cular year to see per month data.
๏ See country level data, and drill down to see state wide data.
๏ Ac+vity monitoring is a key ac+vity here
Case Study: Sobware Support Services
๏ KPIs
๏ Number of open Issues
๏ Mean and Max +me to close a issue
๏ SLA viola+ons
๏ List of long running issues
๏ Issue by last update +me
๏ Data
๏ Collect data when issue is created or updated or closed.
๏ E.g. IssueChange(id, ac+on, +me, userID)
๏ Emit ID as the correla+onID
From data to KPI ๏ Group issues by ID grouping each issue into one ac+vity (Ac+vity Monitoring)
๏ Calculate KPIs for each ac+vity
define partition IssueParition IssueChange.id;
from IssueChange[type=“open”] as s
->IssueChange[“closed”] as e
select e.time-s.time ..
Aggregate data in different level to support drill down
๏ hour, weekly, and monthly
๏ Ver+cal, product,
Aggregate to support Drilldown
Dimension 1 Dimension 2
Dashboard (Build with UES)
Click to drill down
Click to drill down
Click to see details about Issues
Drill Down
๏ Click and drill down issues in each category
๏ Can select a issue and see all ac+vi+es happened within the issue
๏ See who takes most +me on issues
๏ Look at day, week, month trends
Analy+cs Process
Ques+ons?
THANK YOU!!!
37