advanced monitoring with complex stream processing it-tf 29-05-2015... · 2016-07-18 · 29/05/2015...
TRANSCRIPT
![Page 1: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/1.jpg)
Advanced Monitoring With
Complex Stream Processing
Magnoni Luca, Cons Lionel IT-SDC
Reguero Ignacio IT-PES
IT Technical Forum
![Page 2: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/2.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Few words about myself
• Software Engineer
• Experience with monitoring projects
• Worked@PH/ATLAS-TDAQ• Automated/Reactive monitoring at speed
• Work@IT/SDC-MI• Monitoring WLCG with lambda-architecture
• Advanced Monitoring for Messaging services
2
![Page 3: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/3.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Outline
• Messaging service and monitoring
• Technology investigation
• Building a solution
• Summary
3
![Page 4: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/4.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Messaging service @ CERNRun
Services
Provide
Additional Tools
Offer
Advanced
Monitoring
Producer
Consumer
ConsumerMessaging service
4
![Page 5: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/5.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Messaging infrastructure16 Physical,
9 Virtual
Machines
Host
Broker
Broker
Broker
Client
Client
Broker
Host Host
Broker
Broker
ClusterBroker
23 Brokers
10 Clusters
157 Clients
up to 150 Millions
msg/day
5
![Page 6: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/6.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Monitoring: measure everything
Broker
Client
Client
Broker
Host
6
host.syslog
host.iptables
host.SELinux
host.cpu
host.memory
host.swap
…
broker.cpu
broker.heap
broker.connections
broker.received_msg
broker.sent_msg
…
client.received_msg
client.sent_msg
client.stored_msg
broker.stomp
broker.http
broker.httpd
broker.activemq
Log
Metric
Checks
![Page 7: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/7.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing 7
![Page 8: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/8.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Monitoring: measure everything
8
![Page 9: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/9.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
We have a situation… or not?
9
Number of
messages
not consumed
(stored) for a
specific client on
different brokers
YESBetter call
someone!
![Page 10: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/10.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
We have a situation… or not?
10
NO Everything is
fine.
Number of
messages
not consumed
(stored) for a
specific client on
different brokers
![Page 11: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/11.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
We have a situation… or not?
11
NO Everything is
fine.
Number of
messages
received from
a specific client
on different
brokers
![Page 12: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/12.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
We have a situation… or not?
12
YESNumber of
messages
received from
a specific client
on different
brokers
![Page 13: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/13.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
We have a situation… or not?
13
YESUnbalanced
use of
brokers in
a cluster.
Number of
network
connections
on different
brokers
![Page 14: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/14.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
What we need
• Detect problems from metrics, logs and
checks
• Notify users when something is not ok
(but also when it goes back to ok)
• Automate as much as possible
14
![Page 15: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/15.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Monitoring is complicated
• Information is scattered: value not in the single event but in the overall system behavior over time
• Events have to be correlated and aggregated
• Fault diagnosis requires competence and experience
15
![Page 16: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/16.jpg)
Technology investigation
![Page 17: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/17.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Quest for technology• Stream Processing frameworks
• Storm, Spark Streaming, Akka, Samza, etc.
• emphasis on streaming high data volumes
• basic analytic primitives, derived from batch processing
• Quite surprisingly, not much from the monitoring world
• Process streams of information at high rate with complex time-based analysis is not new• financial analysis, sensor networks, business intelligence,…
17
![Page 18: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/18.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Complex Event Processing (CEP)• A.k.a Event Stream Processing (ESP) / SQL streaming analytics
• A set of technology to process and discover complex patterns among streams of events
• A cross between DBMS and Rule Engines • formalized by Prof. D.Luckham (Stanford) in 2002
• specialized system for special purpose (Prof. Stonebraker/StreamBase)
• Several providers (commercial & open source) • Oracle, IBM, TIBCO, Drools Fusion, Siddhi, EsperTech
• Functional properties• stream analysis with Domain Specific Language (SQL-like or custom)
18
![Page 19: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/19.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
CEP: continuous-processing model
19
To a data-driven, in-memory, on-the-fly
analysis
continuous query evaluation model
Move from a traditional pull-based store-and-
analyze model
data stores grows permanently, search
becomes expensive
![Page 20: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/20.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
CEP: a database upside/down
SQL
Store the data.
Compute on demand.
~ SQL
Store the query.
Continuous computation.
select count(*)
from circle
group by color;
20
select count(*)
from circle.win:time(1 hour)
group by color;
![Page 21: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/21.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Esper: open source CEP/ESP engineEsper [...] provide a highly scalable, memory-efficient, in-memory computing, SQL-standard, minimal latency, real-time streaming-capable Big Data processing engine for historical data, or medium to high-velocity data and high-variety data.
• GPL, with commercial support and extensions
• strong community, support and documentation
• long list of big customers (new entry: Ebay with Pulsar)
• in use at CERN:• PH/ATLAS TDAQ: Intelligent monitoring and real-time analytics for ATLAS-TDAQ: a CEP-based solution
• BE/Control: Surveillance of CERN Accelerator Logging Processes with CEP and Esper
21
![Page 22: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/22.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Esper• Lightweight Library
• Java, standalone or embeddable
• input-output connectors: Java API, HTTP, JMS, AMQP, DB, File
• data format: JSON, XML, custom
• supports custom adapter
• strong performance (500000 events/s with 3µs avg latency)
• Feature rich SQL-likeEvent Processing Language
(EPL)
22
![Page 23: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/23.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Esper analytic features
23
Data windowsto instruct the engine on how long to retain events.
(Time or length based, with sliding or batch
behavior)
Patternsfor pattern matching based on logical and temporal
event correlation, or regular expression. (Match-
recognize, timer-control patterns)
Event analysisgrouping, aggregation, rollup, cube, sorting, filtering,
transforming, merging, splitting or duplicating of event
series or streams.
Pluggable functionsfor event pattern and event stream analysis via user-
defined functions, plug-in views, plug-in aggregation
functions. (Java callback, Script support, Lambda
functions)
Time Controlsupports externally-provided time as well as current system
time, allowing applications full control over the concept of time
within an engine and full control over which thread(s) evaluate
timer schedule for queries.
Joining dataacross stream or with external sources, for
easy integration with external data sources such
as web services and plain DB.
![Page 24: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/24.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
EPL: event selection
• How do I look for specific events on a stream, dropping
the unwanted events?
• How do I aggregate several (simple) events from a
stream into a single new (complex) event summarizing
event properties of the simple events?
24
select * from SensorEvent where temperature > 90
select avg(temperature) from SensorEvent
![Page 25: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/25.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
EPL: data windows and patterns
• How can I detect N events within X seconds?
25
select * from MyEvent(type=ERROR).win:time(3 min)
having count(*) >= 5
• How do I use patterns to correlate events?
select * from pattern [every s=StartEvent ->
a=EndEvent(id = s.id)]
• More examples from Esper’s “Solution Patterns”
![Page 26: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/26.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
EPL programming model:
pipes and operators
26
Streams
EPL statementsInput
channel
Output
channel
![Page 27: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/27.jpg)
Building a solution
![Page 28: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/28.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Messaging Monitoring model
28
Metric
Log
Check[ OK | WARNING | CRITICAL
|UNKNOWN ]
(Active or Derived)
StatusService state handling,
flapping, notification,…
[ OK | WARNING | CRITICAL
|UNKNOWN ]
* Deeply inspired by the Nagios model
![Page 29: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/29.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Messaging Monitoring architecture
Messaging
infrastructureStorage /
Gateways
Metis
29
Transport
layer
Log
Check
Metric
Status
![Page 30: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/30.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Metis
30
I
N
P
U
T
O
U
T
P
U
T
Esper
DIR
QJM
S
JM
SF
ILE
DIR
Q
REST/JMX
EPL statements
Spring framework
• Esper-based service
• Java & Spring
• Configurable Input/Output
channels to receive and to
report events
• Digests JSON-structured
monitoring data
• Implements the messaging
monitoring model
• EPL statements as monitoring
directives
• Run-time admin interface
![Page 31: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/31.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Detect a problem from a Metric
31
![Page 32: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/32.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Check generation: threshold
32
//
// Check load per host
//
insert into Check
select host as m.host,
case when m.value>95
then {STATE.CRITICAL, ’Load is too high:’ || m.value}
when m.value>90
then {STATE.WARNING, ’Load is high:’|| m.value}
when m.value>=0
then {STATE.OK, ’Load is ok’}
end as result
from Metric(name=‘load’) as m;
• Check = f (Metric)
![Page 33: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/33.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Detect a problem from multiple Metrics over time
33
![Page 34: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/34.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Metric transformation: time-aggregation and grouping
34
//
// Avg stored msg per broker over sliding 1 hour
//
insert into Metric
select
m.host as host,
m.broker as broker,
’stored_avg' as name,
avg(m.value) as value
from Metric(grp='msgbrk’, name=’stored')
.win:time(1 hour) as m
group by host, broker;
• Metric = f (Metric)
A C B
A A A A
B B B B
C C C C
avg(A) avg(B)
1 hour
![Page 35: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/35.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Metric transformation: counter to rate
35
Plot shows
client
messages
received
(counter)
by
broker
![Page 36: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/36.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Metric transformation: counter to rate
36
Plot shows
client
messages
received
(rate)
by
broker
![Page 37: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/37.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Counter to rate operator
• Metric = f (Metricx, Metricx-1)
37
//
// EPL Lambda function to compute rate
//
create expression computeRate
{ m =>
(m.value - prev(1, m.value)) /
(m.timestamp - prev(1, m.timestamp))
};
BCAAx-1
Ax
![Page 38: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/38.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Hysteresis operator (e.g. WARNING >90 and OK <60)
• Check = f (Metric, Check)
38
//
// Script to compute Hystheresis thresholds
//
create expression double mvel:hyst(target_state,
previous_state, up_threshold, down_threshold) [
if (target_state > previous_state)
return up_threshold;
else
return down_threshold;
];
BCAAx-1
Ax
![Page 39: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/39.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Log as first-class citizen
• Check = f (Log)
• Additional feature:
log parsing
• Log = f (Log)
• Grok library from Logstash
• Re-use Logstash patterns
39
//
// Log parsing using via
// the Grok library
//
insert into StructuredLog
select MetisUtils.GrokParse(log.text)
from Log(name=“syslog”) as log;
![Page 40: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/40.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Status
• Status = f (Check, Status)
• State handling
• Validity
• Flapping
• Notification
40
//
// Check Validity.
// CHECK goes to UNKNOWN after 30
// minutes without new check event
//
insert into Check
select
check.*, STATE.UNKNOWN as value,
"No new check since a while..." as text
from
pattern [ c = Check -> (timer:interval(30 min)
and not c_up=Check(FID=c.FID))];
![Page 41: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/41.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Metis: project status• In production since December 2014
• Data types/instances:
41
6 formats
276 instances
Logs
97 types
6179 instances
Metrics
16 types
673 instances
Checks
• Data-rate:
100 metrics/second 10 logs/second 13 checks/second
![Page 42: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/42.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Metis: features summary
42
• Metric transformation
• grouping and aggregation
across time and brokers
• counter/rate
• Check generation
• threshold
• hysteresis
• patterns
• Status inference
• Notification logic
• when to report to users
and what
• EPL templates as
monitoring directives
• fine grain configuration
![Page 43: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/43.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Does it work?
43
Yes
![Page 44: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/44.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Does it work?
44
Yes
![Page 45: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/45.jpg)
Summary
![Page 46: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/46.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Lessons learned
• CEP: a powerful and versatile technology
• consider CEP when aggregation and
correlation over time is a key feature in your
analysis
• EPL/SQL-like language makes complex
analysis easier
46
![Page 47: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/47.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Lessons learned• Esper is good at what it does
• but needs several things to build a service
• Spring is good to glue all these things together
• Esper has a quite steep learning curve• but this pays off in the end
• Esper features allow for very advanced monitoring rules• getting close to intelligent monitoring
• A flexible transport layer eases integration and testing
47
![Page 48: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/48.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Lessons learned: test everything
48
• Unit testing is crucial
• Esper allows to
control time (wow!)
• Replay old events
• Mock time-based
sequence
• Run development
Metis (e.g. from git
checkout) on the real
data flow on the
preferred IDE…
![Page 49: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/49.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Consideration on scalability
• Metis runs on a modest VM (4 cores/8GB)
• ~ 80 MB Heap, ~ 6% CPU
49
![Page 50: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/50.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Consideration on scalability • Dimension your system to make analysis,
not to store data • Vertical
• EPL processing optimized (decision tree) with fine grain control on multi-threading
• Horizontal• Multiple instances (stream partitioning)
• Ebay/Pulsar architecture (with Kafka and Zookeeper)
• Esper within streaming framework:• Esper/Storm and Esper/Akka are working example
50
![Page 51: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/51.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Roadmap
• GNI/ServiceNowintegration
• Roger integration
• State preservation
• Shadow instance
• Do more with logs
51
©m
e
![Page 52: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/52.jpg)
29/05/2015 Magnoni L. Advanced monitoring with complex stream processing
Summary
• We have an advanced monitoring system
• used by us and by our clients
• Esper has proven to be a good choice
• The same approach could be useful to other services
• Metis provides a generic monitoring model with a modular architecture
52
![Page 53: Advanced Monitoring With Complex Stream Processing IT-TF 29-05-2015... · 2016-07-18 · 29/05/2015 Magnoni L. Advanced monitoring with complex stream processing Quest for technology](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed37174b458607d8231cc3a/html5/thumbnails/53.jpg)
Thank you!
Questions?