Monitor with the Stack

Philipp Krenn�����@xeraa

1

Infrastructure | Developer Advocate

2

Disclaimer

This is not a traininghttps://www.elastic.co/training

3

Who Is UsingElasticsearch

Logstash and KibanaBeats

4

5

6

7

Starting Pointhttps://github.com/xeraa/mongodb-monitoring

8

USB Sticks

9

BoxVagrant Ansible Provisioner

10

Credentialsvagrant & vagrant

11

SSH$ ssh vagrant@127.0.0.1 -p 2222 -o PreferredAuthentications=password

Windows: http://www.putty.org

12

Ansible

$ cd /elastic-stack/$ ls

13

14

REST

$ curl -XGET -u "elastic:changeme" http://localhost:9200/

15

16

Loginhttp://localhost:5601elastic & changeme

17

18

Filebeat

19

Filebeat Modules

20

System Dashboards

21

22

MongoDB Logs/var/log/mongodb/mongod.log

23

/etc/filebeat/filebeat.yml

filebeat.prospectors:- input_type: log paths: - /var/log/mongodb/mongod.log document_type: mongodb

24

PS: Multiline Logs

25

/etc/filebeat/filebeat.yml

filebeat.prospectors:

- input_type: log paths: - /var/log/java-app/*.log document_type: java multiline.pattern: '^[[:space:]]' multiline.negate: false multiline.match: after

26

Test

$ java -jar /opt/pocdriver/bin/POCDriver.jar -k 20 -i 10 -u 10 -b 20

27

Kibana DiscoverLimit Kibana view to the mongodb type

28

29

Grok Patternshttps://github.com/logstash-plugins/logstash-patterns-

core/blob/master/patterns/grok-patterns

30

MongoDB Patternshttps://github.com/logstash-plugins/logstash-patterns-

core/blob/master/patterns/mongodb

31

Building PatternsGrokdebug in Kibana 5.5+ (X-Pack Basic)

https://grokdebug.herokuapp.com

32

33

/etc/filebeat/filebeat.yml

output.elasticsearch: hosts: ["localhost:9200"] username: "{{ elastic_user }}" password: "{{ elastic_password }}" pipeline: "mongodb_log"

34

ConsolePattern in a single line

PUT _ingest/pipeline/mongodb_log{ "description": "Ingest pipeline for MongoDB logs", "processors": [ { "grok": { "field": "message", "patterns": [ "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{MONGO3_SEVERITY:mongodb.severity} %{SPACE}%{MONGO3_COMPONENT:mongodb.component}%{SPACE} (?:[%{DATA:mongodb.context}])?%{SPACE}%{GREEDYDATA:message}" ] } } ]}

35

Filebeat Restart$ sudo service filebeat restart

36

Test

$ java -jar /opt/pocdriver/bin/POCDriver.jar -k 20 -i 10 -u 10 -b 20

37

Refresh Field ListManagement → Index Patterns → filebeat-* → Refresh

field list

38

Visualizemongodb.component of log events

39

40

41

Fetch the Slow Log

42

/etc/logstash/conf.d/00-mongodb-input.conf

input { mongodb { uri => 'mongodb://127.0.0.1:27017/POCDB' placeholder_db_dir => '/var/local/logstash-mongodb/' collection => 'system.profile' batch_size => 500 generateId => true }}

43

/etc/logstash/conf.d/20-elasticsearch-output.conf

output { elasticsearch { hosts => ["localhost:9200"] manage_template => false index => "mongodb-%{+YYYY.MM.dd}" document_type => "slowlog" user => "{{ elastic_user }}" password => "{{ elastic_password }}" }}

44

Logstash Restart$ sudo service logstash restart

45

Debug Logstash

$ less /var/log/logstash/logstash-plain.log

46

Doesn't WorkGetting data from system collections

https://github.com/phutchins/logstash-input-mongodb/issues/8

47

Extension IdeaBuild an exec filter to run slow queries against

MongoDB's .explain()

48

Metricbeat

49

Metricbeat System

50

Metricbeat Service

51

/etc/metricbeat/metricbeat.yml

- module: mongodb metricsets: ["dbstats", "status"] hosts: ["localhost:27017"]

52

Metricbeat Restart$ sudo service metricbeat restart

53

Test

$ java -jar /opt/pocdriver/bin/POCDriver.jar -k 20 -i 10 -u 10 -b 20

54

55

Visual Buildermongodb.status.network.in.bytes vs mongodb.status.network.out.bytes

56

57

Packetbeat

58

Protocols

59

FlowsApplication layer: Unsupported / encrypted (TLS) protocols

IP / TCP / UDPNumber of packets & bytes

RetransmissionsTemporal flow

60

/etc/packetbeat/packetbeat.yml

packetbeat.protocols.mongodb: ports: [27017]

61

Packetbeat Restart$ sudo service packetbeat restart

62

Test

$ java -jar /opt/pocdriver/bin/POCDriver.jar -k 20 -i 10 -u 10 -b 20

63

64

Heartbeat

65

HeartbeatICMP, TCP, HTTP, HTTPS

66

/etc/heartbeat/heartbeat.yml

heartbeat.monitors:- type: tcp hosts: ["127.0.0.1:27017"] schedule: '@every 10s'

67

Heartbeat Restart$ sudo service heartbeat restart

68

Test

$ sudo service mongod stop$ sudo service mongod start

69

VisualizeUp or down and TCP response times

70

71

72

DashboardCombining visualizations

73

74

Winlogbeat

75

libbeathttps://github.com/elastic/beats/tree/master/generate/beat

76

77

X-PackSecurityMonitoringGraph

ReportingAlerting

Machine Learning

78

X-Pack Basic

79

Conclusion

80

81

82

83

Thanks!

Questions?Philipp Krenn�����@xeraa

84