log mining

D i g i n s i g h t

LOG MINING

fanjiang@thoughtworks.com

https://github.com/tcz001

TECH RADAR TREND

2

structured-logging

什么是LOG？

3

> tail -f /usr/local/log

INFO [2014-11-13 12:23:36,173] com.thoughtworks.forcetalk.resources.ContactResource: Updated Contact {"FirstName":"Alper","LastName":"Mermer","Employee_ID__c":"16906","Email":"amermer@thoughtworks.com","Grade__c":"Senior Consultant”}ERROR [2014-11-13 11:45:33,892] com.thoughtworks.forcetalk.validators.ForceQueryResultsValidator: Unable to retrieve Project for Opportunity with id: 0065000000TE2evAADINFO [2014-11-13 12:23:36,505] com.thoughtworks.tetalk.resources.UserResource: Contact Update Response SObjectResponse{successful=true, id='null', errorMessage='null', errorField='null', errorCode='null'}

INFO 2014-11-13 12:23:36,173com.thoughtworks.forcetalk.resources.ContactResource

ERROR

什么是好LOG？

4

▫ http://juliusdavies.ca/logging/llclc.html

Best Logs:▫ Tell you exactly what happened: when, where, and how.▫ Suitable for manual, semi-automated, or automated analysis.▫ Can be analysed without having the application that produced them at hand.▫ Don't slow the system down.▫ Can be proven reliable (if used as evidence).

Avoid Logs:▫ Missing necessary information.▫ Unsuitable for grep because of redundant information.▫ Information split across more than one line (bad for grep).▫ Error reported to user, but not logged.▫ Never include any sensitive data.(for Security !).

DEVOPS的故事

5

> rm -rf ALL_THE_LOGS

DEVOPS的故事

6

We got an angry User! HELP!

BE REACTIVE

7

MONITOR IS FAR FROM “TOP”

8

SAVE OUR LIFE

9

?

SAVE OUR LIFE

10

Splunksaas

LogStashopensource

OR

SAVE OUR LIFE

11

SAVE OUR LIFE

12

WHAT TIME IS IT?130406050529/Apr/2011:07:05:26 +0000Fri, 21 Nov 1997 09:55:06 -0600Oct 11 20:21:47020805 13:51:24 110429.071055,118@4000000037c219bf2ef02e94

DATE FILTER FIXES THIS BULLSHIT

filter {date {# Turn 020805 13:51:24# Into 2002-08-05T13:51:24.000Zmysqltimestamp => "YYMMdd HH:mm:ss"

}}

SAVE OUR LIFE

13

> 23 INPUTS | 18 FILTERS | 40 OUTPUTS

不只是timestamp

▫ LogLevel▫ Source▫ IP=> GeoHash▫ Browser/Platform

SAVE OUR LIFE

14

Logstash-server

input {lumberjack {# The port to listen onport => 5043

# The paths to your ssl cert and keyssl_certificate => "./logstash.crt"ssl_key => "./logstash.key"

# Set this to whatever you want.type => "finance"

}}

filter {if [type] == "finance" {grok {match => [ "message",

"%{LOGLEVEL:loglevel}\s+\[%{TIMESTAMP_ISO8601:timestamp}\] (?<source>(\w|\.)+): (?<msg>(.*))" ]

add_tag => [ "grokked" ]}date {match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss,SSS" ]

}}

}

output {if "_grokparsefailure" not in [tags] {stdout { codec => rubydebug }elasticsearch { host => localhost }

}}

Logstash-forwarder

"network": {"servers": [ "localhost:5043" ],

"ssl ca": "./logstash-forwarder.crt"},"files": [{"paths": ["/usr/local/finance/**/logs/*.log"

],"dead time" : "8760h","fields": { "type": "finance" }

}]

All Our Services

ElasticSearch Clusters

ELASTICSEARCH

15

▫ Restiful API search engine▫ Multi-cluster supported▫ Great community▫ Use it! throw things into it!

ElasticSearch+

Kibana

DIGGING DEEPER

16

curl -XGET 'http://localhost:9200/logstash-*/_search?pretty&search_type=count' -d '{"aggregations": {“source-aggregation”: {"terms": {"field": "source","size": 1000

}}

}}'

Try it!

DIGGING DEEPER

17

http://localhost:8000/

Zoomable Treemap for diging into Logs via source

By Elasticsearch aggregation API

LEARN FROM LOG

18

treat Log as Statistical Data

AUTO REACTIVE

19

Be Responsive to every Exception

OTHER POSSIBILITY

20

Q&A

Thanks~

21