elasticsearch - meetupfiles.meetup.com/7429482/meetuparchitecturepresentation.pdf · •...

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

Elasticsearch

Presented by: Steve Mayzak, Director of Systems Engineering Vince Marino, Account Exec

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedCopyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited

What about Elasticsearch the Company?

!

• Support 100s of Companies in Production environments

• Training Developers and Ops around the world on ELK

• Drive the ELK Projects forward, great things to come!

• Commercial products: Marvel to monitor and manage ELK !

• Backed by the best: Benchmark, Index Ventures, NEA


Little known facts

• Elasticsearch now makes over 70% of the Lucene commits

• Elasticsearch now employs over 100 people in the US, UK, Netherlands, Germany, Poland, Romania, Czech R., Italy, Japan, Canada, France, Spain, ?

• Elasticsearch powers Wikipedia.en search and the over 80 million sites on wordpress.com

http://wordpress.com


Data From Any Source

Instantly Analyze

Actionable Insights

The Elasticsearch ELK Stack

Logstash Elasticsearch Kibana


MarvelMonitor and Manage your Elasticsearch Cluster

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedCopyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibitedCopyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited

Who’s using Elasticsearch?


Search as Navigation• Update searches in real time based on user content • Roll out new features early and often automatically


Start smallES Node

curl –XGET ‘http://localhost:9200/'

{ "status" : 200, "name" : "Turner Century", "version" : { "number" : "1.3.1", "build_hash" : "2de6dc5268c32fb49b205233c138d93aaf772015", "build_timestamp" : "2014-07-28T14:45:15Z", "build_snapshot" : false, "lucene_version" : "4.9" }, "tagline" : "You Know, for Search" }


Add your app

ES NodeYour AppREST over http


Double in size

ES Node

Your App

ES Node

What about load balancing? Cluster


java

JS

Python

.NET

Perl

Ruby

PHP

Clients

ES Node

Your App

ES Node

Cluster


java

JS

Python

.NET

Perl

Ruby

PHP

Scale your app

ES Node

Your App

ES Node

Cluster

Your App

Your App


java

JS

Python

.NET

Perl

Ruby

PHP

Scale the cluster

Your AppES

Node

Cluster

Your App

Your App

ES Node

ES Node

ES Node


java

JS

Python

.NET

Perl

Ruby

PHP

Node Roles

Your AppMaster Node

Cluster

Your App

Your App

Data/Master Node

Data Node

Data Node


java

JS

Python

.NET

Perl

Ruby

PHP

Variations

Your AppMaster Node

Cluster

Your App

Your App

Data/Master Node

Data Node

Data Node

Client Node

Client Node

Servers 1,2 Servers 3,4,5,6


Log Analytics• Index Billions of events per day • Offer Log Analytics as a service to over 3500 customers


Logs,!Packets,!Events,!

Transactions,!!

Timestamp Data

Logstash

What about Logging Analytics?

ES Node

Kibana


Logstash

Scale up a bit

Logstash Forwarder

ShippersShippers

Logstash Forwarder

LogstashMaster Node

Cluster

Data/Master Node

Data Node

Data Node

Kibana


Scale up some more

Logstash Forwarder

ShippersShippers

Logstash Forwarder

Logstash

Master Node

Cluster

Data/Master Node

Data Node

Data Node

Logstash Forwarder

ShippersShippers

Logstash Forwarder

Logstash

Logstash

Logstash

Logstash

Brokers!!

Redis!!

RabbitMQ!!

Kafka

Logstash

Logstash

Kibana


Process as a Stream

Brokers

Log

Ship

Log

Ship LogstashNode

Cluster

Node

NodeNode

Logstash

Logstash

Brokers

Storm!!

Spark!!

Samza

Kibana


Sentiment Analysis• Index and search over millions of social interactions every day • Measure sentiment changes as they happen


Where does Hadoop fit in?

Brokers

Log

Ship

Log

Ship LogstashNode

Cluster

Node

NodeNode

Logstash

Logstash

Brokers

Storm!!

Spark!!

Samza

Kibana

HDFSFlume

es-hadoop


Scaling• Which hardware do I buy?

• Physical vs Virtual

• Sizing a shard - Distributed systems knowledge applies

• Dedicated Masters, Minimum Masters

• Replica - Pros and Cons


Common Pitfalls• Not fencing network, locking down Elasticsearch

• Messing with defaults blindly

• Shoehorn relational model into NoSQL

• The ins and outs of Upserts

• Distributed systems knowledge and architecture a must


Q&A

elasticsearch - meetupfiles.meetup.com/7429482/meetuparchitecturepresentation.pdf · •...

Documents