elasticsearch - meetupfiles.meetup.com/7429482/meetuparchitecturepresentation.pdf · •...

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

Elasticsearch

Presented by: Steve Mayzak, Director of Systems Engineering Vince Marino, Account Exec

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibitedCopyright  Elasticsearch  2014.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

What about Elasticsearch the Company?

!

• Support 100s of Companies in Production environments

• Training Developers and Ops around the world on ELK

• Drive the ELK Projects forward, great things to come!

• Commercial products: Marvel to monitor and manage ELK !

• Backed by the best: Benchmark, Index Ventures, NEA

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

Little known facts

• Elasticsearch now makes over 70% of the Lucene commits

• Elasticsearch now employs over 100 people in the US, UK, Netherlands, Germany, Poland, Romania, Czech R., Italy, Japan, Canada, France, Spain, ?

• Elasticsearch powers Wikipedia.en search and the over 80 million sites on wordpress.com

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibitedCopyright  Elasticsearch  2014.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

Data From Any Source

Instantly Analyze

Actionable Insights

The Elasticsearch ELK Stack

Logstash Elasticsearch Kibana

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibitedCopyright  Elasticsearch  2014.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

MarvelMonitor and Manage your Elasticsearch Cluster

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibitedCopyright  Elasticsearch  2014.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibitedCopyright  Elasticsearch  2014.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

Who’s using Elasticsearch?

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibitedCopyright  Elasticsearch  2014.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

Search as Navigation• Update searches in real time based on user content • Roll out new features early and often automatically

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

Start smallES Node

curl –XGET ‘http://localhost:9200/'

{ "status" : 200, "name" : "Turner Century", "version" : { "number" : "1.3.1", "build_hash" : "2de6dc5268c32fb49b205233c138d93aaf772015", "build_timestamp" : "2014-07-28T14:45:15Z", "build_snapshot" : false, "lucene_version" : "4.9" }, "tagline" : "You Know, for Search" }

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

Add your app

ES NodeYour AppREST over http

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

Double in size

ES Node

Your App

ES Node

What about load balancing? Cluster

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

java

JS

Python

.NET

Perl

Ruby

PHP

Clients

ES Node

Your App

ES Node

Cluster

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

java

JS

Python

.NET

Perl

Ruby

PHP

Scale your app

ES Node

Your App

ES Node

Cluster

Your App

Your App

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

java

JS

Python

.NET

Perl

Ruby

PHP

Scale the cluster

Your AppES

Node

Cluster

Your App

Your App

ES Node

ES Node

ES Node

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

java

JS

Python

.NET

Perl

Ruby

PHP

Node Roles

Your AppMaster Node

Cluster

Your App

Your App

Data/Master Node

Data Node

Data Node

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

java

JS

Python

.NET

Perl

Ruby

PHP

Variations

Your AppMaster Node

Cluster

Your App

Your App

Data/Master Node

Data Node

Data Node

Client Node

Client Node

Servers 1,2 Servers 3,4,5,6

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibitedCopyright  Elasticsearch  2014.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

Log Analytics• Index Billions of events per day • Offer Log Analytics as a service to over 3500 customers

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

Logs,!Packets,!Events,!

Transactions,!!

Timestamp Data

Logstash

What about Logging Analytics?

ES Node

Kibana

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

Logstash

Scale up a bit

Logstash Forwarder

ShippersShippers

Logstash Forwarder

LogstashMaster Node

Cluster

Data/Master Node

Data Node

Data Node

Kibana

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

Scale up some more

Logstash Forwarder

ShippersShippers

Logstash Forwarder

Logstash

Master Node

Cluster

Data/Master Node

Data Node

Data Node

Logstash Forwarder

ShippersShippers

Logstash Forwarder

Logstash

Logstash

Logstash

Logstash

Brokers!!

Redis!!

RabbitMQ!!

Kafka

Logstash

Logstash

Kibana

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

Process as a Stream

Brokers

Log

Ship

Log

Ship LogstashNode

Cluster

Node

NodeNode

Logstash

Logstash

Brokers

Storm!!

Spark!!

Samza

Kibana

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibitedCopyright  Elasticsearch  2014.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

Sentiment Analysis• Index and search over millions of social interactions every day • Measure sentiment changes as they happen

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

Where does Hadoop fit in?

Brokers

Log

Ship

Log

Ship LogstashNode

Cluster

Node

NodeNode

Logstash

Logstash

Brokers

Storm!!

Spark!!

Samza

Kibana

HDFSFlume

es-hadoop

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

Scaling• Which hardware do I buy?

• Physical vs Virtual

• Sizing a shard - Distributed systems knowledge applies

• Dedicated Masters, Minimum Masters

• Replica - Pros and Cons

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

Common Pitfalls• Not fencing network, locking down Elasticsearch

• Messing with defaults blindly

• Shoehorn relational model into NoSQL

• The ins and outs of Upserts

• Distributed systems knowledge and architecture a must

Copyright  Elasticsearch  2013.  Copying,  publishing  and/or  distributing  without  written  permission  is  strictly  prohibited

Q&A