scaling real-time search and analytics with elasticsearch
DESCRIPTION
See the video here: https://www.youtube.com/watch?v=o6lSeNatVFM A look at the elements required by Elasticsearch to turn a simple inverted index into an auto-clustering, horizontally scalable real time search and analytics engine. The talk will start from first principles, explaining how an inverted index works, how to make an inverted index suitable for real time search, how to scale that out, and how to add reliability and failover to the cluster.TRANSCRIPT
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Clinton Gormley @clintongormley
Scaling real time search and analytics with
elasticsearch
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch.org/guide
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch• real-time
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch• real-time • distributed
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch• real-time • distributed • search
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch• real-time • distributed • search • analytics
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how to use it?
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how to use it?
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how does it work?
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 1:making text searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
where content like “%darling%buds%”
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
slow & inflexible
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc 1 Doc 2 Doc 3breathebringsbudsbutbycan…damaskeddarlingdatedaydeafdeathdeclinesdelight
sorted list of unique terms
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc 1 Doc 2 Doc 3breathebringsbudsbutbycan…damaskeddarlingdatedaydeafdeathdeclinesdelight
where they occur
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc 1 Doc 2 Doc 3breathebringsbudsbutbycan…damaskeddarlingdatedaydeafdeathdeclinesdelight
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc 1 Doc 2 Doc 3breathebringsbudsbutbycan…damaskeddarlingdatedaydeafdeathdeclinesdelight
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index• term frequencies
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index• term frequencies » relevance
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index• term frequencies • text length
» relevance
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index• term frequencies • text length
» relevance » doc weight
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index• term frequencies • text length • term positions
» relevance » doc weight
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index• term frequencies • text length • term positions
» relevance » doc weight » word proximity
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index• term frequencies • text length • term positions • char offsets
» relevance » doc weight » word proximity
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index• term frequencies • text length • term positions • char offsets
» relevance » doc weight » word proximity » highlighting
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted indexnot just for text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted indexnumbers, dates, bools, enums
geopoints, geoshapes, etc
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 2:analytics
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
for searchmap values → doc_ids
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
for searchmap values → doc_ids
for analyticsmap doc_ids → values
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
uninvert the index
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
uninvert the indexcache values in memory
called “fielddata”
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
uninvert the indexdata access from RAM
very fast
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
on-the-fly analyticsin the context of
a user’s query
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
on-the-fly analyticsrelevant analytics
for each user
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
calculate metricscount, min, max, sum, avg,
percentiles, cardinality, stddev, variance, sum of squares
!
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
grouped bypopular terms, significant terms, ranges, dates, geolocation, etc
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
grouped bygroups can
… contain subgroups … which contain subgroups
etc
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 3:building the inverted index
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable• cache friendly
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable• cache friendly • reads from RAM
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable• cache friendly • reads from RAM • fielddata never changes
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable• cache friendly • reads from RAM • fielddata never changes • compressible
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable• cache friendly • reads from RAM • fielddata never changes • compressible • no locking
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but, immutable…
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 4:dynamic inverted index
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
in-memory buffer
commit
segment
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable
commit
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable
commit
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit• write new segment
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit• write new segment • write new commit point
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit• write new segment • write new commit point • fsync
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit• write new segment • write new commit point • fsync • clear buffer
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit• write new segment • write new commit point • fsync • clear buffer • reopen index
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit• write new segment • write new commit point • fsync ← expensive! • clear buffer • reopen index
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 5:near real-time search
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
in-memory buffer
flush
segment
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
flush
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
flush
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchablecommit
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene flush
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene flush• write new segment • clear buffer • reopen index !
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene flush• write new segment • clear buffer • reopen index • no fsync
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene flush• write new segment • clear buffer • reopen index • no fsync → lightweight
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…data not safe until fsync’ed!
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 6:don’t lose data
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 6:don’t lose data → transaction log
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
in-memory buffer
flush
segment
translog
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
translog
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
flush
translog
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
translog
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
translog
commit
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
translog
commit point
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch “refresh”• lucene “flush” • makes changes searchable • lightweight !
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
• lucene “commit” • clears transaction log • persists changes • heavy !
elasticsearch “flush”
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
refresh every second
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
near real-time search!
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
near real-time search!near real-time analytics!
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
• slow searches • poor term frequencies • poor compression !
!
too many segments
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 7:reduce segments
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
merge process• many small → one big • removes deleted docs • runs in background • throttled
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.“Any wonder it broke down” by Brian Snelson is licensed under CC BY 2.0
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
sometimes you need another truck
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 8:scale out, not up
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
shard your data
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
shard your datatransparent in elasticsearch
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
many segments
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
one shard
ss
many segments →
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
one shard
ss
many segments
ssssssss
many shards
ss
→
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
one shard
ss
many segments
one index
IIssssssss
many shards
ss
→
→
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“node”running instance of elasticsearch
≈ one server
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“shard”bucket of data
lives on one nodephysical worker unit
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“index”logical namespace
points to one or more shards
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“index”logical namespace
points to one or more shards
shard = hash(_id) % no_of_shards
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
PUT doc _id:1
hash(1) % 3 ⇒ shard_2
node_A
shard_0
node_B
shard_1
node_C
shard_2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
GET doc _id:2
hash(2) % 3 ⇒ shard_0
node_A
shard_0
node_B
shard_1
node_C
shard_2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Search all docs
shard = hash(_id) % no_of_shards
node_A
shard_0
node_B
shard_1
node_C
shard_2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 9:scaling elastically
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
start smallnode_A
shard_0
shard_1
shard_2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add more nodesnode_A
shard_0
shard_1
shard_2
node_B node_C
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
shards migratenode_A
shard_0
shard_1
shard_2
node_B
shard_1
node_C
shard_2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
rebalancednode_A
shard_0
node_B
shard_1
node_C
shard_2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add new indexnode_A
shard_0
shard_1
node_B
shard_1
shard_2
node_C
shard_0
shard_2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…more hardware?
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…more hardware?
more hardware failure
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
at 3am on sunday…node_A
shard_0
shard_1
node_B
shard_1
shard_2
node_C
shard_0
shard_2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
boom!node_A
shard_0
shard_1
node_B
shard_1
shard_2
node_C
shard_0
shard_2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 10:add redundancy
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
for every shard…make a copy
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“primary shard”main shard
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“replica shard(s)”copy of primary shard
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
one nodenode_A
P0
P1
P2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add a nodenode_A
P0
P1
P2
node_B
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add a nodenode_A
P0
P1
P2
node_B
R0
R1
R2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
redundancynode_A
P0
P1
P2
node_B
R0
R1
R2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add a nodenode_A
P0
P1
P2
node_B
R0
R1
R2
node_C
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add a nodenode_A
P0
P1
P2
node_B
R0
R1
R2
R1
node_C
P0
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
rebalancednode_A
P0
P1
P2
node_B
R0
R1
R2
node_C
P0
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lose a nodenode_A
P0
P1
P2
node_B
R0
R2
R1
node_C
P0
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
replica ⇒ primarynode_A
P0
P1
P2
node_B
R0
R2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
replica ⇒ primarynode_A
P0
P1
P2
node_B
P0
R2
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
allocate replicasnode_A
P0
P1
P2
node_B
P0
R2
R0
R1
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
rebalancednode_A
P0
P1
P2
node_B
P0
R2
R0
R1
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
primary shard• just a role • receives doc changes first • forwards new doc to replicas in parallel • number of primaries fixed
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
replica shard• copy of primary shard • serves read/search requests • number of replicas can be changed • more replicas → more read throughput
*if you have more hardware*
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…who controls all this?
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 11:the master node
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.“Master Yoda” by Gonzalo Martín is licensed under CC BY-SA 2.0
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“node”running instance of elastic search
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“node”running instance of elastic search
node_A
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“cluster”one or more nodes
with same cluster name working together
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“cluster”
node_A node_B
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
discover a clusterwith multicast/unicast
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
discover a clusterwith multicast/unicast
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
request routingsend request to any node
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
request routingforwards to correct node
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how?
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how?every node knows where
every document is
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
cluster stateevery node knows where
every document is
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
cluster statecluster level information
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
cluster statecluster level information
indices ⇔ shards ⇔ nodes
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
cluster statecan only be updated by
the master node
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A
master nodeelected when cluster forms
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B
master node
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B
master node
node_C
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
master node
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
master nodejust a role
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
master nodere-elected if master fails
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_B node_C
master node
node_A
re-elected if master fails
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_B node_C
master nodere-elected if master fails
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
master nodeonly manages
cluster level changes
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
master nodenot doc-level
get/put/search
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
the result?
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
distributed real-time
search & analytics
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
which works in the same way on your laptop…
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
…as on your 1,000 node cluster
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
who is using it?
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
• full text search • highlighted search snippets • search-as-you-type • did-you-mean suggestions
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
• combine visitor logs with social network data
• real-time feedback to editors
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
• combines full text search with geolocation
• uses more-like-this to find related questions and answers
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
• search repositories, users, issues, pull requests
• search 130 billion lines of code • track all alerts, events, logs
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
• index and analyse 5TB of log data every day
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
thank you@clintongormley
elasticsearch.org/downloadselasticsearch.com/support
elasticsearch.com/jobs