your data, your search, elasticsearch (euruko 2011)

Your Data,Your Search

Karel Minařík

ElasticSearch

http://karmi.cz

Your Data,Your Search

Karel Minařík and Florian Hanke

ElasticSearch

Search is the primary interfacefor getting information today.

http://www.apple.com/macosx/what-is-macosx/spotlight.html

??????

# https://github.com/rubygems/gemcutter/blob/master/app/models/rubygem.rb#L29-33#def self.search(query) where("versions.indexed and (upper(name) like upper(:query) or upper(versions.description) like upper(:query))", {:query => "%#{query.strip}%"}). includes(:versions). order("rubygems.downloads desc")end

ElasticSearch

Search (mostly) sucks.Why?

How do you implement search?WHY SEARCH SUCKS?

class MyModel include Whatever::Searchend

MyModel.search "something"

class MyModel include Whatever::Searchend

MyModel.search "whatever"

def search @results = MyModel.search params[:q] respond_with @resultsend

ResultResultsQuery

MAGIC +

A personal story...

MyModel.search "(this OR that) AND NOT whatever"

Arel::Table.new(:articles). where(articles[:title].eq('On Search')). where(["published_on => ?", Time.now]). join(comments). on(article[:id].eq(comments[:article_id])) take(5). skip(4). to_sql

Compare your search library with your ORM libraryWHY SEARCH SUCKS?

ElasticSearch

Your data, your search.

A collection of documentsHOW DOES SEARCH WORK?

file_1.txtThe ruby is a pink to blood-‐red colored gemstone ...

file_2.txtRuby is a dynamic, reflective, general-‐purpose object-‐oriented programming language ...

file_3.txt"Ruby" is a song by English rock band Kaiser Chiefs ...

How do you search documents?HOW DOES SEARCH WORK?

File.read('file1.txt').include?('ruby')

The inverted indexHOW DOES SEARCH WORK?

http://en.wikipedia.org/wiki/Index_(search_engine)#Inverted_indices

TOKENS POSTINGS

ruby file_1.txt file_2.txt file_3.txt

pink file_1.txt

gemstone file_1.txt

dynamic file_2.txt

reflective file_2.txt

programming file_2.txt

song file_3.txt

english file_3.txt

rock file_3.txt

pink file_1.txt

gemstone file_1.txt

dynamic file_2.txt

song file_3.txt

english file_3.txt

rock file_3.txt

MySearchLib.search "ruby"

pink file_1.txt

gemstone file_1.txt

dynamic file_2.txt

song file_3.txt

english file_3.txt

rock file_3.txt

MySearchLib.search "song"

module SimpleSearch

def index document, content tokens = analyze content store document, tokens puts "Indexed document #{document} with tokens:", tokens.inspect, "\n" end

def analyze content # >>> Split content by words into "tokens" content.split(/\W/). # >>> Downcase every word map { |word| word.downcase }. # >>> Reject stop words, digits and whitespace reject { |word| STOPWORDS.include?(word) || word =~ /^\d+/ || word == '' } end

def store document_id, tokens tokens.each do |token| # >>> Save the "posting" ( (INDEX[token] ||= []) << document_id ).uniq! end end

def search token puts "Results for token '#{token}':" # >>> Print documents stored in index for this token INDEX[token].each { |document| " * #{document}" } end

INDEX = {} STOPWORDS = %w|a an and are as at but by for if in is it no not of on or that the then there these this to with|

extend self

end A naïve Ruby implementation

SimpleSearch.index "file1", "Ruby is a language. Java is also a language."SimpleSearch.index "file2", "Ruby is a song."SimpleSearch.index "file3", "Ruby is a stone."SimpleSearch.index "file4", "Java is a language."

Indexed document file1 with tokens:["ruby", "language", "java", "also", "language"]

Indexed document file2 with tokens:["ruby", "song"]

Indexed document file3 with tokens:["ruby", "stone"]

Indexed document file4 with tokens:["java", "language"]

Indexing documentsHOW DOES SEARCH WORK?

Words downcased,stopwords removed.

puts "What's in our index?"

p SimpleSearch::INDEX

{ "ruby" => ["file1", "file2", "file3"], "language" => ["file1", "file4"], "java" => ["file1", "file4"], "also" => ["file1"], "stone" => ["file3"], "song" => ["file2"]}

The indexHOW DOES SEARCH WORK?

SimpleSearch.search "ruby"

Results for token 'ruby':* file1* file2* file3

Search the indexHOW DOES SEARCH WORK?

TOKENS POSTINGS

pink file_1.txt

gemstone file_1.txt

dynamic file_2.txt

song file_3.txt

english file_3.txt

rock file_3.txt

ElasticSearch

It is very practical to know how search works.

For instance, now you know thatthe analysis step is very important.

Most of the time, it's more important than the search step.

module SimpleSearch

def index document, content tokens = analyze content store document, tokens puts "Indexed document #{document} with tokens:", tokens.inspect, "\n" end

def analyze content # >>> Split content by words into "tokens" content.split(/\W/). # >>> Downcase every word map { |word| word.downcase }. # >>> Reject stop words, digits and whitespace reject { |word| STOPWORDS.include?(word) || word =~ /^\d+/ || word == '' } end

def store document_id, tokens tokens.each do |token| # >>> Save the "posting" ( (INDEX[token] ||= []) << document_id ).uniq! end end

def search token puts "Results for token '#{token}':" # >>> Print documents stored in index for this token INDEX[token].each { |document| " * #{document}" } end

INDEX = {} STOPWORDS = %w|a an and are as at but by for if in is it no not of on or that the then there these this to with|

extend self

end A naïve Ruby implementation

http://search-engines-book.com

Search EnginesInformation Retrieval in PracticeBruce Croft, Donald Metzler and Trevor StrohmaAddison Wesley, 2009

The Search Engine TextbookHOW DOES SEARCH WORK?

Lucene in ActionMichael McCandless, Erik Hatcher and Otis GospodneticJuly, 2010

The Baseline Information Retrieval ImplementationSEARCH IMPLEMENTATIONS

http://manning.com/hatcher3

http://elasticsearch.org

ElasticSearch

HTTPJSONSchema-freeIndex as ResourceDistributedQueriesFacetsMappingRuby

ELASTICSEARCH FEATURES

# Add documentcurl -‐X POST "http://localhost:9200/articles/article/1" -‐d '{ "title" : "One" }'

# Querycurl -‐X GET "http://localhost:9200/articles/_search?q=One"curl -‐X POST "http://localhost:9200/articles/_search" -‐d '{ "query" : { "terms" : { "tags" : ["ruby", "python"], "minimum_match" : 2 } }}'

# Delete indexcurl -‐X DELETE "http://localhost:9200/articles"

# Create index with settings and mappingcurl -‐X PUT "http://localhost:9200/articles" -‐d '{ "settings" : { "index" : "number_of_shards" : 3, "number_of_replicas" : 2 }},{ "mappings" : { "document" : { "properties" : { "body" : { "type" : "string", "analyzer" : "snowball" } } } }}'

HTTP JSON / Schema-free / Index as Resource / Distributed / Queries / Facets / Mapping / Ruby

INDEX TYPE ID

# Add documentcurl -‐X POST "http://localhost:9200/articles/article/1" -‐d '{ "title" : "One" }'

# Querycurl -‐X GET "http://localhost:9200/articles/_search?q=One"curl -‐X POST "http://localhost:9200/articles/_search" -‐d '{ "query" : { "terms" : { "tags" : ["ruby", "python"], "minimum_match" : 2 } }}'

# Delete indexcurl -‐X DELETE "http://localhost:9200/articles"

# Create index with settings and mappingcurl -‐X PUT "http://localhost:9200/articles" -‐d '{ "settings" : { "index" : "number_of_shards" : 3, "number_of_replicas" : 2 }},{ "mappings" : { "document" : { "properties" : { "body" : { "type" : "string", "analyzer" : "snowball" } } } }}'

HTTP JSON / Schema-free / Index as Resource / Distributed / Queries / Facets / Mapping / Ruby

http {

server {

listen 8080; server_name search.example.com;

error_log elasticsearch-‐errors.log; access_log elasticsearch.log;

location / {

# Deny access to Cluster API if ($request_filename ~ "_cluster") { return 403; break; }

# Pass requests to ElasticSearch proxy_pass http://localhost:9200; proxy_redirect off; proxy_set_header X-‐Real-‐IP $remote_addr; proxy_set_header X-‐Forwarded-‐For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host;

# Authorize access auth_basic "ElasticSearch"; auth_basic_user_file passwords;

# Route all requests to authorized user's own index rewrite ^(.*)$ /$remote_user$1 break; rewrite_log on;

return 403; }

GET http://user:password@localhost:8080/_search?q=* => http://localhost:9200/user/_search?q=*

https://gist.github.com/986390

HTTP / JSON / Schema-free / Index as Resource / Distributed / Queries / Facets / Mapping / Ruby

{ "id" : "abc123",

"title" : "ElasticSearch Understands JSON!",

"body" : "ElasticSearch not only “works” with JSON, it understands it! Let’s first ...",

"published_on" : "2011/05/27 10:00:00", "tags" : ["search", "json"],

"author" : { "first_name" : "Clara", "last_name" : "Rice", "email" : "clara@rice.org" }}

curl -‐X DELETE "http://localhost:9200/articles"; sleep 1curl -‐X POST "http://localhost:9200/articles/article" -‐d '

{ "id" : "abc123",

"author" : { "first_name" : "Clara", "last_name" : "Rice", "email" : "clara@rice.org" }}'curl -‐X POST "http://localhost:9200/articles/_refresh"

curl -‐X GET \ "http://localhost:9200/articles/article/_search?q=author.first_name:clara"

curl -‐X GET "http://localhost:9200/articles/_mapping?pretty=true"{ "articles" : { "article" : { "properties" : { "title" : { "type" : "string" }, // ... "author" : { "dynamic" : "true", "properties" : { "first_name" : { "type" : "string" }, // ... } }, "published_on" : { "format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd", "type" : "date" } } } }}

curl -‐X POST "http://localhost:9200/articles/comment" -‐d '{ "body" : "Wow! Really nice JSON support.",

"published_on" : "2011/05/27 10:05:00",

"author" : { "first_name" : "John", "last_name" : "Pear", "email" : "john@pear.org" }}'curl -‐X POST "http://localhost:9200/articles/_refresh"

curl -‐X GET \ "http://localhost:9200/articles/comment/_search?q=author.first_name:john"

HTTP / JSON / Schema Free / Index as Resource / Distributed / Queries / Facets / Mapping / Ruby

curl -‐X GET \ "http://localhost:9200/articles/comment/_search?q=body:json"

curl -‐X GET \ "http://localhost:9200/articles/_search?q=body:json"

curl -‐X GET \ "http://localhost:9200/articles,users/_search?q=body:json"

curl -‐X GET \ "http://localhost:9200/_search?q=body:json"

curl -‐X DELETE "http://localhost:9200/articles"; sleep 1curl -‐X POST "http://localhost:9200/articles/article" -‐d '{ "id" : "abc123",

"author" : { "first_name" : "Clara", "last_name" : "Rice", "email" : "clara@rice.org" }}'curl -‐X POST "http://localhost:9200/articles/_refresh"

curl -‐X GET "http://localhost:9200/articles/article/1"

{"_index":"articles","_type":"article","_id":"1","_version":1, "_source" : { "id" : "1",

"author" : { "first_name" : "Clara", "last_name" : "Rice", "email" : "clara@rice.org" }}}

The Index Is Your Database.

http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html

my_alias

index_A

index_B

curl -‐X POST 'http://localhost:9200/_aliases' -‐d '{ "actions" : [ { "add" : { "index" : "index_1", "alias" : "myalias" } }, { "add" : { "index" : "index_2", "alias" : "myalias" } } ]}'

Index Aliases

The “Sliding Window” problem

logs_2010_02

logs_2010_03

logs_2010_04

curl -‐X DELETE http://localhost:9200 / logs_2010_01

“We can really store only three months worth of data.”

curl -‐X PUT localhost:9200/_template/bookmarks_template -‐d '{ "template" : "users_*",

"settings" : { "index" : { "number_of_shards" : 1, "number_of_replicas" : 3 } },

"mappings": { "url": { "properties": { "url": { "type": "string", "analyzer": "simple", "boost": 10 }, "title": { "type": "string", "analyzer": "snowball", "boost": 5 } // ... } } }}'

Apply this configurationfor every matchingindex being created

http://www.elasticsearch.org/guide/reference/api/admin-indices-templates.html

Index Templates

Node 1 Node 2 Node 3 Node 4MASTER

Automatic Discovery Protocol

http://www.elasticsearch.org/guide/reference/modules/discovery/

$ cat elasticsearch.yml

cluster: name: <YOUR APPLICATION>

Replicas

Shards

curl -‐XPUT 'http://localhost:9200/A/' -‐d '{ "settings" : { "index" : { "number_of_shards" : 3, "number_of_replicas" : 2 } }}'

Index is split into 3 shards, and duplicated in 2 replicas.

SHARDS

REPLICAS

Improve indexing performance

ve search perfo

rmance

HTTP / JSON / Schema Free / Distributed / Queries / Facets / Mapping / Ruby

Terms appleapple iphone

Phrases "apple iphone"

Proximity "apple safari"~5

Fuzzy apple~0.8

Wildcards app**pp*

Boosting apple^10 safari

Range [2011/05/01 TO 2011/05/31][java TO json]

Boolean

apple AND NOT iphone+apple -‐iphone(apple OR iphone) AND NOT review

Fieldstitle:iphone^15 OR body:iphonepublished_on:[2011/05/01 TO "2011/05/27 10:00:00"]

http://lucene.apache.org/java/3_1_0/queryparsersyntax.html

$ curl -‐X GET "http://localhost:9200/_search?q=<YOUR QUERY>"

curl -‐X POST "http://localhost:9200/articles/_search?pretty=true" -‐d '{ "query" : { "terms" : { "tags" : [ "ruby", "python" ], "minimum_match" : 2 } }}'

Query DSL

http://www.elasticsearch.org/guide/reference/query-dsl/

curl -‐X POST "http://localhost:9200/venues/venue" -‐d '{ "name": "Pizzeria", "pin": { "location": { "lat": 50.071712, "lon": 14.386832 } }}'

curl -‐X POST "http://localhost:9200/venues/_search?pretty=true" -‐d '{

"query" : { "filtered" : { "query" : { "query_string" : { "query" : "pizzeria" } }, "filter" : { "geo_distance" : { "distance" : "0.5km", "pin.location" : { "lat" : 50.071481, "lon" : 14.387284 } } } } }}'

Accepted formats for Geo:

[lon, lat] # Array

"lat,lon" # String

drm3btev3e86 # Geohash

Geo Search

http://www.elasticsearch.org/guide/reference/query-dsl/geo-distance-filter.html

http://blog.linkedin.com/2009/12/14/linkedin-faceted-search/

curl -‐X POST "http://localhost:9200/articles/_search?pretty=true" -‐d '{ "query" : { "query_string" : { "query" : "title:T*"} }, "filter" : { "terms" : { "tags" : ["ruby"] } }, "facets" : { "tags" : { "terms" : { "field" : "tags", "size" : 10 } } }}'

# facets" : {# "tags" : {# "terms" : [ {# "term" : "ruby",# "count" : 2# }, {# "term" : "python",# "count" : 1# }, {# "term" : "java",# "count" : 1# } ]# }# }

User query

“Checkboxes”

Facets

http://www.elasticsearch.org/guide/reference/api/search/facets/index.html

curl -‐X POST "http://localhost:9200/articles/_search?pretty=true" -‐d '{ "facets" : { "published_on" : { "date_histogram" : { "field" : "published", "interval" : "day" } } }}'

curl -‐X POST "http://localhost:9200/venues/_search?pretty=true" -‐d '{ "query" : { "query_string" : { "query" : "pizzeria" } }, "facets" : { "distance_count" : { "geo_distance" : { "pin.location" : { "lat" : 50.071712, "lon" : 14.386832 }, "ranges" : [ { "to" : 1 }, { "from" : 1, "to" : 5 }, { "from" : 5, "to" : 10 } ] } } }}'

Geo Distance Facets

http://www.elasticsearch.org/guide/reference/api/search/facets/geo-distance-facet.html

def analyze content # >>> Split content by words into "tokens" content.split(/\W/). # >>> Downcase every word map { |word| word.downcase }. # ... end

Remember?

curl -‐X DELETE "http://localhost:9200/articles"curl -‐X POST "http://localhost:9200/articles/article" -‐d '{ "mappings": { "article": { "properties": { "tags": { "type": "string", "analyzer": "keyword" }, "content": { "type": "string", "analyzer": "snowball" }, "title": { "type": "string", "analyzer": "snowball", "boost": 10.0 } } } }}'

curl -‐X GET 'http://localhost:9200/articles/_mapping?pretty=true'

http://www.elasticsearch.org/guide/reference/api/admin-indices-create-index.html

curl -‐X DELETE "http://localhost:9200/urls"curl -‐X POST "http://localhost:9200/urls/url" -‐d '{ "settings" : { "index" : { "analysis" : { "analyzer" : { "url_analyzer" : { "type" : "custom", "tokenizer" : "lowercase", "filter" : ["stop", "url_stop", "url_ngram"] } }, "filter" : { "url_stop" : { "type" : "stop", "stopwords" : ["http", "https", "www"] }, "url_ngram" : { "type" : "nGram", "min_gram" : 3, "max_gram" : 5 } } } } }}'

curl -‐X PUT localhost:9200/urls/url/_mapping -‐d '{ "url": { "properties": { "url": { "type": "string", "analyzer": "url_analyzer" } } }

curl -‐X POST localhost:9200/urls/url -‐d '{ "url" : "http://urlaubinkroatien.de" }'curl -‐X POST localhost:9200/urls/url -‐d '{ "url" : "http://besteurlaubinkroatien.de" }'curl -‐X POST localhost:9200/urls/url -‐d '{ "url" : "http://kroatien.de" }'curl -‐X POST localhost:9200/urls/_refresh

curl "http://localhost:9200/urls/_search?pretty=true&q=url:kroatien"

curl "http://localhost:9200/urls/_search?pretty=true&q=url:urlaub"

curl "http://localhost:9200/urls/_search?pretty=true&q=url:(urlaub AND kroatien)"

K R O A T I E N

Trigrams}

HTTP / JSON / Schema Free / Distributed / Queries / Facets / Mapping / RubyELASTICSEARCH FEATURES

Tire.index 'articles' do delete create

store :title => 'One', :tags => ['ruby'], :published_on => '2011-‐01-‐01' store :title => 'Two', :tags => ['ruby', 'python'], :published_on => '2011-‐01-‐02' store :title => 'Three', :tags => ['java'], :published_on => '2011-‐01-‐02' store :title => 'Four', :tags => ['ruby', 'php'], :published_on => '2011-‐01-‐03'

refreshend

s = Tire.search 'articles' do query { string 'title:T*' }

filter :terms, :tags => ['ruby']

sort { title 'desc' }

facet 'global-‐tags' { terms :tags, :global => true }

facet 'current-‐tags' { terms :tags }end

http://github.com/karmi/tire

class Article < ActiveRecord::Base include Tire::Model::Search include Tire::Model::Callbacksend

Article.search do query { string 'love' } facet('timeline') { date :published_on, :interval => 'month' } sort { published_on 'desc' }end

$ rake environment tire:import CLASS='Article'

class Article include Whatever::ORM

include Tire::Model::Search include Tire::Model::Callbacksend

Article.search do query { string 'love' } facet('timeline') { date :published_on, :interval => 'month' } sort { published_on 'desc' }end

$ rake environment tire:import CLASS='Article'

$ rails new tired -‐m "https://gist.github.com/raw/951343/tired.rb"

A “batteries included” installation.Downloads and launches ElasticSearch.Sets up a Rails applicationand and launches it.When you're tired of it, just delete the folder.

Try ElasticSearch and Tire with a one-line command.

Thanks!d

your data, your search, elasticsearch (euruko 2011)

search sucks

search ruby ruby

search params

search step

search library

search rubyresults

search works

eqon search

Technology

replace your google search appliance (gsa) with...

euruko 2009 - dataobjects

elasticsearch aggregations

your data, your search, elasticsearch

analyze your log data with open distro for elasticsearch

analyzing text with amazon elasticsearch service and ... ·...

elasticsearch: accelerating the django admin ·...

choctop lightning talk euruko 2010

managing your content with elasticsearch

meetup elasticsearch : « booster votre magento avec...

managing your security logs with elasticsearch

vote amsterdam for euruko 2012

dell emc search · above elasticsearch. elasticsearch...

elasticsearch documentation€¦ · elasticsearch...

euruko jruby talk 2008

elasticsearch sur azure : make sense of your (big) data !

managing your security logs with elasticsearch

elasticsearch terminology

elasticsearch 5 in amazon elasticsearch service

chapter 1: getting started with elasticsearch...