your data, your search, elasticsearch (euruko 2011)

Your Data,Your Search

Karel Minařík

ElasticSearch

http://karmi.cz

http://karmi.cz

http://karmi.cz

Your Data,Your Search

Karel Minařík and Florian Hanke

ElasticSearch

Search is the primary interfacefor getting information today.

http://www.apple.com/macosx/what-is-macosx/spotlight.html



??????

# https://github.com/rubygems/gemcutter/blob/master/app/models/rubygem.rb#L29-33#def self.search(query) where("versions.indexed and (upper(name) like upper(:query) or upper(versions.description) like upper(:query))", {:query => "%#{query.strip}%"}). includes(:versions). order("rubygems.downloads desc")end

ElasticSearch

Search (mostly) sucks.Why?

How do you implement search?WHY SEARCH SUCKS?

class MyModel include Whatever::Searchend

MyModel.search "something"


class MyModel include Whatever::Searchend

MyModel.search "whatever"

MAGIC


def search @results = MyModel.search params[:q] respond_with @resultsend

ResultResultsQuery


ResultResultsQuery


MAGIC


ResultResultsQuery


MAGIC +

A personal story...

670px

23px

MyModel.search "(this OR that) AND NOT whatever"

Arel::Table.new(:articles). where(articles[:title].eq('On Search')). where(["published_on => ?", Time.now]). join(comments). on(article[:id].eq(comments[:article_id])) take(5). skip(4). to_sql

Compare your search library with your ORM libraryWHY SEARCH SUCKS?

ElasticSearch

Your data, your search.

A collection of documentsHOW DOES SEARCH WORK?

file_1.txtThe ruby is a pink to blood-‐red colored gemstone ...

file_2.txtRuby is a dynamic, reflective, general-‐purpose object-‐oriented programming language ...

file_3.txt"Ruby" is a song by English rock band Kaiser Chiefs ...

How do you search documents?HOW DOES SEARCH WORK?

File.read('file1.txt').include?('ruby')

The inverted indexHOW DOES SEARCH WORK?

http://en.wikipedia.org/wiki/Index_(search_engine)#Inverted_indices

TOKENS POSTINGS

ruby file_1.txt file_2.txt file_3.txt

pink file_1.txt

gemstone file_1.txt

dynamic file_2.txt

reflective file_2.txt

programming file_2.txt

song file_3.txt

english file_3.txt

rock file_3.txt






pink file_1.txt

gemstone file_1.txt

dynamic file_2.txt



song file_3.txt

english file_3.txt

rock file_3.txt

MySearchLib.search "ruby"





pink file_1.txt

gemstone file_1.txt

dynamic file_2.txt



song file_3.txt

english file_3.txt

rock file_3.txt

MySearchLib.search "song"




module SimpleSearch

def index document, content tokens = analyze content store document, tokens puts "Indexed document #{document} with tokens:", tokens.inspect, "\n" end

def analyze content # >>> Split content by words into "tokens" content.split(/\W/). # >>> Downcase every word map { |word| word.downcase }. # >>> Reject stop words, digits and whitespace reject { |word| STOPWORDS.include?(word) || word =~ /^\d+/ || word == '' } end

def store document_id, tokens tokens.each do |token| # >>> Save the "posting" ( (INDEX[token] ||= []) << document_id ).uniq! end end

def search token puts "Results for token '#{token}':" # >>> Print documents stored in index for this token INDEX[token].each { |document| " * #{document}" } end

INDEX = {} STOPWORDS = %w|a an and are as at but by for if in is it no not of on or that the then there these this to with|

extend self

end A naïve Ruby implementation

SimpleSearch.index "file1", "Ruby is a language. Java is also a language."SimpleSearch.index "file2", "Ruby is a song."SimpleSearch.index "file3", "Ruby is a stone."SimpleSearch.index "file4", "Java is a language."

Indexed document file1 with tokens:["ruby", "language", "java", "also", "language"]

Indexed document file2 with tokens:["ruby", "song"]

Indexed document file3 with tokens:["ruby", "stone"]

Indexed document file4 with tokens:["java", "language"]

Indexing documentsHOW DOES SEARCH WORK?

Words downcased,stopwords removed.

puts "What's in our index?"

p SimpleSearch::INDEX

{ "ruby" => ["file1", "file2", "file3"], "language" => ["file1", "file4"], "java" => ["file1", "file4"], "also" => ["file1"], "stone" => ["file3"], "song" => ["file2"]}

The indexHOW DOES SEARCH WORK?

SimpleSearch.search "ruby"

Results for token 'ruby':* file1* file2* file3

Search the indexHOW DOES SEARCH WORK?



TOKENS POSTINGS


pink file_1.txt

gemstone file_1.txt

dynamic file_2.txt



song file_3.txt

english file_3.txt

rock file_3.txt

31

ElasticSearch

It is very practical to know how search works.

For instance, now you know thatthe analysis step is very important.

Most of the time, it's more important than the search step.

module SimpleSearch

def index document, content tokens = analyze content store document, tokens puts "Indexed document #{document} with tokens:", tokens.inspect, "\n" end

def analyze content # >>> Split content by words into "tokens" content.split(/\W/). # >>> Downcase every word map { |word| word.downcase }. # >>> Reject stop words, digits and whitespace reject { |word| STOPWORDS.include?(word) || word =~ /^\d+/ || word == '' } end

def store document_id, tokens tokens.each do |token| # >>> Save the "posting" ( (INDEX[token] ||= []) << document_id ).uniq! end end

def search token puts "Results for token '#{token}':" # >>> Print documents stored in index for this token INDEX[token].each { |document| " * #{document}" } end

INDEX = {} STOPWORDS = %w|a an and are as at but by for if in is it no not of on or that the then there these this to with|

extend self

end A naïve Ruby implementation

http://search-engines-book.com

Search EnginesInformation Retrieval in PracticeBruce Croft, Donald Metzler and Trevor StrohmaAddison Wesley, 2009

The Search Engine TextbookHOW DOES SEARCH WORK?

http://www.search-engines-book.com/

http://www.search-engines-book.com/

Lucene in ActionMichael McCandless, Erik Hatcher and Otis GospodneticJuly, 2010

The Baseline Information Retrieval ImplementationSEARCH IMPLEMENTATIONS

http://manning.com/hatcher3

http://www.manning.com/hatcher3/

http://www.manning.com/hatcher3/

http://elasticsearch.org



ElasticSearch

HTTPJSONSchema-freeIndex as ResourceDistributedQueriesFacetsMappingRuby

{ }

ELASTICSEARCH FEATURES

# Add documentcurl -‐X POST "http://localhost:9200/articles/article/1" -‐d '{ "title" : "One" }'

# Querycurl -‐X GET "http://localhost:9200/articles/_search?q=One"curl -‐X POST "http://localhost:9200/articles/_search" -‐d '{ "query" : { "terms" : { "tags" : ["ruby", "python"], "minimum_match" : 2 } }}'

# Delete indexcurl -‐X DELETE "http://localhost:9200/articles"

# Create index with settings and mappingcurl -‐X PUT "http://localhost:9200/articles" -‐d '{ "settings" : { "index" : "number_of_shards" : 3, "number_of_replicas" : 2 }},{ "mappings" : { "document" : { "properties" : { "body" : { "type" : "string", "analyzer" : "snowball" } } } }}'

HTTP JSON / Schema-free / Index as Resource / Distributed / Queries / Facets / Mapping / Ruby

INDEX TYPE ID


# Add documentcurl -‐X POST "http://localhost:9200/articles/article/1" -‐d '{ "title" : "One" }'

# Querycurl -‐X GET "http://localhost:9200/articles/_search?q=One"curl -‐X POST "http://localhost:9200/articles/_search" -‐d '{ "query" : { "terms" : { "tags" : ["ruby", "python"], "minimum_match" : 2 } }}'

# Delete indexcurl -‐X DELETE "http://localhost:9200/articles"

# Create index with settings and mappingcurl -‐X PUT "http://localhost:9200/articles" -‐d '{ "settings" : { "index" : "number_of_shards" : 3, "number_of_replicas" : 2 }},{ "mappings" : { "document" : { "properties" : { "body" : { "type" : "string", "analyzer" : "snowball" } } } }}'




http {

server {

listen 8080; server_name search.example.com;

error_log elasticsearch-‐errors.log; access_log elasticsearch.log;

location / {

# Deny access to Cluster API if ($request_filename ~ "_cluster") { return 403; break; }

# Pass requests to ElasticSearch proxy_pass http://localhost:9200; proxy_redirect off; proxy_set_header X-‐Real-‐IP $remote_addr; proxy_set_header X-‐Forwarded-‐For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host;

# Authorize access auth_basic "ElasticSearch"; auth_basic_user_file passwords;

# Route all requests to authorized user's own index rewrite ^(.*)$ /$remote_user$1 break; rewrite_log on;

return 403; }

}}

GET http://user:password@localhost:8080/_search?q=* => http://localhost:9200/user/_search?q=*

https://gist.github.com/986390



HTTP / JSON / Schema-free / Index as Resource / Distributed / Queries / Facets / Mapping / Ruby


{ "id" : "abc123",

"title" : "ElasticSearch Understands JSON!",

"body" : "ElasticSearch not only “works” with JSON, it understands it! Let’s first ...",

"published_on" : "2011/05/27 10:00:00", "tags" : ["search", "json"],

"author" : { "first_name" : "Clara", "last_name" : "Rice", "email" : "[email protected]" }}

JSON

curl -‐X DELETE "http://localhost:9200/articles"; sleep 1curl -‐X POST "http://localhost:9200/articles/article" -‐d '

{ "id" : "abc123",




"author" : { "first_name" : "Clara", "last_name" : "Rice", "email" : "[email protected]" }}'curl -‐X POST "http://localhost:9200/articles/_refresh"

curl -‐X GET \ "http://localhost:9200/articles/article/_search?q=author.first_name:clara"



curl -‐X GET "http://localhost:9200/articles/_mapping?pretty=true"{ "articles" : { "article" : { "properties" : { "title" : { "type" : "string" }, // ... "author" : { "dynamic" : "true", "properties" : { "first_name" : { "type" : "string" }, // ... } }, "published_on" : { "format" : "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd", "type" : "date" } } } }}



curl -‐X POST "http://localhost:9200/articles/comment" -‐d '{ "body" : "Wow! Really nice JSON support.",

"published_on" : "2011/05/27 10:05:00",

"author" : { "first_name" : "John", "last_name" : "Pear", "email" : "[email protected]" }}'curl -‐X POST "http://localhost:9200/articles/_refresh"

curl -‐X GET \ "http://localhost:9200/articles/comment/_search?q=author.first_name:john"

HTTP / JSON / Schema Free / Index as Resource / Distributed / Queries / Facets / Mapping / Ruby


curl -‐X GET \ "http://localhost:9200/articles/comment/_search?q=body:json"

curl -‐X GET \ "http://localhost:9200/articles/_search?q=body:json"

curl -‐X GET \ "http://localhost:9200/articles,users/_search?q=body:json"

curl -‐X GET \ "http://localhost:9200/_search?q=body:json"





curl -‐X DELETE "http://localhost:9200/articles"; sleep 1curl -‐X POST "http://localhost:9200/articles/article" -‐d '{ "id" : "abc123",




"author" : { "first_name" : "Clara", "last_name" : "Rice", "email" : "[email protected]" }}'curl -‐X POST "http://localhost:9200/articles/_refresh"

curl -‐X GET "http://localhost:9200/articles/article/1"



{"_index":"articles","_type":"article","_id":"1","_version":1, "_source" : { "id" : "1",




"author" : { "first_name" : "Clara", "last_name" : "Rice", "email" : "[email protected]" }}}

The Index Is Your Database.



http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html

my_alias

index_A

index_B

curl -‐X POST 'http://localhost:9200/_aliases' -‐d '{ "actions" : [ { "add" : { "index" : "index_1", "alias" : "myalias" } }, { "add" : { "index" : "index_2", "alias" : "myalias" } } ]}'

Index Aliases





logs

The “Sliding Window” problem

logs_2010_02

logs_2010_03

logs_2010_04

curl -‐X DELETE http://localhost:9200 / logs_2010_01

“We can really store only three months worth of data.”

curl -‐X PUT localhost:9200/_template/bookmarks_template -‐d '{ "template" : "users_*",

"settings" : { "index" : { "number_of_shards" : 1, "number_of_replicas" : 3 } },

"mappings": { "url": { "properties": { "url": { "type": "string", "analyzer": "simple", "boost": 10 }, "title": { "type": "string", "analyzer": "snowball", "boost": 5 } // ... } } }}'



Apply this configurationfor every matchingindex being created

http://www.elasticsearch.org/guide/reference/api/admin-indices-templates.html

Index Templates





Node 1 Node 2 Node 3 Node 4MASTER

Automatic Discovery Protocol

http://www.elasticsearch.org/guide/reference/modules/discovery/

$ cat elasticsearch.yml

cluster: name: <YOUR APPLICATION>





A

A1

A2

A3

A1'

A2'

A3'

A1''

A2''

A3''

Replicas

Shards

curl -‐XPUT 'http://localhost:9200/A/' -‐d '{ "settings" : { "index" : { "number_of_shards" : 3, "number_of_replicas" : 2 } }}'

Index is split into 3 shards, and duplicated in 2 replicas.



SHARDS

REPLICAS

Improve indexing performance

Impro

ve search perfo

rmance

HTTP / JSON / Schema Free / Distributed / Queries / Facets / Mapping / Ruby


Terms appleapple iphone

Phrases "apple iphone"

Proximity "apple safari"~5

Fuzzy apple~0.8

Wildcards app**pp*

Boosting apple^10 safari

Range [2011/05/01 TO 2011/05/31][java TO json]

Boolean

apple AND NOT iphone+apple -‐iphone(apple OR iphone) AND NOT review

Fieldstitle:iphone^15 OR body:iphonepublished_on:[2011/05/01 TO "2011/05/27 10:00:00"]

http://lucene.apache.org/java/3_1_0/queryparsersyntax.html

$ curl -‐X GET "http://localhost:9200/_search?q=<YOUR QUERY>"



curl -‐X POST "http://localhost:9200/articles/_search?pretty=true" -‐d '{ "query" : { "terms" : { "tags" : [ "ruby", "python" ], "minimum_match" : 2 } }}'



Query DSL

http://www.elasticsearch.org/guide/reference/query-dsl/

JSON



curl -‐X POST "http://localhost:9200/venues/venue" -‐d '{ "name": "Pizzeria", "pin": { "location": { "lat": 50.071712, "lon": 14.386832 } }}'

curl -‐X POST "http://localhost:9200/venues/_search?pretty=true" -‐d '{

"query" : { "filtered" : { "query" : { "query_string" : { "query" : "pizzeria" } }, "filter" : { "geo_distance" : { "distance" : "0.5km", "pin.location" : { "lat" : 50.071481, "lon" : 14.387284 } } } } }}'



Accepted formats for Geo:

[lon, lat] # Array

"lat,lon" # String

drm3btev3e86 # Geohash

Geo Search

http://www.elasticsearch.org/guide/reference/query-dsl/geo-distance-filter.html





http://blog.linkedin.com/2009/12/14/linkedin-faceted-search/

Query



curl -‐X POST "http://localhost:9200/articles/_search?pretty=true" -‐d '{ "query" : { "query_string" : { "query" : "title:T*"} }, "filter" : { "terms" : { "tags" : ["ruby"] } }, "facets" : { "tags" : { "terms" : { "field" : "tags", "size" : 10 } } }}'

# facets" : {# "tags" : {# "terms" : [ {# "term" : "ruby",# "count" : 2# }, {# "term" : "python",# "count" : 1# }, {# "term" : "java",# "count" : 1# } ]# }# }

User query

“Checkboxes”

Facets

http://www.elasticsearch.org/guide/reference/api/search/facets/index.html





curl -‐X POST "http://localhost:9200/articles/_search?pretty=true" -‐d '{ "facets" : { "published_on" : { "date_histogram" : { "field" : "published", "interval" : "day" } } }}'

http://www.elasticsearch.org/blog/2011/05/13/data-visualization-with-elasticsearch-and-protovis.html



curl -‐X POST "http://localhost:9200/venues/_search?pretty=true" -‐d '{ "query" : { "query_string" : { "query" : "pizzeria" } }, "facets" : { "distance_count" : { "geo_distance" : { "pin.location" : { "lat" : 50.071712, "lon" : 14.386832 }, "ranges" : [ { "to" : 1 }, { "from" : 1, "to" : 5 }, { "from" : 5, "to" : 10 } ] } } }}'



Geo Distance Facets

http://www.elasticsearch.org/guide/reference/api/search/facets/geo-distance-facet.html



def analyze content # >>> Split content by words into "tokens" content.split(/\W/). # >>> Downcase every word map { |word| word.downcase }. # ... end



Remember?

curl -‐X DELETE "http://localhost:9200/articles"curl -‐X POST "http://localhost:9200/articles/article" -‐d '{ "mappings": { "article": { "properties": { "tags": { "type": "string", "analyzer": "keyword" }, "content": { "type": "string", "analyzer": "snowball" }, "title": { "type": "string", "analyzer": "snowball", "boost": 10.0 } } } }}'

curl -‐X GET 'http://localhost:9200/articles/_mapping?pretty=true'

http://www.elasticsearch.org/guide/reference/api/admin-indices-create-index.html





curl -‐X DELETE "http://localhost:9200/urls"curl -‐X POST "http://localhost:9200/urls/url" -‐d '{ "settings" : { "index" : { "analysis" : { "analyzer" : { "url_analyzer" : { "type" : "custom", "tokenizer" : "lowercase", "filter" : ["stop", "url_stop", "url_ngram"] } }, "filter" : { "url_stop" : { "type" : "stop", "stopwords" : ["http", "https", "www"] }, "url_ngram" : { "type" : "nGram", "min_gram" : 3, "max_gram" : 5 } } } } }}'






curl -‐X PUT localhost:9200/urls/url/_mapping -‐d '{ "url": { "properties": { "url": { "type": "string", "analyzer": "url_analyzer" } } }

}'

curl -‐X POST localhost:9200/urls/url -‐d '{ "url" : "http://urlaubinkroatien.de" }'curl -‐X POST localhost:9200/urls/url -‐d '{ "url" : "http://besteurlaubinkroatien.de" }'curl -‐X POST localhost:9200/urls/url -‐d '{ "url" : "http://kroatien.de" }'curl -‐X POST localhost:9200/urls/_refresh

curl "http://localhost:9200/urls/_search?pretty=true&q=url:kroatien"

curl "http://localhost:9200/urls/_search?pretty=true&q=url:urlaub"

curl "http://localhost:9200/urls/_search?pretty=true&q=url:(urlaub AND kroatien)"






K R O A T I E N

K R O

R O A

O A T

A T I

T I E

I E N

Trigrams}

HTTP / JSON / Schema Free / Distributed / Queries / Facets / Mapping / RubyELASTICSEARCH FEATURES

Tire.index 'articles' do delete create

store :title => 'One', :tags => ['ruby'], :published_on => '2011-‐01-‐01' store :title => 'Two', :tags => ['ruby', 'python'], :published_on => '2011-‐01-‐02' store :title => 'Three', :tags => ['java'], :published_on => '2011-‐01-‐02' store :title => 'Four', :tags => ['ruby', 'php'], :published_on => '2011-‐01-‐03'

refreshend

s = Tire.search 'articles' do query { string 'title:T*' }

filter :terms, :tags => ['ruby']

sort { title 'desc' }

facet 'global-‐tags' { terms :tags, :global => true }

facet 'current-‐tags' { terms :tags }end

http://github.com/karmi/tire

https://github.com/karmi/tire


class Article < ActiveRecord::Base include Tire::Model::Search include Tire::Model::Callbacksend



Article.search do query { string 'love' } facet('timeline') { date :published_on, :interval => 'month' } sort { published_on 'desc' }end

$ rake environment tire:import CLASS='Article'



class Article include Whatever::ORM

include Tire::Model::Search include Tire::Model::Callbacksend



Article.search do query { string 'love' } facet('timeline') { date :published_on, :interval => 'month' } sort { published_on 'desc' }end

$ rake environment tire:import CLASS='Article'



$ rails new tired -‐m "https://gist.github.com/raw/951343/tired.rb"

A “batteries included” installation.Downloads and launches ElasticSearch.Sets up a Rails applicationand and launches it.When you're tired of it, just delete the folder.

Try ElasticSearch and Tire with a one-line command.

https://gist.github.com/raw/951343/tired.rb

https://gist.github.com/raw/951343/tired.rb

Thanks!d

your data, your search, elasticsearch (euruko 2011)

Technology

search sucks

search ruby ruby

search params

search step

search library

search rubyresults

search works

eqon search