GETTING STARTED WITH ELASTICSEARCH ON WINDOWS

AND .NET WITH NEST

A short introduction

Oslo/NNUG Meetup

Tomas Jansson

29/01/2014

THIS IS ME

Tomas Jansson

Manager & Group Lead .NETBEKK Oslo

@TomasJanssontomas.jansson@bekk.nogithub.com/mastojblog.tomasjansson.com

TL;DR;

https://github.com/mastoj/NestDemo

AUDIENCE

N00b

N00b Expert

Expert

BACKGROUND

This is the data and we need this new application

THE MASTERPLAN

WHAT I WANT TO SHOW YOU IS...

Elasticsearch is awesome

Indexing using NEST

Querying using NEST

... not about advanced elasticsearch hosting

INSTALLATION

Great news, install as a service added in 0.90.5

Powershell to the rescue

NEST

Abstractionover

Elasticsearch

There is an low level abstraction as well called RawElasticClient

Abstractionover

Elasticsearch

NEST

Fluent & Strongly

typed

Functional C#

FUNC DEMO

C:\Dev\git> scriptcs

scriptcs (ctrl-c or blank to exit)

> Func<int, int, int> add = (x, y) => x + y;

> add(1, 3)

4

Func executable

SIMPLE EXPRESSION DEMO

> using System.Linq.Expressions;

> Expression<Func<int, int, int>> addExpr = (x, y) => x + y;

> addExpr(1, 3)

(1,1): error CS1955: Non-invocable member 'addExpr' cannot be used like a method.

> var binExpr = addExpr.Body as BinaryExpression;

> Console.WriteLine(binExpr);

(x + y)

> var add2 = addExpr.Compile();

> add2(3, 1);

4

Expression ”function description”

MORE COMPLEX EXPRESSION DEMO

> public class SomeClass { public string MyString { get; set; } }

> Expression<Func<SomeClass, object>> propExpr = y => y.MyString + y.MyString;

> var compExpr = propExpr.Compile();

> var obj = new SomeClass { MyString = "Hello world" };

> compExpr(obj)

Hello worldHello world

> var body = propExpr.Body as BinaryExpression;

> Console.WriteLine(body);

(y.MyString + y.MyString)

> var left = body.Left as MemberExpression;

> Console.WriteLine(left.Member.Name);

MyString

MORE COMPLEX EXPRESSION DEMO

> public class SomeClass { public string MyString { get; set; } }

> Expression<Func<SomeClass, object>> propExpr = y => y.MyString + y.MyString;

> var compExpr = propExpr.Compile();

> var obj = new SomeClass { MyString = "Hello world" };

> compExpr(obj)

Hello worldHello world

> var body = propExpr.Body as BinaryExpression;

> Console.WriteLine(body);

(y.MyString + y.MyString)

> var left = body.Left as MemberExpression;

> Console.WriteLine(left.Member.Name);

MyString

Enables us to translate from one domain to another in an ”easy” manner

Show me the code!

ELASTICSEARCH CONNECTION

public class ElasticClientWrapper : ElasticClient{

private static string _connectionString = Settings.ElasticSearchServer;

private static ConnectionSettings _settings =new ConnectionSettings(new Uri(_connectionString)) //http://demoserver:9200

.SetDefaultIndex(Settings.Alias) //"customer_product_mapping"

.UsePrettyResponses();

public ElasticClientWrapper(): base(_settings)

{}

}

//usagevar client = new ElasticClientWrapper();

MAPPING

public class Product{

public double UnitPrice { get; set; }public int TotalQuantity { get; set; }[ElasticProperty(Index = FieldIndexOption.not_analyzed)]public string ProductName { get; set; }[ElasticProperty(Index = FieldIndexOption.not_analyzed)]public string CategoryName { get; set; }

}

public class Customer{

public string CustomerID { get; set; }public string CompanyName { get; set; }public string Address { get; set; }public string City { get; set; }public string Country { get; set; }[ElasticProperty(Type = FieldType.nested)]public Product[] Products { get; set; }

}

MAPPING & INDEXING

_client = new ElasticClientWrapper();_client.CreateIndex("indexName", s =>

s.AddMapping<Customer>(m => m.MapFromAttributes()));

var customers = _customerRepo.GetCustomers();_client.IndexMany(customers, "indexName");

Mapping created from attributes

Indexing will use the mapping for the specified index

There is async versions of the methods

ALIAS

_client = new ElasticClientWrapper();_client.Alias("indexName", "aliasName");

Index_01

Alias

SWAPPING

_client = new ElasticClientWrapper();_client.Swap("aliasName", new [] { "Index_01" }, new [] { "Index_02" } );

Index_01 Index_02

Alias Alias

1. Create new index

2. Swap

3. Delete old index

MY QUERY OBJECT (WILL BE USED IN THE EXAMPLES)

public class SearchModel{

private int? _numberToTake;public string Query { get; set; }public Dictionary<string, IEnumerable<string>> Filter { get; set; }

public int? NumberToTake{

get { return _numberToTake.HasValue ? _numberToTake.Value : 25; }set { _numberToTake = value; }

}}

QUERYING

{"query": {

"query_string": {"query": "tomas"

}}

}

_client.Search<Customer>(sd => sd.QueryString(Input.Query));

Elasticsearch NEST

FUZZY

{"query": {

"fuzzy": {"_all": {

"min_similarity": 0.6,"prefix_length": 1,"value": "tomas"

}}

}}

_client.Search<Customer>(sd => sd.Query(q => q

.Fuzzy(fd => fd

.OnField("_all")

.MinSimilarity(0.6)

.PrefixLength(1)

.Value(Input.Query))));

Elasticsearch NEST

Will enable us to search for both «Thomas» and «Tomas» when writing «Tomas»

FUZZY IMPROVED (USING BOOL QUERY) - ELASTICSEARCH

{"query": {

"bool": {"should": [{

"match": {"_all": {

"query": "tomas"}

}},{

"fuzzy": {"_all": {

"boost": 0.1,"min_similarity": 0.6,"prefix_length": 1,"value": "tomas"

}}

}]}

}}

FUZZY IMPROVED (USING BOOL QUERY) - NEST

_client.Search<Customer>(sd => sd.Query(q => q

.Bool(b => b.Should(new Func<QueryDescriptor<Customer>, BaseQuery>[]

{_ => _.Match(m => m

.OnField("_all")

.QueryString(Input.Query)),_ => _.Fuzzy(fd => fd

.OnField("_all")

.MinSimilarity(0.6)

.PrefixLength(1)

.Value(Input.Query)

.Boost(0.1))}))));

HIGHLIGHT RESULT - ELASTICSEARCH

{"query": {

// see previous example},"highlight": {

"pre_tags": ["<span class='highlight'>"

],"post_tags": [

"</span>"],"fields": {

"companyName": {"fragment_size": 100,"number_of_fragments": 1

}}

}}

HIGHLIGHT RESULT - NEST

_client.Search<Customer>(sd => sd.Query( /* See previous example */ ).Highlight(h => h

.PreTags("<span class='highlight'>")

.PostTags("</span>")

.OnFields(new Action<HighlightFieldDescriptor<Customer>>[]{

_ => _.OnField(c => c.CompanyName).NumberOfFragments(1).FragmentSize(100)

})));

FACETS - ELASTICSEARCH

{"query": { /* See previous example */ },"highlight": { /* See previous example */ },"facets": {

"products.productName": {"nested": "products","terms": { "field": "products.productName", "size": 1000 }

},"products.categoryName": {

"nested": "products","terms": { "field": "products.categoryName", "size": 1000 }

},"country": {

"terms": { "field": "country", "size": 1000 }}

}}

FACETS - NEST

_client.Search<Customer>(sd => sd.Query( /* See previous example */ ).Highlight( /* See previous example */ ).FacetTerm(f => f

.Nested(c => c.Products)

.OnField(c => c.Products[0].ProductName)

.Size(1000)).FacetTerm(f => f

.Nested(c => c.Products)

.OnField(c => c.Products[0].CategoryName)

.Size(1000)).FacetTerm(f => f

.OnField(c => c.Country)

.Size(1000)));

http://go-gaga-over-testing.blogspot.no/2011/09/solution-to-warning-in-quality-center.html

FILTERS - ELASTICSEARCH

{"query": {

"filtered": {"query": { /* See previous example */ },"filter": {

"bool": {"must": [{

"terms": { "country": ["usa"] }},{

"nested": {"query": { "terms": { "products.categoryName": ["Condiments", "Seafood"] } },"path": "products"

}},{

"nested": {"query": { "terms": { "products.productName": ["Chai"] } },"path": "products"

}}

]}

}}

},"facets": { /* See previous example */},"highlight": { /* See previous example */ }

}

FILTERS – NEST – PART 1, THE CUSTOMERS FILTER

private static BaseFilter AddCustomerFilter(IEnumerable<string> items, Expression<Func<Customer, object>> propExpr)

{return Filter<Customer>.Terms(propExpr, items.ToArray());

}

FILTERS – NEST – PART 1, THE PRODUCTS FILTER

private static BaseFilter AddProductsFilter(IEnumerable<string> items,Expression<Func<Customer, object>> propExpr)

{return Filter<Customer>.Nested(sel => sel

.Path(c => c.Products)

.Query(q => q.Terms(propExpr, items.ToArray())));}

FILTERS – NEST – PART 1, THE MAGIC DICTIONARY

public Dictionary<string, Func<IEnumerable<string>, BaseFilter>> FilterDesc =new Dictionary<string, Func<IEnumerable<string>, BaseFilter>>(){

{"products.productName", ps => AddProductsFilter(ps, c => c.Products[0].ProductName)},

{"products.categoryName", cs => AddProductsFilter(cs, c => c.Products[0].CategoryName)},

{"country", cs => AddCustomerFilter(cs, c => c.Country)}};

FILTERS – NEST – PART 1, ALL THE HELPERS

private static BaseFilter AddCustomerFilter(IEnumerable<string> items, Expression<Func<Customer, object>> propExpr)

{return Filter<Customer>.Terms(propExpr, items.ToArray());

}

private static BaseFilter AddProductsFilter(IEnumerable<string> items,Expression<Func<Customer, object>> propExpr)

{return Filter<Customer>.Nested(sel => sel

.Path(c => c.Products)

.Query(q => q.Terms(propExpr, items.ToArray())));}

public Dictionary<string, Func<IEnumerable<string>, BaseFilter>> FilterDesc =new Dictionary<string, Func<IEnumerable<string>, BaseFilter>>(){

{"products.productName", ps => AddProductsFilter(ps, c => c.Products[0].ProductName)},

{"products.categoryName", cs => AddProductsFilter(cs, c => c.Products[0].CategoryName)},

{"country", cs => AddCustomerFilter(cs, c => c.Country)}};

FILTERS – NEST – PART 2, THE QUERY

_client.Search<Customer>(sd => sd.Query(q => q

.Filtered(fq =>{

fq.Query(qs =>{

if (!string.IsNullOrEmpty(Input.Query)){

qs.Bool( /* See previous example */ ));}else{

qs.MatchAll();}return qs;

});if (Input.Filter.Count > 0){

var filters =Input.Filter.Select(_ => FilterDesc[_.Key](_.Value)).ToArray();

fq.Filter(fs => fs.Bool(bf => bf.Must(filters)));}

})).Highlight( /* See previous example */ ).FacetTerm( /* See previous example */ ).FacetTerm( /* See previous example */ ).FacetTerm( /* See previous example */ );

Easy installation

Awesome search engine

Strongly typed client

Fluent

Abstraction over Elasticsearch

Elasticsearch NEST

SUMMARY

Demo code: https://github.com/mastoj/NestDemo

Nest documentation: http://nest.azurewebsites.net/

Nest source code: https://github.com/Mpdreamz/NEST

Slideshare: http://www.slideshare.net/mastoj/getting-started-with-elasticsearch-and-net

Sense (great tool to query elastic search in the browser): https://github.com/bleskes/sense

RESOURCES

Questions?

Thank you!

@TomasJansson