scalable full-text search with datastax enterprise
TRANSCRIPT
©2013 DataStax 1
Scalable Full-Text Search with DataStax Enterprise
Piotr KołaczkowskiDataStax
[email protected]@pkolaczk
©2013 DataStax 2
DataStax Enterprise
Apache Cassandra
+
Apache Solr
+
Apache Hadoop
+
...
©2013 DataStax 3
Apache Cassandra
A database system:
• distributed
• replicated & durable
• scalable
• fault-tolerant (no SPOF)
• highly available
©2013 DataStax 4
Apache Cassandra
A database:
• distributed
• replicated & durable
• scalable
• fault-tolerant (no SPOF)
• highly available
• data-center aware
US Europe
©2013 DataStax 5
Apache Cassandra Storage Engine
• Wide tables, up to 2GB per row
• Fast writes
• Fast primary-key searches
• Durability (commit log)
• Secondary indexes
• No full text search
©2013 DataStax 6
Apache Solr
Database system specialized at searching text:
• Language-aware
• e.g. lowercasing ὈΔΥΣΕΎΣ produces ὀδυσεύς
• can do stemming / stop-world elimination, etc.
• Supports relevance scoring
• Supports complex range queries
• Centralized
©2013 DataStax 7
Solr
Apache Solr
Update Handler
Lucene
Request Handler
Lucene Index
Response Writer
search results
documents to index
SearchSearch
©2013 DataStax 8
Classic Partitioning with SPOF
mastermaster
slaveslave
slaveslave
partition 1 partition 2 partition 3 partition 4
routerrouter
clientclient
©2013 DataStax 9
Availability
“High availability implies that a single fault will not bring down your system. Not ‘we’ll recover quickly.’”
-- Ben Coverston: DataStax
“The biggest problem with failover is that you're almost never using it until it really hurts. It's like backups that you never test.”
-- Rick Branson: Instagram
©2013 DataStax 10
Fully Distributed, no SPOF
clientclient
p3p1
p6
p1p3
p1
©2013 DataStax 11
Partitioning
Primary key determines placement*
jim age: 36 car: camaro gender: M
carol age: 37 car: subaru
johnny age: 12 gender: M
suzy age: 10 gender: F
©2013 DataStax 12
jim
carol
johnny
suzy
5e02739678...
a9a0198010...
f4eb27cea7...
78b421309e...
PK MD5 Hash
MD5* hash operation yields a 128-bit number
for keysof any size.
Partitioning
©2013 DataStax 13
The “Token Ring”
©2013 DataStax 14
StartStart EndEndA 0xc000000000..1 0x0000000000..0
B 0x0000000000..1 0x4000000000..0
C 0x4000000000..1 0x8000000000..0
D 0x8000000000..1 0xc000000000..0
jim 5e02739678...
carol a9a0198010...
johnny f4eb27cea7...
suzy 78b421309e...
©2013 DataStax 15
StartStart EndEndA 0xc000000000..1 0x0000000000..0
B 0x0000000000..1 0x4000000000..0
C 0x4000000000..1 0x8000000000..0
D 0x8000000000..1 0xc000000000..0
jim 5e02739678...
carol a9a0198010...
johnny f4eb27cea7...
suzy 78b421309e...
©2013 DataStax 16
StartStart EndEndA 0xc000000000..1 0x0000000000..0
B 0x0000000000..1 0x4000000000..0
C 0x4000000000..1 0x8000000000..0
D 0x8000000000..1 0xc000000000..0
jim 5e02739678...
carol a9a0198010...
johnny f4eb27cea7...
suzy 78b421309e...
©2013 DataStax 17
StartStart EndEndA 0xc000000000..1 0x0000000000..0
B 0x0000000000..1 0x4000000000..0
C 0x4000000000..1 0x8000000000..0
D 0x8000000000..1 0xc000000000..0
jim 5e02739678...
carol a9a0198010...
johnny f4eb27cea7...
suzy 78b421309e...
©2013 DataStax 18
StartStart EndEndA 0xc000000000..1 0x0000000000..0
B 0x0000000000..1 0x4000000000..0
C 0x4000000000..1 0x8000000000..0
D 0x8000000000..1 0xc000000000..0
jim 5e02739678...
carol a9a0198010...
johnny f4eb27cea7...
suzy 78b421309e...
©2013 DataStax 19
Replication
carol a9a0198010...
©2013 DataStax 20
Replication
carol a9a0198010...
©2013 DataStax 21
Replication
carol a9a0198010...
©2013 DataStax 22
Bringing Cassandra and Solr Together
Cassandra Solr
KeyspaceCore
Table (Column Family)
Row Document
Column Field
©2013 DataStax 23
Schema
<schema name="wikipedia" version="1.1">
<types> <fieldType name="string" class="solr.StrField"/> <fieldType name="text" class="solr.TextField"/> </types>
<fields> <field name="part_id" type="string" indexed="true" stored="true"/> <field name="description" type="text" indexed="true" stored="true"/> </fields>
<defaultSearchField>description</defaultSearchField> <uniqueKey>part_id</uniqueKey>
</schema>
schema.xml and solrconfig.xml stored and distributed by Cassandra
©2013 DataStax 24
Data Mapping
part_id description
2N2222 Low power bipolar NPN transistor
TL074 Low noise JFET-input operational amplifier
LM3886 High performance integrated audio power amplifier
Cassandra table
field value
part_id 2N2222
description Low power bipolar NPN transistor
field value
part_id TL074
description Low noise JFET-input operational amplifier
field value
part_id LM3886
description High performance integrated audio power amplifier
©2013 DataStax 25
DSE Search Architecture
DSE daemon
Embedded Tomcat
Solr
Cassandra
Secondary Index API
SolrIndexes
C* Data
C* Commit Log
Update Handler
SolrREST
CQL
Thrift
©2013 DataStax 26
Inserting
clientclient
p3p1
p6
p1p3
p1
Coordinator
Replica
Replica
Replica
©2013 DataStax 27
Inserting through Cassandra API
Coordinator Node
Solr
Cassandra
SI API
Update Handler
1
CQL INSERT
2
Replica Node
Solr
Cassandra
SI API
Update Handler
4
3
©2013 DataStax 28
Inserting through Solr API
Coordinator Node
Solr
Cassandra
SI API
Update Handler
1
HTTP POST
23
Replica Node
Solr
Cassandra
SI API
Update Handler
5
4
©2013 DataStax 29
Querying
clientclient
p3p1
p6
p1p3
p1
Coordinator
Replica
Replica
Replica
p3
p3
p3
p6
p6
©2013 DataStax 30
How many nodes to contact?
• We don't know the primary key
• Theory: contact at least one replica for every token range
• Cassandra contacts all nodes
• Our custom Solr SearchComponent does intelligent shard selection
©2013 DataStax 31
Querying through CQL
SELECT title FROM solr WHERE solr_query='title:natio*';
title-------------------------------------------------------------------------- Bolivia national football team 2002 List of French born footballers who have played for other national teams Lithuania national basketball team at Eurobasket 2009 Bolivia national football team 2000 Kenya national under-20 football team Bolivia national football team 1999 Israel men's national inline hockey team Bolivia national football team 2001
©2013 DataStax 32
Querying through CQL
Coordinator Node
Solr
Cassandra
SI API
2
Replica Node
Solr
Cassandra
SI API
4
31
CQL SELECT
5
contacts all nodes
6
©2013 DataStax 33
Querying through Solr API
Coordinator Node
Solr
Cassandra
SI API
CassandraSearchComponent
Replica Node
Solr
Cassandra
SI API
21
HTTP GET
3
contacts selectednodes
4
©2013 DataStax 34
Shard Selection Algorithm
• Tries to minimize the number of selected shards
optimum number of shards = ⎡N / RF⎤
• Tries to fetch data from the closest nodes
• local node
• nodes on the same rack
• nodes in the same DC
• Balances the load
©2013 DataStax 35
Shard Selection Algorithm
1.Always select the local node first
2.Select the closest node that is covering the highest number of token ranges not yet covered.
3.Repeat the previous step until all ranges are covered.
©2013 DataStax 36
Querying Shards
Solr does not support indexing 128-bit numbers
Cassandra 128-bit token
127 0
63 0 63 0
_token_lhs _token_rhs
©2013 DataStax 37
Querying Shards
_token_rhs
_token_lhs
max
max
start
end
((+_token_lhs:3074457345618258602 +_token_rhs:[3074457345618258604 TO *]) OR (+_token_lhs:[3074457345618258603 TO 6148914691236517204]) OR (+_token_lhs:6148914691236517205 +_token_rhs:[* TO -3074457345618258602]))
©2013 DataStax 38
Workload Separation
C*
C*
C*
C*
clientclient Replication FactorSolr: 2Cassandra: 2
©2013 DataStax 39
Questions?
• http://www.datastax.com/docs
• http://www.datastax.com/products/enterprise
Cassandra Summit 2013, June 11-12, San Francisco, CAhttp://www.datastax.com/company/news-and-events/events/cassandrasummit2013