developing with couchbase: app development with indexes and queries
TRANSCRIPT
1 1
Couchbase Server 2.0 Indexing and Querying
Chris AndersonChief Architect
2
What we’ll talk about
• View basics• Lifecycle of a view
Index definition, build, and query phase Indexing details
• Replica indexes, failover and compaction• Primary and Secondary indexes• View best practices• Additional patterns
3
JSON Documents
• Map more closely to objects or entities• CRUD Operations, lightweight schema
• Stored under an identifier key
{ “fields” : [“with basic types”, 3.14159, true], “like” : “your favorite language”}
client.set(“mydocumentid”, myDocument);mySavedDocument = client.get(“mydocumentid”);
What are Views?
• Extract fields from JSON documents and produce an index of the selected information
Views – The basics
• Define materialized views on JSON documents and then query across the data set • Using views you can define
• Primary indexes
• Simple secondary indexes (most common use case)
• Complex secondary, tertiary and composite indexes
• Aggregations (reduction)
• Indexes are eventually indexed • Queries are eventually consistent with respect to documents• Built using Map/Reduce technology
• Map and Reduce functions are written in Javascript
View LifecycleDefine -> Build -> Query
6 6
• Create design documents on a bucket• Create views within a design document
Buckets & Design docs & Views
7
BUCKET 1
Design document 1
View 1View 1
View 2View 2
View 3View 3
Design document 2
View 4View 4
View 5View 5
Design document 3
View 6View 6
View 7View 7
BUCKET 2
8
3333 22
Eventually indexed Views – Data flow2
Managed Cache
Dis
k Q
ueue
Disk
Replication Queue
App Server
Couchbase Server Node
Doc 1Doc 1
Doc 1
To other node
View engine
Doc 1
9
Couchbase Server Cluster
Distributed Indexing and Querying
User Configured Replica Count = 1
Active
Doc 5
Doc 2
Doc
Doc
Doc
Server 1
REPLICA
Doc 3
Doc 1
Doc 7
Doc
Doc
Doc
App Server 1
COUCHBASE Client LibraryCOUCHBASE Client Library
Cluster Map
COUCHBASE Client LibraryCOUCHBASE Client Library
Cluster Map
App Server 2
Doc 9
• Indexing work is distributed amongst nodes
• Parallelize the effort
• Each node has index for data stored on it
• Queries combine the results from required nodes
Active
Doc 3
Doc 1
Doc
Doc
Doc
Server 2
REPLICA
Doc 6
Doc 4
Doc 9
Doc
Doc
Doc
Doc 8
Active
Doc 4
Doc 6
Doc
Doc
Doc
Server 3
REPLICA
Doc 2
Doc 5
Doc 8
Doc
Doc
Doc
Doc 7
Query
Create Index / View
10
DEFINE Index / View Definition in JavaScript
CREATE INDEX City ON Brewery.City;
10
11
BUILD Distributed Index Build Phase
• Optimized for lookups, in-order access and aggregations
• View reads are from disk (different performance profile than GET/SET)
• Views built against every document on every node–Group them in a design document
• Views are automatically kept up to date
12
QUERY Dynamic Queries with Optional Aggregation
Query ?startkey=“J”&endkey=“K”{“rows”:[{“key”:“Juneau”,“value”:null}]}
• Eventually consistent with respect to document updates• Efficiently fetch a document or group of similar documents • Queries will use cached values from B-tree inner nodes when possible• Take advantage of in-order tree traversal with group_level queries
–All the views within a design document are incrementally updated when the view is accessed or auto-indexing kicks in–Automatic view updates• In addition to forcing an index build at query time, active & replica indexes are
updated every 3 seconds of inactivity if there are at least 5000 new changes (configurable)
–The entire view is recreated if the view definition has changed–Views can be conditionally updated by specifying the “stale”
argument to the view query–The index information stored on disk consists of the combination
of both the key and value information defined within your view.
Index building details
14
Queries run against stale indexes by default
• stale=update_after (default if nothing is specified)–always get fastest response–can take two queries to read your own writes
• stale=ok–auto update will trigger eventually–might not see your own writes for a few minutes– least frequent updates -> least resource impact
• stale=false–Use with “set with persistence” if data needs to be included in
view results–BUT be aware of delay it adds, only use when really required
Views and Replica indexes
• In addition to replicas for data (up to 3 copies), optionally create replica for indexes• Each node manages replica index data structures• Set at a bucket level • Replica index populated from replica data• Replica index is used after a failover
Views and failover
• Replica indexes enabled on failover• Replicas indexes are rebuilt on replica nodes–Automatically incrementally built based on replica data–Updated every 3 seconds of inactivity if there are at least 5000
new changes–Not copied/moved to be consistent with persisted replica data
View Compaction
• Compaction is ONLINE • Reclaims empty allocated space from disk• Indexes are stored on disk for active vBuckets on each
node and updated in append-only manner• Auto-compaction performed in the background–Set the database fragmentation levels–Set the index fragmentation levels–Choose a schedule–Global and bucket specific settings
18
Development vs. Production Views
• Development views index a subset of the data.• Publishing a view builds the
index across the entire cluster.• Queries on production views
are scattered to all cluster members and results are gathered and returned to the client.
19 19
Simple Primary and
Secondary Indexing
Example Document
20
Document ID
21
Define a primary index on the bucket
• Lookup the document ID / key by key, range, prefix, suffix
Index definition
22
Define a secondary index on the bucket
• Lookup an attribute by value, range, prefix, suffix
Index definition
23
Find documents by a specific attribute
• Lets find beers by brewery_id!
24
The index definition
ValueKey
25
The result set: beers keyed by brewery_id
View Best Practices
26 26
function(doc, meta){ if (doc.ingredient) { emit(doc.ingredient.ingredtext, null); }}
function(doc, meta){ if (doc.ingredient) { emit(doc.ingredient.ingredtext, null); }} 27
View writing guidance
• Move frequently used views out to a separate design document– All views in a design document are updated at the same time
– This can result in increase index building time if all views are in a single design document, especially for frequently accessed views.
– However, grouping views into smaller number of design documents improves overall performance
• Try to avoid computing too many things with one view
• Use built-in reduces where possible - custom reduces are not optimized
• Check for attribute existence
28
View writing guidance
function(doc, meta) { if(doc.type == “player”) emit(doc.experience, null);}
function(doc, meta) { if(doc.type == “player”) emit(doc.experience, null);}
• Do not include the document in the view value– Instead either use the GET / SET API or the API that includes documents filtered by
the query [example: willIncludeDocs()]
– Emit either null or the ID instead (meta.id) in your key or value data
• Don’t emit too much data into a view value– Use views to filter documents
– Then use the data path to access the matched documents
• Use Document Types to make views more selective
emit(doc.name, null)emit(doc.name, null)
What impact do views have on the system?
• Complexity of the index CPU • Size of the value emitted and selectivity Disk size, I/O• Replica index Disk size, I/O, CPU• Number of design doc CPU, I/O, Disk size– 4 active and 2 replica design documents are built in parallel by default –Can be changed using the maxParallelIndexers and maxParallelReplicaIndexers parameters
• Compaction of views CPU, I/O • Rebalance time Increases with views to support consistent
query results during rebalance – Can be disabled using the indexAwareRebalanceDisabled parameter
Views and OS caching
• File system cache availability for the index has a big impact performance• Indexes are disk based and should have sufficient file system
cache available for faster query access• In house performance results show that by doubling system
cache availability–query latency reduces to half –throughput increases by 50%
• Runs based on 10 million items with 16GB bucket quota and 4GB, 8GB system RAM availability for indexes
Query PatternBasic Aggregations
31 31
32
Use a built-in reduce function with a group query
• Lets find average abv for each brewery!
33 33
We are reducing doc.abv with _stats
34 34
Group reduce (reduce by unique key)
Query PatternTime-based Rollups
35 35
36
Find patterns in beer comments by time
{ "type": "comment", "about_id": "beer_Enlightened_Black_Ale", "user_id": 525, "text": "tastes like college!", "updated": "2010-07-22 20:00:20"}
{ "id": "f1e62"}
timestamp
37
Query with group_level=2 to get monthly rollups
38
dateToArray() is your friend
• String or Integer based timestamps• Output optimized for group_level queries• array of JSON numbers:
[2012,9,21,11,30,44]
dateToArray()
39
group_level=2 results
• Monthly rollup• Sorted by time—sort the query results in your application if
you want to rank by value—no chained map-reduce
40
group_level=3 - daily results - great for graphing
• Daily, hourly, minute or second rollup all possible with the same index.
41 41
Query PatternLeaderboard
42
Aggregate value stored in a document
• Lets find the top-rated beers!
{ "brewery": "New Belgium Brewing", "name": "1554 Enlightened Black Ale", "abv": 5.5, "description": "Born of a flood...", "category": "Belgian and French Ale", "style": "Other Belgian-Style Ales", "updated": "2010-07-22 20:00:20", “ratings” : { “jchris” : 5, “scalabl3” : 4, “damienkatz” : 1 }, “comments” : [ “f1e62”, “6ad8c” ]}
ratings
43 43
Sort each beer by its average rating
• Lets find the top-rated beers!
average
Questions?
44 44