scaling with couchbase, kafka and apache spark
TRANSCRIPT
Scaling with Couchbase, Kafka and Apache Spark
Matt Ingenthron, Sr. Director SDK Engineering and Developer Advocacy
2
About Me
Matt IngenthronWorked on large site scalability problems at previous company…
memcached contributor
Joined Couchbase very early and helped define key parts of system @ingenthr
A Brief Aside: What is Couchbase
©2015 Couchbase Inc. 4
Couchbase provides a complete Data Management solution
4
High availability cache
Key-value store
Document database
Embedded database
Sync management
Multi-purpose capabilities support a broad range of apps and use cases
Enterprises often start with cache, then broaden usage to other apps and use cases
©2015 Couchbase Inc. 5
What makes Couchbase unique?
Performance & scalability
leaderSub millisecond latency with high throughput; memory-centric architecture
Multi-purpose
Simplified administrationEasy to deploy & manage; integrated Admin Console, single-click cluster expansion & rebalance
Cache, key value store, document database, and local/mobile database in single platform
Always-on availability
Data replication across nodes, clusters, and data centers
Enterprises choose Couchbase for several key advantages
24x365
©2015 Couchbase Inc. 6
Couchbase Server Architecture
Single-node type means easier administration and scaling
Single installation
Two major components/processes: Data manager cluster manager
Data manager: C/C++ Layer consolidation of caching and
persistence
Cluster manager: Erlang/OTP Administration UI’s Out-of-band for data requests
©2014 Couchbase Inc.
ACTIVE ACTIVE ACTIVE
REPLICA REPLICA REPLICA
Couchbase Server 1 Couchbase Server 2 Couchbase Server 3
Basic Operation
7
SHARD5
SHARD2
SHARD9
SHARD SHARD SHARD
SHARD4
SHARD7
SHARD8
SHARD SHARD SHARD
SHARD1
SHARD3
SHARD6
SHARD SHARD SHARD
SHARD4
SHARD1
SHARD8
SHARD SHARD SHARD
SHARD6
SHARD3
SHARD2
SHARD SHARD SHARD
SHARD7
SHARD9
SHARD5
SHARD SHARD SHARD
Application has single logical connection to cluster (client object)
• Data is automatically sharded resulting in even document data distribution across cluster
• Each vbucket replicated 1, 2 or 3 times (“peer-to-peer” replication)
• Docs are automatically hashed by the client to a shard’
• Cluster map provides location of which server a shard is on
• Every read/write/update/delete goes to same node for a given key
• Strongly consistent data access (“read your own writes”)
• A single Couchbase node can achieve 100k’s ops/sec so no need to scale reads
©2014 Couchbase Inc.
Add Nodes to Cluster
8
ACTIVE ACTIVE ACTIVE
REPLICA REPLICA REPLICA
Couchbase Server 1 Couchbase Server 2 Couchbase Server 3
ACTIVE ACTIVE
REPLICA REPLICA
Couchbase Server 4 Couchbase Server 5
SHARD5
SHARD2
SHARD SHARD
SHARD4
SHARD SHARD
SHARD1
SHARD3
SHARD SHARD
SHARD4
SHARD1
SHARD8
SHARD SHARD SHARD
SHARD6
SHARD3
SHARD2
SHARD SHARD SHARD
SHARD7
SHARD9
SHARD5
SHARD SHARD SHARD
SHARD7
SHARD
SHARD6
SHARD
SHARD8
SHARD9
SHARD
READ/WRITE/UPDATE
Application has single logical connection to cluster (client object)
Multiple nodes added or removed at once
One-click operation
Incremental movement of active and replica vbuckets and data
Client library updated via cluster map
Fully online operation, no downtime or loss of performance
Couchbase in Big Data
a.k.a. the main program
©2015 Couchbase Inc. 10
Lambda Architecture
1
4
5
DATA
SERVE
QUERY
New Data Stream
Analysis
All DataPrecompute
Views (Map Reduce)
Process Stream
Incremental Views
BatchRecompute
Real-TimeIncrement
Batch Layer
Serving Layer
Speed Layer
2 BATCH
3 SPEED
©2015 Couchbase Inc. 11
Lambda Architecture
New Data Stream
Merged View
All DataPrecompute
Views (Map Reduce)
Process Stream
Incremental Views
Partial Aggregat
e
Partial Aggregat
e
Partial Aggregat
e
Real-Time Data
BatchRecompute
Batch Views
Real-Time Views
Real-TimeIncrement
Merge
Batch Layer
Serving Layer
Speed Layer
Lambda + Couchbase
New Data Stream
Merged View
All DataPrecompute
Views (Map Reduce)
Process Stream
Incremental Views
Partial Aggregat
e
Partial Aggregat
e
Partial Aggregat
e
Real-Time Data
BatchRecompute
Batch Views
Real-Time Views
Real-TimeIncrement
Merge
Batch Layer
Serving Layer
Speed Layer
Lamba + Hadoop + Spark + Storm
New Data Stream
Merged View
All DataPrecompute
Views (Map Reduce)
Process Stream
Incremental Views
Partial Aggregat
e
Partial Aggregat
e
Partial Aggregat
e
Real-Time Data
BatchRecompute
Batch Views
Real-Time Views
Real-TimeIncrement
Merge
Batch Layer
Serving Layer
Speed Layer
Couchbase HadoopConnector (Sqoop)
COMPLEX EVENT PROCESSING
Real TimeREPOSITORY
PERPETUALSTORE
ANALYTICALDB
BUSINESSINTELLIGENCE
MONITORING
CHAT/VOICESYSTEM
BATCHTRACK
REAL-TIMETRACK
DASHBOARD
TRACKING and COLLECTION
ANALYSIS ANDVISUALIZATION
REST FILTER METRICS
Integration at Scale
17
Requirements for data streaming in modern systems…
• Must support high throughput and low latency• Need to handle failures
• Pick up where you left off• Be efficient about resource usage
Data Sync is the Heart of Any Big Data System
Fundamental piece of the architecture- Data Sync maintains Data Redundancy for High
Availability (HA) & Disaster Recovery (DR)- Protect against failures – node, rack, region etc.
- Data Sync maintains Indexes- Indexing is key to building faster access paths to query data- Spatial, Full-text
DCP and Couchbase Server Architecture
Database Change Protocol: DCP
20
OrderingTo build interesting features a streaming protocol needs to have a concept of when operations happened.
Couchbase operation ordering at the node level:
Each mutation is assigned a sequence numberSequence numbers increase monotonically Sequence numbers are assigned on a per VBucket basis
Restart-abilityNeed to handle failures with grace, in particular being efficient about the amount of data being moved around.
Consistency pointsPoints in time for incremental backup, query consistency.
PerformanceRecognize that durability on a distributed system may have different definitions.
Design Goals
21
Version History (Failover Example)
VB UUID Seqno
X 0
VB UUID Seqno
DCP Replication Stream
X 0
Last Seqno: 100 Last Seqno: 0Last Seqno: 100
Active ReplicaActive
100Y
Last Seqno: 0
22
Restartability Example
VB UUID Seqno
X 0
VB UUID Seqno
DCP Replication Stream
X 0
Last Seqno: 80 Last Seqno: 150
Down Active
100Y
Replica
0 <= 80 <= 100
Y 100
23
Restartability Example (With Rollback)
VB UUID Seqno
X 0
VB UUID Seqno
DCP Replication Stream
X 0
Last Seqno: 110 Last Seqno: 150
Down Active
100Y
Replica
0 <= 110 <= 100
Last Seqno: 100
0 <= 100 <= 100
Y 100
Demo
25
Shopper Tracking(click stream)
Lightweight Analytics:• Department shopped• Tech platform• Click tracks by Income
Heavier Analytics, Develop Profiles
26
And at scale…
N1QL Queryand Apache Spark
Introduction & Integration
What do Application Developers Need
NoSQL databases provide Excellent performance and scalability Simplicity for working with data in aggregates A modern toolset for integration with mobile and web applications
Before N1QL, they tend to come with some unexpected baggage Map-Reduce for processing is great for some types of processing,
but is not expressive enough for other kinds Some say “but no joins” or have odd ways of duplicating data to
make it fit the query model
Key Concepts
What is N1QL?
N1QL is a new query language that systematically extends SQL to
document-oriented data
N1QL extends SQL to handle data that is:
Nested: Contains nested objects, arrays
Heterogeneous: Schema-optional, non-uniform
Distributed: Partitioned across a cluster
Language Highlights
SELECT
Standard SELECT pipeline
SELECT, FROM, WHERE, GROUP BY, ORDER BY, LIMIT, OFFSET
JOINs (by keys)
Subqueries
Aggregation
Set operators
UNION, INTERSECT, EXCEPT
EXPLAIN
Metadata access
Couchbase has metadata about documents in addition to the document values itself.
The META() function allows for this kind of access
select META(`travel-sample`) from `travel-sample` limit 1; { "$1": { "cas": 1.60955901476865e+14, "flags": 0, "id": "landmark_25257", "type": "json" } }
Metadata access
Couchbase has metadata about documents in addition to the document values itself.
The META() function allows for this kind of access
select META(`travel-sample`).id from `travel-sample` limit 10; { "id": "landmark_26021" }, { "id": "airport_1421" },…
Indexing
CREATE / DROP INDEX
Two types of indexes
View indexes
GSI indexes (new secondary indexes)
Document-oriented Extensions
Nested Paths—user.profile.email, children[0], children[0:2]
NEST, UNNEST
Ranging—FOR EVERY child IN children
Transformations—SELECT {“name”: first_name || last_name}
Non-uniform IS MISSING
Type checking and conversion
Distributed Direct lookup—USE KEYS
Efficient joins—ON KEYS
Apache Spark
What is Spark?
Apache is a fast and general engine for large-scale data processing.
Spark Components
Spark Core: RDDs, Clustering, Execution, Fault Management
Spark Components
Spark SQL: Work with structured data, distributed SQL querying
Spark Components
Spark Streaming: Build fault-tolerant streaming applications
Spark Components
Mlib: Machine Learning built in
Spark Components
GraphX: Graph processing and graph-parallel computations
Daytona GraySort Performance
Hadoop MR Record Spark Record
Data Size 102.5 TB 100 TB
Elapsed Time 72 mins 23 mins
# Nodes 2100 206
# Cores 50400 physical 6592 virtual
Cluster Disk Throughput 3150 GB/s 618 GB/s
Network Dedicated DC, 10Gbps EC2, 10Gbps
Sort Rate 1.42 TB/min 4.27 TB/min
Sort Rate/Node 0.67 GB/min 20.7 GB/min
Source: http://databricks.com/blog/2014/11/05/spark-officially-sets-a-new-record-in-large-scale-sorting.htmlBenchmark: http://sortbenchmark.org/
How does it work?
Resilient Distributed Datatypes paper: https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf
RDDCreation
SchedulingDAG
TaskExecution
How does it work?
Resilient Distributed Datatypes paper: https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf
RDDCreation
SchedulingDAG
TaskExecution
How does it work?
Resilient Distributed Datatypes paper: https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf
RDDCreation
SchedulingDAG
TaskExecution
Spark vs Hadoop
• Spark is RAM while Hadoop is HDFS (disk) bound
• API easier to reason about & to develop against
• Fully compatible with Hadoop Input/Output formats
• Hadoop more mature, Spark ecosystem growing fast
Ecosystem Flexibility
RDBMS
StreamsWeb APIs
DCPKVN1QLViews
BatchingArchived Data
OLTP
Infrastructure Consolidation
StreamsWeb APIs
UserInteraction
51
What does this look like?
Read a sequence of RDDs in, and apply a few transformations…
sc.couchbaseGet[JsonDocument(Seq( "21st_amendment_brewery_cafe-21a_ipa", "aass_brewery-genuine_pilsner")) .map(doc => doc.content().getString("name")) .collect() .foreach(println)
52
Reading a Couchbase Secondary Index…
Query a Couchbase View, subfilter, and then map over them…
// Read the first 10 rows and load their full documentsval beers = sc.couchbaseView(ViewQuery.from("beer", "brewery_beers")) .map(_.id) .couchbaseGet[JsonDocument]() .filter(doc => doc.content().getString("type") == "beer") .cache() // Calculate the mean for all beersprintln(beers .map(doc => doc.content().getDouble("abv").asInstanceOf[Double]) .mean())
Couchbase Connector
Spark Core Automatic Cluster and Resource Management Creating and Persisting RDDs Java APIs in addition to Scala (planned before GA)
Spark SQL Easy JSON handling and querying Tight N1QL Integration (dp2)
Spark Streaming Persisting DStreams DCP source (planned before GA)
Connector Facts
• Current Version: 1.0.0-dp• Beta in July• GA planned for Q3
Code:https://github.com/couchbaselabs/couchbase-spark-connector
Docs until GA: https://github.com/couchbaselabs/couchbase-spark-connector/wiki
Demo #2
Questions
Couchbase Spark Samples:https://github.com/couchbaselabs/couchbase-spark-samples
Couchbase Spark Connector:http://blog.couchbase.com/introducing-the-couchbase-spark-connectorhttps://github.com/couchbaselabs/couchbase-spark-connector
Couchbase Kafka Connector:http://blog.couchbase.com/introducing-the-couchbase-kafka-connectorhttps://github.com/couchbase/couchbase-kafka-connector
Actions!
Matt Ingenthron @ingenthr
Thanks