performance & scalability of couchbase server – couchbase connect 2016

Download Performance & scalability of Couchbase Server – Couchbase Connect 2016

Post on 15-Apr-2017

408 views

Category:

Software

3 download

Embed Size (px)

TRANSCRIPT

PowerPoint Presentation

1The Couchbase Connect16 mobile appTake our in-app survey!

2016 Couchbase Inc.

1

Performance & Scalability of Couchbase Server

2

2016 Couchbase Inc.

3Raju SuravarjjalaSenior Director, QE & Perfraju@couchbase.comIMAGE GOES HERE

2016 Couchbase Inc.AgendaWhat is Couchbase?Why you will choose Couchbase?Accessing Data from CouchbasePerformance within the Database & ApplicationTuning N1QLPerformance testing at CouchbaseMeasuring Couchbase PerformanceBenchmarksDemo

2016 Couchbase Inc.

2016 Couchbase Inc.What is Couchbase?Couchbase is a distributed operational database that enables you to develop with agility and operate at any scale.

Managed CacheKey-Value StoreDocument DatabaseEmbedded DatabaseSync Management

2016 Couchbase Inc.

2016 Couchbase Inc.KEY POINT: COUCHBASE HAS YOU COVERED FOR YOUR GENERAL PURPOSE DB NEEDS. FROM CACHING TO KV STORE, TO JSON DOCUMENT STORE, TO MOBILE APPS. NO OTHER NOSQL DB VENDOR HAS THIS BREADTH AND DEPTH OF TECHNOLOGY

The purpose of this slide is to discuss the high level concepts of Couchbase, and if the SE wants to discuss what parts of Couchbase make up each concept. It is not to go over specific technologies like N1QL, ODBC, etc5

Operate at Any ScaleElastic ScalabilityConsistent High PerformanceAlways-on AvailabilityMulti-Data Center DeploymentSimple, Powerful AdministrationEnterprise Grade Security

2016 Couchbase Inc.

2016 Couchbase Inc.Key Point: Couchbase Server enables you to Operate at Any Scale Deploy, Manage, and Scale applications without limits

Elastic ScalabilityCouchbase 4.0 enables enterprises to scale easily, efficiently, and reliably, from a few nodes tohundreds, with push button simplicity.

Consistent High PerformanceCouchbase 4.0 enables enterprises to meet the highest performance requirements with amemory-centric, asynchronous architecture that can support millions of concurrent users.

Always-On AvailabilityCouchbase 4.0 enables enterprises to avoid downtime, maintain high availability, and recoverfrom disaster via multiple DR options.

MultiData Center DeploymentCouchbase 4.0 enables enterprises to leverage infrastructure across geographies to improveperformance by placing data closer to users, and to increase availability by placing data inmultiple locations.

Simple and Powerful AdministrationCouchbase 4.0 enables enterprises to efficiently deploy, manage, and monitor clusters of any size.

Enterprise-Grade SecurityCouchbase 4.0 enables enterprises to control and monitor administrative access, and to securecommunication between applications, administrators, and the database.6

Accessing Data From Couchbase Key access using Document IDOperations are extremely fast with consistent low latencyReads and writes are evenly distributed across Data Service nodesData is cached in built-in Managed Caching layer and stored in persistent storage layerQueries using N1QLSQL-like : SELECT * FROM WHERE, LIKE, GROUP, etc.,JOINsPowerful Extensions (nest, unnest) for JSON to support nested and hierarchical data structures.Multiple access paths Views and global secondary indexesODBC/JDBC drivers availableViews using static queriesPre-computed complex Map-Reduce queriesIncrementally updated to power analytics, reporting and dashboardsStrong for complex custom aggregations

2015 Couchbase Inc.#

2016 Couchbase Inc.

Performance within the DatabaseArchitecture and Design:Shared nothing workload distributionRAM < - > RAM replication and data transfer (DCP)Managed Caching LayerDirect Client < - > Node communication

Efficient Resource Utilization:Asynchronous and multi-threaded processingCore data management written in C/C++Workload isolation between data storage, querying and indexing (MDS)Append-only storage format

2016 Couchbase Inc.

2016 Couchbase Inc.Performance for the ApplicationChoose the right access pattern:Key-ValuePrimary key lookup, whole document or individual fieldsPredictable sub-ms latencies for 100K+ reads/writes per secondLess flexible, application needs more codeN1QL (plus GSI)Operational queries: Low-ms latencies, 10K ops/secAnalytical queries: Variable performancePush-down to data base layerSQL-based language provides much more flexibilityMap-Reduce ViewsAPI-based, incremental map-reduceScatter-Gather = Less scalable for reads, more scalable for writes

2016 Couchbase Inc.

2016 Couchbase Inc.Indexing with Couchbase Server: Views vs GSIFind the Top 10 Most Active Users for Month of Jan

CREATE INDEX ON customer_bucket(customer_name, total_logins.jan_2015) WHERE type=customer_profile;

SELECT customer_name, total_logins.jan_2015 FROM customer_bucket WHERE type=customer_profile ORDER BY total_logins.jan_2015 DESC LIMIT 10;

Q1: Execution Plan on N nodesScatter: Scatter Q1 to N nodesGather: Gather N results from N nodesFinalize: Re-Aggregate Q1 on 1 nodeLocal Indexes (Views)Q1Active @ Jan 2015

Cluster ManagerData ServiceManaged CacheStorageCluster ManagerManaged CacheStorageCluster ManagerManaged CacheStorageCluster ManagerManaged CacheStorageCluster ManagerManaged CacheStorageIndex ServiceQuery ServiceData ServiceIndex ServiceQuery ServiceData ServiceIndex ServiceQuery ServiceData ServiceIndex ServiceQuery ServiceData ServiceIndex ServiceQuery Service

123

2016 Couchbase Inc.

2016 Couchbase Inc.Flip the visual to be more central visual is more compelling10

Indexing with Couchbase Server: Views vs GSIFind the Top 10 Most Active Users for Month of Jan

CREATE INDEX ON Customer_bucket(customer_name, total_logins.jan_2015) WHERE type=customer_profile;

SELECT customer_name, total_logins.jan_2015 FROM customer_bucket WHERE type=customer_profile ORDER BY total_logins.jan_2015 DESC LIMIT 10;

Q1Active @ Jan 2015Q1: Execution Plan on N nodesExecute Q1 on N1QL Service nodeScan index on Index Service nodeReturn results to applicationGlobal Indexes (GSI)

Cluster ManagerIndex ServiceManaged CacheStorageCluster ManagerIndex ServiceManaged CacheStorageCluster ManagerQuery ServiceManaged CacheStorageCluster ManagerData ServiceManaged CacheStorageCluster ManagerData ServiceManaged CacheStorage12

3

2016 Couchbase Inc.

2016 Couchbase Inc.Flip the visual to be more central visual is more compelling11

Tuning N1QL Indexing, Indexing, Indexing!

Global Secondary Indexes/Memory Optimized Indexes:Index any JSON field (including nested and array structures)Compound/Composite IndexesFunctional IndexesPartial IndexesCovering IndexesPartition-able for scaling out

Per-query scan consistency optionsEXPLAIN plan for analysis/optimization

Best Practices for performance:Covering indexesAvoid intersect scansSmaller result sets

2016 Couchbase Inc.

2016 Couchbase Inc.Couchbase Testing Suite - SummaryCouchbase Software goes through rigorous testing

4,500 Functional Tests via continuous integration model Hours of functional testing 12000 for major releases, 4500 for minor releasesPlatform certification 3000 functional tests Scale tests on 30 node on a weekly basis, 130 node being run on a monthly basis30+ different System test setups including some customer workloadsHours of performance testing - 5000+ hours for major, 2000+ hours for minor releases103 performance metric captured for Performance tests Couchbase Testing SuiteSanity & Unit TestingFunctional TestingRegressionTestingSystemTestingScaleTestingPerformanceTesting

2016 Couchbase Inc.

2016 Couchbase Inc.

14

2016 Couchbase Inc.

2016 Couchbase Inc.Performance Testing

2016 Couchbase Inc.

2016 Couchbase Inc.Measuring Couchbase Performance

Hundreds of metrics available from Couchbase Server:Throughput: Data Service: >100k reads/sec/node Data Service: >10k writes/sec/nodeQuery Service: >10k queries/sec/nodeLatency: Data Service: 500s-1ms for reads and writes in RAMData Service: 1ms replication between nodesQuery Service: 1-3ms for simple lookup queriesN1QL:SELECT * FROM system:active_requests;SELECT * FROM system:completed_requests;

Built-into client/SDK:Throughput/Total OpsLatency Histograms

2016 Couchbase Inc.

2016 Couchbase Inc.Benchmarks

2015 Couchbase Inc.#

2016 Couchbase Inc.

17

Latest YCSB BenchmarksCouchbase Key-Value AccessDetails: Standard Workload A as Defined by the Benchmark, 1 bucket 150M items with 1K size, Couchbase Server 4.5, 9 c3.8xlarge nodes

2016 Couchbase Inc.

2016 Couchbase Inc.16 threads for workload A. 128 threads for workload E. We think this is the most important one because it encapsulate the whole product offering end to end. From KV to query, from server to sdk and it will catch most eyes because its a direct blow to Mongo.

18

Latest YCSB BenchmarksCouchbase N1QL QueriesDetails: Standard Workload E as Defined by the Benchmark, 1 bucket 150M items with 1K size, Indexing: Memory Optimized GSI, Couchbase Server 4.5, 9 c3.8xlarge nodes

2016 Couchbase Inc.

2016 Couchbase Inc.

19

Couchbase on Google Cloud Engine

1M writes/sec to disk on Google Cloud Engine, median latency 15ms, 95th at 27ms Couchbase server 3.1.5: 50 nodes, n1-standard-16 VMs, 500GB SSD per node

https://cloudplatform.googleblog.com/2015/05/Couchbase-Server-Hits-One-Million-Writes-Per-Second-with-Just-50-Nodes-of-Google-Compute-Engine.html

2016 Couchbase Inc.

2016 Couchbase Inc.16 threads for workload A. 128 threads for workload E. We think this is the most important one because it encapsulate the whole product offering end to end. From KV to query, from server to sdk