elasticsearch and couchbase: configuration to capabilities to real time bi: couchbase connect 2015

ELASTIC AND COUCHBASE: CONFIGURATION TO CAPABILITIES TO REAL-TIME BIBala VenkatraoSenior DirectorProduct Management @ Elastic

David MaierPrincipal Solutions Engineer

©2015 Couchbase Inc. 2

Agenda

Welcome Introduction to Elastic Demo Behind the scenes What’s new in the 2.0 Plug-in


Welcome!

Bala VenkatraoSenior Director, Product Management

Introduction to Elastic Unlocking the value of your data through search, analytics, and more . . .

www.elastic.co

http://www.elastic.co/

www.elastic.co Copyright Elastic 2015 Copying, publishing and/ordistributing without written permission is strictly prohibited

Problem: Complex Data + Diverse Requirements

Social

Location

User-Activit

y Machine

Documents

5

COMPLEX & DIVERSE DATA

MISSION CRITICALBUSINESS REQUIREMENTS

Developer requirements

Many users / use cases

Fast data processing

Large data volumes

Data quality & integrity

How do you build

scalable, Search/Anal

ytics platform?

Cross-source insights


6

New Systems to Address New Requirements

Schemaless

Horizontal Scale

Real Time Data Availability

Fast Search & Query Execution

Structured and unstructured

Flexibility

Sophisticated Query Language


7

Elasticsearch: Search, Store, and Analyze

Distributed, scalable, and resilient Designed for scale-out; high availability

Developer friendlyAPI-first; schemaless, native JSON, client libraries for any language

Real-time Search & AnalyticsReal-time aggregations, geospatial, full-text search; query structured and unstructured data


8

Introducing the Elastic Product Portfolio

KibanaVisualize and explore data

ElasticsearchStore, search, analyze

LogstashHadoop

ConnectorCollect, parse and enrich data

MarvelMonitor and manage

ShieldSecure and protect

FoundElasticsearch as a Service

Open Source Products

Commercial Products

Training Professional Services Support Subscriptions

WatcherNotify and Alert

Powering diverse Use Cases from Search, Analytics, Log Analysis and more…

9www.elastic.co Copyright Elastic 2015 Copying, publishing and/or

distributing without written permission is strictly prohibited

• Analytics/Search Engine based on Apache Lucene

• Schema-free JSON document store• Every field is

indexed/searchable• Scalable sharded distributed

architecture• Simple REST API• (Near) Real-time Search• Aggregations/Analytics support• Geospatial support• Extensibility via plugins

Why use Elasticsearch and Couchbase together?

Elasticsearch• Key-Value document store• Supports JSON, Binary, XML …• N1QL Query Language =>

SQL + JSON• Scalable, distributed architecture

• Active memory management (built-in cache)

• SDK for language integration in Java, .NET, Node.js, Python…• Uses binary wire protocol for data

access

• Extends to iOS, Android and Xamarin with Couchbase Mobile

Couchbase

Couchbase & ElasticsearchA brief Introduction


Couchbase Server & Elasticsearch

Key Reference

tim #2

bob #3

jim #1

Secondary Index

Key Reference

#1 { name : ‘jim’ }

#2 { name : ‘tim’ }

#3 { name : ‘bob’ }

Key-Value AccessTerm Count Referenc

e

tim 1 #1

is 2 #1, #2

sitting 2 #1, #2

next 2 #1, #2

to 2 #1, #2

bob 2 #1, #2

jim 1 #2

Inverted Index


What is it all about?

+ +

= awesome

Demohttps://github.com/dmaier-couchbase/cb-twitter-

demo


Demo Overview

0

1 2

3 4


Twitter Dashboard

Behind the Scenes


Reference Architecture


XDCR

Bucket A

Bucket B

Bucket C

Cluster 1

Bucket A

Bucket B

Bucket C

Cluster 2

Continuous Replication

Topology Aware Recovery from

Network Failure


The Couchbase Transport Plug-in

Needs to be installed on side of Elasticsearch

REST Service Per default on port 9091 XDCR protocol v1

bin/plugin -install transport-couchbase -url \http://packages.couchbase.com.s3.amazonaws.com/releases/elastic-search-adapter/2.0.0/elasticsearch-transport-couchbase-2.0.0.zip

I speak ‘Couchbase’!


Connecting Couchbase Server to Elasticsearch


Couchbase specific Types


Default Mapping Applies to every index couchbaseCheckpoint

The _source field contains the content of the document not the metadata

Don’t store or index Don’t make it accessible via the _all field

couchbaseDocument Handled by the overridden _default_ mapping The _source field contains the content of the

metadata (document id, ..) Don’t include the metadata in the _all field Index the content by using Elasticsearch’s

_default_ mapping Can be overridden by using Elasticsearch’s REST

interface

E = M(c)


Plug-in Settings - Document Type Selector

couchbase.typeSelector Fully qualified class name of a type selector class DefaultTypeSelector maps to ‘couchbaseDocument’ e.g. RegExp on the document id which matches

‘review-123’

couchbase.typeSelector: org.elasticsearch.transport.couchbase.capi.RegexTypeSelector couchbase.documentTypes.review: ^review-(\d)+$


Plug-in Settings - Advanced couchbase.port couchbase.username couchbase.password couchbase.num_vbuckets couchbase.defaultDocumentType couchbase.checkpointDocumentType couchbase.ignoreDeletes couchbase.wrapCounters …

What’s new in the 2.0 Plug-in


What’s new in the 2.0 Plug-in

Type Selectors Support of more than one document

type Support for filtering documents

Possibility to Ignore deletes Ignore Elasticsearch errors To automatically wrap Couchbase

counter documents with JSON for indexing purposes

Bug fixes

Did you know … ?

Thank you!Q&A

elasticsearch and couchbase: configuration to capabilities to real time bi: couchbase connect 2015

Technology

copyright elastic

couchbase elasticsearch

data elasticsearch store

unstructured data

couchbase mobile couchbase

data access

search analytics

elastic demo