Agility with Data Modeling at DIRECTV Using N1QL in Couchbase with Node.jsFidencio Garrido

@elfidomxGithub: theFido

Isidro Salcedo@isidrosalcedo88

About UsDIRECTV is one of the world's leading providers of digital television entertainment services delivering a premium video experience through state-of-the-art technology, superior entertainment experience, and award winning customer service.

Offers over 3,000 SD, HD, 3D and 4K channels in the US alone

Higher customer satisfaction than the leading cable companies for 14 years running

Global presence servicing multiple countries across the entire western hemisphere.

Over 38 million subscribers around the world.

The Challenge

Developing systems capable of supporting business changes in a fast pace environment.

Supports

• Additional metadata types (different versions of objects)

• Mixed configurations

• Reduction in code changes

Requirements

• Reliable

• Scalable

• High Performance

• Easy Administration (nice to have)

Options

• RDBMS

• Key-Value Stores

• Document DB

Architecture Goals

The overall architecture must perform as if your business depended on it… because it does!

●High Performance

●Reliability

●Scalability

●Easy to Manage

●Never be a constraint

Why Couchbase?

●Performance beast

●Easy setup / administration:●Monitoring tools

●REST Administration API

●All levels of scalability

●Almost flat learning curve (Thanks N1QL!)

●Expressive Language for Query

Couchbase Features

●Document Based

●Persistence + Cache

●Support of Views (map-reduce functions also written in JS)

●Separation of Concerns (index, query, data store)

●Fast Rebalancing

Couchbase N1QL Benefits

●Simplified Code●Reduce Code Complexity: Power of SQL

●Enable Ad-Hoc Queries: interactive queries with low latency with indexes

●Improves granularity of query consistency●Logical-timestamp at run time

●More processing on the server side

Why NodeJS?

●Fast I/O

●Native JSON Speaker

●Soft Typed

●Single threaded but asynchronous

●Light middleware layer provided by express

API Design

●Stateless

●Fully REST Compliant●Resource Oriented (Avoiding Verbs)

●Using the Right Verbs

●Returning the Right HTTP Code

●Using HTTP Headers to Provide Metadata

Starting Design

API

API

APILB

BL

BL

BL

DAS

DAS

DAS

DB

DB

N1QLLB

API BL DAS N1QL

Clients Growing

API

API

APILB

BL

BL

BL

DAS

DAS

DAS

DB

DB

N1QLLB

API BL DAS N1QL

API BL DAS

More Read han Write Operations

API

API

API

LBget

BL

BL

BL

DAS

DAS

DAS

DB

DB

N1QLLB

API BL DAS N1QL

N1QLAPI BL DASLBpostput

delete

More Writes than Read Operations

API

API

API

LBget BL

BL

BL

DAS

DAS

DAS

DB

DB

DBLB

API BL DAS Index

N1QLAPI BL DAS

LBpostput

delete

More Key Based Operation

API

API

APILB

BL

BL

BL

DAS

DAS

DAS

DB

DB

DBLB

API BL DAS

N1QL

N1QL

Index

Geographic Expansion

Enabling an architecture that scales across multiple data centers to support our global business.

Data Center #1 Data Center #2

XDCR

Data Center #4Data Center #3

Using Documents

●NodeJS is enough for the backend●Couchbase is great to store the information

and retrieve it using the document key but…

What happens when you want to use Couchbase as multipurpose Database System?

Can you really have the same capabilities as any other traditional SQL solution?Short answer…

Meet Couchbase N1QL

●A language that extends SQL capabilities to search into documents

●Supports search through complex structures

●Join operations for documents even if they belong to a different bucket

●Advanced features such as nesting and unnesting

●Index Support

Gentle Learning Curve

●A Language we were already familiar with

●Classic SQL

SELECT first_name, last_name FROM customers WHERE email = "someone@directv.com";

●N1QL

SELECT first_name, last_nameFROM bucketWHERE email = "someone@directv.com";

Ad-hoc Querying

{"_metadata": {

"document_type": "program","updated": "2015-06-02","last_user": "admin"

},"data": {

"cast_members" : [{ "person_id" : 14531, "role_id" : 1},{ "person_id " : 5983, "role_id" : 1}

]}

}

SELECT *FROM bucketWHERE _metadata.document_type = "program"AND ANY c_0 IN data.cast_members

SATISFIES c_0.person_id = 15415 END;

Analyze

EXPLAIN SELECT * FROM bucket WHERE _metadata.document_type = "program";

{ "resultset": [ { "input": { "expr": { "left": { "left": { "left": { "path": "dtv-ps-store-dtss", "type": "property" }, "right": { "path": "_metadata", "type": "property" }, "type": "dot_member" },

… }, "input": { "as": "bucket", "bucket": "bucket", "ids": null, "input": { "as": "", "bucket": "bucket", "cover": false, "index": "ix_document_type", "pool": "default",

…

Challenges

●Thinking in Documents●Think in the structure

●Visualize the queries you want to perform{ “user”: “dtv-user”, “phone”: [“(333) 333-3333”, “(444) 444-4444”] }

{ “user”: “dtv-user”, “platform”: [“iOS 7”, “Android 4”, “Windows 8.1”] }

OR{ “user”: “dtv-user”, “platform”: [123, 232, 342] }

●One Data Space{

“_metadata”: {“documentType”: “user”,“updated”: “2015-06-02”,“lastUser”: “admin”

},“data”: {

“user”: “dtv-002”,“state”: “active”,….

}}

Challenges

●Know your data flow: Use explain●Plan your infrastructure

●N1QL does processing… therefore it requires processors. Trust me, it knows what to do with them.

Roadmap

Support expanding business demands and data growth on common, easy to maintain, platform

Upgrade to Couchbase 4.0 to leverage Multi Dimensional Scale

Leverage the power of Global Secondary Indices for instant lookups and query at scale

24

Our development toolkit

Questions