datastax enterprise & apache cassandra – essentials for financial services – 20151006

DataStax Enterprise & Apache Cassandra Essentials for Financial Services

Daniel Cohen Solutions Engineer

Company Confidential© 2015 DataStax, All Rights Reserved. 2

1 Introduction to DataStax Enterprise

2 Technology Overview

3 Architecture in Enterprise

4 Use Cases in Finance

5 Questions?

Company Confidential© 2015 DataStax, All Rights Reserved.

DataStax Enterprise

3

A Purpose-Built Platform on

Apache Cassandra

Company Confidential© 2015 DataStax, All Rights Reserved.

Certified Apache Cassandra

4

• Certified for production environments • Rigorous certification process:

– Extensive quality assurance testing – Performance and scale tests with

1,000 node clusters – 3rd party software validation – Defect resolution

• Pass internal certifications and audits

Company Confidential© 2015 DataStax, All Rights Reserved.

Visual Admin

5Confidential

Company Confidential© 2015 DataStax, All Rights Reserved.

Visual Admin

6

• Visual, browser-based user interface • Installation, configuration, and

administration tasks carried out in point-and-click fashion

• REST API allows scripting of any OpsCenter function

• Works on any device (tablet, etc.)

Company Confidential© 2015 DataStax, All Rights Reserved.

Management Services

7

• Repair service – automatically keeps data consistent across a cluster.

• Capacity service – historical trend analysis and forecasts future resource needs.

• Performance service – provides insight into the performance of a cluster with queryable CQL-based diagnostic objects

• Best Practice service – enforces best practice rules across many areas (e.g. security, configuration, etc.) to ensure proper setup and optimization.

• Proactive alerts and external notifications

Company Confidential© 2015 DataStax, All Rights Reserved.

Enterprise Security

8

Company Confidential© 2015 DataStax, All Rights Reserved.

Enterprise Security

8

Inherited from Open Source Cassandra

Internal authentication (IDs, passwords)

Permission management via GRANT/REVOKE

Client-to-node, node-to-node encryption

Company Confidential© 2015 DataStax, All Rights Reserved.

Enterprise Security

8

Inherited from Open Source Cassandra

Internal authentication (IDs, passwords)

Permission management via GRANT/REVOKE

Client-to-node, node-to-node encryption

DataStax Enterprise Only

External authentication(eg Kerberos, LDAP/Active Directory)Transparent data encryption

Data auditing

Company Confidential© 2015 DataStax, All Rights Reserved.

Enterprise Security

8

Inherited from Open Source Cassandra

Internal authentication (IDs, passwords)

Permission management via GRANT/REVOKE

Client-to-node, node-to-node encryption

DataStax Enterprise Only

External authentication(eg Kerberos, LDAP/Active Directory)Transparent data encryption

Data auditing

Company Confidential© 2015 DataStax, All Rights Reserved.

Enterprise Support

9

• 24x7x365 Technical Support • Covers both production and

non-production environments • Scheduled health checks for

assistance on architecture, design, and tuning

• Hot-fix support for emergency maintenance situations

Company Confidential© 2015 DataStax, All Rights Reserved.

DSE Analytics

10

• Integrated Real-Time Analytics • Integrated Batch Analytics • External Batch Analytics

– external Hadoop connectivity, certified to Cloudera, HortonWorks

– link hot transactional data with cold data from data warehouse / lake

ETL

Company Confidential© 2015 DataStax, All Rights Reserved.

DSE Search

11

• DSE Search inherits all the power and capabilities of Solr and builds on top of it

• Built-in scale out and continuous availability for search operations

• Enabled across multiple data centers and cloud providers

• Search via native API or through CQL

ETL

Company Confidential© 2015 DataStax, All Rights Reserved.

DSE In-Memory

12

• Brings all the goodness of Cassandra to an in-memory database

• In-Memory tables look and act like any Cassandra table

• Great for use cases requiring low latency reads

• Combine with in-memory analytics for full in-memory transactional / analytical processing

Company Confidential© 2015 DataStax, All Rights Reserved.

DataStax Enterprise

13

A Purpose-Built Platform on

Apache Cassandra

Company Confidential© 2015 DataStax, All Rights Reserved. 14

1 Introduction to DataStax Enterprise

2 Technology Overview

3 Architecture in Enterprise

4 Use Cases in Finance

5 Questions?

Company Confidential© 2015 DataStax, All Rights Reserved.

• A Distributed NoSQL Database • Google BigTable ✖ Amazon Dynamo

• Continuously Available • Disaster avoidance, not disaster recovery

• Scale Out with Linear Performance • Just add nodes

• Run on Commodity Hardware • In the cloud, on premise, or hybrid

What is Apache Cassandra?

15

Company Confidential© 2015 DataStax, All Rights Reserved.

• Fully Distributed • Data spread over multiple nodes • All nodes participate in a cluster • Configurable data replication

• Masterless • All nodes are equal • Read from or write to any node

• No Single Point of Failure

Distributed Architecture

16

Node 1 1st copy

Node 4

Node 5 Node 2 2nd copy

Node 3 3rd copy

Company Confidential© 2015 DataStax, All Rights Reserved.

A Quick Vocabulary Lesson

17

Company Confidential© 2015 DataStax, All Rights Reserved.

A Quick Vocabulary Lesson

Replication Factor – RF • Specifies how many copies of a row of data C* distributes across nodes • Highly configurable for SLA and network topology (racks, data centers)

CREATE KEYSPACE "Excalibur" WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'NY1' : 3, 'NY2' : 3};

17

Company Confidential© 2015 DataStax, All Rights Reserved.

A Quick Vocabulary Lesson

Replication Factor – RF • Specifies how many copies of a row of data C* distributes across nodes • Highly configurable for SLA and network topology (racks, data centers)

CREATE KEYSPACE "Excalibur" WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'NY1' : 3, 'NY2' : 3};

17

Consistency Level – CL • Specifies how synchronized a row of data must be across C* nodes • Tunable consistency: client application decides CL for each operation

ANY … ONE … QUORUM … EACH_QUORUM … ALL

Company Confidential© 2015 DataStax, All Rights Reserved.

A Quick Vocabulary Lesson

Replication Factor – RF • Specifies how many copies of a row of data C* distributes across nodes • Highly configurable for SLA and network topology (racks, data centers)

CREATE KEYSPACE "Excalibur" WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'NY1' : 3, 'NY2' : 3};

17

Consistency Level – CL • Specifies how synchronized a row of data must be across C* nodes • Tunable consistency: client application decides CL for each operation

ANY … ONE … QUORUM … EACH_QUORUM … ALL

Company Confidential© 2015 DataStax, All Rights Reserved.

Multiple Data Centers

18

Node 1 1st copy

Node 4

Node 5 Node 2 2nd copy

Node 3 3rd copy

Node 1 1st copy

Node 4

Node 5 Node 2 2nd copy

Node 3

London DCNew York DC

Company Confidential© 2015 DataStax, All Rights Reserved.

• Need more storage? Add more nodes.

• Need greater throughput? Add more nodes.

• Predictable, linear performance gains

Scale Out Linearly

19

Company Confidential© 2015 DataStax, All Rights Reserved.

• Need more storage? Add more nodes.

• Need greater throughput? Add more nodes.

• Predictable, linear performance gains

Scale Out Linearly

19

Company Confidential© 2015 DataStax, All Rights Reserved.

• Need more storage? Add more nodes.

• Need greater throughput? Add more nodes.

• Predictable, linear performance gains

Scale Out Linearly

19

Company Confidential© 2015 DataStax, All Rights Reserved.

Linear Scale Out at Netflix

20

Source: The Netflix Tech Blog http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html

Company Confidential© 2015 DataStax, All Rights Reserved.

Node 1 1st copy

Tunable Consistency

21

Node 2 2nd copy

Node 3 3rd copyNode 4

Node 5

Company Confidential© 2015 DataStax, All Rights Reserved.

Node 1 1st copy

Tunable Consistency

21

Node 2 2nd copy

Node 3 3rd copyNode 4

Node 5

Company Confidential© 2015 DataStax, All Rights Reserved.

Node 1 1st copy

Tunable Consistency

21

WriteConsistency Level = QUORUMReplication Factor = 3 Node 2

2nd copy

Node 3 3rd copyNode 4

Node 5

Company Confidential© 2015 DataStax, All Rights Reserved.

Node 1 1st copy

Tunable Consistency

21

ParallelWrite

WriteConsistency Level = QUORUMReplication Factor = 3 Node 2

2nd copy

Node 3 3rd copyNode 4

Node 5

Company Confidential© 2015 DataStax, All Rights Reserved.

Node 1 1st copy

Tunable Consistency

21

ParallelWrite

WriteConsistency Level = QUORUMReplication Factor = 3

5 μs ack

Node 2 2nd copy

Node 3 3rd copyNode 4

Node 5

Company Confidential© 2015 DataStax, All Rights Reserved.

Node 1 1st copy

Tunable Consistency

21

ParallelWrite

WriteConsistency Level = QUORUMReplication Factor = 3

5 μs ack

12 μs ack

Node 2 2nd copy

Node 3 3rd copyNode 4

Node 5

Company Confidential© 2015 DataStax, All Rights Reserved.

Node 1 1st copy

Tunable Consistency

21

ParallelWrite

WriteConsistency Level = QUORUMReplication Factor = 3

5 μs ack

12 μs ack12 μs ack

Node 2 2nd copy

Node 3 3rd copyNode 4

Node 5

Company Confidential© 2015 DataStax, All Rights Reserved.

Node 1 1st copy

Tunable Consistency

21

ParallelWrite

WriteConsistency Level = QUORUMReplication Factor = 3

5 μs ack

12 μs ack

500 μs ack

12 μs ack

Node 2 2nd copy

Node 3 3rd copyNode 4

Node 5

Company Confidential© 2015 DataStax, All Rights Reserved.

Node 1 1st copy

Tunable Consistency

21

ParallelWrite

WriteConsistency Level = QUORUMReplication Factor = 3

5 μs ack

12 μs ack

500 μs ack

12 μs ack

Node 2 2nd copy

Node 3 3rd copyNode 4

Node 5

Company Confidential© 2015 DataStax, All Rights Reserved.

Continuous Availability

22

Node 1 1st copy

Node 4

Node 5 Node 2 2nd copy

Node 4

Node 2 2nd copy

Node 1 1st copy

Node 3 3rd copyNode 3 3rd copyNode 4

Node 5

Company Confidential© 2015 DataStax, All Rights Reserved.

Continuous Availability

22

Node 1 1st copy

Node 4

Node 5 Node 2 2nd copy

Node 4

Node 2 2nd copy

Node 1 1st copy

Node 3 3rd copyNode 3 3rd copyNode 4

Node 5

Company Confidential© 2015 DataStax, All Rights Reserved.

Continuous Availability

22

Node 1 1st copy

Node 4

Node 5 Node 2 2nd copy

ReadConsistency Level = QUORUMReplication Factor = 3

Node 4

Node 2 2nd copy

Node 1 1st copy

Node 3 3rd copyNode 3 3rd copyNode 4

Node 5

Company Confidential© 2015 DataStax, All Rights Reserved.

Continuous Availability

22

Node 1 1st copy

Node 4

Node 5 Node 2 2nd copy

ReadConsistency Level = QUORUMReplication Factor = 3

Node 4

Node 2 2nd copy

Node 1 1st copy

Node 4

Node 5

Company Confidential© 2015 DataStax, All Rights Reserved.

Continuous Availability

22

Node 1 1st copy

Node 4

Node 5 Node 2 2nd copy

ParallelRead

ReadConsistency Level = QUORUMReplication Factor = 3

Node 4

Node 2 2nd copy

Node 1 1st copy

Node 4

Node 5

Company Confidential© 2015 DataStax, All Rights Reserved.

Continuous Availability

22

Node 1 1st copy

Node 4

Node 5 Node 2 2nd copy

ParallelRead

ReadConsistency Level = QUORUMReplication Factor = 3

Node 4

Node 2 2nd copy

Node 1 1st copy

Node 4

Node 5

Company Confidential© 2015 DataStax, All Rights Reserved.

Continuous Availability

22

Node 1 1st copy

Node 4

Node 5 Node 2 2nd copy

ParallelRead

ReadConsistency Level = QUORUMReplication Factor = 3

Node 4

Node 2 2nd copy

Node 1 1st copy

Node 4

Node 5

Company Confidential© 2015 DataStax, All Rights Reserved.

Continuous Availability

22

Node 1 1st copy

Node 4

Node 5 Node 2 2nd copy

ParallelRead

ReadConsistency Level = QUORUMReplication Factor = 3

Node 4

Node 2 2nd copy

Node 1 1st copy

Node 4

Node 5

Company Confidential© 2015 DataStax, All Rights Reserved.

Continuous Availability

22

Node 1 1st copy

Node 4

Node 5 Node 2 2nd copy

ParallelRead

ReadConsistency Level = QUORUMReplication Factor = 3

Node 4

Node 2 2nd copy

Node 1 1st copy

Node 3 3rd copyNode 3 3rd copyNode 4

Node 5

Company Confidential© 2015 DataStax, All Rights Reserved.

Continuous Availability

22

Node 1 1st copy

Node 4

Node 5 Node 2 2nd copy

ParallelRead

ReadConsistency Level = QUORUMReplication Factor = 3

Hints Node 4

Node 2 2nd copy

Node 1 1st copy

Node 3 3rd copyNode 3 3rd copyNode 4

Node 5

Company Confidential© 2015 DataStax, All Rights Reserved.

Continuous Availability

22

Node 1 1st copy

Node 4

Node 5 Node 2 2nd copy

ParallelRead

ReadConsistency Level = QUORUMReplication Factor = 3

Hints Node 4

Node 2 2nd copy

Node 1 1st copy

Node 3 3rd copyNode 3 3rd copyNode 4

Node 5

Company Confidential© 2015 DataStax, All Rights Reserved.

Disaster Avoidance

23

Singapore

New York

London

Company Confidential© 2015 DataStax, All Rights Reserved.

Disaster Avoidance

23

Singapore

New York

London

Company Confidential© 2015 DataStax, All Rights Reserved.

Disaster Avoidance

23

Singapore

New York

London

Company Confidential© 2015 DataStax, All Rights Reserved.

Disaster Avoidance

23

Singapore

New York

London

Company Confidential© 2015 DataStax, All Rights Reserved.

Mixed Workload DSE Cluster

Cassandra Only DC

Transactions

Workload Isolation with DataStax Enterprise

24

Cassandra + Spark DC Analytics

Analytical Application

Online Application

Company Confidential© 2015 DataStax, All Rights Reserved.

CQL Example

CREATE TABLE market_prices ( symbol TEXT, date TIMESTAMP, price DECIMAL, side INT, PRIMARY KEY (symbol, date)) WITH CLUSTERING ORDER BY (date DESC);

CQL: Cassandra Query Language

• Syntax similar to RDBMS SQL • Create objects via DDL

• e.g. CREATE…

• INSERT, UPDATE, DELETE • GRANT, REVOKE • SELECT…WHERE • Command line and GUI tools

25

Company Confidential© 2015 DataStax, All Rights Reserved.

• Certified by DataStax: • Java • C / C# / C++ • Python • Node.js • Ruby • PHP • ODBC • Connector for Apache Spark

• Many community drivers

Drivers & Connectors

26

Company Confidential© 2015 DataStax, All Rights Reserved. 27

1 Introduction to DataStax Enterprise

2 Technology Overview

3 Architecture in Enterprise

4 Use Cases in Finance

5 Questions?

Company Confidential© 2015 DataStax, All Rights Reserved.

DataStax in the Enterprise Ecosystem

28

Offline Application

External Hadoop Distribution(e.g. Cloudera, Hortonworks)

OpsCenter Services

Monitoring Operations

Operational Application

Real Time Search

Real Time Analytics

Batch Analytics

RDBMS

Analytics Transformations

Certified Apache Cassandra – No single point of failure – Linear Scalability – Disaster Avoidance

Security In-Memory

Company Confidential© 2015 DataStax, All Rights Reserved.

Application Tier Resilience

Example: A Single Session

DC NY1 DC NY2

• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra

• Sessions replicate naturally • Recover on disaster / failure

29

App ServerCache

C* C*

App Server

Cache

Company Confidential© 2015 DataStax, All Rights Reserved.

Application Tier Resilience

Example: A Single Session

DC NY1 DC NY2

• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra

• Sessions replicate naturally • Recover on disaster / failure

29

App ServerCache

C* C*

App Server

Cache

Company Confidential© 2015 DataStax, All Rights Reserved.

Application Tier Resilience

Example: A Single Session

DC NY1 DC NY2

• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra

• Sessions replicate naturally • Recover on disaster / failure

29

App ServerCache

C* C*

App Server

Cache

Company Confidential© 2015 DataStax, All Rights Reserved.

Application Tier Resilience

Example: A Single Session

DC NY1 DC NY2

• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra

• Sessions replicate naturally • Recover on disaster / failure

29

App ServerCache

C* C*

App Server

Cache

session

Company Confidential© 2015 DataStax, All Rights Reserved.

Application Tier Resilience

Example: A Single Session

DC NY1 DC NY2

• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra

• Sessions replicate naturally • Recover on disaster / failure

29

C* C*

App Server

Cache

session

Company Confidential© 2015 DataStax, All Rights Reserved.

Application Tier Resilience

Example: A Single Session

DC NY1 DC NY2

• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra

• Sessions replicate naturally • Recover on disaster / failure

29

C* C*

App Server

Cache

Company Confidential© 2015 DataStax, All Rights Reserved.

Application Tier Resilience

Example: A Single Session

DC NY1 DC NY2

• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra

• Sessions replicate naturally • Recover on disaster / failure

29

C* C*

App Server

Cache

Company Confidential© 2015 DataStax, All Rights Reserved.

Application Tier Resilience

Example: A Single Session

DC NY1 DC NY2

• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra

• Sessions replicate naturally • Recover on disaster / failure

29

C* C*

App Server

Cache

session

Company Confidential© 2015 DataStax, All Rights Reserved.

Application Tier Resilience

Example: A Single Session

DC NY1 DC NY2

• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra

• Sessions replicate naturally • Recover on disaster / failure

29

C* C*

App Server

Cache

session

Company Confidential© 2015 DataStax, All Rights Reserved.

A Resilient Enterprise Architecture

30

DC NY1

C*

DC NY2

C*

DC LDN1

C*DC LDN2

C*

…DC NY1

App App

App App

DC LDN2

App App

App App

Company Confidential© 2015 DataStax, All Rights Reserved.

A Resilient Enterprise Architecture

30

DC NY1

C*

DC NY2

C*

DC LDN1

C*DC LDN2

C*

…DC NY1

App App

App App

DC LDN2

App App

App App

Company Confidential© 2015 DataStax, All Rights Reserved.

A Resilient Enterprise Architecture

30

DC NY1

C*

DC NY2

C*

DC LDN1

C*DC LDN2

C*

…DC NY1

App App

App App

DC LDN2

App App

App App

Company Confidential© 2015 DataStax, All Rights Reserved.

A Resilient Enterprise Architecture

30

DC NY1

C*

DC NY2

C*

DC LDN1

C*DC LDN2

C*

…DC NY1

App App

App App

DC LDN2

App App

App App

Company Confidential© 2015 DataStax, All Rights Reserved.

A Resilient Enterprise Architecture

30

DC NY1

C*

DC NY2

C*

DC LDN1

C*DC LDN2

C*

…DC NY1

App App

App App

DC LDN2

App App

App App

Company Confidential© 2015 DataStax, All Rights Reserved.

A Resilient Enterprise Architecture

30

DC NY1

C*

DC NY2

C*

DC LDN1

C*DC LDN2

C*

…DC NY1

App App

App App

DC LDN2

App App

App App

Company Confidential© 2015 DataStax, All Rights Reserved.

A Resilient Enterprise Architecture

30

DC NY1

C*

DC NY2

C*

DC LDN1

C*DC LDN2

C*

…DC NY1

App App

App App

DC LDN2

App App

App App

Company Confidential© 2015 DataStax, All Rights Reserved. 31

1 Introduction to DataStax Enterprise

2 Technology Overview

3 Architecture in Enterprise

4 Use Cases in Finance

5 Questions?

Company Confidential© 2015 DataStax, All Rights Reserved.

A Few Use Cases in Financial Services

32

• Payments & Transactions • Fraud Prevention & Detection • Real-time Market Risk • Regulatory Compliance & Reporting • Authentication & Authorization • Customer 360° & Portfolio Analysis • Recommendations & Personalization • Global Document Search • Geospatial Search

$ £€ ¥🔐 📈

Company Confidential© 2015 DataStax, All Rights Reserved.

• Cassandra dominates time series data • Extremely high ingest rates • Low latency retrievals • Efficient storage

• Capture every tick and trade • Also important metadata, eg volume

• Real-time risk analysis

Capture Financial Market Data

33

• Example: Blue Mountain Capital Management • http://www.planetcassandra.org/blog/5-minute-c-

interview-bluemountain-capital-management/

Company Confidential© 2015 DataStax, All Rights Reserved. 34

Authentication & Authorization

DC LDN1

C*

DC LDN2

C*

DC AMS1

C*

DC ZUR1

C*DSE

Cluster

Web User

App Service

DC LDN2

App App

App App

DC AMS1

App App

App App ……

Company Confidential© 2015 DataStax, All Rights Reserved. 34

Authentication & Authorization

DC LDN1

C*

DC LDN2

C*

DC AMS1

C*

DC ZUR1

C*DSE

Cluster

Web User

App Service

DC LDN2

App App

App App

DC AMS1

App App

App App ……session?

Company Confidential© 2015 DataStax, All Rights Reserved. 34

Authentication & Authorization

DC LDN1

C*

DC LDN2

C*

DC AMS1

C*

DC ZUR1

C*DSE

Cluster

Web User

App Service

DC LDN2

App App

App App

DC AMS1

App App

App App ……session?

Company Confidential© 2015 DataStax, All Rights Reserved. 34

Authentication & Authorization

session

DC LDN1

C*

DC LDN2

C*

DC AMS1

C*

DC ZUR1

C*DSE

Cluster

Web User

App Service

DC LDN2

App App

App App

DC AMS1

App App

App App ……

Company Confidential© 2015 DataStax, All Rights Reserved. 34

Authentication & Authorization

session

DC LDN1

C*

DC LDN2

C*

DC AMS1

C*

DC ZUR1

C*DSE

Cluster

Web User

App Service

DC LDN2

App App

App App

DC AMS1

App App

App App ……quote?

Company Confidential© 2015 DataStax, All Rights Reserved. 34

Authentication & Authorization

session

DC LDN1

C*

DC LDN2

C*

DC AMS1

C*

DC ZUR1

C*DSE

Cluster

Web User

App Service

DC LDN2

App App

App App

DC AMS1

App App

App App ……quote?

Company Confidential© 2015 DataStax, All Rights Reserved. 34

Authentication & Authorization

session

DC LDN1

C*

DC LDN2

C*

DC AMS1

C*

DC ZUR1

C*DSE

Cluster

Web User

App Service

DC LDN2

App App

App App

DC AMS1

App App

App App ……quote

Company Confidential© 2015 DataStax, All Rights Reserved. 34

Authentication & Authorization

session

DC LDN1

C*

DC LDN2

C*

DC AMS1

C*

DC ZUR1

C*DSE

Cluster

Web User

App Service

DC LDN2

App App

App App

DC AMS1

App App

App App ……quote

Company Confidential© 2015 DataStax, All Rights Reserved.

Unify & Scale Legacy Infrastructure

35

…USA Equities

UK FX

UK Bonds

Global Users

Legacy Systems

USA FX

DataStax Enterprise ClusterC*

User Interface / Application Services

Thank you!Questions?