datastax enterprise & apache cassandra – essentials for financial services – 20151006

DataStax Enterprise & Apache Cassandra Essentials for Financial Services

Daniel Cohen Solutions Engineer

Company Confidential© 2015 DataStax, All Rights Reserved. 2

1 Introduction to DataStax Enterprise

2 Technology Overview

3 Architecture in Enterprise

4 Use Cases in Finance

5 Questions?

Company Confidential© 2015 DataStax, All Rights Reserved.

DataStax Enterprise

3

A Purpose-Built Platform on

Apache Cassandra


Certified Apache Cassandra

4

• Certified for production environments • Rigorous certification process:

– Extensive quality assurance testing – Performance and scale tests with

1,000 node clusters – 3rd party software validation – Defect resolution

• Pass internal certifications and audits


Visual Admin

5Confidential


Visual Admin

6

• Visual, browser-based user interface • Installation, configuration, and

administration tasks carried out in point-and-click fashion

• REST API allows scripting of any OpsCenter function

• Works on any device (tablet, etc.)


Management Services

7

• Repair service – automatically keeps data consistent across a cluster.

• Capacity service – historical trend analysis and forecasts future resource needs.

• Performance service – provides insight into the performance of a cluster with queryable CQL-based diagnostic objects

• Best Practice service – enforces best practice rules across many areas (e.g. security, configuration, etc.) to ensure proper setup and optimization.

• Proactive alerts and external notifications


Enterprise Security

8


Enterprise Security

8

Inherited from Open Source Cassandra

Internal authentication (IDs, passwords)

Permission management via GRANT/REVOKE

Client-to-node, node-to-node encryption


Enterprise Security

8

Inherited from Open Source Cassandra

Internal authentication (IDs, passwords)

Permission management via GRANT/REVOKE

Client-to-node, node-to-node encryption

DataStax Enterprise Only

External authentication(eg Kerberos, LDAP/Active Directory)Transparent data encryption

Data auditing


Enterprise Support

9

• 24x7x365 Technical Support • Covers both production and

non-production environments • Scheduled health checks for

assistance on architecture, design, and tuning

• Hot-fix support for emergency maintenance situations


DSE Analytics

10

• Integrated Real-Time Analytics • Integrated Batch Analytics • External Batch Analytics

– external Hadoop connectivity, certified to Cloudera, HortonWorks

– link hot transactional data with cold data from data warehouse / lake

ETL


DSE Search

11

• DSE Search inherits all the power and capabilities of Solr and builds on top of it

• Built-in scale out and continuous availability for search operations

• Enabled across multiple data centers and cloud providers

• Search via native API or through CQL

ETL


DSE In-Memory

12

• Brings all the goodness of Cassandra to an in-memory database

• In-Memory tables look and act like any Cassandra table

• Great for use cases requiring low latency reads

• Combine with in-memory analytics for full in-memory transactional / analytical processing


DataStax Enterprise

13

A Purpose-Built Platform on

Apache Cassandra






5 Questions?


• A Distributed NoSQL Database • Google BigTable ✖ Amazon Dynamo

• Continuously Available • Disaster avoidance, not disaster recovery

• Scale Out with Linear Performance • Just add nodes

• Run on Commodity Hardware • In the cloud, on premise, or hybrid

What is Apache Cassandra?

15


• Fully Distributed • Data spread over multiple nodes • All nodes participate in a cluster • Configurable data replication

• Masterless • All nodes are equal • Read from or write to any node

• No Single Point of Failure

Distributed Architecture

16

Node 1 1st copy

Node 4

Node 5 Node 2 2nd copy

Node 3 3rd copy


A Quick Vocabulary Lesson

17



Replication Factor – RF • Specifies how many copies of a row of data C* distributes across nodes • Highly configurable for SLA and network topology (racks, data centers)

CREATE KEYSPACE "Excalibur" WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'NY1' : 3, 'NY2' : 3};

17



Replication Factor – RF • Specifies how many copies of a row of data C* distributes across nodes • Highly configurable for SLA and network topology (racks, data centers)

CREATE KEYSPACE "Excalibur" WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'NY1' : 3, 'NY2' : 3};

17

Consistency Level – CL • Specifies how synchronized a row of data must be across C* nodes • Tunable consistency: client application decides CL for each operation

ANY … ONE … QUORUM … EACH_QUORUM … ALL


Multiple Data Centers

18

Node 1 1st copy

Node 4


Node 3 3rd copy

Node 1 1st copy

Node 4


Node 3

London DCNew York DC


• Need more storage? Add more nodes.

• Need greater throughput? Add more nodes.

• Predictable, linear performance gains

Scale Out Linearly

19


Linear Scale Out at Netflix

20

Source: The Netflix Tech Blog http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html

http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html


Node 1 1st copy

Tunable Consistency

21

Node 2 2nd copy

Node 3 3rd copyNode 4

Node 5


Node 1 1st copy

Tunable Consistency

21

WriteConsistency Level = QUORUMReplication Factor = 3 Node 2

2nd copy


Node 5


Node 1 1st copy

Tunable Consistency

21

ParallelWrite

WriteConsistency Level = QUORUMReplication Factor = 3 Node 2

2nd copy


Node 5


Node 1 1st copy

Tunable Consistency

21

ParallelWrite

WriteConsistency Level = QUORUMReplication Factor = 3

5 μs ack

Node 2 2nd copy


Node 5


Node 1 1st copy

Tunable Consistency

21

ParallelWrite


5 μs ack

12 μs ack

Node 2 2nd copy


Node 5


Node 1 1st copy

Tunable Consistency

21

ParallelWrite


5 μs ack

12 μs ack12 μs ack

Node 2 2nd copy


Node 5


Node 1 1st copy

Tunable Consistency

21

ParallelWrite


5 μs ack

12 μs ack

500 μs ack

12 μs ack

Node 2 2nd copy


Node 5


Continuous Availability

22

Node 1 1st copy

Node 4


Node 4

Node 2 2nd copy

Node 1 1st copy

Node 3 3rd copyNode 3 3rd copyNode 4

Node 5



22

Node 1 1st copy

Node 4


ReadConsistency Level = QUORUMReplication Factor = 3

Node 4

Node 2 2nd copy

Node 1 1st copy


Node 5



22

Node 1 1st copy

Node 4



Node 4

Node 2 2nd copy

Node 1 1st copy

Node 4

Node 5



22

Node 1 1st copy

Node 4


ParallelRead


Node 4

Node 2 2nd copy

Node 1 1st copy

Node 4

Node 5



22

Node 1 1st copy

Node 4


ParallelRead


Node 4

Node 2 2nd copy

Node 1 1st copy


Node 5



22

Node 1 1st copy

Node 4


ParallelRead


Hints Node 4

Node 2 2nd copy

Node 1 1st copy


Node 5


Disaster Avoidance

23

Singapore

New York

London


Mixed Workload DSE Cluster

Cassandra Only DC

Transactions

Workload Isolation with DataStax Enterprise

24

Cassandra + Spark DC Analytics

Analytical Application

Online Application


CQL Example

CREATE TABLE market_prices ( symbol TEXT, date TIMESTAMP, price DECIMAL, side INT, PRIMARY KEY (symbol, date)) WITH CLUSTERING ORDER BY (date DESC);

CQL: Cassandra Query Language

• Syntax similar to RDBMS SQL • Create objects via DDL

• e.g. CREATE…

• INSERT, UPDATE, DELETE • GRANT, REVOKE • SELECT…WHERE • Command line and GUI tools

25


• Certified by DataStax: • Java • C / C# / C++ • Python • Node.js • Ruby • PHP • ODBC • Connector for Apache Spark

• Many community drivers

Drivers & Connectors

26






5 Questions?


DataStax in the Enterprise Ecosystem

28

Offline Application

External Hadoop Distribution(e.g. Cloudera, Hortonworks)

OpsCenter Services

Monitoring Operations

Operational Application

Real Time Search

Real Time Analytics

Batch Analytics

RDBMS

Analytics Transformations

Certified Apache Cassandra – No single point of failure – Linear Scalability – Disaster Avoidance

Security In-Memory


Application Tier Resilience

Example: A Single Session

DC NY1 DC NY2

• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra

• Sessions replicate naturally • Recover on disaster / failure

29

App ServerCache

C* C*

App Server

Cache




DC NY1 DC NY2



29

App ServerCache

C* C*

App Server

Cache

session




DC NY1 DC NY2



29

C* C*

App Server

Cache

session




DC NY1 DC NY2



29

C* C*

App Server

Cache




DC NY1 DC NY2



29

C* C*

App Server

Cache

session


A Resilient Enterprise Architecture

30

DC NY1

C*

DC NY2

C*

DC LDN1

C*DC LDN2

C*

…DC NY1

App App

App App

DC LDN2

App App

App App






5 Questions?


A Few Use Cases in Financial Services

32

• Payments & Transactions • Fraud Prevention & Detection • Real-time Market Risk • Regulatory Compliance & Reporting • Authentication & Authorization • Customer 360° & Portfolio Analysis • Recommendations & Personalization • Global Document Search • Geospatial Search

$ £€ ¥🔐 📈


• Cassandra dominates time series data • Extremely high ingest rates • Low latency retrievals • Efficient storage

• Capture every tick and trade • Also important metadata, eg volume

• Real-time risk analysis

Capture Financial Market Data

33

• Example: Blue Mountain Capital Management • http://www.planetcassandra.org/blog/5-minute-c-

interview-bluemountain-capital-management/

http://www.planetcassandra.org/blog/5-minute-c-interview-bluemountain-capital-management/


Authentication & Authorization

DC LDN1

C*

DC LDN2

C*

DC AMS1

C*

DC ZUR1

C*DSE

Cluster

Web User

App Service

DC LDN2

App App

App App

DC AMS1

App App

App App ……



DC LDN1

C*

DC LDN2

C*

DC AMS1

C*

DC ZUR1

C*DSE

Cluster

Web User

App Service

DC LDN2

App App

App App

DC AMS1

App App

App App ……session?



session

DC LDN1

C*

DC LDN2

C*

DC AMS1

C*

DC ZUR1

C*DSE

Cluster

Web User

App Service

DC LDN2

App App

App App

DC AMS1

App App

App App ……



session

DC LDN1

C*

DC LDN2

C*

DC AMS1

C*

DC ZUR1

C*DSE

Cluster

Web User

App Service

DC LDN2

App App

App App

DC AMS1

App App

App App ……quote?



session

DC LDN1

C*

DC LDN2

C*

DC AMS1

C*

DC ZUR1

C*DSE

Cluster

Web User

App Service

DC LDN2

App App

App App

DC AMS1

App App

App App ……quote


Unify & Scale Legacy Infrastructure

35

…USA Equities

UK FX

UK Bonds

Global Users

Legacy Systems

USA FX

DataStax Enterprise ClusterC*

User Interface / Application Services

Thank you!Questions?