5 fundamental strategies for building a data-centered data

© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

5 Fundamental Strategies for Building a Data-centered Data Center

June 5, 2014 Ken Krupa, Chief Field Architect Pete Aven, Principal Sales Engineer

© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. SLIDE: 2

Last generation

OLTP

Warehouse

Data Marts Archives

“Unstructured”

“ ”

Video Audio

Signals, Logs, Streams

Social

Documents, Messages

{ } Metadata

Search 🔍

Reference Data


Summary – The Data-centered Data Center

Elastic: flexible, shared-nothing, scale-out architecture

Cost competitive: low-cost commodity hardware, lower TCO

Converged: single data layer for operational and analytical workloads

Managing data life-cycle in real-time: prioritize your data storage

Governed, not renegade: customizable, transparent, secure


ELASTIC


Data organization in MarkLogic

Data inserted into stands One stand is in-memory Many other stands are on disk A collection of stands is a forest Each forest is an atomic unit and

can be managed and moved


Servers have Multiple Forests


Scale out


Clustering


Migration

Two forests on one node Bring a second node online Replicate a forest Disable the forest on the

original node Original forest on original

node fails over Enable the original forest as

a replica

X


Migration in one step $ cat forest-migrate.json

{ "operation": "forest-migrate”, "forest": [”forest-in-database", ”another-forest-in-database"], "host": ”destination-host” }

$ curl --anyauth --user user:password -X PUT -d @./forest-migrate.json \ -i -H "Content-type: application/json" \ http://anyhostinthecluster:8002/manage/v2/forests


Cluster topologies XA

RDBMS


Knowing where you’re going - and where you’ve been

Business context

What are your SLAs? How many requests per second does the application

have to support? How will the business grow? What will drive growth - and how fast will it go?

As-Built Capacities

How does your system perform under different usage profiles (e.g., QPS tests)?

How often do you hit the cache? What is your peak storage I/O? What is end-to-end recovery objective/capability?


Performance History


Performance History

To handle more requests: • Fix Configuration • Add Disk IO via Volumes or Nodes • Add Ram to decrease Disk IO • Rewrite Query


Scaling out: questions to consider

How do you know when you need to add a node?

CPU/Memory/IO: when you get close to hardware limits, time to grow

High Performance: SLA’s may drive forest sizes; more docs, time to grow

High Capacity: running low on storage, time to grow

Easy (temporary?) fix—add RAM

Cheaper alternative

Increases cache hits for better performance

Fewer than three hosts, local forests MUST move across hosts

Use forest migrate to move forests from one host to another

Faster than backup/restore

Follow distribution pattern:

Don’t just swap masters/replicas on two: if one goes down, load is not split evenly across cluster

Adding a node - or RAM Migrating a forest


LOWERING TCO THROUGH

COMMODITY HARDWARE


Kryder’s Law: The density of hard drives

increases by a factor of 1,000

every 10.5 years.

(doubling every 13 months)


Moore’s Law: The density of transistors on

integrated circuits doubles

every 18 months.


The laws in action At the end of a 3-year life cycle, one new server can do the job of four old

servers.

At 1.5 Years, you can add 100% more capacity for 50% of original spend

For the cost of storing 1TB in 1996, we will be able to store 1PB in 2016.


Commodity hardware will reduce costs


Hardware/sizing recommendations

2U 25 SFF Chassis 2 Socket

8 Core/2.8Ghz

128GB – 256GB RAM

10GB Network

2 2GB RAID Cards

22 10K 900-1200GB Data Drives


VMWare NetApp recommendations (preliminary)

1U 8SFF Chassis 2 Socket

8 Core/2.8Ghz

128GB – 256GB RAM

10GB Network

1 10GB iSCSI 12-16 Spindles per Server, 10K SAS


Storage Economics

SAN/Scale-up

$2 - $10/Gigabyte

$1M gets: 0.5Petabytes

200,000 IOPS 1Gbyte/sec

NAS Filers

$1 - $5/Gigabyte

$1M gets: 1 Petabyte

400,000 IOPS 2Gbyte/sec

Local Compute

$0.20/Gigabyte

$1M gets: 10 Petabytes

5,000,000 IOPS 40 Gbytes/sec

SAN (Scale Up) Commodity (Scale Out)

Public cloud

$0.04/gb/month

$100K/month: 1.25 Petabytes (HA)

1,500,000 IOPS 150 GB/Sec

Cloud


Signs of war: cloud prices have dropped recently

Google Cloud: - $0.04 GB-month for 1000GB

Amazon EBS: - $.055 GB-month (standard) - $.138 GB-month (provisioned)


Leveraging Scale-out Economics Run on existing Infrastructure today

Leverage Scale-Out Commodity Hardware as you grow

Leverage Cloud today or tomorrow

SAME DATABASE, SAME CODE


DATA LAYER CONVERGENCE

FEWER MOVING PARTS =

MORE AGILITY


Last generation

OLTP

Warehouse

Data Marts Archives

“Unstructured”

“ ”

Video Audio


Social

Documents, Messages

{ } Metadata

Search 🔍

Reference Data


RDBMS: One Tool, Many Contortions OLTP

3rd normal form, updates, simple query Reporting DB

Because the OLTP app slowed down during heavy query use Enterprise Data Warehouse

Because we needed a unified view of the enterprise – Star schema enters the picture

Data Marts Because the EDW didn’t have everything – Also star schema

Federated Because it took too long to agree on a standard model

Hybrid Because Federated is too slow


If I run analytics in my OLTP DB then.... Won’t meet my SLA’s Too expensive No common data model Cache won’t ever be right Too expensive to keep around

context necessary for analytics

If I run transactions in my Analytical DB...

Transaction locks will block aggregate reads

Too expensive Why constrain ad-hoc

query? We need to investigate

The old consensus: mixing is bad


The new wisdom: mixing is good Operational with Analytics Risk calculations Underwriting Compliance Content Discovery Fraud

Analytics with Operations Operational BI Archival/E-Discovery Personalization Situational Awareness

SINGLE SOURCE OF TRUTH


Mixing workloads in MarkLogic - how it works

ML as an analytic database - examples and possibilities

Range indexes: in-memory columnar Query load separation Tiered storage and real-time replication Hadoop MapReduce and HDFS Transactions and ACID help manage and

prioritize data - better performance, lower TCO

Operations and analytics in MarkLogic

COPIES, NOT ETL


INFORMATION LIFE-CYCLE MANAGEMENT (FOR REAL)


Understanding the life cycle The older your database,

the more data you have

The older the data, the less likely you will reuse it

Storage requirements increase, but much of what is stored will go untouched


Data life cycle management, in three easy steps

1. Move data off active system to cheaper system.

2. Keep track of what you moved.

3. Provide facility for getting it back.


CERN: implementation is hard in the RDBMS world DBAs / database developers cannot easily

implement these policies by themselves.

Database admins, application developers, and application owners must work together to: Reduce amount of data produced Allow for database structure that can

facilitate archiving Define data availability requirements for

online data and archive Identify how to leverage database features


CERN: archiving RDBMS data is also difficult

The DBA removes old partitions from the production database and moves them to the archive. One option: use partition exchange to table Post-move jobs can implement compression, drop indexes

Sticking points: Set of data must be consistent Must build support in the application Have to validate access to archived data Archived data must remain readable in future

versions


Tiered Storage With Tiered Storage, you can… Define data tiers based on a range index

Have content balanced into forests by tier

Move an entire tier to different storage

Attach a tier to a different database

Query one database on one tier…

…or the other database on the other tier…

…or both at once All with no downtime, and 100% consistency


0

10000

20000

30000

40000

50000

60000

Tier 1 SAN Exadata ML usingDAS

Tier-1

Effective Cost/TB for Production Storage (all copies)

0100020003000400050006000700080009000

FlexPod/VCE NetApp ML usingDAS

Tier-2


GOVERNANCE + PROVENANCE


Data Governance Considerations

Security



Security

Privacy



Security

Privacy

Provenance



Security

Privacy

Provenance

Retention



Security

Privacy Continuity

Provenance

Retention



Security

Privacy Continuity

Provenance Compliance

Retention


Last Generation

OLTP

Warehouse

Data Marts Archives

“Unstructured”

“ ”

Video Audio


Social

Documents, Messages

{ } Metadata

Search 🔍

Reference Data


New Generation

Application


Summary Elastic systems let you respond rapidly to changing loads - and let you keep costs

in line with usage.

Scale-out systems on commodity hardware are much less expensive and more powerful than scale-up systems.

Converging transactional and analytical workloads into single data layer is not only possible - it is often a great idea. A single data layer can increase agility.

Managing information throughout its life cycle means more than choosing the cheapest storage possible - it means being able to manage and query data in real time.

Proper data governance is simpler in an enterprise NoSQL system.