con8862 no sql, json and time series data

NoSQL , JSON and TimeSeries Data - CON8862

Anuj Sahni, Principal Product Manager

Ashok Holla, Senior Sales Consultant


Safe Harbor

The following is intended to outline our general product direction. It is

intended for information purposes only, and may not be incorporated into

any contract. It is not a commitment to deliver any material, code, or

functionality, and should not be relied upon in making purchasing decisions.

The development, release, and timing of any features or functionality

described for Oracle’s products remains at the sole discretion of Oracle.


Agenda

Why JSON ?

Its role in NoSQL and Big Data

NoSQL Use Cases

Oracle NoSQL DB overview

Architecture

Data Modeling using JSON schema

Time Series case study and demo


Brief History of data Interchange

10 years ago XML was primary data exchange format

– Vast improvement over SGML (Standard Generalized Markup Language)

– Enabled people to exchange documents across HTTP

Language of Internet


Brief History of data Interchange (cont…)

Last few years, a bold transformation happening in world of data

interchange

– JSON (JavaScript Object Notation) emerged as an alternative to XML

looks more data-structure like,

Light weight/bandwidth-non-intensive,

language independent,

Language of Internet


What made JSON Cool ?

API

– No business value gain operating in silos

– REST replacing SOAP as data transfer protocol

The Internet of Things

– “JSON better adapted to devices with

limited capabilities”

Full-stack JavaScript

– JavaScript is new hot

– Node.js gone mainstream

Big Data


Thoughts Things Processes

Newer Challenges


Get Fast Answers to New Questions

What is Big data ?


NoSQL and Big Data Where did it come from?

SQL

JDBC,

ODBC

General

Purpose

Managed

Schemas

Security,

Backups

Analytics

…

Distributed

Processing

Distributed,

Replicated

File System

Driver

Application

NoSQL databases

Flexible

Schemas

Sharded,

Replicated

Database

High Speed,

Simple Ops

More Flexible Schema

Management

Globally Distributed,

“Always On” data

Competitive Advantages

of “Fast Data”

Lower TCO,

commodity HW scale-out


Kinds of NoSQL Database Based on Storage Model

Key Value Columnar Document Graph

Oracle NoSQL DB

Riak

Dynamo

Voldemort

Cassandra

HBase

MongoDB

CouchBase

Neo4J

GraphDB


Few common characteristics

They all store data in de-normalized fashion

– Don’t support PK-> FK relationship (no cascade deletes)

– Don’t support complex Joins

– Don’t manage constraints (application’s job)

Maintains data relationships using JSON/BSON representation

– Simple lookup operations by primary keys are generally enough

In NoSQL Databases


Agenda

Why JSON ?


NoSQL Use Cases


Architecture




Point Flexibility Point

Web-Scale

Transaction Processing

High velocity, volume,

variety, low information

density data capture

Web browsing, Shopping

Carts, CDR processing,

Sensor data, Stock data

Web-Scale

Personalization

Guaranteed low latency

lookups for end-customers

Advertising, Product

Recommendations, Online

Catalogs, Social Media, Profile

Management, Personalization

Real-Time

Event Processing

Real time events trigger rule

that perform low latency lookups

Medical Monitoring, Factory

Automation, Oil & Gas, Geo-

location

NoSQL Database Use Case Summary


Use Case – Online Social Gaming

Problem

– Very low latency requirements – Player movement must feel like a real time operation, while

being tracked on the server

– Extreme data velocity – Popular games, large scale user base (Farmville boasts 80 million active

users)

– Highly available – 24/7 sites

– Write heavy workloads

Solution – Where to use a NoSQL Database?

– Player interaction data store – Database to track player movement and game interaction

– Game play statistics – Per player usage statistics

– Persistent chat store – For games that allow player communication via chat, the NoSQL

database is used as a persistent message store (auditing and COPA compliance)

Web Scale Personalization


Use Case – Online Social Gaming Web Scale Personalization

JSON maps well

with HTML5/JS

based Client

Player interaction

stored as simple

JSON events

Easy to store

entire player state

reliably

NoS

QL D

B D

river

Applic

ation

Shard 2

Shard N

Shard 1

Schema Evolution

for agile

development

Transparent load

balancing On-demand

cluster expansion

REST


Use Case – Smart Meter Analytics

Problem

– Real Time Access to time-series data – identity patterns hidden within the terabytes of data

Identify theft,

predict system failures ,

better manage operations

– Large volume of machine data – 1000 fold increase in data collection

Write heavy workloads

– Highly available – 24/7 sites

Solution – Where to use a NoSQL Database?

– Fast time based queries – Daily, weekly, yearly consumption trends.

– Flexible data model – can adapt to various kind of sensors or changes in sensor format

– Scalable and Reliable performance – Easy to scale as workload increases because of more

sensors

Web Scale Transactions


Smart Meter Analytics Web Scale Transactions

Analytic

Engine

User

Interface

Export

back to

Smart

Grid N

oS

QL D

B D

river

Applic

ation

Shard 2

Shard N

Shard 1

Time Series Data

Data coming in

many different

format

Data

coming

fast

Horizontally

scalable

database

JSON

Easy to model

keys for time

bases queries


Agenda

Why JSON ?


Introduction to NoSQL


Architecture




Flexible Key-Value Data Model

ACID transactions

Horizontally Scalable

Highly Available

Elastic Configuration

Simple administration

Intelligent Driver

Commercial grade software and support

Features

Oracle NoSQL Database

Application

Storage Nodes Datacenter B

Storage Nodes Datacenter A

Application

NoSQL DB Driver

Application

NoSQL DB Driver

Application

Java SE 6 (JDK 1.6.0 u25)+; Solaris or Linux

Scalable, Highly Available, Key-Value Database


Simple Data Model

Simple, flexible key-value paradigm

Simple operations – read/insert/update/delete, RMW support

Ordered scan of key ranges

Unordered scan of all data (non-transactional)

Streaming API for LOBs

Java and C APIs

Key-value pairs


Compact, highly efficient serialization

Synergy with Hadoop

Supports serialization from and to JSON strings

Bindings from serialized formats to language constructs

Easy to use mechanism for schema evolution

Schema definition tracked with schema ID of data writer

Schema versions can be opaque to readers

Oracle NoSQL – Why AVRO?


JSON support in Oracle NoSQL DB Schema Creation


JSON support in Oracle NoSQL DB (cont…) Data Manipulation


JSON support in Oracle NoSQL DB (cont…) Schema Evolution


JSON support in Oracle NoSQL DB (cont…) Agile development


Time Series Analysis using

Oracle NoSQL database


Big Data Challenge

High velocity of stock tick data generated

at a massive volume each day

Millions of customers trading stocks on-

line based on the current value and the

historical trends.

How do you ensure these buy/sell

transactions happens in real time?

How do you store large volume of tick

data in a quick and consistent manner?

– So you can run the trend analysis on the time

series data.

UC - Stock Tick Analysis


Introduction – Oracle Investments Inc. Goal

– Ensure fast and consistent storage of

large volume of tick data

– Deliver a platform where customers can:

Analyze historical trends by plotting

time-series data

Trade stocks in real time by placing

buy/sell orders

Challenge

– 100k’s customers trading concurrently.

– 10k’s of ticks each second

– Real time response required for order

execution


Introduction – Oracle Investments Inc.

Value

– Real time:

Data capture and random access to

time sensitive data

Order processing

– Automatic data partitioning for easy

scalability

– Lowest $/ops for a consistent storage

– Highly Available system for 24/7 operation


Big Data Appliance

Oracle NoSQL DB

Oracle NoSQL DB as Data Store

NoSQL Driver

Read, Update

Stores all key interactions required to

drive application. For example:

User Profile

Stock Tick Data

Transaction information

Why Oracle NoSQL Database?

Highly Scalable

Extremely performant

Super low-latency


Architecture

Business Layer

Big Data Appliance

Oracle NoSQL DB

NoSQL Driver

Stock Tick Generator

Tick data every 5 seconds

Trading

Read, Update

Analytics

Time Series

Analysis

Order requests

User Profile Updates

Order processing

Historical Data


NoSQL Data Representation

{"lasttimestamp":"1367251181219“,"lastclose":"45.17","lasthigh":"32.25","lastlow":"31.94","lastopen":"32.21","lastvolume":"26"

,

"stockArray":

[

{"timestamp":"1367245601219“, "value":"32.21“, "open":"32.23“, "high":"32.27“, "low":"32.14“,"volume":"4700"}

{"timestamp":"1367245602220“, "value":"32.15“, "open":"32.14“, "high":"32.27“, "low":"32.09“,"volume":"4000"}

{"timestamp":"1367245603319“, "value":"32.11“, "open":"32.09“, "high":"32.27“, "low":"32.14“,"volume":"3500“}

]

}

Denormalized Data


NoSQL Data Representation

Get Stock Information for a Company

– Key = /SYMBOL/YYYY-MM/-/DD

– (Ex: /ORCL/2013-05/-/02 , Stock data for 2nd of May 13)

Write = put(key, value)

Read = get(key)


Demo


Oracle NoSQL DB Resources

Support via OTN forums and Oracle Support process

OTN Forum:

– Forum Home » Big Data » NoSQL Database

– forums.oracle.com/forums/forum.jspa?forumID=1388

Oracle.com:

– www.oracle.com/us/products/database/nosql/overview/index.html

OTN (including documentation and download):

– www.oracle.com/technetwork/products/nosqldb/overview/index.html

Support

https://forums.oracle.com/forums/index.jspa;jsessionid=8d92106930d7d68fda8aaa414c6a878b845cd0c9e1c5.e38La3uKbh4Oci0Ma3uKa30NbNuMe0?categoryID=84

https://forums.oracle.com/forums/category.jspa;jsessionid=8d92106930d7d68fda8aaa414c6a878b845cd0c9e1c5.e38La3uKbh4Oci0Ma3uKa30NbNuMe0?categoryID=512

https://forums.oracle.com/forums/forum.jspa;jsessionid=8d92106930d7d68fda8aaa414c6a878b845cd0c9e1c5.e38La3uKbh4Oci0Ma3uKa30NbNuMe0?forumID=1388&start=0

https://forums.oracle.com/forums/forum.jspa?forumID=1388



http://www.oracle.com/us/products/database/nosql/overview/index.html

http://www.oracle.com/technetwork/products/nosqldb/overview/index.html


Oracle NoSQL DB Resources

On OTN and in download

– docs.oracle.com/cd/NOSQL/html/index.html

Getting Started Guides

Programmatic API

Installation & Release Notes

FAQ

Documentation

http://docs.oracle.com/cd/NOSQL/html/index.html




References

Why JSON will continue to push XML out of the picture

Interactive Matter Lab

Architecting the Internet of Things(p. 102)

http://blog.appfog.com/why-json-will-continue-to-push-xml-out-of-the-picture/

http://interactive-matter.eu/blog/2010/08/14/ajson-handle-json-with-arduino/

http://www.cui-zy.cn/Recommended/Linux/Architecting_the_Internet_of_Things.pdf


Appendix


Configurable Durability per operation

Configurable Consistency per

operation

ACID by default

Transaction scope is single API call

Records share same major key

Multiple operations supported

Greater Flexibility

Features – Configurable ACID Transactions


Developing Applications Data Modeling

Subject ID Qualifier – Value Type Range

Binary

JSON/Avro

RDF Triples

Tables/Rows

Queries

sensors by section

Pressure Sensor Sensor ID Left Front What a List The Value

Value Types

=

A specific sensor

All measures of a sensor A range of sensor measures

Major Key

/ / / / / /

Minor Key

CT 123-PS3234 LF – psi, irate Timestamp Array of Int = / / / / / /


Increase Data Capacity

– Add more storage nodes

– New shards automatically created

Increase Data Throughput

– More shards = better write throughput

– More replicas/shard = better read throughput

On Demand

Elasticity

NoSQL DB Driver

Application

Master

Replica

Replica

StorageNode StorageNode StorageNode

Shard-1

Master

Replica

Replica

Shard-2

On-Demand Cluster Expansion


Supports heterogeneous storage

topology

Replicas move from over-utilized to

under-utilized storage nodes

Number of shards and replication

factor remain unchanged

Improve Performance

Rebalance an Unbalanced Store

Storage Node 1 Storage Node 2 Storage Node 3

Represents a data fragment


What’s Really Important?

Technical Feature Importance Why

Storage Model Not really Will merge over time

Specific Features Somewhat Application requirements?

Performance Somewhat Rapid changes, YMWV

Integration Critical Long term, Repetitive cost

Reliability/Support Critical Early products, Product

direction

Predictability Critical Production reqs & SLAs


Query NoSQL data from Oracle Database

Access NoSQL data from Hadoop for DW and analytics

Share data with Coherence for extensible in-memory cache grid

Persist history & event streams for processing with OEP

Store & query RDF data using Oracle RDF for NoSQL

Integration Oracle NoSQL Database: Integrated out of the box


Reliability & Support

Decades of widespread, reliable deployment experience

15+ years of mission-critical non-relational database technology

Oracle Support available for both Enterprise and Community Edition

Oracle NoSQL Database: Enterprise-Grade Software & Support


Automatic election of new Master

Rejoining nodes automatically

synchronize with the Master

Isolated nodes can still service reads

All nodes are symmetric

Automatic Failover

Highly Reliable

Replication factor = 5

Rep Node

Master

Rep Node

Replica Rep Node

Replica

Rep Node

Replica

Rep Node

Replica

New Master

con8862 no sql, json and time series data

Technology

oracle andor

data relationships

sensor data

mainstream big data

data transfer protocol

kinds of nosql database

sole discretion of oracle

json cool