gridgain basic turbocharge sql queries

© 2015 The Apache Software Foundation. Apache, Apache Ignite, the Apache feather and the Apache Ignite logo are trademarks of The Apache Software Foundation.

Turbocharge Your SQL Queries In-Memory with Apache Ignite

Agenda

• GridGain & Apache Ignite Project

• Ignite Data Fabric:

• In-Memory Data Grid & SQL

• Compute Grid & Streaming • Hadoop & Spark Integration

• Q & A

© 2016 GridGain Systems, Inc.

Apache Ignite Project• 2007: First version of GridGain

(compute grid)• Oct. 2014: GridGain contributes Ignite

to ASF • Aug. 2015: Ignite is the second fastest

project to graduate after Spark • Today: • 60+ contributors and growing rapidly • Huge development momentum -

Estimated 192 years of effort since the first commit in February, 2014 [Openhub]

• Mature codebase: 700k+ SLOC & more than 16k commits January 2016

https://www.openhub.net/p/apache-ignite/estimated_cost


• GridGain Enterprise Edition• Is a binary build of Apache Ignite™ created by GridGain • Added enterprise features for enterprise deployments • Earlier features and bug fixes by a few weeks • Heavily tested


Customer Use Cases

Automated Trading SystemsReal time analysis of trading positions & market risk. High volume transactions, ultra low latencies.

Financial ServicesFraud Detection, Risk Analysis, Insurance rating and modelling.

Online & Mobile AdvertisingReal time decisions, geo-targeting & retail traffic information.

Big Data AnalyticsCustomer 360 view, real-time analysis of KPIs, up-to-the-second operational BI.

Online Gaming

Real-time back-ends for mobile and massively parallel games.

SaaS Platforms & AppsHigh performance next-generation architectures for Software as a Service Application vendors.

Travel & E-CommerceHigh performance next-generation architectures for online hotel booking.


What is an IMDF?

High-performance distributed in-memory platform for computing and transacting on large-scale data sets in near real-time.


What is an IMDF?‣ HPC‣ Machine learning‣ Risk analysis‣ Grid computing

‣ HA API Services‣ Scalable

Middleware

‣ Web-session clustering

‣ Distributed caching‣ In-Memory SQL

‣ Real-time Analytics

‣ Big Data‣ Monitoring tools

‣ Big Data‣ Realtime

Analytics‣ Batch processing

‣ Distributed In-Memory File System

‣ Node2Node & Topic-based Messaging

‣ Fault Tolerance‣ Multiple backups‣ Cluster groups‣ Auto Rebalancing

‣ Complex event processing

‣ Event driven design

‣ Distributed queues

‣ Atomic variables‣ Dist. Semaphore


IMDF: High-Speed Data & Compute Layer

Data Grid

Batch Data

Compute Grid

Transactional & Analytical Workloads

Transactional & Analytical workloads

Streaming Data

External Persistency

External APIs

Back-end users, third-party clients and downstream systems

Downstream Systems

Clients accessing a high-speed distributed multi-facet service


In-Memory Data Grid


Scalability & Resilience with Ignite

Data Grid

Batch Data

Compute Grid



Streaming Data


External APIs


Downstream Systems



Fault Tolerance & Scalability

Replicated Cache Partitioned Cache


Tiered Memory & Local Store

• Tiered Memory

• On-Heap -> Off-Heap -> Disk

• Persistent On-Disk Store

• Fast Recovery

• Local Data Reload

• Eliminate Network and Db impacts when reloading in-memory store


• Unlimited Vertical Scale • Avoid Java Garbage Collection

Pauses • Small On-Heap Footprint • Configurable eviction policies • Off-Heap Indexes • Full RAM Utilisation • Simple Configuration

Off-Heap Memory


• Multiple (up to 32) Data Centres • Complex Replication

Technologies • Active-Active & Active-Passive • Smart Conflict Resolution • Durable Persistent Queues • Automatic Throttling • GridGain Enterprise

Data Center Replication


Storage and Caching using Ignite

Data Grid

Batch Data

Compute Grid



Streaming Data


External APIs


Downstream Systems



• 100% JCache Compliant (JSR 107) – Basic Cache Operations

– Concurrent Map APIs

– Collocated Processing (EntryProcessor) – Events and Metrics

– Pluggable Persistence

• Ignite Data Grid – Fault Tolerance and Scalability – Distributed Key-Value Store – SQL Queries (ANSI 99) – ACID Transactions

– In-Memory Indexes

– RDBMS / NoSQL Integration

In-Memory Data Grid


External Persistence

• Read-through & Write-through

• Support for Write-behind • Configurable eviction policies • DB schema mapping wizard: • Generates all the XML configuration and

Java POJOs


Cache APIs & Queries

• Predicate-based Scan Queries • Text Queries based on Lucene indexing • Query configuration using annotations,

Spring XML or simple Java code • SQL Queries • Memcached (PHP, Java, Python, Ruby) • HTTP REST API • JDBC


• ANSI-99 SQL • In-Memory Indexes (On and Off-

Heap) • Automatic Group By, Aggregations,

Sorting • Cross-Cache Joins, Unions • Use local H2 engine

SQL Support (ANSI 99)


Data Grid: Transactions

• Fully ACID • Support for Transactional & Atomic • Cross-cache transactions • Optimistic and Pessimistic

concurrency modes with multiple isolation levels

• Deadlock protection • JTA Integration


Distributed Java Structures

• Distributed Map (cache) • Distributed Set • Distributed Queue • CountDownLatch • AtomicLong • AtomicSequence • AtomicReference • Distributed ExecutorService


Continuous Queries

• Execute a query and get notified on data changes captured in the filter

• Remote filter to evaluate event and local listener to receive notification

• Guarantees exactly once delivery of an event


• Create chains of event processors & transform an object through various states • Synchronous or asynchronous execution of remote filters & listeners with thread control

Payment Validator Payment Verifier Payment Processor

Ignite Cache

Event Processing using Ignite


In-Memory Compute Grid & the rest


• Branching Pipelines

• Sliding Windows for CEP/Continuous Query

• JMS, Kafka, MQTT, Flume, Camel data streamer integrations

In-Memory Streaming and CEP


• Direct API for MapReduce

• Cron-like Task Scheduling

• State Checkpoints

• Load Balancing • Round-robin • Random & weighted

• Automatic Failover • Per-node Shared State • Zero Deployment • Distributed class loading

In-Memory Compute Grid


Client-Server vs. Affinity Colocation

12

4

3 Data 1

Job 1

2

3Data 2

Job 2

Processing Node 1

Processing Node 2

Client Node

Data Node 1

Data Node 2

Processing Node 1

1

3

4

Data 1

Data 2

2

2

1. Initial Request 2. Fetch data from remote nodes 3. Process entire data-set 4. Return to client

1. Initial Request 2. Co-locating processing with data 3. Return partial result 4. Reduce & return to client


Messaging & Events

• Topic-based messaging • Ordered & Unordered messages • Local & Remote message

listeners • Local & Remote event listeners • Trigger actions from any cluster

events or operations • Query events via IgniteEvents API


• Singletons on the Cluster

– Cluster Singleton

– Node Singleton

– Key Singleton

• Guaranteed Availability

– Auto Redeployment in Case of Failures

In-Memory Service Grid


• Resilience - Build an in-memory resilient service layer between your client application and the grid

• Only expose application APIs and not direct grid APIs

• Service Chaining - Call services internally via compute tasks to create service chains

In-Memory Service Grid


Hadoop & Spark Integration


• Ignite In-Memory File System (IGFS) – Hadoop-compliant

– Easy to Install – On-Heap and Off-Heap

– Caching Layer for HDFS

– Write-through and Read-through HDFS

– Performance Boost

IGFS: In-Memory File System

MR HIVE PIG

In-Memory MapReduce

IGFS

HDFS

IGFS

YARN }Any Hadoop Distro


Hadoop Accelerator: Map Reduce

• In-Memory Performance

• Zero Code Change

• Use existing MR code

• Use existing Hive queries

• No Name Node

• No Network Noise

• In-Process Data Colocation

• Eager Push Scheduling

User Application

Hadoop Client

Ignite Client

Hadoop Jobtracker

Hadoop Name Node

Hadoop Tasktracker

Hadoop Tasktracker

Ignite Data Node (IGFS)

Ignite Data Node (IGFS)

Hadoop Data Node

(HDFS)

Hadoop Data Node

(HDFS)Ignite PathHadoop Path


• IgniteRDD

– Share RDD across jobs on the host

– Share RDD across jobs in the application

– Share RDD globally

• Faster SQL

– In-Memory Indexes

– SQL on top of Shared RDD

Spark & Ignite Integration

Spark Application

Spark Worker

Spark Job

Spark Job

Ignite Node

Yarn Mesos Docker Cloud

Server

Spark Worker

Spark Job

Spark Job

Ignite Node

Server

Spark Worker

Spark Job

Spark Job

Ignite Node

Server

In-Memory Shared RDDs


• Docker • Amazon AWS • Azure Marketplace • Google Cloud • Apache JClouds • Mesos • YARN • Apache Karaf (OSGi)

Deployment


Thank You!

www.gridgain.com

@gridgain

#gridgain

Thank you for joining us. Follow the conversation.

Author: Christos Erotocritou

http://www.gridgain.com

gridgain basic turbocharge sql queries

Documents