Five In-depth Technology and Architecture Sessions on Data Virtualization

Session 3: Scalability

Today’s Speaker

■ Jesus Barrasa

Senior Solutions Architect, Denodo

Architect-to-Architect Series

■ Series of five webinars over 3 months

■ Deeper look into Denodo Platform

■ Architectural Overview

■ Performance

■ Scalability (today’s session)

■ Data Discovery and Governance

■ Security

Denodo Express

■ Denodo Express

■ Free to Download

■ Fully functioning Data Virtualization Platform

■ Single user, supports common data sources

■ Many of the same capabilities of Denodo Platform

■ Performance, Data Discovery, Governance, internal Security, Publishing, …

Scalability – Architecture Modules

Scalability – HA and Clustering

■ Denodo Cache

■ The Denodo Platform incorporates an advanced cache system to improve performance

■ For data materialization it makes use of an external JDBC database (physical or in-memory)

■ Denodo High-Availability

■ Set of features provided by the Denodo Platform to support High Availability and Load Balancing

■ Denodo Cluster Management

■ Tool for monitoring and managing a cluster of Denodo servers

Scalability – HA and Clustering

■ Singleton

■ High Availability Architecture

■ Active-Passive

■ Active-Active

■ Vertical Scalability

■ VM Support

■ Metadata Sharing

■ Horizontal Scalability

■ Clustering

■ Shared Cache or Individual Cache

• A single Data Virtualization server instance

…

Data sources

LDAP Auth

Cache Database

Client ApplicationESB/BPM/Portal…

Singleton

…

Data sources

LDAP Auth

Cache Database

Client ApplicationESB/BPM/Portal…

HA – Active-Passive

Load Balancer

HA – Active-Passive

■ All queries sent to Active server

■ Passive server running, but not handling requests

■ Both servers ‘pinged’ by Load Balancer with appliance health check or using ping script

■ When Load Balancer detects Active server failure

■ Requests redirected to Passive server

■ Passive server becomes ‘Active’

■ When former Active server restored

■ Load Balancer determines new roles…

…

Data sources

LDAP Auth

Cache Database

Client ApplicationESB/BPM/Portal…

HA – Active-Active

Load Balancer

HA – Active-Active

■ Load Balancer distributes queries between both servers

■ Both servers running and handling requests

■ Both servers ‘pinged’ by Load Balancer with appliance health check or using ping script

■ When Load Balancer detects server failure

■ All requests redirected to other server

■ When ‘failed’ server restored

■ Added back to load balancing pool

■ Starts receiving queries

Vertical Scaling

■ Java limitations inhibit vertical scaling for singletons

■ Virtualization

■ Run Denodo Platform instances in VM

■ VMWare, Hyper-V, Xen, …

■ Very common scenario

■ Multiple Instances

■ Install multiple instances of Denodo Platform

■ Run multiple instances of Denodo Platform from a single Denodo Platform installation

■ Share metadata and cache

Vertical Scaling (Cont’d)

Cache DatabaseMetadata

Master

Slave-1

Slave-2

Slave-3

Vertical Scaling (Cont’d)

■ First started instance is ‘master’

■ All others are ‘slaves’

■ Only ‘master’ instance can modify metadata

■ Slaves read metadata at startup time

■ Pros

■ Only one copy of metadata

■ Cons

■ Port configurations need modifying

■ SPoF

Horizontal Scaling – Clustering

Load Balancer

…

Dat

a so

urc

es

Virtual ServerVDP Cluster: 192.168.0.10:9999Web Container Cluster: 192.168.0.10:9090

VDP Cluster:• Denodo1:9999• Denodo2:9999• Denodo3:9999• Denodo4:9999• Denodo5:9999

Web Cont. Cluster:• Denodo1:9090• Denodo2:9090• Denodo3:9090• Denodo4:9090• Denodo5:9090

Clustering – Shared Cache

Load Balancer

Virtual ServerVDP Cluster: 192.168.0.10:9999Web Container Cluster: 192.168.0.10:9090

VDP Cluster:• Denodo1:9999• Denodo2:9999• Denodo3:9999• Denodo4:9999• Denodo5:9999

Web Cont. Cluster:• Denodo1:9090• Denodo2:9090• Denodo3:9090• Denodo4:9090• Denodo5:9090

Cache Database

Clustering – Separate Caches

Load Balancer

Virtual ServerVDP Cluster: 192.168.0.10:9999Web Container Cluster: 192.168.0.10:9090

VDP Cluster:• Denodo1:9999• Denodo2:9999• Denodo3:9999• Denodo4:9999• Denodo5:9999

Web Cont. Cluster:• Denodo1:9090• Denodo2:9090• Denodo3:9090• Denodo4:9090• Denodo5:9090

Cache Database

Cache Database

Cache Database

Cache Database

Cache Database

Clustering - Caches

■ Shared Cache

■ Any server can populate cache

■ All servers can access cached data

■ Increased ‘hit ratio’

■ Cache can become performance bottleneck

■ Separate Caches

■ Each server has its own cache

■ Avoids potential performance bottleneck

■ Access only data it has cached

■ Removes SPoF

Distributed Environments

■ Local caches help to achieve performance goals when servers are in geographically dispersed locations

New York City

Omaha

San Francisco

Houston

Distributed Environments (Cont’d)

■ A remote Denodo server can be viewed as a data source to another server

■ Promotes multi-layered architecture

■ Queries can be delegated to remote servers to reduce network data movement

■ Same query delegation mechanism as talked about in last session

Distributed Environments (Cont’d)

■ Remote query delegation…

Local Data Sources

Cache Cache

Local Data Sources

New York City HoustonClient

10 M

rows

5 M

rows

JOIN Result:

20,000 rowsJOIN Result:

2,500 rows

2 M

rows

Distributed Environments (Cont’d)

■ Shared views…

Local Data Sources

Cache Cache

Local Data Sources

New York City HoustonClient

10 M

rows

5 M

rows

JOIN Result:

20,000 rowsJOIN Result:

2,500 rows

2 M

rows

New York City Houston

Disaster Recovery

■ Business Continuity policies usually require that critical IT systems are backed up to a DR center

■ Denodo supports scripted synchronization of metadata with DR systems

■ Using Export and Import tools

■ Sync up can be scheduled (time-based) or on an event (promotion to production)

Scalability - Summary

■ Denodo Platform scales from single instance to geographically distributed system

■ Grow system without reconfiguring

■ Supports HA configurations

■ Active-Passive, Active-Active

■ Vertical scalability options

■ Support for DR site synchronization

Q&A

Data Virtualization – Next Steps

Move forward at your own pace

Download Denodo Express –

The fastest way to Data Virtualization

Denodo Community: Documents, Videos, Tutorials, and more.

Attend Architect-to-Architect Series

Performance

Scalability

Move forward with one of our Data Virtualization experts

Phone: (+1) 877-556-2531 (NA)

Phone: (+44) (0)20 7869 8053 (EMEA)

Email: info@denodo.com | www.denodo.com

Data Discovery and Governance

Security

Five In-depth Technology and Architecture Sessions on Data Virtualization

Thank You!

Next SessionSession 4

Denodo Platform: Data Discoveryand Governance