rackspace analytical compute grid (acg)

April 12, 2023

Analytical Compute Grid (ACG)

Elastic “Big Data” Infrastructure

by Natasha Gajic

Big Data on Open Cloud

2RACKSPACE® HOSTING | WWW.RACKSPACE.COM

Rackspace’s EBI Environment

Current EnvironmentWindows and Linux

operating systemsOracle and Microsoft

databases solutionsMicrosoft and Oracle

replication technologySSISInformaticaDedicated serversRapid data set growth

“Big Data” ProblemCost of purchasing

additional licensesTime required to set up

new hardwareIncreased demand for DBA

resourcesSystem performanceSystem scalabilityCapacity

Analytical Compute Grid (ACG) Features

•Host ever growing set of data•Quick data collection and retrieval•Rapid scalability•Ease of maintenance•Provide standard data access API

Analytical Compute Grid (ACG) Features

•Ability to provide variety of storage types:

• Columnar

• Relational

• HDFS

•Enable users to select optimal storage type for information collected

•Leverage Rackspace® Private Cloud powered by OpenStack® and open source technology

Analytical Compute Grid (ACG) Quality Attributes

ACG on Rackspace® Private Cloud powered by OpenStack®

High Level Architecture

Database Engine Selection

Columnar Cassandra

Relational PostgreSQL

HDFS Hadoop

Controller

Indexing Structure

What is ACG Indexing Structure?

• System entry point

• Set of pointers ultimately addressing database entities

Indexing Structure

What is ACG Indexing Structure?

• System entry point• Set of pointers ultimately addressing database entities

Where is Indexing Structure Located?

• It is a part of ACG so it resides on Open Cloud• ACG Controller manages Indexing Structure

Indexing Structure

What ACG Indexing Structure Enables?

• Splitting of large data sets across many instances• Query parallelization• Controlled data store size• Optimal data store configuration• Uniform access to data residing in various storage types• System scalability as it expands horizontally and vertically to address ever growing data set

Quality Attributes

Quality Attributes - Performance

Rackspace® Private Cloud powered by OpenStack®

Creates ACG node in 30 secondsCreates ACG nodes concurrentlyRe-size ACG nodes adding CPUs

Quality Attributes - Performance

Creates ACG node in 30 secondsCreates ACG nodes concurrentlyRe-size ACG nodes adding CPUs

Indexing structure and controlled data set size allow for: Quick data distribution Query parallelization

Quality Attributes – Availability

Rapidly replace failed ACG nodes

Quality Attributes – Availability

Rapidly replace failed ACG nodes

Deploys data store native availability mechanisms (replication, data distribution…)

Quality Attributes – Maintainability

Adding ACG nodes expands: Storage capacity CPU power MemoryNo DBA or system administrators activity required

Quality Attributes – Maintainability

Adding ACG nodes expands: Storage capacity CPU power RAM No DBA or system administrators activity required

Controlled data set size enables: Optimal and stable data store configuration Reducing demand for managing data store objects Stable query execution plans

Quality Attributes – Flexibility

Variety of storage types:Columnar – Cassandra : time series dataRelational – PostgreSQL : relational dataHDFS – Hadoop : un-structured data

Ability to select optimal storage type for individual use case

Quality Attributes – Usability

Standard interfaces: SQL language JDBC API ODBC

ACG Management Console

ACG Monitoring Console Loader utility implementing: Bulk Loader Insert Loader

Current State

ACG Controller

•ACG Manager•Rule Engine•Node Manager•ACG Management Console•ACG Monitoring

Columnar Implementation

•Data Store Controller•JDBC extended to work with supercolumn•Loader integrated with Informatica

Relational Implementation

•Data Store Controller•JDBC driver extended with distributed query rewrite•Loader integrated with Informatica•ODBC (In Progress)

HDFS Implementation

•Will start soon

Rackspace Use Case

• Subject:

• Complex availability calculation sourcing 3 months of monitoring data and creating 1 billion records in initial calculation

Rackspace Use Case

• Environment 1

• Data Warehouse Microsoft SQL server database• SSIS data loading• SQL server with 24 CPUs and 250GB RAM was dedicated to the initial calculation

• SQL server stored procedure performed the calculation

• Source and result are stored in traditional data warehouse structure

Rackspace Use Case

• Environment 2

• ACG running two Cassandra clusters 4 nodes each

• Informatica with Cassandra bulk loader• Each ACG node has 2CPUs and 8GB RAM• Java program running on instance with 4CPUs and 8GB RAM

• Source and result are stored in columnar structure suitable for time series data

Rackspace Use Case

• Calculation Duration

•Microsoft SQL Server lasted 5 days•ACG calculation completed in 3.5 hours

• Storage Size• Microsoft SQL server 500GB •ACG 20 GB

• Complexity of the calculation•Columnar data store is optimal for time series data. Sourcing from columnar data store resulted in relatively simple Java calculation process comparing to SQL server stored procedure

Rackspace Use Case - Result

• Selecting optimal data store for use case resulted in:

• Substantial performance improvement• Reduced storage demand•Simplified processes•Ability to process terabytes of data per day close to real-time and on-demand

•Improved trending and reporting:• enhances support capabilities

• improved Rackspace customer experience

• Significant cost reduction

Rackspace Use Case - Conclusion

RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218

US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM

RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM

ACG UI

rackspace analytical compute grid (acg)

open cloud acg controller

acg indexing structure

data set rackspace hosting

cpus rackspace hosting

openstackapi rackspace

openstack7 rackspace

openstackadding acg

openstackcreates acg

Technology

acg european capital tour: introducing acg

control - wordpress.com · 2018-07-30 · inventory,...

rackspace: driving openstack forward

automation for everyone - events.redhat.com · red hat...

young rackspace

cloudu certificate- rackspace

rackspace it summit 2012

rackspace-#45658-v15- master rackspace master services

rackspace: unlock your cloud - rightscale compute 2013

cisco support services for cloudcenter service...

cloudharmony - centurylink cloud · compute instances from...

rackspace unlocked.io q3'13

rackspace managed cloud for elastic path · companies trust...

rackspace slides

superintendent’s manual - rackspace technology

nvme over fabrics- - open compute project · switch compute...

rackspace private cloud installation...

rackspace howto

xero rackspace presentation

pair programming at rackspace