biwa2015-bringingoraclebigdatasqltoobieeandodi-150129143259-conversion-gate01.pdf

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

Bringing Oracle Big Data SQLto OBIEE 11g and ODI12c Mark Rittman, CTO, Rittman Mead BIWA Summit 2015, January 2015, San Francisco





About the Speaker

Mark Rittman, Co-Founder of Rittman Mead Oracle ACE Director, specialising in Oracle BI&DW 14 Years Experience with Oracle Technology Regular columnist for Oracle Magazine Author of two Oracle Press Oracle BI books Oracle Business Intelligence Developers Guide Oracle Exalytics Revealed Writer for Rittman Mead Blog :http://www.rittmanmead.com/blog

Email : [email protected] Twitter : @markrittman





About Rittman Mead

Oracle BI and DW Gold partner Winner of five UKOUG Partner of the Year awards in 2013 - including BI World leading specialist partner for technical excellence, solutions delivery and innovation in Oracle BI

Approximately 80 consultants worldwide All expert in Oracle BI and DW Offices in US (Atlanta), Europe, Australia and India Skills in broad range of supporting Oracle tools: OBIEE, OBIA ODIEE Essbase, Oracle OLAP GoldenGate Endeca



Introducing Hadoop

A new approach to data processing and data storage Rather than a small number of large, powerful servers, it spreads processing overlarge numbers of small, cheap, redundant servers

Spreads the data youre processing over lots of distributed nodes

Has scheduling/workload process that sends parts of a job to each of the nodes- a bit like Oracle Parallel Execution

And does the processing where the data sits - a bit like Exadata storage servers

Shared-nothing architecture Low-cost and highly horizontal scalable

Job Tracker

Task Tracker Task Tracker Task Tracker Task Tracker

Data Node Data Node Task Tracker Task Tracker



Apache Hive : SQL Access + Table Metadata Over HDFS

Apache Hive provides a SQL layer over Hadoop, once we understand the structure (schema) of the data were working with

Exposes HDFS and other Hadoop data as tables and columns Provides a simple SQL dialect for queries called HiveQL SQL queries are turned into MapReduce jobs under-the-covers JDBC and ODBC drivers provideaccess to BI and ETL tools

Hive metastore (data dictionary)leveraged by many other Hadoop tools Apache Pig Cloudera Impala etc

SELECT a, sum(b) FROM myTable WHERE a



Hive SerDes & Storage Handlers

Plug-in technologies that extend Hive to handle new data formats and semi-structured sources Typically distributed as JAR files, hosted on sites such as GitHub Can be used to parse log files, access data in NoSQL databases, Amazon S3 etc

CREATE EXTERNAL TABLE apachelog ( host STRING, identity STRING, user STRING, time STRING, request STRING, status STRING, size STRING, referer STRING, agent STRING) ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe' WITH SERDEPROPERTIES ( "input.regex" = "([^ ]*) ([^ ]*) ([^ ]*) (-|\\[[^\\]]*\\]) ([^ \"]*|\"[^\"]*\") (-|[0-9]*) (-|[0-9]*)(?: ([^ \"]*|\"[^\"]*\") ([^ \"]*|\"[^\"]*\"))?", "output.format.string" = "%1$s %2$s %3$s %4$s %5$s %6$s %7$s %8$s %9$s" ) STORED AS TEXTFILE LOCATION '/user/root/logs';

CREATE TABLE tweet_data( interactionId string, username string, content string, author_followers int) ROW FORMAT SERDE 'com.mongodb.hadoop.hive.BSONSerDe' STORED BY 'com.mongodb.hadoop.hive.MongoStorageHandler' WITH SERDEPROPERTIES ( 'mongo.columns.mapping'='{"interactionId":"interactionId", "username":"interaction.interaction.author.username", "content":\"interaction.interaction.content", "author_followers_count":"interaction.twitter.user.followers_count"}' ) TBLPROPERTIES ( 'mongo.uri'='mongodb://cdh51-node1:27017/datasiftmongodb.rm_tweets' )



Oracles Big Data Products

Oracle Big Data Appliance Optimized hardware for Hadoop processing Cloudera Distribution incl. Hadoop Oracle Big Data Connectors, ODI etc

Oracle Big Data Connectors Oracle NoSQL Database Oracle Data Integrator Oracle R Distribution OBIEE, BI Publisher and Endeca Info Discovery



Oracle Big Data Connectors

Oracle-licensed utilities to connect Hadoop to Oracle RBDMS Bulk-extract data from Hadoop to Oracle, or expose HDFS / Hive data as external tables Run R analysis and processing on Hadoop

Leverage Hadoop compute resources to offload ETL and other work from Oracle RBDMS Enable Oracle SQL to access and load Hadoop data



Oracle Business Analytics and Big Data Sources

OBIEE 11g can also make use of big data sources OBIEE 11.1.1.7+ supports Hive/Hadoop as a data source, via specific Hive ODBC driversand Apache Hive Physical Layer database type

But practically, it comes with limitations Current 11.1.1.7 version of OBIEE only ships with HiveServer1 ODBC drivers HiveQL is a limited subset of ISO/Oracle SQL and Hive access is really slow



Oracle Big Data SQL

Part of Oracle Big Data 4.0 (BDA-only) Also requires Oracle Database 12c, Oracle Exadata Database Machine

Extends Oracle Data Dictionary to cover Hive Extends Oracle SQL and SmartScan to Hadoop Extends Oracle Security Model over Hadoop Fine-grained access control Data redaction, data masking

Uses fast c-based readers where possible(vs. Hive MapReduce generation) Map Hadoop parallelism to Oracle PQ Big Data SQL engine works on top of YARN Like Spark, Tez, MR2

Exadata Storage Servers

HadoopCluster

Exadata DatabaseServer

Oracle Big Data SQL

SQL Queries

SmartScan SmartScan



View Hive Table Metadata in the Oracle Data Dictionary

Oracle Database 12c 12.1.0.2.0 with Big Data SQL option can view Hive table metadata Linked by Exadata configuration steps to one or more BDA clusters

DBA_HIVE_TABLES and USER_HIVE_TABLES exposes Hive metadata Oracle SQL*Developer 4.0.3, with Cloudera Hive drivers, can connect to Hive metastore

SQL> col database_name for a30 SQL> col table_name for a30 SQL> select database_name, table_name 2 from dba_hive_tables; DATABASE_NAME TABLE_NAME ------------------------------ ------------------------------ default access_per_post default access_per_post_categories default access_per_post_full default apachelog default categories default countries default cust default hive_raw_apache_access_log



Big Data SQL Server Dataflow

Read data from HDFS Data Node Direct-path reads C-based readers when possible Use native Hadoop classes otherwise

Translate bytes to Oracle

Apply SmartScan to Oracle bytes Apply filters Project columns Parse JSON/XML Score models Disks%

Data$Node$

Big$Data$SQL$Server$

External$Table$Services$

Smart$Scan$

RecordReader%

SerDe%

1011

0010

%10

1100

10%

1011

0010

%

1%

2%

3%

1

2

3



Hive Access through Oracle External Tables + Hive Driver

Big Data SQL accesses Hive tables through external table mechanism ORACLE_HIVE external table type imports Hive metastore metadata ORACLE_HDFS requires metadata to be specified

Access parameters cluster and tablename specify Hive table source and BDA clusterCREATE TABLE access_per_post_categories( hostname varchar2(100), request_date varchar2(100), post_id varchar2(10), title varchar2(200), author varchar2(100), category varchar2(100), ip_integer number) organization external (type oracle_hive default directory default_dir access parameters(com.oracle.bigdata.tablename=default.access_per_post_categories));



Use Rich Oracle SQL Dialect over Hadoop (Hive) Data

Ranking Functions rank, dense_rank, cume_dist, percent_rank, ntile

Window Aggregate Functions Avg, sum, min, max, count, variance, first_value, last_value

LAG/LEAD Functions Reporting Aggregate Functions Sum, Avg, ratio_to_report

Statistical Aggregates Correlation, linear regression family, covariance

Linear Regression Fitting of ordinary-least-squares regression line to set of number pairs

Descriptive Statistics Correlations Pearsons correlation coefficients

Crosstabs Chi squared, phi coefficinet

Hypothesis Testing Student t-test, Bionomal test

Distribution Anderson-Darling test - etc.



Leverages Hive Metastore for Hadoop Java Access Classes

As with other next-gen SQL access layers, uses common Hive metastore table metadata Provides route to underlying Hadoop data for Oracle Big Data SQL c-based SmartScan



Extending SmartScan, and Oracle SQL, Across All Data

Brings query-offloading features of Exadatato Oracle Big Data Appliance

Query across both Oracle and Hadoop sources Intelligent query optimisation applies SmartScanclose to ALL data

Use same SQL dialect across both sources Apply same security rules, policies, user access rights across both sources



Example : Using Big Data SQL to Add Dimensions to Hive Data

We want to add country and post details to a Hive table containing page accesses Post and Country details are stored in Oracle RBDMS reference tables

Hive Weblog Activity tableOracle Dimension lookup tables

Combined output in report form



Create ORACLE_HIVE External Table over Hive Table

Use the ORACLE_HIVE access driver type to create Oracle external table over Hive table ACCESS_PER_POST_EXTTAB now appears in Oracle data dictionary



Import Oracle Tables, Create RPD joining Tables Together

No need to use Hive ODBC drivers - Oracle OCI connection instead No issue around HiveServer1 vs HiveServer2; also Big Data SQL handles authenticationwith Hadoop cluster in background, Kerberos etc

Transparent to OBIEE - all appear as Oracle tables Join across schemas if required



Create Physical Data Model from Imported Table Metadata

Join ORACLE_HIVE external table containing log data, to reference tables from Oracle DB



Create Business Model and Presentation Layers

Map incoming physical tables into a star schema Add aggregation method for fact measures Add logical keys for logical dimension tables Remove columns from fact table that arent measures



Create Initial Analyses Against Combined Dataset

Create analyses usingfull SQL features

Access to Oracle RDBMSAdvanced Analytics functionsthrough EVALUATE,EVALUATE_AGGR etc

Big Data SQL SmartScan featureprovides fast, ad-hoc accessto Hive data, avoiding MapReduce



Oracle / Hive Query Federation at the RDBMS Level

Oracle Big Data SQL feature (not BI Server) takes care of query federation SQL required for fact table (web log activity) access sent to Big Data SQL agent on BDA Only columns (projection) and rows (filtering) required to answer query sent back to Exadata Storage Indexes used on both Exadata Storage Servers and BDA nodes to skip block reads for irrelevant data

HDFS caching used to speed-upaccess to commonly-usedHDFS data



Access to Full Set of Oracle Join Types

No longer restricted to HiveQL equi-joins - Big Data SQL supports all Oracle join operators Use to join Hive data (using View over external table) to a IP range country lookup tableusing BETWEEN join operator



Add In Time Dimension Table

Enables time-series reporting; pre-req for forecasting (linear regression-type queries) Map to Date field in view over ORACLE_HIVE table Convert incoming Hive STRING field to Oracle DATE for better time-series manipulation



Now Enables Time-Series Reporting Incl. Country Lookups



Oracle Data Integrator 12c

Oracles data integration tool for loading, transforming and integrating enterprise data Successor to Oracle Warehouse Builder, part of wider Oracle DI platform Connectivity to most RBDMS, file and application sources



Part of the Wider Oracle Data Integration Platform

Oracle Data Integrator for large-scale data integration across heterogenous sources and targets

Oracle GoldenGate for heterogeneous data replication and changed data capture Oracle Enterprise Data Quality for data profiling and cleansing Oracle Data Services Integratorfor SOA message-baseddata federation



Oracle Data Integrator on Hadoop

ODI provides an excellent framework for running Hadoop ETL jobs ELT approach pushes transformations down to Hadoop - leveraging power of cluster

Hive, HBase, Sqoop and OLH/ODCH KMs provide native Hadoop loading / transformation Whilst still preserving RDBMS push-down Extensible to cover Pig, Spark etc

Process orchestration Data quality / error handling Metadata and model-driven



Combining Oracle and Hadoop (Hive) Data in Mappings

Example scenario : log data in Hadoop needs to be enriched with customer data in Oracle Hadoop (Hive) contains log activity and customer etc IDs Reference / customer data held in Oracle RBDMS

How do we create a mapping that joins both datasets?

movieapp_log_odistage.custid = CUSTOMER.CUSTID



Options for Importing Oracle / RDBMS Data into Hadoop

Could export RBDMS data to file, and load using IKM File to Hive

Oracle Big Data Connectors only export to Oracle, not import to Hadoop

One option is to use Apache Sqoop, and new IKM SQL to Hive-HBase-File knowledge module Hadoop-native, automatically runs in parallel Uses native JDBC drivers, or OraOop (for example)

Bi-directional in-and-out of Hadoop to RDBMS Join performed in Hive, using HiveQL With HiveQL limitations (only equi-joins)

movieapp_log_odistage.custid = customer.custid

Sqoop extract



New Option - Using Oracle Big Data SQL

Oracle Big Data SQL provides ability for Exadata to reference Hive tables Use feature to create join in Oracle, bringing across Hive data through ORACLE_HIVE table



Oracle Big Data SQL and Data Integration

Gives us the ability to easily bring in Hadoop (Hive) data into Oracle-based mappings Allows us to create Hive-based mappings that use Oracle SQL for transforms, joins Faster access to Hive data for real-time ETL scenarios Through Hive, bring NoSQL and semi-structured data access to Oracle ETL projects For our scenario - join weblog + customer data in Oracle RDBMS, no need to stage in Hive



Using Big Data SQL in an ODI12c Mapping

By default, Hive table has to be exposed as an ORACLE_HIVE external table in Oracle first Then register that Oracle external table in ODI repository + model

External table creation in Oracle

Logical Mapping using just Oracle tables

1

2

Register in ODI Model

3



Custom KM : LKM Hive to Oracle (Big Data SQL)

ODI12c Big Data SQL example on BigDataLite VM uses a custom KM for Big Data SQL LKM Hive to Oracle (Big Data SQL) - KM code downloadable from java.net Allows Hive+Oracle joins by auto-creating ORACLE_HIVE extttab definition to enable Big Data SQL Hive table access



ODI12c Mapping Creates Temp Exttab, Joins to Oracle

1

2

Register in ODI Model3

4

Hive table AP uses LKM Hive to Oracle (Big Data SQL)

IKM Oracle Insert

Big Data SQL Hive External Table created as temp object

Main integration SQL routines uses regular Oracle SQL join



Summary

Oracle Big Data SQL extends Exadata capabilities over Hadoop and Hive Makes Hive Metastore visible through Oracle Data Dictionary Register Hive tables as ORACLE_HIVE external tables and include in Oracle SQL queries Used with OBIEE, allows RPDs to be created across both Oracle + Hive data, withquery federation handled by Oracle RBDMS rather than BI Server

Enables use of Oracle Advanced Analytics functions over Hadoop data Useful for ODI as way of using full set of join operators on Hive data, and simplifyingthe addition of Hive data to Oracle mappings

For developers working with Exadata + BDA, useful addition to the data access toolkit





Thank You for Attending!

Thank you for attending this presentation, and more information can be found at http://www.rittmanmead.com

Contact us at [email protected] or [email protected] Look out for our book, Oracle Business Intelligence Developers Guide out now! Follow-us on Twitter (@rittmanmead) or Facebook (facebook.com/rittmanmead)



Bringing Oracle Big Data SQLto OBIEE 11g and ODI12c Mark Rittman, CTO, Rittman Mead BIWA Summit 2015, January 2015, San Francisco

biwa2015-bringingoraclebigdatasqltoobieeandodi-150129143259-conversion-gate01.pdf

Documents