oracle’s big data solutions
DESCRIPTION
Oracle’s Big Data solutions. Jean-Philippe Breysse Oracle Suisse. - PowerPoint PPT PresentationTRANSCRIPT
<Insert Picture Here>
Oracle’s Big Data solutionsJean-Philippe BreysseOracle Suisse
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remain at the sole discretion of Oracle.
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 134
USE CASE 3: LOGS ANALYSIS OF SERVERS
Short Description :– Daily logs analysis
Issues:– Find correlations on what drives to failures– Log files stored as flat files
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 135
Oracle Technology mapped to Analytics Landscape
Acquire AnalyzeOrganizeData Decide
Stru
ctur
edS
emi-s
truct
ured
Uns
truct
ured
Master & Reference
Transactions
Machine Generated
Text, Image, Video, Audio
Oracle 12g
Files
Oracle NoSQL
Oracle Hadoop HDFS
Oracle Data Integrator
Oracle Hadoop
MapReduce
Oracle 12g
Oracle Essbas
e
Oracle R Enterprise & Oracle Data
Mining
Oracle BI Enterprise
Oracle Real Time Decisions
Oracle Endeca Information Discovery
Oracle Times Ten
Endeca MDEX
Oracle Golden Gate
Agenda
• Big Data• Solution Spectrum• Inside the Big Data Appliance• Big Data Applications Software• Big Data Analytics• Conclusions
<Insert Picture Here>
Big DataWhy Everyone Should Care
Tapping into Diverse Data Sets
Transactions
Information Architectures
Today:Decisions based on database data
Big Data:Decisions based on all your data
Video and Images
Machine-Generated DataSocial Data
Documents
9
A bit of history ...
: Developed initially by Doug Cutting (Nutch - Opensource websearch engine) and Yahoo -> inspired by Google’s papers on MapReduce and GFS (2003-2004) resulted in Apache Hadoop (2006)Amazon Dynamo (2007): distributed systems technologiesCassandra: was developed at Facebook (2008) to power their Inbox Search feature (columnar oriented distributed DB) based initially on Dynamo and Bigtable (built by Google)Voldemort: is a distributed data store that is designed as a key-value store used by LinkedIn for high-scalability storage (NoSql key value)Cloudera: . It contributes to Hadoop and related Apache projects and provides a commercial distribution of Hadoop
10
So What is Big Data Anyway?It’s a matter of perspective. Big Data is both:
• LARGE AND VARIABLE DATASETS that are difficult for traditional database tools to easily manage – including datasets that once seemed not important or too problematic to deal with. Big Data datasets include:
− Extremely large files of unstructured or semi-structured data − Large and highly distributed datasets that are otherwise difficult
to manage as a single unit of information
• NEW SET OF TECHNOLOGIES that can economically capture, store, manage, and extract value from Big Data datasets – thus facilitating better, more informed business decisions
Structured Data vs. Unstructured DataRelational databases work best with structured data – data which has underlying structure (schema) and size that easily fits the specific confines of database columns and rows. Unstructured data is highly variable, lacks fixed structure, and is often too large to easily handle by RDBMS systems.
Source: IDC Digital Universe Study, Extracting Value from Chaos, June 2011 (sponsored by EMC)
<Insert Picture Here>
Drive Value from Big DataBuilding a Big Data Platform
Divided Solution Spectrum
Acquire AnalyzeOrganize
MapReduceSolutions
DBMS (DW)
DBMS (OLTP)
Advanced Analytics
DistributedFile Systems
Transaction (Key-Value)
Stores
ETL
NoSQL Flexible
SpecializedDeveloper
Centric
SQL TrustedSecure
Administered
Schema-less
Unstructured
DataVariety
Schema
Hadoop to Oracle – Bridging the Gap
Acquire AnalyzeOrganize
HadoopMapReduce
HDFS
Cassandra
RDBMS(OLTP)
RDBMS(DW)
Advanced Analytics
ETL
Oracle Loader for Hadoop
Schema-less
Unstructured
DataVariety
Schema
Oracle Integrated Software Solution
Acquire AnalyzeOrganize
Oracle (DW)
Oracle (OLTP)
Schema-less
Unstructured
DataVariety
Schema
HadoopHDFS
Oracle NoSQL DB
OracleAnalytics:
Data Mining
RSpatialGraph
mapreduce
OBI EE
OracleData Integrator
Oracle Loader for Hadoop
<Insert Picture Here>
Inside the Big Data ApplianceOverview
16 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 8
Oracle Engineered Solutions
Acquire AnalyzeOrganize
OracleDatabase
(DW)
OracleDatabase
(OLTP)
In-DBAnalytics
“R”Mining
TextGraphSpatial
OracleBI EE
Oracle NoSQL DB
HDFS Hadoop
OracleData Integrator
Oracle Loader for Hadoop
DataVariety
InformationDensity
Unstructured
Schema
Big Data Appliance• Hadoop• NoSQL Database• Oracle Loader for hadoop• Oracle Data Integrator
Oracle Exadata• OLTP & DW• Data Mining & Oracle R• Semantics• Spatial
Exalytics• Speed of
ThoughtAnalytics
Big Data ApplianceUsage Model
InfiniBand
Oracle Big Data Appliance
Oracle Exadata
Acquire Organize Analyze & VisualizeStream
Oracle Exalytics
InfiniBand
•18 Sun X4270 M2 Servers– 48 GB memory per node = 864 GB memory – 12 Intel cores per node = 216 cores – 24 TB storage per node = 432 TB storage
•40 Gb p/sec InfiniBand•10 Gb p/sec Ethernet
Oracle Big Data Appliance Hardware
Scale Out to Infinity
Scale out by connecting racksto each other using Infiniband
• Expand up to eight racks without additional switches
• Scale beyond eight racks by adding an additional switch
• Oracle Enterprise Linux 5.6• Oracle Hotspot Java VM• Cloudera’s Distribution
including Apache Hadoop• Cloudera Manager• Open Source Distribution of R• Oracle NoSQL Database
Community Edition
Oracle Big Data Appliance Software
<Insert Picture Here>
Big Data Application SoftwareAcquire New Information
Key-Value Store Workloads
• Large dynamic schema based data repositories• Data capture
• Web applications (click-through capture)• Online retail• Sensor/statistics/network capture (factory automation for example)• Backup services for mobile devices
• Data services• Scalable authentication• Real-time communication (MMS, SMS, routing)• Personalization• Social Networks
Oracle NoSQL DB A distributed, scalable key-value database
• Simple Data Model• Key-value pair with major+sub-key paradigm• Read/insert/update/delete operations
• Scalability• Dynamic data partitioning and distribution• Optimized data access via intelligent driver
• High availability• One or more replicas• Disaster recovery through location of replicas• Resilient to partition master failures• No single point of failure
• Transparent load balancing• Reads from master or replicas• Driver is network topology & latency aware
• Elastic (Planned for Release 2)• Online addition/removal of Storage Nodes • Automatic data redistribution
Storage NodesData Center A
Storage NodesData Center B
NoSQLDB Driver
Application
NoSQLDB Driver
Application
<Insert Picture Here>
Big Data Application SoftwareOrganizing Data for Analysis
30 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Oracle Loader for Hadoop Features
• Load data into a partitioned or non-partitioned table– Single level, composite or interval partitioned table– Support for scalar datatypes of Oracle Database– Load into Oracle Database 11g Release 2
• Runs as a Hadoop job and supports standard options
• Pre-partitions and sorts data on Hadoop
• Online and offline load modes
31 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Oracle Loader for Hadoop
SHUFFLE/SORT
SHUFFLE/SORT
MAP
MAP
MAP
MAPSHUFFLE
/SORT
REDUCE
REDUCE
SHUFFLE/SORT
SHUFFLE/SORT
REDUCE
REDUCE
REDUCE
INPUT2
INPUT1
MAP
MAP
MAP
MAP
MAP
REDUCE
REDUCE
REDUCE
MAP
MAP
MAP
MAP
MAP
MAP
REDUCE
REDUCE
MAP
MAP
MAP
MAP
MAP
REDUCE
REDUCE
REDUCE
ORACLE LOADER FOR HADOOP
32 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Oracle Loader for Hadoop: Online Option
SHUFFLE/SORT
SHUFFLE/SORT
REDUCE
REDUCE
REDUCE
MAP
MAP
MAP
MAP
MAP
MAP
REDUCE
REDUCE
ORACLE LOADER FOR HADOOP Connect to the database from reducer nodes, load into database partitions in parallel
Read target table metadata from the database
Perform partitioning, sorting, and data conversion
33 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Oracle Loader for Hadoop: Offline Option
SHUFFLE/SORT
SHUFFLE/SORT
REDUCE
REDUCE
REDUCE
MAP
MAP
MAP
MAP
MAP
MAP
REDUCE
REDUCE
ORACLE LOADER FOR HADOOPRead target table metadata from the database
Perform partitioning, sorting, and data conversion
Write from reducer nodes to Oracle Data Pump files
Import into the database in parallel using external table mechanism
35 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Selection Output Option for Use Case
Oracle Loader for Hadoop Output Option Use Case Characteristics
Online load with JDBC The simplest use case for non partitioned tables
Online load with Direct Path Fast online load for partitioned tables
Offline load with datapump files Fastest load method for external tables
On Oracle Big Data ApplianceDirect HDFS
Leave data on HDFSParallel access from databaseImport into database when needed
36 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Automate Usage of Oracle Loader for Hadoop
• ODI has knowledge modules to – Generate data transformation code to run on Hive/Hadoop– Invoke Oracle Loader for Hadoop
• Use the drag-and-drop interface in ODI to– Include invocation of Oracle Loader for Hadoop in any ODI
packaged flow
Oracle Data Integrator (ODI)
37 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
<Insert Picture Here>
Big Data AnalyticsReal Time Analytics Platform
R Statistical Programming Language
Open source language and environment
Used for statistical computing and graphics
Strength in easily producing publication-quality plots
Highly extensible with open source community R packages
<Insert Picture Here>
Drive Value from Big DataConclusions
Big Data ApplianceBig Data for the Enterprise
• Optimized and Complete• Everything you need to store and integrate
your lower information density data• Integrated with Oracle Exadata
• Analyze all your data• Easy to Deploy
• Risk Free, Quick Installation and Setup• Single Vendor Support
• Full Oracle support for the entire system and software set
DECIDE
Oracle Analytic
Applications
Oracle Integrated Solution Stack for Big Data
ACQUIRE
Oracle NoSQL
Database
HDFS
Enterprise Applications
ORGANIZE
Hadoop(MapReduce)
Oracle Loader
for Hadoop
Oracle DataIntegrator
ANALYZE
In-D
atab
ase
Ana
lytic
s
DataWarehouse
Oracle: Big Data for the Enterprise
• The most comprehensive solution• Includes everything needed to acquire, organize and
analyze all your data• Optimized for Extreme Analytics
• Deepest analytics portfolio with access to all data• Engineered to Work Together
• Eliminate deployment risk and support risk• Enterprise Ready
• Deliver extreme performance and scalability
Questions