oracle exadata x2-8: a critical review
DESCRIPTION
Oracle ACE, Mike Ault, gives a high level overview of what the Oracle Exadata is and its best-fit applications as well as offer alternatives to it..TRANSCRIPT
The World’s Fastest Storage®Texas Memory Systems, Inc.
Oracle Exadata X2-8: A Critical Review
Mike AultOracle Guru
Texas Memory Systems
The World’s Fastest Storage®Texas Memory Systems, Inc.
Michael R. AultOracle Guru
- Nuclear Navy 6 years- Nuclear Chemist/Programmer 10 years - Kennedy Western University Graduate- Bachelors Degree Computer Science- Certified in all Oracle Versions Since 6- Oracle DBA, author, since 1990- Oracle Ace
The World’s Fastest Storage®Texas Memory Systems, Inc.
Books by Michael R. Ault
The World’s Fastest Storage®Texas Memory Systems, Inc.
Statspackanalyzer.comFree Statspack/AWR Analysis
Sponsored by Texas Memory Systems
- Looks for IO bottlenecks and other configuration issues.
- Straightforward tuning advice
The World’s Fastest Storage®Texas Memory Systems, Inc.
What Is Exadata X2-8?
• Is it software?• Is it hardware?• Is it the Borg?
The World’s Fastest Storage®Texas Memory Systems, Inc.
Exadata as Borg?
The World’s Fastest Storage®Texas Memory Systems, Inc.
Not Really (at least not yet…)
• Exadata is a combination of hardware and software
• Exadata takes state of the art disk, computer and flash technology and combines it with specially designed software from Oracle.
• Exadata hardware can only run Oracle software
The World’s Fastest Storage®Texas Memory Systems, Inc.
Exadata X2-8 Hardware
(Looks to be the SUN X4800)
The World’s Fastest Storage®Texas Memory Systems, Inc.
Traditional Setup
The World’s Fastest Storage®Texas Memory Systems, Inc.
Exadata Setup
Each Exadata Cell has 4-96 GB
flash cards
The World’s Fastest Storage®Texas Memory Systems, Inc.
Exadata X2-8 Storage Hardware• Sun x4270 M2 servers that contain dual six-core Xeon L5640 processors
running at 2.26 GHz, with 24GB of memory• 4-96 MB flash cards for Smart flash cache used to accelerate disk reads• Disks are either high performance low capacity 15K or low performance
high capacity 7.5K• Lose 66% or more of raw space for redundancy options or formatting
losses– High Perf: 7.2 down to 2 TB per cell– Low Perf: 24 down to 7 TB per cell
• You pay license on a per disk basis – $10,000.00/disk, $120,000.00 per cell
• Get IOPS based on large number of disks (168 to get 50K IOPS)– 200 IOPS/DISK*168 DISK=33,600 IOPS so this is doubtful
• Promises of 1,000,000 IOPS from FLASH (read-only) • Full X2-8 hardware cost: $1,500,000.00 (doesn’t include software!)
All prices/data taken from:
Oracle Exadata and Exalogic Pricelist, July 1, 2011
Oracle Technology Global Pricelist, July 1, 2011, Software Investment Guide
Exadata Smart Flash Cache and the Sun Oracle Database Machine, Oracle, Oct 2009
The World’s Fastest Storage®Texas Memory Systems, Inc.
Exadata X2-8 Specific Software
The World’s Fastest Storage®Texas Memory Systems, Inc.
Something to Note:
The World’s Fastest Storage®Texas Memory Systems, Inc.
Exadata X2-8 Oracle Software• License is per CPU
– Approximately $51,500.00 per CPU for all Oracle software
– 128 CPU * $51,500.00 = $6,592,000.00– Add the Cell license cost of $1,680,000.00– Total license cost: $8,272,000.00
• Total Hardware cost: $1,500,000.00• Total cost (plus shipping, handling and
installation, support): $9,772,000.00All prices taken from:
Oracle Exadata and Exalogic Pricelist, July 1, 2011
Oracle Technology Global Pricelist, July 1, 2011, Software Investment Guide
The World’s Fastest Storage®Texas Memory Systems, Inc.
Exadata Specific Software Features:Smartscan
• Goal reduce the data volume into the SGA as will only match the result set
• Less load on IB and RAC interconnects+CPU• Predicate Filtering processes WHERE in storage• Column Projections only returns columns = sql• Bloom filter Joins passes matching values to storage
cells to match only requested values• Storage Indexes High/Low Values @1MB disk• Data Mining Scoring functions run on storage
The World’s Fastest Storage®Texas Memory Systems, Inc.
Smartscan• First: Traditional Processing (non-indexed select):
– The client issues a SELECT statement with a predicate to filter and return only rows of interest.
– The database kernel maps this request to the file and extents containing the table being scanned.
– The database kernel issues the I/O to read the blocks. – All the blocks of the table being queried are read into
memory. Then SQL processing is done against the raw blocks searching for the rows that satisfy the predicate.
– Lastly the rows are returned to the client.
From: Oracle, A Technical Overview of the Oracle Exadata Database Machine and Exadata Storage Server, April 2011
The World’s Fastest Storage®Texas Memory Systems, Inc.
Smartscan• Smartscan:
– The client issues a SELECT statement with a predicate to filter and return only rows of interest.
– The database kernel determines that Exadata storage is available and constructs an iDB command representing the SQL command issued and sends it the Exadata storage.
– The CELLSRV component of the Exadata software scans the data blocks to identify those rows and columns that satisfy the SQL issued.
– Only the rows satisfying the predicate and the requested columns are read into memory.
– The database kernel consolidates the result sets from across the Exadata cells.
– Lastly, the rows are returned to the client.
From: Oracle, A Technical Overview of the Oracle Exadata Database Machine and Exadata Storage Server, April 2011
The World’s Fastest Storage®Texas Memory Systems, Inc.
SO…• Smartscan is good for full table scan issues where no indexes
are available• Have heard that Oracle suggests removing indexes, but how
does smartscan resolve FK-PK issues?• In complex OLTP or DSS/DWH queries smartscan may not help• Depends on min/max storage indexes which can have issues
with changing data• May not be efficient on partitioned tables when a partitioned-
wise join can’t be used• In frequently updated tables smartscan may be bypassed, or,
blocks will be passed that are inconsistent, requiring the DB server to filter them and retrieve the proper block from the undo segments
From: Oracle: A Technical Overview of the Oracle Exadata Database Machine and Exadata Storage Server, April 2011Teradata: Exadata is Still Oracle, March 2011
The World’s Fastest Storage®Texas Memory Systems, Inc.
Exadata Specific Software Features:Smart Flash Cache
• Seems to be lots of confusion between Flash cache in the server and flash cache at the DB level
• Flash cache is turned on at the server in Oracle Linux and Sun using initialization parameters
• Smart flash cache is automatic with the Exadata cell and requires no operator intervention
The World’s Fastest Storage®Texas Memory Systems, Inc.
Exadata Smart Flash Cache• Each cell comes with 384 GB of Exadata Smart Flash Cache.• In the Database Machine X2-8 and Full Rack X2-2 there is 5.3
TB of Flash • Provides a 10x improvement in response time for reads and
100x improvement in IOPS for reads over regular disk• Delivers a 10x blended performance average of read and write
operations.• Doesn’t use Least Recently Used (LRU) algorithm• The Storage Server Software keeps track of data access
patterns and caches to avoid polluting the cache. • Managed automatically and does not allow manual tuning. • Specific objects key to performance can be pinned in cache.• Only for EXADATA
From: An Oracle White Paper, October 2009,Exadata Smart Flash Cache and the Sun Oracle Database Machine
The World’s Fastest Storage®Texas Memory Systems, Inc.
Database Flash Cache• A victim cache, it stores clean (unmodified) blocks that have
been evicted from the SGA buffer cache • If a block has been modified, it is first written to disk, then it is
written to the Database Smart Flash Cache.• Blocks can later be returned to the SGA as required. • If a block cannot be found in the SGA buffer cache or the
Database Smart Flash Cache, it is retrieved from disk.• This is only done when DBWR has time to perform the
operations• Only works (in supported mode) on Oracle Linux and Oracle
Sun Solaris• Set up with two-initialization parameters:
– DB_FLASH_CACHE_FILE– DB_FLASH_CACHE_SIZE
FROM: An Oracle White Paper, September 2010, Oracle Database Smart Flash Cache
The World’s Fastest Storage®Texas Memory Systems, Inc.
SO…
• Exadata flash cache can benefit both OLTP and DSS/DWH
• Only for non-changing, frequently accessed data
• Most helpful for reads• Is basically a write-through cache using access
statistics to optimize storage of hot blocks• On external flash so not as fast as database
flash cache
The World’s Fastest Storage®Texas Memory Systems, Inc.
IORM(IO Resource manager)
• DBRM has been enhanced for use with Exadata. • DBRM lets the user define and manage intra and inter-database I/O
bandwidth. • Consumer groups are allocated a percent of the available I/O bandwidth
and the DBRM ensures these targets are delivered. • The database tags I/O with the database and consumer group. • The intra-database I/O allocations are defined and managed at the
database • The inter-database I/O allocations are defined within the software in the
Exadata cell and managed by the I/O Resource Manager (IORM). • The Exadata cell software ensures that inter-database I/O resources are
managed and properly allocated within, and between, databases. • Overall, DBRM ensures each database receives its specified amount of I/O
resources and user defined SLAs are met.• Note that IORM currently provides no management of objects cached in
FlashFROM:An Oracle White Paper, April 2011, A Technical Overview of the Oracle Exadata Database Machine andExadata Storage ServerMixed Consolidated Workloads and the Sun Oracle Database Machine By Robert Stackowiak February 6, 2010
The World’s Fastest Storage®Texas Memory Systems, Inc.
IORMDBRM are created at the DatabaseIORM are created at the CELLCreating IORM Plans on Oracle Exadata: Create a simple plan:
CellCLI> alter iormplan dbplan=((name=db1_prod, level=1, allocation=70),(name=db2_prod, level=1, allocation=30),(name=db1_dev, level=2, allocation=100),(name=db1_test, level=3, allocation=70),(name=other, level=3, allocation=40)),catplan=''
CellCLI> alter iormplan activeFrom: http://pauloferreiraportugal.blogspot.com/2011/04/creating-iorm-plans-on-oracle-exadata.html
The World’s Fastest Storage®Texas Memory Systems, Inc.
Other IO Features
• Utilizes creation of cell disks from whole physical disks
• Cell disks are then split into ASM DISKS with HOT and COLD ASM DISKS
• The Hot ASM DISKS (outer edge) are grouped together and used for hot data
• The Cold ASM DISKS are grouped together and used for cold data
The World’s Fastest Storage®Texas Memory Systems, Inc.
SO…
• IORM is only for Exadata at the Cell level• IORM doesn’t regulate or understand flash
cache operations• Can be used in concert with DBRM to control
resources
The World’s Fastest Storage®Texas Memory Systems, Inc.
Exadata Specific Software Features:Storage Indexes
• The Storage Server Software creates and maintains the Storage Index
• Tracks min and max values of columns for tables stored on that cell.
• When a query specifies a WHERE clause, but before any I/O, the Exadata software uses the Storage Index to determine if rows with the specified column value exist in the cell
• Many SQL Operations will run dramatically faster because large numbers of I/O operations are automatically replaced by a few lookups.
• Storage Indexes are created and maintained by the Exadata Storage Server Software.
• Indexes must be rebuilt every shutdown/startup operation• Can cause false positives, but not false negatives
The World’s Fastest Storage®Texas Memory Systems, Inc.
Exadata Specific Software Features:Hybrid Columnar Compression
• Combination of both row and columnar methods for storing data.
• Achieves the compression benefits of columnar storage, avoiding the performance issues of a pure columnar format.
• A logical construct called the compression unit is used to store a set of Exadata Hybrid Columnar-compressed rows.
• When data is loaded, column values are detached from the set of rows, ordered and grouped together and then compressed.
• After the column data for a set of rows has been compressed, it is fit into the compression unit.
• Not good for volatile data
From: An Oracle White Paper: December 2009, Exadata Hybrid Columnar Compression
The World’s Fastest Storage®Texas Memory Systems, Inc.
Exadata Specific Software Features:Hybrid Columnar Compression
• Data must be loaded using data warehouse bulk loading techniques.
• Exadata HCC tables can still be modified using conventional Data Manipulation Language (DML) operations, such as INSERT and UPDATE.
• Examples of bulk load operations commonly used in data warehouse environments are: – Insert statements with the APPEND hint – Parallel DML – Direct Path SQL*LDR – Create Table as Select (CTAS)
From: An Oracle White Paper: December 2009, Exadata Hybrid Columnar Compression
The World’s Fastest Storage®Texas Memory Systems, Inc.
Exadata Specific Software Features:HCC Warehouse Compression
Warehouse Compression Syntax CREATE TABLE emp ( emp_id NUMBER , first_name VARCHAR2(128) , last_name VARCHAR2(128) ) COMPRESS FOR QUERY [ LOW | HIGH ];
Low gives 6XHigh gives 10X
From: An Oracle White Paper: December 2009, Exadata Hybrid Columnar Compression
The World’s Fastest Storage®Texas Memory Systems, Inc.
Exadata Specific Software Features:HCC Archive Compression
Online Archival Compression Syntax CREATE TABLE emp ( emp_id NUMBER , first_name VARCHAR2(128) , last_name VARCHAR2(128) ) COMPRESS FOR ARCHIVE [ LOW | HIGH ];
Gives up to 15XFor infrequently accessed dataWill cause notable decrease in performance
From: An Oracle White Paper: December 2009, Exadata Hybrid Columnar Compression
The World’s Fastest Storage®Texas Memory Systems, Inc.
SO…
• HCC good for infrequently accessed data– Use OLTP compression for OLTP only 4X
• Have seen reports that the needed decompression/recompression due to changing data can be quite expensive performance wise
• Must do initial load with pre-built blocks
The World’s Fastest Storage®Texas Memory Systems, Inc.
Data Mining in Oracle Exadata
• Scoring is the process of applying a data mining model to data to generate predictions.
• Scoring processes require significant system resources.
• Vast amounts of data may be involved, and algorithmic processing may be very complex.
• With Oracle Data Mining, scoring can be off-loaded to intelligent Oracle Exadata Storage Servers
The World’s Fastest Storage®Texas Memory Systems, Inc.
Overall
From: Teradata, Exadata is Still Oracle, March 2011
The World’s Fastest Storage®Texas Memory Systems, Inc.
Overall
• Low– Simple star schemas– Simple joins– Fixed data
• High– Hybrid or snowflake schemas– Complex joins, summaries, etc.– Changing data
The World’s Fastest Storage®Texas Memory Systems, Inc.
Exadata Strengths
• One vendor• High bandwidth data path• Moves processing (in some cases) closer to
the data• Offers advanced compression for
archival/fixed data• Offers 5-20X acceleration for simple queries
against relatively calm data• Within the Exadata X2-2 family, easy upgrades
The World’s Fastest Storage®Texas Memory Systems, Inc.
Exadata Weaknesses
• Limited to one vendor for hardware and software (limited flexibility)
• Not good for rapidly changing data• Expensive• Complex hardware and software• Daisy cutter approach to upgrade from existing
system • Heavy metal approach to performance• For DP if you go Dataguard, must use Exadata
The World’s Fastest Storage®Texas Memory Systems, Inc.
Summary• For a new shop or project with no existing hardware
Exadata is a good option if you can afford it• For consolidation Exadata may be a good fit• If you intend on keeping your licenses where they are
and just getting an Exadata, it is really expensive• Uses “old” technology (disks)• May have issues with rapidly changing data (frequent
IUD) negating the new performance features• Can get similar or better performance improvements
without getting rid of existing technologies.
The World’s Fastest Storage®Texas Memory Systems, Inc.
Comparison of Exadata X2-8 to A RamSan710 Based Database System
The World’s Fastest Storage®Texas Memory Systems, Inc.
Main Storage-RamSan710
• 1-U 1-5TB SLC Flash • 400,000 IOPS• 5 GB/s bandwidth• 4 QDR Infiniband ports• 175 microsecond latency worst case (4k)• Base cost 5TB - $150,000.00• 12 - $1,800,000.00 (Raid 0) 30 TB
The World’s Fastest Storage®Texas Memory Systems, Inc.
How About Some CPUs?
• 2-SUN Fire X4800 (Same as Exadata)• Each with:
– 8-8 core 2.26 GHz 7560 CPUs– 1 TB memory– 8 PCIe modules
• Cost: $268,392.00
The World’s Fastest Storage®Texas Memory Systems, Inc.
Some Licenses(Same base Oracle licenses as Exadata)• 128 CPUS*$51,500.00=$6,592,000• No need for Cell licenses since no Cells!
The World’s Fastest Storage®Texas Memory Systems, Inc.
Misc(Support and connection)
• Switches• Rack• Cables, etc• $40,000.00
The World’s Fastest Storage®Texas Memory Systems, Inc.
Some Database Flash cache
• 6 – RamSan-70 Gorillas• 5.4 TB total of database Flash cache (2.7 TB
per server)• 600,000 IOPS/card• 70 Microsecond read latency• $134,400.00
The World’s Fastest Storage®Texas Memory Systems, Inc.
Total
RamSan710s $1,800,000.00
Servers $268,392.00Oracle $6,592,000.00RamSan70s $134,400.00Misc $40,000.00Total $8,834,792.00(Plus shipping, handling, installation, support)Of course this is street price, once discounts for bulk purchases and other things are removed, probably closer to 6-7 million.
The World’s Fastest Storage®Texas Memory Systems, Inc.
A Million here, a Million There• So, what are you getting for a million less?
– 5 GB/s*6=30 GB/s bandwidth– 30 TB usable raided flash with 175 microsecond latency for
all transactions– 400,000*6=2,400,000 IOPS from Storage– 2,400,000 IOPS from Cache– 128 CPUS and 2 TB of memory– Database Independence– OS independence– Less complexity– Better Performance– Lower operating costs (lower license, electric and cooling)– Great performance on OLTP/DWH/DSS
The World’s Fastest Storage®Texas Memory Systems, Inc.
All-In-One-Place
Exadata X2-8 RamSan Solution Difference
IOPS (Storage) 50,000 2,400,000 48X more
IOPS (Cache) 1,500,000 5,400,000 3.6X more
CPUs 128 128 0
Memory 2 TB 2 TB 0
Storage Capacity 28 TB Disk* 30 TB Flash 1.1X larger
Storage Latency 5 ms 0.175 ms 28X smaller
Flash Cache 5.2 TB at Cells 5.4 TB at database ~Equal
Cost $9,772,000.00 $8,834,792.00 11% less
*Note: Various sources quote the Exadata X2-8 HP capacity from 28-45 usable TB. Adding 15 TB RS710 RAIDed Flash: $900,000.00
The World’s Fastest Storage®Texas Memory Systems, Inc.
Summary
• Exadata uses old technology (disk), write through flash caches and new software to brute force performance
• New technology such as Flash and SSD as storage can get the same benefits, cheaper
• Exadata locks you into Oracle technology and hardware
• Using newer technology keeps your options open with better performance
The World’s Fastest Storage®Texas Memory Systems, Inc.
Questions/Comments?
The World’s Fastest Storage®Texas Memory Systems, Inc.
Thank You!
Mike [email protected]
http://www.statspackanalyzer.com/