data warehousing and bitmap indexes - more than just some bits

2014 © Trivadis

BASEL BERN BRUGG LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN

2014 © Trivadis

MORE THAN JUST SOME BITS…Data Warehousing and Bitmap Indexes

Dani Schnider, Trivadis AG

Oracle Open World 2011,San Francisco

02.10.2011Data Warehouseing & Bitmap Indexes

1

2014 © Trivadis

Fehlertolerante Ladeprozesse2

Dani Schnider Principal Consultant und

DWH/BI Lead Architectbei Trivadis in Zürich

Teacher for courses about Data Warehousing, SQL Optimization and Oracle Warehouse Builder

Co-author of book «Data Warehousing mit Oracle»

18. April 2012

2014 © Trivadis

11 Trivadis Niederlassungen mitüber 600 Mitarbeitenden

200 Service Level Agreements

Mehr als 4'000 Trainingsteilnehmer

Forschungs- und Entwicklungs-budget: CHF 5.0 / EUR 4 Mio.

Finanziell unabhängig undnachhaltig profitabel

Erfahrung aus mehr als 1'900 Projekten pro Jahr bei über 800 Kunden

Stand 12/2012

Hamburg

Düsseldorf

Frankfurt

FreiburgMünchen

Wien

Basel

ZürichBernLausanne

3

Stuttgart

Mit über 600 IT- und Fachexperten bei Ihnen vor Ort


3

2014 © Trivadis


4

SOME BITS ABOUT

BITMAP INDEXES

AND THEIR USAGE IN

DATA WAREHOUSES

2014 © Trivadis

Bitmap Indexes – Concept

Oracle® Database Concepts 11g Release 2 (11.2)


5

2014 © Trivadis

Bitmap Indexes – Concept


6

2014 © Trivadis

Bitmap Indexes – Example


7

2014 © Trivadis

Comparison B-tree Index / Bitmap Index


8

2014 © Trivadis

Comparison B-tree Index / Bitmap Index


9

2014 © Trivadis

Concatenated Bitmap Indexes

Bitmap Indexes on two or more columns are possible, but not useful

Why?

Concatenated bitmap index can be used only when all columns are filtered

Two or more separate bitmap indexes are more flexible and can be used for all combinations


10

2014 © Trivadis

Index Strategy on Star Schema

Dimension Tables: Unique b-tree index on primary key column Bitmap indexes on additional columns (optional)

Fact Tables: Bitmap indexes on all dimension key (foreign key) columns Bitmap join indexes on often used filter columns (optional) Usually no primary key on fact tables


11

2014 © Trivadis

Index Strategy on Star Schema


12

2014 © Trivadis

Query Optimization on Star Schemas

Typical Queries: Filter criteria on (multiple) dimension

tables Facts are selected by join with

dimensions

Problem: Tables with restrictions should be

read first Only two tables can be joined at a

time No relationships between dimension

tables


13

2014 © Trivadis

Star Transformation – Concept


14

2014 © Trivadis

Star Transformation – Example

What is the total revenue for product “Shorts” for all customers in Germany, grouped by male and female customers?


15

2014 © Trivadis

Star Transformation – Example


16

2014 © Trivadis

Bitmap Join Index – Concept

Bitmap index is defined on fact table

Indexed column is part of dimension table

Possible for star and snowflake schemas


17

2014 © Trivadis

Bitmap Indexes and ETL

Bitmap indexes are efficient for set-based ETL Index maintenance is deferred until end of each DML operation

Efficient index maintenance for Parallel DML

Avoid “hand-made” parallel ETL with multiple sessions Locking behavior of bitmap indexes

For large data loads: Rebuild indexes after ETL operation Set all indexes to UNUSABLE Load data into table Rebuild all indexes


19

2014 © Trivadis

Bitmap Indexes and Partitioning

Bitmap Indexes must always be LOCAL

Local indexes are recommended in Data Warehouse Only purpose of global indexes in DWH: unique indexes without partition key Restriction of local bitmap indexes is not a problem


20

2014 © Trivadis

Bitmap Indexes and Compression

Bitmap Indexes are always compressed Zeros are compressed, Ones are not compressed Bitmap contains number of zeros to next 1 bit

Bitmap Indexes need less disk space than B-tree Indexes Even for columns with high number of distinct keys

Detail description see «Bitmap Index Internals», © Julian Dyke http://www.juliandyke.com/Presentations/BitmapIndexInternals.ppt


21

http://www.juliandyke.com/Presentations/BitmapIndexInternals.ppt

2014 © Trivadis


Number of distinct keys has impact on size of index Bitmap index is usually smaller than b-tree index For high number of keys, size of bitmap index increases Size of b-tree index is more or less constant Sort order affects size of bitmap index, but not of b-tree index


22

Demo

10 100 1000 10000 1000000

50

100

150

200

250

300

350

Bitmap (scattered) Bitmap (sorted) B-tree index

2014 © Trivadis


Enabling table compression When a table is enabled for compression, all bitmap indexes must be disabled

Reason: Hakan factor of table Maximum number of rows that can be stored in a table block Hakan factor changes when table is enabled for compression (only first time)02.10.2011

Data Warehouseing & Bitmap Indexes23

Demo

2014 © Trivadis

Bitmap Indexes – Conclusion

Bitmap Indexes are a powerful Data Warehouse feature Can be combined in ad hoc queries Star Transformation Useful for selective and non-selective columns

Bitmap Indexes are useful for large databases Efficient for set-based ETL and Parallel DML Local bitmap indexes on partitioned tables Compression of bitmap indexes

There are almost no reasons to use b-tree indexes in a Data Warehouse B-tree indexes are mainly used for Primary Key or Unique Key constraints


24

2014 © Trivadis

References

Oracle® Database Data Warehousing Guide 11g Release 2 (11.2) Oracle Corporation, http://www.oracle.com/pls/db112/

Oracle® Database Concepts 11g Release 2 (11.2) Oracle Corporation, http://www.oracle.com/pls/db112/

Christian Antognini: Troubleshooting Oracle Performance Apress, http://www.apress.com/9781590599174

Jonathan Lewis: Cost-Based Oracle Fundamentals Apress, http://www.apress.com/9781590596364

Julian Dyke: Bitmap Index Internals http://www.juliandyke.com/Presentations/BitmapIndexInternals.ppt


25

http://www.oracle.com/pls/db112/

http://www.oracle.com/pls/db112/

http://www.apress.com/9781590599174

http://www.apress.com/9781590596364

http://www.juliandyke.com/Presentations/BitmapIndexInternals.ppt

2014 © Trivadis

BASEL BERN BRUGG LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN

2014 © Trivadis

THANK YOU.Trivadis AG

Dani Schnider

Europa-Strasse 5CH-8152 Glattbrugg/ZürichSwitzerland

Tel. +41 44 808 70 20Fax +41 44 808 70 21

[email protected]


26

data warehousing and bitmap indexes - more than just some bits

Presentations & Public Speaking