olap systems and multidimensional queries iolap servers relational olap (rolap), multidimensional...

OLAP Systems and Multidimensional Queries I

Krzysztof Dembczynski

Intelligent Decision Support Systems Laboratory (IDSS)Poznan University of Technology, Poland

Software Development TechnologiesMaster studies, second semester

Academic year 2014/15 (winter course)

1 / 45

Review of the previous lectures

• Mining of massive datasets

• Evolution of database systems: operational vs. analytical systems.

• Dimensional modeling.

• Extraction, transformation and load of data.

2 / 45

Outline

1 Motivation

2 OLAP Servers

3 ROLAP

4 SQL

5 Summary

3 / 45

Outline

1 Motivation

2 OLAP Servers

3 ROLAP

4 SQL

5 Summary

4 / 45

OLAP systems

• The next step is to provide solutions for querying and reportingmultidimensional analytical data.

• The goal is to provide efficient solutions for physical representationand processing of these data.

5 / 45

Multidimensional reports

• OLAP servers provide an effective solution for accessing andprocessing large volumes of high dimensional data.

• OLAP systems provide tools for multidimensional reporting.

6 / 45

Outline

1 Motivation

2 OLAP Servers

3 ROLAP

4 SQL

5 Summary

7 / 45

Multidimensional cube

• The proper data model for multidimensional reporting is themultidimensional one.

8 / 45

Operators in multidimensional data model

• Roll up – summarize dataalong a dimension hierarchy.

• Drill down – go from higherlevel summary to lower levelsummary or detailed data.

• Slice and dice – correspondsto selection and projection.

• Pivot – reorient cube.

• Raking, Time functions,etc..

9 / 45

Lattice of cuboids

• Different degrees of summarizations are presented as a lattice ofcuboids.

Example for the dimensions: time, product, location, supplier.

Using this structure, one can easily show roll up and drill down operations.

10 / 45

Total number of cuboids

• For an n-dimensional data cube, the total number of cuboids that canbe generated is:

T =∏

i=1,...,n

(Li + 1),

where Li is the number of levels associated with dimension i(excluding the virtual top level ”all” since generalizing to ”all” isequivalent to the removal of a dimension).

• For example, if the cube has 10 dimensions and each dimension has 4levels, the total number of cuboids that can be generated will be:

l = 510 = 9, 8× 106.

11 / 45


• Example: Consider a simple database with two dimensions:

I Columns in Date dimension: day, month, yearI Columns in Localization dimension: street, city, country.I Without any information about hierarchies, the number of all possible

group-bys is 26:

∅ ∅day street

month cityyear country

day, month ./ street, cityday, year street, country

month, year city, countryday, month, year street, city, country

12 / 45


• Example: Consider a simple database with two dimensions:I Columns in Date dimension: day, month, yearI Columns in Localization dimension: street, city, country.

I Without any information about hierarchies, the number of all possiblegroup-bys is 26:

∅ ∅day street




12 / 45


• Example: Consider a simple database with two dimensions:I Columns in Date dimension: day, month, yearI Columns in Localization dimension: street, city, country.I Without any information about hierarchies, the number of all possible

group-bys is

26:

∅ ∅day street




12 / 45


• Example: Consider a simple database with two dimensions:I Columns in Date dimension: day, month, yearI Columns in Localization dimension: street, city, country.I Without any information about hierarchies, the number of all possible

group-bys is 26:

∅ ∅day street




12 / 45


• Example: Consider the same relations but with defined hierarchies:

I day → month → yearI street → city → countryI Many combinations of columns can be excluded, e.g. group by day,

year, street, country.I The number of group-bys is then 42:

∅ ∅year ./ country


13 / 45


• Example: Consider the same relations but with defined hierarchies:I day → month → yearI street → city → country

I Many combinations of columns can be excluded, e.g. group by day,




13 / 45


• Example: Consider the same relations but with defined hierarchies:I day → month → yearI street → city → countryI Many combinations of columns can be excluded, e.g. group by day,

year, street, country.I The number of group-bys is then

42:



13 / 45


• Example: Consider the same relations but with defined hierarchies:I day → month → yearI street → city → countryI Many combinations of columns can be excluded, e.g. group by day,




13 / 45

Three types of aggregate functions

• distributive: count(), sum(),max(),min(),

• algebraic: ave(), std dev(),

• holistic: median(),mode(), rank().

14 / 45

OLAP servers

• Relational OLAP (ROLAP),

• Multidimensional OLAP (MOLAP),

• Hybrid OLAP (HOLAP).

15 / 45

Outline

1 Motivation

2 OLAP Servers

3 ROLAP

4 SQL

5 Summary

16 / 45

ROLAP

• ROLAP servers use a relational or post-relational databasemanagement system to store and manage warehouse data.

• ROLAP systems use SQL and its OLAP extensions.

• Optimization techniques:I Denormalization,I Materialized views,I Partitioning,I Joins,I Indexes,I Query processing.

17 / 45

ROLAP

• Advantages of ROLAP Servers:I Scalable with respect to the number of dimensions,I Scalable with respect to the size of data,I Sparsity is not a problem (fact tables contain only facts),I Mature and well-developed technology.

• Disadvantage of ROLAP Servers:I Worse performance than MOLAP,I Additional data structures and optimization techniques used to improve

the performance.

18 / 45

Grouping

• Group-by is usually performed in the following way:

I Partition tuples on grouping attributes: tuples in same group areplaced together, and in different groups separated,

I Scan tuples in each partition and compute aggregate expressions.

• Two techniques for partitioningI Sorting

• Sort by the grouping attributes,• All tuples with same grouping attributes will appear together in sorted

list.

I Hashing• Hash by the grouping attributes,• All tuples with same grouping attributes will hash to same bucket,• Sort or re-hash within each bucket to resolve collisions.

• In OLAP queries use intermediate results to compute more generalgroup-bys

19 / 45

Grouping

• Group-by is usually performed in the following way:I Partition tuples on grouping attributes: tuples in same group are

placed together, and in different groups separated,I Scan tuples in each partition and compute aggregate expressions.

• Two techniques for partitioningI Sorting

• Sort by the grouping attributes,• All tuples with same grouping attributes will appear together in sorted

list.

I Hashing• Hash by the grouping attributes,• All tuples with same grouping attributes will hash to same bucket,• Sort or re-hash within each bucket to resolve collisions.

• In OLAP queries use intermediate results to compute more generalgroup-bys

19 / 45

Grouping

• Example: Grouping by sorting (Month, City):

Month City Sale

March Poznan 105March Warszawa 135March Poznan 50April Poznan 150April Krakow 175May Warszawa 100May Poznan 70May Warszawa 75

→

Month City Sale

March Poznan 105March Poznan 50March Warszawa 135April Krakow 175April Poznan 150May Poznan 70May Warszawa 100May Warszawa 75

→

Month City Sale

March Poznan 155March Warszawa 135April Krakow 175April Poznan 150May Poznan 70May Warszawa 175

20 / 45

Grouping

• Example: Grouping by sorting (Month; City; Month, City):

Month City Sale

March Poznan 155March Warszawa 135April Krakow 175April Poznan 150May Poznan 70May Warszawa 175

→

Month Sale

March 155March 135April 150April 175May 175May 70

→

Month Sale

March 285April 325May 245

↓City Sale

Krakow 175Poznan 155Poznan 150Poznan 70

Warszawa 135Warszawa 175

→

City Sale

Krakow 175Poznan 375

Warszawa 310

21 / 45

Outline

1 Motivation

2 OLAP Servers

3 ROLAP

4 SQL

5 Summary

22 / 45

SQL queries

• Querying the star schema

23 / 45

SQL queries

SQL – group by

SELECT Name, AVG(Grade)

FROM Students grades G, Student S

WHERE G.Student = S.ID

GROUP BY Name;

Name AVG(Grade)

Inmon 4.8Kimball 4.7Gates 4.0Todman 4.5

24 / 45

SQL queries

SQL– group by

SELECT Academic year, Name, AVG(Grade)

FROM Students grades G, Academic year A, Professor P

WHERE G.Professor = P.ID and G.Academic year = A.ID

GROUP BY Academic year, Name;

Academic year Name AVG(Grade)

2001/2 Stefanowski 4.22002/3 Stefanowski 4.02003/4 Stefanowski 3.92001/2 S lowinski 4.12002/3 S lowinski 3.82003/4 S lowinski 3.62003/4 Dembczynski 4.8

25 / 45

SQL queries

• OLAP extensions in SQL:

I GROUP BY ROLLUP,I GROUP BY CUBE,I GROUP BY GROUPING SETSI GROUPING and DECODE/CASEI OVERI Ranking functions

26 / 45

SQL queries

• GROUP BY CUBE

SELECT Time, Product, Location, Supplier, SUM(Gain)

FROM Sales

GROUP BY CUBE (Time, Product, Location, Supplier);

27 / 45

SQL queries

• GROUP BY CUBE


FROM Sales

GROUP BY Time, Product, Location, Supplier

UNION ALL

SELECT Time, Product, Location, ’’*’’, SUM(Gain)

FROM Sales

GROUP BY Time, Product, Location

UNION ALL

SELECT Time, Product, ’’*’’, Location, SUM(Gain)

FROM Sales


UNION ALL

. . .UNION ALL

SELECT ’*’, ’*’, ’*’, ’*’, SUM(Gain)

FROM Sales;

28 / 45

SQL queries

• GROUP BY CUBE


FROM Students grades GROUP BY CUBE(Academic year, Name);


2001/2 Stefanowski 4.22001/2 S lowinski 4.12002/3 Stefanowski 4.02002/3 S lowinski 3.82003/4 Stefanowski 3.92003/4 S lowinski 3.62003/4 Dembczynski 4.82001/2 NULL 4.152002/3 NULL 3.852003/4 NULL 3.8NULL Stefanowski 3.9NULL S lowinski 3.6NULL Dembczynski 4.8NULL NULL 3.95

29 / 45

SQL queries

• GROUP BY ROLLUP


FROM Sales

GROUP BY ROLLUP (Time, Product, Location, Supplier);

30 / 45

SQL queries

• GROUP BY ROLLUP


FROM Sales

GROUP BY Time, Product, Location, Supplier

UNION ALL

SELECT Time, Product, Location, ’’*’’, SUM(Gain)

FROM Sales


UNION ALL

SELECT Time, Product, ’’*’’, ’’*’’, SUM(Gain)

FROM Sales

GROUP BY Time, Product

UNION ALL

SELECT Time, ’’*’’, ’’*’’, ’’*’’, SUM(Gain)

FROM Sales

GROUP BY Time

UNION ALL

SELECT ’*’, ’*’, ’*’, ’*’, SUM(Gain)

FROM Sales;

31 / 45

SQL queries

• GROUP BY ROLLUP


FROM Students grades G

GROUP BY ROLLUP(Academic year, Name);


2001/2 Stefanowski 4.22001/2 S lowinski 4.12002/3 Stefanowski 4.02002/3 S lowinski 3.82003/4 Stefanowski 3.92003/4 S lowinski 3.62003/4 Dembczynski 4.82001/2 NULL 4.152002/3 NULL 3.852003/4 NULL 3.8NULL NULL 3.95

32 / 45

SQL queries

• GROUP BY GROUPING SETS


FROM Sales

GROUP BY GROUPING SETS ((Time), (Product), (Location), (Supplier));

33 / 45

SQL queries


SELECT Time, ’’*’’, ’’*’’, ’’*’’, SUM(Gain)

FROM Sales

GROUP BY Time

UNION ALL

SELECT ’’*’’, Product, ’’*’’, ’’*’’, SUM(Gain)

FROM Sales

GROUP BY Product

UNION ALL

SELECT ’’*’’, ’’*’’, Location, ’’*’’, SUM(Gain)

FROM Sales

GROUP BY Location

UNION ALL

SELECT ’*’, ’*’, ’*’, Supplier, SUM(Gain)

FROM Sales GROUP BY Supplier;

34 / 45

SQL queries



FROM Students grades GROUP BY GROUPING SETS ((Academic year), (Name),());


2001/2 NULL 4.152002/3 NULL 3.852003/4 NULL 3.8NULL Stefanowski 3.9NULL S lowinski 3.6NULL Dembczynski 4.8NULL NULL 3.95

35 / 45

SQL queries

• GROUPING(<column expression>)I Returns a value of 1 if the value of expression in the row is a null

representing the set of all values.I <column expression> is a column or an expression that contains a

column in a GROUP BY clause.I GROUPING is used to distinguish the null values that are returned by

ROLLUP, CUBE or GROUPING SETS from standard null values.I The NULL returned as the result of a ROLLUP, CUBE or GROUPING SETS

operation is a special use of NULL.

36 / 45

SQL queries

• GROUPING(<column expression>)

SELECT Extra scholarship, AVG(Grade), GROUPING(Extra scholarship) as

Grouping

FROM Students grades

GROUP BY ROLL UP(Extra scholarship);

Extra scholarship AVG(Grade) Grouping

Yes 4.15 0No 3.61 0NULL 4.03 0NULL 3.89 1

36 / 45

SQL queries

• DECODE(expression , search , result [, search ,result]... [, default] )

I If the value of expression is equal to search, then result isreturned, otherwise default is returned.

I The functionality is similar to CASE expression,I The results of GROUPING() can be passed into a DECODE function or

the CASE expression.

37 / 45

SQL queries

• DECODE(expression , search , result [, search ,

result]... [, default] )

SELECT DECODE(GROUPING(Extra scholarship), 1, "Total Average",

Extra scholarship) as Extra scholarship, AVG(Grade)


GROUP BY ROLL UP(Extra scholarship);

Extra scholarship AVG(Grade)

Yes 4.15No 3.61NULL 4.03Total average 3.89

37 / 45

SQL queries

• OVER():

I Determines the partitioning and ordering of a rowset before theassociated window function is applied.

I The OVER clause defines a window or user-specified set of rows within aquery result set.

I A window function then computes a value for each row in the window.I The OVER clause can be used with functions to compute aggregated

values such as moving averages, cumulative aggregates, running totals,or a top N per group results.

I Syntax:OVER (

[ <PARTITION BY clause> ]

[ <ORDER BY clause> ]

[ <ROW or RANGE clause> ]

)

38 / 45

SQL queries

• OVER():I Determines the partitioning and ordering of a rowset before the

associated window function is applied.

I The OVER clause defines a window or user-specified set of rows within aquery result set.



I Syntax:OVER (




)

38 / 45

SQL queries


associated window function is applied.I The OVER clause defines a window or user-specified set of rows within a

query result set.



I Syntax:OVER (




)

38 / 45

SQL queries



query result set.I A window function then computes a value for each row in the window.

I The OVER clause can be used with functions to compute aggregatedvalues such as moving averages, cumulative aggregates, running totals,or a top N per group results.

I Syntax:OVER (




)

38 / 45

SQL queries



query result set.I A window function then computes a value for each row in the window.I The OVER clause can be used with functions to compute aggregated


I Syntax:OVER (




)

38 / 45

SQL queries

• OVER():

I PARTITION BY:

• Divides the query result set into partitions. The window function isapplied to each partition separately and computation restarts for eachpartition.

I ORDER BY:

• Defines the logical order of the rows within each partition of the resultset, i.e., it specifies the logical order in which the window functioncalculation is performed.

I ROW | RANGE:

• Further limits the rows within the partition by specifying start and endpoints within the partition.

• This is done by specifying a range of rows with respect to the currentrow either by logical association or physical association.

• The ROWS clause limits the rows within a partition by specifying a fixednumber of rows preceding or following the current row.

• The RANGE clause logically limits the rows within a partition byspecifying a range of values with respect to the value in the current row.

• Preceding and following rows are defined based on the ordering in theORDER BY clause.

39 / 45

SQL queries

• OVER():I PARTITION BY:


I ORDER BY:

• Defines the logical order of the rows within each partition of the resultset, i.e., it specifies the logical order in which the window functioncalculation is performed.

I ROW | RANGE:






39 / 45

SQL queries



I ORDER BY:• Defines the logical order of the rows within each partition of the result

set, i.e., it specifies the logical order in which the window functioncalculation is performed.

I ROW | RANGE:






39 / 45

SQL queries





I ROW | RANGE:• Further limits the rows within the partition by specifying start and end

points within the partition.





39 / 45

SQL queries






points within the partition.• This is done by specifying a range of rows with respect to the current

row either by logical association or physical association.




39 / 45

SQL queries







row either by logical association or physical association.• The ROWS clause limits the rows within a partition by specifying a fixed

number of rows preceding or following the current row.



39 / 45

SQL queries








number of rows preceding or following the current row.• The RANGE clause logically limits the rows within a partition by

specifying a range of values with respect to the value in the current row.


39 / 45

SQL queries








number of rows preceding or following the current row.• The RANGE clause logically limits the rows within a partition by

specifying a range of values with respect to the value in the current row.• Preceding and following rows are defined based on the ordering in the

ORDER BY clause.

39 / 45

SQL queries

• Ranking functions:

I RANK () OVER:

• Returns the rank of each row within the partition of a result set. Therank of a row is one plus the number of ranks that come before the rowin question.

I DENSE RANK () OVER:

• Returns the rank of rows within the partition of a result set, withoutany gaps in the ranking. The rank of a row is one plus the number ofdistinct ranks that come before the row in question.

I NTILE (integer expression) OVER:

• Distributes the rows in an ordered partition into a specified number ofgroups. The groups are numbered, starting at one. For each row,NTILE returns the number of the group to which the row belongs.

I ROW NUMBER () OVER:

• Returns the sequential number of a row within a partition of a resultset, starting at 1 for the first row in each partition.

40 / 45

SQL queries

• Ranking functions:I RANK () OVER:


I DENSE RANK () OVER:

• Returns the rank of rows within the partition of a result set, withoutany gaps in the ranking. The rank of a row is one plus the number ofdistinct ranks that come before the row in question.





40 / 45

SQL queries



I DENSE RANK () OVER:• Returns the rank of rows within the partition of a result set, without

any gaps in the ranking. The rank of a row is one plus the number ofdistinct ranks that come before the row in question.





40 / 45

SQL queries





I NTILE (integer expression) OVER:• Distributes the rows in an ordered partition into a specified number of

groups. The groups are numbered, starting at one. For each row,NTILE returns the number of the group to which the row belongs.



40 / 45

SQL queries





I NTILE (integer expression) OVER:• Distributes the rows in an ordered partition into a specified number of

groups. The groups are numbered, starting at one. For each row,NTILE returns the number of the group to which the row belongs.

I ROW NUMBER () OVER:• Returns the sequential number of a row within a partition of a result

set, starting at 1 for the first row in each partition.

40 / 45

SQL queries

• Examples:I Ranking of the students

SELECT Student, Avg(Grade), RANK () OVER (ORDER BY Avg(Grade) DESC)

FROM Students grades GROUP BY Student;

I To sort according to rank, we need to order the resulting relation:

SELECT Student, Avg(Grade), RANK () OVER (ORDER BY Avg(Grade) DESC) AS

rank of grades

FROM Students grades GROUP BY Student ORDER BY rank of grades;

41 / 45

SQL queries

• Examples:I Ranking of students partitioned by instructors.

SELECT Instructor Name, Student, Avg(Grade), RANK () OVER (PARTITION BY

Instructor Name ORDER BY Avg(Grade) DESC) AS rank 1


GROUP BY Student,Instructor Name

ORDER BY Instructor Name, rank 1;

I Moving average of a student:

SELECT Student, Academic year, AVG (grades) OVER (PARTITION BY Student

ORDER BY Academic year DESC ROWS UNBOUNDED PRECEDING)


ORDER BY Student, Academic year;

42 / 45

Outline

1 Motivation

2 OLAP Servers

3 ROLAP

4 SQL

5 Summary

43 / 45

Summary

• OLAP Systems: Relational OLAP.

• SQL for analytical queries.

44 / 45

Bibliography

• J. Han and M. Kamber. Data Mining: Concepts and Techniques.

Morgan Kaufmann Publishers, second edition, 2006

45 / 45

olap systems and multidimensional queries iolap servers relational olap (rolap), multidimensional...

Documents