implementing data cubes efficiently
Post on 05-Jan-2016
76 Views
Preview:
DESCRIPTION
TRANSCRIPT
1
Vicky :: Cao Hui PingSherman :: Chow Sze Ming
CTH :: Chong Tsz HoRonald :: Woo Lok Yan
Ken :: Yiu Man Lung
ImplementingData Cubes Efficiently
2
Content
Background Introduction of Datacube Problem defined Lattice model Greedy algorithm
How to do? How good? How bad ?
Evaluations Conclusion
3
Background
DSS (Decision Support System)Gain competitiveness for business
Data warehouseMaintain historical informationUse “Data cube” to summarize results Identify trendsPerformance issue (time and space)Need to reuse result (materialization of views)
4
Introduction of datacube Datacube
Dimensionality (number of GROUP-BYs)Aggregated data: Values in each cellDimension of datacube Detail of summaryHigher Dimension Higher detail
Common operationsDrill down: Look in more detailRoll up: Look in less detail
5
What is a data cube?Date
Produ
ct
Cou
ntr
ysum
sum TV
VCRPC
1Qtr 2Qtr 3Qtr 4Qtr
U.S.A
Canada
Mexico
sum
Total annual salesof TV in U.S.A.
6
Our problem
Physically materialize the whole data cubeBest query response Heavy pre-computing, large storage space i.e. Time efficient but space inefficient
Materialize nothingWorse query responseDynamic query evaluation, less storage space i.e. Space efficient but time inefficient
7
Problem on materialized views
Materialize only part of the data cubeBalance the storage space and responseWhat is the best subject to materialize?Addressed in this paper
Source Size Time (sec) Ratio
From cell itself 1 2.07 N/A
View (s) 10,000 2.38 0.000031
View (p,s) 800,000 20.77 0.000023
View (p,s,c) 6,000,000 226.23 0.000037
8
Data? View?
We use data cube to modify aggregate data.
So what we use to model view?
Lattice!
9
Example of lattice diagram
8 possible grouping on the dimensionsp for Parts for Supplierc for Customer# of rows of data shown
next to the grouping
psc 6M
pc 6M ps 0.8M sc 6M
p 0.2M s 0.01M c 0.1M
none 1
An example of Regular Lattice
10
≼ operator
Suppose c d≼ The view d can be used to derive the view c c is the ancestor of d in lattice diagram
Impose a partial order on the views Usage on dimensions
(part) (part,customer) ≼ (part) (customer) ⋠
Usage within attribute value (year) (quarter) (month) (day)≼ ≼ ≼ (year) (quarter) (week) (day)≼ ≼ ≼
week month
day
year
quarter
An example of Irregular Lattice
11
Regular lattices with equal domain size
Grouping attributes: A1,A2,…, An (domain: r) Attribute for aggregation: B Efficient algorithm
m: # of rows in top viewsk = log ⌈ r m⌉
Strategy k, j, and n Space Time
Space-optimal M m2n
Time-optimal k>j (2rr/(r+1))n (2rr/(r+1))n
k<j and k ≤ n/2 m m2n
k<j and k > n/2 m nCj rj
12
The problem
The previous technique cannot be applied to the irregular lattices
Irregular lattices is common in data warehouse The optimization of views for irregular lattice is
NP-complete problem (inefficient!) Use Greedy Algorithm i.e. use heuristics to obtain approximate solution
13
Greedy algorithm
Being as greedy as possible in each step!!
Simple example: Use the smallest number of coins to pay $50 cents
Suppose we have many coins of 20 cents, 10 cents and 5 cents.
14
How to be greedy?
Common sense approach:Select the largest coin: 20 centsSelect the largest coin again: 20 centsRemaining amount = 50 – 20 – 20 = 10 centsWe cannot select the largest coin again.We choose the 2nd largest coin 10 cents instead.
Only 3 coins are needed! Optimal solution!
15
Definition of “benefit of view”
C(v) denotes cost of view (v) B(v,S) denotes benefit of a view (v) relativ
e to a set of views (S) For each w v≼
Let u be the view of least cost in S such that w u≼
Bw = max{ C(u) – C(v) ,0}
B(v,S) = ∑w v≼ Bw
16
Greedy algorithm
In each step Select the view with the most benefit Add it to the result
AlgorithmS={top view};for i=1 to k {
select view v not in S such that B(v,S) is maximizedS = S union {v}
}return S;
17
Selecting the first view
After selecting coins, let us back to our problem, selecting views.
We must materialize the top view i.e. the view grouping by all attributesCannot be constructed from other viewsAvoid going to the raw data
18
Selecting k views more
Space is limited! Suppose we can only select k more views.
For each view which is not yet selected, calculate the benefit of materializing it.
Pick the one with maximum benefit!!!
Let’s set k = 2 for examples.
19
Example
a
b c
d e f
g h
100
50 75
20 40
30
1 10
E.g. The cost of constructing view b given the view A is 100
If we choose b to materialize, the new cost of constructing view b is 50.
20
First round
a
b c
d e f
g h
100
50 75
20 40
30
1 10
Notice that not only b, but also d, e, g and h can be calculated from b
So the total benefit is (100 – 50) x 5 = 250
21
Continue… Similarly, the benefit
of materializing c is (100 – 75) x 5 = 125a
b c
d e f
g h
100
50 75
20 40
30
1 10
Benefit
b 250
c 125
22
Not yet finish… For e,
Benefit =
(100-30) x 3
= 210
a
b c
d e f
g h
100
50 75
20 40
30
1 10
Benefit
b 250
c 125
e 210
23
Let’s choose b!
a
b c
d e f
g h
100
50 75
20 40
30
1 10
For d and f ,
Benefit =
(100-20) x 2
= 160 and
(100-40) x 2 =
120 respectively.
Benefit
b 250
c 125
d 160
e 210
f 120
24
Next round?
Seems we should choose e, as it has the second largest benefit.
Let’s see what will happen in the second round. Benefit
b 250
c 125
d 160
e 210
f 120
25
Second round!
a
b c
d e f
g h
100
50 75
20 40
30
1 10
Now, only c and f get benefit if we materialize c (since e, g and h can be more efficiently calculated by using b)
Benefit
= (100 – 75) x 2 = 50
Benefit
c 50
26
How about choosing f?
a
b c
d e f
g h
100
50 75
40
30
1 10
If we choose f, we found that h can be effectively calculated by using f instead of b.
Benefit
= (100 – 40) + (50 – 40)
Benefit
c 50
f 7020
27
Easy to work out others
Benefit of d
= (50 – 20) x 2 = 60 Benefit of e
= (50 – 30) x 3 = 60 Benefit of g
= 50 – 1 = 49 Benefit of h
= 50 – 10 = 40
a
b c
d e f
g h
100
50 75
20 40
30
1 10
28
Observation
In the first round, the benefit of choosing f (only 120) is far from the best choice (250)
But in second round, choosing f gives the maximum benefit!1st round Benefit
b 250
c 125
d 160
e 210
f 120
2nd round Benefit
c 50
d 60
e 70
f 70
g 49
29
Simple? Optimal?
Trade off again! This simple algorithm is not optimal in all cases!
Consider the following case…
30
Bad example
a
b dc
200
100 100
20 nodes
Total 1000
99
31
Bad example
a
b dc
200
100 100
20 nodes
Total 1000
99
Choose c Benefit
= (200-99) x (1 + 20 + 20)= 4141= maximum
32
Bad example
a
b dc
200
100 100
20 nodes
Total 1000
99
Now choose either 1 of b and d (same benefit)
33
Bad example
a
b dc
200
100 100
20 nodes
Total 1000
99
How about these? Very expensive!!!
34
Optimal solution should be…
a
b dc
200
100 100
20 nodes
Total 1000
99
Only c is a little bit expensive.
35
Some theoretical result
It can be proved that we can get at least (e – 1 ) / e % (which is about 63%) of the benefit of the optimal algorithm.
36
Extensions (1)
ProblemThe views in a lattice are unlikely to have the
same probability of being requested in a query.
Solution:We can weight each benefit by its probability.
37
Extensions (2)
Problem Instead of asking for some fixed number (k) of
views to materialize, we might instead allocate a fixed amount of space to views.
SolutionWe can consider the “benefit of each view per
unit space”.
38
Conclusions
Materialization of views is an essential query optimization strategy for decision-support applications.
Reason to materialize some part of the data cube but not all of the cube.
A lattice framework that models multidimensional analysis very well.
39
Conclusions (cont.)
Finding optimal solution in the case of irregular lattice is NP-hard.
Introduction of greedy algorithm Greedy algorithm work on this lattice and
pick the almost right views to materialize.
40
Conclusions (the end)
There exists cases which greedy algorithm fails to produce optimal solution.
But greedy algorithm has guaranteed performance
Expansion of greedy algorithm.
41
Reference
Venky Harinarayan, Anand Rajaraman, Jeffrey D. Ullman. Implementing Data Cubes Efficiently. SIGMOD’96:205-216.
42
Thank you~
Q & A Section
top related