query optimization in distributed database systems
DESCRIPTION
Query optimization in distributed database systems. Framework for query optimization. The selection of a query processing strategy involves: determining the physical copies of the fragments upon which to execute the query - PowerPoint PPT PresentationTRANSCRIPT
Query optimization in distributed database systems
2
Framework for query optimization • The selection of a query processing strategy
involves:– determining the physical copies of the fragments upon
which to execute the query
– selecting the order of the execution of operations, particularly, this involves the determination of a „good” sequence of joins
– selecting the method for executing each operation
3
Transmission cost
• Transmission requirements are neutral with respect to systems; they are typically a function of the amount of data transmitted among sites
• The optimization of a distributed query can be partitioned into two independent problems: the distribution of the access strategy among sites, which is done considering transmission only, and the determination of local access strategies at each site, which use traditional methods of centralized databases
• Transmission cost:
TC(X) = C0 + C1 * x
4
Database Profile
Database profile:• The number of tuples in each relation Ri (card(Ri))
• The size of each attribute A (size(A) )
• The size of Ri (size(Ri)) is sum of the sizes of its attributes
• For each attribute A in each relation Ri: the number of distinct values appearing in Ri (val(A[Ri])), max and min
LDBS1 LDBS2
Supply1 Dept1
Supply2 Dept2
5
Database Profile
SNUM PNUM DEPTNUM QUANsize 6 7 2 10val 3000 1000 30 500
Supply card(Supply)=50 000
DEPTNUM NAME AREA MGRNUMsize 2 15 1 7val 30 30 6 30
Dept card(dept)= 30
6
Database Profile
SNUM PNUM DEPTNUM QUANsize 6 7 2 10val 1800 1000 20 500
DEPTNUM NAME AREA MGRNUMsize 2 15 1 7val 10 10 2 10
Supply1 card(Supply1)=30 000 site(Supply1) = 1
Dept1 card(dept)= 10 site(Dept1) = 2
7
Profile of partial results of algebraic
operations - SELECTION
Let S denote the result of performing a unary relation over a relation R
• Cardinality - to each selection we associate a selectivity factor which is the fraction of tuples satisfying it
In simple selection attribute = value (A=v), can be defined as follows:
= 1/val(A[Ri])
under the assumptions that values are homogeneously distributed. Thus
card(S) = * card(R)
8
Profile of partial results of algebraic
operations - SELECTION
• Size: selection does not affect the size of relations
size(S) = size(R)
• Distinct values : depends on the selection criterion
Consider an attribute B which is not used in selection formula. The determination of val(B[S]) may be as follows
Given n=card(R) - objects uniformly distributed over m = val(B[R]) colors. How many different colors c= val(B[S]) are selected if we take just r objects?
9
Profile of partial results of algebraic
operations - SELECTION
• Yao approximation:
r, for r < m/2
c(n, m, r) = (r+m)/3 for m/2 < r < 2m
m, for r > 2m
10
Profile of partial results of algebraic operations - PROJECTION
Let S denote the result of performing a unary relation over a relation R
• Cardinality – projection affects the cardinality of operands since duplicates are eliminated from the result. This effect is difficult to evaluate, the following three rules can be applied– If the projection involves a single attribute A, set
card(S) = val(A[R])
– If the product AiAttr(S) val(Ai[R]) is less than card(R), where Attr(S) are the attributes in the result of the projection, set
card(S) = AiAttr(S) val(Ai[R])
11
Profile of partial results of algebraic operations - PROJECTION
– If the projection includes a key of R, set
card(S) = card(R)
• Note that if the system does not eliminate duplicates, the cardinality of the result is the same as the cardinality of the operand relation
• Size: the size of the result of a projection is reduced to the sum of the sizes of attributes in its specification
• Distinct values : the distinct values of projected attributes are the same as in the operand relation
12
Profile of partial results of algebraic operations – GROUP BY
Let G denote the attributes on which the grouping is performed, AF indicates the aggregate functions to be evaluated
• Cardinality – we give an upper bound on the cardinality of S:
card(S) < AiG val(Ai[R])
• Size: for all attributes A appearing in G
size(R.A) = size (S.A)
• Distinct values : for all attributes A appearing in G
val(A[S]) = val(A[R])
13
Profile of partial results of algebraic operations – UNION
• Cardinality – we have:
card(T) < card(R) + card(S)
Equality holds when duplicates are not eliminated
• Size: we have
size(T) = size(R) = size(S)
• Distinct values : an upper bound is
val(A[T]) < val(A[R]) + val(A[S])
14
Profile of partial results of algebraic operations – DIFFERENCE
• Cardinality – we have:
max(0, card((R)-card(S)) < card(T) < card(R)
• Size: we have
size(T) = size(R) = size(S)
• Distinct values : an upper bound is
val(A[T]) < val(A[R])
15
Profile of partial results of algebraic operations – CARTESIAN PRODUCT
• Cardinality – we have:
card(T) < card(R) x card(S)
• Size: we have
size(T) = size(R) + size(S)
• Distinct values : the distinct values of attributes are the same as in the operand relation
16
Profile of partial results of algebraic operations – JOIN
• Cardinality – estimating precisely the cardinality of T is very complex; we can give an upper bound to card(T) because card(T) < card(R) x card(S), but this value is usually much higher than the actual cardinality. Assuming that all the values of A in R appear also as values of B in S and vice versa and that the two attributes are both uniformly distributed over tuples of R and S, we have
card(T) = (card(R) x card(S))/val(A[R])
if one of the two attributes, say A, is a key of R, then
card(T) = card(S)
17
Profile of partial results of algebraic operations – JOIN
• Size: we have
size(T) = size(R) + size(S)
In the case of natural join the size of the join attribute must be subtracted from the size of the result
• Distinct values : if A is a join attribute, an upper bound is
val(A[T]) < min(val(A[R]), val(B[S]) )
if A is not a join attribute, an upper bound is
val(A[T]) < val(A[R]) + val(B[S])
18
Profile of partial results of algebraic operations – SEMIJOIN
Consider the semijoin T=R SJ A=B S
• Cardinality – the estimation of the cardinality of T is similar to that of a selection operation; we denote with the selectivity of the semijoin operation, which measures the fraction of the tuples of R which belong to the result. The estimation is the following:
= 1/val(A[S]) / val(dom[A])
Given
card(T) = * card(R)
19
Profile of partial results of algebraic operations – SEMIJOIN
• Size: The size of the result of a semijoin is the same as the size of its first operand
size(T) = size(R)
• Distinct values : the number of distinct values of attributes which do not belong to the semijoin specification can be estimated using Yao’s formula with n= card(R), m=val(A[R]), and r =card(T). If A is the only attribute appearing in the semijoin specification, then
val(A[T]) = * val(A[R])
20
Architecture of a Query Processing
Parser
Query
QueryRewrite
Query Optimizer
Internal rep.
Internal rep.
Catalog
PlanRefinement
QueryExecutionEngine
result
Base data
plan query executionplan
21
Architecture of a Query Processing
• Parser: the query is parsed and translated into an internal representation (flex and bison can be used for the construction of SQL parser)
• Query Rewrite: query rewrite transforms a query in order to carry out optimizations that are good regardless of the physical state of the system (elimination of redundant predicates, unnesting of subqueries, simplification of expressions). Query rewrite is carried out by a rule engine
• Query Optimizer: this component carries out optimizations that depend on the physical state of the system. QO decides which index, which method, and in which order to execute operations of a query.
22
Architecture of a Query Processing• Query optimizer: in distributed system QO must decide at
which site each operation is to be executed. QO enumerates alternative plans and chooses the best plan using a cost estimation model
• Plan: specifies precisely how the query is to be executed. The nodes are operators, and every operator carries out one particular operation. The edges represent consumer-producer relationships of operators.
• Plan Refinement: this component transforms the plan into an executable plan. In DB2 this transformation involves the generation of an assembler-like code to evaluate expressions and predicates efficiently
23
Query evaluation planPJ A1
NLJ A2=B2
scan
temp
receivereceive
Site 0
Inxscan(A)
PJ A3
send
Scan(B)
SL C=cos
PJ B3
send
24
Query evaluation plan
• Fragment reducers: a set of unary operations which apply to the same fragment are collected into programs
• Binary operations: joins and unions
• Optimization graph: nodes represent reduced fragments, and joins (unions) are represented by edges (hypernodes)
A B
A2=B2
25
Query Optimization (1)
• Plan enumeration with Dynamic Programming
Input: SPJ query q on relations R1, ..., Rn
Output: A query plan for q
1. for i=1 to n do {
2. optPlan({Ri}) = accessPlans(Ri)
3. prunePlans(optPlan({Ri}))
4. }
5. for i=2 to n do {
6. for all S {R1, ..., Rn} such that |S| = i do {
7. optPlan(S) =
26
Query Optimization (2)
8. for all O S do {
9. optPlan(S) = optPlan(S)
joinPlans(optPlan(O), optPlan(S-O))
10. prunePlans(optPlan(S))
11. }
12. }
13. }
14. return optPlan({R1, ..., Rn})
Problem: alternative plans cannot be immediately pruned
27
Query Optimization (3)
• Optimization criteria:– Classic cost model (total time, total resource
consumption) – estimate the cost of every individual operator of the plan and then sum up these costs – this model is useful to estimate the overall throughput of a system
– Mean response time model – estimate the lowest response time of a query
28
Query Execution Techniques
• Row blocking – implementation of send and receive operators is based on TCP/IP, UDP protocols;
idea: ship tuples in a blockwise fashion
• Optimization of Multicasts: send data sequentially instead of sending data twice (NY Berlin Poznan)
• Joins with Horizontally Partitioned Data –
(A1 A2) JN B or (A1 JN B) (A2 JN B)
If A and B are both partitioned than we have more plans
• Semijoin and Bloojoin programs
29
Semijoin Programs
• Semijoin between R and S over two attributes A and B is defined as follows:
( R SJ A=B S) JN A=B S is equal R JN A=B S
1. Send PJ B (S) to site R at a cost
C0 + C1 * size(B) * val(B(S))
2. Compute semijoin on R at a null cost; Let R’= R SJ A=B S
3. Send R’ to site S at a cost
C0 + C1 * size(R) * card(R’)
4. Compute the join on site S at a null value
30
Reducers
• Semijoin programs can be regarded as reducers, i.e. Operations that can be applied to reduce the cardinality of their operands
• Let RED(Q, R) denote the set of reducer programs that can be built for a given relation R in a given query Q
• There is one reducer program, element of RED(Q, R), which reduces R more than all other programs – full reducer
• The problem : find all full reducers for the relations of a query (difficult task)
• Acyclic (tree queries) versus cyclic queries
31
Reducers• Is it possible to give a limitation to the length of the full
reducer?
• Tree queries – YES
The limitation on the length of the full reducer amounts to n-1, where n is the number of nodes of the tree
• Cyclic queries – NO
The limitation on the length of the ‘best’ reducer is linearly bound by the number of tuples of some relations of the query
• Best reducer does not mean full reducer
32
Example (1)
A B1 a2 b3 c
R
B Ca xb yc z
S
C Ax 2y 3z 4
T
S
R T
A=A
B=B C=C
The final result is empty relation; the length of the reducers is 3*(m-1), where m is the number of tuples
Cyclic query
33
Example (2)
A B1 a2 b3 e
R
B Ca xb yc z
S
C Dx 10p 20q 30
T
S
R TB=B C=C
The final result - one tuple (a, x)
Acyclic query
34
Testing the graph for cycles
• There are two cases in which cycles can be broken without changing the meaning of the query
1. In the cycle (R.A=S.B), (S.B=T.C), (T.C=R.A), in which R, S, T are relation names, and A, B, C are attributes, any one of the edges can be dropped, as any edge can be obtained from the remaining ones by transitivity.
2. In the cycle (R.A=S.B), (S.B=T.C), (T.C=R.D), we can substitute (R.A=R.D) for (T.C=R.D) because, by transitivity, T.C must equal R.A; the remaining graph contains two edges (R.S) and (S.T) and is acyclic, because an interrelation clause can be sabstituted by an intrarelation clause