on the optimization of recursive relational queries. - dig seminar · 2019. 3. 9. · the...

98
On the optimization of recursive relational queries. DIG Seminar Louis Jachiet Centre de Recherche en Informatique, Signal et Automatique de Lille 1

Upload: others

Post on 11-Nov-2020

2 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

On the optimization of recursive relational

queries.

DIG Seminar

Louis Jachiet

Centre de Recherche en Informatique, Signal et Automatique de Lille

1

Page 2: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

The relational algebra

Page 3: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Generalities

The relational algebra [Cod70]

• a set of base relations

the tables in SQL

• combined through operators

union, projection, filter, join, etc.

• operates on named tuples

2

Page 4: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Syntax

ϕ ::= term

| X relation variable

| |c → v | constant

| ∅ empty set

| ϕ1 ∪ ϕ2 union

| ϕ1 ./ ϕ2 join

| ϕ1 . ϕ2 antijoin

| σfilter (ϕ) filter

| ρba (ϕ) rename

| πP (ϕ) projection

Figure 1: Syntax of the relational algebra

3

Page 5: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Examples

T

from to

Lille Paris

Lille Saclay

Paris Grenoble

Paris Saclay

Saclay Grenoble

πto (T )

to

Paris

Saclay

Grenoble

4

Page 6: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Examples

T

from to

Lille Paris

Lille Saclay

Paris Grenoble

Paris Saclay

Saclay Grenoble

ρstepto (T )

from step

Lille Paris

Lille Saclay

Paris Grenoble

Paris Saclay

Saclay Grenoble

4

Page 7: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Examples

T

from to

Lille Paris

Lille Saclay

Paris Grenoble

Paris Saclay

Saclay Grenoble

ρstepto (T ) ./ ρstep

from (T )

from step to

Lille Paris Saclay

Lille Paris Grenoble

Lille Saclay Grenoble

Paris Saclay Grenoble

4

Page 8: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Examples

T

from to

Lille Paris

Lille Saclay

Paris Grenoble

Paris Saclay

Saclay Grenoble

σfrom=Lille (T )

from to

Lille Paris

Lille Saclay

4

Page 9: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Examples

T

from to

Lille Paris

Lille Saclay

Paris Grenoble

Paris Saclay

Saclay Grenoble

σfrom=Paris (T ) ∪ σfrom=Lille (T )

from to

Lille Paris

Lille Saclay

Paris Saclay

Paris Grenoble

4

Page 10: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Examples

T

from to

Lille Paris

Lille Saclay

Paris Grenoble

Paris Saclay

Saclay Grenoble

πto (T ) . σfrom=Lille (T )

to

Grenoble

4

Page 11: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Recursive relational algebra

Page 12: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Syntax

ϕ ::= term

| X relation variable

| |c → v | constant

| ∅ empty set

| ϕ1 ∪ ϕ2 union

| ϕ1 ./ ϕ2 join

| ϕ1 . ϕ2 antijoin

| σfilter (ϕ) filtering

| ρba (ϕ) rename

| πc1,...,cn (ϕ) projection

Figure 2: Syntax of our relational algebra5

Page 13: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Syntax

ϕ ::= term

| X relation variable

| |c → v | constant

| ∅ empty set

| ϕ1 ∪ ϕ2 union

| ϕ1 ./ ϕ2 join

| ϕ1 . ϕ2 antijoin

| σfilter (ϕ) filtering

| ρba (ϕ) rename

| π̃c (ϕ) anti-projection

Figure 2: Syntax of our relational algebra5

Page 14: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Syntax

ϕ ::= term

| X relation variable

| |c → v | constant

| ∅ empty set

| ϕ1 ∪ ϕ2 union

| ϕ1 ./ ϕ2 join

| ϕ1 . ϕ2 antijoin

| σfilter (ϕ) filtering

| ρba (ϕ) rename

| π̃c (ϕ) anti-projection

| βba (ϕ) duplication

Figure 2: Syntax of our relational algebra5

Page 15: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Syntax

ϕ ::= term

| X relation variable

| |c → v | constant

| ∅ empty set

| ϕ1 ∪ ϕ2 union

| ϕ1 ./ ϕ2 join

| ϕ1 . ϕ2 antijoin

| σfilter (ϕ) filtering

| ρba (ϕ) rename

| π̃c (ϕ) anti-projection

| βba (ϕ) duplication

| µ(X = ϕ) fixpoint

Figure 2: Syntax of our relational algebra5

Page 16: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Differences

Anti-projectionRemove a column

π̃d1 (. . . π̃dk(ϕ) . . . ) = πc1...cn (ϕ)

DuplicationCopy a column

βba (ϕ) = σa=b

(ϕ ./ ρb

a (ϕ))

FixpointsCompute the least fixpoint of a function S → ϕ[X/S ]

Jµ(X = ϕ)KV = limn→∞Un U0 = ∅Un+1 = Un ∪ JϕKV [X/Un]

6

Page 17: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Differences

Anti-projectionRemove a column

π̃d1 (. . . π̃dk(ϕ) . . . ) = πc1...cn (ϕ)

DuplicationCopy a column

βba (ϕ) = σa=b

(ϕ ./ ρb

a (ϕ))

FixpointsCompute the least fixpoint of a function S → ϕ[X/S ]

Jµ(X = ϕ)KV = limn→∞Un U0 = ∅Un+1 = Un ∪ JϕKV [X/Un]

6

Page 18: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Differences

Anti-projectionRemove a column

π̃d1 (. . . π̃dk(ϕ) . . . ) = πc1...cn (ϕ)

DuplicationCopy a column

βba (ϕ) = σa=b

(ϕ ./ ρb

a (ϕ))

FixpointsCompute the least fixpoint of a function S → ϕ[X/S ]

Jµ(X = ϕ)KV = limn→∞Un U0 = ∅Un+1 = Un ∪ JϕKV [X/Un]

6

Page 19: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Examples

T

from to

Lille Paris

Lille Saclay

Paris Grenoble

Paris Saclay

Saclay Grenoble

π̃from (T )

to

Paris

Saclay

Grenoble

7

Page 20: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Examples

T

from to

Lille Paris

Lille Saclay

Paris Grenoble

Paris Saclay

Saclay Grenoble

βtofrom (π̃to (T ))

from to

Lille Lille

Saclay Saclay

Paris Paris

7

Page 21: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Examples

T+ = µ(X = T ∪ T/X )

T

from to

Lille Paris

Paris Saclay

Saclay Lyon

Lyon Grenoble

from to

Lille Paris

Paris Saclay

Saclay Lyon

Lyon Grenoble

Lille Saclay

Paris Lyon

Saclay Grenoble

Lille Lyon

Paris Grenoble

Lille Grenoble

8

Page 22: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Examples

T+ = µ(X = T ∪ π̃s (ρsto (X ) ./ ρs

from (T )))

T

from to

Lille Paris

Paris Saclay

Saclay Lyon

Lyon Grenoble

from to

Lille Paris

Paris Saclay

Saclay Lyon

Lyon Grenoble

Lille Saclay

Paris Lyon

Saclay Grenoble

Lille Lyon

Paris Grenoble

Lille Grenoble

8

Page 23: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Examples

T+ = µ(X = T ∪ π̃s (ρsto (X ) ./ ρs

from (T )))

T

from to

Lille Paris

Paris Saclay

Saclay Lyon

Lyon Grenoble

from to

Lille Paris

Paris Saclay

Saclay Lyon

Lyon Grenoble

Lille Saclay

Paris Lyon

Saclay Grenoble

Lille Lyon

Paris Grenoble

Lille Grenoble

8

Page 24: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Examples

T+ = µ(X = T ∪ π̃s (ρsto (X ) ./ ρs

from (T )))

T

from to

Lille Paris

Paris Saclay

Saclay Lyon

Lyon Grenoble

from to

Lille Paris

Paris Saclay

Saclay Lyon

Lyon Grenoble

Lille Saclay

Paris Lyon

Saclay Grenoble

Lille Lyon

Paris Grenoble

Lille Grenoble

8

Page 25: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Examples

T+ = µ(X = T ∪ π̃s (ρsto (X ) ./ ρs

from (T )))

T

from to

Lille Paris

Paris Saclay

Saclay Lyon

Lyon Grenoble

from to

Lille Paris

Paris Saclay

Saclay Lyon

Lyon Grenoble

Lille Saclay

Paris Lyon

Saclay Grenoble

Lille Lyon

Paris Grenoble

Lille Grenoble8

Page 26: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Limitations

Limitations on fixpoints

• recursive variables must appear positively

No µ(X = R . X )

• no join between recursive terms

No µ(X = X on X )

• no mutually recursive fixpoints

No µ(X = µ(Y = X ∪ Y ))

→ corresponds to linear datalog!

→ superset of WITH RECURSIVE in SQL!

9

Page 27: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Limitations

Limitations on fixpoints

• recursive variables must appear positively

No µ(X = R . X )

• no join between recursive terms

No µ(X = X on X )

• no mutually recursive fixpoints

No µ(X = µ(Y = X ∪ Y ))

→ corresponds to linear datalog!

→ superset of WITH RECURSIVE in SQL!

9

Page 28: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Limitations

Limitations on fixpoints

• recursive variables must appear positively

No µ(X = R . X )

• no join between recursive terms

No µ(X = X on X )

• no mutually recursive fixpoints

No µ(X = µ(Y = X ∪ Y ))

→ corresponds to linear datalog!

→ superset of WITH RECURSIVE in SQL!

9

Page 29: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Limitations

Limitations on fixpoints

• recursive variables must appear positively

No µ(X = R . X )

• no join between recursive terms

No µ(X = X on X )

• no mutually recursive fixpoints

No µ(X = µ(Y = X ∪ Y ))

→ corresponds to linear datalog!

→ superset of WITH RECURSIVE in SQL!9

Page 30: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Performance of recursive queries

Page 31: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:Lille :TGV/:Bus∗ ?o

10

Page 32: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:Lille :TGV/:Bus∗ ?o

L

10

Page 33: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:Lille :TGV/:Bus∗ ?o

L

10

Page 34: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:Lille :TGV/:Bus∗ ?o

L

10

Page 35: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:Lille :TGV/:Bus∗ ?o

L

10

Page 36: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:Lille :TGV/:Bus∗ ?o

L

10

Page 37: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:Lille :TGV/:Bus∗ ?o

L

10

Page 38: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:Lille :TGV/:Bus∗ ?o

L

10

Page 39: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:Lille :TGV/:Bus∗ ?o

L

10

Page 40: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:Lille :TGV/:Bus∗ ?o

L

10

Page 41: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:Lille :TGV/:Bus∗ ?o

L

10

Page 42: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:Lille :TGV/:Bus∗ ?o

L

10

Page 43: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:Lille :TGV/:Bus∗ ?o

L

10

Page 44: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:Lille :TGV/:Bus∗ ?o

L

10

Page 45: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:Lille :TGV/:Bus∗ ?o

L

10

Page 46: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:Lille :TGV/:Bus∗ ?o

L

10

Page 47: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:L :TGV/:Bus∗ ?o

π̃?s (σ?s=:L (:TGV/µ(X = βos (AllNodes) ∪ X/:Bus)))

π̃?s (σ?s=:L (:TGV/µ(X = βos (AllNodes) ∪ :Bus/X )))

11

Page 48: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:L :TGV/:Bus∗ ?o

π̃?s (σ?s=:L (:TGV/µ(X = βos (AllNodes) ∪ X/:Bus)))

π̃?s (σ?s=:L (:TGV/µ(X = βos (AllNodes) ∪ :Bus/X )))

11

Page 49: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:L :TGV/:Bus∗ ?o

π̃?s (σ?s=:L (:TGV/µ(X = βos (AllNodes) ∪ X/:Bus)))

π̃?s (σ?s=:L (:TGV/µ(X = βos (AllNodes) ∪ :Bus/X )))

11

Page 50: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Benchmarking

102 103 104 105 10610−2

10−1

100

101

number of nodes n

Tim

e(s

)

PostgresSQLite

VirtuosoARQ1

ARQ2

DLV1

DLV2

Ramsdell2Ramsdell1

Vlog1Vlog2

Figure 3: Time for : N :TGV/:Bus∗ ?o on n nodes

12

Page 51: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Benchmarking

Logicblox

• Materialization of all intermediate predicate

• ... except for “on demand” predicate

• therefore manual optimization

13

Page 52: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Rewrite rules

Rewrite rules for fixpoints

• pushing filters?

σfilter (µ(X = ϕ))?= µ(X = σfilter (ϕ))

• pushing joins?

ψ ./ µ(X = ϕ)?= µ(X = ψ ./ ϕ)

• pushing antijoins?

µ(X = ϕ) . ψ?= µ(X = ϕ . ψ)

• pushing anti-projections?

π̃p (µ(X = ϕ))?= µ(X = π̃p (ϕ))

• combine fixpoints?

µ(X = ψ ∪ κ) ./ µ(X = ϕ ∪ ξ)?= µ(X = ψ ./ ϕ ∪ ξ ∪ κ)

14

Page 53: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Rewrite rules

Rewrite rules for fixpoints

• pushing filters?

σfilter (µ(X = ϕ))?= µ(X = σfilter (ϕ))

• pushing joins?

ψ ./ µ(X = ϕ)?= µ(X = ψ ./ ϕ)

• pushing antijoins?

µ(X = ϕ) . ψ?= µ(X = ϕ . ψ)

• pushing anti-projections?

π̃p (µ(X = ϕ))?= µ(X = π̃p (ϕ))

• combine fixpoints?

µ(X = ψ ∪ κ) ./ µ(X = ϕ ∪ ξ)?= µ(X = ψ ./ ϕ ∪ ξ ∪ κ)

14

Page 54: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Rewrite rules

Rewrite rules for fixpoints

• pushing filters?

σfilter (µ(X = ϕ))?= µ(X = σfilter (ϕ))

• pushing joins?

ψ ./ µ(X = ϕ)?= µ(X = ψ ./ ϕ)

• pushing antijoins?

µ(X = ϕ) . ψ?= µ(X = ϕ . ψ)

• pushing anti-projections?

π̃p (µ(X = ϕ))?= µ(X = π̃p (ϕ))

• combine fixpoints?

µ(X = ψ ∪ κ) ./ µ(X = ϕ ∪ ξ)?= µ(X = ψ ./ ϕ ∪ ξ ∪ κ)

14

Page 55: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Rewrite rules

Rewrite rules for fixpoints

• pushing filters?

σfilter (µ(X = ϕ))?= µ(X = σfilter (ϕ))

• pushing joins?

ψ ./ µ(X = ϕ)?= µ(X = ψ ./ ϕ)

• pushing antijoins?

µ(X = ϕ) . ψ?= µ(X = ϕ . ψ)

• pushing anti-projections?

π̃p (µ(X = ϕ))?= µ(X = π̃p (ϕ))

• combine fixpoints?

µ(X = ψ ∪ κ) ./ µ(X = ϕ ∪ ξ)?= µ(X = ψ ./ ϕ ∪ ξ ∪ κ)

14

Page 56: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Rewrite rules

Rewrite rules for fixpoints

• pushing filters?

σfilter (µ(X = ϕ))?= µ(X = σfilter (ϕ))

• pushing joins?

ψ ./ µ(X = ϕ)?= µ(X = ψ ./ ϕ)

• pushing antijoins?

µ(X = ϕ) . ψ?= µ(X = ϕ . ψ)

• pushing anti-projections?

π̃p (µ(X = ϕ))?= µ(X = π̃p (ϕ))

• combine fixpoints?

µ(X = ψ ∪ κ) ./ µ(X = ϕ ∪ ξ)?= µ(X = ψ ./ ϕ ∪ ξ ∪ κ)

14

Page 57: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Rewrite rules

Rewrite rules for fixpoints

• Reverse fixpoints?

• pushing filters?

σfilter (µ(X = ϕ))?= µ(X = σfilter (ϕ))

• pushing joins?

ψ ./ µ(X = ϕ)?= µ(X = ψ ./ ϕ)

• pushing antijoins?

µ(X = ϕ) . ψ?= µ(X = ϕ . ψ)

• pushing anti-projections?

π̃p (µ(X = ϕ))?= µ(X = π̃p (ϕ))

14

Page 58: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:L :TGV/:Bus∗ ?o

π̃?s (σ?s=:L (:TGV/µ(X = βos (AllNodes) ∪ X/:Bus)))

π̃?s (σ?s=:L (µ(X = :TGV/βos (AllNodes) ∪ X/:Bus)))

π̃?s (σ?s=:L (µ(X = :TGV ∪ X/:Bus)))

π̃?s (µ(X = σ?s=:L (:TGV) ∪ X/:Bus))

µ(X = π̃?s (σ?s=:L (:TGV)) ∪ X/:Bus)

15

Page 59: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:L :TGV/:Bus∗ ?o

π̃?s (σ?s=:L (:TGV/µ(X = βos (AllNodes) ∪ X/:Bus)))

π̃?s (σ?s=:L (µ(X = :TGV/βos (AllNodes) ∪ X/:Bus)))

π̃?s (σ?s=:L (µ(X = :TGV ∪ X/:Bus)))

π̃?s (µ(X = σ?s=:L (:TGV) ∪ X/:Bus))

µ(X = π̃?s (σ?s=:L (:TGV)) ∪ X/:Bus)

15

Page 60: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:L :TGV/:Bus∗ ?o

π̃?s (σ?s=:L (:TGV/µ(X = βos (AllNodes) ∪ X/:Bus)))

π̃?s (σ?s=:L (µ(X = :TGV/βos (AllNodes) ∪ X/:Bus)))

π̃?s (σ?s=:L (µ(X = :TGV ∪ X/:Bus)))

π̃?s (µ(X = σ?s=:L (:TGV) ∪ X/:Bus))

µ(X = π̃?s (σ?s=:L (:TGV)) ∪ X/:Bus)

15

Page 61: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:L :TGV/:Bus∗ ?o

π̃?s (σ?s=:L (:TGV/µ(X = βos (AllNodes) ∪ X/:Bus)))

π̃?s (σ?s=:L (µ(X = :TGV/βos (AllNodes) ∪ X/:Bus)))

π̃?s (σ?s=:L (µ(X = :TGV ∪ X/:Bus)))

π̃?s (µ(X = σ?s=:L (:TGV) ∪ X/:Bus))

µ(X = π̃?s (σ?s=:L (:TGV)) ∪ X/:Bus)

15

Page 62: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:L :TGV/:Bus∗ ?o

π̃?s (σ?s=:L (:TGV/µ(X = βos (AllNodes) ∪ X/:Bus)))

π̃?s (σ?s=:L (µ(X = :TGV/βos (AllNodes) ∪ X/:Bus)))

π̃?s (σ?s=:L (µ(X = :TGV ∪ X/:Bus)))

π̃?s (µ(X = σ?s=:L (:TGV) ∪ X/:Bus))

µ(X = π̃?s (σ?s=:L (:TGV)) ∪ X/:Bus)

15

Page 63: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

An example

:L :TGV/:Bus∗ ?o

π̃?s (σ?s=:L (:TGV/µ(X = βos (AllNodes) ∪ X/:Bus)))

π̃?s (σ?s=:L (µ(X = :TGV/βos (AllNodes) ∪ X/:Bus)))

π̃?s (σ?s=:L (µ(X = :TGV ∪ X/:Bus)))

π̃?s (µ(X = σ?s=:L (:TGV) ∪ X/:Bus))

µ(X = π̃?s (σ?s=:L (:TGV)) ∪ X/:Bus)

15

Page 64: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Other methods of optimization

Datalog?No combination of fixpoints

Automata based techniques?Step by Step and no conjunction

(a/b/c)+ vs ((a/b)/c)+ vs ((a/b/c))+

Special joins (RDF-3X ferari)?Compute efficiently A ./ (B)∗ but same problem...

WaveguideEfficient on a single RPQ but cannot optimize across RPQ.

16

Page 65: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Theoretical framework

Page 66: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Decomposition

Decomposed fixpointsGiven a fixpoint µ(X = ϕ) it can be rewritten to

µ(X = ϕcon ∪ ϕrec ) with:

• ϕcon constant, i.e. JϕconKV [X/∅] = JϕconKV [X/S]

• ϕrec recursive, i.e. JϕconKV [X/∅] = ∅

17

Page 67: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Lineage

Linearity of fixpointsGiven a fixpoint µ(X = ϕ):

JϕKV [X/S] = JϕKV [X/∅]⋃

w∈S

JϕKV [X/{w}]

LineageFor each m ∈ Ui+1 \ Ui we can find w ∈ Ui such that m ∈ f (w)

with f (w) = JϕKV [X/{w}] \ JϕKV [X/∅].

u u1f

w w1

w2

f

w3 w4f

ff

18

Page 68: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Lineage

Linearity of fixpointsGiven a fixpoint µ(X = ϕ):

JϕKV [X/S] = JϕKV [X/∅]⋃

w∈S

JϕKV [X/{w}]

LineageFor each m ∈ Ui+1 \ Ui we can find w ∈ Ui such that m ∈ f (w)

with f (w) = JϕKV [X/{w}] \ JϕKV [X/∅].

u u1f

w w1

w2

f

w3 w4f

ff

18

Page 69: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Examples

LYS/GRE

SAC/LYS SAC/GRE

PAR/SAC PAR/LYS PAR/GRE

LIL/PAR LIL/SAC LIL/LYO LIL/GRE

T+ = µ(X = T ∪ X/T )

T

from to

Lille Paris

Paris Saclay

Saclay Lyon

Lyon Grenoble 19

Page 70: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Invariant

How the elements of f (w) depend on w?

StabilizersFor each w , m ∈ f (w) and c ∈ stab(ϕ): m(c) = w(c).

u u1f

w w1

w2

f

w3 w4f

ff

σfilter (µ(X = ϕ)) = µ(X = σfilter (ϕ))

when filter operates on stab(ϕ)

20

Page 71: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Invariant

How the elements of f (w) depend on w?

StabilizersFor each w , m ∈ f (w) and c ∈ stab(ϕ): m(c) = w(c).

u u1f

w w1

w2

f

w3 w4f

ff

σfilter (µ(X = ϕ)) = µ(X = σfilter (ϕ))

when filter operates on stab(ϕ)

20

Page 72: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Invariant

How the elements of f (w) depend on w?

StabilizersFor each w , m ∈ f (w) and c ∈ stab(ϕ): m(c) = w(c).

u u1f

w w1

w2

f

w3 w4f

ff

σfilter (µ(X = ϕ)) = µ(X = σfilter (ϕ))

when filter operates on stab(ϕ)

20

Page 73: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Examples

LYS/GRE

SAC/LYS SAC/GRE

PAR/SAC PAR/LYS PAR/GRE

LIL/PAR LIL/SAC LIL/LYO LIL/GRE

σfrom=Lille

(T+)

= µ(X = σfrom=Lille (T ) ∪ X/T )

σto=Lille

(T+)6= µ(X = σfrom=Lille (T ) ∪ X/T )

21

Page 74: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Invariant

How the elements of f (w) depend on w?

Added columnsFor each c ∈ add(ϕ): f (w) ./ |c → v | = f (w ./ |c → v |)

u u1f

w w1

w2

f

w3 w4f

ff

ψ ./ µ(X = ϕ) = µ(X = ψ ./ ϕ)

when sort(ψ) ⊆ stab(ϕ)

and sort(ψ) ⊆ add(ϕ) ∪ sort(µ(X = ϕ))

22

Page 75: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Rewrite rules

Rewrite rules for fixpoints

• pushing filters

σfilter (µ(X = ϕ))?= µ(X = σfilter (ϕ))

• pushing joins

ψ ./ µ(X = ϕ)?= µ(X = ψ ./ ϕ)

• pushing antijoins

µ(X = ϕ) . ψ?= µ(X = ϕ . ψ)

• pushing anti-projections

π̃p (µ(X = ϕ))?= µ(X = π̃p (ϕ))

• combine fixpoints

µ(X = ψ ∪ κ) ./ µ(X = ϕ ∪ ξ)?= µ(X = ψ ./ ϕ ∪ ξ ∪ κ)

23

Page 76: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Streams

Page 77: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Streams

Streams are one-way communication channels

RS

end

24

Page 78: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Streams

Streams are one-way communication channels

RS

end

24

Page 79: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Streams

Streams are one-way communication channels

RS

end

24

Page 80: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Streams

Streams are one-way communication channels

RS

end

24

Page 81: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Streams

Streams are one-way communication channels

RS

end

24

Page 82: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Streams: a a good abstraction for iterative distributed execu-

tion

• no order of messages

• not necessarily a DAG

• fast single machine communication and slow inter-machine

communication

• partial typing of message content (for fast serialization)

25

Page 83: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Streams: a a good abstraction for iterative distributed execu-

tion

• no order of messages

• not necessarily a DAG

• fast single machine communication and slow inter-machine

communication

• partial typing of message content (for fast serialization)

25

Page 84: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Streams: a a good abstraction for iterative distributed execu-

tion

• no order of messages

• not necessarily a DAG

• fast single machine communication and slow inter-machine

communication

• partial typing of message content (for fast serialization)

25

Page 85: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Streams: a a good abstraction for iterative distributed execu-

tion

• no order of messages

• not necessarily a DAG

• fast single machine communication and slow inter-machine

communication

• partial typing of message content (for fast serialization)

25

Page 86: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Execution of µ-algebra terms with streams

start

Xn

. . .X1

I2

I1

ϕ out

26

Page 87: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Streams

Streams for µ(X = X/T ∪ T )

X/TT

X

∪∪ out

27

Page 88: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Benchmarking

Page 89: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Q2

102 103 104 105 106 107 10810−2

10−1

100

101

102

Number of nodes

Tim

e(s

)

PrototypeRamsdell

DLVVlog

PostgresMariaDB

Figure 4: ?a (P1+)/(P5+) ?b.28

Page 90: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Q3

102 103 104 105 106 107 10810−2

10−1

100

101

102

Number of nodes

Tim

e(s

)

PrototypeRamsdell

DLVVlog

PostgresMariaDB

Figure 5: ?a (P1+)/P2 ?b . ?b P3 + ?c .29

Page 91: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Q7

102 103 104 105 106 107 10810−2

10−1

100

101

102

Number of nodes

Tim

e(s

)

PrototypeRamsdell

DLVVlog

PostgresMariaDB

Figure 6: N0 P1/(P2+) ?a30

Page 92: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Q10

102 103 104 105 106 107 10810−2

10−1

100

101

102

Number of nodes

Tim

e(s

)

PrototypeRamsdell

DLVVlog

PostgresMariaDB

Figure 7: ?a (P4+)/(P5+)/(P3+) ?b31

Page 93: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Why recursive queries?

• Evaluate property paths

• Evaluate general recursive queries

• OBDA without rewriting nor materialization

32

Page 94: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

What am I doing now?

33

Page 95: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Questions?

34

Page 96: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Edgar F Codd.

A relational model of data for large shared data banks.

Communications of the ACM, 13(6):377–387, 1970.

Page 97: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Semantics

Page 98: On the optimization of recursive relational queries. - DIG Seminar · 2019. 3. 9. · The relational algebra [Cod70] a set of base relations the tables in SQL combined through operators

Semantics

Jϕ1 ./ ϕ2KV = {m1 + m2 | m1 ∈ Jϕ1KV ∧m2 ∈ Jϕ2KV ∧m1 ∼ m2}Jϕ1 ∪ ϕ2KV = Jϕ1KV ∪ Jϕ2KV

Jϕ1 . ϕ2KV = {m ∈ Jϕ1KV | ∀m′ ∈ Jϕ2KV ¬(m′ ∼ m)}Jπ̃a (ϕ)KV =

{{c → v ∈ m | c 6= a}

∣∣∣ m ∈ JϕKV

}JX KV = V (X )

Jβba (ϕ)KV =

{{c → v ∈ m | c 6= b} ∪ {b → v | a→ v ∈ m}∣∣∣ m ∈ JϕKV

}Jσfilter (ϕ)KV = {m | m ∈ JϕKV ∧ filter(m) = >}

Jµ(X = ϕ)KV = JX KV [X/U∞], U0 = ∅, Ui+1 = Ui ∪ JϕKV [X/Ui ]