2014 10-23-oopsla-i3ql

i3QL: Language-Integrated Live Data Views

Ralf Mitschke Sebastian Erdweg Mirko Köhler Mira Mezini Guido Salvaneschi

��

��

��

��

��

��

��

��

��

��

2

World airportsFlight data

2

World airportsFlight data

From each city, how many flights to Portland in 2014?

3

World airports

Flight data

FROM Airport a1, Airport a2, Flight fWHERE f.from = a1.id AND f.to = a2.id AND a2.code = ‘PDX’ !

!

!


3

World airports

Flight data


!

!

AND f.takeoff >= '2014-01-01' AND f.takeoff <= '2014-12-31'


3

World airports

Flight data


!

!


GROUP BY a1.city


3

World airports

Flight data


!

!

SELECT a1.city AS From COUNT(*) AS Flights


GROUP BY a1.city


4

World airports

Flight data

SELECT a1.city AS From COUNT(*) AS FlightsFROM Airport a1, Airport a2, Flight fWHERE f.from = a1.id AND f.to = a2.id AND a2.code = ‘PDX’ AND f.takeoff >= '2014-01-01' AND f.takeoff <= '2014-12-31'GROUP BY a1.city


5

World airports

Flight data


ID Code City1 AMS Amsterdam2 BOS Boston3 SFO San Francisco 4 NRT Tokyo5 PDX Portland

5

World airports

Flight data



From To Take off1 5 2014-01-01 10:00am1 5 2014-01-03 10:00am1 2 2014-01-01 1:20pm4 5 2013-12-31 4:00pm4 5 2014-01-01 4:00pm2 5 2014-12-31 5:05pm

5

World airports

Flight data

From Flights

Amsterdam 360

Boston 350

San Francisco 3467

Tokyo 349




6

World airports

Flight data

From Flights

Amsterdam 360

Boston 350

San Francisco 3467

Tokyo 349




6

World airports

Flight data

From Flights

Amsterdam 360

Boston 350

San Francisco 3467

Tokyo 349



From To Take off1 5 2014-01-01 10:00am1 5 2014-01-03 10:00am1 2 2014-01-01 1:20pm4 5 2013-12-31 4:00pm4 5 2014-01-01 4:00pm2 5 2014-12-31 5:05pm3 5 2014-09-14 8:15pm Additional flight

6

World airports

Flight data

From Flights

Amsterdam 360

Boston 350

San Francisco 3467

Tokyo 349



From To Take off1 5 2014-01-01 10:00am1 5 2014-01-03 10:00am1 2 2014-01-01 1:20pm4 5 2013-12-31 4:00pm4 5 2014-01-01 4:00pm2 5 2014-12-31 5:05pm3 5 2014-09-14 8:15pm Additional flight

+1

From To Take off1 5 2014-01-01 10:00am1 5 2014-01-03 10:00am1 2 2014-01-01 1:20pm4 5 2013-12-31 4:00pm4 5 2014-01-01 4:00pm2 5 2014-12-31 5:05pm3 5 2014-09-14 8:15pm

7

World airports

Flight data

From Flights

Amsterdam 360

Boston 350

San Francisco 3468

Tokyo 349



From To Take off1 5 2014-01-01 10:00am1 5 2014-01-03 10:00am1 2 2014-01-01 1:20pm4 5 2013-12-31 4:00pm4 5 2014-01-01 4:00pm2 5 2014-12-31 5:05pm3 5 2014-09-14 8:15pm2 5 2015-01-01 11:05am

7

World airports

Flight data

From Flights

Amsterdam 360

Boston 350

San Francisco 3468

Tokyo 349



Rescheduled

From To Take off1 5 2014-01-01 10:00am1 5 2014-01-03 10:00am1 2 2014-01-01 1:20pm4 5 2013-12-31 4:00pm4 5 2014-01-01 4:00pm2 5 2014-12-31 5:05pm3 5 2014-09-14 8:15pm2 5 2015-01-01 11:05am

7

World airports

Flight data

From Flights

Amsterdam 360

Boston 350

San Francisco 3468

Tokyo 349



-1

Rescheduled

8

ChangeUpdate

9

Maintain live data views

9

Maintain live data views

= Incremental computations

10

Incremental computations

at our fingertips

10

Incremental computations

at our fingertips(inside PL)

11

i3QL!integrated! ! ! ! !in-memory! ! ! ! !incremental! ! !

11


Scala-embedded DSL

11


Scala-embedded DSLcollections API

11


Scala-embedded DSLcollections APIbased on relational algebra

11



case class Airport(id: Int, code: String, city: String)case class Flight(from: Int, to: Int, takeoff: Date)

11




val airport: Table[Airport]val flight: Table[Flight]

11





airport += Airport(1, "AMS", "Amsterdam")airport += Airport(2, "BOS", "Boston")airport += Airport(5, "PDX", “Portland")

11





airport += Airport(1, "AMS", "Amsterdam")airport += Airport(2, "BOS", "Boston")airport += Airport(5, "PDX", “Portland")

flight += Flight(1, 5, new Date(2014, 1, 1, 10, 0))flight += Flight(1, 5, new Date(2014, 1, 3, 10, 0))flight += Flight(2, 5, new Date(2014, 12, 31, 17, 5))

12

i3QL queriesval airport: Table[Airport]val flight: Table[Flight]!

val q = ( !

!

!

!

!

!

!

!

!

!

!

)

FROM (airport, airport, flight)WHERE ((a1, a2, f) => f.from == a1.id AND f.to == a2.id AND a2.code == "PDX" !

)

12


val q = ( !

!

!

!

!

!

!

!

!

!

!

)


)

a2.code == "PDX" AND f.takeoff >= new Date(2014, 01, 01) AND f.takeoff < new Date(2015, 01, 01)

12


val q = ( !

!

!

!

!

!

!

!

!

!

!

)


)

a2.code == "PDX" AND f.takeoff >= new Date(2014, 01, 01) AND f.takeoff < new Date(2015, 01, 01) GROUP BY ((a1: Rep[Airport], a2: Rep[Airport], f: Rep[Flight]) => a1.city)

12


val q = ( !

!

!

!

!

!

!

!

!

!

!

)


)

a2.code == "PDX" AND f.takeoff >= new Date(2014, 01, 01) AND f.takeoff < new Date(2015, 01, 01) GROUP BY ((a1: Rep[Airport], a2: Rep[Airport], f: Rep[Flight]) => a1.city)

SELECT ((city: Rep[String]) => city, COUNT(*))

13

i3QL queriesval airport: Table[Airport] val flight: Table[Flight] val q = SELECT … FROM … WHERE …

13


val result = q.asMaterialized

13


val result = q.asMaterialized

From Flights

Amsterdam 360

Boston 350

San Francisco 3467

Tokyo 349

14

i3QL queriesval airport: Table[Airport] val flight: Table[Flight] !

val q = SELECT … FROM … WHERE … !

val result = q.asMaterialized !

flight += Flight(3, 5, new Date(2014, 9, 14, 20, 15))

From Flights

Amsterdam 360

Boston 350

San Francisco 3467

Tokyo 349

14




flight += Flight(3, 5, new Date(2014, 9, 14, 20, 15))

From Flights

Amsterdam 360

Boston 350

San Francisco 3467

Tokyo 349

+1

15




flight += Flight(3, 5, new Date(2014, 9, 14, 20, 15))flight ~= Flight(2, 5, new Date(2014, 12, 31, 17, 5)) -> Flight(2, 5, new Date(2015, 1, 1, 11, 5))

From Flights

Amsterdam 360

Boston 350

San Francisco 3468

Tokyo 349

15




flight += Flight(3, 5, new Date(2014, 9, 14, 20, 15))flight ~= Flight(2, 5, new Date(2014, 12, 31, 17, 5)) -> Flight(2, 5, new Date(2015, 1, 1, 11, 5))

From Flights

Amsterdam 360

Boston 350

San Francisco 3468

Tokyo 349

-1

16

i3QL query execution

16


1.Translate i3QL query to relational-algebra tree

16



2.Use incremental relational-algebra operatorsthat propagate change events

17

×

airport flight

17

×

airport flight

σ f.from == a1.idf.to == a2.ida2.code == “PDX”f.takeoff >= new Date(2014, 01, 01)f.takeoff < new Date(2015, 01, 01)

17

×

airport flight


γ GROUP a1.cityCOUNT(*)

17

×

airport flight



add((3, 5, new Date(2014, 9, 14, 20, 15)))

17

×

airport flight



add((3, 5, new Date(2014, 9, 14, 20, 15)))

for (a1, a2 <- airport) add(a1, a2, (3, 5, new Date(2014, 9, 14, 20, 15)))

17

×

airport flight



add((3, 5, new Date(2014, 9, 14, 20, 15)))


add((3, “SFO”), (5, “PDX”), (3, 5, new Date(2014, 9, 14, 20, 15)))

17

×

airport flight



add((3, 5, new Date(2014, 9, 14, 20, 15)))


add((3, “SFO”), (5, “PDX”), (3, 5, new Date(2014, 9, 14, 20, 15)))

update((“San Francisco”, 3467) -> (“San Francisco”, 3468))

18



!


18



!


} based on LMSby Rompf et al.

1.

2.Optimize relational-algebra tree !! - algebraic laws!! - automatic indexing !! - partial evaluation !! - common subquery elimination

19

×

airport flight

σf.from == a1.idf.to == a2.id

a2.code == “PDX”f.takeoff >= new Date(2014, 01, 01)f.takeoff < new Date(2015, 01, 01)

γGROUP a1.cityCOUNT(*)

optimize

19

×

airport flight




airport flight

optimize

σcode==“PDX” σ takeoff >= new Date(2014, 01, 01)takeoff < new Date(2015, 01, 01)

19

×

airport flight




airport flight

optimize

σcode==“PDX” σ takeoff >= new Date(2014, 01, 01)takeoff < new Date(2015, 01, 01)

⋈ f.from == a1.idf.to == a2.id

automatic indexing

19

×

airport flight




airport flight

optimize

σcode==“PDX”


σ takeoff >= new Date(2014, 01, 01)takeoff < new Date(2015, 01, 01)

⋈ f.from == a1.idf.to == a2.id

automatic indexing

20

i3QL operatorsOperator Use

Selection SELECT ... FROM t1 WHERE pred(t1.f1)

Projection SELECT f1 FROM t1

Unnesting SELECT ... FROM UNNEST (t1, .f1)

Aggregation SELECT ... FROM t1 GROUP BY t1.f1

Dupl. elimination SELECT DISTINCT f1 FROM t1

Recursion WITH RECURSIVE(...)

Cartesian product SELECT ... FROM (t1, t2)

Join SELECT ... FROM (t1, t2)WHERE t1.f1 === t2.f2

Semi-join SELECT ... FROM t1 WHERE EXISTS(SELECT ... FROM t2WHERE t1.f1 === t2.f2)

Anti semi-join SELECT ... FROM t1 WHERENOT EXISTS(SELECT ... FROM t2WHERE t1.f1 === t2.f2)

Difference SELECT ... FROM t1EXCEPT SELECT ... FROM t2

Union SELECT ... FROM t1UNION SELECT ... FROM t2

Union all SELECT ... FROM t1UNION ALL SELECT ... FROM t2

Intersection SELECT ... FROM t1INTERSECT SELECT ... FROM t2

Table 1: i3QL operators.

The basis of the change propagation is the Observer inter-face depicted in Figure 3. The interface declares three atomicchange events for addition, removal and update.

1 trait Observer[V] {2 def added(v: V)3 def removed(v: V)4 def updated(oldV: V, newV: V)5 }

Figure 3: The observer interface for change propagation.

Every i3QL operator implements the Observer trait and acorresponding Observable trait. The implementation of theObserver defines incremental semantics of an operator. Anoperator registers itself as an observer of its predecessors inthe graph and receives atomic change events through any ofthe three methods of Observer.

Selection Operator To illustrate the implementation of op-erators, we show the implementation of the selection oper-ator in Figure 4. The selection operator observes an under-lying relation containing elements of type V and filters theelements using a predicate function. Each operator has a type

for resulting elements, defined by the type argument passedto Observable. In case of selections,V is the input and outputtype of the operator. The change propagation of selection iscomparatively simple: Selection only propagates a changeto its observers if the involved value satisfies the predicate.Finally, selection registers itself as an observer of the under-lying relation.

1 class Selection[V](rel: Relation[V], predicate: V => Boolean)2 extends Observer[V] with Observable[V] {3 def added(v: V) { if(predicate(v)) notify added(v) }4 def removed(v: V) { if(predicate(v)) notify removed(v) }5 def updated(oldV: V, newV: V) {6 if(predicate(oldV)) {7 if(predicate(newV)) notify updated(oldV, newV)8 else notify removed(oldV) }9 else if(predicate(newV)) notify added(newV) }

10 rel addObserver this11 }

Figure 4: The incremental selection operator.

Join Operator The join operator is a binary operator thatcorrelates elements of a left-hand and right-hand side rela-tion. The results are pairs of elements from the two rela-tions. Instead of a binary predicate expressing the join con-dition, our join operator uses two functions that map ele-ments to key values; the operator joins two elements if theyare mapped to the same key. For example, in the join exam-ple shown in Sec. 2, we compared the properties firstName

and lastName of elements from two relations. i3QL automat-ically translates this condition into two key functions of theform (s:Student) => (s.firstName, s.lastName), where the key isa tuple of all values that have to be equal.

1 class Join[V1, V2, Key](2 left: Relation[V1], right: Relation[V2],3 leftKeyFun: V1 => Key, rightKeyFun: V2 => Key)4 extends Observable[(V1,V2)] {5 val leftIndex : Map[Key, List[V1]] = ...6 val rightIndex : Map[Key, List[V2]] = ...7 object LeftObserver extends Observer[V1] {8 def added(v: V1) {9 val key = leftKeyFun(v)

10 rightIndex.get(key) match {11 case Some(list) =>12 list.foreach( u => notify added ((u,v)) )13 case => // no changes propagated14 }}...}15 object RightObserver extends Observer[V2] { ... }16 }

Figure 5: Excerpt of the incremental join operator.

Figure 5 shows an excerpt of the implementation of the in-cremental join operator. The join—and any binary operatorin general—observes the input relations via two differentlytyped observers (line 7 and 15). To correlate elements, a join

Operator Use































20
































Operator Use































Support recursion WITH RECURSIVE (init UNION ALL query)

20
































Operator Use































Support recursion WITH RECURSIVE (init UNION ALL query)Extensible ! Add new operators! Add new optimizations

20
































Operator Use































Support recursion WITH RECURSIVE (init UNION ALL query)Extensible ! Add new operators! Add new optimizations

Use regular Scala code for nonincremental subcomputations

21

Performance evaluation

22

Implemented 15 FindBugsJVM bytecode analyses in i3QL

UpdateChange

22


Test data: Vespucci revision history! 381 class files initially! 242 changes (add, remove, update)! 1 change affects 1 to 312 class filesUpdateChange

22


Test data: Vespucci revision history! 381 class files initially! 242 changes (add, remove, update)! 1 change affects 1 to 312 class files

For each analysis! Analyze initial 381 class files! Replay changes in order

UpdateChange

23

0 50 100 150 200 250 300

050

100

150

200

250

Change size (in number of affected classes)

Tim

e (m

s)

(a) Update time per change.

0 50 100 150 200 250 300

−100

−50

050

100


Mem

ory

chan

ge (K

iB)

(b) Variation in memory consumption per change.

Figure 13: Mean time and memory consumption per changefor all 15 Findbugs analyses implemented with i3QL.

ory consumption averaged over all 15 analyses (the standarddeviation was low). The benchmarks have been performed 5times for each analysis with an additional 4 warmup itera-tions. The end result for each analysis is the average of those5 measurements.

Figure 13a shows the time it takes to update the analysisresult for each change, that is all classes that are changed si-multaneously. The x-axis shows the size of a change and they-axis shows the time that is needed to update the analysisresult. We ordered data points according to the change size.The plot shows that the time it takes to update the overallanalysis result is roughly linear in the change size; it doesnot seem to depend on the history of changes already pro-cessed or on the size of the initial revision.

A comparison with the non-incremental analyses im-plemented in FindBugs unsurprisingly shows an order-of-magnitude improvement in run-time performance. Runningan analysis in FindBugs for the initial revision takes 534 mson average, whereas i3QL only takes 129 ms on average.Since the FindBugs implementation will take roughly the

same time for each revision, checking all revisions amountsto about 130 seconds, whereas i3QL only requires little morethan 2 seconds. This means that on our test data i3QL pro-vides a speedup of 65x compared to the non-incrementalFindBugs analyses.

Figure 13b shows the variation in memory for eachchange. Like in the graphic above, the x-axis displays thesize of one change. The y-axis shows how much more (orless) memory is needed after the change has been processed.We run the garbage collector after the replay and propaga-tion of each change to get accurate measurements. The plotshows that the memory impact of a change is moderate (atmost 120 KiB for a change affecting more than 250 classes),which means that auxiliary data stored to enable incremen-talization does not have a significant effect. For comparison,the class files of the initial revision account for 1.39 MiB.

Effect of Optimizations. To measure the effect of opti-mizations in our implementation, we run the analyses in twoconfigurations of i3QL: with and without the optimizationspresented in Section 4. The results are shown in Table 2. Therows of the table are indexed by the analyses (A through O).The execution times of the analyses with i3QL optimizationsturned off respectively on are shown in the first respectivelythe second column of the table. The last column shows thespeedup achieved by turning optimization on.

By comparing the results, we observe that in 6 analy-ses, the optimizations improved performance significantly(> 20%), most prominently for analyses A, C, D, and E.In the other analyses, the effect of optimization is less sig-nificant. In case of analysis B, the optimizations even hada small negative effect. The reason for such a difference istwofold.

First, analyses A, C, D, E, F, G, K, N contain queries overthe byte-code instruction of the codebase—a much largertable than the table of method declarations, for example.Thus, the optimizations have a greater effect. Second, A,C and D use equi-joins, which further amplifies this effect.Equi-joins are only created with optimizations; without op-timizations those relations are simply cross products and se-lections. Equi-joins are very benefiting for large relations(e.g. instructions), since the join operator hashes its values—opposed to cross product and selection, which have to iter-ate over all elements in order to find equal elements. Thedifference between analysis A and C, D is that analysis Ajoins two large instruction relations while C and D join aninstruction relation with other relations that are not as large.

Summary. Our performance evaluation shows that i3QLqueries run an order of magnitude faster than their non-incremental counterparts, without requiring much memory.The run time for a change is linear in the change size andindependent of size of the initial revision of the input. Theevaluation clearly shows that optimizations bring significantperformance improvements; especially, they ensure that theanalyses execution time scales well for big sets of input data.

Update time per change

Mean update time for all 15 analyses

23

0 50 100 150 200 250 300

050

100

150

200

250


Tim

e (m

s)


0 50 100 150 200 250 300

−100

−50

050

100


Mem

ory

chan

ge (K

iB)













Update time ~ linear in change size!! ! Small change => small effect! ! Larger change => larger effect


23

0 50 100 150 200 250 300

050

100

150

200

250


Tim

e (m

s)


0 50 100 150 200 250 300

−100

−50

050

100


Mem

ory

chan

ge (K

iB)













Update time ~ linear in change size!! ! Small change => small effect! ! Larger change => larger effect


Comparison incremental vs nonincrementalFindBugs

i3QL sequentiali3QL incremental

1 10 100time (seconds), log-scale

130s31s

2s

24

Effect of optimizationstim

e (s

econ

ds),

log-

scal

e

1

10

100

1000

A B C D E F G H I J K L M N O

i3QL without optimizations i3QL with optimizations

24


e (s

econ

ds),

log-

scal

e

1

10

100

1000



Speedup up to 575xSlowdown of 4% in one case

24


e (s

econ

ds),

log-

scal

e

1

10

100

1000



Speedup up to 575xSlowdown of 4% in one case

When optimizations apply, huge effect

25

In the paper

• Detailed description of incremental operators

25

In the paper


• Case study:Complete definition of incremental parser

25

In the paper



• Evaluation of memory consumption

25

In the paper



• Evaluation of memory consumption

• Detailed discussion of related work!- embedded query languages!- in-memory collections!- constraints and functional dependencies!- view maintenance in DB systems!- optimization techniques

i3QL: Language-Integrated Live Data Views

��

��

��

��

��

��

��

��

��

��

integrated!! ! ! !in-memory! ! ! ! !incremental!! ! high-performance applicable

Scala-embedded DSL!collections API!based on relational algebrausing algebraic laws and partial evaluation used for FindBugs analyses on JVM byte code

https://github.com/seba--/i3QL/

https://github.com/seba--/i3QL/

2014 10-23-oopsla-i3ql

Software