jinq: linq-style queries in java 8 - oracle.com · linq syntax •use lambdas as operations over...
TRANSCRIPT
Jinq:LINQ-style Queries in Java 8
JVM Language Summit 2015
Ming-Yee Iu
1
Why are Databases Interesting to Language Designers?
•Databases are amazing• Interesting language issues• Query languages are DSLs
FROM Product p
WHERE p.price < 100
SELECT p.name
2
Database
Embedding Database DSLs in Java 8
•Many ways to embed query language DSLs into Java• Problems with corner cases
• dynamically generated queries
• complicated expressions
• subqueries
3
@Query(query=“FROM Product p ” +“WHERE p.price < 100 ” +“SELECT p.name”)
List<String> findCheapProducts();
FROM Product pWHERE p.price < 100SELECT p.name
LINQ-Style Queries
• LINQ• Microsoft’s system for database queries in C#
•Express queries using lambdas
4
FROM Product pWHERE p.price < 100SELECT p.name
db.products().where(p -> p.price < 100).select(p -> p.name)
• LINQ-style queries for Java• Translates Java code to SQL queries
•No compiler or VM changes
What is Jinq?
5
FROM Product pWHERE p.price < 100SELECT p.name
db.products().where(p -> p.price < 100).select(p -> p.name)
Why Should I Care?
•Databases are important• Rails helped make Ruby popular• Programmers love Microsoft’s LINQ database queries
•Good example of metaprogramming on Java• Jinq shows how far you can push existing facilities• Practical system built on runtime bytecode analysis
6
Syntax and SemanticsTranslation
Complications
7
LINQ Syntax
•Use lambdas as operations over collections• Automatic Java type checking and syntax checking• Uses familiar semantics
8
FROM Product p db.products()
WHERE p.price < 100 .where(p -> p.price < 100)
SELECT p.name .select(p -> p.name)
LINQ Syntax
•Use lambdas as operations over collections• Automatic Java type checking and syntax checking• Uses familiar semantics
9
FROM Product p db.products()
WHERE p.price < 100 .where(p -> p.price < 100)
SELECT p.name .select(p -> p.name)
filter
LINQ Syntax
•Use lambdas as operations over collections• Automatic Java type checking and syntax checking• Uses familiar semantics
10
FROM Product p db.products()
WHERE p.price < 100 .where(p -> p.price < 100)
SELECT p.name .select(p -> p.name)map
filter
Conceptual Model: LINQ-Style Queries
11
db
Conceptual Model: LINQ-Style Queries
12
db .products()
Conceptual Model: LINQ-Style Queries
13
CPU$200
RAM$80
HD$90
db .products()
Conceptual Model: LINQ-Style Queries
14
CPU$200
RAM$80
HD$90
db .products()
.where(p -> p.price < 100)
Conceptual Model: LINQ-Style Queries
15
CPU$200
RAM$80
HD$90
CPU$200
RAM$80
HD$90
db .products()
.where(p -> p.price < 100)
Conceptual Model: LINQ-Style Queries
16
CPU$200
RAM$80
HD$90
CPU$200
RAM$80
HD$90
db .products()
.where(p -> p.price < 100) .select(p -> p.name)
Conceptual Model: LINQ-Style Queries
17
CPU$200
RAM$80
HD$90
CPU$200
RAM$80
HD$90 $80 $90
RAM HD
db .products()
.where(p -> p.price < 100) .select(p -> p.name)
In Reality: Lazy Evaluation of DB Queries
18
db
In Reality: Lazy Evaluation of DB Queries
19
db .products()
In Reality: Lazy Evaluation of DB Queries
20
db .products()
FROMSELECT
Products pp
In Reality: Lazy Evaluation of DB Queries
21
db .products()
.where(p -> p.price < 100)
FROMSELECT
Products pp
In Reality: Lazy Evaluation of DB Queries
22
db .products()
.where(p -> p.price < 100)
FROMSELECT
Products pp
FROMWHERESELECT
Products pp.price < 100p
In Reality: Lazy Evaluation of DB Queries
23
db .products()
.where(p -> p.price < 100) .select(p -> p.name)
FROMSELECT
Products pp
FROMWHERESELECT
Products pp.price < 100p
In Reality: Lazy Evaluation of DB Queries
24
db .products()
.where(p -> p.price < 100) .select(p -> p.name)
FROMSELECT
Products pp
FROMWHERESELECT
Products pp.price < 100p
FROMWHERESELECT
Products pp.price < 100p.name
In Reality: Lazy Evaluation of DB Queries
25
.get(0)
db .products()
.where(p -> p.price < 100) .select(p -> p.name)
FROMSELECT
Products pp
FROMWHERESELECT
Products pp.price < 100p
FROMWHERESELECT
Products pp.price < 100p.name
In Reality: Lazy Evaluation of DB Queries
26
.get(0)
db .products()
.where(p -> p.price < 100) .select(p -> p.name)
FROMSELECT
Products pp
FROMWHERESELECT
Products pp.price < 100p
FROMWHERESELECT
Products pp.price < 100p.name
In Reality: Lazy Evaluation of DB Queries
27
.get(0)
RAM HD
db .products()
.where(p -> p.price < 100) .select(p -> p.name)
FROMSELECT
Products pp
FROMWHERESELECT
Products pp.price < 100p
FROMWHERESELECT
Products pp.price < 100p.name
In Reality: Lazy Evaluation of DB Queries
28
.get(0)
RAM HD
db .products()
.where(p -> p.price < 100) .select(p -> p.name)
FROMSELECT
Products pp
FROMWHERESELECT
Products pp.price < 100p
FROMWHERESELECT
Products pp.price < 100p.name
?How are lambdas
translated to queries?
Syntax and Semantics
TranslationComplications
29
How Does This All Work?
•Use bytecode analysis• Bytecode format is fairly static
•But it’s tricky• Different compilers produce slightly different output• How to translate imperative code to declarative code?
• Use symbolic execution
30
Same Code, Different “Bytecode”
31
Java Codedb.products()
.where( p -> p.getPrice() < 100)
Same Code, Different “Bytecode”
32
Java Codedb.products()
.where( p -> p.getPrice() < 100)
Extract bytecode of lambda in where() at
runtime
Same Code, Different “Bytecode”
33
Java Codedb.products()
.where( p -> p.getPrice() < 100)
Compiled Code 1a = 100b = p.getPrice()p = b < areturn p
Compiled Code 2a = 100b = p.getPrice()if (b < a) goto True:return 0True:return 1
Extract bytecode of lambda in where() at
runtime
Same Code, Different “Bytecode”
34
Java Codedb.products()
.where( p -> p.getPrice() < 100)
Compiled Code 1a = 100b = p.getPrice()p = b < areturn p
Compiled Code 2a = 100b = p.getPrice()if (b < a) goto True:return 0True:return 1
Extract bytecode of lambda in where() at
runtime
What does the compiled code
do?
Understand Code by Executing It
35
Compiled Code 1a = 100b = p.getPrice()p = b < areturn p
Understand Code by Executing It
36
Compiled Code 1a = 100b = p.getPrice()p = b < areturn p
p = CPU, $110
Understand Code by Executing It
37
Compiled Code 1a = 100b = p.getPrice()p = b < areturn p
p = CPU, $110
Side Effects
Understand Code by Executing It
38
function(Product p):a = 100b = p.getPrice()p = b < areturn p
p = CPU, $110
Side Effects
Understand Code by Executing It
39
function(Product p):a = 100b = p.getPrice()p = b < areturn p
p = CPU, $110
p = CPU, $110
Side Effects
Understand Code by Executing It
40
function(Product p):a = 100b = p.getPrice()p = b < areturn p
p = CPU, $110
p = CPU, $110
a = 100
Side Effects
Understand Code by Executing It
41
function(Product p):a = 100b = p.getPrice()p = b < areturn p
p = CPU, $110
p = CPU, $110
a = 100
b = 110
Side Effects
Understand Code by Executing It
42
function(Product p):a = 100b = p.getPrice()p = b < areturn p
p = CPU, $110
a = 100
b = 110
p = false
Side Effects
Understand Code by Executing It
43
function(Product p):a = 100b = p.getPrice()p = b < areturn p
p = CPU, $110
a = 100
b = 110
returns false
p = false
Side Effects
Understand Code by Executing It
44
function(Product p):a = 100b = p.getPrice()p = b < areturn p
p = CPU, $110
a = 100
b = 110
returns false
p = false
Side Effects
Execute Code Symbolically
45
function(Product p):a = 100b = p.getPrice()p = b < areturn p
p = input
Side Effects
Execute Code Symbolically
46
function(Product p):a = 100b = p.getPrice()p = b < areturn p
p = input
p = input
Side Effects
Execute Code Symbolically
47
function(Product p):a = 100b = p.getPrice()p = b < areturn p
p = input
p = input
a = 100
Side Effects
Execute Code Symbolically
48
function(Product p):a = 100b = p.getPrice()p = b < areturn p
p = input
p = input
a = 100
b = input.getPrice()
Side Effects
Execute Code Symbolically
49
function(Product p):a = 100b = p.getPrice()p = b < areturn p
p = input
a = 100
b = input.getPrice()
p = input.getPrice() < 100
Side Effects
Execute Code Symbolically
50
function(Product p):a = 100b = p.getPrice()p = b < areturn p
p = input
a = 100
b = input.getPrice()
returns input.getPrice() < 100
p = input.getPrice() < 100
Side Effects
Execute Code Symbolically
51
function(Product p):a = 100b = p.getPrice()p = b < areturn p
p = input
a = 100
b = input.getPrice()
returns input.getPrice() < 100
p = input.getPrice() < 100
Side Effects
Execute Code Symbolically
52
function(Product p):a = 100b = p.getPrice()p = b < areturn p
p = input
a = 100
b = input.getPrice()
returns input.getPrice() < 100
p = input.getPrice() < 100
Side Effects
Execute Code Symbolically
53
function(Product p):a = 100b = p.getPrice()p = b < areturn p
p = input
a = 100
b = input.getPrice()
returns input.getPrice() < 100
p = input.getPrice() < 100returns
input.getPrice() < 100
Query Generation
54
db.products()
.where( p -> p.getPrice() < 100 )
Query Generation
55
db.products()
.where( p -> p.getPrice() < 100 )
FROM Product P
SELECT P
Generated Query
Query Generation
56
db.products()
.where( p -> p.getPrice() < 100 )
FROM Product P
SELECT P
Generated Query
Query Generation
57
db.products()
.where( p -> p.getPrice() < 100 )
FROM Product P
SELECT P
Generated Query
returns input.getPrice() < 100
Query Generation
58
db.products()
.where( p -> p.getPrice() < 100 )
FROM Product P
WHERE P.Price < 100
SELECT P
Generated Query
returns input.getPrice() < 100
Query Generation
59
db.products()
.where( p -> p.getPrice() < 100 )
FROM Product P
WHERE P.Price < 100
SELECT P
Generated Query
returns input.getPrice() < 100
Same Code, Different “Bytecode”
60
Java Codedb.products()
.where( p -> p.getPrice() < 100)
Compiled Code 1a = 100b = p.getPrice()p = b < areturn d
Compiled Code 2a = 100b = p.getPrice()if (b < a) goto True:return 0True:return 1
Same Code, Different “Bytecode”
61
Java Codedb.products()
.where( p -> p.getPrice() < 100)
Compiled Code 2a = 100b = p.getPrice()if (b < a) goto True:return 0True:return 1
Control Flow Graph
62
Compiled Code 2a = 100b = p.getPrice()if (b < a) goto True:return 0True:return 1
true false
b = p.getPrice()
if (b < a)
a = 100
return 1 return 0
true false
b = p.getPrice()
if (b < a)
a = 100
return 1 return 0
Paths through Control Flow Graph
63
true false
b = p.getPrice()
if (b < a)
a = 100
return 1 return 0
Paths through Control Flow Graph
64
true
b = p.getPrice()
if (b < a)
a = 100
return 1
true false
b = p.getPrice()
if (b < a)
a = 100
return 1 return 0
Paths through Control Flow Graph
65
true
b = p.getPrice()
if (b < a)
a = 100
return 1
false
b = p.getPrice()
if (b < a)
a = 100
return 0
Preconditions
Side Effects
Execute Code Symbolically in Each Path
66
p = input
true
b = p.getPrice()
if (b < a)
a = 100
return 1
Preconditions
Side Effects
Execute Code Symbolically in Each Path
67
p = input
p = input
true
b = p.getPrice()
if (b < a)
a = 100
return 1
Preconditions
Side Effects
Execute Code Symbolically in Each Path
68
p = input
p = input
a = 100
true
b = p.getPrice()
if (b < a)
a = 100
return 1
Preconditions
Side Effects
Execute Code Symbolically in Each Path
69
p = input
p = input
a = 100
b = input.getPrice()
true
b = p.getPrice()
if (b < a)
a = 100
return 1
Preconditions
Side Effects
Execute Code Symbolically in Each Path
70
p = input
p = input
a = 100
b = input.getPrice()
true
b = p.getPrice()
if (b < a)
a = 100
return 1
input.getPrice() < 100
Preconditions
Side Effects
Execute Code Symbolically in Each Path
71
p = input
p = input
a = 100
b = input.getPrice()
returns 1
true
b = p.getPrice()
if (b < a)
a = 100
return 1
input.getPrice() < 100
Preconditions
Side Effects
Execute Code Symbolically in Each Path
72
p = input
p = input
a = 100
b = input.getPrice()
returns 1
true
b = p.getPrice()
if (b < a)
a = 100
return 1
input.getPrice() < 100
Preconditions
Side Effects
Execute Code Symbolically in Each Path
73
p = input
p = input
a = 100
b = input.getPrice()
returns 1
true
b = p.getPrice()
if (b < a)
a = 100
return 1
input.getPrice() < 100
Preconditions
Side Effects
Execute Code Symbolically in Each Path
74
p = input
p = input
a = 100
b = input.getPrice()
returns 1
true
b = p.getPrice()
if (b < a)
a = 100
return 1
input.getPrice() < 100
if input.getPrice() < 100:return true
true false
b = p.getPrice()
if (b < a)
a = 100
return 1 return 0
Analyze Both Paths
75
true false
b = p.getPrice()
if (b < a)
a = 100
return 1 return 0
Analyze Both Paths
76
true false
b = p.getPrice()
if (b < a)
a = 100
return 1 return 0
Analyze Both Paths
77
true
b = p.getPrice()
if (b < a)
a = 100
return 1
false
b = p.getPrice()
if (b < a)
a = 100
return 0
true false
b = p.getPrice()
if (b < a)
a = 100
return 1 return 0
Analyze Both Paths
78
true
b = p.getPrice()
if (b < a)
a = 100
return 1
false
b = p.getPrice()
if (b < a)
a = 100
return 0
if input.getPrice() < 100:return true
true false
b = p.getPrice()
if (b < a)
a = 100
return 1 return 0
Analyze Both Paths
79
true
b = p.getPrice()
if (b < a)
a = 100
return 1
false
b = p.getPrice()
if (b < a)
a = 100
return 0
if input.getPrice() < 100:return true
if input.getPrice() >= 100:return false
Merge Analysis of Paths
80
true
b = p.getPrice()
if (b < a)
a = 100
return 1
false
b = p.getPrice()
if (b < a)
a = 100
return 0
if input.getPrice() < 100:return true
if input.getPrice() >= 100:return false
db.products().where( p ->
p.getPrice() < 100)
Merge Analysis of Paths
81
true
b = p.getPrice()
if (b < a)
a = 100
return 1
false
b = p.getPrice()
if (b < a)
a = 100
return 0
if input.getPrice() < 100:return true
if input.getPrice() >= 100:return false
db.products().where( p ->
p.getPrice() < 100)Returns:
true == true ANDinput.getPrice() < 100
OR
false == true ANDinput.getPrice() >= 100
Merge Analysis of Paths
82
true
b = p.getPrice()
if (b < a)
a = 100
return 1
false
b = p.getPrice()
if (b < a)
a = 100
return 0
if input.getPrice() < 100:return true
if input.getPrice() >= 100:return false
db.products().where( p ->
p.getPrice() < 100)Returns:
true == true ANDinput.getPrice() < 100
OR
false == true ANDinput.getPrice() >= 100
Merge Analysis of Paths
83
true
b = p.getPrice()
if (b < a)
a = 100
return 1
false
b = p.getPrice()
if (b < a)
a = 100
return 0
if input.getPrice() < 100:return true
if input.getPrice() >= 100:return false
db.products().where( p ->
p.getPrice() < 100)returns input.getPrice() < 100
Different “Bytecode”, Same Analysis
84
Java Codedb.products()
.where( p -> p.getPrice() < 100)
Compiled Code 1a = 100b = p.getPrice()p = b < areturn d
Compiled Code 2a = 100b = p.getPrice()if (b < a) goto True:return 0True:return 1
Different “Bytecode”, Same Analysis
85
Java Codedb.products()
.where( p -> p.getPrice() < 100)
Compiled Code 1a = 100b = p.getPrice()p = b < areturn d
Compiled Code 2a = 100b = p.getPrice()if (b < a) goto True:return 0True:return 1
returns input.getPrice() < 100
Different “Bytecode”, Same Analysis
86
Java Codedb.products()
.where( p -> p.getPrice() < 100)
Compiled Code 1a = 100b = p.getPrice()p = b < areturn d
Compiled Code 2a = 100b = p.getPrice()if (b < a) goto True:return 0True:return 1
returns input.getPrice() < 100
returns input.getPrice() < 100
Syntax and SemanticsTranslation
Complications
87
References to Lambdas, but No Reflection
• Java 8 lambdas are opaque• Reflection doesn’t work on them• Don’t know which lambda we’ve received or what it does
88
Lambda Serialization
• Lambdas can be serialized• Serialization format is documented and fixed• Lambdas serialize into a regular object SerializedLambda
• Magically deserialize back into a lambda
• Serialization format stores• Underlying method
• External variables
• etc.
89
Lambda Deserialization
•Writing deserialization code is annoying• Serialize
• Change name of object
• Deserialize into a regular object
• Extract the parameters you need
90
00: ACED 0005 7372 0021 6A61 7661 2E6C 616E
¬ í . . s r . ! j a v a . l a n
10: 672E 696E 766F 6B65 2E53 6572 6961 6C69
g . i n v o k e . S e r i a l i
20: 7A65 644C 616D 6264 616F 61D0 942C 2936
z e d L a m b d a o a Ð . , ) 6
30: 8502 000A 4900 0E69 6D70 6C4D 6574 686F
. . . . I . . i m p l M e t h o
40: 644B 696E 645B 000C 6361 7074 7572 6564
d K i n d [ . . c a p t u r e d
50: 4172 6773 7400 135B 4C6A 6176 612F 6C61
Serialization Limitations
•Methods must accept serializable lambdas• Not regular lambdas• Can’t reuse any existing APIs
•External closure variables must be serializable• Users find that confusing
•More overhead• Must serialize and deserialize each lambda
• Java only serializes the name of the lambda and its variables
• Still imposes overhead though
• Performance hit acceptable for database accesses• Acceptable for other systems?
91
Syntax and SemanticsTranslation
Complications
Conclusion
92
Conclusion
• LINQ-style queries in Java• It works
• No compiler, VM, or other changes required
• Jinq embeds a rich and complex DSL in Java• Can we use the same techniques in other systems?
•http://www.jinq.org
93