low overhead concurrency control for partitioned main ...low overhead concurrency control for...

Low Overhead Concurrency Control for Partitioned Main

Memory DatabasesEvan P. C. Jones Daniel J. Abadi Samuel Madden"

Banks"Payment Processing"Airline Reservations"

E-Commerce"Web 2.0"

Problem:"Millions of transactions per second"

Problem:"Millions of transactions per second"

=!

$$$$!

Alternative: H-Store Project"

Redesign specifically for OLTP"

Prototype: ~10X throughput"Idea: Remove un-needed features"

Source: Stonebraker et. al, “The End of an Architectural Era”, VLDB 2007. "

H-Store: High Throughput OLTP"

Redesign DB specifically for OLTP"Prototype: ~10X throughput"Main memory database"

Concurrency control consumes ~30-40% of CPU time"

CPU Cycle Breakdown for Shore on TPC-C New Order!Source: Harizopoulos, Abadi, Madden and Stonebraker,"

“OLTP Under the Looking Glass”, SIGMOD 2008"

!"#$%&'() "$$*+'#,) -*.) "/*+#) 012) *-) #%3) &',#.+$#&*',4) "'5) &,) 6.&7

8".&9:) &86*.#"'#) &') #%3)$.3"#3) .3$*.5)"'5);7#.33) 9**<+6)6*.#&*',

*-)#%3)#."',"$#&*'=)>%&,)&,)/3$"+,3)#%3)/+--3.)6**9)?+,35)&')$.3"#3@

"'5) ;7#.33,) ".3) #%3) 6.&8".:) ,%".35) 5"#") ,#.+$#+.3,) #%"#) 8+,#) /3

6.*#3$#35)A&#%)9"#$%3,=

B&'"99:4)*+.)/+--3.)8"'"(3.)8*5&-&$"#&*',)"$$*+'#)-*.)"/*+#)1C2

*-) #%3) #*#"9) &',#.+$#&*')$*+'#=)D3$"99) #%"#)A&#%) #%&,),3#)*-)8*5&-&7

$"#&*',4)'3A).3$*.5,)".3)"99*$"#35)5&.3$#9:)A&#%)8"99*$4)"'5)6"(3

9**<+6,) '*) 9*'(3.) %"E3) #*) (*) #%.*+(%) #%3) /+--3.) 6**9) &') 8*,#

$",3,=)>%&,)8"<3,).3$*.5)"99*$"#&*')3,,3'#&"99:)-.334)"'5),+/,#"'7

#&"99:)&86.*E3,)#%3)63.-*.8"'$3)*-)*#%3.)$*86*'3'#,)#%"#)63.-*.8

-.3F+3'#)9**<+6,4)9&<3);7#.33)9**<+6)"'5)+65"#3=

G#) #%&,)6*&'#4) #%3) .38"&'&'()<3.'39) .3F+&.3,)"/*+#)H2)?-*.)")ICJ

63.-*.8"'$3) ("&'K@) *-) #%3) #*#"9) &'&#&"9) &',#.+$#&*') $*+'#4) "'5) &,

"/*+#)L)#&83,)#%3)#*#"9)&',#.+$#&*',)*-)*+.)M*6#&8"9N),:,#38=)>%&,

"'"9:,&,)93"5,)#*)#A*)*/,3.E"#&*',O)-&.,#4)"99),&J)*-)#%3)8"P*.)$*87

6*'3'#,)".3),&('&-&$"'#4)3"$%)"$$*+'#&'()-*.)0Q)#%*+,"'5)*.)8*.3

&',#.+$#&*',) *-) #%3) &'&#&"9) 0QC) #%*+,"'5=) R3$*'54) +'#&9) "99) *-) *+.

*6#&8&S"#&*',)".3)"669&354)#%3).35+$#&*')&')&',#.+$#&*')$*+'#)&,)'*#

5."8"#&$O)/3-*.3)*+.)9",#),#36)*-).38*E&'()#%3)/+--3.)8"'"(3.4)#%3

.38"&'&'()$*86*'3'#,)+,35)"/*+#)")-"$#*.)*-)#%.33)-3A3.)&',#.+$7

#&*',) #%"') #%3) /",39&'3) ,:,#38) ?E3.,+,) ") -"$#*.) *-) IC) A%3') #%3

/+--3.)8"'"(3.)&,).38*E35@=

!"#"#$$%&'$()*&)

G) ,&8&9".) /.3"<5*A') *-) #%3) &',#.+$#&*') $*+'#) &') #%3)T3A)U.53.

#."',"$#&*') &,) ,%*A') &') B&(+.3) LV) B&(+.3) W) ,%*A,) ") 53#"&935

"$$*+'#&'() *-) "99) 00) 8*5&-&$"#&*',) "'5) *6#&8&S"#&*',) A3) 63.7

-*.835=)>%&,)#."',"$#&*')+,3,)"/*+#)0C)#&83,)",)8"':)&',#.+$#&*',

",)#%3)X":83'#)#."',"$#&*'4).3F+&.&'()01);7#.33)&',3.#,4)0I).3$*.5

$.3"#&*') *63."#&*',4) 00) +65"#3,4) I1) 6&'Y+'6&') *63."#&*',4) "'5) I1

;7#.33)9**<+6,=)>%3)8"&')5&--3.3'$3,)&')#%3)"99*$"#&*')*-)&',#.+$7

#&*',)#*)8"P*.)*6#&8&S"#&*',)/3#A33')T3A)U.53.)"'5)X":83'#)".3

&');7#.33)<3:)$*534) 9*((&'(4)"'5) 9*$<&'(=)R&'$3)T3A)U.53.)"55,

;7#.33) &',3.#&*',) &') #%3)8&J) *-) *63."#&*',4) #%3.3) &,)8*.3) .39"#&E3

/3'3-&#) #*) /3) %"5) /:) *6#&8&S&'() #%3) <3:) 3E"9+"#&*') $*53) ?"/*+#

0L2@=)!*((&'()"'5)9*$<&'()'*A)*'9:)"$$*+'#)-*.)"/*+#)0I2)"'5

0L2)*-)#%3)#*#"9)&',#.+$#&*',V)#%&,)&,)9".(39:)/3$"+,3)#%3)#*#"9)-."$7

#&*') *-) #&83) ,63'#) &') *63."#&*',)A%3.3) 9*((&'() "'5) 9*$<&'() 63.7

-*.8)")9*#)*-)A*.<)&,)8+$%),8"993.)&')#%&,)$",3=

>%3)/+--3.)8"'"(3.)*6#&8&S"#&*',),#&99).36.3,3'#)#%3)8*,#),&('&-&7

$"'#)A&')%3.34)"("&')/3$"+,3)A3)".3)"/93)#*)/:6",,)#%3)%&(%)*E3.7

%3"5) *-) .3$*.5) $.3"#&*'=) !**<&'() "#) #%3) 53#"&935) /.3"<5*A') &'

B&(+.3) W) -*.) #%3) /+--3.)8"'"(3.) *6#&8&S"#&*') .3E3"9,) ,*83#%&'(

,+.6.&,&'(O) $%"'(&'() -.*8) QZ) #*) 1IZ) 6"(3,) ?9"/39935) M,8"99

6"(3N@) 6.*E&53,) "98*,#) ") 0[2) .35+$#&*') &') #%3) #*#"9) &',#.+$#&*'

$*+'#=) >%&,) ,&8693) *6#&8&S"#&*') \) A%&$%) ,3.E3,) #*) .35+$3) #%3

-.3F+3'$:)*-)6"(3)"99*$"#&*',)"'5)53$.3",3);7#.33)536#%)\)*--3.,

"),&S3"/93)("&'=

!"#"!$$+,-.)/0.12,-$3-"$4506&-

]"E&'()9**<35)"#)#%3)53#"&935)/.3"<5*A')*-)&',#.+$#&*')$*+'#,)&'

#%3) X":83'#) "'5) T3A)U.53.) #."',"$#&*',4) A3) '*A) $*86".3) #%3

-."$#&*') *-) #&83) ?$:$93,@) ,63'#) &') 3"$%) 6%",3) *-) #%3) T3A)U.53.

#."',"$#&*')#*)#%3)-."$#&*')*-)&',#.+$#&*',)+,35)&')3"$%)6%",3=)>%3

.3,+9#,) ".3) ,%*A') &') B&(+.3) Q=) G,) A3) '*#35) 3".9&3.4) A3) 5*) '*#

3J63$#) #%3,3) #A*) -."$#&*',) #*) /3) &53'#&$"9) -*.) ") (&E3') 6%",34

/3$"+,3) $"$%3) 8&,,3,) "'5) 6&639&'3) ,#"99,) ?#:6&$"99:) 5+3) #*

/."'$%3,@) $"') $"+,3) ,*83) &',#.+$#&*',) #*) #"<3)8*.3) $:$93,) #%"'

*#%3.,=)B*.)3J"86934);7#.33)*6#&8&S"#&*',).35+$3)$:$93,)93,,)#%"'

#%3:).35+$3)&',#.+$#&*',4)/3$"+,3)#%3)R%*.3);7#.33)$*53)*E3.%3"5

A3) .38*E3) &,)8"&'9:) *--,3#) $"9$+9"#&*',)A&#%) -3A) $"$%3)8&,,3,=

^*'E3.,39:4)*+.) .3,&5+"9)M<3.'39N)+,3,)") 9".(3.) -."$#&*')*-)$:$93,

#%"') &#) 5*3,) &',#.+$#&*',4) /3$"+,3) &#) &,) /."'$%7%3"E:4) $*',&,#&'(

8*,#9:) *-) -+'$#&*') $"99,=) R&8&9".9:4) 9*((&'() +,3,) ,&('&-&$"'#9:

8*.3)$:$93,)/3$"+,3)&#)#*+$%3,)")9*#)*-)838*.:)$.3"#&'()"'5)A.&#7

&'() 9*() .3$*.5,) ?5&,<) _YU) #&83) &,) '*#) &'$9+535) &') 3&#%3.) (."6%@=

B&'"99:4) 9*$<&'()"'5) #%3)/+--3.)8"'"(3.)$*',+83)"/*+#) #%3),"83

63.$3'#"(3)*-)$:$93,)",)#%3:)5*)&',#.+$#&*',=)

!"#

!$#

!%#

!&#

!'#

(!"#

(!$#

(!%#

(!&#

(!'#

!"#$%&'$()"#

!"#$%&'()*+

*$%++

,)--("-

,)'.("-

,/$'0("-

1&22+%

3/"/-+%

,!"&&%-"(.

-"(."//.,,

%+3/("("-4)5+%0+/6

0#,1%&'(

0#2%&''13-

4"/5#'$,

.+7#

!"#$%&'() *+,-./".#'0%&-1/23.'42%'5&3'6%/&%'78&&'9&:;"2.'<)='42%';>&'?-0&?8'2.';>&'?&4;':[email protected])

!"#

!$#

!%#

!&#

!'#

(!"#

(!$#

(!%#

(!&#

!"#

!)#

(!"#

(!)#

$!"#

$!)#

*!"#

*!)#

!"#$%&'$()"#

87',+#

*$%++

,)--("-

,)'.("-

,/$'0("-

1&22+%3/"/-+%

.+7#

*$%++

,)--("-

,)'.("-

,/$'0("-

1&22+%3/"/-+%

.+7#

!"#$%&'B) C.8;%$:;"2.8'7?&4;A'D8)'EF:?&8'7%"#>;A'42%'5&3'6%/&%)

L=Q2

1[=L2

0[=I2

0L=12

00=`2

0L=I2

0I=12

I`=L2

0C=I2

0Q=W2

I02

Q=02

concurrency control! 30-40%

Speculative Concurrency Control"

Eliminate fine-grained access tracking (locks or read/write sets)"

Eliminate undo logs (where possible)"

Up to 2X faster than locking for appropriate workloads!

Why Support Concurrency?"

main memory"stored procedures"

partition per core"

donʼt do them"

Use idle resources:"" disk stalls"" user stalls"

Physical resources:"" multiple CPUs"" multiple disks"

Long running txns:"

H-Store: Single thread engine"

Assumptions:!

Database divided into partitions"Transactions access one partition (mostly)"

Mapping procedures to partitions is given"Total data fits in memory of N machines"Partitions are replicated on 2 machines"

System Overview"

Clients

H-Store

Partition 2

Partition 2

Client Library Client Library Client Library

Primary

Backup

Partition 1

Partition 1

Primary

Backup

Coordinator

Single Partition Transaction"Client

Primary

Backup

1


Primary

Backup

1

2


Primary

Backup

1

2

execute


Primary

Backup

1

2 3

execute


Primary

Backup

1

2 3

4

execute


Primary

Backup

1

2 3

4

execute execute

Single Partition Transaction"

Clients

H-Store

Partition 2

Partition 2


Primary

Backup

Partition 1

Partition 1

Primary

Backup

Coordinator1


Clients

H-Store

Partition 2

Partition 2


Primary

Backup

Partition 1

Partition 1

Primary

Backup

Coordinator1

2


Clients

H-Store

Partition 2

Partition 2


Primary

Backup

Partition 1

Partition 1

Primary

Backup

Coordinator

3

1

2


Clients

H-Store

Partition 2

Partition 2


Primary

Backup

Partition 1

Partition 1

Primary

Backup

Coordinator

3

1

2

4

Not Perfectly Partionable?"

Example: users and groups"

Many applications are mostly partitionable"

e.g. TPC-C: 11% multi-partition transactions"

Distributed Transactions"

Need two-phase commit (consensus)"

Simple solution:"block until the transaction finishes"

Introduces network stall (bad)"

Blocking Multi-Partition"

Clients

H-Store

Partition 2

Partition 2


Primary

Backup

Partition 1

Partition 1

Primary

Backup

Coordinator

1


Clients

H-Store

Partition 2

Partition 2


Primary

Backup

Partition 1

Partition 1

Primary

Backup

Coordinator

1

2 2

Blocking Multi-Partition"Client

Coordinator

P1 Primary

P1 Backup

2

1


Coordinator

P1 Primary

P1 Backup

2

3

1


Coordinator

P1 Primary

P1 Backup

2

3

1

execute


Coordinator

P1 Primary

P1 Backup

2

3

1

execute

4


Coordinator

P1 Primary

P1 Backup

2

3

1

execute

4

5


Coordinator

P1 Primary

P1 Backup

2

3

1

execute

4

5

6

6


Coordinator

P1 Primary

P1 Backup

2

3

1

execute

4

5

6

6

execute


Coordinator

P1 Primary

P1 Backup

2

3

1

execute

4

5

6

6

execute

network stall


Clients

H-Store

Partition 2

Partition 2


Primary

Backup

Partition 1

Partition 1

Primary

Backup

Coordinator

3

1

2 2

3


Clients

H-Store

Partition 2

Partition 2


Primary

Backup

Partition 1

Partition 1

Primary

Backup

Coordinator

3

1

2 2

4 3 4


Clients

H-Store

Partition 2

Partition 2


Primary

Backup

Partition 1

Partition 1

Primary

Backup

Coordinator

3

1

2 2

4 3 4

55


Clients

H-Store

Partition 2

Partition 2


Primary

Backup

Partition 1

Partition 1

Primary

Backup

Coordinator

3

1

2 2

4 3 4

5 56 6

6

Two-Phase Locking"

+ Execute non-conflicting txns during stall"+ No need to order in advance!– Locking overhead"– Deadlocks"

Optimization: turn off locks and undo logging when no multi-partition transactions"

Speculative CC"

While waiting for commit/abort, speculatively execute other transactions"

+ No locks; no read/write sets"– Need global transaction order"– Cascading aborts"

Speculative Multi-Partition"Client

Coordinator

P1 Primary

P1 Backup

2

3

1

execute

4

5


Coordinator

P1 Primary

P1 Backup

2

3

1

execute

4

5

execute


Coordinator

P1 Primary

P1 Backup

2

3

1

execute

4

5

execute

6

6

Speculation Limitation"

Transactions with multiple “rounds” of work: need network stall"

Example:"1. Read x on partition 1, y on partition 2"2. Update x = f(x, y); y = f(x, y)"

Speculative Multi-Partition"

Clients

H-Store

Partition 2

Partition 2


Primary

Backup

Partition 1

Partition 1

Primary

Backup

Coordinator

3

1

2 2

3


Clients

H-Store

Partition 2

Partition 2


Primary

Backup

Partition 1

Partition 1

Primary

Backup

Coordinator

3

1

2 2


Clients

H-Store

Partition 2

Partition 2


Primary

Backup

Partition 1

Partition 1

Primary

Backup

Coordinator

3

1

2 2

4 3 4


Clients

H-Store

Partition 2

Partition 2


Primary

Backup

Partition 1

Partition 1

Primary

Backup

Coordinator

3

1

2 2

4 3 4

5 5


Clients

H-Store

Partition 2

Partition 2


Primary

Backup

Partition 1

Partition 1

Primary

Backup

Coordinator

3

1

2 2

4 3 4

5 56 6

6

Microbenchmark"

Primary

Backup

Primary

Backup

Partition 1 Partition 2

Coordinator

Client Load Generator

Two partitions of a single table" (id INTEGER PRIMARY KEY, value INTEGER)

Microbenchmark"

Single partition transaction:"" read/write keys on one partition"Multi-partition transaction:"" access half keys from each partition"

single partition work = multi-partition work"No deadlocks, no aborts, no conflicts"

0"

5000"

10000"

15000"

20000"

25000"

30000"

0" 20" 40" 60" 80" 100"% Multi-Partition"

Locking"

Speculation"

Blocking"

Throughput (transactions/s)"

Locking 40% Conflicts"

0

5000

10000

15000

20000

25000

30000

0% 20% 40% 60% 80% 100%

Tra

nsa

ctio

ns/s

eco

nd

Multi-Partition Transactions

Speculation

Locking

Blocking

TPC-C Based"

~11% multi-partition transactions"More complex locking"Many conflicts"Some deadlocks"Some aborts"

0"

5000"

10000"

15000"

20000"

25000"

30000"

2" 4" 6" 8" 10" 12" 14" 16" 18" 20"Warehouses"

Locking"

Speculation"

Blocking"

Throughput (transactions/s)"

0

5000

10000

15000

20000

25000

2 4 6 8 10 12 14 16 18 20

Transactions/second

Warehouses

SpeculationBlockingLocking

Speculative CC"

! better for “mostly partitionable” apps on main memory DBs!

Up to 2X throughput"No locking overhead"No deadlocks"

0

5000

10000

15000

20000

25000

30000

0% 20% 40% 60% 80% 100%

Tran

sact

ions

/sec

ond


Speculation 0% abortsSpeculation 1% abortsSpeculation 3% abortsSpeculation 5% aborts

Speculation 10% abortsBlocking 10% abortsLocking 10% aborts

0

5000

10000

15000

20000

25000

30000

0% 20% 40% 60% 80% 100%

Tran

sact

ions

/sec

ond


Locking 0% conflictLocking 20% conflictLocking 60% conflict

Locking 100% conflictSpeculation

Blocking

0

5000

10000

15000

20000

25000

30000

0% 20% 40% 60% 80% 100%

Tran

sact

ions

/sec

ond



0

5000

10000

15000

20000

25000

30000

35000

0% 20% 40% 60% 80% 100%

Tran

sact

ions

/sec

ond


Model BlockingModel Speculate

Model LockingMeasured Blocking

Measured SpeculateMeasured Locking

0

5000

10000

15000

20000

25000

30000

35000

0% 20% 40% 60% 80% 100%

Tran

sact

ions

/sec

ond



0

5000

10000

15000

20000

25000

30000

0% 20% 40% 60% 80% 100%

Tran

sact

ions

/sec

ond


SpeculationLocal Speculation

BlockingLocking

Optimistic

0

5000

10000

15000

20000

25000

2 4 6 8 10 12 14 16 18 20

Transactions/second

Warehouses


Optimistic

low overhead concurrency control for partitioned main ...low overhead concurrency control for...

Documents