data tieringSqueezing scale out of MySQL

Julien, VP Engineering at HouseTrip

github.com/mezis

disclaimer

IANADBA*I’m not a database administrator

!3

the scene

!4

!5

Load balancer(ELB)

30x10web workers

(Passenger/Rack)

6x20job workers

(DJ)

Memcache4x7GB

MySQL400GB

5x replica

MondoDB120GB

2x replica

S3TBs

85k qpm

3k rpm

predictable traffic ~25% searches

!6

search = [destination, start date, end date]

↓[ [property, price], … ]

!7

property!

destination_id

availability!

property_id rate_id

start_date end_date

rate!

price

search → big bulky join

booking → long transaction+ business logic

+ many small R/W

!8

property!

destination_id

availability!

property_id rate_id

start_date end_date

rate!

price

the crisis

!9

peak traffic 7pm - 10pm write queries !

read queries "

write IO ⛅️

cpu load ⛅️

memory ☀️

!10

contention

noun (kənˈtɛnʃən)

1. a struggling between opponents

2. competition for limited resources

!11

slow reads ? poor use of indices

during large write transactions

!12

http://dev.mysql.com/doc/refman/5.5/en/optimizing-innodb-transaction-management.html http://dev.mysql.com/doc/refman/5.5/en/glossary.html#glos_covering_index

slow writes ? load+locking on rollback segments

!13

http://dev.mysql.com/doc/refman/5.1/en/innodb-multi-versioning.htmlhttp://dev.mysql.com/doc/refman/5.5/en/glossary.html#glos_rollback_segment

digging & deeper SHOW ENGINE INNODB STATUS '

!14

---TRANSACTION 72C, ACTIVE 755 sec 4 lock struct(s), …, 3 row lock(s), undo log entries 12 TABLE LOCK …RECORD LOCKS …RECORD LOCKS … locks rec but not gap RECORD LOCKS … lock_mode X locks gap before rec 

http://www.mysqlperformanceblog.com/2012/03/27/innodbs-gap-locks/http://dev.mysql.com/doc/refman/5.1/en/innodb-monitors.html

!15

(not) fixing it

!16

horizontal scaling “throw money at it”

→ does not work

→ ops + maintenance cost

!17

fine-tunethe DB engine

“bring in the experts”

→ only works short-term

!18

vertical scaling “throw more money at it”

→ bigger database instances~ +4k$/mo

→ only 2 cartridgesin that gun

!19

put it in a service de-normalise the data

use that noSQL thing

!20

webscalewebscale

webscale

good solution ? → live in 1-2 weeks

→ buys 6-12 months

!21http://xkcd.com/1205/

( intermission (

!22

frame tearing (not tiering)

!23

frame tearing caused by “simple buffering”

!24

render scene in buffert

draw to screen from buffer t

frame(buffer) tiering aka “double buffering”

!25

render scene in buffert

draw to screen from buffer t

data tiering separate read and write tables

!26

availabilities_back

http://en.wikipedia.org/wiki/File:Comparison_double_triple_buffering.svg

availabilities_frontread

availabilitiesread/write

� swap

copy

how it works

!27

tables availabilitiesavailabilities_0availabilities_1

data_tiering_switchesdata_tiering_sync_logs 

!28

using tiered tables DataTiering::Switch.new.active_scope_for(Availability)

equivalent one of

Availability.scoped(:from => 'properties_0')Availability.scoped(:from => 'properties_1')

!29

availabilities_back

availabilities_frontread

availabilitiesread/write

� swap

copy

syncing DataTiering::Sync.sync_and_switch!

regularly* scheduled task(every 5 min for us, takes ~ 60s)

*depend on acceptable staleness

!30

availabilities_back

availabilities_frontread

availabilitiesread/write

� swap

copy

syncing: schema lazily (no migrations) :- create missing /.*_[01]/ tables

- compare schemas with SHOW CREATE TABLE

!31

availabilities_back

availabilities_frontread

availabilitiesread/write

� swap

copy

syncing: bulk - run TRUNCATE TABLE then INSERT INTO … SELECT FROM

- too slow at runtime (only for setup / after migrations)

!32

availabilities_back

availabilities_frontread

availabilitiesread/write

� swap

copy

syncing: deltas - deletions : SELECT id … LEFT JOIN … DELETE … WHERE id IN …

- insertions & updates : REPLACE INTO … row_touched_at > X

- remember last sync in data_tiering_sync_logs 

- row_touched_at “magic” TIMESTAMP column

!33

availabilities_back

availabilities_frontread

availabilitiesread/write

� swap

copy

swapping - renaming tables not transactional

- atomically change a pointer instead (and cache it)

→ data_tiering_switches

!34

availabilities_back

availabilities_frontread

availabilitiesread/write

� swap

copy

gem by

@kratob@hubb@mconnell@danielgrieve

!35

outcome

!36

outcome averageDB time unchanged

nominal at peak traffic

deadlocks timeoutsfaster reads

!37

epilogue

!38

so long, data tiering ! you served us well

!39

keep calm ♡

refactor

!40http://dev.housetrip.com/

@mezis_fr