drupal camp london 2014 john ennew concurrency session deeson online

C O N C U R R E N C YA N I N T R O D U C T I O N T O

J O H N E N N E W, S O L U T I O N S A R C H I T E C T, D E E S O N O N L I N E C E N G O N D . O . @ D E E S O N _ L A B S H T T P : / / W W W. D E E S O N - O N L I N E . C O . U K / L A B S

… O R H O W I M A D E M I G R AT E FA S T E R

W H AT I S C O N C U R R E N C Y ?

DatabaseFilestore

Web service

Application

DatabaseFilestore

Web service

Application

B R E A K I N G T H E P R O B L E M D O W N …

– E D W A R D A . L E E

“… a folk definition of insanity is to do the same thing over and over again and to expect the results to be different. By this definition, we in fact require

that programmers of multithreaded systems be insane. Were they sane, they could not understand

their programs …”

T H E P R O B L E M W I T H T H R E A D S , 2 0 0 6

T H E H T T P : / / W W W. E E C S . B E R K E L E Y. E D U / P U B S / T E C H R P T S / 2 0 0 6 /E E C S - 2 0 0 6 - 1 . H T M L W I T H T H R E A D S , 2 0 0 6

T H E D I N I N G P H I L O S O P H E R S

H T T P : / / D E E S O N - O N L I N E . C O . U K / L A B S / M U LT I - P R O C E S S I N G - PA R T- 1 -H O W - M A K E - D R U S H - R U S H

• lock_acquire()

• drush_invoke_process()

• DB transactions (innodb)

D R U PA L 7 T O O L S

M A K I N G M I G R AT E M O V E …

W H Y S P E E D U P M I G R A T I O N S ?

• Reduce the time a migration takes

• Discover and fix problems quicker

• Remigrate faster / regular migrations

T H E P R O C E S S …

• https://github.com/johnennewdeeson/drush-multi-processing

• Download mt.drush.inc and place in sites/all/drush

• Update to migrate 7.x-2.6-rc1

• Configure your migration class for batched operation

• Create a custom drush import command (or modify mtm.drush.inc also at github URL above)

• Full instructions online at: http://deeson-online.co.uk/labs/multi-processing-part-2-how-make-migrate-move

Configuring your migration class

Configuring your migration class

Creating a Drush command

Starting the process thread manager

Process setup …

Process teardown …

L I V E D E M O …

F I N E T U N I N G …

• Ramp up slowly - monitor server load and db load

• /etc/my.cnfinnodb_locks_unsafe_for_binlog = 1 innodb_flush_log_at_trx_commit=2

• Use alternative cache mechanism to ease your database load (e.g. memcache)

• $conf['lock_inc'] = 'sites/all/modules/memcache/memcache-lock.inc';

• $conf['memcache_stampede_protection'] = TRUE;

T H A N K S ! … A N Y Q U E S T I O N S ?

J O H N E N N E W, S O L U T I O N S A R C H I T E C T, D E E S O N O N L I N E C E N G O N D . O . @ D E E S O N _ L A B S H T T P : / / W W W. D E E S O N - O N L I N E . C O . U K / L A B S