stm on pypy

STM

Francisco Fernandez Castano@fcofdezc

Francisco.fernandez.castano@gmail.comUpclose.me

The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software

Herb Sutterhttp://www.gotw.ca/publications/concurrency-ddj.htm

Inside the Python GIL

David Beazleyhttps://www.youtube.com/watch?v=ph374fJqFPE

Removing the GIL

● Fine-grained locking● Shared-nothing● STM

BEGIN;

UPDATE accounts SET balance = balance - 100.00

WHERE name = 'Alice';

UPDATE accounts SET balance = balance + 100.00

WHERE name = 'Bob';

COMMIT;

def update_account(src_account_id, dst_account_id, amount):

with atomic:src_account = accounts[src_account_id]dst_account = acconts[dst_account_id]dst_account.deposit(account.withdraw(amount))

● Hardware and Software support● Some modern processors (Haswell) support it

– https://software.intel.com/en-us/blogs/2012/02/07/transactional-synchronization-in-haswell

● Transaction is started● Memory read are logged into read set● Memory writes are logged into write set● Commit is attempted

PyPy approach

Illustration taken from Armin Rigo

Problems

● Too experimental● Only runs in 64 bit Linux and patched version of

Clang (just for translation)● Limited to 1.5 GB of Ram (Configurable)● JIT warm up still pretty bad● Different GC from regular PyPy (Less efficient)● Long running processes crash

Demo

References

● Rajwar, Herlihy and Lai: Virtualizing Transactional Memory

● Herlihy and Moss: Transactional Memory: Architectural Support for Lock-Free Data Structures

● Dice, Shalev, Shavit: Transactional Locking II

Donations

● http://pypy.org/tmdonate2.html