Concurrency with Multiprocessing in Python

PhillyPug & Philly.rb RedSnake MeetingFebruary 8th, 2011

Gavin M. RoymyYearbook.com

green threads in threading

from threading import Threadimport time

class MyThread(Thread):

def run(self): print "%s running" % self.name time.sleep(5) print "%s done" % self.name for x in xrange(0,10): thread = MyThread() thread.start() thread.join(0)

gmr-0x04:pika gmr$ python threads.py MyThread-1 runningMyThread-2 runningMyThread-3 runningMyThread-4 runningMyThread-5 runningMyThread-6 runningMyThread-7 runningMyThread-8 runningMyThread-9 runningMyThread-10 runningMyThread-1 doneMyThread-2 doneMyThread-3 doneMyThread-4 doneMyThread-5 doneMyThread-6 doneMyThread-7 doneMyThread-8 doneMyThread-9 doneMyThread-10 done

threading

• Locks

• Reentrant Locks

• Conditions

• Semaphores

• Events

• Timers

Reentrant knows who own the locks and the recursion level

Additional classes like Queue.Queue

This GIL From David Beeazley’s GIL Visualizationhttp://www.dabeaz.com/GIL/gilvis/fourthread.html

enter multiprocessing

multiprocessing module

• All the things threading has

• Exchanging Objects

• Queues and Pipes

• Shared State

• Pipes and Queues

• Pools

• Connections

• Managers

• SyncManager

• Logging

TCP Server coordinating shared objects

Process Sync

from multiprocessing import Processimport time

class MyThread(Process):

def run(self): print "%s running" % self.name time.sleep(5) print "%s done" % self.name for x in xrange(0,10): thread = MyThread() thread.start() thread.join(0)

gmr-0x04:pika gmr$ python processes.py Thread-1 runningThread-2 runningThread-3 runningThread-4 runningThread-5 running Thread-6 running Thread-7 runningThread-8 runningThread-9 runningThread-10 runningThread-1 doneThread-2 doneThread-3 doneThread-4 doneThread-5 doneThread-6 doneThread-7 doneThread-8 doneThread-9 doneThread-10 done

Photo By Susan NYC: http://www.flickr.com/photos/en321/33868864/

multiprocessing.reduction

# Process #1from multiprocessing.reduction import reduce_handleimport socket

# Create a socketsock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)

# Do stuff here, client or server wise

# Create the pickled socket handlehandle = reduce_handle(sock.fileno)

# Process #2from multiprocessing.reduction import rebuild_handleimport socket

# In other processfd = rebuild_handle(handle)sock = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)

# Now I can read and write from the socket too