In general, processes take longer to start than threads. This makes sense if you think about it – a thread lives within the memory space of its parent process, so it takes less work to set one up. (This is a gross oversimplification, but to be honest I find the details of process management incredibly uninteresting in 2008.) I assumed that this difference would hold for the Python processing module. Apparently it doesn't, at least on Mac OS X. Surprise!
Spawning 100 children with Thread took 1.04s Spawning 100 children with Process took 0.60s
The above result is for starting and joining the children serially. I get the same results in all of these variations:
I don't know whether this is due to goodness in OS X, or processing, or fork(), or just Unix in general. In any case, it's very good news. I'd dismissed processing for use on the client side of BitBacker because "process management is hard and they're too heavyweight." Clearly at least one of those complaints is invalid; maybe the other is as well. It would be a wonderful relief if I could use processes. I'm going to need parallelization of one form or another soon, and I'm definitely not going to start sprinkling threads around. Only madness lies down that path.
Here's the code that generated those results, in case you're interested:
import time, threading, processing for cls in [threading.Thread, processing.Process]: start = time.time() for _ in range(100): child = cls(target=lambda: None) child.start() child.join() print 'Spawning 100 children with %s took %.2fs' % ( cls.__name__, time.time() - start)