Processes spawn faster than threads?
In general, processes take longer to start than threads. This makes sense if you think about it - a thread lives within the memory space of its parent process, so it takes less work to set one up. (This is a gross oversimplification, but to be honest I find the details of process management incredibly uninteresting in 2008.) I assumed that this difference would hold for the Python processing module. Apparently it doesn't, at least on Mac OS X. Surprise!
Spawning 100 children with Thread took 1.04s
Spawning 100 children with Process took 0.60sThe above result is for starting and joining the children serially. I get the same results in all of these variations:
- Starting them all at once, then joining them all at once.
- Using 10 children or 1,000 children.
- Having each child sleep for one second (to ensure that they're all actually alive at the same time).
I don't know whether this is due to goodness in OS X, or processing, or fork(), or just Unix in general. In any case, it's very good news. I'd dismissed processing for use on the client side of BitBacker because "process management is hard and they're too heavyweight." Clearly at least one of those complaints is invalid; maybe the other is as well. It would be a wonderful relief if I could use processes. I'm going to need parallelization of one form or another soon, and I'm definitely not going to start sprinkling threads around. Only madness lies down that path.
Here's the code that generated those results, in case you're interested:
import time, threading, processing
for cls in [threading.Thread, processing.Process]:
start = time.time()
for _ in range(100):
child = cls(target=lambda: None)
child.start()
child.join()
print 'Spawning 100 children with %s took %.2fs' % (
cls.__name__, time.time() - start)