-
Notifications
You must be signed in to change notification settings - Fork 6
1 What's the Point?
There are 2 main things that this project is trying to solve:
- improve Python's multi-core story
- fill the gap in supported concurrency models
The proposed solution will resolve both but requires effort and expands the language a little. So it's important that there is a clear justification. Why is it worth it?
- true multi-core parallelism in CPython
- human-friendly concurrency model
- combines the benefits of threads (in-process) and multiprocessing (isolation)
- requires secondary improvements to the runtime, leading to:
- better runtime performance
- better startup time
- more consistency in runtime, improving maintainability
- better discoverability in runtime code, improving maintainability
- improved isolation in runtime gives embedders less problems
- helps extension modules be more isolated/self-contained
- enables other runtime/language improvements
- most of the secondary improvements stand on their own merits
- provides a low-level, thread-safe, in-process serialization mechanism for Python objects
- does not require substantial changes to CPython's code (subinterpreters already exist in C-API)
- does not break backward compatibility for extension modules
- does not negatively impact performance
- subinterpreters have been part of the CPython C-API for over 20 years
- we're not ripping it out at this point
- current users of the C-API will benefit from a stdlib module
Since I started this project in 2014 (at the encouragement of Nick Coghlan), I've been confident that it's something people want. However, I hadn't really solicited feedback/support until recently (other than on a few mailing list threads).
That has changed since the core sprint at Facebook, September 2017. The ball has been rolling since then, including the creation of PEP 554. At the 2018 language summit I presented on this project and the writeup showed up on LWN. Throughout all 9 days of PyCon 2018 I made a concerted effort to explain the project to as many people as possible and ask for their feedback. I spoke with many folks, including Guido, Davin Potts, Travis Oliphant, Jake VanderPlas, Nathaniel Smith, Lukasz Langa, and other core devs, plus various folks involved in web frameworks. The response was overwhelmingly positive. Many folks were visibly excited by the idea. Also, at this point several people are helping me on the project (most notably Emily Morehouse).
The only negative reaction I've gotten from the Python community has been from 3 individuals: Guido, Nathaniel Smith, and Sturla Molden. Guido was was unconvinced that subinterpreters provided any improvement over multiprocessing and worried about complexity. Nathaniel echoed those concerns, worried about the extra burden on extension authors, and doubted the goals of the project could be achieved without breaking compatibility. Sturla advocated strongly for multiprocessing (fork w/ COW).
In all honesty, I have serious reservations about the future of this project if Guido doesn't support the idea. He has language design superpowers! So even if he doesn't call the shots anymore, I want Guido's support. This page is, for better or worse, the centerpiece of the effort to make the case (to Guido and anyone else) for the value of this project.
Aside from some of the benefits I've noted above (and concerns below), here are the most interesting bits of feedback I received at PyCon:
- subinterpreters is a more elegant API for concurrency than the others
- threads vs. subinterpreters is the same argument as threads vs. procs
- what is the future of multiprocessing?
- likely that few extension modules will break
- burden on extension module authors is a red herring
- the community will get more excited once there's something they can try out
There are still a number of individuals with whom I'd like to speak regarding how they could make use of subinterpreters in their area of expertise. In the area of numerical/scientific computing I'm hoping to speak with Matt Rocklin (Dask). Most notably I need to get more feedback from web framework folks, e.g. Andrew Godwin.
- someone has to do the work :)
- adds cognitive burden to Python users
- some might see it as yet-another-concurrency-model-to-understand
- adds to Python's maintenance burden
- cost of per-interpreter state
- subinterpreter startup is relatively expensive vs. multiprocessing (fork + COW)
- per-interpreter state is relatively expensive vs. threads/multiprocessing
- no segfault isolation
- why not keep using threads/multiprocessing/async/etc.?
- could it break embedders that rely on GIL?
- won't subinterpreters add extra burden on extension module authors?
- subinterpreters add unnecessary complexity to the CPython implementation?
- why not "just" get rid of the GIL?
Rebuttals:
- I'm doing the work, albeit slowly, and others have volunteered to help.
- Every new feature adds to the cognitive burden on Python users. This project does not add much and the benefits to Python users are significant.
- This is also true of every new feature. Furthermore, the net change in maintenance burden should actually be a reduction since a lot of the work involved brings consistency to the runtime code. PEP 554 will help normalize use of subinterpreters in the test suite. Finally, most of this project will involve work that should be done regardless. The remainder to actually facilitate multi-core subinterpreters is quite small.
- This will need to be addressed. However, the extra costs involved are relatively small and should not impact most uses of subinterpreters.
- It's unclear where this would actually be a problem in practice. However, dealing with segfaulting processes isn't a problem that's unique to subinterpreters...
- See the section below on concurrency models.
- I'd expect this to not be a problem. We will likely still have a GIL for some operations. Furthermore, if an embedder is using a single interpreter then nothing will change for them.
- It shouldn't. Most extension modules will be unaffected by this project. For those that are affected, there are a number of possible ways that we can avoid the problem. Most significantly, extension module authors can indicate that they do not support subinterpreters.
- That ship sailed years ago. We really no longer in a position to remove subinterpreters from the C-API. Furthermore they are being used more frequently, including in major projects. At this point we should fix broken corner cases, close gaps, and provide a Python API.
- Good luck with that. :)
The various concurrency models in Python have strengths and weaknesses/deficiencies. There's also a meaningful gap that subinterpreters can fill.
- greatly impacted by the GIL
- easy to cause problems that are hard to debug (e.g. race conditions)
- coming from other languages, folks reach for threads
- in data science (e.g. Dask), threshold where MPI (distributed) makes more sense than threads in ~3?
- ...
- to folks coming from other languages, multiprocessing seems like overkill/bad
- leads to higher devops costs
- large data + MP -> expensive serialization
- fork start -> much undefined behavior (e.g. combine fork/threads)
- in data science, multiprocessing doesn't really factor in (they either use threads or MPI)
- Copy-On-Write benefits are mostly eliminated due to refcounting
- <Use cases vs multiprocessing>
- ...
- ...
Why haven't folks been using subinterpreters, historically?
- mod_wsgi
Why are projects starting to use subinterpreters now?