This might be a dumb question, but why would removing the GIL break FFI? Is it j...

kortex · on June 15, 2023

> C programs unavoidably interact directly with the GIL?

Bingo. They don't have to, but often the point of C extensions is performance, which usually means turning on parallelism. E.g. Numpy will release the GIL in order to use machine threads on compute-heavy tasks. I'm not worried about the big 5 (numpy, scipy, pandas, pytorch, and sklearn), they have enough support that they can react to a GILectomy. It's everyone else that touches the GIL but may not have the capacity or ability to update in a timely manner.

I don't think this is something which can be shimmed either or ABI-versioned either. It's deeeep and touches huge swaths of the cpython codebase.

saltminer · on June 15, 2023

Thanks, that explains a lot. Sounds like a task that would have to be done in Python 4, if ever it exists.

notatallshaw · on June 15, 2023

> or is there a fundamental requirement, e.g. C programs unavoidably interact directly with the GIL?

Both C programs can use the GIL for thread safety and can make assumptions about the safety of interacting with a Python object.

Some of those assumptions are not real guarantees from the GIL but in practise are good enough, they would no longer be good enough in a no-GIL world.

> I know that the C-API is only stable between minor releases [0] compiled in the same manner [1], so it's not like the ecosystem is dependent upon it never changing.

There is a limited API tagged as abi3[1] which is unchanging and doesn't require recompiling and any attempt to remove the GIL so far would break that.

> so it's not like the ecosystem is dependent upon it never changing

But the wider C-API does not change much between major versions, it's not like the way you interact with the garbage collector completely changes causing you to rethink how you have to write concurrency. This allows the many projects which use Python's C-API to relatively quickly update to new major versions of Python.

> I have found a no-GIL interpreter that works with numpy, scikit, etc. [2][3] so it doesn't seem to be a hard limit.

The version of nogil Python you are linking is the product of years of work by an expert funded to work full time on this by Meta, the knowledge is sourcing many previous attempts to remove the GIL including the "gilectomy". Also you are linking to the old version based on Python 3.9, there is a new version based on Python 3.12[2]

This strays away from the points I was making, but with this specific attempt to remove the GIL if it is adopted it is unlikely to be switched over in a "big bang", e.g. Python 3.13 followed by Python 4.0 with no backwards compatibility on C extensions. The Python community does not want to repeat the mistakes of the Python 2 to 3 transition.

So far more likely is to try and find a way to have a bridge version that supports both styles of extensions. There is a lot of complexity in this though, including how to mark these in packaging, how to resolve dependencies between packages which do or do not support nogil, etc.

And even this attempt to remove the GIL is likely to make things slower in some applications, both in terms of real-world performance as some benchmarks such as MyPy show a nearly 50% slowdown and there may be even worse edge cases not discovered yet, and in terms of lost development as the Faster CPython project will unlikely be able to land a JIT in 3.13 or 3.14 as they plan right now.

[1]: https://docs.python.org/3/c-api/stable.html#c.Py_LIMITED_API [2]: https://github.com/colesbury/nogil-3.12