Creating C Modules for free-threaded interpreter

[matj-sag]

Bug report

Bug description:

Hi all,

We've just moved to 3.13 and I'm looking at the free-threaded interpreter stuff. We're embedding CPython in our own process and dynamically loading the shared library, so I can switch it to load the 3.13t library instead. However, we also have internally to the program some C modules that we expose into Python. These modules aren't separate .so files that get loaded in, they're part of the program which is then added with PyModule_Create2.

I obviously need to switch that to PyModuleDef_Init, but then I run into the problem that PyModuleDef_HEAD_INIT (via PyObject_HEAD_INIT) is defined differently in free threaded vs GIL mode. I would like to be able to compile the module once (or even have two code paths creating different objects), but currently that's impossible, since only one HEAD_INIT structure is defined at any given time.

I can make my module not rely on the GIL, but I'd like it to be able to load in both interpreter versions. Is this something that you are planning, or could plan, to support in future?

CPython versions tested on:

3.13

Operating systems tested on:

Linux

[ZeroIntensity]

Short answer: No, you can't compile something for 3.13 and then load it in 3.13t, or vice versa.

Long answer: We're sort of working on this, but not for the non-limited API. It sounds like you want a limited API that's compatible across FT and non-FT builds. @encukou has been doing that work, but as he said, PyModuleDef is inherently broken because it relies on the object structure. PEP 793 is the first step to getting this working.

[matj-sag]

Yes, I'd love for it to be in the limited API. I could live with it being in the non-limited API (and subject to change between versions), but that would be even better. (I'm already having to break the limited API to support separate sub-interpreter GILs, since Py_NewInterpreterWithConfig is not in the limited API).

Good to know it's being worked on, thanks

[colesbury]

As @ZeroIntensity wrote, you cannot mix Python libraries between builds free threaded and GIL-enabled builds. It's not enough to change a few things like PyModuleDef_HEAD_INIT.

You need to build your interpreter with --disable-gil if you are using the autoconf/configure based build. The important thing is that Py_GIL_DISABLED is defined to 1 when building both the interpreter and extension modules.