Add minimal C-api implementation that builds with Pyo3 by bschoenmaeckers · Pull Request #7562 · RustPython/RustPython

bschoenmaeckers · 2026-04-05T12:20:57Z

This is my shot at implementing a minimal Cpython compatible C-api. I've implemented the bare minimum to get the included Pyo3 example running where most of the api is stubbed. I'm not familiar with the rest of the RustPython code base so let me know what you think and where I did stupid things.

Please take extra care reviewing the pylifecycle.rs & pystate.rs files where I try to setup the RustPython interpreter.

xref #5604

coderabbitai · 2026-04-05T12:21:06Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: b6b67fcb-38aa-448a-bce3-b40ce5bbf56a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

youknowone · 2026-04-05T12:47:20Z

Could you try with hpy if you dont mind? We are looking for sustainable way

bschoenmaeckers · 2026-04-05T13:04:40Z

Could you try with hpy if you dont mind? We are looking for sustainable way

That would require significant changes in codebase that are using the CPython api spec. I would like to explore the possibility to use the ab3/abi3t api. Would you be willing to accept that?

youknowone

This way will not work.

Just in case, the code can be generated, but I hope we carefully review the decisions. unlikely-to-happen decisions must be reviewed and justified by document(comment) the decisions.
If you are not familiar enough to the project to decide good way, please start from a smaller and simpler issue.

We only need abi3t, because we are compatible to free-threading.

We have to minimize the surface of C API. if HPy helps it, we need HPy.

Ideally c api must be very thin wrappers to RustPython features. "code" must not be included in this crate.

youknowone · 2026-04-06T01:35:03Z

crates/capi/src/object.rs

+use std::sync::LazyLock;
+
+pub struct PyTypeObject {
+    ty: LazyLock<&'static Py<PyType>>,


This will cause significant performance draw down.
PyType also already has flags inside.
Will you create all the new type wrapper for every static types? That will not work.

Will you create all the new type wrapper for every static types? That will not work.

I was just trying to get the static type objects working. These are static exported pointers so I came up with this. If you have a better idea to statically create pointers to the TypeZoo, please, I'm very open to suggestions.

PyType also already has flags inside.

I got rid of the flags field in PyTypeObject, and map the RustPythons PyTypeFlags to the format PyType_GetFlags expects. Note that this does currently not map all the available flags.

The static type pointers are now dynamically set at runtime. They are now basically a static pointer to a Py<PyType> instance. What do you think?

Good try 👍 That will be better.

youknowone · 2026-04-06T01:47:39Z

crates/capi/src/pystate.rs

+use std::cell::RefCell;
+
+thread_local! {
+    static VM: RefCell<Option<ThreadedVirtualMachine>> = const { RefCell::new(None) };


C API must be simple API set. It must not contains any state ideally. And never the VM.

The VM_CURRENT static in vm/thread.rs is scoped. So I cannot set and unset it in PyGILState_Ensure/ PyGILState_Release. (These methods have nothing to do with the GIL in free-threaded world despite GIL in the name)

I could use VM_STACK instead... 🤔

Indeed. Let's try to expose it

VM_STACK only stores pointers (NonNull<VirtualMachine>) to a vm, so I need some other place to store it. What do you recommend?

bschoenmaeckers · 2026-04-07T07:21:51Z

Ideally c api must be very thin wrappers to RustPython features. "code" must not be included in this crate.

I wanted to create as little of modificaties to the existing code base without significant motivation. But I definitely agree with you.

bschoenmaeckers · 2026-04-07T07:37:43Z

crates/capi/src/pylifecycle.rs

+    let (response_tx, response_rx) = mpsc::channel();
+    tx.send(response_tx).expect("Failed to send VM request");
+    response_rx.recv().expect("Failed to receive VM response")
+}


This is also somewhat cursed. I want to be able to create ThreadedVirtualMachine scoped to the thread where PyGILState_Ensure is called. This may be from a different thread than where the main interpreter is running. So I need a way to share the Interpreter between different threads.

youknowone · 2026-04-08T08:15:02Z

I wanted to create as little of modificaties to the existing code base without significant motivation. But I definitely agree with you.

That's actually a good view in general. In this case, C APIs must be jsut C API for real RustPython features.
If RustPython lacks features to support C API, we have to consider to add it inside the core crate. Not 100% sure. C API is broad and something must not be fit in this way. Let's investigate it if we can make it or not.

bschoenmaeckers added 11 commits April 4, 2026 17:21

export c api

54156f6

type mapping v1

2250a1c

Type mapping v2

bc96211

Implement type flags

46307f6

Implement PyUnicode_FromStringAndSize

b08d41a

Do not use C-unwind

34c13f4

implement PyLong_AsLong

24b2c63

Add with_vm helper

384b378

Move PyThreadState to pystate.rs

f49e859

Basic multi-threading support

e18eef7

Cleanup

565bf5e

Attach thread in Py_InitializeEx

85b2928

youknowone requested changes Apr 6, 2026

View reviewed changes

bschoenmaeckers commented Apr 7, 2026

View reviewed changes

bschoenmaeckers force-pushed the c-api branch from a717679 to 6d20c1c Compare April 7, 2026 09:21

Map RustPython type flags to CPython type flags

c603d37

bschoenmaeckers force-pushed the c-api branch from 6d20c1c to c603d37 Compare April 7, 2026 09:24

bschoenmaeckers added 5 commits April 7, 2026 15:47

Use *const Py<PyType> instead of *mut PyTypeObject

cee9403

Implement PyUnicode_AsUTF8AndSize

0a50fcc

Implement PyType_GetName

f588979

Implement alternative PyLong constructors

edaca0c

Move init_static_type_pointers to pylifecycle.rs

e4c8bd7

bschoenmaeckers mentioned this pull request Apr 7, 2026

Use Py_GetConstantBorrowed for python True & False on abi3 PyO3/pyo3#5956

Merged

bschoenmaeckers force-pushed the c-api branch from 137617c to 7ea97d1 Compare April 7, 2026 19:01

Add support for None, True, False, Ellipsis & NotImplemented

26c2777

bschoenmaeckers force-pushed the c-api branch from 7ea97d1 to 26c2777 Compare April 7, 2026 19:06

bschoenmaeckers added 3 commits April 8, 2026 11:51

Add support for PyBytes

09466e2

Add basic exception support

94cc39b

Add import support

2ef64d2

bschoenmaeckers force-pushed the c-api branch from ed320f5 to 2ef64d2 Compare April 8, 2026 13:48

Add basic call support

55b1e14

bschoenmaeckers force-pushed the c-api branch 3 times, most recently from 16f8074 to 850cd07 Compare April 8, 2026 19:41

Use pyo3 as test harness

85b1ed4

bschoenmaeckers force-pushed the c-api branch from 850cd07 to 85b1ed4 Compare April 8, 2026 19:45

Conversation

bschoenmaeckers commented Apr 5, 2026

Uh oh!

coderabbitai bot commented Apr 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

youknowone commented Apr 5, 2026

Uh oh!

bschoenmaeckers commented Apr 5, 2026

Uh oh!

youknowone left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bschoenmaeckers Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bschoenmaeckers commented Apr 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

youknowone commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

coderabbitai bot commented Apr 5, 2026 •

edited

Loading

bschoenmaeckers Apr 7, 2026 •

edited

Loading

youknowone commented Apr 8, 2026 •

edited

Loading