Skip to content

Add support for creating functions with the c-api#7984

Open
bschoenmaeckers wants to merge 5 commits into
RustPython:mainfrom
bschoenmaeckers:c-api-functions
Open

Add support for creating functions with the c-api#7984
bschoenmaeckers wants to merge 5 commits into
RustPython:mainfrom
bschoenmaeckers:c-api-functions

Conversation

@bschoenmaeckers
Copy link
Copy Markdown
Contributor

@bschoenmaeckers bschoenmaeckers commented May 26, 2026

Summary by CodeRabbit

  • New Features

    • Added C API-compatible support for creating and invoking native Python-callable methods, including multiple calling conventions and bound-self support.
  • Bug Fixes

    • Improved validation and UTF-8 handling for method names and documentation to reduce runtime errors.
  • Chores

    • Updated dependencies and internal method-construction plumbing for more reliable native-callable method creation.

Review Change Stack

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 26, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: 1d76ac84-808d-42a0-90bc-1582d1e52cf9

📥 Commits

Reviewing files that changed from the base of the PR and between 30d3f4d and cc4b1c7.

📒 Files selected for processing (1)
  • crates/capi/src/methodobject.rs

📝 Walkthrough

Walkthrough

Adds a new C-API methodobject module with PyMethodDef and calling-convention adapters, exports PyCMethod_New/PyCFunction_New/PyCFunction_NewEx, and updates the VM builder to accept/thread an optional self (zelf) into constructed native function objects.

Changes

C-API Method Object Implementation

Layer / File(s) Summary
Method types and dispatch orchestration
crates/capi/Cargo.toml, crates/capi/src/lib.rs, crates/capi/src/methodobject.rs (lines 1–137)
Adds bitflags dependency, exposes methodobject module, defines PyMethodDef and PyMethodPointer, and implements build_method_def to validate ml_name/ml_doc, parse ml_flags, enforce abi3 constraints, and choose the appropriate calling-convention wrapper (with self-binding when present).
Calling convention adapters and utilities
crates/capi/src/methodobject.rs (lines 139–266)
Implements unsafe adapters for PyCFunction, PyCFunctionWithKeywords, PyCFunctionFast, and PyCFunctionFastWithKeywords; packs C-API args into VM representations, handles extracting/omitting self, and converts returned native pointers into PyResult with ret_ptr_to_pyresult and take_self_arg.
Public constructors and tests
crates/capi/src/methodobject.rs (lines 268–369)
Exports PyCMethod_New, PyCFunction_New, and PyCFunction_NewEx that call build_method_def(...).build_function(vm, ...) to produce callable objects; includes a disabled #[cfg(false)] test module.

VM Method Construction Integration

Layer / File(s) Summary
Method builder signature and VM call site
crates/vm/src/function/method.rs, crates/vm/src/vm/vm_new.rs
HeapMethodDef::build_function now takes zelf: Option<PyObjectRef> and assigns it to the resulting PyNativeFunction; VirtualMachine::new_function updated to call def.build_function(self, None).

Sequence Diagram

sequenceDiagram
  participant C_Ext as C extension
  participant capi as capi::PyCFunction_New*
  participant Orchestrator as build_method_def
  participant VM as VirtualMachine
  participant Native as PyNativeFunction

  C_Ext->>capi: call PyCFunction_New(ml, self)
  capi->>Orchestrator: build_method_def(ml, has_self)
  Orchestrator->>VM: .build_function(vm, optional_zelf)
  VM->>Native: create PyNativeFunction(zelf assigned)
  Native-->>C_Ext: callable PyObject pointer
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Suggested reviewers

  • youknowone
  • ShaharNaveh

Poem

A rabbit hops through method pointers bright,
Wrapping C calls in Rust by moonlight,
Names checked, flags parsed, returns made right,
Zelf tucked softly into native sight,
Callables sprout and dance into midnight 🐰✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title directly describes the main addition: new C-API support for creating functions via PyCMethod_New, PyCFunction_New, and PyCFunction_NewEx in the methodobject module.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/capi/src/methodobject.rs`:
- Around line 91-114: The VARARGS/VARARGS|KEYWORDS/FASTCALL/FASTCALL|KEYWORDS
branches always call take_self_arg which strips the first positional arg even
when has_self is false; fix by capturing the local has_self (the same way NOARGS
and O do) and either (A) pass has_self into take_self_arg so the adapters
(call_function, call_fast_function, call_function_with_keywords,
call_fast_function_with_keywords) can honor it, or (B) branch each case on
has_self and create different callable signatures (one that calls take_self_arg
when has_self==true and one that forwards args untouched when false) before
calling vm.ctx.new_method_def with the appropriate callable.
- Around line 78-90: The NOARGS branch (PyMethodFlags::NOARGS) incorrectly
creates and passes an empty tuple for args when has_self is true; change the
closure created in that branch (the callable that takes zelf: PyObjectRef) to
call call_function with None for the args parameter (i.e., pass None so it
becomes NULL) instead of Some(vec![zelf]) wrapped as an empty tuple, or
alternatively implement a small helper (e.g., call_function_noargs) that always
forwards NULL for the args parameter; update the callable and keep using
vm.ctx.new_method_def(name, callable, flags, doc) so behavior matches the
has_self=false path.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: 9893aed3-e881-4cbe-841a-48f858dae982

📥 Commits

Reviewing files that changed from the base of the PR and between dcb273b and eba367d.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (5)
  • crates/capi/Cargo.toml
  • crates/capi/src/lib.rs
  • crates/capi/src/methodobject.rs
  • crates/vm/src/function/method.rs
  • crates/vm/src/vm/vm_new.rs

Comment thread crates/capi/src/methodobject.rs
Comment on lines +91 to +114
PyMethodFlags::VARARGS => {
let callable = move |args: PosArgs, vm: &VirtualMachine| unsafe {
call_function(vm, method, flags, Some(args))
};
Ok(vm.ctx.new_method_def(name, callable, flags, doc))
},
PyMethodFlags::VARARGS | PyMethodFlags::KEYWORDS => {
let callable = move | args: FuncArgs, vm: &VirtualMachine| unsafe {
call_function_with_keywords(vm, method, flags, args)
};
Ok(vm.ctx.new_method_def(name, callable, flags, doc))
},
PyMethodFlags::FASTCALL | PyMethodFlags::KEYWORDS => {
let callable = move |args: FuncArgs, vm: &VirtualMachine| unsafe {
call_fast_function_with_keywords(vm, method, flags, args)
};
Ok(vm.ctx.new_method_def(name, callable, flags, doc))
},
PyMethodFlags::FASTCALL => {
let callable = move |args: PosArgs, vm: &VirtualMachine| unsafe {
call_fast_function(vm, method, flags, args)
};
Ok(vm.ctx.new_method_def(name, callable, flags, doc))
},
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical | 🏗️ Heavy lift

Argument corruption when has_self=false for VARARGS/FASTCALL conventions.

For VARARGS, VARARGS|KEYWORDS, FASTCALL, and FASTCALL|KEYWORDS, the wrapper always calls take_self_arg, which removes the first positional argument unless the STATIC flag is set. However, when has_self=false:

  1. The VM does not prepend any self (since zelf=None)
  2. The callable receives the user's original arguments
  3. take_self_arg incorrectly removes the first user argument

Compare to NOARGS (lines 78-90) and O (lines 115-130) which correctly branch on has_self to create different callable signatures.

The fix is to capture has_self and pass it to take_self_arg, or branch on has_self for each calling convention like NOARGS and O do.

Proposed fix: Update take_self_arg to consider has_self

One approach - pass has_self to the call adapters:

 unsafe fn call_function<A: Into<FuncArgs>>(
     vm: &VirtualMachine,
     method: PyMethodPointer,
     flags: PyMethodFlags,
     args: Option<A>,
+    has_self: bool,
 ) -> PyResult {
     let f = unsafe { method.PyCFunction };
     let (slf, arg_tuple) = if let Some(mut args) = args.map(Into::into) {
-        let slf = take_self_arg(&mut args, flags);
+        let slf = take_self_arg(&mut args, flags, has_self);
         let arg_tuple = vm.ctx.new_tuple(args.args);
         (slf, Some(arg_tuple))
     } else {
         (None, None)
     };
     // ...
 }

-fn take_self_arg(args: &mut FuncArgs, flags: PyMethodFlags) -> Option<PyObjectRef> {
-    if flags.contains(PyMethodFlags::STATIC) {
+fn take_self_arg(args: &mut FuncArgs, flags: PyMethodFlags, has_self: bool) -> Option<PyObjectRef> {
+    if !has_self || flags.contains(PyMethodFlags::STATIC) {
         None
     } else {
         args.take_positional()
     }
 }

Then update the VARARGS case to capture has_self:

 PyMethodFlags::VARARGS => {
-    let callable = move |args: PosArgs, vm: &VirtualMachine| unsafe {
-        call_function(vm, method, flags, Some(args))
+    let callable = move |args: PosArgs, vm: &VirtualMachine| unsafe {
+        call_function(vm, method, flags, Some(args), has_self)
     };
     Ok(vm.ctx.new_method_def(name, callable, flags, doc))
 },

Apply similar changes to VARARGS|KEYWORDS, FASTCALL, and FASTCALL|KEYWORDS branches.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/capi/src/methodobject.rs` around lines 91 - 114, The
VARARGS/VARARGS|KEYWORDS/FASTCALL/FASTCALL|KEYWORDS branches always call
take_self_arg which strips the first positional arg even when has_self is false;
fix by capturing the local has_self (the same way NOARGS and O do) and either
(A) pass has_self into take_self_arg so the adapters (call_function,
call_fast_function, call_function_with_keywords,
call_fast_function_with_keywords) can honor it, or (B) branch each case on
has_self and create different callable signatures (one that calls take_self_arg
when has_self==true and one that forwards args untouched when false) before
calling vm.ctx.new_method_def with the appropriate callable.

Copy link
Copy Markdown
Contributor

@JamesClarke7283 JamesClarke7283 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice groundwork on the C API surface. Build is clean and clippy passes the workspace lint set, which already includes pedantic and a fair bit of nursery. Most of what's left is semantic.

1. VARARGS / FASTCALL paths strip a real user argument as self when the function has no bound self.

PyMethodFlags::VARARGS => {
    let callable = move |args: PosArgs, vm: &VirtualMachine| unsafe {
        call_function(vm, method, flags, Some(args))
    };
    ...
},
fn take_self_arg(args: &mut FuncArgs, flags: PyMethodFlags) -> Option<PyObjectRef> {
    if flags.contains(PyMethodFlags::STATIC) { None } else { args.take_positional() }
}

Callable::call for PyNativeFunction only prepends zelf when zelf.is_some() && !STATIC. After PyCFunction_New(ml, NULL) with METH_VARARGS, calling fn(a, b) makes take_self_arg consume a as self and the C function sees args = (b,). Silent data corruption with no error. Same shape for the three other branches at lines 95 to 118. NOARGS and O already split on has_self; the VARARGS / FASTCALL family needs the same treatment, or pass has_self into take_self_arg.

2. module is silently dropped, so __module__ is always None for anything built through this API.

pub unsafe extern "C" fn PyCMethod_New(
    ml: *mut PyMethodDef,
    slf: *mut PyObject,
    _module: *mut PyObject,
    _cls: *mut PyTypeObject,
) -> *mut PyObject {

CPython does op->m_module = Py_XNewRef(module) and exposes it via __module__. Here it is bound to _module and never read. HeapMethodDef::build_function also lacks a module parameter, so even threading it through would not reach PyNativeFunction.module. Breaks inspect.getmodule, pickling and traceback module attribution.

3. assert! and .expect() across an extern \"C\" boundary.

with_vm(|vm| -> PyResult {
    assert!(
        _cls.is_null(),
        \"PyCMethod_New does not support METH_METHOD on abi3\"
    );
vm.take_raised_exception()
    .expect(\"Native function returned NULL, but there was no exception set\")

with_vm does not catch panics, and neither function is extern \"C-unwind\", so a panic here aborts the process. Any caller that passes a non-null cls (the PEP 573 path with METH_METHOD | METH_FASTCALL | METH_KEYWORDS) takes the interpreter down. Likewise for any buggy extension that returns NULL without PyErr_SetString. Both should be vm.new_system_error(...) plus a NULL return, matching _Py_CheckFunctionResult in CPython.

4. PyMethodFlags::from_bits rejects valid CPython flag combinations.

let flags = PyMethodFlags::from_bits(ml.ml_flags as u32)
    .ok_or_else(|| vm.new_system_error(\"PyMethodDef contains unknown flags\"))?;

METH_COEXIST (0x40) and METH_STACKLESS (0x100) are commented out of PyMethodFlags in crates/vm/src/function/method.rs. CPython masks unknown bits and treats them as ignored, so a textbook pattern like {\"__contains__\", impl, METH_O | METH_COEXIST, NULL} (used by several stdlib C modules) fails registration outright. Either switch to from_bits_truncate or declare the bits.

5. CStr::from_ptr(ml.ml_name) with no null check.

let name = unsafe { CStr::from_ptr(ml.ml_name) }
    .to_str()
    .map_err(|_| vm.new_system_error(\"Method name was not valid UTF-8\"))?;

Null ml_name is UB. ml_doc is already guarded with NonNull::new two lines down, so this is also an internal inconsistency. Same guard plus a SystemError.

Minor: the #[cfg(false)] mod tests block at lines 309 to 369 is dead code that references pyo3 types not declared as a dev dependency. Either wire it into CI or drop it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants