Which issue does this PR close?

Related to #1612. This PR does not close it, but provides the FFI query planner and codec plumbing that a datafusion-distributed integration can build on.

Rationale for this change

Extension libraries (for example distributed execution engines) need to supply their own QueryPlanner to a SessionContext without compiling against the datafusion-python crate. This PR exposes the query planner over the FFI boundary, following the same PyCapsule pattern used for table providers and catalogs.

Supporting a foreign planner also surfaced a codec problem: a query can involve three independent native libraries (datafusion-python, a provider library, and a planner library), and each library needs its extension codecs active on the session at the same time. Previously, installing a logical or physical extension codec replaced the prior codec, so the second library's install silently discarded the first — plans then failed later with a confusing decode error. Codecs now compose.

What changes are included in this PR?

FFI query planner

• SessionContext.with_query_planner(planner) installs a planner exported via a __datafusion_query_planner__ PyCapsule, preserving existing session state and codec settings.
• SessionContext.__datafusion_query_planner__() exports the current planner so another planner can wrap it as an explicit fallback (a session holds exactly one planner; layering is explicit delegation).
• A RuntimeAwareQueryPlanner adapter binds foreign planners to the Tokio runtime owned by datafusion-python.
• New example crate datafusion-ffi-query-planner-example demonstrating a real three-library plan exchange (host, provider library, planner library as separate cdylibs), including session config transfer via SessionConfig.with_extension.

Composable extension codecs

• with_logical_extension_codec / with_physical_extension_codec now prepend to a codec chain instead of replacing the prior codec. The most recently installed codec is consulted first, falling through codec by codec to DataFusion's default codec. A codec signals "not mine" by returning an error.
• Encoding runs each codec against a scratch buffer so failed attempts leave no partial bytes, and treats Ok-with-no-bytes (encode by name) as "no opinion" so later codecs still get a chance.
• When every codec in the chain fails, the errors are aggregated so the owning codec's diagnostic is not masked by the default codec's generic error.
• Fixed a latent bug where installing a codec silently reset python_udf_inlining back to enabled.

Atomic extension installation: SessionContext.with_extensions

Working on the Ballista integration showed that chaining the low-level with_* methods is easy to get wrong: FFI codecs and planners carry a weak task-context provider bound to the context they were created against, so components can end up bound to an intermediate context that is later garbage collected. Queries then fail with TaskContextProvider went out of scope over FFI boundary, or worse, silently read stale session state.

• SessionContext.with_extensions(*extensions) installs one or more extension bundles atomically. Each bundle implements the new __datafusion_session_extension__(ctx) protocol: it receives the destination context, creates fresh components bound to that exact context, and returns them as a SessionExtensionComponents (new public dataclass; SessionExtensionExportable is the matching typing protocol).
• Internally one destination context is created, factories bind against it, all codecs are installed, and the planner is bound against the final codec chains in a single state write through that context's own state_ref(). No context is derived after any provider is created, so every weak provider targets the returned context.
• Validation happens before any state change: a failing factory or invalid capsule leaves the source context untouched.
• If no bundle supplies a planner, an existing FFI planner on the source context is rebound to the new codec chains; at most one bundle may supply a planner.
• The ownership contract is documented and tested: the returned context is the strong owner, and a DataFrame outliving it fails with a clean out-of-scope error rather than crashing.
• MyPlannerExtension in the example crate is a complete Rust implementation of the protocol, including extracting the host's task-context provider from the supplied context. Its codecs record the config value they resolve through the weak provider, letting tests prove the provider targets the returned context rather than the source.

Documentation and tests

• docs/source/contributor-guide/ffi.md gains sections on composable codecs (family-prefix discipline, registration order between libraries no longer matters), planner layering (install all codecs before exporting or chaining planners, since a planner capsule captures the codecs at export time), and extension bundles (with_extensions documented as the preferred API, low-level chaining kept as advanced), including a full three-library registration recipe.
• Rust unit tests for chain dispatch semantics; Python integration tests for codec composition on both layers plus an end-to-end three-library test with composed codecs under a foreign planner.
• with_extensions tests cover validation, codec precedence between bundles, planner rebinding, identity (two live contexts with diverged config), GC of the source context and bundle objects, and failure atomicity.

Are there any user-facing changes?

Yes:

• New APIs: SessionContext.with_query_planner, SessionContext.__datafusion_query_planner__, SessionContext.with_extensions, SessionExtensionComponents, and SessionExtensionExportable.
• Behavior change: with_logical_extension_codec / with_physical_extension_codec now compose with previously installed codecs instead of replacing them. Sessions that install a single codec are unaffected.
• These methods no longer reset the with_python_udf_inlining setting.
• New example package datafusion-ffi-query-planner-example in the examples folder (not shipped in the wheel).

🤖 Generated with Claude Code