Which issue does this PR close?

Related to #1612. This PR does not close it, but provides the FFI query planner and codec plumbing that a datafusion-distributed integration can build on.

Rationale for this change

Extension libraries (for example distributed execution engines) need to supply their own QueryPlanner to a SessionContext without compiling against the datafusion-python crate. This PR exposes the query planner over the FFI boundary, following the same PyCapsule pattern used for table providers and catalogs.

Supporting a foreign planner also surfaced a codec problem: a query can involve three independent native libraries (datafusion-python, a provider library, and a planner library), and each library needs its extension codecs active on the session at the same time. Previously, installing a logical or physical extension codec replaced the prior codec, so the second library's install silently discarded the first — plans then failed later with a confusing decode error. Codecs now compose.

What changes are included in this PR?

FFI query planner

• SessionContext.with_query_planner(planner) installs a planner exported via a __datafusion_query_planner__ PyCapsule, preserving existing session state and codec settings.
• SessionContext.__datafusion_query_planner__() exports the current planner so another planner can wrap it as an explicit fallback (a session holds exactly one planner; layering is explicit delegation).
• A RuntimeAwareQueryPlanner adapter binds foreign planners to the Tokio runtime owned by datafusion-python.
• New example crate datafusion-ffi-query-planner-example demonstrating a real three-library plan exchange (host, provider library, planner library as separate cdylibs), including session config transfer via SessionConfig.with_extension.

Composable extension codecs

• with_logical_extension_codec / with_physical_extension_codec now prepend to a codec chain instead of replacing the prior codec. The most recently installed codec is consulted first, falling through codec by codec to DataFusion's default codec. A codec signals "not mine" by returning an error.
• Encoding runs each codec against a scratch buffer so failed attempts leave no partial bytes, and treats Ok-with-no-bytes (encode by name) as "no opinion" so later codecs still get a chance.
• When every codec in the chain fails, the errors are aggregated so the owning codec's diagnostic is not masked by the default codec's generic error.
• Fixed a latent bug where installing a codec silently reset python_udf_inlining back to enabled.

Documentation and tests

• docs/source/contributor-guide/ffi.md gains sections on composable codecs (family-prefix discipline, registration order between libraries no longer matters) and planner layering (install all codecs before exporting or chaining planners, since a planner capsule captures the codecs at export time), including a full three-library registration recipe.
• Rust unit tests for chain dispatch semantics; Python integration tests for codec composition on both layers plus an end-to-end three-library test with composed codecs under a foreign planner.

Are there any user-facing changes?

Yes:

• New APIs: SessionContext.with_query_planner and SessionContext.__datafusion_query_planner__.
• Behavior change: with_logical_extension_codec / with_physical_extension_codec now compose with previously installed codecs instead of replacing them. Sessions that install a single codec are unaffected.
• These methods no longer reset the with_python_udf_inlining setting.
• New example package datafusion-ffi-query-planner-example in the examples folder (not shipped in the wheel).

🤖 Generated with Claude Code