Support pluggable AWS credentials for the S3 object store (credentials provider callback) #116

Is your feature request related to a problem or challenge?

ObjectStoreOptions.S3 currently supports two credential modes: static values (accessKeyId / secretAccessKey / sessionToken) passed once at registration, or falling through to object_store's built-in provider chain (IMDS instance profile, ECS task endpoint, EKS pod identity, web identity).

For a long-lived SessionContext — the intended usage pattern for DataFusion — this leaves no working credential story in several common deployments:

• STS-assumed roles: IMDS only serves the instance-profile role. Credentials for an assumed role (cross-account, scoped-down, role chaining) can only be passed statically, and expire after 1–12h with no refresh. Queries then start failing with ExpiredToken until the store is somehow rebuilt.
• AWS Lambda: there is no IMDS endpoint. Credentials arrive via environment variables, which object_store captures once as a StaticCredentialProvider. A SessionContext cached across warm invocations goes stale when the environment credentials rotate underneath it.
• SSO / profiles / credential_process / custom credential vendors: not in object_store's chain at all.

Upstream has explicitly declined to widen the built-in chain: apache/arrow-rs#5143 (use the official AWS SDK) and apache/arrow-rs-object-store#47 (credential_process support) were both closed as not planned. The sanctioned extension point is
AmazonS3Builder::with_credentials(AwsCredentialProvider). Quoting the object_store maintainer in those threads:

"Users can provide their own credential providers […] This could even be using the AWS SDK if they so wish."

"Encouraging the downstreams to expose, or otherwise utilize the object_store credential provider API would avoid this entirely."

"I'd naturally prefer the solution that gives users the most flexibility and avoids needing to revisit this again when someone comes along requesting SSO or similar."

The Java binding does not expose this extension point, so Java users are locked out of everything the AWS SDK for Java's default credential chain already handles. There is direct precedent for fixing this at the binding layer: arrow-rs-object-store#47 was closed because polars exposed a credential provider
callback in its Python API ("Polars exposed it as part of their API and configured it to use boto when available. I'm going to close this…").

Describe the solution you'd like

An S3-specific credentials provider callback on the S3 options builder, bridged to AmazonS3Builder::with_credentials:

/**
 * Supplies current AWS credentials. Invoked by the native layer when its cached
 * credentials approach expiry; implementations must be thread-safe.
 */
@FunctionalInterface
public interface AwsCredentialsProvider {
  AwsCredentials get();
}

public record AwsCredentials(
    String accessKeyId,
    String secretAccessKey,
    Optional<String> sessionToken,
    Optional<Instant> expiration) {}

SessionContext.builder()
    .registerObjectStore(
        ObjectStoreOptions.s3()
            .bucket("my-bucket")
            .credentialsProvider(provider) // new
            .build())
    .build();

// Typical usage delegates to the AWS SDK for Java, which brings the full default chain (SSO, profiles, credential_process, STS auto-refresh, Lambda env) for free:

var sdk = software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.create();
var options = ObjectStoreOptions.s3()
    .bucket("my-bucket")
    .credentialsProvider(() -> AwsCredentials.from(sdk.resolveCredentials()))
    .build();

Native side. A Rust struct holding a JNI GlobalRef to the Java provider implements object_store::CredentialProvider<Credential = AwsCredential> and is passed to AmazonS3Builder::with_credentials in build_s3 (native/src/object_store.rs). The bridge caches the returned credentials keyed on expiration (no expiration
= cache indefinitely) with a min-TTL refresh-ahead check, so the JVM upcall fires roughly once per credential rotation, not per request. The repo already has the required upcall machinery: native/src/udf.rs holds a GlobalRef plus cached bridge class and calls attach_current_thread() to invoke Java from DataFusion
execution threads — the credentials bridge follows the same pattern. (object_store's own TokenCache is pub(crate), so the bridge implements its own small equivalent, as delta-rs does.)

Since registration currently crosses JNI as a serialized protobuf ObjectStoreRegistration, the provider jobject would be passed alongside the message rather than inside it — the same way UDF registration passes the UDF object.

Semantics.

• The provider is called on native (tokio worker) threads; implementations must be thread-safe.
• Returning an expiration opts into refresh: the native layer re-invokes the provider before expiry. Because every request resolves credentials at sign time, rotation is safe mid-query — long scans and multipart uploads spanning a rotation just sign later requests with the newer credentials.
• Mutually exclusive with static accessKeyId/secretAccessKey on the same builder — setting both is a build-time error.
• Java exceptions from the provider are translated into object_store::Error::Generic and surface as query errors.
• The GlobalRef is released when the registered store is dropped.

Non-goals.

• No generic cross-backend credential abstraction: with_credentials is typed per backend (AwsCredential vs GcpCredential bearer tokens), so this API belongs on the S3 builder only.
• GCS is out of scope: service-account keys and the metadata server already self-refresh in object_store. A gcs().credentialsProvider(...) (bearer-token supplier) can follow the same pattern later if a concrete need appears.
• No replication of the AWS credential chain inside the binding — resolving credentials is the AWS SDK for Java's job; the binding only ferries the result. The binding takes no dependency on the AWS SDK; a convenience adapter could be a follow-up.

Describe alternatives you've considered

1. Do nothing / document the built-in chain. Leaves STS-assumed roles, Lambda with a cached context, SSO, and credential_process broken; upstream has frozen the built-in chain (object_store: Use the official AWS SDK for s3 interactions arrow-rs#5143, [object-store]: Implement credential_process support for S3 arrow-rs-object-store#47), so this gap is permanent.
2. Imperative refreshCredentials(newCreds) method. Requires the application to run its own expiry timers, is not atomic with respect to in-flight queries without extra locking in the binding, and puts lifecycle behavior on an immutable options object. The pull-based provider + cache is race-free by
   construction (each request resolves current credentials) and matches the design every AWS SDK converged on.
3. Re-register the object store with new options. Same timer problem, plus undefined behavior for scans already planned against the previously registered store.

Additional context

• Maintainer guidance to downstream bindings: object_store: Use the official AWS SDK for s3 interactions arrow-rs#5143 (comments from 2024-08-29 onward) and [object-store]: Implement credential_process support for S3 arrow-rs-object-store#47.
• Precedent: polars exposed exactly this callback in its Python API, which is what closed arrow-rs-object-store#47 (Allow Overriding Object Store Credential Provider pola-rs/polars#18979).
• Rust-side reference implementation gluing the AWS SDK to a CredentialProvider: delta-rs credentials.rs.
• Existing Rust→Java upcall pattern in this repo: native/src/udf.rs (GlobalRef + attach_current_thread).

I'm happy to implement this if the approach is agreed.