An opt-in, read-only FileIO backend using Alibaba Cloud OSS C++ SDK v2 for HEAD, Range GET and paginated LIST. OpenDAL remains the default.
Install the SDK, then build the bridge and enable the Rust feature:
cmake -S integrations/oss-cpp -B build/oss-cpp \
-DCMAKE_PREFIX_PATH=/path/to/oss-sdk-install
cmake --build build/oss-cpp
cargo build -p paimon --features storage-oss-cppThe Python binding already includes this feature, but neither the bridge nor
the SDK is bundled in its wheel. Install both on each worker. Tested on Linux
with SDK 0.2.0 (c13a82038b0696f99c32f4e86b623ffcc9e62140).
Set the native table/catalog FileIO options:
fs.oss.impl = cpp
fs.oss.cpp.library.path = /absolute/path/liboss_cpp_bridge.so
fs.oss.endpoint = https://oss-cn-shanghai-internal.aliyuncs.com
fs.oss.region = cn-shanghai
Use the existing fs.oss.accessKeyId, fs.oss.accessKeySecret and optional
fs.oss.securityToken credentials. REST catalog credentials still refresh;
static FileIO credentials do not.
Python bindings also discover liboss_cpp_bridge.so (Linux) or
liboss_cpp_bridge.dylib (macOS) beside pypaimon_oss_cpp/__init__.py.
An explicit fs.oss.cpp.library.path takes precedence. The companion package
is not yet published.
| Option | Default |
|---|---|
fs.oss.cpp.max.concurrent.requests |
8 |
fs.oss.cpp.connect.timeout-ms |
10000 |
fs.oss.cpp.request.timeout-ms |
30000 |
fs.oss.cpp.retry.max-attempts |
3 |
fs.oss.cpp.path-style |
false |
The SDK handles retries. Timeouts are per connection/read-write, not end-to-end. Concurrency is bounded per FileIO, not per process; cancelled calls retain a slot until the SDK returns. Use spawned workers: SDK access after fork is rejected. Mutations and presigning are unsupported. Python-side image downloads are unaffected.
cargo test -p paimon --features storage-oss-cpp --lib io::storage_oss_cpp
OSS_CPP_BRIDGE_LIBRARY="$PWD/build/oss-cpp/liboss_cpp_bridge.so" \
cargo test -p paimon --features storage-oss-cpp --lib io::storage_oss_cpp \
-- --ignoredThe ignored tests use the real SDK, fake credentials and a local HTTP server; no cloud account is needed.