Skip to content

Latest commit

 

History

History

oss-cpp

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 

README.md

OSS C++ SDK backend (experimental, read-only)

An opt-in, read-only FileIO backend using Alibaba Cloud OSS C++ SDK v2 for HEAD, Range GET and paginated LIST. OpenDAL remains the default.

Build

Install the SDK, then build the bridge and enable the Rust feature:

cmake -S integrations/oss-cpp -B build/oss-cpp \
  -DCMAKE_PREFIX_PATH=/path/to/oss-sdk-install
cmake --build build/oss-cpp
cargo build -p paimon --features storage-oss-cpp

The Python binding already includes this feature, but neither the bridge nor the SDK is bundled in its wheel. Install both on each worker. Tested on Linux with SDK 0.2.0 (c13a82038b0696f99c32f4e86b623ffcc9e62140).

Configure

Set the native table/catalog FileIO options:

fs.oss.impl = cpp
fs.oss.cpp.library.path = /absolute/path/liboss_cpp_bridge.so
fs.oss.endpoint = https://oss-cn-shanghai-internal.aliyuncs.com
fs.oss.region = cn-shanghai

Use the existing fs.oss.accessKeyId, fs.oss.accessKeySecret and optional fs.oss.securityToken credentials. REST catalog credentials still refresh; static FileIO credentials do not.

Python bindings also discover liboss_cpp_bridge.so (Linux) or liboss_cpp_bridge.dylib (macOS) beside pypaimon_oss_cpp/__init__.py. An explicit fs.oss.cpp.library.path takes precedence. The companion package is not yet published.

Option Default
fs.oss.cpp.max.concurrent.requests 8
fs.oss.cpp.connect.timeout-ms 10000
fs.oss.cpp.request.timeout-ms 30000
fs.oss.cpp.retry.max-attempts 3
fs.oss.cpp.path-style false

The SDK handles retries. Timeouts are per connection/read-write, not end-to-end. Concurrency is bounded per FileIO, not per process; cancelled calls retain a slot until the SDK returns. Use spawned workers: SDK access after fork is rejected. Mutations and presigning are unsupported. Python-side image downloads are unaffected.

Test

cargo test -p paimon --features storage-oss-cpp --lib io::storage_oss_cpp

OSS_CPP_BRIDGE_LIBRARY="$PWD/build/oss-cpp/liboss_cpp_bridge.so" \
  cargo test -p paimon --features storage-oss-cpp --lib io::storage_oss_cpp \
  -- --ignored

The ignored tests use the real SDK, fake credentials and a local HTTP server; no cloud account is needed.