Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 34 additions & 11 deletions mkdocs/docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,20 +189,43 @@ For the FileIO there are several configuration options available:

### Alibaba Cloud Object Storage Service (OSS)

<!-- markdown-link-check-disable -->

PyIceberg uses [S3FileSystem](https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html) class to connect to OSS bucket as the service is [compatible with S3 SDK](https://www.alibabacloud.com/help/en/oss/developer-reference/use-amazon-s3-sdks-to-access-oss) as long as the endpoint is addressed with virtual hosted style.

| Key | Example | Description |
| -------------------- | ------------------- | ------------------------------------------------ |
| s3.endpoint | <https://s3.oss-your-bucket-region.aliyuncs.com/> | Configure an endpoint of the OSS service for the FileIO to access. Be sure to use S3 compatible endpoint as given in the example. |
| s3.access-key-id | admin | Configure the static access key id used to access the FileIO. |
| s3.secret-access-key | password | Configure the static secret access key used to access the FileIO. |
| s3.session-token | AQoDYXdzEJr... | Configure the static session token used to access the FileIO. |
| s3.force-virtual-addressing | True | Whether to use virtual addressing of buckets. This is set to `True` by default as OSS can only be accessed with virtual hosted style address. |
| s3.anonymous | True | Configure whether to use anonymous connection. If False (default), uses key/secret if configured or standard AWS configuration methods. |
| Key | Example | Description |
| ---------------------------- | -------------------------------------------------- | ----------- |
| s3.endpoint | <https://s3.oss-your-bucket-region.aliyuncs.com/> | Configure an endpoint of the OSS service for the FileIO to access. Be sure to use S3 compatible endpoint as given in the example. |
| s3.access-key-id | admin | Configure the static access key id used to access the FileIO. |
| s3.secret-access-key | password | Configure the static secret access key used to access the FileIO. |
| s3.session-token | AQoDYXdzEJr... | Configure the static session token used to access the FileIO. |
| s3.force-virtual-addressing | True | Whether to use virtual addressing of buckets. This is set to `True` by default as OSS can only be accessed with virtual hosted style address. |
| s3.anonymous | True | Configure whether to use anonymous connection. If False (default), uses key/secret if configured or standard AWS configuration methods. |

<!-- markdown-link-check-enable-->
### Tencent Cloud Object Storage (COS)

Tencent Cloud Object Storage (COS) is S3-compatible and can be used with PyIceberg using the existing S3FileIO / PyArrowFileIO implementation.

No additional dependencies are required.

| Key | Example | Description |
| -------------------- | ---------------------------------------- | ----------- |
| s3.endpoint | <https://cos.ap-guangzhou.myqcloud.com> | Tencent COS S3-compatible endpoint |
| s3.access-key-id | admin | Access key for COS |
| s3.secret-access-key | password | Secret key for COS |
| s3.session-token | AQoDYXdzEJr... | Optional session token |

#### Example

```python
catalog = load_catalog(
**{
"s3.endpoint": "https://cos.ap-guangzhou.myqcloud.com",
"s3.access-key-id": "admin",
"s3.secret-access-key": "password",
}
)
```

<!-- markdown-link-check-disable -->

### Hugging Face

Expand Down