Skip to content

Commit 4ac38ca

Browse files
yagagagagayagagagaga
authored
[typo](docs) add a python example for stream load. (apache#20697)
1 parent 550584e commit 4ac38ca

3 files changed

Lines changed: 118 additions & 0 deletions

File tree

‎docs/en/docs/data-operate/import/import-way/stream-load-manual.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -382,6 +382,22 @@ Cluster situation: The concurrency of Stream load is not affected by cluster siz
382382
curl --location-trusted -u user:password -T /home/store_sales -H "label:abc" http://abc.com:8030/api/bj_sales/store_sales/_stream_load
383383
```
384384
385+
### Coding with StreamLoad
386+
387+
You can initiate HTTP requests for Stream Load using any language. Before initiating HTTP requests, you need to set several necessary headers:
388+
389+
```http
390+
Content-Type: text/plain; charset=UTF-8
391+
Expect: 100-continue
392+
Authorization: Basic <Base64 encoded username and password>
393+
```
394+
395+
`<Base64 encoded username and password>`: a string consist with Doris's `username`, `:` and `password` and then do a base64 encode.
396+
397+
Additionally, it should be noted that if you directly initiate an HTTP request to FE, as Doris will redirect to BE, some frameworks will remove the `Authorization` HTTP header during this process, which requires manual processing.
398+
399+
Doris provides StreamLoad examples in three languages: [Java](https://github.com/apache/doris/tree/master/samples/stream_load/java), [Go](https://github.com/apache/doris/tree/master/samples/stream_load/go), and [Python](https://github.com/apache/doris/tree/master/samples/stream_load/python) for reference.
400+
385401
## Common Questions
386402

387403
* Label Already Exists

‎docs/zh-CN/docs/data-operate/import/import-way/stream-load-manual.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -397,6 +397,22 @@ timeout = 1000s 等于 10G / 10M/s
397397
curl --location-trusted -u user:password -T /home/store_sales -H "label:abc" http://abc.com:8030/api/bj_sales/store_sales/_stream_load
398398
```
399399

400+
### 使用代码调用 StreamLoad
401+
402+
你可以使用任意代码发起 http 请求进行 Stream Load，在发起 http 请求前，需要设置几个必要的 Header：
403+
404+
```http
405+
Content-Type: text/plain; charset=UTF-8
406+
Expect: 100-continue
407+
Authorization: Basic <base64编码后的用户名密码>
408+
```
409+
410+
其中，`<base64编码后的用户名密码>`是指 Doris 的`username`+`:`+`password`拼接成的字符串进行 base64 编码后得到的值。
411+
412+
另外，需要注意的是，如果你直接对 FE 发起 http 请求，由于 Doris 会重定向到 BE，在这个过程中，某些框架会把`Authorization`这个 http Header 移除掉，这个时候需要你进行手动处理。
413+
414+
Doris 提供了 [Java](https://github.com/apache/doris/tree/master/samples/stream_load/java)、[Go](https://github.com/apache/doris/tree/master/samples/stream_load/go)、[Python](https://github.com/apache/doris/tree/master/samples/stream_load/python) 三种语言的 StreamLoad Example 供参考。
415+
400416
## 常见问题
401417

402418
- Label Already Exists

‎samples/stream_load/python/DorisStreamLoad.py‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
# coding=utf-8
2+
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing,
14+
# software distributed under the License is distributed on an
15+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
# KIND, either express or implied. See the License for the
17+
# specific language governing permissions and limitations
18+
# under the License.
19+
20+
import requests
21+
from requests.auth import HTTPBasicAuth
22+
import base64
23+
24+
# This script is a python demo for doris stream load
25+
#
26+
# How to use:
27+
# 1. create a table in doris with any mysql client
28+
# CREATE TABLE `db0`.`t_user` (
29+
# `id` int,
30+
# `name` string
31+
# )
32+
# DUPLICATE KEY(`id`)
33+
# DISTRIBUTED BY HASH(`id`) BUCKETS 1
34+
# PROPERTIES (
35+
# "replication_num" = "1"
36+
# );
37+
#
38+
# 2. change the Doris cluster, db, user config in this class
39+
#
40+
# 3. run this script, you should see the following output:
41+
#
42+
# 200 OK
43+
# {
44+
# "TxnId": 14017,
45+
# "Label": "2486da70-94bb-47cc-a810-70791add2b8c",
46+
# "TwoPhaseCommit": "false",
47+
# "Status": "Success",
48+
# "Message": "OK",
49+
# "NumberTotalRows": 2,
50+
# "NumberLoadedRows": 2,
51+
# "NumberFilteredRows": 0,
52+
# "NumberUnselectedRows": 0,
53+
# "LoadBytes": 13,
54+
# "LoadTimeMs": 54,
55+
# "BeginTxnTimeMs": 1,
56+
# "StreamLoadPutTimeMs": 12,
57+
# "ReadDataTimeMs": 0,
58+
# "WriteDataTimeMs": 11,
59+
# "CommitAndPublishTimeMs": 28
60+
# }
61+
if __name__ == '__main__':
62+
database, table = 'db0', 't_user'
63+
username, password = 'root', ''
64+
url = 'http://127.0.0.1:8030/api/%s/%s/_stream_load' % (database, table)
65+
headers = {
66+
'Content-Type': 'text/plain; charset=UTF-8',
67+
# 'label': 'your_custom_label',
68+
'format': 'csv',
69+
"column_separator": ',',
70+
'Expect': '100-continue',
71+
# 'Authorization': 'Basic ' + base64.b64encode((username + ':' + password).encode('utf-8')).decode('ascii')
72+
}
73+
auth = HTTPBasicAuth(username, password)
74+
session = requests.sessions.Session()
75+
session.should_strip_auth = lambda old_url, new_url: False # Don't strip auth
76+
77+
data='1,Tom\n2,Jelly'
78+
79+
resp = session.request(
80+
'PUT', url=url,
81+
data=data, # open('/path/to/your/data.csv', 'rb'),
82+
headers=headers, auth=auth
83+
)
84+
85+
print(resp.status_code, resp.reason)
86+
print(resp.text)

0 commit comments

Comments
 (0)