Skip to content

Commit 738d2d2

Browse files
yangzhgyangzhg
authored
[refactor] update parent pom version and optimize build scripts (#7548)
1 parent 9ddcf06 commit 738d2d2

23 files changed

Lines changed: 472 additions & 472 deletions

File tree

‎.github/workflows/build-extension.yml‎

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -58,15 +58,23 @@ jobs:
5858
- name: Build spark connector v2
5959
run: |
6060
thrift --version
61-
cd extension/spark-doris-connector/ && /bin/bash build.sh 2
61+
cd extension/spark-doris-connector/ && /bin/bash build.sh 2.3.4 2.11
6262
6363
- name: Build spark connector v3
6464
run: |
65-
cd extension/spark-doris-connector/ && /bin/bash build.sh 3
65+
cd extension/spark-doris-connector/ && /bin/bash build.sh 3.1.2 2.12
6666
67-
- name: Build flink connector
67+
- name: Build flink connector 1.11
6868
run: |
69-
cd extension/flink-doris-connector/ && /bin/bash build.sh
69+
cd extension/flink-doris-connector/ && /bin/bash build.sh 1.11.6 2.12
70+
71+
- name: Build flink connector 1.12
72+
run: |
73+
cd extension/flink-doris-connector/ && /bin/bash build.sh 1.12.7 2.12
74+
75+
- name: Build flink connector 1.13
76+
run: |
77+
cd extension/flink-doris-connector/ && /bin/bash build.sh 1.13.5 2.12
7078
7179
- name: Build docs
7280
run: |

‎docs/.vuepress/sidebar/en.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -683,6 +683,7 @@ module.exports = [
683683
"subscribe-mail-list",
684684
"feedback",
685685
"how-to-contribute",
686+
"how-to-deploy-to-maven",
686687
"committer-guide",
687688
"commit-format-specification",
688689
"pull-request",

‎docs/.vuepress/sidebar/zh-CN.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -687,6 +687,7 @@ module.exports = [
687687
"subscribe-mail-list",
688688
"feedback",
689689
"how-to-contribute",
690+
"how-to-deploy-to-maven",
690691
"committer-guide",
691692
"commit-format-specification",
692693
"pull-request",

‎docs/en/community/how-to-deploy-to-maven.md‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
---
2+
{
3+
"title": "How to deploy to Maven Central Repository",
4+
"language": "en"
5+
}
6+
---
7+
8+
<!--
9+
Licensed to the Apache Software Foundation (ASF) under one
10+
or more contributor license agreements. See the NOTICE file
11+
distributed with this work for additional information
12+
regarding copyright ownership. The ASF licenses this file
13+
to you under the Apache License, Version 2.0 (the
14+
"License"); you may not use this file except in compliance
15+
with the License. You may obtain a copy of the License at
16+
17+
http://www.apache.org/licenses/LICENSE-2.0
18+
19+
Unless required by applicable law or agreed to in writing,
20+
software distributed under the License is distributed on an
21+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
22+
KIND, either express or implied. See the License for the
23+
specific language governing permissions and limitations
24+
under the License.
25+
-->
26+
27+
## Prepare
28+
29+
### 0. Requirements
30+
31+
1. apache account ID
32+
2. apache password
33+
3. gpg key
34+
35+
### 1. Prepare the local maven environment
36+
37+
1. Generate the master password: `mvn --encrypt-master-password <password>` This password is only used to encrypt other subsequent passwords, and the output is similar to `{VSb+6+76djkH/43...}` and then create` ~/.m2/settings-security.xml` file, the content is
38+
39+
```
40+
<settingsSecurity>
41+
<master>{VSb+6+76djkH/43...}</master>
42+
</settingsSecurity>
43+
```
44+
45+
2. Encrypt apache password: `mvn --encrypt-password <password>` This password is the password of the apache account. The output is similar to the above `{GRKbCylpwysHfV...}` and added in `~/.m2/settings.xml`
46+
```
47+
<servers>
48+
<!-- To publish a snapshot of your project -->
49+
<server>
50+
<id>apache.snapshots.https</id>
51+
<username>yangzhg</username>
52+
<password>{GRKbCylpwysHfV...}</password>
53+
</server>
54+
<!-- To stage a release of your project -->
55+
<server>
56+
<id>apache.releases.https</id>
57+
<username>yangzhg</username>
58+
<password>{GRKbCylpwysHfV...}</password>
59+
</server>
60+
</servers>
61+
```
62+
63+
3. Optional, (you can also pass -Darguments="-Dgpg.passphrase=xxxx" during deployment), add the following content in `~/.m2/settings.xml`, if the profiles tag already exists, this is only required Just add profile to profiles, activeProfiles is the same as above, xxxx is the passphrase of the gpg key
64+
```
65+
<profiles>
66+
<profile>
67+
<id>gpg</id>
68+
<properties>
69+
<gpg.executable>gpg</gpg.executable>
70+
<gpg.passphrase>xxxx</gpg.passphrase>
71+
</properties>
72+
</profile>
73+
</profiles>
74+
<activeProfiles>
75+
<activeProfile>gpg</activeProfile>
76+
</activeProfiles>
77+
```
78+
### Publish to SNAPSHOT
79+
### 1. Deploy flink connector
80+
81+
Switch to the flink connector directory, let’s take flink version 1.11.6 and scalar 2.12 as examples
82+
83+
```
84+
cd incubator-doris/extension/flink-doris-connector
85+
export DORIS_HOME=$PWD/../../
86+
source ${DORIS_HOME}/env.sh
87+
if [ -f ${DORIS_HOME}/custom_env.sh ]; then source ${DORIS_HOME}/custom_env.sh; fi
88+
export FLINK_VERSION="1.11.6"
89+
export SCALA_VERSION="2.12"
90+
mvn deploy
91+
```
92+
93+
94+
95+
### 2. Deploy Spark connector
96+
97+
Switch to the spark connector directory, let’s take spark version 2.3.4 and scalar 2.11 as examples
98+
99+
```
100+
cd incubator-doris/extension/spark-doris-connector
101+
export DORIS_HOME=$PWD/../../
102+
source ${DORIS_HOME}/env.sh
103+
if [ -f ${DORIS_HOME}/custom_env.sh ]; then source ${DORIS_HOME}/custom_env.sh; fi
104+
export SPARK_VERSION="2.3.4"
105+
export SCALA_VERSION="2.11"
106+
mvn deploy
107+
```

‎docs/en/extending-doris/flink-doris-connector.md‎

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,8 +34,9 @@ Flink Doris Connector can support read and write data stored in Doris through Fl
3434

3535
| Connector | Flink | Doris | Java | Scala |
3636
| --------- | ----- | ------ | ---- | ----- |
37-
| 1.0.0 | 1.11.2 | 0.13+ | 8 | 2.12 |
38-
| 1.0.0 | 1.13.x | 0.13.+ | 8 | 2.12 |
37+
| 1.11.6-2.12-xx | 1.11.x | 0.13+ | 8 | 2.12 |
38+
| 1.12.7-2.12-xx | 1.12.x | 0.13.+ | 8 | 2.12 |
39+
| 1.13.5-2.12-xx | 1.13.x | 0.13.+ | 8 | 2.12 |
3940

4041
**For Flink 1.13.x version adaptation issues**
4142

@@ -63,7 +64,7 @@ Execute following command in dir `extension/flink-doris-connector/`:
6364
2. It is recommended to compile under the docker compile environment `apache/incubator-doris:build-env-1.2` of doris, because the JDK version below 1.3 is 11, there will be compilation problems.
6465

6566
```bash
66-
sh build.sh
67+
sh build.sh 1.11.6 2.12 # flink 1.11.6 scala 2.12
6768
```
6869

6970
After successful compilation, the file `doris-flink-1.0.0-SNAPSHOT.jar` will be generated in the `output/` directory. Copy this file to `ClassPath` in `Flink` to use `Flink-Doris-Connector`. For example, `Flink` running in `Local` mode, put this file in the `jars/` folder. `Flink` running in `Yarn` cluster mode, put this file in the pre-deployment package.
@@ -78,6 +79,21 @@ conf/fe.conf
7879
```
7980
enable_http_server_v2 = true
8081
```
82+
## Using Maven
83+
84+
Add Dependency
85+
86+
```
87+
<dependency>
88+
<groupId>org.apache.doris</groupId>
89+
<artifactId>doris-flink-connector</artifactId>
90+
<version>1.11.6-2.12-SNAPSHOT</version>
91+
</dependency>
92+
```
93+
94+
**Remarks**
95+
96+
`1.11.6 ` can be substitute with `1.12.7` or `1.13.5` base on flink version you are using
8197

8298

8399
## How to use

‎docs/en/extending-doris/spark-doris-connector.md‎

Lines changed: 23 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -35,10 +35,11 @@ Spark Doris Connector can support reading data stored in Doris and writing data
3535

3636
## Version Compatibility
3737

38-
| Connector | Spark | Doris | Java | Scala |
39-
| --------- | ----- | ------ | ---- | ----- |
40-
| 1.0.0 | 2.x | 0.12+ | 8 | 2.11 |
41-
| 1.0.0 | 3.x | 0.12.+ | 8 | 2.12 |
38+
| Connector | Spark | Doris | Java | Scala |
39+
|---------------| ----- | ------ | ---- | ----- |
40+
| 2.3.4-2.11.xx | 2.x | 0.12+ | 8 | 2.11 |
41+
| 3.1.2-2.12.xx | 3.x | 0.12.+ | 8 | 2.12 |
42+
4243

4344

4445
## Build and Install
@@ -51,12 +52,28 @@ Execute following command in dir `extension/spark-doris-connector/`:
5152
2. It is recommended to compile under the docker compile environment `apache/incubator-doris:build-env-1.2` of doris, because the JDK version below 1.3 is 11, there will be compilation problems.
5253

5354
```bash
54-
sh build.sh 3 ## spark 3.x version, the default is 3.1.2
55-
sh build.sh 2 ## soark 2.x version, the default is 2.3.4
55+
sh build.sh 2.3.4 2.11 ## spark 2.3.4 version, and scala 2.11
56+
sh build.sh 3.1.2 2.12 ## spark 3.1.2 version, and scala 2.12
5657
```
5758

5859
After successful compilation, the file `doris-spark-1.0.0-SNAPSHOT.jar` will be generated in the `output/` directory. Copy this file to `ClassPath` in `Spark` to use `Spark-Doris-Connector`. For example, `Spark` running in `Local` mode, put this file in the `jars/` folder. `Spark` running in `Yarn` cluster mode, put this file in the pre-deployment package.
5960

61+
## Using Maven
62+
63+
Add dependency
64+
```
65+
<dependency>
66+
<groupId>org.apache.doris</groupId>
67+
<artifactId>doris-spark-connector</artifactId>
68+
<version>2.3.4-2.11-SNAPSHOT</version>
69+
</dependency>
70+
```
71+
72+
**Remark**
73+
74+
`2.3.4-2.11` can be repacled with `3.1.2-2.12` base on you spark and scala version
75+
76+
6077
## Example
6178
### Read
6279

‎docs/zh-CN/community/how-to-deploy-to-maven.md‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
---
2+
{
3+
"title": "如何部署到Maven 仓库",
4+
"language": "zh-CN"
5+
}
6+
---
7+
8+
<!--
9+
Licensed to the Apache Software Foundation (ASF) under one
10+
or more contributor license agreements. See the NOTICE file
11+
distributed with this work for additional information
12+
regarding copyright ownership. The ASF licenses this file
13+
to you under the Apache License, Version 2.0 (the
14+
"License"); you may not use this file except in compliance
15+
with the License. You may obtain a copy of the License at
16+
17+
http://www.apache.org/licenses/LICENSE-2.0
18+
19+
Unless required by applicable law or agreed to in writing,
20+
software distributed under the License is distributed on an
21+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
22+
KIND, either express or implied. See the License for the
23+
specific language governing permissions and limitations
24+
under the License.
25+
-->
26+
27+
## 准备
28+
29+
### 0. Requirements
30+
31+
1. apache 账号ID
32+
2. apache 账号密码
33+
3. gpg key
34+
35+
### 1. 准备本地maven 环境
36+
37+
1. 生成主密码： `mvn --encrypt-master-password <password>` 这个密码仅用作加密后续的其他密码使用, 输出类似 `{VSb+6+76djkH/43...}` 之后创建 `~/.m2/settings-security.xml` 文件，内容是
38+
39+
```
40+
<settingsSecurity>
41+
<master>{VSb+6+76djkH/43...}</master>
42+
</settingsSecurity>
43+
```
44+
45+
2. 加密 apache 密码： `mvn --encrypt-password <password>` 这个密码是apache 账号的密码 输出和上面类似`{GRKbCylpwysHfV...}` 在`~/.m2/settings.xml` 中加入
46+
47+
```
48+
<servers>
49+
<!-- To publish a snapshot of your project -->
50+
<server>
51+
<id>apache.snapshots.https</id>
52+
<username>yangzhg</username>
53+
<password>{GRKbCylpwysHfV...}</password>
54+
</server>
55+
<!-- To stage a release of your project -->
56+
<server>
57+
<id>apache.releases.https</id>
58+
<username>yangzhg</username>
59+
<password>{GRKbCylpwysHfV...}</password>
60+
</server>
61+
</servers>
62+
```
63+
64+
3. 可选，（也可以在部署时传递-Darguments="-Dgpg.passphrase=xxxx"），在`~/.m2/settings.xml` 中加入如下内容，如果已经存在profiles 标签， 这只需要将profile 加入profiles 中即可，activeProfiles 同上， xxxx 是gpg 密钥的passphrase
65+
66+
```
67+
<profiles>
68+
<profile>
69+
<id>gpg</id>
70+
<properties>
71+
<gpg.executable>gpg</gpg.executable>
72+
<gpg.passphrase>xxxx</gpg.passphrase>
73+
</properties>
74+
</profile>
75+
</profiles>
76+
<activeProfiles>
77+
<activeProfile>gpg</activeProfile>
78+
</activeProfiles>
79+
```
80+
### 发布到SNAPSHOT
81+
### 1. 部署 flink connector
82+
83+
切换到 flink connector 目录， 我们以 flink 版本 1.11.6， scalar 2.12 为例
84+
85+
```
86+
cd incubator-doris/extension/flink-doris-connector
87+
export DORIS_HOME=$PWD/../../
88+
source ${DORIS_HOME}/env.sh
89+
if [ -f ${DORIS_HOME}/custom_env.sh ]; then source ${DORIS_HOME}/custom_env.sh; fi
90+
export FLINK_VERSION="1.11.6"
91+
export SCALA_VERSION="2.12"
92+
mvn deploy
93+
```
94+
95+
96+
97+
### 2. 部署 Spark connector
98+
99+
切换到 spark connector 目录， 我们以 spark 版本 2.3.4， scalar 2.11 为例
100+
101+
```
102+
cd incubator-doris/extension/spark-doris-connector
103+
export DORIS_HOME=$PWD/../../
104+
source ${DORIS_HOME}/env.sh
105+
if [ -f ${DORIS_HOME}/custom_env.sh ]; then source ${DORIS_HOME}/custom_env.sh; fi
106+
export SPARK_VERSION="2.3.4"
107+
export SCALA_VERSION="2.11"
108+
mvn deploy
109+
```

0 commit comments

Comments
 (0)