Skip to content

Add AWS MSK Routine Load docs in EN/ZH/JA#3520

Merged
liaoxin01 merged 3 commits intoapache:masterfrom
0AyanamiRei:routineload-aws-msk
Apr 10, 2026
Merged

Add AWS MSK Routine Load docs in EN/ZH/JA#3520
liaoxin01 merged 3 commits intoapache:masterfrom
0AyanamiRei:routineload-aws-msk

Conversation

@0AyanamiRei
Copy link
Copy Markdown
Contributor

@0AyanamiRei 0AyanamiRei commented Apr 6, 2026

Versions

  • dev
  • 4.x
  • 3.x
  • 2.1

Languages

  • Chinese
  • English
  • Japanese

Docs Checklist

  • Checked by AI
  • Test Cases Built

liaoxin01 pushed a commit to apache/doris that referenced this pull request Apr 8, 2026
### What problem does this PR solve?
**_Overview_**:
This PR adds AWS MSK IAM authentication for Kafka Routine Load in Apache
Doris. You can connect to Amazon MSK using IAM credentials (including
Assume Role and cross-account) with SASL_SSL and OAUTHBEARER.

**_What It Solves_**:
1. Consume AWS MSK data from Doris via Routine Load.
2. Support three credential modes: explicit AK/SK, same-account Instance
Profile Assume Role, and cross-account AK/SK Assume Role.
3. Align with AWS MSK IAM (SigV4-signed OAUTHBEARER tokens).

**_SQL Examples_**
1. MSK IAM with explicit Access Key and Secret Key (same account)
```
CREATE ROUTINE LOAD my_msk_load ON my_db.my_table
COLUMNS (id, name, dt)
PROPERTIES (
    "desired_concurrent_number" = "2",
    "max_error_number" = "1000"
)
FROM KAFKA
(
    "kafka_broker_list" = "b-1.xxx.kafka.us-east-1.amazonaws.com:9098,b-2.xxx.kafka.us-east-1.amazonaws.com:9098",
    "kafka_topic" = "my-topic",
    "property.security.protocol" = "SASL_SSL",
    "property.sasl.mechanism" = "OAUTHBEARER",
    "aws.region" = "us-east-1",
    "aws.access_key" = "XXX",
    "aws.secret_key" = "XXX"
);
```

2. MSK IAM with Assume Role (e.g. EC2 Instance Profile, same account)
```
CREATE ROUTINE LOAD my_msk_load ON my_db.my_table
COLUMNS (id, name, dt)
PROPERTIES (
    "desired_concurrent_number" = "2"
)
FROM KAFKA
(
    "kafka_broker_list" = "b-1.xxx.kafka.us-east-1.amazonaws.com:9098",
    "kafka_topic" = "my-topic",
    "property.security.protocol" = "SASL_SSL",
    "property.sasl.mechanism" = "OAUTHBEARER",
    "aws.region" = "us-east-1",
    "aws.credentials_provider" = "xxx",
    "aws.role_arn" = "arn:aws:iam::123456789012:role/MyMSKConsumerRole"
);
```

3. MSK IAM with cross-account Assume Role (AK/SK of account B to assume
role in account A)
```
CREATE ROUTINE LOAD my_msk_load ON my_db.my_table
COLUMNS (id, name, dt)
PROPERTIES (
    "desired_concurrent_number" = "2"
)
FROM KAFKA
(
    "kafka_broker_list" = "b-1.xxx.kafka.us-east-1.amazonaws.com:9098",
    "kafka_topic" = "my-topic",
    "property.security.protocol" = "SASL_SSL",
    "property.sasl.mechanism" = "OAUTHBEARER",

    "aws.region" = "us-east-1",
    "aws.role_arn" = "arn:aws:iam::111111111111:role/CrossAccountMSKRole",
    "aws.access_key" = "XXX",
    "aws.secret_key" = "XXX"
);
```

Validation rules (FE): When any aws.* property is set, aws.region is
required, and property.security.protocol must be SASL_SSL and
property.sasl.mechanism must be OAUTHBEARER. If you use explicit
credentials, both aws.access.key and aws.secret.key must be set
together.

doc pr:apache/doris-website#3520
github-actions Bot pushed a commit to apache/doris that referenced this pull request Apr 8, 2026
### What problem does this PR solve?
**_Overview_**:
This PR adds AWS MSK IAM authentication for Kafka Routine Load in Apache
Doris. You can connect to Amazon MSK using IAM credentials (including
Assume Role and cross-account) with SASL_SSL and OAUTHBEARER.

**_What It Solves_**:
1. Consume AWS MSK data from Doris via Routine Load.
2. Support three credential modes: explicit AK/SK, same-account Instance
Profile Assume Role, and cross-account AK/SK Assume Role.
3. Align with AWS MSK IAM (SigV4-signed OAUTHBEARER tokens).

**_SQL Examples_**
1. MSK IAM with explicit Access Key and Secret Key (same account)
```
CREATE ROUTINE LOAD my_msk_load ON my_db.my_table
COLUMNS (id, name, dt)
PROPERTIES (
    "desired_concurrent_number" = "2",
    "max_error_number" = "1000"
)
FROM KAFKA
(
    "kafka_broker_list" = "b-1.xxx.kafka.us-east-1.amazonaws.com:9098,b-2.xxx.kafka.us-east-1.amazonaws.com:9098",
    "kafka_topic" = "my-topic",
    "property.security.protocol" = "SASL_SSL",
    "property.sasl.mechanism" = "OAUTHBEARER",
    "aws.region" = "us-east-1",
    "aws.access_key" = "XXX",
    "aws.secret_key" = "XXX"
);
```

2. MSK IAM with Assume Role (e.g. EC2 Instance Profile, same account)
```
CREATE ROUTINE LOAD my_msk_load ON my_db.my_table
COLUMNS (id, name, dt)
PROPERTIES (
    "desired_concurrent_number" = "2"
)
FROM KAFKA
(
    "kafka_broker_list" = "b-1.xxx.kafka.us-east-1.amazonaws.com:9098",
    "kafka_topic" = "my-topic",
    "property.security.protocol" = "SASL_SSL",
    "property.sasl.mechanism" = "OAUTHBEARER",
    "aws.region" = "us-east-1",
    "aws.credentials_provider" = "xxx",
    "aws.role_arn" = "arn:aws:iam::123456789012:role/MyMSKConsumerRole"
);
```

3. MSK IAM with cross-account Assume Role (AK/SK of account B to assume
role in account A)
```
CREATE ROUTINE LOAD my_msk_load ON my_db.my_table
COLUMNS (id, name, dt)
PROPERTIES (
    "desired_concurrent_number" = "2"
)
FROM KAFKA
(
    "kafka_broker_list" = "b-1.xxx.kafka.us-east-1.amazonaws.com:9098",
    "kafka_topic" = "my-topic",
    "property.security.protocol" = "SASL_SSL",
    "property.sasl.mechanism" = "OAUTHBEARER",

    "aws.region" = "us-east-1",
    "aws.role_arn" = "arn:aws:iam::111111111111:role/CrossAccountMSKRole",
    "aws.access_key" = "XXX",
    "aws.secret_key" = "XXX"
);
```

Validation rules (FE): When any aws.* property is set, aws.region is
required, and property.security.protocol must be SASL_SSL and
property.sasl.mechanism must be OAUTHBEARER. If you use explicit
credentials, both aws.access.key and aws.secret.key must be set
together.

doc pr:apache/doris-website#3520
Copy link
Copy Markdown
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01 liaoxin01 merged commit 446ab03 into apache:master Apr 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants