Skip to content

feat: Add possibility to set local profile to use in S3 Destination spec #19913

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kodiakhq merged 2 commits into from
Dec 10, 2024
Merged

feat: Add possibility to set local profile to use in S3 Destination spec #19913

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0e3f15f
add possibility to set local profile to use in spec
blesniewski Dec 9, 2024
57cb501
Merge branch 'main' into feat/s3-destination-local-profile-support
kodiakhq[bot] Dec 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion plugins/destination/s3/client/client.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,14 @@ func New(ctx context.Context, logger zerolog.Logger, s []byte, opts plugin.NewCl
}
c.Client = filetypesClient

cfg, err := config.LoadDefaultConfig(ctx, config.WithDefaultRegion("us-east-1"))
configFns := []func(*config.LoadOptions) error{
config.WithDefaultRegion("us-east-1"),
}
if c.spec.LocalProfile != "" {
configFns = append(configFns, config.WithSharedConfigProfile(c.spec.LocalProfile))
}

cfg, err := config.LoadDefaultConfig(ctx, configFns...)
if err != nil {
return nil, fmt.Errorf("unable to load AWS SDK config: %w", err)
}
Expand Down
7 changes: 7 additions & 0 deletions plugins/destination/s3/client/spec/schema.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

18 changes: 18 additions & 0 deletions plugins/destination/s3/client/spec/spec.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,24 @@ type Spec struct {
// Region where bucket is located.
Region string `json:"region,omitempty" jsonschema:"required,minLength=1"`

// [Local profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) to use to authenticate this account with.
// Should be set to the name of the profile.
//
// For example, with the following credentials file:
//
// ```ini copy
// [default]
// aws_access_key_id=xxxx
// aws_secret_access_key=xxxx
//
// [user1]
// aws_access_key_id=xxxx
// aws_secret_access_key=xxxx
// ```
//
// `local_profile` should be set to either `default` or `user1`.
LocalProfile string `json:"local_profile,omitempty" jsonschema:"example=my_aws_profile"`

// Path to where the files will be uploaded in the above bucket, for example `path/to/files/{{TABLE}}/{{UUID}}.parquet`.
// The path supports the following placeholder variables:
//
Expand Down
1 change: 1 addition & 0 deletions plugins/destination/s3/docs/_authentication.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ Then, you can either export the `AWS_PROFILE` environment variable (On Linux/Mac
export AWS_PROFILE=myprofile
```

You can also use the `local_profile` field in plugin configuration (can be helpful for syncing between different accounts).

### IAM Roles for AWS Compute Resources

Expand Down
19 changes: 19 additions & 0 deletions plugins/destination/s3/docs/overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,25 @@ This is the (nested) spec used by the CSV destination Plugin.

Region where bucket is located.

- `local_profile` (`string`) (optional) (default: will use current credentials)

[Local profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) to use to authenticate this account with.
Please note this should be set to the name of the profile.

For example, with the following credentials file:

```toml copy
[default]
aws_access_key_id=xxxx
aws_secret_access_key=xxxx

[user1]
aws_access_key_id=xxxx
aws_secret_access_key=xxxx
```

`local_profile` should be set to either `default` or `user1`.

- `path` (`string`) (required)

Path to where the files will be uploaded in the above bucket, for example `path/to/files/{{TABLE}}/{{UUID}}.parquet`.
Expand Down