-
Notifications
You must be signed in to change notification settings - Fork 544
feat: Add JSON schema to elasticsearch destination
#16489
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
f47fecf
feat: Add JSON schema to elasticsearch destination
disq 68adaad
Add default
disq 7a047e7
Merge branch 'main' into feat/jsonschema/elasticsearch
kodiakhq[bot] 1652ead
Update plugins/destination/elasticsearch/client/spec.go
candiduslynx 9595f22
gen
candiduslynx File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,31 +1,54 @@ | ||
| package client | ||
|
|
||
| import "runtime" | ||
| import ( | ||
| _ "embed" | ||
| "runtime" | ||
| ) | ||
|
|
||
| const ( | ||
| defaultBatchSize = 1000 | ||
| defaultBatchSizeBytes = 5 * 1024 * 1024 | ||
| ) | ||
|
|
||
| type Spec struct { | ||
| Addresses []string `json:"addresses"` // A list of Elasticsearch nodes to use. | ||
| Username string `json:"username"` // Username for HTTP Basic Authentication. | ||
| Password string `json:"password"` // Password for HTTP Basic Authentication. | ||
| // A list of Elasticsearch nodes to use. | ||
| Addresses []string `json:"addresses" jsonschema:"default=http://localhost:9200"` | ||
|
|
||
| // Username for HTTP Basic Authentication. | ||
| Username string `json:"username"` | ||
|
|
||
| // Password for HTTP Basic Authentication. | ||
| Password string `json:"password"` | ||
|
|
||
| // Endpoint for the Elastic Service (https://elastic.co/cloud). | ||
| CloudID string `json:"cloud_id"` | ||
|
|
||
| // Base64-encoded token for authorization; if set, overrides username/password and service token. | ||
| APIKey string `json:"api_key"` | ||
|
|
||
| CloudID string `json:"cloud_id"` // Endpoint for the Elastic Service (https://elastic.co/cloud). | ||
| APIKey string `json:"api_key"` // Base64-encoded token for authorization; if set, overrides username/password and service token. | ||
| ServiceToken string `json:"service_token"` // Service token for authorization; if set, overrides username/password. | ||
| CertificateFingerprint string `json:"certificate_fingerprint"` // SHA256 hex fingerprint given by Elasticsearch on first launch. | ||
| // Service token for authorization; if set, overrides username/password. | ||
| ServiceToken string `json:"service_token"` | ||
|
|
||
| // SHA256 hex fingerprint given by Elasticsearch on first launch. | ||
| CertificateFingerprint string `json:"certificate_fingerprint"` | ||
|
|
||
| // PEM-encoded certificate authorities. | ||
| // When set, an empty certificate pool will be created, and the certificates will be appended to it. | ||
| CACert string `json:"ca_cert"` | ||
|
|
||
| Concurrency int `json:"concurrency"` // Number of concurrent worker goroutines to use for indexing. (Default: number of CPUs) | ||
| BatchSize int `json:"batch_size"` // Number of documents to batch together per request. (Default: 1000) | ||
| BatchSizeBytes int `json:"batch_size_bytes"` // Number of bytes to batch together per request. (Default: 5 MiB) | ||
| // Number of concurrent worker goroutines to use for indexing. (Default: number of CPUs) | ||
| Concurrency int `json:"concurrency" jsonschema:"minimum=1"` | ||
|
|
||
| // Number of documents to batch together per request. | ||
| BatchSize int `json:"batch_size" jsonschema:"minimum=1,default=1000"` | ||
|
|
||
| // Number of bytes to batch together per request. | ||
| BatchSizeBytes int `json:"batch_size_bytes" jsonschema:"minimum=1,default=5242880"` | ||
| } | ||
|
|
||
| //go:embed schema.json | ||
| var JSONSchema string | ||
|
|
||
| func (s *Spec) SetDefaults() { | ||
| if len(s.Addresses) == 0 { | ||
| s.Addresses = []string{"http://localhost:9200"} | ||
|
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is very basic and Validate() is empty... doesn't make a good json schema :) |
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| package main | ||
|
|
||
| import ( | ||
| "fmt" | ||
| "log" | ||
| "path" | ||
| "runtime" | ||
|
|
||
| "github.com/cloudquery/cloudquery/plugins/destination/elasticsearch/client" | ||
| cqjsonschema "github.com/cloudquery/codegen/jsonschema" | ||
| ) | ||
|
|
||
| func main() { | ||
| fmt.Println("Generating JSON schema for plugin spec") | ||
| cqjsonschema.GenerateIntoFile(new(client.Spec), path.Join(currDir(), "../..", "schema.json"), | ||
| cqjsonschema.WithAddGoComments("github.com/cloudquery/cloudquery/plugins/destination/elasticsearch/client", path.Join(currDir(), "../..")), | ||
| ) | ||
| } | ||
|
|
||
| func currDir() string { | ||
| _, filename, _, ok := runtime.Caller(0) | ||
| if !ok { | ||
| log.Fatal("Failed to get caller information") | ||
| } | ||
| return path.Dir(filename) | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| package client | ||
|
|
||
| import ( | ||
| "testing" | ||
|
|
||
| "github.com/cloudquery/codegen/jsonschema" | ||
| ) | ||
|
|
||
| func TestJSONSchema(t *testing.T) { | ||
| jsonschema.TestJSONSchema(t, JSONSchema, []jsonschema.TestCase{ | ||
| { | ||
| Name: "empty spec", | ||
| Spec: `{}`, | ||
| }, | ||
| { | ||
| Name: "spec with str addresses", | ||
| Spec: `{"addresses": "address"}`, | ||
| Err: true, | ||
| }, | ||
| { | ||
| Name: "spec with valid addresses", | ||
| Spec: `{"addresses": ["address"]}`, | ||
| }, | ||
| { | ||
| Name: "spec with bool batch_size", | ||
| Spec: `{"batch_size":false}`, | ||
| Err: true, | ||
| }, | ||
| { | ||
| Name: "spec with null batch_size", | ||
| Spec: `{"batch_size":null}`, | ||
| Err: true, | ||
| }, | ||
| { | ||
| Name: "spec with string batch_size", | ||
| Spec: `{"batch_size":"str"}`, | ||
| Err: true, | ||
| }, | ||
| { | ||
| Name: "spec with array batch_size", | ||
| Spec: `{"batch_size":["abc"]}`, | ||
| Err: true, | ||
| }, | ||
| { | ||
| Name: "spec with unknown field", | ||
| Spec: `{"unknown": "test"}`, | ||
| Err: true, | ||
| }, | ||
| }) | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.