diff --git a/README.md b/README.md index f44dfaaa..b4f1b9fd 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ # Deploy, Manage and Query Cloud Infrastructure using SQL -[Documentation](https://docs.stackql.io/)] [Developer Guide](/docs/developer_guide.md)] +[[Documentation](https://docs.stackql.io/)] [[Developer Guide](/docs/developer_guide.md)] [[BYO Providers](/docs/registry_contribution.md)] ## Cloud infrastructure coding using SQL @@ -21,6 +21,14 @@ ---- +## Provider development + +Keen to expose some new functionality though `stackql`? We are very keen on this! + +Please see [registry_contribution.md](/docs/registry_contribution.md). + +--- + ## Design [HLDD](/docs/high-level-design.md) diff --git a/build/.gitignore b/build/.gitignore index c96a04f0..cf26ca47 100644 --- a/build/.gitignore +++ b/build/.gitignore @@ -1,2 +1,3 @@ * -!.gitignore \ No newline at end of file +!.gitignore +!build.md \ No newline at end of file diff --git a/build/build.md b/build/build.md new file mode 100644 index 00000000..45d71c38 --- /dev/null +++ b/build/build.md @@ -0,0 +1,4 @@ + +## Build Directory + +This directory is the location for building and testing artifacts. Various walkthroughs and tests depend upon the executable being pre-built to this directory. diff --git a/docs/developer_guide.md b/docs/developer_guide.md index 50a3d42a..bcca33e4 100644 --- a/docs/developer_guide.md +++ b/docs/developer_guide.md @@ -1,6 +1,26 @@ # StackQL Developer Guide +## Provider development + +Keen to expose some new functionality though `stackql`? We are very keen on this! + +Please see [registry_contribution.md](/docs/registry_contribution.md). + +## Provider Authentication + +At this stage, authentication config must be specified for each provider, even for unauthorized ones. Supported auth types are: + +- `api_key`. +- `basic`. +- `interactive` for interactive oAuth, thus far only google supported via `gcloud` command line tool. +- `service_account` for json style private keys (eg: google service accounts). +- `null_auth` for unauthenticated providers. + +If you want further auth types or discover bugs, please raise an issue. + +Examples are present [here](/docs/examples.md). + ## Concurrency considerations In server mode, a thread pool issues one thread to handle each connection. diff --git a/docs/registry_contribution.md b/docs/registry_contribution.md new file mode 100644 index 00000000..f45723ff --- /dev/null +++ b/docs/registry_contribution.md @@ -0,0 +1,78 @@ + +# Provider Registry Contribution + +Community development of provider functionality is key to `stackql` and our mission. +Here, we describe the process to develop, test and integrate a new provider from zero to General Availability. + +We include an example integration for a simple Provider, `publicapis`. + +## Workflow + +1. Create a Provider definition document. +2. Create at least one Service definition document. +3. Run `stackql` with config to support local Provider development. +4. Iterate and verify that `stackql` works as expected against your new provider. +5. Submit a Pull Request against the Provider Registry repository. + + +### 1. Create a Provider definition document + +This is a yaml or json document which contains Provider metadata and reference(s) to any Service document(s) through which it will expose functionality. + +Example as per [examples/registry/src/publicapis/v1/provider.yaml](/examples/registry/src/publicapis/v1/provider.yaml). + +### 2. Create at least one Service definition document. + +This is an [openapi document spec](https://swagger.io/specification/) in either yaml or json format, **plus** a legal annotation `components.x-stackQL-resources`, which defines the Resource portion of the `stackql` heirarchy. + +Example as per [examples/registry/src/publicapis/v1/services/api-v1.yaml](/examples/registry/src/publicapis/v1/services/api-v1.yaml). + + +### 3. Run `stackql` with config to support local Provider development. + +Configure `stackql` to use your docs, bia the `--registry` command line argument. The [example integration section](#Configuring-StackQL-to-consume-a-local-development-registry) expands upon this. + +### 4. Iterate and verify that `stackql` works as expected against your new provider. + +Iterate upon steps 2 and 3 until API coverage and functionality fulfil your requirements. + +### 5. Submit a Pull Request against the Provider Registry repository. + +As per [the Provider Registry Contributor Guide](https://github.com/stackql/stackql-provider-registry/blob/main/.github/CONTRIBUTING.md). The team will review as rapidly as possible and work with you to complete the integration. Once this is done, your functionality with be available via `registry pull...` + +## Example integration + +In this walkthrough, we demonstrate the bare bones of integrating a simple Provider, +`publicapis`, which is included here within source control. + +The assumptions for this walkthrough are: + +1. You have either built (eg: via `cmake` as per [the readme](/README.md#build)) or copied the `stackql` executable for your platform into the [build](/build) directory. +2. The current working directory is same as this document, otherwise relative paths to `pwd` will need to be adjusted. + +### Configuring StackQL to consume a local development registry + +Registry access config: + +```bash +PROVIDER_REGISTRY_ROOT_DIR="$(pwd)/../examples/registry" + +REG_STR='{ "url": "file://'${PROVIDER_REGISTRY_ROOT_DIR}'", "localDocRoot": "'${PROVIDER_REGISTRY_ROOT_DIR}'", "verifyConfig": {"nopVerify": true } }' +``` + +Provider auth config as per [developer guide](/docs/developer_guide.md#provider-authentication): + +```bash +AUTH_STR='{ "publicapis": { "type": "null_auth" } }' +``` + +### Interacting with the local development registry + +```bash +$(pwd)/../build/stackql --auth="${AUTH_STR}" --registry="${REG_STR}" exec "select API from publicapis.api.apis where API like 'Dog%' limit 10;" + +$(pwd)/../build/stackql --auth="${AUTH_STR}" --registry="${REG_STR}" exec "select API from publicapis.api.random where title = 'Dog';" + +## The single, anonmyous column returned from selecing an array of strings is a core issue to fix separate to this +$(pwd)/../build/stackql --auth="${AUTH_STR}" --registry="${REG_STR}" exec "select * from publicapis.api.categories limit 5;" +``` diff --git a/examples/README.md b/examples/README.md deleted file mode 100644 index 7c5e54a9..00000000 --- a/examples/README.md +++ /dev/null @@ -1,38 +0,0 @@ - -## BYO Registry - - -```bash - -PROVIDER_REGISTRY_ROOT_DIR="$(pwd)/../examples/registry" - -REG_STR='{ "url": "useEmbedded": false }' - -``` - - -## BYO Registry - - -```bash - -PROVIDER_REGISTRY_ROOT_DIR="$(pwd)/../examples/registry" - -REG_STR='{ "url": "file://'${PROVIDER_REGISTRY_ROOT_DIR}'", "localDocRoot": "'${PROVIDER_REGISTRY_ROOT_DIR}'", "useEmbedded": false, "verifyConfig": {"nopVerify": true } }' - -``` - -## Interacting with the registry - -```bash -## All auth required ahead of time at this stage, even for no auth providers -AUTH_STR='{ "publicapis": { "type": "null_auth" } }' - -./stackql --auth="${AUTH_STR}" --registry="${REG_STR}" exec "select API from publicapis.api.apis where API like 'Dog%' limit 10;" - -./stackql --auth="${AUTH_STR}" --registry="${REG_STR}" exec "select API from publicapis.api.random where title = 'Dog';" - -## The single, anonmyous column returned from selecing an array of strings is a core issue to fix separate to this -./stackql --auth="${AUTH_STR}" --registry="${REG_STR}" exec "select * from publicapis.api.categories limit 5;" - -``` \ No newline at end of file diff --git a/internal/stackql/cmd/root.go b/internal/stackql/cmd/root.go index b6ed557f..fdee7a23 100644 --- a/internal/stackql/cmd/root.go +++ b/internal/stackql/cmd/root.go @@ -46,7 +46,7 @@ var ( ) const ( - defaultRegistryUrlString string = "https://cdn.statically.io/gh/stackql/stackql-provider-registry/intial-devel/providers" + defaultRegistryUrlString string = "https://cdn.statically.io/gh/stackql/stackql-provider-registry/main/providers" ) var SemVersion string = fmt.Sprintf("%s.%s.%s", BuildMajorVersion, BuildMinorVersion, BuildPatchVersion)