Skip to content

Commit 2d36b9b

Browse files
erezrokaherezrokah
authored
feat: Add package command (#15726)
 #### Summary Follow up to #15720 that upgraded the Python SDK. This PR adds the metadata needed for the package command to work, and adds tests for it. I copied the docs from `website/` and we can remove them later once the plugins are published to the Hub. <!--
1 parent a857746 commit 2d36b9b

13 files changed

Lines changed: 169 additions & 14 deletions

File tree

‎.github/workflows/source_square.yml‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,12 @@ jobs:
3636
pip install -r requirements.txt
3737
- name: Check formatting
3838
run: make fmt-check
39+
40+
-
41+
# Required for the package command tests to work
42+
name: Set up Docker Buildx
43+
uses: docker/setup-buildx-action@v3
44+
3945
- name: Run tests
4046
run: make test
4147

@@ -47,10 +53,10 @@ jobs:
4753
REGISTRY: ghcr.io
4854
steps:
4955
- name: Set up Docker Buildx
50-
uses: docker/setup-buildx-action@4c0219f9ac95b02789c1075625400b2acbff50b1
56+
uses: docker/setup-buildx-action@v3
5157

5258
- name: Build and push Docker image
53-
uses: docker/build-push-action@f2a1d5e99d037542a71f64918e516c093c6f3fc4
59+
uses: docker/build-push-action@v5
5460
with:
5561
context: "{{defaultContext}}:plugins/source/square"
5662
load: true

‎.github/workflows/source_typeform.yml‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,13 @@ jobs:
3636
pip install -r requirements.txt
3737
- name: Check formatting
3838
run: make fmt-check
39+
-
40+
# Required for the package command tests to work
41+
name: Set up Docker Buildx
42+
uses: docker/setup-buildx-action@v3
43+
44+
- name: Run tests
45+
run: make test
3946
validate-release:
4047
timeout-minutes: 10
4148
runs-on: ubuntu-latest
@@ -44,10 +51,10 @@ jobs:
4451
REGISTRY: ghcr.io
4552
steps:
4653
- name: Set up Docker Buildx
47-
uses: docker/setup-buildx-action@4c0219f9ac95b02789c1075625400b2acbff50b1
54+
uses: docker/setup-buildx-action@v3
4855

4956
- name: Build and push Docker image
50-
uses: docker/build-push-action@f2a1d5e99d037542a71f64918e516c093c6f3fc4
57+
uses: docker/build-push-action@v5
5158
with:
5259
context: "{{defaultContext}}:plugins/source/typeform"
5360
load: true

‎plugins/source/square/docs/_authentication.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
In order to fetch information from Square, `cloudquery` needs to authenticate using an access token.

‎plugins/source/square/docs/_configuration.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
```yaml copy
2+
kind: source
3+
# Common source-plugin configuration
4+
spec:
5+
name: square
6+
registry: docker
7+
path: ghcr.io/cloudquery/square:VERSION_SOURCE_SQUARE
8+
tables: ["*"]
9+
destinations: ["DESTINATION_NAME"]
10+
# Square-specific configuration
11+
spec:
12+
environment: "sandbox" # sandbox or production
13+
access_token: "<YOUR_SECRET_ACCESS_TOKEN_HERE>"
14+
concurrency: 100
15+
queue_size: 10000
16+
```
17+
18+
:::callout{type="info"}
19+
The Square plugin is distributed as a Docker image. This requires a Docker runtime to be installed on the same machine as the CloudQuery CLI, and a CLI version that supports the `docker` registry type (`v3.12.0` and higher).
20+
:::

‎plugins/source/square/docs/overview.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
The CloudQuery Square plugin pulls data from [Square](https://www.squareup.com/) and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and [more](https://hub.cloudquery.io/plugins/destination)).
2+
3+
See [tables](/docs/plugins/sources/square/tables) for a list of resources supported.
4+
5+
## Example Configuration
6+
7+
This example syncs from Square to a Postgres destination. The (top level) source spec section is described in the [Source Spec Reference](/docs/reference/source-spec).
8+
9+
:configuration
10+
11+
## Authentication
12+
13+
:authentication
14+
15+
## Configuration Reference
16+
17+
This is the (nested) spec used by the Square source plugin:
18+
19+
- `access_token` (string, required):
20+
21+
Your access token from Square.
22+
23+
- `environment` (string, required):
24+
25+
The environment to use. Can be `production` or `sandbox`.
26+
27+
- `concurrency` (integer, optional. Default: 100):
28+
29+
Maximum number of requests to perform concurrently.
30+
31+
- `queue_size` (integer, optional. Default: 10000):
32+
33+
Maximum number of items to have in the queue before waiting for an unfinished request to complete.

‎plugins/source/square/plugin/plugin.py‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -14,17 +14,21 @@
1414

1515
class SquarePlugin(plugin.Plugin):
1616
def __init__(self) -> None:
17-
super().__init__(PLUGIN_NAME, PLUGIN_VERSION)
17+
super().__init__(
18+
PLUGIN_NAME,
19+
PLUGIN_VERSION,
20+
opts=plugin.plugin.Options(team="cloudquery", kind="source"),
21+
)
1822
self._logger = structlog.get_logger()
1923

2024
def set_logger(self, logger) -> None:
2125
self._logger = logger
2226

23-
def init(self, spec_bytes, no_connection: bool = True):
27+
def init(self, spec, no_connection: bool = True):
2428
if no_connection:
2529
return
26-
if spec_bytes:
27-
self._spec_json = json.loads(spec_bytes)
30+
if spec:
31+
self._spec_json = json.loads(spec)
2832
else:
2933
# without this, an empty spec returns an obscure error
3034
self._spec_json = {}

‎plugins/source/square/tests/plugin/test_plugin.py‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,14 @@
1-
def test_plugin():
2-
# TODO: Implement test
3-
pass
1+
import os
2+
from cloudquery.sdk import serve
3+
from plugin import SquarePlugin
4+
5+
6+
def test_plugin_package():
7+
p = SquarePlugin()
8+
cmd = serve.PluginCommand(p)
9+
cmd.run(["package", "-m", "test", "v1.0.0", "."])
10+
assert os.path.isfile("dist/tables.json")
11+
assert os.path.isfile("dist/package.json")
12+
assert os.path.isfile("dist/docs/overview.md")
13+
assert os.path.isfile("dist/plugin-square-v1.0.0-linux-amd64.tar")
14+
assert os.path.isfile("dist/plugin-square-v1.0.0-linux-arm64.tar")

‎plugins/source/typeform/docs/_authentication.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
In order to fetch information from Typeform, `cloudquery` needs to authenticate using a [personal access token for Typeform's APIs](https://www.typeform.com/developers/get-started/personal-access-token/). Follow the instructions on the Typeform website and create a read-only token.

‎plugins/source/typeform/docs/_configuration.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
```yaml copy
2+
kind: source
3+
# Common source-plugin configuration
4+
spec:
5+
name: typeform
6+
registry: docker
7+
path: ghcr.io/cloudquery/cq-source-typeform:VERSION_SOURCE_TYPEFORM
8+
tables: ["typeform_forms"]
9+
destinations: ["DESTINATION_NAME"]
10+
# Typeform-specific configuration
11+
spec:
12+
access_token: "<YOUR_SECRET_ACCESS_TOKEN_HERE>"
13+
base_url: "https://api.typeform.com" # use https://api.eu.typeform.com for EU accounts
14+
# Optional configuration
15+
# concurrency: 100
16+
# queue_size: 10000
17+
```
18+
19+
:::callout{type="info"}
20+
The Typeform plugin is distributed as a Docker image. This requires a Docker runtime to be installed on the same machine as the CloudQuery CLI, and a CLI version that supports the `docker` registry type (`v3.12.0` and higher).
21+
:::

‎plugins/source/typeform/docs/overview.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
The CloudQuery Typeform plugin pulls data from [Typeform](https://www.typeform.com/) and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and [more](https://hub.cloudquery.io/plugins/destination)).
2+
3+
See [tables](/docs/plugins/sources/typeform/tables) for a list of resources supported.
4+
5+
## Example Configuration
6+
7+
This example syncs from Typeform to a Postgres destination. The (top level) source spec section is described in the [Source Spec Reference](/docs/reference/source-spec).
8+
9+
:configuration
10+
11+
## Authentication
12+
13+
:authentication
14+
15+
## Configuration Reference
16+
17+
This is the (nested) spec used by the Typeform source plugin:
18+
19+
- `access_token` (string, required):
20+
21+
Your [personal access token](https://www.typeform.com/developers/get-started/personal-access-token/) from the Typeform Dashboard.
22+
23+
- `base_url` (string, optional. Default: `https://api.typeform.com`):
24+
25+
The base URL to fetch data from. Use https://api.eu.typeform.com if your account is stored in the EU.
26+
27+
- `concurrency` (integer, optional. Default: 100):
28+
29+
Maximum number of requests to perform concurrently.
30+
31+
- `queue_size` (integer, optional. Default: 10000):
32+
33+
Maximum number of items to have in the queue before waiting for an unfinished request to complete.

0 commit comments

Comments
 (0)