Skip to content

Commit d501cf3

Browse files
marianogappaerezrokah
marianogappa
and
erezrokah
authored
feat: Implement SQLite-Python destination plugin. (#18970)
This is meant as the simplest destination plugin so that we have an example destination plugin in Python. Missing: - release-please, etc, not yet done. - SDK changes need to be merged first. - I haven't been able to do the round-trip test yet; it requires SDK changes that are tricky; but I'm pretty sure read/write work well now. Note second run relies on state to do virtually nothing: ``` Starting sync for: tenable (cloudquery/tenable@v2.2.4) -> [sqlite (grpc@localhost:7777)] Sync completed successfully. Resources: 214303, Errors: 0, Warnings: 0, Time: 5m15s ✓ Code/test-tenable-sqlite-sync $ cli sync . ⏱ 16:46:49 Loading spec(s) from . Starting sync for: tenable (cloudquery/tenable@v2.2.4) -> [sqlite (grpc@localhost:7777)] Sync completed successfully. Resources: 19, Errors: 0, Warnings: 0, Time: 5s ✓ Code/test-tenable-sqlite-sync $ ``` ``` $ cli sync . ⏱ 16:35:35 Loading spec(s) from . Starting sync for: xkcd (cloudquery/xkcd@v1.0.6) -> [sqlite-python (grpc@localhost:7777)] Sync completed successfully. Resources: 2973, Errors: 0, Warnings: 0, Time: 20s ✓ Code/test-xkcd-sqlitepython-sync $ ``` State stored and read properly ![Screenshot 2024-08-20 at 17 03 16](https://github.com/user-attachments/assets/ce003adf-5271-4119-84e4-ecc7cc6b80b4) Dates & Extension types managed properly ![Screenshot 2024-08-20 at 17 05 05](https://github.com/user-attachments/assets/f2c8d6aa-faba-4e9c-8784-a0660d272d30) --------- Co-authored-by: Erez Rokah <erezrokah@users.noreply.github.com>
1 parent f9b0a1a commit d501cf3

22 files changed

Lines changed: 961 additions & 0 deletions

‎.github/pr_labeler.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,8 @@ area/plugin/destination/snowflake:
4040
- plugins/destination/snowflake/**/*
4141
area/plugin/destination/sqlite:
4242
- plugins/destination/sqlite/**/*
43+
area/plugin/destination/sqlite-python:
44+
- plugins/destination/sqlite/**/*
4345
area/plugin/source/airtable:
4446
- plugins/source/airtable/**/*
4547
area/plugin/source/bitbucket:

‎.github/workflows/dest_sqlite_python.yml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
name: Destination Plugin SQLite-Python Workflow
2+
3+
concurrency:
4+
group: ${{ github.workflow }}-${{ github.ref }}
5+
cancel-in-progress: true
6+
7+
on:
8+
pull_request:
9+
paths:
10+
- "plugins/destination/sqlite-python/**"
11+
- ".github/workflows/dest_sqlite_python.yml"
12+
push:
13+
branches:
14+
- main
15+
paths:
16+
- "plugins/destination/sqlite-python/**"
17+
- ".github/workflows/dest_sqlite_python.yml"
18+
19+
jobs:
20+
plugins-destination-sqlite-python:
21+
timeout-minutes: 30
22+
name: "plugins/destination/sqlite-python"
23+
runs-on: ubuntu-latest
24+
defaults:
25+
run:
26+
working-directory: ./plugins/destination/sqlite-python
27+
steps:
28+
- uses: actions/checkout@v4
29+
- name: Set up Python
30+
uses: actions/setup-python@v5
31+
with:
32+
python-version: '3.11'
33+
- name: Install dependencies
34+
run: |
35+
pip install --upgrade pip
36+
pip install -r requirements.txt
37+
- name: Check formatting
38+
run: make fmt-check
39+
-
40+
# Required for the package command tests to work
41+
name: Set up Docker Buildx
42+
uses: docker/setup-buildx-action@v3
43+
44+
- name: Run tests
45+
run: make test
46+

‎plugins/destination/sqlite-python/.dockerignore‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# Byte-compiled / optimized / DLL files
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
6+
# C extensions
7+
*.so
8+
9+
# Distribution / packaging
10+
.Python
11+
build/
12+
develop-eggs/
13+
dist/
14+
downloads/
15+
eggs/
16+
.eggs/
17+
lib/
18+
lib64/
19+
parts/
20+
sdist/
21+
var/
22+
wheels/
23+
*.egg-info/
24+
.installed.cfg
25+
*.egg
26+
MANIFEST
27+
28+
# PyInstaller
29+
# Usually these files are written by a python script from a template
30+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
31+
*.manifest
32+
*.spec
33+
34+
# Installer logs
35+
pip-log.txt
36+
pip-delete-this-directory.txt
37+
38+
# Unit test / coverage reports
39+
htmlcov/
40+
.tox/
41+
.nox/
42+
.coverage
43+
.coverage.*
44+
.cache
45+
nosetests.xml
46+
coverage.xml
47+
*.cover
48+
*.py,cover
49+
.hypothesis/
50+
.pytest_cache/
51+
52+
# Translations
53+
*.mo
54+
*.pot
55+
56+
# Django stuff:
57+
*.log
58+
local_settings.py
59+
db.sqlite3
60+
db.sqlite3-journal
61+
62+
# Flask stuff:
63+
instance/
64+
.webassets-cache
65+
66+
# Scrapy stuff:
67+
.scrapy
68+
69+
# Sphinx documentation
70+
docs/_build/
71+
72+
# PyBuilder
73+
target/
74+
75+
# pyenv
76+
.python-version
77+
78+
# celery beat schedule file
79+
celerybeat-schedule
80+
81+
# SageMath parsed files
82+
*.sage.py
83+
84+
# Environments
85+
.env
86+
.venv
87+
env/
88+
venv/
89+
ENV/
90+
91+
# Spyder project settings
92+
.spyderproject
93+
.spyproject
94+
95+
# Rope project settings
96+
.ropeproject
97+
98+
# mkdocs documentation
99+
/site
100+
101+
# mypy
102+
.mypy_cache/
103+
.dmypy.json
104+
dmypy.json
105+
106+
# Pyre type checker
107+
.pyre/
108+
109+
# profiling data
110+
.prof

‎plugins/destination/sqlite-python/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
venv
2+
__pycache__

‎plugins/destination/sqlite-python/CHANGELOG.md‎

Whitespace-only changes.

‎plugins/destination/sqlite-python/Dockerfile‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Start from a base image
2+
FROM python:3.11-slim
3+
4+
# (Optional) Set a working directory
5+
WORKDIR /app
6+
7+
# Copy requirements.txt and install the Python dependencies
8+
COPY requirements.txt .
9+
RUN pip3 install --no-cache-dir -r requirements.txt
10+
11+
# Copy the rest of the code
12+
COPY plugin plugin
13+
COPY main.py .
14+
15+
# (Optional) Expose any ports your app uses
16+
EXPOSE 7777
17+
18+
ENTRYPOINT ["python3", "main.py"]
19+
20+
# Specify the command to run when the container starts
21+
CMD ["serve", "--address", "[::]:7777", "--log-format", "json", "--log-level", "info"]

‎plugins/destination/sqlite-python/Makefile‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
current_dir=$(shell basename $(CURDIR))
2+
3+
test:
4+
pytest .
5+
6+
fmt:
7+
black .
8+
9+
fmt-check:
10+
black --check .
11+
12+
build-docker:
13+
docker build -t sqlite-python:latest .

‎plugins/destination/sqlite-python/README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# CloudQuery SQLite-Python Destination Plugin
2+
3+
This destination plugin let's you sync data from a CloudQuery source to an SQLite database. This can be useful for local data exploration as no db/service is required.
4+
5+
## Links
6+
7+
- [User Guide](https://cloudquery.io/docs/plugins/destinations/sqlite-python/overview)

‎plugins/destination/sqlite-python/docs/_configuration.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
This is a basic configuration that will save all your sync resources to `db.sql`.
2+
3+
```yaml copy
4+
kind: destination
5+
spec:
6+
name: sqlite-python
7+
path: cloudquery/sqlite-python
8+
registry: cloudquery
9+
version: "VERSION_DESTINATION_SQLITE_PYTHON"
10+
# Learn more about the configuration options at https://cql.ink/sqlite-python_destination
11+
spec:
12+
connection_string: ./db.sql
13+
```
14+
15+
After running `cloudquery sync`, you can explore the data locally with the SQLite CLI: `sqlite ./db.sql`.

‎plugins/destination/sqlite-python/docs/overview.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
---
2+
name: SQLite-Python
3+
stage: GA
4+
title: SQLite-Python Destination Plugin
5+
description: CloudQuery SQLite destination plugin documentation
6+
---
7+
# SQLite-Python Destination Plugin
8+
9+
:badge
10+
11+
This destination plugin lets you sync data from a CloudQuery source to a SQLite database. This can be useful for local data exploration as no other database or service is required.
12+
13+
## Example Config
14+
15+
:configuration
16+
17+
## SQLite Spec
18+
19+
This is the top level spec used by the SQLite-Python destination Plugin.
20+
21+
- `connection_string` (`string`) (required)
22+
23+
Path to a file, such as `./mydb.sql`.

0 commit comments

Comments
 (0)