Skip to content

Commit 89fa42f

Browse files
abhizerabhizer
authored
py: set up sphinx for auto generated documentation (feldera#1772)
* py: set up sphinx for auto generated documentation Signed-off-by: Abhinav Gyawali <22275402+abhizer@users.noreply.github.com> * py: update the documentation Signed-off-by: Abhinav Gyawali <22275402+abhizer@users.noreply.github.com> * py: remove autogenerated rst and bat files Signed-off-by: Abhinav Gyawali <22275402+abhizer@users.noreply.github.com> * py: doc pages for Introduction and Examples * also use ReadTheDocs theme * also add requirements-dev.txt for docs generation Signed-off-by: Abhinav Gyawali <22275402+abhizer@users.noreply.github.com> --------- Signed-off-by: Abhinav Gyawali <22275402+abhizer@users.noreply.github.com>
1 parent 60b17a7 commit 89fa42f

16 files changed

Lines changed: 404 additions & 3 deletions

‎python/.gitignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,3 +2,9 @@
22
__pycache__
33
test.log
44
.idea
5+
.venv
6+
docs/_build
7+
docs/make.bat
8+
docs/modules.rst
9+
docs/feldera.rst
10+
docs/feldera.rest.rst

‎python/README.md‎

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,35 @@
11
# Feldera Python SDK
22

3-
Feldera Python is the Feldera SDK for Python developers.
3+
Feldera Python is the Feldera SDK for Python developers.
4+
5+
## Installation
6+
7+
```bash
8+
pip install git+https://github.com/feldera/feldera#subdirectory=python
9+
```
10+
11+
Similarly, to install from a specific branch:
12+
13+
```bash
14+
$ pip install git+https://github.com/feldera/feldera@{BRANCH_NAME}#subdirectory=python
15+
```
16+
17+
Replace `{BRANCH_NAME}` with the name of the branch you want to install from.
18+
19+
Checkout the docs [here](./feldera/__init__.py) for an example on how to use the SDK.
20+
21+
## Documentation
22+
23+
To build the html documentation run:
24+
25+
Ensure that you have sphinx installed. If not, install it using `pip install sphinx`.
26+
27+
Then run the following commands:
28+
29+
```bash
30+
cd docs
31+
sphinx-apidoc -o . ../feldera
32+
make html
33+
```
34+
35+
To clean the build, run `make clean`.

‎python/docs/Makefile‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Minimal makefile for Sphinx documentation
2+
#
3+
4+
# You can set these variables from the command line, and also
5+
# from the environment for the first two.
6+
SPHINXOPTS ?=
7+
SPHINXBUILD ?= sphinx-build
8+
SOURCEDIR = .
9+
BUILDDIR = _build
10+
11+
# Put it first so that "make" without argument is like "make help".
12+
help:
13+
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
14+
15+
.PHONY: help Makefile
16+
17+
# Catch-all target: route all unknown targets to Sphinx using the new
18+
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
19+
%: Makefile
20+
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

‎python/docs/conf.py‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# Configuration file for the Sphinx documentation builder.
2+
#
3+
# For the full list of built-in configuration values, see the documentation:
4+
# https://www.sphinx-doc.org/en/master/usage/configuration.html
5+
6+
# -- Project information -----------------------------------------------------
7+
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
8+
9+
import os
10+
import sys
11+
sys.path.insert(0, os.path.abspath('..'))
12+
13+
project = 'Feldera Python SDK'
14+
copyright = '2024, Feldera'
15+
author = 'Feldera'
16+
release = '0.0.1'
17+
18+
# -- General configuration ---------------------------------------------------
19+
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
20+
21+
extensions = [
22+
'sphinx.ext.autodoc',
23+
'sphinx.ext.viewcode',
24+
'sphinx.ext.todo',
25+
'sphinx.ext.doctest',
26+
]
27+
28+
templates_path = ['_templates']
29+
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
30+
31+
32+
33+
# -- Options for HTML output -------------------------------------------------
34+
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
35+
36+
html_theme = 'sphinx_rtd_theme'
37+
html_static_path = ['_static']

‎python/docs/examples.rst‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
Examples
2+
========
3+
4+
Pandas
5+
*******
6+
7+
8+
Working wth pandas DataFrames in Feldera is fairly straight forward.
9+
You can use :meth:`.SQLContext.connect_source_pandas` to connect a
10+
DataFrame to a feldera table as the data source.
11+
12+
To listen for response from feldera, in the form of DataFrames, it is necessary
13+
to to call :meth:`.SQLContext.listen` before you call
14+
:meth:`.SQLContext.start` or :meth:`.SQLContext.run_to_completion`.
15+
16+
.. highlight:: python
17+
.. code-block:: python
18+
19+
from feldera import FelderaClient, SQLContext, SQLSchema
20+
import pandas as pd
21+
22+
# Create a client
23+
client = FelderaClient("https://try.feldera.com", api_key="YOUR_API_KEY")
24+
25+
# Create a SQLContext
26+
sql = SQLContext("notebook", client).get_or_create()
27+
28+
TBL_NAMES = ["students", "grades"]
29+
view_name = "average_scores"
30+
31+
df_students = pd.read_csv("students.csv")
32+
df_grades = pd.read_csv("grades.csv")
33+
34+
# register an input table
35+
# tables receive data from the source, therefore they need a schema
36+
sql.register_table(TBL_NAMES[0], SQLSchema({"name": "STRING", "id": "INT"}))
37+
38+
sql.register_table(TBL_NAMES[1], SQLSchema({
39+
"student_id": "INT",
40+
"science": "INT",
41+
"maths": "INT",
42+
"art": "INT"
43+
}))
44+
45+
# here, we provide a query, that gets registered as a view in feldera
46+
# this query will be executed on the data in the table
47+
query = f"SELECT name, ((science + maths + art) / 3) as average FROM {TBL_NAMES[0]} JOIN {TBL_NAMES[1]} on id = student_id ORDER BY average DESC"
48+
sql.register_view(view_name, query)
49+
50+
# connect the source (a pandas Dataframe in this case) to the tables
51+
sql.connect_source_pandas(TBL_NAMES[0], df_students)
52+
sql.connect_source_pandas(TBL_NAMES[1], df_grades)
53+
54+
# listen for the output of the view here in the notebook
55+
# you do not need to call this if you are forwarding the data to a sink
56+
out = sql.listen(view_name)
57+
58+
# run this to completion
59+
# note that if the source is a stream, this will run indefinitely
60+
sql.run_to_completion()
61+
62+
# finally, convert the output to a pandas Dataframe
63+
df = out.to_pandas()
64+
65+
# see the result
66+
print(df)

‎python/docs/index.rst‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
.. feldera documentation master file, created by
2+
sphinx-quickstart on Tue May 21 18:28:09 2024.
3+
You can adapt this file completely to your liking, but it should at least
4+
contain the root `toctree` directive.
5+
6+
Welcome to feldera's documentation!
7+
===================================
8+
9+
.. toctree::
10+
:maxdepth: 2
11+
:caption: Contents:
12+
13+
introduction
14+
examples
15+
16+
.. toctree::
17+
:maxdepth: 2
18+
:caption: API Reference:
19+
20+
modules
21+
22+
Indices and tables
23+
==================
24+
25+
* :ref:`genindex`
26+
* :ref:`modindex`
27+
* :ref:`search`

‎python/docs/introduction.rst‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
Introduction
2+
============
3+
4+
The Feldera Python SDK is meant to provide an easy and convenient way of
5+
interacting with Feldera.
6+
7+
8+
Please submit any feature request / bug reports to:
9+
https://github.com/feldera/feldera
10+
11+
12+
Installation
13+
*************
14+
15+
.. code-block:: bash
16+
17+
pip install git+https://github.com/feldera/feldera#subdirectory=python
18+
19+
20+
Similarly, to install from a specific branch:
21+
22+
.. code-block:: bash
23+
24+
$ pip install git+https://github.com/feldera/feldera@{BRANCH_NAME}#subdirectory=python
25+
26+
Replace ``{BRANCH_NAME}`` with the name of the branch you want to install from.
27+
28+
29+
Key Concepts
30+
************
31+
32+
* :class:`feldera.FelderaClient` or :class:`.Client`
33+
- This is the actual HTTP client used to make requests to your Feldera
34+
instance.
35+
- creating an instance of :class:`.Client` is usually the first thing you
36+
will do while working with Feldera.
37+
38+
- Example:
39+
40+
.. code-block:: python
41+
42+
from feldera import FelderaClient
43+
44+
client = FelderaClient("https://try.feldera.com", api_key="YOUR_API_KEY")
45+
46+
- The API key may not be required if you are running Feldera locally.
47+
48+
49+
50+
* :class:`.SQLContext`
51+
- This represents the current context of your SQL program, data sources
52+
and sinks. In Feldera terminology, this represents both a Program and a
53+
Pipeline.
54+
55+
- Example:
56+
57+
.. code-block:: python
58+
59+
from feldera import SQLContext
60+
61+
sql = SQLContext("getting_started", client).get_or_create()
62+
63+
- The first parameter is the name of this SQL context. By default, this is
64+
the name used in both Feldera Program and Pipeline.
65+
- The second parameter here is :class:`.Client` that we created above.
66+
67+
* :meth:`.SQLContext.run_to_completion`
68+
- Runs this Feldera pipeline to completion. Normally this means until the EoF
69+
has been reached for this input source.
70+
71+
- Example:
72+
73+
.. code-block:: python
74+
75+
from feldera import SQLSchema
76+
77+
tbl_name = "user_data"
78+
view_name = "select_view"
79+
80+
# Declare input tables
81+
sql.register_table(tbl_name, SQLSchema({"name": "STRING"}))
82+
83+
# Register Views based on your queries
84+
query = f"SELECT * FROM {tbl_name}"
85+
sql.register_view(view_name, query)
86+
87+
# name for this connector
88+
in_con = "delta_input_conn"
89+
90+
# the configuration for this input source
91+
in_cfg = {...}
92+
93+
sql.connect_source_delta_table(tbl_name, in_con, in_cfg)
94+
95+
# name for this connector
96+
out_con = "delta_output_con"
97+
98+
# the configuration for this input source
99+
out_cfg = {...}
100+
101+
sql.connect_sink_delta_table(view_name, out_con, out_cfg)
102+
103+
sql.run_to_completion()
104+
105+
- Here, we register a data table which receives data from input sources.
106+
- Then, we register a view that performs operations on this input data.
107+
You can also register other views on top of existing views.
108+
- Then, we connect a source delta table to the previously defined table.
109+
- Then, we connect a sink delta table to the previously defined view.
110+
- Finally, we run the pipeline to completion. Feldera will fetch data from
111+
the source, perform the query you supplied and passes this data to the
112+
sink delta table.
113+
114+
.. warning::
115+
If the data source is streaming, this will block forever.
116+
In such cases, use :meth:`.SQLContext.start` instead.
117+
118+
* :meth:`.SQLContext.start`
119+
- Starts the Feldera Pipeline and keeps it running indefinitely.
120+
- Example:
121+
122+
.. code-block:: python
123+
124+
sql.start()
125+
126+
- This tells Feldera to go ahead and start processing the data.
127+
128+
Checkout the :doc:`/examples`.

‎python/feldera/output_handler.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,8 @@ def run(self):
2525
"""
2626
The main loop of the thread. It listens to the pipeline and appends the data to the buffer.
2727
Doesn't do integration, just takes the data and ignores if they are `insert`s or `delete`s.
28+
29+
:meta private:
2830
"""
2931

3032
ack: _OutputHandlerInstruction = self.queue.get()

‎python/feldera/rest/__init__.py‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,11 @@
1+
"""
2+
This is the lower level REST client for Feldera.
3+
4+
This is a thin wrapper around the Feldera REST API.
5+
6+
It is recommended to use the higher level abstractions in the `feldera` package,
7+
instead of using the REST client directly.
8+
9+
"""
10+
111
from feldera.rest.client import Client

‎python/feldera/rest/attached_connector.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,13 @@ class AttachedConnector:
88
"""
99

1010
def __init__(self, connector_name: str, relation_name: str, is_input: bool, name: Optional[str] = None):
11+
"""
12+
:param connector_name: The name of the connector.
13+
:param relation_name: The name of the relation / table / view to attach to.
14+
:param is_input: True if the connector is to be used for input.
15+
:param name: A unique name for this connector instance.
16+
"""
17+
1118
self.name: str = name or str(uuid.uuid4())
1219
self.is_input: bool = is_input
1320
self.connector_name: str = connector_name

0 commit comments

Comments
 (0)