Skip to content

Commit 77d9838

Browse files
github-actions[bot]github-actions[bot]
committed
Publish built docs triggered by 5646693 
1 parent a451068 commit 77d9838

23 files changed

Lines changed: 887 additions & 206 deletions

File tree

‎_sources/autoapi/datafusion/catalog/index.rst.txt‎

Lines changed: 27 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ Module Contents
6363

6464

6565

66-
.. py:method:: register_schema(name, schema) -> Schema | None
66+
.. py:method:: register_schema(name: str, schema: Schema | SchemaProvider | SchemaProviderExportable) -> Schema | None
6767
6868
Register a schema with this catalog.
6969

@@ -162,9 +162,9 @@ Module Contents
162162

163163

164164

165-
.. py:method:: register_table(name, table) -> None
165+
.. py:method:: register_table(name: str, table: Table | datafusion.context.TableProviderExportable | datafusion.DataFrame | pyarrow.dataset.Dataset) -> None
166166
167-
Register a table provider in this schema.
167+
Register a table in this schema.
168168

169169

170170

@@ -191,7 +191,7 @@ Module Contents
191191
Abstract class for defining a Python based Schema Provider.
192192

193193

194-
.. py:method:: deregister_table(name, cascade: bool) -> None
194+
.. py:method:: deregister_table(name: str, cascade: bool) -> None
195195
196196
Remove a table from this schema.
197197

@@ -208,9 +208,9 @@ Module Contents
208208

209209

210210

211-
.. py:method:: register_table(name: str, table: Table) -> None
211+
.. py:method:: register_table(name: str, table: Table | datafusion.context.TableProviderExportable | Any) -> None
212212
213-
Add a table from this schema.
213+
Add a table to this schema.
214214

215215
This method is optional. If your schema provides a fixed list of tables, you do
216216
not need to implement this method.
@@ -241,11 +241,20 @@ Module Contents
241241

242242

243243

244-
.. py:class:: Table(table: datafusion._internal.catalog.RawTable)
244+
.. py:class:: Table(table: Table | datafusion.context.TableProviderExportable | datafusion.DataFrame | pyarrow.dataset.Dataset)
245245
246-
DataFusion table.
246+
A DataFusion table.
247247

248-
This constructor is not typically called by the end user.
248+
Internally we currently support the following types of tables:
249+
250+
- Tables created using built-in DataFusion methods, such as
251+
reading from CSV or Parquet
252+
- pyarrow datasets
253+
- DataFusion DataFrames, which will be converted into a view
254+
- Externally provided tables implemented with the FFI PyCapsule
255+
interface (advanced)
256+
257+
Constructor.
249258

250259

251260
.. py:method:: __repr__() -> str
@@ -258,8 +267,16 @@ Module Contents
258267
:staticmethod:
259268

260269

261-
Turn a pyarrow Dataset into a Table.
270+
Turn a :mod:`pyarrow.dataset` ``Dataset`` into a :class:`Table`.
271+
272+
262273

274+
.. py:attribute:: __slots__
275+
:value: ('_inner',)
276+
277+
278+
279+
.. py:attribute:: _inner
263280
264281
265282
.. py:property:: kind
@@ -276,6 +293,3 @@ Module Contents
276293
Returns the schema associated with this table.
277294

278295

279-
.. py:attribute:: table
280-
281-

‎_sources/autoapi/datafusion/context/index.rst.txt‎

Lines changed: 9 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -663,14 +663,10 @@ Module Contents
663663

664664

665665

666-
.. py:method:: read_table(table: datafusion.catalog.Table) -> datafusion.dataframe.DataFrame
666+
.. py:method:: read_table(table: datafusion.catalog.Table | TableProviderExportable | datafusion.dataframe.DataFrame | pyarrow.dataset.Dataset) -> datafusion.dataframe.DataFrame
667667
668668
Creates a :py:class:`~datafusion.dataframe.DataFrame` from a table.
669669

670-
For a :py:class:`~datafusion.catalog.Table` such as a
671-
:py:class:`~datafusion.catalog.ListingTable`, create a
672-
:py:class:`~datafusion.dataframe.DataFrame`.
673-
674670

675671

676672
.. py:method:: register_avro(name: str, path: str | pathlib.Path, schema: pyarrow.Schema | None = None, file_extension: str = '.avro', table_partition_cols: list[tuple[str, str | pyarrow.DataType]] | None = None) -> None
@@ -809,23 +805,23 @@ Module Contents
809805

810806

811807

812-
.. py:method:: register_table(name: str, table: datafusion.catalog.Table) -> None
808+
.. py:method:: register_table(name: str, table: datafusion.catalog.Table | TableProviderExportable | datafusion.dataframe.DataFrame | pyarrow.dataset.Dataset) -> None
813809
814-
Register a :py:class: `~datafusion.catalog.Table` as a table.
810+
Register a :py:class:`~datafusion.Table` with this context.
815811

816-
The registered table can be referenced from SQL statement executed against.
812+
The registered table can be referenced from SQL statements executed against
813+
this context.
817814

818815
:param name: Name of the resultant table.
819-
:param table: DataFusion table to add to the session context.
816+
:param table: Any object that can be converted into a :class:`Table`.
820817

821818

822819

823-
.. py:method:: register_table_provider(name: str, provider: TableProviderExportable) -> None
820+
.. py:method:: register_table_provider(name: str, provider: datafusion.catalog.Table | TableProviderExportable | datafusion.dataframe.DataFrame | pyarrow.dataset.Dataset) -> None
824821
825822
Register a table provider.
826823

827-
This table provider must have a method called ``__datafusion_table_provider__``
828-
which returns a PyCapsule that exposes a ``FFI_TableProvider``.
824+
Deprecated: use :meth:`register_table` instead.
829825

830826

831827

@@ -855,7 +851,7 @@ Module Contents
855851

856852
.. py:method:: register_view(name: str, df: datafusion.dataframe.DataFrame) -> None
857853
858-
Register a :py:class: `~datafusion.detaframe.DataFrame` as a view.
854+
Register a :py:class:`~datafusion.dataframe.DataFrame` as a view.
859855

860856
:param name: The name to register the view under.
861857
:type name: str

‎_sources/autoapi/datafusion/dataframe/index.rst.txt‎

Lines changed: 120 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@ Classes
1818

1919
datafusion.dataframe.Compression
2020
datafusion.dataframe.DataFrame
21+
datafusion.dataframe.DataFrameWriteOptions
22+
datafusion.dataframe.InsertOp
2123
datafusion.dataframe.ParquetColumnOptions
2224
datafusion.dataframe.ParquetWriterOptions
2325

@@ -240,10 +242,20 @@ Module Contents
240242
241243
Drop arbitrary amount of columns.
242244

243-
:param columns: Column names to drop from the dataframe.
245+
Column names are case-sensitive and do not require double quotes like
246+
other operations such as `select`. Leading and trailing double quotes
247+
are allowed and will be automatically stripped if present.
248+
249+
:param columns: Column names to drop from the dataframe. Both ``column_name``
250+
and ``"column_name"`` are accepted.
244251

245252
:returns: DataFrame with those columns removed in the projection.
246253

254+
Example Usage::
255+
256+
df.drop('ID_For_Students') # Works
257+
df.drop('"ID_For_Students"') # Also works (quotes stripped)
258+
247259

248260

249261
.. py:method:: except_all(other: DataFrame) -> DataFrame
@@ -361,9 +373,19 @@ Module Contents
361373

362374

363375

364-
.. py:method:: into_view() -> pyarrow.Table
376+
.. py:method:: into_view() -> datafusion.catalog.Table
365377
366-
Convert DataFrame as a ViewTable which can be used in register_table.
378+
Convert ``DataFrame`` into a :class:`~datafusion.Table`.
379+
380+
.. rubric:: Examples
381+
382+
>>> from datafusion import SessionContext
383+
>>> ctx = SessionContext()
384+
>>> df = ctx.sql("SELECT 1 AS value")
385+
>>> view = df.into_view()
386+
>>> ctx.register_table("values_view", view)
387+
>>> df.collect() # The DataFrame is still usable
388+
>>> ctx.sql("SELECT value FROM values_view").collect()
367389

368390

369391

@@ -436,6 +458,27 @@ Module Contents
436458

437459

438460

461+
.. py:method:: parse_sql_expr(expr: str) -> datafusion.expr.Expr
462+
463+
Creates logical expression from a SQL query text.
464+
465+
The expression is created and processed against the current schema.
466+
467+
Example::
468+
469+
from datafusion import col, lit
470+
df.parse_sql_expr("a > 1")
471+
472+
should produce:
473+
474+
col("a") > lit(1)
475+
476+
:param expr: Expression string to be converted to datafusion expression
477+
478+
:returns: Logical expression .
479+
480+
481+
439482
.. py:method:: repartition(num: int) -> DataFrame
440483
441484
Repartition a DataFrame into ``num`` partitions.
@@ -693,60 +736,119 @@ Module Contents
693736

694737

695738

696-
.. py:method:: write_csv(path: str | pathlib.Path, with_header: bool = False) -> None
739+
.. py:method:: write_csv(path: str | pathlib.Path, with_header: bool = False, write_options: DataFrameWriteOptions | None = None) -> None
697740
698741
Execute the :py:class:`DataFrame` and write the results to a CSV file.
699742

700743
:param path: Path of the CSV file to write.
701744
:param with_header: If true, output the CSV header row.
745+
:param write_options: Options that impact how the DataFrame is written.
702746

703747

704748

705-
.. py:method:: write_json(path: str | pathlib.Path) -> None
749+
.. py:method:: write_json(path: str | pathlib.Path, write_options: DataFrameWriteOptions | None = None) -> None
706750
707751
Execute the :py:class:`DataFrame` and write the results to a JSON file.
708752

709753
:param path: Path of the JSON file to write.
754+
:param write_options: Options that impact how the DataFrame is written.
710755

711756

712757

713-
.. py:method:: write_parquet(path: str | pathlib.Path, compression: str, compression_level: int | None = None) -> None
714-
write_parquet(path: str | pathlib.Path, compression: Compression = Compression.ZSTD, compression_level: int | None = None) -> None
715-
write_parquet(path: str | pathlib.Path, compression: ParquetWriterOptions, compression_level: None = None) -> None
758+
.. py:method:: write_parquet(path: str | pathlib.Path, compression: str, compression_level: int | None = None, write_options: DataFrameWriteOptions | None = None) -> None
759+
write_parquet(path: str | pathlib.Path, compression: Compression = Compression.ZSTD, compression_level: int | None = None, write_options: DataFrameWriteOptions | None = None) -> None
760+
write_parquet(path: str | pathlib.Path, compression: ParquetWriterOptions, compression_level: None = None, write_options: DataFrameWriteOptions | None = None) -> None
716761
717762
Execute the :py:class:`DataFrame` and write the results to a Parquet file.
718763

764+
Available compression types are:
765+
766+
- "uncompressed": No compression.
767+
- "snappy": Snappy compression.
768+
- "gzip": Gzip compression.
769+
- "brotli": Brotli compression.
770+
- "lz4": LZ4 compression.
771+
- "lz4_raw": LZ4_RAW compression.
772+
- "zstd": Zstandard compression.
773+
774+
LZO compression is not yet implemented in arrow-rs and is therefore
775+
excluded.
776+
719777
:param path: Path of the Parquet file to write.
720778
:param compression: Compression type to use. Default is "ZSTD".
721-
Available compression types are:
722-
- "uncompressed": No compression.
723-
- "snappy": Snappy compression.
724-
- "gzip": Gzip compression.
725-
- "brotli": Brotli compression.
726-
- "lz4": LZ4 compression.
727-
- "lz4_raw": LZ4_RAW compression.
728-
- "zstd": Zstandard compression.
729-
:param Note: LZO is not yet implemented in arrow-rs and is therefore excluded.
730779
:param compression_level: Compression level to use. For ZSTD, the
731780
recommended range is 1 to 22, with the default being 4. Higher levels
732781
provide better compression but slower speed.
782+
:param write_options: Options that impact how the DataFrame is written.
733783

734784

735785

736-
.. py:method:: write_parquet_with_options(path: str | pathlib.Path, options: ParquetWriterOptions) -> None
786+
.. py:method:: write_parquet_with_options(path: str | pathlib.Path, options: ParquetWriterOptions, write_options: DataFrameWriteOptions | None = None) -> None
737787
738788
Execute the :py:class:`DataFrame` and write the results to a Parquet file.
739789

740790
Allows advanced writer options to be set with `ParquetWriterOptions`.
741791

742792
:param path: Path of the Parquet file to write.
743793
:param options: Sets the writer parquet options (see `ParquetWriterOptions`).
794+
:param write_options: Options that impact how the DataFrame is written.
795+
796+
797+
798+
.. py:method:: write_table(table_name: str, write_options: DataFrameWriteOptions | None = None) -> None
799+
800+
Execute the :py:class:`DataFrame` and write the results to a table.
801+
802+
The table must be registered with the session to perform this operation.
803+
Not all table providers support writing operations. See the individual
804+
implementations for details.
744805

745806

746807

747808
.. py:attribute:: df
748809
749810
811+
.. py:class:: DataFrameWriteOptions(insert_operation: InsertOp | None = None, single_file_output: bool = False, partition_by: str | collections.abc.Sequence[str] | None = None, sort_by: datafusion.expr.Expr | datafusion.expr.SortExpr | collections.abc.Sequence[datafusion.expr.Expr] | collections.abc.Sequence[datafusion.expr.SortExpr] | None = None)
812+
813+
Writer options for DataFrame.
814+
815+
There is no guarantee the table provider supports all writer options.
816+
See the individual implementation and documentation for details.
817+
818+
Instantiate writer options for DataFrame.
819+
820+
821+
.. py:attribute:: _raw_write_options
822+
823+
824+
.. py:class:: InsertOp(*args, **kwds)
825+
826+
Bases: :py:obj:`enum.Enum`
827+
828+
829+
Insert operation mode.
830+
831+
These modes are used by the table writing feature to define how record
832+
batches should be written to a table.
833+
834+
835+
.. py:attribute:: APPEND
836+
837+
Appends new rows to the existing table without modifying any existing rows.
838+
839+
840+
.. py:attribute:: OVERWRITE
841+
842+
Overwrites all existing rows in the table with the new rows.
843+
844+
845+
.. py:attribute:: REPLACE
846+
847+
Replace existing rows that collide with the inserted rows.
848+
849+
Replacement is typically based on a unique key or primary key.
850+
851+
750852
.. py:class:: ParquetColumnOptions(encoding: Optional[str] = None, dictionary_enabled: Optional[bool] = None, compression: Optional[str] = None, statistics_enabled: Optional[str] = None, bloom_filter_enabled: Optional[bool] = None, bloom_filter_fpp: Optional[float] = None, bloom_filter_ndv: Optional[int] = None)
751853
752854
Parquet options for individual columns.

0 commit comments

Comments
 (0)