diff --git a/README.md b/README.md index 6face6c61..1b4c602a5 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,14 @@ -GraphFrames Logo +GraphFrames Logo [![Scala CI](https://github.com/graphframes/graphframes/actions/workflows/scala-ci.yml/badge.svg)](https://github.com/graphframes/graphframes/actions/workflows/scala-ci.yml) [![Python CI](https://github.com/graphframes/graphframes/actions/workflows/python-ci.yml/badge.svg)](https://github.com/graphframes/graphframes/actions/workflows/python-ci.yml) [![pages-build-deployment](https://github.com/graphframes/graphframes/actions/workflows/pages/pages-build-deployment/badge.svg)](https://github.com/graphframes/graphframes/actions/workflows/pages/pages-build-deployment) [![scala-central-publish](https://github.com/graphframes/graphframes/actions/workflows/scala-publish.yml/badge.svg)](https://github.com/graphframes/graphframes/actions/workflows/scala-publish.yml) [![python-pypi-publish](https://github.com/graphframes/graphframes/actions/workflows/python-publish.yml/badge.svg)](https://github.com/graphframes/graphframes/actions/workflows/python-publish.yml) +![GitHub Release](https://img.shields.io/github/v/release/graphframes/graphframes) +![GitHub License](https://img.shields.io/github/license/graphframes/graphframes) + + # GraphFrames: DataFrame-based Graphs @@ -26,19 +30,19 @@ GraphFrames scala core and Spark-Connect plugin are published in the Sonatype Ce # Interactive Scala/Java # For Spark 3.5.x, scala 2.12 -$ spark-shell --packages io.graphframes:graphframes-spark3_2.12:0.9.0 +$ spark-shell --packages io.graphframes:graphframes-spark3_2.12:0.9.1 # For Spark 3.5.x, scala 2.13 -$ spark-shell --packages io.graphframes:graphframes-spark3_2.13:0.9.0 +$ spark-shell --packages io.graphframes:graphframes-spark3_2.13:0.9.1 # For Spark 4.0.x -$ spark-shell --packages io.graphframes:graphframes-spark4_2.13:0.9.0 +$ spark-shell --packages io.graphframes:graphframes-spark4_2.13:0.9.1 # Interactive Python, Spark 3.5.x -$ pyspark --packages io.graphframes:graphframes-spark3_2.12:0.9.0 +$ pyspark --packages io.graphframes:graphframes-spark3_2.12:0.9.1 # Interactive Python, Spark 4.0.x -$ pyspark --packages io.graphframes:graphframes-spark4_2.13:0.9.0 +$ pyspark --packages io.graphframes:graphframes-spark4_2.13:0.9.1 ``` ### GraphFrames Python API @@ -58,7 +62,7 @@ To add GraphFrames to your spark connect server, you need to specify the plugin ```bash ./sbin/start-connect-server.sh \ --conf spark.connect.extensions.relation.classes=org.apache.spark.sql.graphframes.GraphFramesConnect \ - --packages io.graphframes.graphframes-connect-spark4_2.13:0.9.0 + --packages io.graphframes.graphframes-connect-spark4_2.13:0.9.1 --conf spark.checkpoint.dir=${CHECKPOINT_DIR} ``` @@ -208,14 +212,14 @@ Maven: io.graphframes graphframes-spark4_2.13 - 0.9.0 + 0.9.1 ``` SBT: ```sbt -libraryDependencies += "io.graphframes" %% "graphframes-spark4" % "0.9.0" +libraryDependencies += "io.graphframes" %% "graphframes-spark4" % "0.9.1" ``` **WARNING!** diff --git a/docs/_config.yml b/docs/_config.yml index 4c1ab075c..868f146b9 100644 --- a/docs/_config.yml +++ b/docs/_config.yml @@ -13,7 +13,7 @@ include: # These allow the documentation to be updated with newer releases # of Spark, Scala, and Mesos. -GRAPHFRAMES_VERSION: 0.8.4 +GRAPHFRAMES_VERSION: 0.9.1 #SCALA_BINARY_VERSION: "2.10" #SCALA_VERSION: "2.10.4" #MESOS_VERSION: 0.21.0 diff --git a/docs/_plugins/copy_api_dirs.rb b/docs/_plugins/copy_api_dirs.rb index 820809902..f744a330f 100644 --- a/docs/_plugins/copy_api_dirs.rb +++ b/docs/_plugins/copy_api_dirs.rb @@ -37,7 +37,7 @@ # Copy over the unified ScalaDoc for all projects to api/scala. # This directory will be copied over to _site when `jekyll` command is run. - source = "../target/scala-2.12/api" + source = "../core/target/scala-2.13/api" dest = "api/scala" puts "Making directory " + dest diff --git a/python/README.md b/python/README.md index 0ab7cd4ba..e2f02cf55 100644 --- a/python/README.md +++ b/python/README.md @@ -1,17 +1,43 @@ -# GraphFrames `graphframes-py` Python Package +# GraphFrames Python Package -The is the officila [graphframes-py PyPI package](https://pypi.org/project/graphframes-py/), which is a Python wrapper for the Scala GraphFrames library. This package is maintained by the GraphFrames project and is available on PyPI. +![PyPI - Downloads](https://img.shields.io/pypi/dm/graphframes-py) ![PyPI - License](https://img.shields.io/pypi/l/graphframes-py) ![PyPI - Version](https://img.shields.io/pypi/v/graphframes-py) -For instructions on GraphFrames, check the project [../README.md](../README.md). See [Installation and Quick-Start](#installation-and-quick-start) for the best way to install and use GraphFrames. + + +https://graphframes.io/ + +The is the officila [graphframes-py PyPI package](https://pypi.org/project/graphframes-py/), which is a Python wrapper for the Scala GraphFrames library. +This package is maintained by the GraphFrames project and is available on PyPI. + +For instructions on GraphFrames, check the project [README.md](https://github.com/graphframes/graphframes?tab=readme-ov-file#graphframes-dataframe-based-graphs). + +See [Installation and Quick-Start](https://github.com/graphframes/graphframes?tab=readme-ov-file#installation-and-quick-start) for the best way to install and use GraphFrames. + +## Installation + +```bash +pip install graphframes-py +``` + +**NOTE!** *Python distribution does not include JVM-core. You need to add it to your cluster or Spark-Connect server!* ## Running `graphframes-py` You should use GraphFrames via the `--packages` argument to `pyspark` or `spark-submit`, but this package is helpful in development environments. ```bash -# Interactive Python -$ pyspark --packages graphframes:graphframes:0.8.4-spark3.5-s_2.12 +# Interactive Python, Spark 3.5.x +$ pyspark --packages io.graphframes:graphframes-spark3_2.12:0.9.1 -# Submit a script in Scala/Java/Python -$ spark-submit --packages graphframes:graphframes:0.8.4-spark3.5-s_2.12 script.py +# Interactive Python, Spark 4.0.x +$ pyspark --packages io.graphframes:graphframes-spark4_2.13:0.9.1 ``` + +## Documentation + +- [API Reference](https://graphframes.io/api/python/index.html) + +## Spark-Connect Note + +GraphFrames PySpark is choosing connect or classic implementation implicitly based on the result of `is_remote()`. +To enforce usage of connect-based implementation, you may export this variable `SPARK_CONNECT_MODE_ENABLED=1`