-{% highlight python %}
+```python
# G17: A directed 3-path is a surprisingly diverse graphlet
paths = g.find("(a)-[e1]->(b); (b)-[e2]->(c); (d)-[e3]->(c)")
-{% endhighlight %}
-
+```
Let's count the number of instances by type for of this path in the graph. To let you know of a hard-won tip: alias the edge with its pattern. This makes it easier to read the results, even when C points to A or B, not D.
-
-{% highlight python %}
+```python
# Visualize the four-path by counting instances of paths by node / edge type
graphlet_type_df = paths.select(
F.col("a.Type").alias("A_Type"),
@@ -535,11 +483,9 @@ graphlet_count_df = (
.withColumn("count", F.format_number(F.col("count"), 0))
)
graphlet_count_df.show()
-{% endhighlight %}
-
+```
-
-{% highlight python %}
+```
+--------+-------------+--------+-------------+--------+-------------+--------+-------+
| A_Type|(a)-[e1]->(b)| B_Type|(b)-[e2]->(c)| C_Type|(d)-[e3]->(c)| D_Type| count|
+--------+-------------+--------+-------------+--------+-------------+--------+-------+
@@ -565,22 +511,18 @@ graphlet_count_df.show()
| User| Posts| Answer| Answers|Question| Answers| Answer| 12,105|
+--------+-------------+--------+-------------+--------+-------------+--------+-------+
only showing top 20 rows
-{% endhighlight %}
-
+```
How many of these motifs are there in the graph? Let's count them.
-
-{% highlight python %}
+```python
graphlet_count_df.count()
104
-{% endhighlight %}
-
+```
104 - we hit the network motif jackpot with this pattern! Let's order by the successive elements of the pattern to group them logically.
-
-{% highlight python %}
+```python
graphlet_count_df.orderBy([
"A_Type",
"(a)-[e1]->(b)",
@@ -590,11 +532,9 @@ graphlet_count_df.orderBy([
"(d)-[e3]->(c)",
"D_Type",
], ascending=False).show(104)
-{% endhighlight %}
-
+```
-
-{% highlight python %}
+```
+--------+-------------+--------+-------------+--------+-------------+--------+-------+
| A_Type|(a)-[e1]->(b)| B_Type|(b)-[e2]->(c)| C_Type|(d)-[e3]->(c)| D_Type| count|
+--------+-------------+--------+-------------+--------+-------------+--------+-------+
@@ -703,13 +643,11 @@ graphlet_count_df.orderBy([
| Answer| Answers|Question| Duplicates|Question| Asks| User| 110|
| Answer| Answers|Question| Duplicates|Question| Answers| Answer| 1,174|
+--------+-------------+--------+-------------+--------+-------------+--------+-------+
-{% endhighlight %}
-
+```
-The fourth row catches my eye - there are 300,017 matches for the votes cast for linked questions:
-{% highlight python %}
+```python
# A user answers an answer that answers a question that links to an answer.
linked_vote_paths = paths.filter(
(F.col("a.Type") == "Vote") &
@@ -726,22 +664,18 @@ linked_vote_paths = paths.filter(
linked_vote_paths.count()
300017
-{% endhighlight %}
-
+```
We start by using aggregation to count the total votes cast for each end of a link questions. To get the count for Question B, get the distinct 3-paths, group by it's ID and count the votes.
-
-{% highlight python %}
+```python
b_vote_counts = linked_vote_paths.select("a", "b").distinct().groupBy("b").count()
c_vote_counts = linked_vote_paths.select("c", "d").distinct().groupBy("c").count()
-{% endhighlight %}
-
+```
Now join the counts to the links to get the total votes for each pair of linked questions. Then run `pyspark.sql.DataFrame.stats.corr()` to get the correlation between the vote counts for linked questions. We'll use the `Vote.VoteTypeId` to ensure only positive votes are counted.
-
-{% highlight python %}
+```python
linked_vote_counts = (
linked_vote_paths
.filter((F.col("a.VoteTypeId") == 2) & (F.col("d.VoteTypeId") == 2))
@@ -753,12 +687,12 @@ linked_vote_counts = (
)
linked_vote_counts.stat.corr("b_count", "c_count")
0.4287709940689788
-{% endhighlight %}
+```
We conclude there is a moderate correlation in the vote counts of linked questions. This makes sense. Note that this is only the fourth row, there are many more patterns to be examined and considered.
This is just one type of aggregation you can employ - but hopefully it illustrates the way properties and aggregation and other relational operators can transform simple pattern matching into a powerful tool for exploring a knowledge graph.
-
Conclusion
+# Conclusion
In this tutorial, we learned to use GraphFrames to find network motifs in a property graph. We saw how to combine graph and relational queries to find complex patterns in the graph. We also saw how to use the properties of the nodes and edges in the paths to filter, group, and aggregate the results to form complex
property graph motifs. Motif finding in GraphFrames is a powerful technique that can be used to explore and understand complex networks. Network motifs are the building blocks of complex networks.
diff --git a/docs/src/03-tutorials/directory.conf b/docs/src/03-tutorials/directory.conf
new file mode 100644
index 000000000..211123256
--- /dev/null
+++ b/docs/src/03-tutorials/directory.conf
@@ -0,0 +1 @@
+laika.title = Tutorials
\ No newline at end of file
diff --git a/docs/src/04-user-guide/01-creating-graphframes.md b/docs/src/04-user-guide/01-creating-graphframes.md
new file mode 100644
index 000000000..c266d9a99
--- /dev/null
+++ b/docs/src/04-user-guide/01-creating-graphframes.md
@@ -0,0 +1,88 @@
+# Creating GraphFrames
+
+Users can create GraphFrames from vertex and edge DataFrames.
+
+* *Vertex DataFrame*: A vertex DataFrame should contain a special column named "id" which specifies unique IDs for each vertex in the graph.
+* *Edge DataFrame*: An edge DataFrame should contain two special columns: "src" (source vertex ID of edge) and "dst" (destination vertex ID of edge).
+
+Both DataFrames can have arbitrary other columns. Those columns can represent vertex and edge attributes.
+
+A GraphFrame can also be constructed from a single DataFrame containing edge information. The vertices will be inferred from the sources and destinations of the edges.
+
+The following example demonstrates how to create a GraphFrame from vertex and edge DataFrames.
+
+## Python API
+
+```python
+# Vertex DataFrame
+v = spark.createDataFrame([
+ ("a", "Alice", 34),
+ ("b", "Bob", 36),
+ ("c", "Charlie", 30),
+ ("d", "David", 29),
+ ("e", "Esther", 32),
+ ("f", "Fanny", 36),
+ ("g", "Gabby", 60)
+], ["id", "name", "age"])
+
+# Edge DataFrame
+e = spark.createDataFrame([
+ ("a", "b", "friend"),
+ ("b", "c", "follow"),
+ ("c", "b", "follow"),
+ ("f", "c", "follow"),
+ ("e", "f", "follow"),
+ ("e", "d", "friend"),
+ ("d", "a", "friend"),
+ ("a", "e", "friend")
+], ["src", "dst", "relationship"])
+# Create a GraphFrame
+g = GraphFrame(v, e)
+```
+
+The GraphFrame constructed above is available in the GraphFrames package (not available in the Spark-Connect mode:
+
+```python
+from graphframes.examples import Graphs
+
+g = Graphs(spark).friends() # Get example graph
+```
+
+## Scala API
+
+```scala
+import org.graphframes.GraphFrame
+
+// Vertex DataFrame
+val v = spark.createDataFrame(List(
+ ("a", "Alice", 34),
+ ("b", "Bob", 36),
+ ("c", "Charlie", 30),
+ ("d", "David", 29),
+ ("e", "Esther", 32),
+ ("f", "Fanny", 36),
+ ("g", "Gabby", 60)
+)).toDF("id", "name", "age")
+
+// Edge DataFrame
+val e = spark.createDataFrame(List(
+ ("a", "b", "friend"),
+ ("b", "c", "follow"),
+ ("c", "b", "follow"),
+ ("f", "c", "follow"),
+ ("e", "f", "follow"),
+ ("e", "d", "friend"),
+ ("d", "a", "friend"),
+ ("a", "e", "friend")
+)).toDF("src", "dst", "relationship")
+// Create a GraphFrame
+val g = GraphFrame(v, e)
+```
+
+The GraphFrame constructed above is available in the GraphFrames package:
+
+```scala
+import org.graphframes.{examples, GraphFrame}
+
+val g: GraphFrame = examples.Graphs.friends
+```
\ No newline at end of file
diff --git a/docs/src/04-user-guide/02-basic-operations.md b/docs/src/04-user-guide/02-basic-operations.md
new file mode 100644
index 000000000..cb808fac1
--- /dev/null
+++ b/docs/src/04-user-guide/02-basic-operations.md
@@ -0,0 +1,104 @@
+# Basic Graph Operations
+
+GraphFrames provide several simple graph queries, such as node degree. Also, since GraphFrames represent graphs as pairs of vertex and edge DataFrames, it is easy to make powerful queries directly on the vertex and edge DataFrames. Those DataFrames are made available as `vertices` and `edges` fields in the GraphFrame.
+
+## Python API
+
+```python
+from graphframes.examples import Graphs
+
+g = Graphs(spark).friends() # Get example graph
+
+# Display the vertex DataFrame
+g.vertices.show()
+
+# +--+-------+---+
+# |id| name|age|
+# +--+-------+---+
+# | a| Alice| 34|
+# | b| Bob| 36|
+# | c|Charlie| 30|
+# | d| David| 29|
+# | e| Esther| 32|
+# | f| Fanny| 36|
+# | g| Gabby| 60|
+# +--+-------+---+
+
+# Display the edge DataFrame
+g.edges.show()
+
+# +---+---+------------+
+# |src|dst|relationship|
+# +---+---+------------+
+# | a| b| friend|
+# | b| c| follow|
+# | c| b| follow|
+# | f| c| follow|
+# | e| f| follow|
+# | e| d| friend|
+# | d| a| friend|
+# | a| e| friend|
+# +---+---+------------+
+
+# Get a DataFrame with columns "id" and "inDegree" (in-degree)
+vertexInDegrees = g.inDegrees
+
+# Find the youngest user's age in the graph
+# This queries the vertex DataFrame
+g.vertices.groupBy().min("age").show()
+
+# Count the number of "follows" in the graph
+# This queries the edge DataFrame
+numFollows = g.edges.filter("relationship = 'follow'").count()
+```
+
+## Scala API
+
+```scala
+import org.graphframes.{examples,GraphFrame}
+
+val g: GraphFrame = examples.Graphs.friends // get example graph
+
+// Display the vertex and edge DataFrames
+g.vertices.show()
+// +--+-------+---+
+// |id| name|age|
+// +--+-------+---+
+// | a| Alice| 34|
+// | b| Bob| 36|
+// | c|Charlie| 30|
+// | d| David| 29|
+// | e| Esther| 32|
+// | f| Fanny| 36|
+// | g| Gabby| 60|
+// +--+-------+---+
+
+g.edges.show()
+// +---+---+------------+
+// |src|dst|relationship|
+// +---+---+------------+
+// | a| b| friend|
+// | b| c| follow|
+// | c| b| follow|
+// | f| c| follow|
+// | e| f| follow|
+// | e| d| friend|
+// | d| a| friend|
+// | a| e| friend|
+// +---+---+------------+
+
+// import Spark SQL package
+import org.apache.spark.sql.DataFrame
+
+// Get a DataFrame with columns "id" and "inDeg" (in-degree)
+val vertexInDegrees: DataFrame = g.inDegrees
+vertexInDegrees.show()
+
+// Find the youngest user's age in the graph.
+// This queries the vertex DataFrame.
+g.vertices.groupBy().min("age").show()
+
+// Count the number of "follows" in the graph.
+// This queries the edge DataFrame.
+val numFollows = g.edges.filter("relationship = 'follow'").count()
+```
diff --git a/docs/src/04-user-guide/03-centralities.md b/docs/src/04-user-guide/03-centralities.md
new file mode 100644
index 000000000..1d7882b41
--- /dev/null
+++ b/docs/src/04-user-guide/03-centralities.md
@@ -0,0 +1,120 @@
+# Centrality Metrics
+
+## Degrees
+
+GraphFrames provides three main APIs for computing degrees:
+
+- `inDegrees`
+- `outDegrees`
+- `degrees`
+
+### Python API
+
+```python
+from graphframes.examples import Graphs
+
+g = Graphs(spark).friends()
+in_degrees = g.inDegrees()
+out_degrees = g.outDegrees()
+degrees = g.degrees()
+```
+
+### Scala API
+
+```scala
+import org.graphframes.{examples,GraphFrame}
+
+val g: GraphFrame = examples.Graphs.friends
+
+val inDegrees: DataFrame = g.inDegrees
+val outDegrees: DataFrame = g.outDegrees
+val degrees: DataFrame = g.degrees
+```
+
+## PageRank
+
+There are two implementations of PageRank.
+
+* The first one uses the `org.apache.spark.graphx.graph` interface with `aggregateMessages` and runs PageRank for a fixed number of iterations. This can be executed by setting `maxIter`.
+* The second implementation uses the `org.apache.spark.graphx.Pregel` interface and runs PageRank until convergence and this can be run by setting `tol`.
+
+Both implementations support non-personalized and personalized PageRank, where setting a `sourceId` personalizes the results for that vertex.
+
+See [Wikipedia](https://en.wikipedia.org/wiki/PageRank) for a background.
+
+**NOTE:** The `pageRank` API at the moment is the only API in GraphFrames that returns a `GraphFrame` object instead of a `DataFrame`. Most probably, this behavior will change in the nearest major release for the API consistency. It is strongly recommended do not rely on the returned `edges` at all.
+
+### Python API
+
+For API details, refer to the @:pydoc(graphframes.GraphFrame.pageRank).
+
+```python
+from graphframes.examples import Graphs
+
+g = Graphs(spark).friends() # Get example graph
+
+# Run PageRank until convergence to tolerance "tol"
+
+results = g.pageRank(resetProbability=0.15, tol=0.01)
+
+# Display resulting pageranks and final edge weights
+
+# Note that the displayed pagerank may be truncated, e.g., missing the E notation
+
+# In Spark 1.5+, you can use show(truncate=False) to avoid truncation
+
+results.vertices.select("id", "pagerank").show()
+results.edges.select("src", "dst", "weight").show()
+
+# Run PageRank for a fixed number of iterations
+
+results2 = g.pageRank(resetProbability=0.15, maxIter=10)
+
+# Run PageRank personalized for vertex "a"
+
+results3 = g.pageRank(resetProbability=0.15, maxIter=10, sourceId="a")
+
+# Run PageRank personalized for vertex ["a", "b", "c", "d"] in parallel
+
+results4 = g.parallelPersonalizedPageRank(resetProbability=0.15, sourceIds=["a", "b", "c", "d"], maxIter=10)
+```
+
+### Scala API
+
+For API details, refer to the @:scaladoc(org.graphframes.lib.PageRank).
+
+```scala
+import org.graphframes.{examples,GraphFrame}
+
+val g: GraphFrame = examples.Graphs.friends // get example graph
+
+// Run PageRank until convergence to tolerance "tol".
+val results: GraphFrame = g.pageRank.resetProbability(0.15).tol(0.01).run()
+
+// Display resulting pageranks and final edge weights
+// Note that the displayed pagerank may be truncated, e.g., missing the E notation.
+// In Spark 1.5+, you can use show(truncate=false) to avoid truncation.
+results.vertices.select("id", "pagerank").show()
+results.edges.select("src", "dst", "weight").show()
+
+// Run PageRank for a fixed number of iterations.
+val results2 = g.pageRank.resetProbability(0.15).maxIter(10).run()
+
+// Run PageRank personalized for vertex "a"
+val results3 = g.pageRank.resetProbability(0.15).maxIter(10).sourceId("a").run()
+
+// Run PageRank personalized for vertex ["a", "b", "c", "d"] in parallel
+val results4 = g.parallelPersonalizedPageRank.resetProbability(0.15).maxIter(10).sourceIds(Array("a", "b", "c", "d"))
+.run()
+results4.vertices.show()
+results4.edges.show()
+```
+
+### Parallel personalized PageRank
+
+GraphFrames also supports parallel personalized PageRank that allows users to compute ranks "from the subset of source vertices".
+
+For the API details refer to:
+
+* Scala API: @:scaladoc(org.graphframes.lib.ParallelPersonalizedPageRank)
+* Python API: @:pydoc(graphframes.GraphFrame.parallelPersonalizedPageRank)
\ No newline at end of file
diff --git a/docs/src/04-user-guide/04-motif-finding.md b/docs/src/04-user-guide/04-motif-finding.md
new file mode 100644
index 000000000..dd39222a6
--- /dev/null
+++ b/docs/src/04-user-guide/04-motif-finding.md
@@ -0,0 +1,160 @@
+# Motif finding
+
+Motif finding refers to searching for structural patterns in a graph. For an example of real-world use, check out the [Motif Finding Tutorial](/03-tutorials/02-motif-tutorial.md).
+
+GraphFrame motif finding uses a simple Domain-Specific Language (DSL) for expressing structural queries. For example, `graph.find("(a)-[e]->(b); (b)-[e2]->(a)")` will search for pairs of vertices `a,b` connected by edges in both directions. It will return a `DataFrame` of all such structures in the graph, with columns for each of the named elements (vertices or edges) in the motif. In this case, the returned columns will be "a, b, e, e2."
+
+DSL for expressing structural patterns:
+
+* The basic unit of a pattern is an edge.
+ For example, `"(a)-[e]->(b)"` expresses an edge `e` from vertex `a` to vertex `b`.
+ Note that vertices are denoted by parentheses `(a)`, while edges are denoted by
+ square brackets `[e]`.
+* A pattern is expressed as a union of edges. Edge patterns can be joined with semicolons.
+ Motif `"(a)-[e]->(b); (b)-[e2]->(c)"` specifies two edges from `a` to `b` to `c`.
+* Within a pattern, names can be assigned to vertices and edges. For example,
+ `"(a)-[e]->(b)"` has three named elements: vertices `a,b` and edge `e`.
+ These names serve two purposes:
+ * The names can identify common elements among edges. For example,
+ `"(a)-[e]->(b); (b)-[e2]->(c)"` specifies that the same vertex `b` is the destination
+ of edge `e` and source of edge `e2`.
+ * The names are used as column names in the result `DataFrame`. If a motif contains
+ named vertex `a`, then the result `DataFrame` will contain a column "a" which is a
+ `StructType` with sub-fields equivalent to the schema (columns) of
+ `GraphFrame.vertices`. Similarly, an edge `e` in a motif will produce a column "e"
+ in the result `DataFrame` with sub-fields equivalent to the schema (columns) of
+ `GraphFrame.edges`.
+ * Be aware that names do *not* identify *distinct* elements: two elements with different
+ names may refer to the same graph element. For example, in the motif
+ `"(a)-[e]->(b); (b)-[e2]->(c)"`, the names `a` and `c` could refer to the same vertex.
+ To restrict named elements to be distinct vertices or edges, use post-hoc filters
+ such as `resultDataframe.filter("a.id != c.id")`.
+* It is acceptable to omit names for vertices or edges in motifs when not needed.
+ E.g., `"(a)-[]->(b)"` expresses an edge between vertices `a,b` but does not assign a name
+ to the edge. There will be no column for the anonymous edge in the result `DataFrame`.
+ Similarly, `"(a)-[e]->()"` indicates an out-edge of vertex `a` but does not name
+ the destination vertex. These are called *anonymous* vertices and edges.
+* An edge can be negated to indicate that the edge should *not* be present in the graph.
+ E.g., `"(a)-[]->(b); !(b)-[]->(a)"` finds edges from `a` to `b` for which there is *no*
+ edge from `b` to `a`.
+
+Restrictions:
+
+* Motifs are not allowed to contain edges without any named elements: `"()-[]->()"` and `"!()-[]->()"` are prohibited terms.
+* Motifs are not allowed to contain named edges within negated terms (since these named edges would never appear within results). E.g., `"!(a)-[ab]->(b)"` is invalid, but `"!(a)-[]->(b)"` is valid.
+
+More complex queries, such as queries which operate on vertex or edge attributes,
+can be expressed by applying filters to the result `DataFrame`.
+
+This can return duplicate rows. E.g., a query `"(u)-[]->()"` will return a result for each
+matching edge, even if those edges share the same vertex `u`.
+
+## Python API
+
+For API details, refer to the @:pydoc(graphframes.GraphFrame.find).
+
+```python
+from graphframes.examples import Graphs
+
+g = Graphs(spark).friends() # Get example graph
+
+# Search for pairs of vertices with edges in both directions between them
+motifs = g.find("(a)-[e]->(b); (b)-[e2]->(a)")
+motifs.show()
+
+# More complex queries can be expressed by applying filters
+motifs.filter("b.age > 30").show()
+```
+
+## Scala API
+
+For API details, refer to the @:scaladoc(org.graphframes.GraphFrame).
+
+```scala
+import org.apache.spark.sql.DataFrame
+import org.graphframes.{examples,GraphFrame}
+
+val g: GraphFrame = examples.Graphs.friends // get example graph
+
+// Search for pairs of vertices with edges in both directions between them.
+val motifs: DataFrame = g.find("(a)-[e]->(b); (b)-[e2]->(a)")
+motifs.show()
+
+// More complex queries can be expressed by applying filters.
+motifs.filter("b.age > 30").show()
+```
+
+## Examples
+
+Many motif queries are stateless and simple to express, as in the examples above. The next examples demonstrate more complex queries which carry state along a path in the motif. These queries can be expressed by combining GraphFrame motif finding with filters on the result, where the filters use sequence operations to construct a series of `DataFrame` `Column`s.
+
+For example, suppose one wishes to identify a chain of 4 vertices with some property defined by a sequence of functions. That is, among chains of 4 vertices `a->b->c->d`, identify the subset of chains matching this complex filter:
+
+* Initialize state on path.
+* Update state based on vertex `a`.
+* Update state based on vertex `b`.
+* Etc. for `c` and `d`.
+* If final state matches some condition, then the chain is accepted by the filter.
+
+The below code snippets demonstrate this process, where we identify chains of 4 vertices
+such that at least 2 of the 3 edges are "friend" relationships.
+In this example, the state is the current count of "friend" edges; in general, it could be any
+`DataFrame Column`.
+
+### Python API
+
+```python
+from pyspark.sql.functions import col, lit, when
+from pyspark.sql.types import IntegerType
+from graphframes.examples import Graphs
+
+
+g = Graphs(spark).friends() # Get example graph
+
+chain4 = g.find("(a)-[ab]->(b); (b)-[bc]->(c); (c)-[cd]->(d)")
+
+# Query on sequence, with state (cnt)
+# (a) Define method for updating state given the next element of the motif
+sumFriends =\
+ lambda cnt,relationship: when(relationship == "friend", cnt+1).otherwise(cnt)
+
+# (b) Use sequence operation to apply method to sequence of elements in motif
+# In this case, the elements are the 3 edges
+condition =\
+ reduce(lambda cnt,e: sumFriends(cnt, col(e).relationship), ["ab", "bc", "cd"], lit(0))
+
+# (c) Apply filter to DataFrame
+chainWith2Friends2 = chain4.where(condition >= 2)
+chainWith2Friends2.show()
+```
+
+### Scala API
+
+```scala
+import org.apache.spark.sql.{Column, DataFrame}
+import org.apache.spark.sql.functions.{col, when}
+import org.graphframes.{examples,GraphFrame}
+
+val g: GraphFrame = examples.Graphs.friends // get example graph
+
+// Find chains of 4 vertices.
+val chain4: DataFrame = g.find("(a)-[ab]->(b); (b)-[bc]->(c); (c)-[cd]->(d)")
+chain4.show()
+
+// Query on sequence, with state (cnt)
+// (a) Define method for updating state given the next element of the motif.
+def sumFriends(cnt: Column, relationship: Column): Column = {
+ when(relationship === "friend", cnt + 1).otherwise(cnt)
+}
+// (b) Use sequence operation to apply method to sequence of elements in motif.
+// In this case, the elements are the 3 edges.
+val condition = { Seq("ab", "bc", "cd")
+ .foldLeft(lit(0))((cnt, e) => sumFriends(cnt, col(e)("relationship"))) }
+// (c) Apply filter to DataFrame.
+val chainWith2Friends2 = chain4.where(condition >= 2)
+chainWith2Friends2.show()
+```
+
+### Conclusion
+
+The above example demonstrated a stateful motif for a fixed-length chain. Currently, in order to search for variable-length motifs, users need to run one query for each possible length. However, the above query patterns allow users to re-use the same code for each length, with the only change being to update the sequence of motif elements ("ab", "bc", "cd" above).
diff --git a/docs/src/04-user-guide/05-traversals.md b/docs/src/04-user-guide/05-traversals.md
new file mode 100644
index 000000000..52b6313af
--- /dev/null
+++ b/docs/src/04-user-guide/05-traversals.md
@@ -0,0 +1,179 @@
+# Traversals and Connectivity
+
+## Shortest paths
+
+Computes shortest paths from each vertex to the given set of landmark vertices, where landmarks are specified by the vertex ID. Note that this takes an edge direction into account.
+
+See [Wikipedia](https://en.wikipedia.org/wiki/Shortest_path_problem) for a background.
+
+### Python API
+
+For API details, refer to the @:pydoc(graphframes.GraphFrame.shortestPaths).
+
+```python
+from graphframes.examples import Graphs
+
+g = Graphs(spark).friends() # Get example graph
+
+results = g.shortestPaths(landmarks=["a", "d"])
+results.select("id", "distances").show()
+```
+
+### Scala API
+
+For API details, refer to the @:scaladoc(org.graphframes.lib.ShortestPaths).
+
+```scala
+import org.graphframes.{examples, GraphFrame}
+
+val g: GraphFrame = examples.Graphs.friends // get example graph
+
+val results = g.shortestPaths.landmarks(Seq("a", "d")).run()
+results.select("id", "distances").show()
+```
+
+## Breadth-first search (BFS)
+
+Breadth-first search (BFS) finds the shortest path(s) from one vertex (or a set of vertices) to another vertex (or a set of vertices). The beginning and end vertices are specified as Spark DataFrame expressions.
+
+See [Wikipedia on BFS](https://en.wikipedia.org/wiki/Breadth-first_search) for more background.
+
+The following code snippets use BFS to find the path between vertex with name "Esther" to a vertex with `age < 32`.
+
+### Python API
+
+For API details, refer to the @:pydoc(graphframes.GraphFrame.bfs).
+
+```python
+g = Graphs(spark).friends() # Get example graph
+
+# Search from "Esther" for users of age < 32
+
+paths = g.bfs("name = 'Esther'", "age < 32")
+paths.show()
+
+# Specify edge filters or max path lengths
+
+g.bfs("name = 'Esther'", "age < 32",\
+edgeFilter="relationship != 'friend'", maxPathLength=3)
+```
+
+### Scala API
+
+For API details, refer to @:scaladoc(org.graphframes.lib.BFS).
+
+```scala
+import org.graphframes.{examples,GraphFrame}
+
+val g: GraphFrame = examples.Graphs.friends // get example graph
+
+// Search from "Esther" for users of age < 32.
+val paths = g.bfs.fromExpr("name = 'Esther'").toExpr("age < 32").run()
+paths.show()
+
+// Specify edge filters or max path lengths.
+val paths = { g.bfs.fromExpr("name = 'Esther'").toExpr("age < 32")
+.edgeFilter("relationship != 'friend'")
+.maxPathLength(3).run() }
+paths.show()
+```
+
+## Connected components
+
+Computes the connected component membership of each vertex and returns a graph with each vertex assigned a component ID.
+
+See [Wikipedia](https://en.wikipedia.org/wiki/Connected_component_(graph_theory)) for the background.
+
+**NOTE:** With GraphFrames 0.3.0 and later releases, the default Connected Components algorithm requires setting a Spark checkpoint directory. Users can revert to the old algorithm using `connectedComponents.setAlgorithm("graphx")`. Starting from GraphFrames 0.9.3 release, users can also use `localCheckpoints` that does not require setting a Spark checkpoint directory. To use `localCheckpoints` users can set the config `spark.graphframes.useLocalCheckpoints` to `true` or use the API `connectedComponents.setUseLocalCheckpoints(true)`. While `localCheckpoints` provides better performance they are not as reliable as the persistent checkpointing.
+
+### Python API
+
+For API details, refer to the @:pydoc(graphframes.GraphFrame.connectedComponents).
+
+```python
+from graphframes.examples import Graphs
+
+sc.setCheckpointDir("/tmp/spark-checkpoints")
+
+g = Graphs(spark).friends() # Get example graph
+
+result = g.connectedComponents()
+result.select("id", "component").orderBy("component").show()
+```
+
+### Scala API
+
+For API details, refer to the @:scaladoc(org.graphframes.lib.ConnectedComponents).
+
+```scala
+import org.graphframes.{examples,GraphFrame}
+
+val g: GraphFrame = examples.Graphs.friends // get example graph
+
+val result = g.connectedComponents.setUseLocalCheckpoints(true).run()
+result.select("id", "component").orderBy("component").show()
+```
+
+### Strongly connected components
+
+Compute the strongly connected component (SCC) of each vertex and return a graph with each vertex assigned to the SCC containing that vertex. At the moment, SCC in GraphFrames is a wrapper around GraphX implementation.
+
+See [Wikipedia](https://en.wikipedia.org/wiki/Strongly_connected_component) for the background.
+
+#### Python API
+
+For API details, refer to the @:pydoc(graphframes.GraphFrame.stronglyConnectedComponents).
+
+```python
+from graphframes.examples import Graphs
+
+sc.setCheckpointDir("/tmp/spark-checkpoints")
+
+g = Graphs(spark).friends() # Get example graph
+
+result = g.stronglyConnectedComponents(maxIter=10)
+result.select("id", "component").orderBy("component").show()
+```
+
+#### Scala API
+
+For API details, refer to the @:scaladoc(org.graphframes.lib.StronglyConnectedComponents).
+
+```scala
+import org.graphframes.{examples,GraphFrame}
+
+val g: GraphFrame = examples.Graphs.friends // get example graph
+
+val result = g.stronglyConnectedComponents.maxIter(10).run()
+result.select("id", "component").orderBy("component").show()
+```
+
+## Triangle count
+
+Computes the number of triangles passing through each vertex.
+
+### Python API
+
+For API details, refer to the @:pydoc(graphframes.GraphFrame.triangleCount).
+
+```python
+from graphframes.examples import Graphs
+
+g = Graphs(spark).friends() # Get example graph
+
+results = g.triangleCount()
+results.select("id", "count").show()
+```
+
+### Scala API
+
+For API details, refer to the @:scaladoc(org.graphframes.lib.TriangleCount).
+
+```scala
+import org.graphframes.{examples, GraphFrame}
+
+val g: GraphFrame = examples.Graphs.friends // get example graph
+
+val results = g.triangleCount.run()
+results.select("id", "count").show()
+```
diff --git a/docs/src/04-user-guide/06-graph-clustering.md b/docs/src/04-user-guide/06-graph-clustering.md
new file mode 100644
index 000000000..76588ff02
--- /dev/null
+++ b/docs/src/04-user-guide/06-graph-clustering.md
@@ -0,0 +1,52 @@
+# Community Detection
+
+## Label Propagation (LPA)
+
+Run a static Label Propagation Algorithm for detecting communities in networks. Each node in the network is initially assigned to its own community. At every superstep, nodes send their community affiliation to all neighbors and update their state to the mode community affiliation of incoming messages. LPA is a standard community detection algorithm for graphs. It is very inexpensive computationally, although (1) convergence is not guaranteed and (2) one can end up with trivial solutions (all nodes are identified into a single community).
+
+See [Wikipedia](https://en.wikipedia.org/wiki/Label_Propagation_Algorithm) for the background.
+
+### Python API
+
+For API details, refer to the @:pydoc(graphframes.GraphFrame.labelPropagation).
+
+```python
+from graphframes.examples import Graphs
+
+g = Graphs(spark).friends() # Get example graph
+
+result = g.labelPropagation(maxIter=5)
+result.select("id", "label").show()
+```
+
+### Scala API
+
+For API details, refer to the @:scaladoc(org.grapimport org.graphframes.lib.LabelPropagation).
+
+```scala
+import org.graphframes.{examples,GraphFrame}
+
+val g: GraphFrame = examples.Graphs.friends // get example graph
+
+val result = g.labelPropagation.maxIter(5).run()
+result.select("id", "label").show()
+```
+
+## Power Iteration Clustering (PIC)
+
+GraphFrames provides a wrapper for the [Power Iteration Clustering](https://www.cs.cmu.edu/~frank/papers/icml2010-pic-final.pdf) algorithm from the SparkML library.
+
+### Python API
+
+```python
+g = GraphFrame(vertices, edges)
+g.powerIterationClustering(k=2, maxIter=40, weightCol="weight")
+```
+
+### Scala API
+
+```scala
+val gf = GraphFrame(vertices, edges)
+val clusters = gf
+ .powerIterationClustering(k = 2, maxIter = 40, weightCol = Some("weight"))
+```
diff --git a/docs/src/04-user-guide/07-subgraphs.md b/docs/src/04-user-guide/07-subgraphs.md
new file mode 100644
index 000000000..94399041b
--- /dev/null
+++ b/docs/src/04-user-guide/07-subgraphs.md
@@ -0,0 +1,81 @@
+# Subgraphs
+
+In GraphX, the `subgraph()` method takes an edge triplet (edge, src vertex, and dst vertex, plus attributes) and allows the user to select a subgraph based on triplet and vertex filters.
+
+GraphFrames provide an even more powerful way to select subgraphs based on a combination of motif finding and DataFrame filters. We provide three helper methods for subgraph selection. `filterVertices(condition)`, `filterEdges(condition)`, and `dropIsolatedVertices()`.
+
+## Simple subgraph
+The following example shows how to select a subgraph based upon vertex and edge filters.
+
+### Python API
+
+```python
+from graphframes.examples import Graphs
+
+g = Graphs(spark).friends() # Get example graph
+
+# Select subgraph of users older than 30, and relationships of type "friend"
+# Drop isolated vertices (users) which are not contained in any edges (relationships)
+g1 = g.filterVertices("age > 30").filterEdges("relationship = 'friend'").dropIsolatedVertices()
+```
+
+### Scala API
+
+```scala
+import org.graphframes.{examples,GraphFrame}
+
+val g: GraphFrame = examples.Graphs.friends
+
+// Select subgraph of users older than 30, and relationships of type "friend".
+// Drop isolated vertices (users) which are not contained in any edges (relationships).
+val g1 = g.filterVertices("age > 30").filterEdges("relationship = 'friend'").dropIsolatedVertices()
+```
+
+## Complex subgraph: triplet filters
+
+The following example shows how to select a subgraph based upon triplet filters which operate on an edge and its src and dst vertices. This example could be extended to go beyond triplets by using more complex motifs.
+
+### Python API
+
+```python
+from graphframes.examples import Graphs
+
+g = Graphs(spark).friends() # Get example graph
+
+# Select subgraph based on edges "e" of type "follow"
+# pointing from a younger user "a" to an older user "b"
+paths = g.find("(a)-[e]->(b)")\
+ .filter("e.relationship = 'follow'")\
+ .filter("a.age < b.age")
+
+# "paths" contains vertex info. Extract the edges
+e2 = paths.select("e.src", "e.dst", "e.relationship")
+
+# In Spark 1.5+, the user may simplify this call
+# val e2 = paths.select("e.*")
+# Construct the subgraph
+g2 = GraphFrame(g.vertices, e2)
+```
+
+### Scala API
+
+```scala
+import org.graphframes.{examples,GraphFrame}
+
+val g: GraphFrame = examples.Graphs.friends // get example graph
+
+// Select subgraph based on edges "e" of type "follow"
+// pointing from a younger user "a" to an older user "b".
+val paths = { g.find("(a)-[e]->(b)")
+ .filter("e.relationship = 'follow'")
+ .filter("a.age < b.age") }
+// "paths" contains vertex info. Extract the edges.
+val e2 = paths.select("e.*")
+
+// Construct the subgraph
+val g2 = GraphFrame(g.vertices, e2)
+```
+
+## Property Graphs
+
+For more advanced subgraph selection, see the [Property Graphs](/04-user-guide/11-property-graphs.md) section.
diff --git a/docs/src/04-user-guide/08-saving-and-loading.md b/docs/src/04-user-guide/08-saving-and-loading.md
new file mode 100644
index 000000000..8ced2c867
--- /dev/null
+++ b/docs/src/04-user-guide/08-saving-and-loading.md
@@ -0,0 +1,46 @@
+# Saving and loading
+
+At the moment, GraphFrames does not support out-of-the-box saving and loading, and one needs to use Spark SQL to save and load `vertices` and `edges` as DataFrames. Refer to the [Spark SQL User Guide on datasources](http://spark.apache.org/docs/latest/sql-programming-guide.html#data-sources) for more details.
+
+**NOTE:** Maintainers of `GraphFrames` are working at the moment on adding support for saving and loading GraphFrames into various formats, like GraphML, Apache GraphAr (incubating), and others.
+
+The below example shows how to save and then load a graph as parquet files for vertices and edges.
+
+## Python API
+
+```python
+from graphframes.examples import Graphs
+
+
+g = Graphs(spark).friends() # Get example graph
+
+# Save vertices and edges as Parquet to some location
+g.vertices.write.parquet("hdfs://myLocation/vertices")
+g.edges.write.parquet("hdfs://myLocation/edges")
+
+# Load the vertices and edges back
+sameV = spark.read.parquet("hdfs://myLocation/vertices")
+sameE = spark.read.parquet("hdfs://myLocation/edges")
+
+# Create an identical GraphFrame
+sameG = GraphFrame(sameV, sameE)
+```
+
+## Scala API
+
+```scala
+import org.graphframes.{examples,GraphFrame}
+
+val g: GraphFrame = examples.Graphs.friends // get example graph
+
+// Save vertices and edges as Parquet to some location.
+g.vertices.write.parquet("hdfs://myLocation/vertices")
+g.edges.write.parquet("hdfs://myLocation/edges")
+
+// Load the vertices and edges back.
+val sameV = spark.read.parquet("hdfs://myLocation/vertices")
+val sameE = spark.read.parquet("hdfs://myLocation/edges")
+
+// Create an identical GraphFrame.
+val sameG = GraphFrame(sameV, sameE)
+```
\ No newline at end of file
diff --git a/docs/src/04-user-guide/09-aggregate-messages.md b/docs/src/04-user-guide/09-aggregate-messages.md
new file mode 100644
index 000000000..92921c1e0
--- /dev/null
+++ b/docs/src/04-user-guide/09-aggregate-messages.md
@@ -0,0 +1,55 @@
+# Message passing
+
+Like GraphX, GraphFrames provides primitives for developing graph algorithms. The two key components are:
+
+* `aggregateMessages`: Send messages between vertices, and aggregate messages for each vertex. GraphFrames provides a native `aggregateMessages` method implemented using DataFrame operations. This may be used analogously to the GraphX API.
+* joins: Join message aggregates with the original graph. GraphFrames rely on `DataFrame` joins, which provide the full functionality of GraphX joins.
+
+The below code snippets show how to use `aggregateMessages` to compute the sum of the ages of adjacent users.
+
+## Python API
+
+For API details, refer to the @:pydoc(graphframes.GraphFrame.aggregateMessages).
+
+```python
+from graphframes.lib import AggregateMessages as AM
+from graphframes.examples import Graphs
+from pyspark.sql.functions import sum as sqlsum
+
+
+g = Graphs(spark).friends() # Get example graph
+
+# For each user, sum the ages of the adjacent users
+msgToSrc = AM.dst["age"]
+msgToDst = AM.src["age"]
+agg = g.aggregateMessages(
+ sqlsum(AM.msg).alias("summedAges"),
+ sendToSrc=msgToSrc,
+ sendToDst=msgToDst)
+agg.show()
+```
+
+## Scala API
+
+For API details, refer to the @:scaladoc(org.graphframes.lib.AggregateMessages).
+
+```scala
+import org.graphframes.{examples,GraphFrame}
+import org.graphframes.lib.AggregateMessages
+
+val g: GraphFrame = examples.Graphs.friends // get example graph
+
+// We will use AggregateMessages utilities later, so name it "AM" for short.
+val AM = AggregateMessages
+
+// For each user, sum the ages of the adjacent users.
+val msgToSrc = AM.dst("age")
+val msgToDst = AM.src("age")
+val agg = { g.aggregateMessages
+ .sendToSrc(msgToSrc) // send destination user's age to source
+ .sendToDst(msgToDst) // send source user's age to destination
+ .agg(sum(AM.msg).as("summedAges")) } // sum up ages, stored in AM.msg column
+agg.show()
+```
+
+For a more complex example, look at the code used to implement the @:pydoc(graphframes.examples.BeliefPropagation).
\ No newline at end of file
diff --git a/docs/src/04-user-guide/10-pregel.md b/docs/src/04-user-guide/10-pregel.md
new file mode 100644
index 000000000..301317838
--- /dev/null
+++ b/docs/src/04-user-guide/10-pregel.md
@@ -0,0 +1,69 @@
+# Pregel API
+
+Pregel API is one of the core backbones of GraphFrames. It is based on the implementation of the [Pregel algorithm](https://blog.lavaplanets.com/wp-content/uploads/2023/12/p135-malewicz.pdf) in terms of relational operations using the Apache Spark `DataFrame` / `Dataset` API.
+
+## API
+
+For the API details, please refer to:
+
+- Scala API: @:scaladoc(org.graphframes.lib.Pregel)
+- Python API: @:pydoc(graphframes.lib.Pregel)
+
+## Arguments
+
+### Message Columns
+
+Internally, GraphFrames handles all the messages via a specific column:
+
+```scala
+val pregelMessageColumn = org.graphframes.lib.Pregel.msg
+val pregelMessageColumnName = org.graphframes.lib.Pregel.MSG_COL_NAME
+```
+
+This name is applied to both raw and aggregated messages and should be used in conditions and aggregations. For example, in the simplest implementation of the PageRank algorithm it may look like this:
+
+```scala
+val aggregateMessagesExpression = sum(Pregel.msg)
+val updateExpression = coalesce(Pregel.msg, lit(0.0)) * (1 - alpha) + alpha / numVertices
+```
+
+### Triplet Column
+
+At the stage of message generation, GraphFrames internally creates triplets, that contains `StructType` columns for source vertex, edge and destination vertex. These structs contain all the columns from the original source, destination and edge, including not only initial attributes, but also Pregel columns, like activity column, all the vertex columns, etc. To access these data, user should rely on the following API:
+
+```scala
+val sourceVertexColumn1 = Pregel.src("vertexColumn1")
+val sourceVertexColumn2 = Pregel.src("vertexColumn2")
+val destinationVertexColumn1 = Pregel.dst("vertexColumn1")
+val edgeWeight = Pregel.edge("weight")
+```
+
+Under the hood, the passed name of the column will be resolved to get the corresponding element of the triplet structs.
+
+### Sending Messages
+
+GraphFrames Pregel API support arbitrary number of messages per vertex. Inside the Pregel API **graphs are always considered directed**. This means that if a vertex has an outgoing edge to another vertex, then the message will be sent to the destination vertex. To emulate the behavior of the undirected graph, the user can send the same message to both the source and the destination vertex.
+
+```scala
+graph
+ .pregel
+ .sendMsgToDst(Pregel.src("vertexColumn")) // Sending the vertex column of the destination vertex to the source vertex.
+ .sendMsgToSrc(Pregel.dst("vertexColumn")) // Sending the vertex column of the source vertex to the destination vertex.
+ .run()
+```
+
+### Aggregation
+
+GraphFrames Pregel API requires the user to specify the aggregation function for the messages. It is called over all the messages sent to the vertex no matter the direction.
+
+```scala
+graph.pregel.aggMsgs(sum(Pregel.msg))
+```
+
+### Termination Conditions
+
+GraphFrames Pregel API provides the following termination conditions:
+
+- **By a number of iterations.** Users can specify the maximum number of iterations with `setMaxIter(value: Int)`.
+- **In case of no new messages are sent.** User can say GF to terminate the computations if all the messages sent on the iteration are empty (`null`). To do this, user should specify `setEarlyStopping(value: Boolean)`. **Be careful, because the checking of nullity is a not free operation, but Apache Spark action!** So, for example, if messages cannot be empty, this condition should be set to `false`. For example, in algorithms like `ShortestPaths`, this condition should be set to `true`, but for algorithms like `PageRank`, this condition should be set to `false` because the messages cannot be empty.
+- **By vertex voting.** Users can specify the participation condition per vertex with `setInitialActiveVertexExpression(expression: Column` and `setUpdateActiveVertexExpression(expression: Column)`. In the case if `stopIfAllNonActiveVertices(value: Boolean)` is set to `true`, the computation will stop if all the vertices are inactive. This is useful for algorithms like `LabelPropagation`, when messages are always not `null`, but if no vertex changed a label on the last iteration, the computation should stop. **Be careful, because the checking of vertex status is a not free operation, but Apache Spark action!**
\ No newline at end of file
diff --git a/docs/src/04-user-guide/11-property-graphs.md b/docs/src/04-user-guide/11-property-graphs.md
new file mode 100644
index 000000000..76c9f1298
--- /dev/null
+++ b/docs/src/04-user-guide/11-property-graphs.md
@@ -0,0 +1,129 @@
+# Property Graphs
+
+## Overview
+
+From the [Wikipedia](https://en.wikipedia.org/wiki/Property_graph), a property graph, labeled property graph, or attributed graph is a data model, where pairs of entities are associated by relationships, and entities and relationships can have properties.
+
+@:image(/img/property-graph/property-graph-example.png) {
+ intrinsicWidth = 600
+ alt = "An example of the property graph with different kinds of edges and vertices"
+ title = "Property Graph Example"
+}
+
+To make this concrete, let's imagine an example of the “movie fan social network.” The diagram above shows how this looks as a property graph. There are people and movies – two types of vertices, each with their own properties. People can like movies (undirected edges), send messages to each other (directed edges), and follow directors. There are also actors and directors as separate vertex types. Movies can be connected as sequels. All these relationships and entities are easy to represent in a property graph, as shown in the diagram.
+
+The property graph model is very universal. Take a banking payments network: there are legal entities, government services, individuals, exchanges, and goods. Each is a different vertex type. Payments are directed, weighted edges with properties like date and amount. Two legal entities sharing a board member form an undirected, unweighted edge with the director’s details as properties. This structure is great for compliance, KYC, and anti-fraud. It helps to see who is connected to whom and how closely.
+
+Or consider an online marketplace. There are buyers, sellers, and products. Buyers purchase products from sellers. Sellers offer many products. All of this fits naturally into a property graph. This structure works well for recommendation systems. Recommending a product is basically a link prediction problem in the graph.
+
+Another example is Organizational Network Analysis (ONA). Companies have departments, teams, and people, all connected in different ways. Teams assign tasks to each other. People have both formal and informal relationships. There are official and real hierarchies. ONA can reveal key employees, process bottlenecks, and even predict conflicts. It also helps improve knowledge sharing across the organization.
+
+## Property Graphs in GraphFrames
+
+GraphFrames represent a property graph as a combination of multiple logical structures named **Vertex Property Group** linked by multiple logical structures named **Edge Property Group**.
+
+### Vertex Property Group
+
+For API details see @:scaladoc(org.graphframes.propertygraph.property.VertexPropertyGroup). It contains a name of the property group, for example, `movies`, a name of ID column and underlying data in the form of a `DataFrame`.
+
+The simple example below creates two property groups: `people` and `movies`.
+
+```scala
+import org.graphframes.propertygraph.property.VertexPropertyGroup
+
+val peopleData = spark
+ .createDataFrame(
+ Seq((1L, "Alice"), (2L, "Bob"), (3L, "Charlie"), (4L, "David"), (5L, "Eve")))
+ .toDF("id", "name")
+
+val peopleGroup = VertexPropertyGroup("people", peopleData, "id")
+
+val moviesData = spark
+ .createDataFrame(Seq((1L, "Matrix"), (2L, "Inception"), (3L, "Interstellar")))
+ .toDF("id", "title")
+
+val moviesGroup = VertexPropertyGroup("movies", moviesData, "id")
+```
+
+### Edge Property Group
+
+For API details see @:scaladoc(org.graphframes.propertygraph.property.EdgePropertyGroup). It contains a name of the property group, links to the source and target vertex property groups, direction of the edges (`directed` or `undirected`), and underlying data in the form of a `DataFrame`. Optionally, it can contain a column with edge weights as well as names of source and target vertex ID columns.
+
+The simple example below creates an edge property group with the name `likes` and links to the `people` and `movies` vertex property groups as well as `messages` property group that links people to people.
+
+```scala
+import org.graphframes.propertygraph.property.EdgePropertyGroup
+
+val likesData = spark
+ .createDataFrame(Seq((1L, 1L), (1L, 2L), (2L, 1L), (3L, 2L), (4L, 3L), (5L, 2L)))
+ .toDF("src", "dst")
+
+val likesGroup = EdgePropertyGroup(
+ "likes",
+ likesData,
+ peopleGroup,
+ moviesGroup,
+ isDirected = false,
+ "src",
+ "dst",
+ lit(1.0))
+
+val messagesData = spark
+ .createDataFrame(
+ Seq((1L, 2L, 5.0), (2L, 3L, 8.0), (3L, 4L, 3.0), (4L, 5L, 6.0), (5L, 1L, 9.0)))
+ .toDF("src", "dst", "weight")
+
+val messagesGroup = EdgePropertyGroup(
+ "messages",
+ messagesData,
+ peopleGroup,
+ peopleGroup,
+ isDirected = true,
+ "src",
+ "dst",
+ col("weight"))
+```
+
+### Property GraphFrame
+
+Having defined the property groups, we can create a `PropertyGraphFrame` by passing the property groups to the constructor.
+
+```scala
+import org.graphframes.propertygraph.PropertyGraphFrame
+
+peopleMoviesGraph =
+ PropertyGraphFrame(Seq(peopleGroup, moviesGroup), Seq(likesGroup, messagesGroup))
+```
+
+### Conversion to GraphFrames
+
+The `PropertyGraphFrame` can be converted to a `GraphFrame` by calling `toGraphFrame`. Users can select a subset of vertex and edge property groups to be included in the resulting `GraphFrame`. Under the hood, the conversion will take care about handling potential vertex and edge ID collisions by applying hashing to both vertex and edge IDs.
+
+```scala
+val graph = peopleMoviesGraph.toGraphFrame(
+ Seq("people"),
+ Seq("messages"),
+ Map("messages" -> lit(true)),
+ Map("people" -> lit(true)))
+```
+
+For more details see @:scaladoc(org.graphframes.propertygraph.PropertyGraphFrame).
+
+This operation is not free, so user can also explicitly specify for each of `VertexGroup` does it need to be hashed or not.
+
+```scala
+val moviesData = spark
+ .createDataFrame(Seq((1L, "Matrix"), (2L, "Inception"), (3L, "Interstellar")))
+ .toDF("id", "title")
+
+val moviesGroup = VertexPropertyGroup("movies", moviesData, "id", applyMaskOnId = false)
+```
+
+### Projection
+
+The `PropertyGraphFrame` support projection of edges groups to a new edge group. For example, if we have a property graph, where people can like movies, we can do a projection of such a bi-graph to a graph of only peoples where two people are connected if they like the same movie. This operation creates a new `PropertyGraphFrame` with a new edge group and without the original edge group through which the projection was done.
+
+```scala
+val projectedGraph = peopleMoviesGraph.projectionBy("people", "movies", "likes")
+```
+
diff --git a/docs/src/04-user-guide/12-graphx-conversion.md b/docs/src/04-user-guide/12-graphx-conversion.md
new file mode 100644
index 000000000..39f75bf1d
--- /dev/null
+++ b/docs/src/04-user-guide/12-graphx-conversion.md
@@ -0,0 +1,52 @@
+# GraphX conversions
+
+We provide utilities for converting between GraphFrame and GraphX graphs. See the [GraphX User Guide](http://spark.apache.org/docs/latest/graphx-programming-guide.html) for details on GraphX.
+
+## GraphFrame to GraphX
+
+Conversion to GraphX creates a GraphX `Graph` which has `Long` vertex IDs and attributes of type `Row`.
+
+Vertex and edge attributes are the original rows in `vertices` and `edges`, respectively.
+
+Note that vertex (and edge) attributes include vertex IDs (and source, destination IDs) to support non-Long vertex IDs. If the vertex IDs are not convertible to Long values, then the values are indexed to generate corresponding Long vertex IDs (which is an
+expensive operation).
+
+The column ordering of the returned `Graph` vertex and edge attributes are specified by `GraphFrame.vertexColumns` and `GraphFrame.edgeColumns`, respectively.
+
+## GraphX to GraphFrame
+
+GraphFrame provides two conversion methods. The first takes any GraphX graph and converts the vertex and edge `RDD`s into `DataFrame`s using schema inference. Those DataFrames are then used to create a GraphFrame.
+
+The second conversion method is more complex and is useful for users with existing GraphX code. Its main purpose is to support workflows of the following form: (1) convert a GraphFrame to GraphX, (2) run GraphX code to augment the GraphX graph with new vertex or edge attributes, and (3) merge the new attributes back into the original GraphFrame.
+
+For example, given:
+
+* GraphFrame `originalGraph`
+* GraphX `Graph[String, Int]` `graph` with a String vertex attribute we want to call "category" and an Int edge attribute we want to call "count"
+
+We can call `fromGraphX(originalGraph, graph, Seq("category"), Seq("count"))` to produce a new GraphFrame. The new GraphFrame will be an augmented version of `originalGraph`, with new `GraphFrame.vertices` column "category" and new `GraphFrame.edges` column "count" added.
+
+For example usage, look at the code used to implement the @:scaladoc(org.graphframes.examples.BeliefPropagation).
+
+## Example conversions
+
+The below example demonstrates simple GraphFrame-GraphX conversions.
+
+For API details, refer to the API docs for @:scaladoc(org.graphframes.GraphFrame).
+
+```scala
+import org.apache.spark.graphx.Graph
+import org.apache.spark.sql.Row
+import org.graphframes.{examples,GraphFrame}
+
+val g: GraphFrame = examples.Graphs.friends // get example graph
+
+// Convert to GraphX
+val gx: Graph[Row, Row] = g.toGraphX
+
+// Convert back to GraphFrame.
+// Note that the schema is changed because of constraints in the GraphX API.
+val g2: GraphFrame = GraphFrame.fromGraphX(gx)
+```
+
+These conversions are only supported in Scala since GraphX does not have a Python API.
\ No newline at end of file
diff --git a/docs/configurations.md b/docs/src/04-user-guide/13-configurations.md
similarity index 87%
rename from docs/configurations.md
rename to docs/src/04-user-guide/13-configurations.md
index 2ae4ea67f..4b9a267e9 100644
--- a/docs/configurations.md
+++ b/docs/src/04-user-guide/13-configurations.md
@@ -1,14 +1,4 @@
----
-layout: global
-displayTitle: GraphFrames Configurations
-title: Configurations
-description: GraphFrames GRAPHFRAMES_VERSION configurations documentation
----
-
-* Table of contents
-{:toc}
-
-# GraphFrames Configurations
+# Configurations
GraphFrames provides several configuration options that can be used to tune the behavior of algorithms and operations. This page documents all available configurations, their descriptions, default values, and usage examples.
@@ -33,10 +23,9 @@ GraphFrames configurations can be set in several ways:
You can set configurations when creating a SparkSession:
-
+#### Scala API
-
-{% highlight scala %}
+```scala
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder()
@@ -44,11 +33,11 @@ val spark = SparkSession.builder()
.config("spark.graphframes.connectedComponents.algorithm", "graphframes")
.config("spark.graphframes.connectedComponents.checkpointinterval", 3)
.getOrCreate()
-{% endhighlight %}
-
+```
+
+#### Python API
-
-{% highlight python %}
+```python
from pyspark.sql import SparkSession
spark = SparkSession.builder \
@@ -56,41 +45,32 @@ spark = SparkSession.builder \
.config("spark.graphframes.connectedComponents.algorithm", "graphframes") \
.config("spark.graphframes.connectedComponents.checkpointinterval", 3) \
.getOrCreate()
-{% endhighlight %}
-
-
-
+```
### Runtime Configuration
You can also set configurations at runtime:
-
+#### Scala API
-
-{% highlight scala %}
+```scala
spark.conf.set("spark.graphframes.connectedComponents.algorithm", "graphframes")
spark.conf.set("spark.graphframes.connectedComponents.checkpointinterval", 3)
-{% endhighlight %}
-
+```
-
-{% highlight python %}
+#### Python API
+```python
spark.conf.set("spark.graphframes.connectedComponents.algorithm", "graphframes")
spark.conf.set("spark.graphframes.connectedComponents.checkpointinterval", 3)
-{% endhighlight %}
-
-
-
+```
## Example: Connected Components with Custom Configurations
This example shows how to run the Connected Components algorithm with custom configurations:
-
+### Scala API
-
-{% highlight scala %}
+```scala
import org.graphframes.GraphFrame
import org.graphframes.examples
@@ -105,11 +85,11 @@ spark.conf.set("spark.graphframes.useLocalCheckpoints", true)
// Run connected components with custom configurations
val result = g.connectedComponents.run()
result.show()
-{% endhighlight %}
-
+```
-
-{% highlight python %}
+### Python API
+
+```python
from graphframes.examples import Graphs
# Get example graph
@@ -123,15 +103,11 @@ spark.conf.set("spark.graphframes.useLocalCheckpoints", "true")
# Run connected components with custom configurations
result = g.connectedComponents()
result.show()
-{% endhighlight %}
-
-
-
+```
## Notes on Configuration Usage
- **Checkpoint Directory**: For configurations related to checkpointing, make sure to set a checkpoint directory using `spark.sparkContext.setCheckpointDir("path/to/checkpoint/dir")` before running algorithms that use checkpointing.
-
- **Storage Levels**: When setting the `spark.graphframes.connectedComponents.intermediatestoragelevel` configuration, use one of the following values: `MEMORY_ONLY`, `MEMORY_AND_DISK`, `MEMORY_ONLY_SER`, `MEMORY_AND_DISK_SER`, `DISK_ONLY`, `MEMORY_ONLY_2`, `MEMORY_AND_DISK_2`, etc.
-
-- **Algorithm Selection**: The choice of algorithm for connected components can significantly impact performance. The "graphframes" algorithm is generally more scalable for large graphs, while the "graphx" algorithm may be faster for smaller graphs.
\ No newline at end of file
+- **Algorithm Selection**: The choice of algorithm for connected components can significantly impact performance. The "graphframes" algorithm is generally more scalable for large graphs, while the "graphx" algorithm may be faster for smaller graphs.
+- **Local Checkpoints**: Local checkpoints are faster and less error prone, but can put strain on the local disk if insufficiently large. Because local checkpoints do not require to set `checkpointDir` it is a recommended option.
diff --git a/docs/src/04-user-guide/directory.conf b/docs/src/04-user-guide/directory.conf
new file mode 100644
index 000000000..8d3d987bc
--- /dev/null
+++ b/docs/src/04-user-guide/directory.conf
@@ -0,0 +1 @@
+laika.title = User Guide
\ No newline at end of file
diff --git a/docs/src/05-blog/01-index.md b/docs/src/05-blog/01-index.md
new file mode 100644
index 000000000..bbabbd1ab
--- /dev/null
+++ b/docs/src/05-blog/01-index.md
@@ -0,0 +1,9 @@
+# Blog
+
+It is an official blog of the GraphFrames project.
+
+[Subscribe to the feed](feed.xml)
+
+## All posts:
+
+- [GraphFrames is back!](1000-graphframes-is-back.md)
\ No newline at end of file
diff --git a/docs/src/05-blog/1000-graphframes-is-back.md b/docs/src/05-blog/1000-graphframes-is-back.md
new file mode 100644
index 000000000..33a2103d4
--- /dev/null
+++ b/docs/src/05-blog/1000-graphframes-is-back.md
@@ -0,0 +1,89 @@
+# GraphFrames is back!
+
+- **Published:** 2025-08-01T00:00:00Z
+- **Title:** GraphFrames is back!
+- **Summary:** GraphFrames 0.9.2 is out on PyPi as graphframes-py and as io.graphframes on Maven Sonatype Central! Documentation is now available on graphframes.io… and we even have a new logo!
+
+*Originally published in the [Graphlet Ai Blog](https://blog.graphlet.ai/graphframes-is-back-with-v0-9-2-5773d55d3291)*
+
+GraphFrames 0.9.2 is out on [PyPi](https://pypi.org/project/graphframes-py/) as [graphframes-py](https://pypi.org/project/graphframes-py/) and as [io.graphframes](https://central.sonatype.com/namespace/io.graphframes) on [Maven Sonatype Central](https://central.sonatype.com/search?q=io.graphframes)! Documentation is now available on [graphframes.io](https://graphframes.io/)… and we even have a new logo!
+
+@:image(/img/logo-dark.png) {
+ intrinsicWidth = 600
+ alt = "The new GraphFrames logo is new for this release :)"
+ title = The new GraphFrames logo
+}
+
+## GraphFrames is BACK!
+
+You can see below that GraphFrames is back! It has seen contributions every week for most of the year — we have half a dozen active contributors now. This release is due to the efforts of many people but I need to express our deep gratitude to [Sem Sinchenko](https://www.linkedin.com/in/semyon-a-sinchenko/), who drove this release.
+
+@:image(https://miro.medium.com/v2/resize:fit:1100/format:webp/1*xJnVeh6LbQ3gNoPVQ4gwmw.png) {
+ intrinsicWidth = 600
+ alt = "GraphFrames is back! Semyon Sinchenko deserves the appreciation and respect of all GraphFrames users :)The project has gone from dead to lively since GraphX was deprecated from Spark — prompting us to work on a replacement."
+ title = "GraphFrames commits chart"
+}
+
+The project has gone from *effectively dead* to *vibrant* in the six months since [GraphX was deprecated](https://lists.apache.org/thread/qrvo6xrt8zvp5ss73z5spt9q89r0htwo) from Spark, which prompted us to get to work on an all-DataFrame replacement. You can see in the chart below that there is more frequent contributions than since the project’s inception!
+
+@:image(https://miro.medium.com/v2/resize:fit:1100/format:webp/1*GYINCDjQN9LIng9ntA27iQ.png) {
+ intrinsicWidth = 600
+ alt = "Contributions chart of the GraphFrames project"
+ title = "After a six year gap in additions, GraphFrames is back with Spark Connect support!"
+}
+
+## New Features in GraphFrames 0.9.2
+
+It was necessary for GraphFrames to support both Spark 4 and Spark Connect to remain integral to the Spark community. There were many issues resolved in the release, but the core of it was:
+
+- [Spark Connect support](https://github.com/graphframes/graphframes/pull/506)
+- [Spark 4.x support](https://github.com/graphframes/graphframes/pull/608)
+- [Performance improvements in Connected Components](https://github.com/graphframes/graphframes/pull/552)
+- [Updated API for Pregel](https://github.com/graphframes/graphframes/issues?q=is%3Aissue+state%3Aclosed+Pregel)
+- DataFrame implementation of [LabelPropagation](https://graphframes.io/api/scaladoc/org/graphframes/lib/LabelPropagation.html), GraphX-free
+- DataFrame implementation of [ShortestPaths](https://graphframes.io/api/scaladoc/org/graphframes/lib/ShortestPaths.html), GraphX-free
+- [New groupId](https://central.sonatype.com/namespace/io.graphframes) `io.graphframes`
+- [New PyPi ID](https://pypi.org/project/graphframes-py/): `graphframes-py`
+- A new website: [graphframes.io](https://graphframes.io/) with [Updated documentation](https://graphframes.io/)
+- A new [Network Motif Finding Tutorial](/03-tutorials/02-motif-tutorial.md)
+- [A lot of additional changes and fixes](https://github.com/graphframes/graphframes/releases/tag/v0.9.0)
+
+## State of the Union
+
+The GraphFrames community has achieved our first goal: make the project viable again! Still in the future?
+
+## Property Graphs
+
+Sem has [started implementing](https://github.com/graphframes/graphframes/pull/613) Property Graphs for GraphFrames, which currently has `relationship` for edges but not `type` for nodes. In current practice, this means property graph processing requires you to merge all your node schemas together into a kitchen sink schema before using GraphFrames’ algorithms. it is a real drag… property graphs will be a huge improvement! Sem recently [outlined a beautiful vision](https://semyonsinchenko.github.io/ssinchenko/post/dreams-about-graph-in-lakehouse/) for property graphs as part of the Open Lakehouse. Check it out!
+
+## Inclusion in Spark
+
+This is actively debated: it would be a lot of trouble to release with Spark, but based on the number of search hits for [GraphX](https://www.google.com/search?q=GraphX) versus [GraphFrames](https://www.google.com/search?q=GraphFrames), it would get us 10x as many users. When I put that way, GraphFrames in Spark sounds pretty good!
+
+## GraphX is Deprecated
+
+Spark deprecating GraphX was the call to action that led us to revive GraphFrames, and we heard it well. We’re building DataFrame implementations of all GraphX components. GraphX has already been removed [from ShortestPaths](https://github.com/graphframes/graphframes/pull/587) and [from LabelPropagation](https://graphframes.io/api/python/graphframes.html#graphframes.GraphFrame.labelPropagation). The rest of the work is being tracked [here](https://github.com/graphframes/graphframes/issues/556) and is underway. **GraphX will be deprecated from GraphFrames** as of 1.0. GraphFrames 2.0 will remove GraphX completely. Soon GraphFrames will be entirely built on DataFrames!
+
+## The Sedona Alliance!
+
+Developers from [Apache Sedona](https://sedona.apache.org/latest/) joined the development of GraphFrames 0.9. Sedona 1.80 [will depend on](https://github.com/apache/sedona/pull/2098) the new version. They’ve been a huge help! [James Willis](https://www.linkedin.com/in/james-willis/), [Adam Binford](https://www.linkedin.com/in/adam-binford-a10b0321/) and the Apache Sedona team gave us new configurations, helped us fix our CI to enable the 0.9 release and drove Spark 4 support. James Willis became an official maintainer of GraphFrames to coordinate efforts between these projects.
+
+## New Contributors
+
+We have a lot of new contributors for this release!
+
+- [@bjornjorgensen](https://github.com/bjornjorgensen) made their first contribution in [#471](https://github.com/graphframes/graphframes/pull/471)
+- [@Nassizouz](https://github.com/Nassizouz) made their first contribution in [#474](https://github.com/graphframes/graphframes/pull/474)
+- [@SauronShepherd](https://github.com/SauronShepherd) made their first contribution in [#495](https://github.com/graphframes/graphframes/pull/495)
+- [@SemyonSinchenko](https://github.com/SemyonSinchenko) made their first contribution in [#487](https://github.com/graphframes/graphframes/pull/487)
+- [@dmatrix](https://github.com/dmatrix) made their first contribution in [#535](https://github.com/graphframes/graphframes/pull/535)
+- [@architch](https://github.com/architch) made their first contribution in [#320](https://github.com/graphframes/graphframes/pull/320)
+- [@james-willis](https://github.com/james-willis) made their first contribution in [#563](https://github.com/graphframes/graphframes/pull/563)
+- [@Conor0Callaghan](https://github.com/@Conor0Callaghan) made their first contribution in [#592](https://github.com/graphframes/graphframes/pull/592)
+- [@Kimahriman](https://github.com/Kimahriman) made their first contribution in [#608](https://github.com/graphframes/graphframes/pull/608)
+
+## A Call for Help
+
+We are building a [list of dependent projects](https://github.com/graphframes/graphframes/discussions/616), so if you use GraphFrames, please let us know! We want your help testing new versions before the release.
+
+Got questions or concerns? Let us know what you think! Find us on Discord in [#graphframes on GraphGeeks](https://discord.com/channels/1162999022819225631/1326257052368113674), or join the [GraphFrames Google Group](https://groups.google.com/g/graphframes/).
\ No newline at end of file
diff --git a/docs/src/05-blog/directory.conf b/docs/src/05-blog/directory.conf
new file mode 100644
index 000000000..0d3ec6702
--- /dev/null
+++ b/docs/src/05-blog/directory.conf
@@ -0,0 +1 @@
+laika.title = GraphFrames Blog
\ No newline at end of file
diff --git a/docs/src/06-contributing/01-contributing-guide.md b/docs/src/06-contributing/01-contributing-guide.md
new file mode 100644
index 000000000..aa3cd8de8
--- /dev/null
+++ b/docs/src/06-contributing/01-contributing-guide.md
@@ -0,0 +1 @@
+# Contributing Guide
\ No newline at end of file
diff --git a/docs/src/06-contributing/02-community.md b/docs/src/06-contributing/02-community.md
new file mode 100644
index 000000000..d740265a3
--- /dev/null
+++ b/docs/src/06-contributing/02-community.md
@@ -0,0 +1 @@
+# GraphFrames community
\ No newline at end of file
diff --git a/docs/src/06-contributing/directory.conf b/docs/src/06-contributing/directory.conf
new file mode 100644
index 000000000..ba8870be3
--- /dev/null
+++ b/docs/src/06-contributing/directory.conf
@@ -0,0 +1 @@
+laika.title = Contributing
\ No newline at end of file
diff --git a/docs/src/api/directory.conf b/docs/src/api/directory.conf
new file mode 100644
index 000000000..3156e1102
--- /dev/null
+++ b/docs/src/api/directory.conf
@@ -0,0 +1 @@
+laika.targetFormats = [html]
\ No newline at end of file
diff --git a/docs/img/4-node-directed-graphlets.png b/docs/src/img/4-node-directed-graphlets.png
similarity index 100%
rename from docs/img/4-node-directed-graphlets.png
rename to docs/src/img/4-node-directed-graphlets.png
diff --git a/docs/img/Directed-Graphlet-G17.png b/docs/src/img/Directed-Graphlet-G17.png
similarity index 100%
rename from docs/img/Directed-Graphlet-G17.png
rename to docs/src/img/Directed-Graphlet-G17.png
diff --git a/docs/img/Directed-Graphlet-G22.png b/docs/src/img/Directed-Graphlet-G22.png
similarity index 100%
rename from docs/img/Directed-Graphlet-G22.png
rename to docs/src/img/Directed-Graphlet-G22.png
diff --git a/docs/img/Directed-Graphlet-G30.png b/docs/src/img/Directed-Graphlet-G30.png
similarity index 100%
rename from docs/img/Directed-Graphlet-G30.png
rename to docs/src/img/Directed-Graphlet-G30.png
diff --git a/docs/img/G11_motif.png b/docs/src/img/G11_motif.png
similarity index 100%
rename from docs/img/G11_motif.png
rename to docs/src/img/G11_motif.png
diff --git a/docs/img/G4_and_G5_directed_network_motif.png b/docs/src/img/G4_and_G5_directed_network_motif.png
similarity index 100%
rename from docs/img/G4_and_G5_directed_network_motif.png
rename to docs/src/img/G4_and_G5_directed_network_motif.png
diff --git a/docs/img/GraphFrames-Logo-Dark-Small.png b/docs/src/img/GraphFrames-Logo-Dark-Small.png
similarity index 100%
rename from docs/img/GraphFrames-Logo-Dark-Small.png
rename to docs/src/img/GraphFrames-Logo-Dark-Small.png
diff --git a/docs/img/GraphFrames-Logo-Large.png b/docs/src/img/GraphFrames-Logo-Large.png
similarity index 100%
rename from docs/img/GraphFrames-Logo-Large.png
rename to docs/src/img/GraphFrames-Logo-Large.png
diff --git a/docs/img/GraphFrames-Logo-Small.png b/docs/src/img/GraphFrames-Logo-Small.png
similarity index 100%
rename from docs/img/GraphFrames-Logo-Small.png
rename to docs/src/img/GraphFrames-Logo-Small.png
diff --git a/docs/img/android-chrome-192x192.png b/docs/src/img/android-chrome-192x192.png
similarity index 100%
rename from docs/img/android-chrome-192x192.png
rename to docs/src/img/android-chrome-192x192.png
diff --git a/docs/img/android-chrome-512x512.png b/docs/src/img/android-chrome-512x512.png
similarity index 100%
rename from docs/img/android-chrome-512x512.png
rename to docs/src/img/android-chrome-512x512.png
diff --git a/docs/img/apple-touch-icon.png b/docs/src/img/apple-touch-icon.png
similarity index 100%
rename from docs/img/apple-touch-icon.png
rename to docs/src/img/apple-touch-icon.png
diff --git a/docs/img/directed_graphlets.webp b/docs/src/img/directed_graphlets.webp
similarity index 100%
rename from docs/img/directed_graphlets.webp
rename to docs/src/img/directed_graphlets.webp
diff --git a/docs/img/favicon-16x16.png b/docs/src/img/favicon-16x16.png
similarity index 100%
rename from docs/img/favicon-16x16.png
rename to docs/src/img/favicon-16x16.png
diff --git a/docs/img/favicon-32x32.png b/docs/src/img/favicon-32x32.png
similarity index 100%
rename from docs/img/favicon-32x32.png
rename to docs/src/img/favicon-32x32.png
diff --git a/docs/img/favicon.ico b/docs/src/img/favicon.ico
similarity index 100%
rename from docs/img/favicon.ico
rename to docs/src/img/favicon.ico
diff --git a/docs/src/img/graphframes-internals/graphframes-overview.png b/docs/src/img/graphframes-internals/graphframes-overview.png
new file mode 100644
index 000000000..32ab58228
Binary files /dev/null and b/docs/src/img/graphframes-internals/graphframes-overview.png differ
diff --git a/docs/src/img/graphframes-internals/pregel-pagerank.png b/docs/src/img/graphframes-internals/pregel-pagerank.png
new file mode 100644
index 000000000..7baa405b1
Binary files /dev/null and b/docs/src/img/graphframes-internals/pregel-pagerank.png differ
diff --git a/docs/img/graphframes_logo.png b/docs/src/img/graphframes_logo.png
similarity index 100%
rename from docs/img/graphframes_logo.png
rename to docs/src/img/graphframes_logo.png
diff --git a/docs/img/java-sm.png b/docs/src/img/java-sm.png
similarity index 100%
rename from docs/img/java-sm.png
rename to docs/src/img/java-sm.png
diff --git a/docs/src/img/logo-dark.png b/docs/src/img/logo-dark.png
new file mode 100644
index 000000000..2c3bb92be
Binary files /dev/null and b/docs/src/img/logo-dark.png differ
diff --git a/docs/src/img/property-graph/property-graph-example.png b/docs/src/img/property-graph/property-graph-example.png
new file mode 100644
index 000000000..0d2cf7543
Binary files /dev/null and b/docs/src/img/property-graph/property-graph-example.png differ
diff --git a/docs/img/python-sm.png b/docs/src/img/python-sm.png
similarity index 100%
rename from docs/img/python-sm.png
rename to docs/src/img/python-sm.png
diff --git a/docs/img/scala-sm.png b/docs/src/img/scala-sm.png
similarity index 100%
rename from docs/img/scala-sm.png
rename to docs/src/img/scala-sm.png
diff --git a/docs/img/site.webmanifest b/docs/src/img/site.webmanifest
similarity index 100%
rename from docs/img/site.webmanifest
rename to docs/src/img/site.webmanifest
diff --git a/docs/user-guide.md b/docs/user-guide.md
deleted file mode 100644
index 2ec964da3..000000000
--- a/docs/user-guide.md
+++ /dev/null
@@ -1,1058 +0,0 @@
----
-layout: global
-displayTitle: GraphFrames User Guide
-title: User Guide
-description: GraphFrames GRAPHFRAMES_VERSION user guide
----
-
-This page gives examples of how to use GraphFrames for basic queries, motif finding, and general graph algorithms. This includes code examples in Scala and Python.
-
-* Table of contents (This text will be scraped.)
-{:toc}
-
-*Note: Most examples use the GraphFrame from the first subsection:
-[Creating GraphFrames](user-guide.html#creating-graphframes).*
-
-# Creating GraphFrames
-
-Users can create GraphFrames from vertex and edge DataFrames.
-
-* *Vertex DataFrame*: A vertex DataFrame should contain a special column named "id" which specifies
- unique IDs for each vertex in the graph.
-* *Edge DataFrame*: An edge DataFrame should contain two special columns: "src" (source vertex ID
- of edge) and "dst" (destination vertex ID of edge).
-
-Both DataFrames can have arbitrary other columns. Those columns can represent vertex and edge
-attributes.
-
-A GraphFrame can also be constructed from a single DataFrame containing edge information.
-The vertices will be inferred from the sources and destinations of the edges.
-
-
-
-The following example demonstrates how to create a GraphFrame from vertex and edge DataFrames.
-
-
-{% highlight scala %}
-import org.graphframes.GraphFrame
-// Vertex DataFrame
-val v = spark.createDataFrame(List(
- ("a", "Alice", 34),
- ("b", "Bob", 36),
- ("c", "Charlie", 30),
- ("d", "David", 29),
- ("e", "Esther", 32),
- ("f", "Fanny", 36),
- ("g", "Gabby", 60)
-)).toDF("id", "name", "age")
-
-// Edge DataFrame
-val e = spark.createDataFrame(List(
- ("a", "b", "friend"),
- ("b", "c", "follow"),
- ("c", "b", "follow"),
- ("f", "c", "follow"),
- ("e", "f", "follow"),
- ("e", "d", "friend"),
- ("d", "a", "friend"),
- ("a", "e", "friend")
-)).toDF("src", "dst", "relationship")
-// Create a GraphFrame
-val g = GraphFrame(v, e)
-{% endhighlight %}
-
-The GraphFrame constructed above is available in the GraphFrames package:
-{% highlight scala %}
-import org.graphframes.{examples,GraphFrame}
-
-val g: GraphFrame = examples.Graphs.friends
-{% endhighlight %}
-
-
-
-{% highlight python %}
-# Vertex DataFrame
-v = spark.createDataFrame([
- ("a", "Alice", 34),
- ("b", "Bob", 36),
- ("c", "Charlie", 30),
- ("d", "David", 29),
- ("e", "Esther", 32),
- ("f", "Fanny", 36),
- ("g", "Gabby", 60)
-], ["id", "name", "age"])
-
-# Edge DataFrame
-e = spark.createDataFrame([
- ("a", "b", "friend"),
- ("b", "c", "follow"),
- ("c", "b", "follow"),
- ("f", "c", "follow"),
- ("e", "f", "follow"),
- ("e", "d", "friend"),
- ("d", "a", "friend"),
- ("a", "e", "friend")
-], ["src", "dst", "relationship"])
-# Create a GraphFrame
-g = GraphFrame(v, e)
-{% endhighlight %}
-
-The GraphFrame constructed above is available in the GraphFrames package:
-{% highlight python %}
-from graphframes.examples import Graphs
-
-g = Graphs(spark).friends() # Get example graph
-{% endhighlight %}
-
-
-
-
-# Basic graph and DataFrame queries
-
-GraphFrames provide several simple graph queries, such as node degree.
-
-Also, since GraphFrames represent graphs as pairs of vertex and edge DataFrames, it is easy to make
-powerful queries directly on the vertex and edge DataFrames. Those DataFrames are made available
-as `vertices` and `edges` fields in the GraphFrame.
-
-
-
-
-{% highlight scala %}
-import org.graphframes.{examples,GraphFrame}
-
-val g: GraphFrame = examples.Graphs.friends // get example graph
-
-// Display the vertex and edge DataFrames
-g.vertices.show()
-// +--+-------+---+
-// |id| name|age|
-// +--+-------+---+
-// | a| Alice| 34|
-// | b| Bob| 36|
-// | c|Charlie| 30|
-// | d| David| 29|
-// | e| Esther| 32|
-// | f| Fanny| 36|
-// | g| Gabby| 60|
-// +--+-------+---+
-
-g.edges.show()
-// +---+---+------------+
-// |src|dst|relationship|
-// +---+---+------------+
-// | a| b| friend|
-// | b| c| follow|
-// | c| b| follow|
-// | f| c| follow|
-// | e| f| follow|
-// | e| d| friend|
-// | d| a| friend|
-// | a| e| friend|
-// +---+---+------------+
-
-// import Spark SQL package
-import org.apache.spark.sql.DataFrame
-
-// Get a DataFrame with columns "id" and "inDeg" (in-degree)
-val vertexInDegrees: DataFrame = g.inDegrees
-vertexInDegrees.show()
-
-// Find the youngest user's age in the graph.
-// This queries the vertex DataFrame.
-g.vertices.groupBy().min("age").show()
-
-// Count the number of "follows" in the graph.
-// This queries the edge DataFrame.
-val numFollows = g.edges.filter("relationship = 'follow'").count()
-{% endhighlight %}
-
-
-
-{% highlight python %}
-from graphframes.examples import Graphs
-
-g = Graphs(spark).friends() # Get example graph
-
-# Display the vertex DataFrame
-g.vertices.show()
-
-# +--+-------+---+
-# |id| name|age|
-# +--+-------+---+
-# | a| Alice| 34|
-# | b| Bob| 36|
-# | c|Charlie| 30|
-# | d| David| 29|
-# | e| Esther| 32|
-# | f| Fanny| 36|
-# | g| Gabby| 60|
-# +--+-------+---+
-
-# Display the edge DataFrame
-g.edges.show()
-
-# +---+---+------------+
-# |src|dst|relationship|
-# +---+---+------------+
-# | a| b| friend|
-# | b| c| follow|
-# | c| b| follow|
-# | f| c| follow|
-# | e| f| follow|
-# | e| d| friend|
-# | d| a| friend|
-# | a| e| friend|
-# +---+---+------------+
-
-# Get a DataFrame with columns "id" and "inDegree" (in-degree)
-vertexInDegrees = g.inDegrees
-
-# Find the youngest user's age in the graph
-# This queries the vertex DataFrame
-g.vertices.groupBy().min("age").show()
-
-# Count the number of "follows" in the graph
-# This queries the edge DataFrame
-numFollows = g.edges.filter("relationship = 'follow'").count()
-{% endhighlight %}
-
-
-
-
-# Motif finding
-
-Motif finding refers to searching for structural patterns in a graph. For an example of real-world use, check out the [Motif Finding Tutorial](motif-tutorial.html).
-
-GraphFrame motif finding uses a simple Domain-Specific Language (DSL) for expressing structural queries. For example, `graph.find("(a)-[e]->(b); (b)-[e2]->(a)")` will search for pairs of vertices `a,b` connected by edges in both directions. It will return a `DataFrame` of all such structures in the graph, with columns for each of the named elements (vertices or edges) in the motif. In this case, the returned columns will be "a, b, e, e2."
-
-DSL for expressing structural patterns:
-
-* The basic unit of a pattern is an edge.
- For example, `"(a)-[e]->(b)"` expresses an edge `e` from vertex `a` to vertex `b`.
- Note that vertices are denoted by parentheses `(a)`, while edges are denoted by
- square brackets `[e]`.
-* A pattern is expressed as a union of edges. Edge patterns can be joined with semicolons.
- Motif `"(a)-[e]->(b); (b)-[e2]->(c)"` specifies two edges from `a` to `b` to `c`.
-* Within a pattern, names can be assigned to vertices and edges. For example,
- `"(a)-[e]->(b)"` has three named elements: vertices `a,b` and edge `e`.
- These names serve two purposes:
- * The names can identify common elements among edges. For example,
- `"(a)-[e]->(b); (b)-[e2]->(c)"` specifies that the same vertex `b` is the destination
- of edge `e` and source of edge `e2`.
- * The names are used as column names in the result `DataFrame`. If a motif contains
- named vertex `a`, then the result `DataFrame` will contain a column "a" which is a
- `StructType` with sub-fields equivalent to the schema (columns) of
- `GraphFrame.vertices`. Similarly, an edge `e` in a motif will produce a column "e"
- in the result `DataFrame` with sub-fields equivalent to the schema (columns) of
- `GraphFrame.edges`.
- * Be aware that names do *not* identify *distinct* elements: two elements with different
- names may refer to the same graph element. For example, in the motif
- `"(a)-[e]->(b); (b)-[e2]->(c)"`, the names `a` and `c` could refer to the same vertex.
- To restrict named elements to be distinct vertices or edges, use post-hoc filters
- such as `resultDataframe.filter("a.id != c.id")`.
-* It is acceptable to omit names for vertices or edges in motifs when not needed.
- E.g., `"(a)-[]->(b)"` expresses an edge between vertices `a,b` but does not assign a name
- to the edge. There will be no column for the anonymous edge in the result `DataFrame`.
- Similarly, `"(a)-[e]->()"` indicates an out-edge of vertex `a` but does not name
- the destination vertex. These are called *anonymous* vertices and edges.
-* An edge can be negated to indicate that the edge should *not* be present in the graph.
- E.g., `"(a)-[]->(b); !(b)-[]->(a)"` finds edges from `a` to `b` for which there is *no*
- edge from `b` to `a`.
-
-Restrictions:
-
-* Motifs are not allowed to contain edges without any named elements: `"()-[]->()"` and
- `"!()-[]->()"` are prohibited terms.
-* Motifs are not allowed to contain named edges within negated terms (since these named
- edges would never appear within results). E.g., `"!(a)-[ab]->(b)"` is invalid, but
- `"!(a)-[]->(b)"` is valid.
-
-More complex queries, such as queries which operate on vertex or edge attributes,
-can be expressed by applying filters to the result `DataFrame`.
-
-This can return duplicate rows. E.g., a query `"(u)-[]->()"` will return a result for each
-matching edge, even if those edges share the same vertex `u`.
-
-
-
-
-
-For API details, refer to the [API docs](api/scala/index.html#org.graphframes.GraphFrame).
-
-{% highlight scala %}
-import org.apache.spark.sql.DataFrame
-import org.graphframes.{examples,GraphFrame}
-
-val g: GraphFrame = examples.Graphs.friends // get example graph
-
-// Search for pairs of vertices with edges in both directions between them.
-val motifs: DataFrame = g.find("(a)-[e]->(b); (b)-[e2]->(a)")
-motifs.show()
-
-// More complex queries can be expressed by applying filters.
-motifs.filter("b.age > 30").show()
-{% endhighlight %}
-
-
-
-
-For API details, refer to the [API docs](api/python/graphframes.html#graphframes.GraphFrame.find).
-
-{% highlight python %}
-from graphframes.examples import Graphs
-
-g = Graphs(spark).friends() # Get example graph
-
-# Search for pairs of vertices with edges in both directions between them
-motifs = g.find("(a)-[e]->(b); (b)-[e2]->(a)")
-motifs.show()
-
-# More complex queries can be expressed by applying filters
-motifs.filter("b.age > 30").show()
-{% endhighlight %}
-
-
-
-
-Many motif queries are stateless and simple to express, as in the examples above.
-The next examples demonstrate more complex queries which carry state along a path in the motif.
-These queries can be expressed by combining GraphFrame motif finding with filters on the result,
-where the filters use sequence operations to construct a series of `DataFrame` `Column`s.
-
-For example, suppose one wishes to identify a chain of 4 vertices with some property defined
-by a sequence of functions. That is, among chains of 4 vertices `a->b->c->d`, identify the subset
-of chains matching this complex filter:
-
-* Initialize state on path.
-* Update state based on vertex `a`.
-* Update state based on vertex `b`.
-* Etc. for `c` and `d`.
-* If final state matches some condition, then the chain is accepted by the filter.
-
-The below code snippets demonstrate this process, where we identify chains of 4 vertices
-such that at least 2 of the 3 edges are "friend" relationships.
-In this example, the state is the current count of "friend" edges; in general, it could be any
-`DataFrame Column`.
-
-
-
-
-{% highlight scala %}
-import org.apache.spark.sql.{Column, DataFrame}
-import org.apache.spark.sql.functions.{col, when}
-import org.graphframes.{examples,GraphFrame}
-
-val g: GraphFrame = examples.Graphs.friends // get example graph
-
-// Find chains of 4 vertices.
-val chain4: DataFrame = g.find("(a)-[ab]->(b); (b)-[bc]->(c); (c)-[cd]->(d)")
-chain4.show()
-
-// Query on sequence, with state (cnt)
-// (a) Define method for updating state given the next element of the motif.
-def sumFriends(cnt: Column, relationship: Column): Column = {
- when(relationship === "friend", cnt + 1).otherwise(cnt)
-}
-// (b) Use sequence operation to apply method to sequence of elements in motif.
-// In this case, the elements are the 3 edges.
-val condition = { Seq("ab", "bc", "cd")
- .foldLeft(lit(0))((cnt, e) => sumFriends(cnt, col(e)("relationship"))) }
-// (c) Apply filter to DataFrame.
-val chainWith2Friends2 = chain4.where(condition >= 2)
-chainWith2Friends2.show()
-{% endhighlight %}
-
-
-
-{% highlight python %}
-from pyspark.sql.functions import col, lit, when
-from pyspark.sql.types import IntegerType
-from graphframes.examples import Graphs
-
-
-g = Graphs(spark).friends() # Get example graph
-
-chain4 = g.find("(a)-[ab]->(b); (b)-[bc]->(c); (c)-[cd]->(d)")
-
-# Query on sequence, with state (cnt)
-# (a) Define method for updating state given the next element of the motif
-sumFriends =\
- lambda cnt,relationship: when(relationship == "friend", cnt+1).otherwise(cnt)
-
-# (b) Use sequence operation to apply method to sequence of elements in motif
-# In this case, the elements are the 3 edges
-condition =\
- reduce(lambda cnt,e: sumFriends(cnt, col(e).relationship), ["ab", "bc", "cd"], lit(0))
-
-# (c) Apply filter to DataFrame
-chainWith2Friends2 = chain4.where(condition >= 2)
-chainWith2Friends2.show()
-{% endhighlight %}
-
-
-
-
-The above example demonstrated a stateful motif for a fixed-length chain. Currently, in order to
-search for variable-length motifs, users need to run one query for each possible length.
-However, the above query patterns allow users to re-use the same code for each length, with the
-only change being to update the sequence of motif elements ("ab", "bc", "cd" above).
-
-# Subgraphs
-
-In GraphX, the `subgraph()` method takes an edge triplet (edge, src vertex, and dst vertex, plus
-attributes) and allows the user to select a subgraph based on triplet and vertex filters.
-
-GraphFrames provide an even more powerful way to select subgraphs based on a combination of
-motif finding and DataFrame filters. We provide three helper methods for subgraph selection.
-`filterVertices(condition)`, `filterEdges(condition)`, and `dropIsolatedVertices()`.
-
-**Simple subgraph: vertex and edge filters**:
-The following example shows how to select a subgraph based upon vertex and edge filters.
-
-
-
-
-{% highlight scala %}
-import org.graphframes.{examples,GraphFrame}
-
-val g: GraphFrame = examples.Graphs.friends
-
-// Select subgraph of users older than 30, and relationships of type "friend".
-// Drop isolated vertices (users) which are not contained in any edges (relationships).
-val g1 = g.filterVertices("age > 30").filterEdges("relationship = 'friend'").dropIsolatedVertices()
-
-{% endhighlight %}
-
-
-
-{% highlight python %}
-from graphframes.examples import Graphs
-
-g = Graphs(spark).friends() # Get example graph
-
-# Select subgraph of users older than 30, and relationships of type "friend"
-# Drop isolated vertices (users) which are not contained in any edges (relationships)
-g1 = g.filterVertices("age > 30").filterEdges("relationship = 'friend'").dropIsolatedVertices()
-
-{% endhighlight %}
-
-
-
-
-**Complex subgraph: triplet filters**:
-The following example shows how to select a subgraph based upon triplet filters which
-operate on an edge and its src and dst vertices. This example could be extended to go beyond
-triplets by using more complex motifs.
-
-
-
-
-{% highlight scala %}
-import org.graphframes.{examples,GraphFrame}
-
-val g: GraphFrame = examples.Graphs.friends // get example graph
-
-// Select subgraph based on edges "e" of type "follow"
-// pointing from a younger user "a" to an older user "b".
-val paths = { g.find("(a)-[e]->(b)")
- .filter("e.relationship = 'follow'")
- .filter("a.age < b.age") }
-// "paths" contains vertex info. Extract the edges.
-val e2 = paths.select("e.*")
-
-// Construct the subgraph
-val g2 = GraphFrame(g.vertices, e2)
-{% endhighlight %}
-
-
-
-{% highlight python %}
-from graphframes.examples import Graphs
-
-g = Graphs(spark).friends() # Get example graph
-
-# Select subgraph based on edges "e" of type "follow"
-# pointing from a younger user "a" to an older user "b"
-paths = g.find("(a)-[e]->(b)")\
- .filter("e.relationship = 'follow'")\
- .filter("a.age < b.age")
-
-# "paths" contains vertex info. Extract the edges
-e2 = paths.select("e.src", "e.dst", "e.relationship")
-
-# In Spark 1.5+, the user may simplify this call
-# val e2 = paths.select("e.*")
-# Construct the subgraph
-g2 = GraphFrame(g.vertices, e2)
-{% endhighlight %}
-
-
-
-
-# Graph algorithms
-
-GraphFrames provides the same suite of standard graph algorithms as GraphX, plus some new ones.
-We provide brief descriptions and code snippets below.
-See the [API docs](api/scala/index.html#org.graphframes.lib.package) for more details.
-
-Some of the algorithms are currently wrappers around GraphX implementations, so they may not be
-more scalable than GraphX. More algorithms will be migrated to native GraphFrames implementations
-in the future.
-
-## Breadth-first search (BFS)
-
-Breadth-first search (BFS) finds the shortest path(s) from one vertex (or a set of vertices)
-to another vertex (or a set of vertices). The beginning and end vertices are specified as
-Spark DataFrame expressions.
-
-See [Wikipedia on BFS](https://en.wikipedia.org/wiki/Breadth-first_search) for more background.
-
-
-
-The following code snippets uses BFS to find path between
-vertex with name "Esther" to a vertex with age < 32.
-
-
-
-For API details, refer to the [API docs](api/scala/index.html#org.graphframes.lib.BFS).
-
-{% highlight scala %}
-import org.graphframes.{examples,GraphFrame}
-
-val g: GraphFrame = examples.Graphs.friends // get example graph
-
-// Search from "Esther" for users of age < 32.
-val paths = g.bfs.fromExpr("name = 'Esther'").toExpr("age < 32").run()
-paths.show()
-
-// Specify edge filters or max path lengths.
-val paths = { g.bfs.fromExpr("name = 'Esther'").toExpr("age < 32")
- .edgeFilter("relationship != 'friend'")
- .maxPathLength(3).run() }
-paths.show()
-{% endhighlight %}
-
-
-
-
-For API details, refer to the [API docs](api/python/graphframes.html#graphframes.GraphFrame.bfs).
-
-{% highlight python %}
-from graphframes.examples import Graphs
-
-
-g = Graphs(spark).friends() # Get example graph
-
-# Search from "Esther" for users of age < 32
-paths = g.bfs("name = 'Esther'", "age < 32")
-paths.show()
-
-# Specify edge filters or max path lengths
-g.bfs("name = 'Esther'", "age < 32",\
- edgeFilter="relationship != 'friend'", maxPathLength=3)
-{% endhighlight %}
-
-
-
-
-## Connected components
-
-Computes the connected component membership of each vertex and returns a graph with each vertex
-assigned a component ID.
-
-See [Wikipedia](https://en.wikipedia.org/wiki/Connected_component_(graph_theory)) for background.
-
-NOTE: With GraphFrames 0.3.0 and later releases, the default Connected Components algorithm
-requires setting a Spark checkpoint directory. Users can revert to the old algorithm using
-`connectedComponents.setAlgorithm("graphx")`.
-
-
-
-
-
-For API details, refer to the [API docs](api/scala/index.html#org.graphframes.lib.ConnectedComponents).
-
-{% highlight scala %}
-import org.graphframes.{examples,GraphFrame}
-
-sc.setCheckpointDir("/tmp/spark-checkpoints")
-
-val g: GraphFrame = examples.Graphs.friends // get example graph
-
-val result = g.connectedComponents.run()
-result.select("id", "component").orderBy("component").show()
-{% endhighlight %}
-
-
-
-
-For API details, refer to the [API docs](api/python/graphframes.html#graphframes.GraphFrame.connectedComponents).
-
-{% highlight python %}
-from graphframes.examples import Graphs
-
-sc.setCheckpointDir("/tmp/spark-checkpoints")
-
-g = Graphs(spark).friends() # Get example graph
-
-result = g.connectedComponents()
-result.select("id", "component").orderBy("component").show()
-{% endhighlight %}
-
-
-
-
-### Strongly connected components
-
-Compute the strongly connected component (SCC) of each vertex and return a graph with each vertex
-assigned to the SCC containing that vertex.
-
-See [Wikipedia](https://en.wikipedia.org/wiki/Strongly_connected_component) for background.
-
-
-
-
-
-For API details, refer to the [API docs](api/scala/index.html#org.graphframes.lib.StronglyConnectedComponents).
-
-{% highlight scala %}
-import org.graphframes.{examples,GraphFrame}
-
-val g: GraphFrame = examples.Graphs.friends // get example graph
-
-val result = g.stronglyConnectedComponents.maxIter(10).run()
-result.select("id", "component").orderBy("component").show()
-{% endhighlight %}
-
-
-
-
-For API details, refer to the [API docs](api/python/graphframes.html#graphframes.GraphFrame.stronglyConnectedComponents).
-
-{% highlight python %}
-from graphframes.examples import Graphs
-
-
-sc.setCheckpointDir("/tmp/spark-checkpoints")
-
-g = Graphs(spark).friends() # Get example graph
-
-result = g.stronglyConnectedComponents(maxIter=10)
-result.select("id", "component").orderBy("component").show()
-{% endhighlight %}
-
-
-
-
-## Label Propagation Algorithm (LPA)
-
-Run static Label Propagation Algorithm for detecting communities in networks.
-
-Each node in the network is initially assigned to its own community. At every superstep, nodes
-send their community affiliation to all neighbors and update their state to the mode community
-affiliation of incoming messages.
-
-LPA is a standard community detection algorithm for graphs. It is very inexpensive
-computationally, although (1) convergence is not guaranteed and (2) one can end up with
-trivial solutions (all nodes are identified into a single community).
-
-See [Wikipedia](https://en.wikipedia.org/wiki/Label_Propagation_Algorithm) for background.
-
-
-
-
-
-For API details, refer to the [API docs](api/scala/index.html#org.graphframes.lib.LabelPropagation).
-
-{% highlight scala %}
-import org.graphframes.{examples,GraphFrame}
-
-val g: GraphFrame = examples.Graphs.friends // get example graph
-
-val result = g.labelPropagation.maxIter(5).run()
-result.select("id", "label").show()
-{% endhighlight %}
-
-
-
-
-For API details, refer to the [API docs](api/python/graphframes.html#graphframes.GraphFrame.labelPropagation).
-
-{% highlight python %}
-from graphframes.examples import Graphs
-
-
-g = Graphs(spark).friends() # Get example graph
-
-result = g.labelPropagation(maxIter=5)
-result.select("id", "label").show()
-{% endhighlight %}
-
-
-
-
-## PageRank
-
-There are two implementations of PageRank.
-
-* The first one uses the `org.apache.spark.graphx.graph` interface with `aggregateMessages` and runs
-PageRank for a fixed number of iterations. This can be executed by setting `maxIter`.
-* The second implementation uses the `org.apache.spark.graphx.Pregel` interface and runs PageRank until
-convergence and this can be run by setting `tol`.
-
-Both implementations support non-personalized and personalized PageRank, where setting a `sourceId`
-personalizes the results for that vertex.
-
-See [Wikipedia](https://en.wikipedia.org/wiki/PageRank) for background.
-
-
-
-
-
-For API details, refer to the [API docs](api/scala/index.html#org.graphframes.lib.PageRank).
-
-{% highlight scala %}
-import org.graphframes.{examples,GraphFrame}
-
-val g: GraphFrame = examples.Graphs.friends // get example graph
-
-// Run PageRank until convergence to tolerance "tol".
-val results: GraphFrame = g.pageRank.resetProbability(0.15).tol(0.01).run()
-
-// Display resulting pageranks and final edge weights
-// Note that the displayed pagerank may be truncated, e.g., missing the E notation.
-// In Spark 1.5+, you can use show(truncate=false) to avoid truncation.
-results.vertices.select("id", "pagerank").show()
-results.edges.select("src", "dst", "weight").show()
-
-// Run PageRank for a fixed number of iterations.
-val results2 = g.pageRank.resetProbability(0.15).maxIter(10).run()
-
-// Run PageRank personalized for vertex "a"
-val results3 = g.pageRank.resetProbability(0.15).maxIter(10).sourceId("a").run()
-
-// Run PageRank personalized for vertex ["a", "b", "c", "d"] in parallel
-val results4 = g.parallelPersonalizedPageRank.resetProbability(0.15).maxIter(10).sourceIds(Array("a", "b", "c", "d")).run()
-results4.vertices.show()
-results4.edges.show()
-{% endhighlight %}
-
-
-
-
-For API details, refer to the [API docs](api/python/graphframes.html#graphframes.GraphFrame.pageRank).
-
-{% highlight python %}
-from graphframes.examples import Graphs
-
-
-g = Graphs(spark).friends() # Get example graph
-
-# Run PageRank until convergence to tolerance "tol"
-results = g.pageRank(resetProbability=0.15, tol=0.01)
-
-# Display resulting pageranks and final edge weights
-# Note that the displayed pagerank may be truncated, e.g., missing the E notation
-# In Spark 1.5+, you can use show(truncate=False) to avoid truncation
-results.vertices.select("id", "pagerank").show()
-results.edges.select("src", "dst", "weight").show()
-
-# Run PageRank for a fixed number of iterations
-results2 = g.pageRank(resetProbability=0.15, maxIter=10)
-
-# Run PageRank personalized for vertex "a"
-results3 = g.pageRank(resetProbability=0.15, maxIter=10, sourceId="a")
-
-# Run PageRank personalized for vertex ["a", "b", "c", "d"] in parallel
-results4 = g.parallelPersonalizedPageRank(resetProbability=0.15, sourceIds=["a", "b", "c", "d"], maxIter=10)
-{% endhighlight %}
-
-
-
-
-
-## Shortest paths
-
-Computes shortest paths from each vertex to the given set of landmark vertices, where landmarks are
-specified by vertex ID. Note that this takes edge direction into account.
-
-See [Wikipedia](https://en.wikipedia.org/wiki/Shortest_path_problem) for background.
-
-
-
-
-
-For API details, refer to the [API docs](api/scala/index.html#org.graphframes.lib.ShortestPaths).
-
-{% highlight scala %}
-import org.graphframes.{examples,GraphFrame}
-
-val g: GraphFrame = examples.Graphs.friends // get example graph
-
-val results = g.shortestPaths.landmarks(Seq("a", "d")).run()
-results.select("id", "distances").show()
-{% endhighlight %}
-
-
-
-
-For API details, refer to the [API docs](api/python/graphframes.html#graphframes.GraphFrame.shortestPaths).
-
-{% highlight python %}
-from graphframes.examples import Graphs
-
-
-g = Graphs(spark).friends() # Get example graph
-
-results = g.shortestPaths(landmarks=["a", "d"])
-results.select("id", "distances").show()
-{% endhighlight %}
-
-
-
-
-## Triangle count
-
-Computes the number of triangles passing through each vertex.
-
-
-
-
-
-For API details, refer to the [API docs](api/scala/index.html#org.graphframes.lib.TriangleCount).
-
-{% highlight scala %}
-import org.graphframes.{examples,GraphFrame}
-
-val g: GraphFrame = examples.Graphs.friends // get example graph
-
-val results = g.triangleCount.run()
-results.select("id", "count").show()
-{% endhighlight %}
-
-
-
-
-For API details, refer to the [API docs](api/python/graphframes.html#graphframes.GraphFrame.triangleCount).
-
-{% highlight python %}
-from graphframes.examples import Graphs
-
-
-g = Graphs(spark).friends() # Get example graph
-
-results = g.triangleCount()
-results.select("id", "count").show()
-{% endhighlight %}
-
-
-
-
-# Saving and loading GraphFrames
-
-Since GraphFrames are built around DataFrames, they automatically support saving and loading
-to and from the same set of datasources.
-Refer to the [Spark SQL User Guide on datasources](http://spark.apache.org/docs/latest/sql-programming-guide.html#data-sources)
-for more details.
-
-The below example shows how to save and then load a graph.
-
-
-
-
-{% highlight scala %}
-import org.graphframes.{examples,GraphFrame}
-
-val g: GraphFrame = examples.Graphs.friends // get example graph
-
-// Save vertices and edges as Parquet to some location.
-g.vertices.write.parquet("hdfs://myLocation/vertices")
-g.edges.write.parquet("hdfs://myLocation/edges")
-
-// Load the vertices and edges back.
-val sameV = spark.read.parquet("hdfs://myLocation/vertices")
-val sameE = spark.read.parquet("hdfs://myLocation/edges")
-
-// Create an identical GraphFrame.
-val sameG = GraphFrame(sameV, sameE)
-{% endhighlight %}
-
-
-
-{% highlight python %}
-from graphframes.examples import Graphs
-
-
-g = Graphs(spark).friends() # Get example graph
-
-# Save vertices and edges as Parquet to some location
-g.vertices.write.parquet("hdfs://myLocation/vertices")
-g.edges.write.parquet("hdfs://myLocation/edges")
-
-# Load the vertices and edges back
-sameV = spark.read.parquet("hdfs://myLocation/vertices")
-sameE = spark.read.parquet("hdfs://myLocation/edges")
-
-# Create an identical GraphFrame
-sameG = GraphFrame(sameV, sameE)
-{% endhighlight %}
-
-
-
-
-# Message passing via AggregateMessages
-
-Like GraphX, GraphFrames provides primitives for developing graph algorithms.
-The two key components are:
-
-* `aggregateMessages`: Send messages between vertices, and aggregate messages for each vertex.
- GraphFrames provides a native `aggregateMessages` method implemented using DataFrame operations.
- This may be used analogously to the GraphX API.
-* joins: Join message aggregates with the original graph.
- GraphFrames rely on `DataFrame` joins, which provide the full functionality of GraphX joins.
-
-The below code snippets show how to use `aggregateMessages` to compute the sum of the ages
-of adjacent users.
-
-
-
-
-
-For API details, refer to the [API docs](api/scala/index.html#org.graphframes.lib.AggregateMessages).
-
-{% highlight scala %}
-import org.graphframes.{examples,GraphFrame}
-import org.graphframes.lib.AggregateMessages
-
-val g: GraphFrame = examples.Graphs.friends // get example graph
-
-// We will use AggregateMessages utilities later, so name it "AM" for short.
-val AM = AggregateMessages
-
-// For each user, sum the ages of the adjacent users.
-val msgToSrc = AM.dst("age")
-val msgToDst = AM.src("age")
-val agg = { g.aggregateMessages
- .sendToSrc(msgToSrc) // send destination user's age to source
- .sendToDst(msgToDst) // send source user's age to destination
- .agg(sum(AM.msg).as("summedAges")) } // sum up ages, stored in AM.msg column
-agg.show()
-{% endhighlight %}
-
-For a more complex example, look at the code used to implement the
-[Belief Propagation example](api/scala/index.html#org.graphframes.examples.BeliefPropagation$).
-
-
-
-
-
-For API details, refer to the
-[API docs](api/python/graphframes.html#graphframes.GraphFrame.aggregateMessages).
-
-{% highlight python %}
-from graphframes.lib import AggregateMessages as AM
-from graphframes.examples import Graphs
-from pyspark.sql.functions import sum as sqlsum
-
-
-g = Graphs(spark).friends() # Get example graph
-
-# For each user, sum the ages of the adjacent users
-msgToSrc = AM.dst["age"]
-msgToDst = AM.src["age"]
-agg = g.aggregateMessages(
- sqlsum(AM.msg).alias("summedAges"),
- sendToSrc=msgToSrc,
- sendToDst=msgToDst)
-agg.show()
-
-{% endhighlight %}
-
-For a more complex example, look at the code used to implement the
-[Belief Propagation example](api/python/graphframes.examples.html#graphframes.examples.BeliefPropagation).
-
-
-
-
-
-# GraphX-GraphFrame conversions
-
-We provide utilities for converting between GraphFrame and GraphX graphs.
-See the [GraphX User Guide](http://spark.apache.org/docs/latest/graphx-programming-guide.html)
-for details on GraphX.
-
-## GraphFrame to GraphX
-
-Conversion to GraphX creates a GraphX `Graph` which has `Long` vertex IDs and attributes
-of type `Row`.
-
-Vertex and edge attributes are the original rows in `vertices` and `edges`, respectively.
-
-Note that vertex (and edge) attributes include vertex IDs (and source, destination IDs)
-in order to support non-Long vertex IDs. If the vertex IDs are not convertible to Long values,
-then the values are indexed in order to generate corresponding Long vertex IDs (which is an
-expensive operation).
-
-The column ordering of the returned `Graph` vertex and edge attributes are specified by
-`GraphFrame.vertexColumns` and `GraphFrame.edgeColumns`, respectively.
-
-## GraphX to GraphFrame
-
-GraphFrame provides two conversions methods. The first takes any GraphX graph and converts
-the vertex and edge `RDD`s into `DataFrame`s using schema inference. Those DataFrames
-are then used to create a GraphFrame.
-
-The second conversion method is more complex and is useful for users with existing GraphX code.
-Its main purpose is to support workflows of the following form: (1) convert a GraphFrame to GraphX,
-(2) run GraphX code to augment the GraphX graph with new vertex or edge attributes, and
-(3) merge the new attributes back into the original GraphFrame.
-
-For example, given:
-
-* GraphFrame `originalGraph`
-* GraphX `Graph[String, Int]` `graph` with a String vertex attribute we want to call "category"
- and an Int edge attribute we want to call "count"
-
-We can call `fromGraphX(originalGraph, graph, Seq("category"), Seq("count"))` to produce
-a new GraphFrame. The new GraphFrame will be an augmented version of `originalGraph`,
-with new `GraphFrame.vertices` column "category" and new `GraphFrame.edges` column "count" added.
-
-For example usage, look at the code used to implement the
-[Belief Propagation example](api/scala/index.html#org.graphframes.examples.BeliefPropagation$).
-
-## Example conversions
-
-The below example demonstrates simple GraphFrame-GraphX conversions.
-
-
-
-
-For API details, refer to the API docs for:
-
-* [toGraphX](api/scala/index.html#org.graphframes.GraphFrame)
-* [fromGraphX](api/scala/index.html#org.graphframes.GraphFrame$)
-
-{% highlight scala %}
-import org.apache.spark.graphx.Graph
-import org.apache.spark.sql.Row
-import org.graphframes.{examples,GraphFrame}
-
-val g: GraphFrame = examples.Graphs.friends // get example graph
-
-// Convert to GraphX
-val gx: Graph[Row, Row] = g.toGraphX
-
-// Convert back to GraphFrame.
-// Note that the schema is changed because of constraints in the GraphX API.
-val g2: GraphFrame = GraphFrame.fromGraphX(gx)
-{% endhighlight %}
-
-
-
-These conversions are only supported in Scala since GraphX does not have a Python API.
-
-# GraphFrames Internals
-
-To learn how GraphFrames works internally to combine graph and relational queries, check out the paper [GraphFrames: An Integrated API for Mixing Graph and
-Relational Queries, Dave et al. 2016](https://people.eecs.berkeley.edu/~matei/papers/2016/grades_graphframes.pdf).
diff --git a/index.html b/index.html
deleted file mode 100644
index 297fb054b..000000000
--- a/index.html
+++ /dev/null
@@ -1,9 +0,0 @@
-
-
-
-
- Your browser should automatically redirect you to the docs.
- Click here if you aren't redirected within 3 seconds.
-
-
-
diff --git a/project/LaikaCustomDirectives.scala b/project/LaikaCustomDirectives.scala
new file mode 100644
index 000000000..461cdc901
--- /dev/null
+++ b/project/LaikaCustomDirectives.scala
@@ -0,0 +1,54 @@
+import cats.syntax.all.*
+import laika.api.bundle.BlockDirectives
+import laika.api.bundle.DirectiveRegistry
+import laika.api.bundle.LinkDirectives
+import laika.api.bundle.SpanDirectives
+import laika.api.bundle.SpanDirectives.dsl.*
+import laika.api.bundle.TemplateDirectives
+import laika.ast.*
+
+object LaikaCustomDirectives extends DirectiveRegistry {
+ val pydocDirective: SpanDirectives.Directive = SpanDirectives.create("pydoc") {
+ (attribute(0).as[String], cursor, source).mapN { (pyClass, cursor, source) =>
+ cursor.config
+ .get[String]("pydoc.baseUri")
+ .fold(
+ error => InvalidSpan(s"Invalid PyDocs baseUri: $error", source),
+ baseUri => {
+ val companion = if (pyClass.startsWith("graphframes.lib")) {
+ SpanLink.external(s"$baseUri/graphframes/lib.html#$pyClass")
+ } else if (pyClass.startsWith("graphframes.examples")) {
+ SpanLink.external(s"$baseUri/graphframes/examples.html#$pyClass")
+ } else {
+ SpanLink.external(s"$baseUri/graphframes.html#$pyClass")
+ }
+
+ companion(pyClass)
+ })
+ }
+ }
+ val scaladocDirective: SpanDirectives.Directive = SpanDirectives.create("scaladoc") {
+ (attribute(0).as[String], cursor, source).mapN { (className, cursor, source) =>
+ cursor.config
+ .get[String]("scaladoc.baseUri")
+ .fold(
+ error => InvalidSpan(s"Invalid Scaladoc baseUri: $error", source),
+ baseUri => {
+ SpanLink.external(s"$baseUri/${className.replace(".", "/")}.html")(className)
+ })
+ }
+ }
+ val sourceCodeLinkDirective: SpanDirectives.Directive = SpanDirectives.create("srcLink") {
+ (attribute(0), cursor, source).mapN { (path, cursor, source) =>
+ cursor.config
+ .get[String]("github.baseUri")
+ .fold(
+ error => InvalidSpan(s"Invalid GitHub baseUri: $error", source),
+ baseUri => SpanLink.external(s"$baseUri/blob/main/$path")(""))
+ }
+ }
+ val spanDirectives = Seq(pydocDirective, scaladocDirective, sourceCodeLinkDirective)
+ val blockDirectives: Seq[BlockDirectives.Directive] = Seq()
+ val templateDirectives: Seq[TemplateDirectives.Directive] = Seq()
+ val linkDirectives: Seq[LinkDirectives.Directive] = Seq()
+}
diff --git a/project/LaikaCustoms.scala b/project/LaikaCustoms.scala
new file mode 100644
index 000000000..76a0b4614
--- /dev/null
+++ b/project/LaikaCustoms.scala
@@ -0,0 +1,260 @@
+import io.circe.Json
+import io.circe.parser.parse
+import laika.ast.*
+import laika.ast.Path.Root
+import laika.config.LaikaKeys
+import laika.config.LinkConfig
+import laika.config.SourceLinks
+import laika.helium.Helium
+import laika.helium.config.*
+import laika.sbt.LaikaConfig
+import laika.theme.ThemeProvider
+
+import java.nio.file.Files
+import java.nio.file.Path
+import java.nio.file.StandardCopyOption
+import scala.util.Try
+import scala.util.Using
+import scala.util.matching.Regex
+
+import scala.collection.JavaConverters.*
+
+object LaikaCustoms {
+ private val gfDescription: String = "Scalable Graph Processing on top of Apache Spark"
+ val thisVersionShortRegex: Regex = """^([0-9]+\.[0-9]+\.[0-9]+)(.*)$""".r
+
+ def generateAtomFeed(blogDir: Path, baseUrl: String): Unit = {
+ val posts = Files
+ .walk(blogDir)
+ .iterator()
+ .asScala
+ .filter(_.getFileName.toString.endsWith(".md"))
+ .filter(_.getFileName.toString != "01-index.md")
+ .toSeq
+
+ val titleRegex = """(?m)^-\s\*\*Title:\*\*\s(.+)$""".r
+ val publishedRegex = """(?m)^-\s\*\*Published:\*\*\s(.+)$""".r
+ val summaryRegex = """(?m)^-\s\*\*Summary:\*\*\s(.+)$""".r
+
+ val header =
+ """
+ |
+ |""".stripMargin
+
+ def genEntry(
+ title: String,
+ link: String,
+ updated: String,
+ id: String,
+ summary: String): String = {
+ s"""
+ |\t$title
+ |\t
+ |\t$id
+ |\t$updated
+ |\t$summary
+ |
+ |""".stripMargin
+ }
+
+ val feed =
+ posts.sortBy(_.getFileName.toString).foldLeft(new StringBuilder(header)) { (builder, post) =>
+ {
+ println(s"Generating feed entry for ${post.getFileName}")
+ val content = Using(scala.io.Source.fromFile(post.toFile)) { source => source.mkString }
+ .getOrElse("")
+
+ val id =
+ java.util.UUID.nameUUIDFromBytes(post.getFileName.toString.getBytes("UTF-8")).toString
+ val title = titleRegex.findAllMatchIn(content) match {
+ case titleMatch if titleMatch.hasNext => titleMatch.next().group(1)
+ case _ => "No title"
+ }
+ val link = s"$baseUrl/05-blog/${post.getFileName.toString.replaceAll("\\.md", "\\.html")}"
+ val updated = publishedRegex.findAllMatchIn(content) match {
+ case publishedMatch if publishedMatch.hasNext => publishedMatch.next().group(1)
+ case _ => "1970-01-01"
+ }
+ val summary = summaryRegex.findAllMatchIn(content) match {
+ case summaryMatch if summaryMatch.hasNext => summaryMatch.next().group(1)
+ case _ => "No summary"
+ }
+ builder.append(genEntry(title, link, updated, id, summary))
+ }
+ }
+
+ val feedContent = feed.append("").mkString
+ val feedFile = blogDir.resolve("feed.xml")
+ Files.write(feedFile, feedContent.getBytes)
+ }
+
+ def laikaConfig(benchmarksFile: Path): LaikaConfig = {
+ val baseConfig = LaikaConfig.defaults.withRawContent
+ .withConfigValue(LaikaKeys.site.apiPath, "api/scaladoc")
+ .withConfigValue("github.baseUri", "https://github.com/graphframes/graphframes")
+ .withConfigValue(
+ LinkConfig.empty.addSourceLinks(SourceLinks(
+ baseUri = "https://github.com/graphframes/graphframes/tree/master/core/src/main/scala/",
+ suffix = "scala").withPackagePrefix("org.graphframes")))
+
+ if (!Files.exists(benchmarksFile)) {
+ println(s"File $benchmarksFile does not exist. Skipping.")
+ return baseConfig
+ }
+ Using(scala.io.Source.fromFile(benchmarksFile.toFile)) { source =>
+ {
+ parse(source.mkString)
+ .getOrElse(Json.Null)
+ .asArray
+ .map(array =>
+ array.foldLeft(baseConfig) { (config, bench) =>
+ {
+ val name =
+ bench.hcursor.downField("benchmark").as[String].getOrElse("").split("\\.").last
+ val measurements =
+ bench.hcursor.downField("measurementIterations").as[Int].getOrElse(-1)
+ val metric = bench.hcursor
+ .downField("primaryMetric")
+ .downField("score")
+ .as[Double]
+ .getOrElse(0.0)
+ val stdErr = bench.hcursor
+ .downField("primaryMetric")
+ .downField("scoreError")
+ .as[Double]
+ .getOrElse(0.0)
+ val quantiles = bench.hcursor
+ .downField("primaryMetric")
+ .downField("scorePercentiles")
+ .as[Map[String, Double]]
+ .getOrElse(Map.empty)
+
+ val confidence = bench.hcursor
+ .downField("primaryMetric")
+ .downField("scoreConfidence")
+ .as[Array[Double]]
+ .getOrElse(Array.empty)
+
+ quantiles.foldLeft(
+ config
+ .withConfigValue(s"benchmarks.$name.metric", f"$metric%.4f")
+ .withConfigValue(s"benchmarks.$name.measurements", measurements)
+ .withConfigValue(
+ s"benchmarks.$name.ciLeft",
+ f"${Try(confidence(0)).getOrElse(0.0)}%.4f")
+ .withConfigValue(
+ s"benchmarks.$name.ciRight",
+ f"${Try(confidence(1)).getOrElse(0.0)}%.4f")
+ .withConfigValue(s"benchmarks.$name.stdErr", f"$stdErr%.4f")) {
+ (conf, quantile) =>
+ {
+ conf
+ .withConfigValue(
+ s"benchmarks.$name.quantiles.${quantile._1}",
+ f"${quantile._2}.4f")
+ }
+ }
+ }
+ })
+ }
+ }.toOption.flatten.getOrElse(baseConfig)
+ }
+
+ def heliumTheme: String => ThemeProvider = (v: String) => {
+ Helium.defaults.all
+ .metadata(
+ title = Some("GraphFrames"),
+ description = Some(gfDescription),
+ version = Some(v),
+ language = Some("en"))
+ .all
+ .tableOfContent("Table of Content", depth = 4)
+ .site
+ .topNavigationBar(navLinks = Seq(
+ IconLink.external("https://github.com/graphframes/graphframes", HeliumIcon.github),
+ IconLink.external(
+ "https://discord.com/channels/1162999022819225631/1326257052368113674",
+ HeliumIcon.chat)))
+ .site
+ .landingPage(
+ title = Some("GraphFrames"),
+ subtitle = Some("Distributed graph processing on top of Apache Spark"),
+ logo = Some(Image.internal(Root / "img" / "logo-dark.png")),
+ latestReleases = Seq(
+ ReleaseInfo(
+ "Latest Stable Release",
+ v match {
+ case thisVersionShortRegex(v, _) => v
+ case v => v
+ })),
+ license = Some("Apache-2.0"),
+ linkPanel = Some(
+ LinkPanel(
+ title = "Documentation",
+ TextLink.internal(Root / "01-about" / "01-index.md", "About GraphFrames"),
+ TextLink.internal(Root / "02-quick-start" / "01-installation.md", "Quick Start"),
+ TextLink.internal(Root / "03-tutorials" / "01-tutorials.md", "Tutorials"),
+ TextLink
+ .internal(Root / "04-user-guide" / "01-creating-graphframes.md", "User Guide"),
+ TextLink.internal(Root / "05-blog" / "01-index.md", "Blog"),
+ TextLink
+ .internal(Root / "06-contributing" / "01-contributing-guide.md", "Contributing"),
+ TextLink.internal(Root / "api" / "scaladoc" / "index.html", "API (Scaladoc)"),
+ TextLink.internal(Root / "api" / "python" / "index.html", "API (Python)"))),
+ projectLinks = Seq(
+ TextLink.external("https://github.com/graphframes/graphframes", "GitHub"),
+ TextLink.external(
+ "https://discord.com/channels/1162999022819225631/1326257052368113674",
+ "Discord")),
+ teasers = Seq(
+ Teaser(
+ "Scalable",
+ "Fully distributed algorithms. GraphFrames scales as well as your Spark Cluster is scaling"),
+ Teaser(
+ "Property Graphs Model",
+ "Support flexible definition of property graphs with groups and support graphs with multiple kind of vertices and edges"),
+ Teaser(
+ "Feature reach",
+ "A lot of ready to use graph algorithms, like PageRank and Label Propagation"),
+ Teaser(
+ "Mature",
+ "Project is old enough already and used in a lot of places as part of another libraries and platforms or as a tool for graph processing by end users"),
+ Teaser(
+ "PySpark support",
+ "Not only scala, but also Python with a full support of Spark Connect protocol")))
+ .build
+ }
+
+ def copyAll(targetDir: Path, sourceDir: Path): Unit = {
+ if (!Files.exists(sourceDir)) {
+ println(s"Directory $sourceDir does not exist. Skipping.")
+ return
+ }
+ val directoryConf = targetDir.resolve("directory.conf")
+
+ if (Files.exists(targetDir)) {
+ println(
+ s"Removing files in the existing documentation directory: ${targetDir.toAbsolutePath}")
+ Files
+ .walk(targetDir)
+ .sorted(java.util.Comparator.reverseOrder())
+ .filter(!_.equals(targetDir))
+ .filter(!_.equals(directoryConf))
+ .forEach(f => Files.delete(f))
+ }
+
+ println(
+ s"Copying files of the documentation from ${sourceDir.toAbsolutePath} to ${targetDir.toAbsolutePath}")
+ Files
+ .walk(sourceDir)
+ .forEach { source =>
+ val target = targetDir.resolve(sourceDir.relativize(source))
+ if (Files.isDirectory(source)) {
+ Files.createDirectories(target)
+ } else {
+ Files.createDirectories(target.getParent)
+ Files.copy(source, target, StandardCopyOption.REPLACE_EXISTING)
+ }
+ }
+ }
+}
diff --git a/project/plugins.sbt b/project/plugins.sbt
index 21a1c43a5..47334f8d6 100644
--- a/project/plugins.sbt
+++ b/project/plugins.sbt
@@ -5,9 +5,9 @@ ThisBuild / libraryDependencySchemes ++= Seq(
"org.scala-lang.modules" %% "scala-xml" % VersionScheme.Always
)
-addSbtPlugin("org.scoverage" % "sbt-scoverage" % "2.0.10")
+addSbtPlugin("org.scoverage" % "sbt-scoverage" % "2.3.1")
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "2.3.1")
-addSbtPlugin("org.scalameta" % "sbt-scalafmt" % "2.5.4")
+addSbtPlugin("org.scalameta" % "sbt-scalafmt" % "2.5.5")
// Protobuf things needed for the Spark Connect
addSbtPlugin("com.thesamet" % "sbt-protoc" % "1.0.7")
@@ -21,3 +21,19 @@ addSbtPlugin("com.github.sbt" % "sbt-ci-release" % "1.11.1")
// JMH & benchmarking
addSbtPlugin("pl.project13.scala" % "sbt-jmh" % "0.4.7")
+
+// Laika
+addSbtPlugin("org.typelevel" % "laika-sbt" % "1.3.2")
+addSbtPlugin("org.scalameta" % "sbt-mdoc" % "2.7.2")
+
+// Typelevel helper
+addSbtPlugin("org.typelevel" % "sbt-tpolecat" % "0.5.2")
+
+// JSONs processing for benchmarks
+val circeVersion = "0.13.0"
+
+libraryDependencies ++= Seq(
+ "io.circe" %% "circe-core",
+ "io.circe" %% "circe-generic",
+ "io.circe" %% "circe-parser"
+).map(_ % circeVersion)
\ No newline at end of file
diff --git a/python/graphframes/lib/pregel.py b/python/graphframes/lib/pregel.py
index 642b45fe6..11bf12e5e 100644
--- a/python/graphframes/lib/pregel.py
+++ b/python/graphframes/lib/pregel.py
@@ -30,7 +30,7 @@
class Pregel(JavaWrapper):
- r"""
+ """
Implements a Pregel-like bulk-synchronous message-passing API based on DataFrame operations.
See `Malewicz et al., Pregel: a system for large-scale graph processing
`_
- {{site.GRAPHFRAMES_VERSION}}
- 
+ 
+ 
+ 