®
+
®
[](https://gitter.im/locationtech/rasterframes?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
@@ -6,7 +6,7 @@ RasterFrames® brings together Earth-observation (EO) data access, cloud computi
RasterFrames provides a DataFrame-centric view over arbitrary raster data, enabling spatiotemporal queries, map algebra raster operations, and compatibility with the ecosystem of Spark ML algorithms. By using DataFrames as the core cognitive and compute data model, it is able to deliver these features in a form that is both accessible to general analysts and scalable along with the rapidly growing data footprint.
-
+
Please see the [Getting Started](http://rasterframes.io/getting-started.html) section of the Users' Manual to start using RasterFrames.
@@ -17,7 +17,6 @@ Please see the [Getting Started](http://rasterframes.io/getting-started.html) se
* [Gitter Channel](https://gitter.im/locationtech/rasterframes)
* [Submit an Issue](https://github.com/locationtech/rasterframes/issues)
-
## Contributing
Community contributions are always welcome. To get started, please review our [contribution guidelines](https://github.com/locationtech/rasterframes/blob/develop/CONTRIBUTING.md), [code of conduct](https://github.com/locationtech/rasterframes/blob/develop/CODE_OF_CONDUCT.md), and reach out to us on [gitter](https://gitter.im/locationtech/rasterframes) so the community can help you get started!
@@ -62,6 +61,11 @@ Additional, Python sepcific build instruction may be found at [pyrasterframes/sr
## Copyright and License
-RasterFrames is released under the Apache 2.0 License, copyright Astraea, Inc. 2017-2019.
+RasterFrames is released under the commercial-friendly Apache 2.0 License, copyright Astraea, Inc. 2017-2021.
+
+## Commercial Support
+
+As the sponsors and developers of RasterFrames, [Astraea, Inc.](https://astraea.earth/) is uniquely positioned to expand its capabilities. If you need additional functionality or just some architectural guidance to get your project off to the right start, we can provide a full range of [consulting and development services](https://astraea.earth/services/) around RasterFrames. We can be reached at [info@astraea.io](mailto:info@astraea.io).
+
diff --git a/RELEASE.md b/RELEASE.md
new file mode 100644
index 000000000..99337c8c8
--- /dev/null
+++ b/RELEASE.md
@@ -0,0 +1,23 @@
+# RasterFrames Release Process
+
+1. Make sure `release-notes.md` is updated.
+2. Use `git flow release start x.y.z` to create release branch.
+3. Manually edit `version.sbt` and `version.py` to set value to `x.y.z` and commit changes.
+4. Do `docker login` if necessary.
+5. `sbt` shell commands:
+ a. `clean`
+ b. `test it:test`
+ c. `makeSite`
+ d. `publishSigned` (LocationTech credentials required)
+ e. `sonatypeReleaseAll`. It can take a while, but should eventually show up [here](https://search.maven.org/search?q=g:org.locationtech.rasterframes).
+ f. `docs/ghpagesPushSite`
+ g. `rf-notebook/publish`
+6. `cd pyrasterframes/target/python/dist`
+7. `python3 -m twine upload pyrasterframes-x.y.z-py2.py3-none-any.whl`
+8. Commit any changes that were necessary.
+9. `git-flow finish release`. Make sure to push tags, develop and master
+ branches.
+10. On `develop`, update `version.sbt` and `version.py` to next development
+ version (`x.y.(z+1)-SNAPSHOT` and `x.y.(z+1).dev0`). Commit and push.
+11. In GitHub, create a new release with the created tag. Copy relevant
+ section of release notes into the description.
diff --git a/bench/build.sbt b/bench/build.sbt
index 36eb61323..e01ea663d 100644
--- a/bench/build.sbt
+++ b/bench/build.sbt
@@ -11,7 +11,7 @@ libraryDependencies ++= Seq(
jmhIterations := Some(5)
jmhWarmupIterations := Some(8)
jmhTimeUnit := None
-javaOptions in Jmh := Seq("-Xmx4g")
+Jmh / javaOptions := Seq("-Xmx4g")
// To enable profiling:
// jmhExtraOptions := Some("-prof jmh.extras.JFR")
diff --git a/bench/src/main/scala/org/locationtech/rasterframes/bench/CatalystSerializerBench.scala b/bench/src/main/scala/org/locationtech/rasterframes/bench/CatalystSerializerBench.scala
deleted file mode 100644
index 12a6b0486..000000000
--- a/bench/src/main/scala/org/locationtech/rasterframes/bench/CatalystSerializerBench.scala
+++ /dev/null
@@ -1,92 +0,0 @@
-/*
- * This software is licensed under the Apache 2 license, quoted below.
- *
- * Copyright 2019 Astraea, Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License"); you may not
- * use this file except in compliance with the License. You may obtain a copy of
- * the License at
- *
- * [http://www.apache.org/licenses/LICENSE-2.0]
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
- * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
- * License for the specific language governing permissions and limitations under
- * the License.
- *
- * SPDX-License-Identifier: Apache-2.0
- *
- */
-
-package org.locationtech.rasterframes.bench
-
-import java.util.concurrent.TimeUnit
-
-import geotrellis.proj4.{CRS, LatLng, Sinusoidal}
-import org.apache.spark.sql.Row
-import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
-import org.locationtech.rasterframes.encoders.{CatalystSerializer, StandardEncoders}
-import org.openjdk.jmh.annotations._
-
-@BenchmarkMode(Array(Mode.AverageTime))
-@State(Scope.Benchmark)
-@OutputTimeUnit(TimeUnit.MICROSECONDS)
-class CatalystSerializerBench extends SparkEnv {
-
- val serde = CatalystSerializer[CRS]
-
- val epsg: CRS = LatLng
- val epsgEnc: Row = serde.toRow(epsg)
- val proj4: CRS = Sinusoidal
- val proj4Enc: Row = serde.toRow(proj4)
-
- var crsEnc: ExpressionEncoder[CRS] = _
-
- @Setup(Level.Trial)
- def setupData(): Unit = {
- crsEnc = StandardEncoders.crsEncoder.resolveAndBind()
- }
-
- @Benchmark
- def encodeEpsg(): Row = {
- serde.toRow(epsg)
- }
-
- @Benchmark
- def encodeProj4(): Row = {
- serde.toRow(proj4)
- }
-
- @Benchmark
- def decodeEpsg(): CRS = {
- serde.fromRow(epsgEnc)
- }
-
- @Benchmark
- def decodeProj4(): CRS = {
- serde.fromRow(proj4Enc)
- }
-
- @Benchmark
- def exprEncodeEpsg(): InternalRow = {
- crsEnc.toRow(epsg)
- }
-
- @Benchmark
- def exprEncodeProj4(): InternalRow = {
- crsEnc.toRow(proj4)
- }
-
-// @Benchmark
-// def exprDecodeEpsg(): CRS = {
-//
-// }
-//
-// @Benchmark
-// def exprDecodeProj4(): CRS = {
-//
-// }
-
-}
diff --git a/bench/src/main/scala/org/locationtech/rasterframes/bench/CellTypeBench.scala b/bench/src/main/scala/org/locationtech/rasterframes/bench/CellTypeBench.scala
index dfc88f855..3a4d9f3f1 100644
--- a/bench/src/main/scala/org/locationtech/rasterframes/bench/CellTypeBench.scala
+++ b/bench/src/main/scala/org/locationtech/rasterframes/bench/CellTypeBench.scala
@@ -21,10 +21,9 @@
package org.locationtech.rasterframes.bench
import java.util.concurrent.TimeUnit
-
import geotrellis.raster.{CellType, DoubleUserDefinedNoDataCellType, IntUserDefinedNoDataCellType}
import org.apache.spark.sql.catalyst.InternalRow
-import org.locationtech.rasterframes.encoders.CatalystSerializer._
+import org.locationtech.rasterframes.encoders.StandardEncoders
import org.openjdk.jmh.annotations._
@BenchmarkMode(Array(Mode.AverageTime))
@@ -37,16 +36,12 @@ class CellTypeBench {
def setupData(): Unit = {
ct = IntUserDefinedNoDataCellType(scala.util.Random.nextInt())
val o: CellType = DoubleUserDefinedNoDataCellType(scala.util.Random.nextDouble())
- row = o.toInternalRow
+ row = StandardEncoders.cellTypeEncoder.createSerializer()(o)
}
@Benchmark
- def fromRow(): CellType = {
- row.to[CellType]
- }
+ def fromRow(): CellType = StandardEncoders.cellTypeEncoder.createDeserializer()(row)
@Benchmark
- def intoRow(): InternalRow = {
- ct.toInternalRow
- }
+ def intoRow(): InternalRow = StandardEncoders.cellTypeEncoder.createSerializer()(ct)
}
diff --git a/bench/src/main/scala/org/locationtech/rasterframes/bench/RasterRefBench.scala b/bench/src/main/scala/org/locationtech/rasterframes/bench/RasterRefBench.scala
index 448fab9c3..c7e36d985 100644
--- a/bench/src/main/scala/org/locationtech/rasterframes/bench/RasterRefBench.scala
+++ b/bench/src/main/scala/org/locationtech/rasterframes/bench/RasterRefBench.scala
@@ -28,8 +28,7 @@ import org.apache.spark.sql._
import org.locationtech.rasterframes._
import org.locationtech.rasterframes.expressions.generators.RasterSourceToRasterRefs
import org.locationtech.rasterframes.expressions.transformers.RasterRefToTile
-import org.locationtech.rasterframes.model.TileDimensions
-import org.locationtech.rasterframes.ref.RasterSource
+import org.locationtech.rasterframes.ref.RFRasterSource
import org.openjdk.jmh.annotations._
@BenchmarkMode(Array(Mode.AverageTime))
@@ -43,11 +42,11 @@ class RasterRefBench extends SparkEnv with LazyLogging {
@Setup(Level.Trial)
def setupData(): Unit = {
- val r1 = RasterSource(remoteCOGSingleband1)
- val r2 = RasterSource(remoteCOGSingleband2)
+ val r1 = RFRasterSource(remoteCOGSingleband1)
+ val r2 = RFRasterSource(remoteCOGSingleband2)
singleDF = Seq((r1, r2)).toDF("B1", "B2")
- .select(RasterRefToTile(RasterSourceToRasterRefs(Some(TileDimensions(r1.dimensions)), Seq(0), $"B1", $"B2")))
+ .select(RasterRefToTile(RasterSourceToRasterRefs(Some(r1.dimensions), Seq(0), $"B1", $"B2")))
expandedDF = Seq((r1, r2)).toDF("B1", "B2")
.select(RasterRefToTile(RasterSourceToRasterRefs($"B1", $"B2")))
diff --git a/bench/src/main/scala/org/locationtech/rasterframes/bench/TileCellScanBench.scala b/bench/src/main/scala/org/locationtech/rasterframes/bench/TileCellScanBench.scala
index 350ac811a..737e0c9b2 100644
--- a/bench/src/main/scala/org/locationtech/rasterframes/bench/TileCellScanBench.scala
+++ b/bench/src/main/scala/org/locationtech/rasterframes/bench/TileCellScanBench.scala
@@ -23,9 +23,9 @@ package org.locationtech.rasterframes.bench
import java.util.concurrent.TimeUnit
+import geotrellis.raster.Dimensions
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.rf.TileUDT
-import org.locationtech.rasterframes.tiles.InternalRowTile
import org.openjdk.jmh.annotations._
@BenchmarkMode(Array(Mode.AverageTime))
@@ -56,20 +56,9 @@ class TileCellScanBench extends SparkEnv {
@Benchmark
def deserializeRead(): Double = {
val tile = TileType.deserialize(tileRow)
- val (cols, rows) = tile.dimensions
- tile.getDouble(cols - 1, rows - 1) +
- tile.getDouble(cols/2, rows/2) +
- tile.getDouble(0, 0)
- }
-
- @Benchmark
- def internalRowRead(): Double = {
- val tile = new InternalRowTile(tileRow)
- val cols = tile.cols
- val rows = tile.rows
+ val Dimensions(cols, rows) = tile.dimensions
tile.getDouble(cols - 1, rows - 1) +
tile.getDouble(cols/2, rows/2) +
tile.getDouble(0, 0)
}
}
-
diff --git a/bench/src/main/scala/org/locationtech/rasterframes/bench/TileEncodeBench.scala b/bench/src/main/scala/org/locationtech/rasterframes/bench/TileEncodeBench.scala
index 20e255b06..d49027206 100644
--- a/bench/src/main/scala/org/locationtech/rasterframes/bench/TileEncodeBench.scala
+++ b/bench/src/main/scala/org/locationtech/rasterframes/bench/TileEncodeBench.scala
@@ -24,13 +24,11 @@ package org.locationtech.rasterframes.bench
import java.net.URI
import java.util.concurrent.TimeUnit
-import org.locationtech.rasterframes.ref.RasterRef.RasterRefTile
-import org.locationtech.rasterframes.ref.RasterRef
import geotrellis.raster.Tile
import geotrellis.vector.Extent
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
-import org.locationtech.rasterframes.ref.{RasterRef, RasterSource}
+import org.locationtech.rasterframes.ref.{RasterRef, RFRasterSource}
import org.openjdk.jmh.annotations._
@BenchmarkMode(Array(Mode.AverageTime))
@@ -53,24 +51,23 @@ class TileEncodeBench extends SparkEnv {
@Setup(Level.Trial)
def setupData(): Unit = {
cellTypeName match {
- case "rasterRef" ⇒
+ case "rasterRef" =>
val baseCOG = "https://s3-us-west-2.amazonaws.com/landsat-pds/c1/L8/149/039/LC08_L1TP_149039_20170411_20170415_01_T1/LC08_L1TP_149039_20170411_20170415_01_T1_B1.TIF"
val extent = Extent(253785.0, 3235185.0, 485115.0, 3471015.0)
- tile = RasterRefTile(RasterRef(RasterSource(URI.create(baseCOG)), 0, Some(extent), None))
- case _ ⇒
+ tile = RasterRef(RFRasterSource(URI.create(baseCOG)), 0, Some(extent), None)
+ case _ =>
tile = randomTile(tileSize, tileSize, cellTypeName)
}
}
@Benchmark
def encode(): InternalRow = {
- tileEncoder.toRow(tile)
+ tileEncoder.createSerializer.apply(tile)
}
@Benchmark
def roundTrip(): Tile = {
- val row = tileEncoder.toRow(tile)
- boundEncoder.fromRow(row)
+ val row = tileEncoder.createSerializer().apply(tile)
+ boundEncoder.createDeserializer().apply(row)
}
}
-
diff --git a/bench/src/main/scala/org/locationtech/rasterframes/bench/package.scala b/bench/src/main/scala/org/locationtech/rasterframes/bench/package.scala
index 65d8ab88f..8296cad37 100644
--- a/bench/src/main/scala/org/locationtech/rasterframes/bench/package.scala
+++ b/bench/src/main/scala/org/locationtech/rasterframes/bench/package.scala
@@ -37,10 +37,10 @@ package object bench {
val cellType = CellType.fromName(cellTypeName)
val tile = ArrayTile.alloc(cellType, cols, rows)
if(cellType.isFloatingPoint) {
- tile.mapDouble(_ ⇒ rnd.nextGaussian())
+ tile.mapDouble(_ => rnd.nextGaussian())
}
else {
- tile.map(_ ⇒ {
+ tile.map(_ => {
var c = NODATA
do {
c = rnd.nextInt(255)
diff --git a/build.sbt b/build.sbt
index f941ea060..52c544e8a 100644
--- a/build.sbt
+++ b/build.sbt
@@ -19,6 +19,15 @@
*
*/
+// Leave me and my custom keys alone!
+Global / lintUnusedKeysOnLoad := false
+ThisBuild / versionScheme := Some("semver-spec")
+ThisBuild / dynverVTagPrefix := false
+ThisBuild / dynverSonatypeSnapshots := true
+ThisBuild / publishMavenStyle := true
+ThisBuild / Test / publishArtifact := false
+
+
addCommandAlias("makeSite", "docs/makeSite")
addCommandAlias("previewSite", "docs/previewSite")
addCommandAlias("ghpagesPushSite", "docs/ghpagesPushSite")
@@ -28,19 +37,17 @@ addCommandAlias("console", "datasource/console")
lazy val IntegrationTest = config("it") extend Test
lazy val root = project
- .in(file("."))
.withId("RasterFrames")
- .aggregate(core, datasource, pyrasterframes, experimental)
- .enablePlugins(RFReleasePlugin)
+ .aggregate(core, datasource)
.settings(
- publish / skip := true,
- clean := clean.dependsOn(`rf-notebook`/clean).value
- )
+ publish / skip := true)
lazy val `rf-notebook` = project
.dependsOn(pyrasterframes)
+ .disablePlugins(CiReleasePlugin)
.enablePlugins(RFAssemblyPlugin, DockerPlugin)
- .settings(publish / skip := true)
+ .settings(
+ publish / skip := true)
lazy val core = project
.enablePlugins(BuildInfoPlugin)
@@ -50,43 +57,61 @@ lazy val core = project
.settings(
moduleName := "rasterframes",
libraryDependencies ++= Seq(
+ `slf4j-api`,
shapeless,
+ circe("core").value,
+ circe("generic").value,
+ circe("parser").value,
+ circe("generic-extras").value,
+ frameless excludeAll ExclusionRule(organization = "com.github.mpilquist"),
`jts-core`,
+ `spray-json`,
geomesa("z3").value,
geomesa("spark-jts").value,
- `geotrellis-contrib-vlm`,
- `geotrellis-contrib-gdal`,
spark("core").value % Provided,
spark("mllib").value % Provided,
spark("sql").value % Provided,
- geotrellis("spark").value,
- geotrellis("raster").value,
- geotrellis("s3").value,
+ // TODO: scala-uri brings an outdated simulacrum dep
+ // Fix it in GT
+ geotrellis("spark").value excludeAll ExclusionRule(organization = "com.github.mpilquist"),
+ geotrellis("raster").value excludeAll ExclusionRule(organization = "com.github.mpilquist"),
+ geotrellis("s3").value excludeAll ExclusionRule(organization = "com.github.mpilquist"),
geotrellis("spark-testkit").value % Test excludeAll (
ExclusionRule(organization = "org.scalastic"),
- ExclusionRule(organization = "org.scalatest")
+ ExclusionRule(organization = "org.scalatest"),
+ ExclusionRule(organization = "com.github.mpilquist")
),
scaffeine,
- scalatest
+ sparktestingbase().value % Test excludeAll ExclusionRule("org.scala-lang.modules", "scala-xml_2.12"),
+ `scala-logging`
),
+ libraryDependencies ++= {
+ val gv = rfGeoTrellisVersion.value
+ if (gv.startsWith("3")) Seq[ModuleID](
+ geotrellis("gdal").value excludeAll ExclusionRule(organization = "com.github.mpilquist"),
+ geotrellis("s3-spark").value excludeAll ExclusionRule(organization = "com.github.mpilquist")
+ )
+ else Seq.empty[ModuleID]
+ },
buildInfoKeys ++= Seq[BuildInfoKey](
- moduleName, version, scalaVersion, sbtVersion, rfGeoTrellisVersion, rfGeoMesaVersion, rfSparkVersion
+ version, scalaVersion, rfGeoTrellisVersion, rfGeoMesaVersion, rfSparkVersion
),
buildInfoPackage := "org.locationtech.rasterframes",
buildInfoObject := "RFBuildInfo",
buildInfoOptions := Seq(
BuildInfoOption.ToMap,
- BuildInfoOption.BuildTime,
BuildInfoOption.ToJson
)
)
lazy val pyrasterframes = project
- .dependsOn(core, datasource, experimental)
+ .dependsOn(core, datasource)
+ .disablePlugins(CiReleasePlugin)
.enablePlugins(RFAssemblyPlugin, PythonBuildPlugin)
.settings(
+ publish / skip := true,
libraryDependencies ++= Seq(
- geotrellis("s3").value,
+ geotrellis("s3").value excludeAll ExclusionRule(organization = "com.github.mpilquist"),
spark("core").value % Provided,
spark("mllib").value % Provided,
spark("sql").value % Provided
@@ -100,16 +125,26 @@ lazy val datasource = project
.settings(
moduleName := "rasterframes-datasource",
libraryDependencies ++= Seq(
- geotrellis("s3").value,
+ compilerPlugin("org.scalamacros" % "paradise" % "2.1.1" cross CrossVersion.full),
+ compilerPlugin("org.typelevel" % "kind-projector" % "0.13.2" cross CrossVersion.full),
+ sttpCatsCe2,
+ stac4s,
+ framelessRefined excludeAll ExclusionRule(organization = "com.github.mpilquist"),
+ geotrellis("s3").value excludeAll ExclusionRule(organization = "com.github.mpilquist"),
spark("core").value % Provided,
spark("mllib").value % Provided,
- spark("sql").value % Provided
+ spark("sql").value % Provided,
+ `better-files`
),
- initialCommands in console := (initialCommands in console).value +
+ Compile / console / scalacOptions ~= { _.filterNot(Set("-Ywarn-unused-import", "-Ywarn-unused:imports")) },
+ Test / console / scalacOptions ~= { _.filterNot(Set("-Ywarn-unused-import", "-Ywarn-unused:imports")) },
+ console / initialCommands := (console / initialCommands).value +
"""
|import org.locationtech.rasterframes.datasource.geotrellis._
|import org.locationtech.rasterframes.datasource.geotiff._
- |""".stripMargin
+ |""".stripMargin,
+ IntegrationTest / fork := true,
+ IntegrationTest / javaOptions := Seq("-Xmx3g -XX:+UseG1GC")
)
lazy val experimental = project
@@ -120,35 +155,36 @@ lazy val experimental = project
.settings(
moduleName := "rasterframes-experimental",
libraryDependencies ++= Seq(
- geotrellis("s3").value,
+ geotrellis("s3").value excludeAll ExclusionRule(organization = "com.github.mpilquist"),
spark("core").value % Provided,
spark("mllib").value % Provided,
spark("sql").value % Provided
),
- fork in IntegrationTest := true,
- javaOptions in IntegrationTest := Seq("-Xmx2G"),
- parallelExecution in IntegrationTest := false
+ IntegrationTest / fork := true,
+ IntegrationTest / javaOptions := (datasource / IntegrationTest / javaOptions).value
)
lazy val docs = project
.dependsOn(core, datasource, pyrasterframes)
+ .disablePlugins(CiReleasePlugin)
.enablePlugins(SiteScaladocPlugin, ParadoxPlugin, ParadoxMaterialThemePlugin, GhpagesPlugin, ScalaUnidocPlugin)
.settings(
- apiURL := Some(url("http://rasterframes.io/latest/api")),
+ publish / skip := true,
+ apiURL := Some(url("https://rasterframes.io/latest/api")),
autoAPIMappings := true,
ghpagesNoJekyll := true,
ScalaUnidoc / siteSubdirName := "latest/api",
paradox / siteSubdirName := ".",
paradoxProperties ++= Map(
"version" -> version.value,
- "scaladoc.org.apache.spark.sql.rf" -> "http://rasterframes.io/latest",
+ "scaladoc.org.apache.spark.sql.rf" -> "https://rasterframes.io/latest",
"github.base_url" -> ""
),
paradoxNavigationExpandDepth := Some(3),
Compile / paradoxMaterialTheme ~= { _
.withRepository(uri("https://github.com/locationtech/rasterframes"))
.withCustomStylesheet("assets/custom.css")
- .withCopyright("""© 2017-2019 + +## Related Links + +* [Gitter Channel](https://gitter.im/locationtech/rasterframes) +* [Scala API Documentation](latest/api/index.html) +* [GitHub Repository](https://github.com/locationtech/rasterframes) +* [Astraea, Inc.](http://www.astraea.earth/), the company behind RasterFrames + +## Detailed Contents + +@@ toc { depth=3 } + +@@@ index +* @ref:[Overview](description.md) +* @ref:[Getting Started](getting-started.md) +* @ref:[Concepts](concepts.md) +* @ref:[Raster Data I/O](raster-io.md) +* @ref:[Vector Data](vector-data.md) +* @ref:[Raster Processing](raster-processing.md) +* @ref:[Machine Learning](machine-learning.md) +* @ref:[Numpy and Pandas](numpy-pandas.md) +* @ref:[IPython Extensions](ipython.md) +* @ref:[Scala and SQL](languages.md) +* @ref:[Function Reference](reference.md) +* @ref:[Release Notes](release-notes.md) +@@@ + diff --git a/pyrasterframes/src/main/python/docs/machine-learning.md b/docs/src/main/paradox/machine-learning.md similarity index 100% rename from pyrasterframes/src/main/python/docs/machine-learning.md rename to docs/src/main/paradox/machine-learning.md diff --git a/pyrasterframes/src/main/python/docs/raster-io.md b/docs/src/main/paradox/raster-io.md similarity index 100% rename from pyrasterframes/src/main/python/docs/raster-io.md rename to docs/src/main/paradox/raster-io.md diff --git a/pyrasterframes/src/main/python/docs/raster-processing.md b/docs/src/main/paradox/raster-processing.md similarity index 85% rename from pyrasterframes/src/main/python/docs/raster-processing.md rename to docs/src/main/paradox/raster-processing.md index fc6353e37..31fbbd105 100644 --- a/pyrasterframes/src/main/python/docs/raster-processing.md +++ b/docs/src/main/paradox/raster-processing.md @@ -4,10 +4,11 @@ * @ref:[Local Map Algebra](local-algebra.md) * @ref:["NoData" Handling](nodata-handling.md) +* @ref:[Masking](masking.md) * @ref:[Zonal Map Algebra](zonal-algebra.md) * @ref:[Aggregation](aggregation.md) * @ref:[Time Series](time-series.md) -* @ref:[Machine Learning](machine-learning.md) +* @ref:[Raster Join](raster-join.md) @@@ diff --git a/pyrasterframes/src/main/python/docs/reference.pymd b/docs/src/main/paradox/reference.md similarity index 65% rename from pyrasterframes/src/main/python/docs/reference.pymd rename to docs/src/main/paradox/reference.md index 195b7e5e0..545455161 100644 --- a/pyrasterframes/src/main/python/docs/reference.pymd +++ b/docs/src/main/paradox/reference.md @@ -66,6 +66,23 @@ See also GeoMesa [st_envelope](https://www.geomesa.org/documentation/user/spark/ Convert an extent to a Geometry. The extent likely comes from @ref:[`st_extent`](reference.md#st-extent) or @ref:[`rf_extent`](reference.md#rf-extent). + +### rf_xz2_index + + Long rf_xz2_index(Geometry geom, CRS crs) + Long rf_xz2_index(Extent extent, CRS crs) + Long rf_xz2_index(ProjectedRasterTile proj_raster) + +Constructs a XZ2 index in WGS84/EPSG:4326 from either a Geometry, Extent, ProjectedRasterTile and its CRS. This function is useful for [range partitioning](http://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=registerjava#pyspark.sql.DataFrame.repartitionByRange). + +### rf_z2_index + + Long rf_z2_index(Geometry geom, CRS crs) + Long rf_z2_index(Extent extent, CRS crs) + Long rf_z2_index(ProjectedRasterTile proj_raster) + +Constructs a Z2 index in WGS84/EPSG:4326 from either a Geometry, Extent, ProjectedRasterTile and its CRS. First the native extent is extracted or computed, and then center is used as the indexing location. This function is useful for [range partitioning](http://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=registerjava#pyspark.sql.DataFrame.repartitionByRange). See @ref:[Reading Raster Data](raster-read.md#spatial-indexing-and-partitioning) section for details on how to have an index automatically added when reading raster data. + ## Tile Metadata and Mutation Functions to access and change the particulars of a `tile`: its shape and the data type of its cells. See section on @ref:["NoData" handling](nodata-handling.md) for additional discussion of cell types. @@ -103,6 +120,12 @@ Fetches the extent (bounding box or envelope) of a `ProjectedRasterTile` or `Ras Fetch CRS structure representing the coordinate reference system of a `ProjectedRasterTile` or `RasterSource` type tile columns, or from a column of strings in the form supported by @ref:[`rf_mk_crs`](reference.md#rf-mk-crs). +### rf_proj_raster + + ProjectedRasterTile rf_proj_raster(Tile tile, Extent extent, CRS crs) + +Construct a `proj_raster` structure from individual Tile, Extent, and CRS columns. + ### rf_mk_crs Struct rf_mk_crs(String crsText) @@ -131,12 +154,30 @@ Change the interpretation of the `tile_col`'s cell values according to specified ### rf_resample - Tile rf_resample(Tile tile, Double factor) - Tile rf_resample(Tile tile, Int factor) - Tile rf_resample(Tile tile, Tile shape_tile) + Tile rf_resample(Tile tile, Double factor, [String method]) + Tile rf_resample(Tile tile, Int factor, [String method]) + Tile rf_resample(Tile tile, Tile shape_tile, [String method]) + +In __SQL__, three parameters are required for `rf_resample`.: + + Tile rf_resample(Tile tile, Double factor, String method) + Tile rf_resample(Tile tile, Int factor, String method) + Tile rf_resample(Tile tile, Tile shape_tile, String method) + Tile rf_resample_nearest(Tile tile, Double factor) + Tile rf_resample_nearest(Tile tile, Int factor) + Tile rf_resample_nearest(Tile tile, Tile shape_tile) -Change the tile dimension. Passing a numeric `factor` will scale the number of columns and rows in the tile: 1.0 is the same number of columns and row; less than one downsamples the tile; and greater than one upsamples the tile. Passing a `shape_tile` as the second argument outputs `tile` having the same number of columns and rows as `shape_tile`. All resampling is by nearest neighbor method. +Change the tile dimension by upsampling or downsampling. Passing a numeric `factor` will scale the number of columns and rows in the tile: 1.0 is the same number of columns and row; less than one downsamples the tile; and greater than one upsamples the tile. Passing a tile as the second argument resamples such that the output has the same dimension (number of columns and rows) as `shape_tile`. + +There are two categories: point resampling methods and aggregating resampling methods. +Resampling method to use can be specified by one of the following strings, possibly in a column. +The point resampling methods are: `"nearest_neighbor"`, `"bilinear"`, `"cubic_convolution"`, `"cubic_spline"`, and `"lanczos"`. +The aggregating resampling methods are: `"average"`, `"mode"`, `"median"`, `"max"`, "`min`", or `"sum"`. + +Note the aggregating methods are intended for downsampling. For example a 0.25 factor and `max` method returns the maximum value in a 4x4 neighborhood. + +If `tile` has an integer `CellType`, the returned tile will be coerced to a floating point with the following methods: bilinear, cubic_convolution, cubic_spline, lanczos, average, and median. ## Tile Creation @@ -158,7 +199,6 @@ Tile rf_make_ones_tile(Int tile_columns, Int tile_rows, [CellType cell_type]) Tile rf_make_ones_tile(Int tile_columns, Int tile_rows, [String cell_type_name]) ``` - Create a `tile` of shape `tile_columns` by `tile_rows` full of ones, with the optional cell type; default is float64. See @ref:[this discussion](nodata-handling.md#cell-types) on cell types for info on the `cell_type` argument. All arguments are literal values and not column expressions. ### rf_make_constant_tile @@ -166,7 +206,6 @@ Create a `tile` of shape `tile_columns` by `tile_rows` full of ones, with the op Tile rf_make_constant_tile(Numeric constant, Int tile_columns, Int tile_rows, [CellType cell_type]) Tile rf_make_constant_tile(Numeric constant, Int tile_columns, Int tile_rows, [String cell_type_name]) - Create a `tile` of shape `tile_columns` by `tile_rows` full of `constant`, with the optional cell type; default is float64. See @ref:[this discussion](nodata-handling.md#cell-types) on cell types for info on the `cell_type` argument. All arguments are literal values and not column expressions. @@ -174,7 +213,6 @@ Create a `tile` of shape `tile_columns` by `tile_rows` full of `constant`, with Tile rf_rasterize(Geometry geom, Geometry tile_bounds, Int value, Int tile_columns, Int tile_rows) - Convert a vector Geometry `geom` into a Tile representation. The `value` will be "burned-in" to the returned `tile` where the `geom` intersects the `tile_bounds`. Returned `tile` will have shape `tile_columns` by `tile_rows`. Values outside the `geom` will be assigned a NoData value. Returned `tile` has cell type `int32`, note that `value` is of type Int. Parameters `tile_columns` and `tile_rows` are literals, not column expressions. The others are column expressions. @@ -183,7 +221,7 @@ Parameters `tile_columns` and `tile_rows` are literals, not column expressions. Tile rf_array_to_tile(Array arrayCol, Int numCols, Int numRows) -Python only. Create a `tile` from a Spark SQL [Array](http://spark.apache.org/docs/2.3.2/api/python/pyspark.sql.html#pyspark.sql.types.ArrayType), filling values in row-major order. +Python only. Create a `tile` from a Spark SQL [Array][Array], filling values in row-major order. ### rf_assemble_tile @@ -198,7 +236,7 @@ SQL implementation does not accept a cell_type argument. It returns a float64 ce ## Masking and NoData -See @ref:[NoData handling](nodata-handling.md) for conceptual discussion of cell types and NoData. +See the @ref:[masking](masking.md) page for conceptual discussion of masking operations. There are statistical functions of the count of data and NoData values per `tile` and aggregate over a `tile` column: @ref:[`rf_data_cells`](reference.md#rf-data-cells), @ref:[`rf_no_data_cells`](reference.md#rf-no-data-cells), @ref:[`rf_agg_data_cells`](reference.md#rf-agg-data-cells), and @ref:[`rf_agg_no_data_cells`](reference.md#rf-agg-no-data-cells). @@ -206,14 +244,55 @@ Masking is a raster operation that sets specific cells to NoData based on the va ### rf_mask - Tile rf_mask(Tile tile, Tile mask) + Tile rf_mask(Tile tile, Tile mask, bool inverse) Where the `mask` contains NoData, replace values in the `tile` with NoData. Returned `tile` cell type will be coerced to one supporting NoData if it does not already. +`inverse` is a literal not a Column. If `inverse` is true, return the `tile` with NoData in locations where the `mask` _does not_ contain NoData. Equivalent to @ref:[`rf_inverse_mask`](reference.md#rf-inverse-mask). + See also @ref:[`rf_rasterize`](reference.md#rf-rasterize). +### rf_mask_by_value + + Tile rf_mask_by_value(Tile data_tile, Tile mask_tile, Int mask_value, bool inverse) + +Generate a `tile` with the values from `data_tile`, with NoData in cells where the `mask_tile` is equal to `mask_value`. + +`inverse` is a literal not a Column. If `inverse` is true, return the `data_tile` with NoData in locations where the `mask_tile` value is _not equal_ to `mask_value`. Equivalent to @ref:[`rf_inverse_mask_by_value`](reference.md#rf-inverse-mask-by-value). + +### rf_mask_by_values + + Tile rf_mask_by_values(Tile data_tile, Tile mask_tile, Array mask_values) + Tile rf_mask_by_values(Tile data_tile, Tile mask_tile, list mask_values) + +Generate a `tile` with the values from `data_tile`, with NoData in cells where the `mask_tile` is in the `mask_values` Array or list. `mask_values` can be a [`pyspark.sql.ArrayType`][Array] or a `list`. + +### rf_mask_by_bit + + Tile rf_mask_by_bits(Tile tile, Tile mask_tile, Int bit_position, Bool mask_value) + +Applies a mask using bit values in the `mask_tile`. Working from the right, the bit at `bit_position` is @ref:[extracted](reference.md#rf_local_extract_bits) from cell values of the `mask_tile`. In all locations where these are equal to the `mask_value`, the returned tile is set to NoData; otherwise the original `tile` cell value is returned. + +This is a single-bit version of @ref:[`rf_mask_by_bits`](reference.md#rf-mask-by-bits). + +### rf_mask_by_bits + + Tile rf_mask_by_bits(Tile tile, Tile mask_tile, Int start_bit, Int num_bits, Array mask_values) + Tile rf_mask_by_bits(Tile tile, Tile mask_tile, Int start_bit, Int num_bits, list mask_values) + +Applies a mask from blacklisted bit values in the `mask_tile`. Working from the right, the bits from `start_bit` to `start_bit + num_bits` are @ref:[extracted](reference.md#rf_local_extract_bits) from cell values of the `mask_tile`. In all locations where these are in the `mask_values`, the returned tile is set to NoData; otherwise the original `tile` cell value is returned. + +This function is not available in the SQL API. The below is equivalent: + +```sql +SELECT rf_mask_by_values( + tile, + rf_local_extract_bits(mask_tile, start_bit, num_bits), + mask_values + ), +``` ### rf_inverse_mask @@ -221,12 +300,12 @@ See also @ref:[`rf_rasterize`](reference.md#rf-rasterize). Where the `mask` _does not_ contain NoData, replace values in `tile` with NoData. -### rf_mask_by_value - Tile rf_mask_by_value(Tile data_tile, Tile mask_tile, Int mask_value) +### rf_inverse_mask_by_value -Generate a `tile` with the values from `data_tile`, with NoData in cells where the `mask_tile` is equal to `mask_value`. + Tile rf_inverse_mask_by_value(Tile data_tile, Tile mask_tile, Int mask_value) +Generate a `tile` with the values from `data_tile`, with NoData in cells where the `mask_tile` is not equal to `mask_value`. In other words, only keep `data_tile` cells in locations where the `mask_tile` is equal to `mask_value`. ### rf_is_no_data_tile @@ -374,6 +453,70 @@ Returns a `tile` column containing the element-wise equality of `tile1` and `rhs Returns a `tile` column containing the element-wise inequality of `tile1` and `rhs`. +### rf_local_is_in + + Tile rf_local_is_in(Tile tile, Array array) + Tile rf_local_is_in(Tile tile, list l) + +Returns a `tile` column with cell values of 1 where the `tile` cell value is in the provided array or list. The `array` is a Spark SQL [Array][Array]. A python `list` of numeric values can also be passed. + +### rf_local_extract_bits + + Tile rf_local_extract_bits(Tile tile, Int start_bit, Int num_bits) + Tile rf_local_extract_bits(Tile tile, Int start_bit) + +Extract value from specified bits of the cells' underlying binary data. Working from the right, the bits from `start_bit` to `start_bit + num_bits` are extracted from cell values of the `tile`. The `start_bit` is zero indexed. If `num_bits` is not provided, a single bit is extracted. + +A common use case for this function is covered by @ref:[`rf_mask_by_bits`](reference.md#rf-mask-by-bits). + + +### rf_local_min + + Tile rf_local_min(Tile tile, Tile max) + Tile rf_local_min(Tile tile, Numeric max) + +Performs cell-wise minimum two tiles or a tile and a scalar. + +### rf_local_max + + Tile rf_local_max(Tile tile, Tile max) + Tile rf_local_max(Tile tile, Numeric max) + +Performs cell-wise maximum two tiles or a tile and a scalar. + +### rf_local_clamp + + Tile rf_local_clamp(Tile tile, Tile min, Tile max) + Tile rf_local_clamp(Tile tile, Numeric min, Tile max) + Tile rf_local_clamp(Tile tile, Tile min, Numeric max) + Tile rf_local_clamp(Tile tile, Numeric min, Numeric max) + +Return the tile with its values limited to a range defined by min and max, inclusive. + +### rf_where + + Tile rf_where(Tile condition, Tile x, Tile y) + +Return a tile with cell values chosen from `x` or `y` depending on `condition`. +Operates cell-wise in a similar fashion to Spark SQL `when` and `otherwise`. + +### rf_rescale + + Tile rf_rescale(Tile tile) + Tile rf_rescale(Tile tile, Double min, Double max) + +Rescale cell values such that the minimum is zero and the maximum is one. Other values will be linearly interpolated into the range. +If specified, the `min` parameter will become the zero value and the `max` parameter will become 1. See @ref:[`rf_agg_stats`](reference.md#rf_agg_stats). +Values outside the range will be set to 0 or 1. +If `min` and `max` are not specified, the __tile-wise__ minimum and maximum are used; this can result in inconsistent values across rows in a tile column. + +### rf_standardize + + rf_standardize(Tile tile) + rf_standardize(Tile tile, Double mean, Double stddev) + +Standardize cell values such that the mean is zero and the standard deviation is one. If specified, the `mean` and `stddev` are applied to all tiles in the column. See @ref:[`rf_agg_stats`](reference.md#rf_agg_stats). If not specified, each tile will be standardized according to the statistics of its cell values; this can result in inconsistent values across rows in a tile column. + ### rf_round Tile rf_round(Tile tile) @@ -434,6 +577,13 @@ Performs cell-wise logarithm with base 2. Performs natural logarithm of cell values plus one. Inverse of @ref:[`rf_expm1`](reference.md#rf-expm1). + +### rf_sqrt + + Tile rf_sqrt(Tile tile) + +Perform cell-wise square root. + ## Tile Statistics The following functions compute a statistical summary per row of a `tile` column. The statistics are computed across the cells of a single `tile`, within each DataFrame Row. @@ -453,7 +603,7 @@ Computes the sum of cells in each row of column `tile`, ignoring NoData values. Computes the mean of cells in each row of column `tile`, ignoring NoData values. -### rf_tile_min +### rf_tile_min Double rf_tile_min(Tile tile) @@ -531,7 +681,7 @@ Aggregates over the `tile` and return the mean of cell values, ignoring NoData. Long rf_agg_data_cells(Tile tile) -_SQL_: @ref:[`rf_agg_stats`](reference.md#rf-agg-stats)`(tile).dataCells` +_SQL_: @ref:[`rf_agg_stats`](reference.md#rf-agg-stats)`(tile).data_cells` Aggregates over the `tile` and return the count of data cells. Equivalent to @ref:[`rf_agg_stats`](reference.md#rf-agg-stats)`.dataCells`. @@ -539,7 +689,7 @@ Aggregates over the `tile` and return the count of data cells. Equivalent to @re Long rf_agg_no_data_cells(Tile tile) -_SQL_: @ref:[`rf_agg_stats`](reference.md#rf-agg-stats)`(tile).dataCells` +_SQL_: @ref:[`rf_agg_stats`](reference.md#rf-agg-stats)`(tile).no_data_cells` Aggregates over the `tile` and return the count of NoData cells. Equivalent to @ref:[`rf_agg_stats`](reference.md#rf-agg-stats)`.noDataCells`. C.F. @ref:[`rf_no_data_cells`](reference.md#rf-no-data-cells) a row-wise count of no data cells. @@ -557,6 +707,26 @@ Aggregates over the `tile` and returns statistical summaries of cell values: num Aggregates over all of the rows in DataFrame of `tile` and returns a count of each cell value to create a histogram with values are plotted on the x-axis and counts on the y-axis. Related is the @ref:[`rf_tile_histogram`](reference.md#rf-tile-histogram) function which operates on a single row at a time. +### rf_agg_approx_quantiles + + Array[Double] rf_agg_approx_quantiles(Tile tile, List[float] probabilities, float relative_error) + +__Not supported in SQL.__ + +Calculates the approximate quantiles of a tile column of a DataFrame. `probabilities` is a list of float values at which to compute the quantiles. These must belong to [0, 1]. For example 0 is the minimum, 0.5 is the median, 1 is the maximum. Returns an array of values approximately at the specified `probabilities`. + +### rf_agg_extent + + Extent rf_agg_extent(Extent extent) + +Compute the naive aggregate extent over a column. Assumes CRS homogeneity. With mixed CRS in the column, or if you are unsure, use @ref:[`rf_agg_reprojected_extent`](reference.md#rf-agg-reprojected-extent). + + +### rf_agg_reprojected_extent + + Extent rf_agg_reprojected_extent(Extent extent, CRS source_crs, String dest_crs) + +Compute the aggregate extent over the `extent` and `source_crs` columns. The `dest_crs` is given as a string. Each row's extent will be reprojected to the `dest_crs` before aggregating. ## Tile Local Aggregate Statistics @@ -621,13 +791,13 @@ Python only. As with @ref:[`rf_explode_tiles`](reference.md#rf-explode-tiles), b Array rf_tile_to_array_int(Tile tile) -Convert Tile column to Spark SQL [Array](http://spark.apache.org/docs/2.3.2/api/python/pyspark.sql.html#pyspark.sql.types.ArrayType), in row-major order. Float cell types will be coerced to integral type by flooring. +Convert Tile column to Spark SQL [Array][Array], in row-major order. Float cell types will be coerced to integral type by flooring. ### rf_tile_to_array_double Array rf_tile_to_arry_double(Tile tile) -Convert tile column to Spark [Array](http://spark.apache.org/docs/2.3.2/api/python/pyspark.sql.html#pyspark.sql.types.ArrayType), in row-major order. Integral cell types will be coerced to floats. +Convert tile column to Spark [Array][Array], in row-major order. Integral cell types will be coerced to floats. ### rf_render_ascii @@ -639,21 +809,58 @@ Pretty print the tile values as plain text. String rf_render_matrix(Tile tile) -Render Tile cell values as numeric values, for debugging purposes. +Render Tile cell values as a string of numeric values, for debugging purposes. +### rf_render_png -### rf_rgb_composite + Array rf_render_png(Tile red, Tile green, Tile blue) + +Converts three tile columns to a three-channel PNG-encoded image `bytearray`. First evaluates [`rf_rgb_composite`](reference.md#rf-rgb-composite) on the given tile columns, and then encodes the result. For more about rendering these in a Jupyter or IPython environment, see @[Writing Raster Data](raster-write.md#rendering-samples-with-color). - Tile rf_rgb_composite(Tile red, Tile green, Tile blue) +### rf_render_color_ramp_png + + Array rf_render_png(Tile tile, String color_ramp_name) -Merges three bands into a single byte-packed RGB composite. It first scales each cell to fit into an unsigned byte, in the range 0-255, and then merges all three channels to fit into a 32-bit unsigned integer. This is useful when you want an RGB tile to render or to process with other color imagery tools. +Converts given tile into a PNG image, using a color ramp of the given name to convert cells into pixels. `color_ramp_name` can be one of the following: + + * "BlueToOrange" + * "LightYellowToOrange" + * "BlueToRed" + * "GreenToRedOrange" + * "LightToDarkSunset" + * "LightToDarkGreen" + * "HeatmapYellowToRed" + * "HeatmapBlueToYellowToRedSpectrum" + * "HeatmapDarkRedToYellowWhite" + * "HeatmapLightPurpleToDarkPurpleToWhite" + * "ClassificationBoldLandUse" + * "ClassificationMutedTerrain" + * "Magma" + * "Inferno" + * "Plasma" + * "Viridis" + * "Greyscale2" + * "Greyscale8" + * "Greyscale32" + * "Greyscale64" + * "Greyscale128" + * "Greyscale256" + +Further descriptions of these color ramps can be found in the [Geotrellis Documentation](https://geotrellis.readthedocs.io/en/latest/guide/rasters.html#built-in-color-ramps). For more about rendering these in a Jupyter or IPython environment, see @[Writing Raster Data](raster-write.md#rendering-samples-with-color). + +### rf_agg_overview_raster + + Tile rf_agg_overview_raster(Tile proj_raster_col, int cols, int rows, Extent aoi) + Tile rf_agg_overview_raster(Tile tile_col, int cols, int rows, Extent aoi, Extent tile_extent_col, CRS tile_crs_col) +Construct an overview _tile_ of size `cols` by `rows`. Data is filtered to the specified `aoi` which is given in web mercator. Uses bi-linear sampling method. The `tile_extent_col` and `tile_crs_col` arguments are optional if the first argument has its Extent and CRS embedded. -### rf_render_png +### rf_rgb_composite - Array rf_render_png(Tile red, Tile green, Tile blue) + Tile rf_rgb_composite(Tile red, Tile green, Tile blue) + +Merges three bands into a single byte-packed RGB composite. It first scales each cell to fit into an unsigned byte, in the range 0-255, and then merges all three channels to fit into a 32-bit unsigned integer. This is useful when you want an RGB tile to render or to process with other color imagery tools. -Runs [`rf_rgb_composite`](reference.md#rf-rgb-composite) on the given tile columns and then encodes the result as a PNG byte array. - [RasterFunctions]: org.locationtech.rasterframes.RasterFunctions [scaladoc]: latest/api/index.html +[Array]: http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.types.ArrayType diff --git a/docs/src/main/paradox/release-notes.md b/docs/src/main/paradox/release-notes.md index 77f3d1e55..01962a62b 100644 --- a/docs/src/main/paradox/release-notes.md +++ b/docs/src/main/paradox/release-notes.md @@ -1,16 +1,98 @@ # Release Notes +## 0.10.x + +### 0.10.1 + +* Fix UDTs registration ordering [#573](https://github.com/locationtech/rasterframes/pull/573) + +### 0.10.0 + +* Upgraded to Scala 2.12 , Spark 3.1.2, and GeoTrellis 3.6.0 (a subtantial accomplishment!) +* Added buffered tile support +* Added focal operations: `rf_focal_mean`, `rf_focal_median`,`rf_focal_mode`, `rf_focal_max`, `rf_focal_min`, `rf_focal_stddev`, `rf_focal_moransi`, `rf_convolve`, `rf_slope`, `rf_aspect`, `rf_hillshade` +* Added STAC API DataFrames implementation + +Special thanks to @pomadchin and @echeipesh for these substantial new contributions! + +## 0.9.x + +### 0.9.1 + +* Upgraded to Spark 2.4.7 +* Added `pyspark.sql.DataFrame.display(num_rows:int, truncate:bool)` extension method when `rf_ipython` is imported. +* Added users' manual section on IPython display enhancements. +* Added `method_name` parameter to the `rf_resample` method. + * __BREAKING__: In SQL, the function `rf_resample` now takes 3 arguments. You can use `rf_resample_nearest` with two arguments or refactor to `rf_resample(t, v, "nearest")`. +* Added resample method parameter to SQL and Python APIs. @ref:[See updated docs](raster-join.md). +* Upgraded many of the pyrasterframes dependencies, including: + `descartes`, `fiona`, `folium`, `geopandas`, `matplotlib`, `numpy`, `pandas`, `rasterio`, `shapely` +* Changed `rasterframes.prefer-gdal` configuration parameter to default to `False`, as JVM GeoTIFF performs just as well for COGs as the GDAL one. +* Fixed [#545](https://github.com/locationtech/rasterframes/issues/545). + +### 0.9.0 + +* Upgraded to GeoTrellis 3.3.0. This includes a number of _breaking_ changes enumerated as a part of the [PR's](https://github.com/locationtech/rasterframes/pull/398) change log. These include: + - Add `Int` type parameter to `Grid` + - Add `Int` type parameter to `CellGrid` + - Add `Int` type parameter to `GridBounds`... or `TileBounds` + - Use `GridBounds.toGridType` to coerce from `Int` to `Long` type parameter + - Update imports for layers, particularly `geotrellis.spark.tiling` to `geotrellis.layer` + - Update imports for `geotrellis.spark.io` to `geotrellis.spark.store...` + - Removed `FixedRasterExtent` + - Removed `FixedDelegatingTile` + - Removed `org.locationtech.rasterframes.util.Shims` + - Change `Extent.jtsGeom` to `Extent.toPolygon` + - Change `TileLayerMetadata.gridBounds` to `TileLayerMetadata.tileBounds` + - Add `geotrellis-gdal` dependency + - Remove any conversions between JTS geometry and old `geotrellis.vector` geometry + - Changed `org.locationtech.rasterframes.encoders.StandardEncoders.crsEncoder` to `crsSparkEncoder` + - Change `(cols, rows)` dimension destructuring to `Dimensions(cols, rows)` + - Revisit use of `Tile` equality since [it's more strict](https://github.com/locationtech/geotrellis/pull/2991) + - Update `reference.conf` to use `geotrellis.raster.gdal` namespace. + - Replace all uses of `TileDimensions` with `geotrellis.raster.Dimensions[Int]`. +* Upgraded to `gdal-warp-bindings` 1.0.0. +* Upgraded to Spark 2.4.5 +* Formally abandoned support for Python 2. Python 2 is dead. Long live Python 2. +* Introduction of type hints in Python API. +* Add functions for changing cell values based on either conditions or to achieve a distribution of values. ([#449](https://github.com/locationtech/rasterframes/pull/449)) + * Add `rf_local_min`, `rf_local_max`, and `rf_local_clip` functions. + * Add cell value scaling functions `rf_rescale` and `rf_standardize`. + * Add `rf_where` function, similar in spirit to numpy's `where`, or a cell-wise version of Spark SQL's `when` and `otherwise`. +* Add `rf_sqrt` function to compute cell-wise square root. + ## 0.8.x +### 0.8.5 + +* Added `rf_z2_index` for constructing a Z2 index on types with bounds. +* _Breaking_: `rf_spatial_index` renamed `rf_xz2_index` to differentiate between XZ2 and Z2 variants. +* Added `withSpatialIndex` to RasterSourceDataSource to pre-partition tiles based on tile extents mapped to a Z2 space-filling curve +* Add `rf_mask_by_bit`, `rf_mask_by_bits` and `rf_local_extract_bits` to deal with bit packed quality masks. Updated the masking documentation to demonstrate the use of these functions. +* Added `toDF` extension method to `MultibandGeoTiff` +* Added `rf_agg_extent` and `rf_agg_reprojected_extent` to compute the aggregate extent of a column +* Added `rf_proj_raster` for constructing a `proj_raster` structure from individual CRS, Extent, and Tile columns. +* Added `rf_render_color_ramp_png` to compute PNG byte array for a single tile column, with specified color ramp. +* In `rf_ipython`, improved rendering of dataframe binary contents with PNG preamble. +* Throw an `IllegalArgumentException` when attempting to apply a mask to a `Tile` whose `CellType` has no NoData defined. ([#409](https://github.com/locationtech/rasterframes/issues/384)) +* Add `rf_agg_approx_quantiles` function to compute cell quantiles across an entire column. + ### 0.8.4 * Upgraded to Spark 2.4.4 +* Add `rf_mask_by_values` and `rf_local_is_in` raster functions; added optional `inverse` argument to `rf_mask` functions. ([#403](https://github.com/locationtech/rasterframes/pull/403), [#384](https://github.com/locationtech/rasterframes/issues/384)) +* Added forced truncation of WKT types in Markdown/HTML rendering. ([#408](https://github.com/locationtech/rasterframes/pull/408)) +* Add `rf_local_is_in` raster function. ([#400](https://github.com/locationtech/rasterframes/pull/400)) +* Added partitioning to catalogs before processing in RasterSourceDataSource ([#397](https://github.com/locationtech/rasterframes/pull/397)) +* Fixed bug where `rf_tile_dimensions` would cause unnecessary reading of tiles. ([#394](https://github.com/locationtech/rasterframes/pull/394)) +* _Breaking_ (potentially): removed `GeoTiffCollectionRelation` due to usage limitation and overlap with `RasterSourceDataSource` functionality. ### 0.8.3 * Updated to GeoTrellis 2.3.3 and Proj4j 1.1.0. * Fixed issues with `LazyLogger` and shading assemblies ([#293](https://github.com/locationtech/rasterframes/issues/293)) * Updated `rf_crs` to accept string columns containing CRS specifications. ([#366](https://github.com/locationtech/rasterframes/issues/366)) +* Added `rf_spatial_index` function. ([#368](https://github.com/locationtech/rasterframes/issues/368)) * _Breaking_ (potentially): removed `pyrasterframes.create_spark_session` in lieu of `pyrasterframes.utils.create_rf_spark_session` ### 0.8.2 diff --git a/experimental/src/it/resources/log4j.properties b/experimental/src/it/resources/log4j.properties index 4a81f524a..cbbdd4af2 100644 --- a/experimental/src/it/resources/log4j.properties +++ b/experimental/src/it/resources/log4j.properties @@ -37,6 +37,8 @@ log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO log4j.logger.org.locationtech.rasterframes=INFO log4j.logger.org.apache.parquet.hadoop.ParquetRecordReader=OFF +log4j.logger.geotrellis.spark=INFO +log4j.logger.geotrellis.raster.gdal=ERROR # SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL diff --git a/experimental/src/it/scala/org/locationtech/rasterframes/experimental/datasource/awspds/L8CatalogRelationTest.scala b/experimental/src/it/scala/org/locationtech/rasterframes/experimental/datasource/awspds/L8CatalogRelationTest.scala deleted file mode 100644 index ea202b726..000000000 --- a/experimental/src/it/scala/org/locationtech/rasterframes/experimental/datasource/awspds/L8CatalogRelationTest.scala +++ /dev/null @@ -1,135 +0,0 @@ -/* - * This software is licensed under the Apache 2 license, quoted below. - * - * Copyright 2018 Astraea. Inc. - * - * Licensed under the Apache License, Version 2.0 (the "License"); you may not - * use this file except in compliance with the License. You may obtain a copy of - * the License at - * - * [http://www.apache.org/licenses/LICENSE-2.0] - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT - * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the - * License for the specific language governing permissions and limitations under - * the License. - * - * - */ - -package org.locationtech.rasterframes.experimental.datasource.awspds - -import geotrellis.proj4.LatLng -import geotrellis.vector.Extent -import org.apache.spark.sql.functions._ -import org.locationtech.rasterframes._ -import org.locationtech.rasterframes.datasource.raster._ -import org.locationtech.rasterframes.expressions.aggregates.TileRasterizerAggregate - -/** - * Test rig for L8 catalog stuff. - * - * @since 5/4/18 - */ -class L8CatalogRelationTest extends TestEnvironment { - import spark.implicits._ - - val catalog = spark.read.l8Catalog.load() - - val scenes = catalog - .where($"acquisition_date" === to_timestamp(lit("2017-04-04 15:12:55.394"))) - .where($"path" === 11 && $"row" === 12) - .cache() - - describe("Representing L8 scenes as a Spark data source") { - it("should provide a non-empty catalog") { - scenes.count() shouldBe 1 - } - - it("should provide 11 band + 1 QA urls") { - scenes.schema.count(_.name.startsWith("B")) shouldBe 12 - } - - it("should construct valid URLs") { - val urlStr = scenes.select("B11").as[String].first - val code = TestSupport.urlResponse(urlStr) - code should be (200) - } - - it("should work with SQL and spatial predicates") { - catalog.createOrReplaceTempView("l8_catalog") - val scenes = spark.sql(""" - SELECT st_geometry(bounds_wgs84) as geometry, acquisition_date, B1, B2 - FROM l8_catalog - WHERE - st_intersects(st_geometry(bounds_wgs84), st_geomFromText('LINESTRING (-39.551 -7.1881, -72.2461 -45.7062)')) AND - acquisition_date > to_timestamp('2017-11-01') AND - acquisition_date <= to_timestamp('2017-12-13') - """) - - scenes.count() shouldBe > (100L) - } - - it("should construct expected extents") { - catalog.createOrReplaceTempView("l8_catalog") - - catalog.filter($"bounds_wgs84.xmin" > $"bounds_wgs84.xmax").count() shouldBe (0) - catalog.filter($"bounds_wgs84.ymin" > $"bounds_wgs84.ymax").count() shouldBe (0) - - val geo_area_row = spark.sql( - """ - SELECT min(st_area(st_geometry(bounds_wgs84))) AS area - FROM l8_catalog - WHERE st_intersects(st_geometry(bounds_wgs84), st_geomFromText('LINESTRING(-78.035 39.004,-80.166 37.241)')) AND - acquisition_date > to_timestamp('2017-11-01') AND - acquisition_date <= to_timestamp('2017-11-16') - """).first() - val geo_area = geo_area_row.getDouble(0) - geo_area shouldBe < (6.5) - geo_area shouldBe > (4.5) - } - } - - describe("Read L8 scenes from PDS") { - it("should be compatible with raster DataSource") { - val df = spark.read.raster - .fromCatalog(scenes, "B1", "B3") - .withTileDimensions(512, 512) - .load() - - // Further refine down to a tile - val sub = df.select($"B3") - .where(st_contains(st_geometry(rf_extent($"B1")), st_makePoint(574965, 7679175))) - - val stats = sub.select(rf_agg_stats($"B3")).first - - stats.data_cells should be (512L * 512L) - stats.mean shouldBe > (10000.0) - } - - it("should construct an RGB composite") { - val aoiLL = Extent(31.115, 29.963, 31.148, 29.99) - - val scene = catalog - .where( - to_date($"acquisition_date") === to_date(lit("2019-07-03")) && - st_intersects(st_geometry($"bounds_wgs84"), geomLit(aoiLL.jtsGeom)) - ) - .orderBy("cloud_cover_pct") - .limit(1) - - val df = spark.read.raster - .fromCatalog(scene, "B4", "B3", "B2") - .withTileDimensions(256, 256) - .load() - .limit(1) - - noException should be thrownBy { - val raster = TileRasterizerAggregate.collect(df, LatLng, Some(aoiLL), None) - raster.tile.bandCount should be (3) - raster.extent.area > 0 - } - } - } -} diff --git a/experimental/src/it/scala/org/locationtech/rasterframes/experimental/datasource/awspds/MODISCatalogRelationTest.scala b/experimental/src/it/scala/org/locationtech/rasterframes/experimental/datasource/awspds/MODISCatalogRelationTest.scala deleted file mode 100644 index 8499bbe44..000000000 --- a/experimental/src/it/scala/org/locationtech/rasterframes/experimental/datasource/awspds/MODISCatalogRelationTest.scala +++ /dev/null @@ -1,107 +0,0 @@ -/* - * This software is licensed under the Apache 2 license, quoted below. - * - * Copyright 2018 Astraea. Inc. - * - * Licensed under the Apache License, Version 2.0 (the "License"); you may not - * use this file except in compliance with the License. You may obtain a copy of - * the License at - * - * [http://www.apache.org/licenses/LICENSE-2.0] - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT - * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the - * License for the specific language governing permissions and limitations under - * the License. - * - * - */ - -package org.locationtech.rasterframes.experimental.datasource.awspds -import java.sql.Timestamp - -import geotrellis.proj4.LatLng -import org.apache.spark.sql.functions._ -import org.locationtech.rasterframes._ -import org.locationtech.rasterframes.datasource.raster._ -import org.locationtech.rasterframes.stats.CellStatistics - -/** - * Test rig for MODIS catalog stuff. - * - * @since 5/4/18 - */ -class MODISCatalogRelationTest extends TestEnvironment { - import spark.implicits._ - val catalog = spark.read.modisCatalog.load() - val scenes = catalog - .where($"acquisition_date".as[Timestamp] === to_timestamp(lit("2018-1-1"))) - .where($"granule_id".contains("h24v03")) - .cache() - - describe("Representing MODIS scenes as a Spark data source") { - it("should provide a non-empty catalog") { - scenes.count() should be (1) - } - - it("should provide 7 band and 7 qa urls") { - scenes.schema.count(_.name.startsWith("B")) should be (14) - } - - it("should construct valid URLs") { - val urlStr = scenes.select("B03").as[String].first - val code = TestSupport.urlResponse(urlStr) - withClue(urlStr) { - code should be(200) - } - } - } - - describe("Read MODIS scenes from PDS") { - it("should be compatible with raster DataSource") { - val df = spark.read.raster - .fromCatalog(scenes, "B03") - .withTileDimensions(128, 128) - .load() - - // Further refine down to a tile - val sub = df.select($"B03", st_centroid(st_geometry(rf_extent($"B03")))) - .where(st_contains(st_geometry(rf_extent($"B03")), st_makePoint(7175787.353582373, 6345530.965564346))) - .withColumn("stats", rf_tile_stats(rf_tile($"B03"))) - - val stats = sub.select($"stats".as[CellStatistics]).first() - - stats.data_cells shouldBe < (128L * 128L) - stats.data_cells shouldBe > (128L) - stats.mean shouldBe > (1000.0) - } - it("should compute aggregate statistics") { - // This is copied from the docs. - import spark.implicits._ - - val modis = spark.read.format("aws-pds-modis-catalog").load() - - val red_nir_monthly_2017 = modis - .select($"granule_id", month($"acquisition_date") as "month", $"B01" as "red", $"B02" as "nir") - .where(year($"acquisition_date") === 2017 && (dayofmonth($"acquisition_date") === 15) && $"granule_id" === "h21v09") - - val red_nir_tiles_monthly_2017 = spark.read.raster - .fromCatalog(red_nir_monthly_2017, "red", "nir") - .load() - .cache() - - val result = red_nir_tiles_monthly_2017 - .where(st_intersects( - st_reproject(rf_geometry($"red"), rf_crs($"red"), LatLng), - st_makePoint(34.870605, -4.729727) - )) - .groupBy("month") - .agg(rf_agg_stats(rf_normalized_difference($"nir", $"red")) as "ndvi_stats") - .orderBy("month") - .select("month", "ndvi_stats.*") - - result.show() - } - } -} diff --git a/experimental/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister b/experimental/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister deleted file mode 100644 index 97275f043..000000000 --- a/experimental/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister +++ /dev/null @@ -1,2 +0,0 @@ -org.locationtech.rasterframes.experimental.datasource.awspds.L8CatalogDataSource -org.locationtech.rasterframes.experimental.datasource.awspds.MODISCatalogDataSource diff --git a/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/CachedDatasetRelation.scala b/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/CachedDatasetRelation.scala deleted file mode 100644 index 1fac7699a..000000000 --- a/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/CachedDatasetRelation.scala +++ /dev/null @@ -1,53 +0,0 @@ -/* - * This software is licensed under the Apache 2 license, quoted below. - * - * Copyright 2019 Astraea, Inc. - * - * Licensed under the Apache License, Version 2.0 (the "License"); you may not - * use this file except in compliance with the License. You may obtain a copy of - * the License at - * - * [http://www.apache.org/licenses/LICENSE-2.0] - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT - * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the - * License for the specific language governing permissions and limitations under - * the License. - * - * SPDX-License-Identifier: Apache-2.0 - * - */ - -package org.locationtech.rasterframes.experimental.datasource - -import org.apache.hadoop.fs.{FileSystem, Path => HadoopPath} -import org.apache.spark.rdd.RDD -import org.apache.spark.sql.sources.BaseRelation -import org.apache.spark.sql.{Dataset, Row, SaveMode} -import org.locationtech.rasterframes.util._ - -/** - * Mix-in for a data source that is cached as a parquet file. - * - * @since 8/24/18 - */ -trait CachedDatasetRelation extends ResourceCacheSupport { self: BaseRelation ⇒ - protected def cacheFile: HadoopPath - protected def constructDataset: Dataset[Row] - - def buildScan(): RDD[Row] = { - val conf = sqlContext.sparkContext.hadoopConfiguration - implicit val fs: FileSystem = FileSystem.get(conf) - val catalog = cacheFile.when(p => fs.exists(p) && !expired(p)) - .map(p ⇒ {logger.debug("Reading " + p); p}) - .map(p ⇒ sqlContext.read.parquet(p.toString)) - .getOrElse { - val scenes = constructDataset - scenes.write.mode(SaveMode.Overwrite).parquet(cacheFile.toString) - scenes - } - - catalog.rdd - } -} diff --git a/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/DownloadSupport.scala b/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/DownloadSupport.scala deleted file mode 100644 index e66d8f659..000000000 --- a/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/DownloadSupport.scala +++ /dev/null @@ -1,64 +0,0 @@ -/* - * This software is licensed under the Apache 2 license, quoted below. - * - * Copyright 2018 Astraea, Inc. - * - * Licensed under the Apache License, Version 2.0 (the "License"); you may not - * use this file except in compliance with the License. You may obtain a copy of - * the License at - * - * [http://www.apache.org/licenses/LICENSE-2.0] - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT - * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the - * License for the specific language governing permissions and limitations under - * the License. - * - * SPDX-License-Identifier: Apache-2.0 - * - */ - -package org.locationtech.rasterframes.experimental.datasource - -import java.io._ -import java.net - -import com.typesafe.scalalogging.Logger -import org.apache.commons.httpclient._ -import org.apache.commons.httpclient.methods._ -import org.apache.commons.httpclient.params.HttpMethodParams -import org.slf4j.LoggerFactory -import spray.json._ - - -/** - * Common support for downloading data. - * This is probably in the "insanely inefficient" category. Currently just a proof of concept. - * - * @since 5/5/18 - */ -trait DownloadSupport { - @transient protected lazy val logger = Logger(LoggerFactory.getLogger(getClass.getName)) - - private def applyMethodParams[M <: HttpMethodBase](method: M): M = { - method.getParams.setParameter(HttpMethodParams.RETRY_HANDLER, new DefaultHttpMethodRetryHandler(3, true)) - method.getParams.setIntParameter(HttpMethodParams.BUFFER_WARN_TRIGGER_LIMIT, 1024 * 1024 * 100) - method - } - - private def doGet[T](uri: java.net.URI, handler: HttpMethodBase ⇒ T): T = { - val client = new HttpClient() - val method = applyMethodParams(new GetMethod(uri.toASCIIString)) - logger.debug("Requesting " + uri) - val status = client.executeMethod(method) - status match { - case HttpStatus.SC_OK ⇒ handler(method) - case _ ⇒ throw new FileNotFoundException(s"Unable to download '$uri': ${method.getStatusLine}") - } - } - - protected def getBytes(uri: net.URI): Array[Byte] = doGet(uri, _.getResponseBody) - protected def getJson(uri: net.URI): JsValue = doGet(uri, _.getResponseBodyAsString.parseJson) - -} diff --git a/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/ResourceCacheSupport.scala b/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/ResourceCacheSupport.scala deleted file mode 100644 index 2f4d72fa5..000000000 --- a/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/ResourceCacheSupport.scala +++ /dev/null @@ -1,101 +0,0 @@ -/* - * This software is licensed under the Apache 2 license, quoted below. - * - * Copyright 2019 Astraea, Inc. - * - * Licensed under the Apache License, Version 2.0 (the "License"); you may not - * use this file except in compliance with the License. You may obtain a copy of - * the License at - * - * [http://www.apache.org/licenses/LICENSE-2.0] - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT - * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the - * License for the specific language governing permissions and limitations under - * the License. - * - * SPDX-License-Identifier: Apache-2.0 - * - */ - -package org.locationtech.rasterframes.experimental.datasource - -import java.net.URI -import java.time.{Duration, Instant} - -import org.apache.commons.io.FilenameUtils -import org.apache.hadoop.fs.{FileSystem, Path => HadoopPath} -import org.apache.hadoop.io.MD5Hash -import org.locationtech.rasterframes.util._ - -import scala.util.Try -import scala.util.control.NonFatal - -/** - * Support for downloading scene files from AWS PDS and caching them. - * - * @since 5/4/18 - */ -trait ResourceCacheSupport extends DownloadSupport { - - def maxCacheFileAgeHours: Int = sys.props.get("rasterframes.resource.age.max") - .flatMap(v ⇒ Try(v.toInt).toOption) - .getOrElse(24) - - protected def expired(p: HadoopPath)(implicit fs: FileSystem): Boolean = { - if(!fs.exists(p)) { - logger.debug(s"'$p' does not yet exist") - true - } - else { - - val time = fs.getFileStatus(p).getModificationTime - val exp = Instant.ofEpochMilli(time).plus(Duration.ofHours(maxCacheFileAgeHours)).isBefore(Instant.now()) - if(exp) logger.debug(s"'$p' is expired with mod time of '$time'") - exp - } - } - - protected def cacheDir(implicit fs: FileSystem): HadoopPath = { - val home = fs.getHomeDirectory - val cacheDir = new HadoopPath(home, ".rf_cache") - if(!fs.exists(cacheDir)) fs.mkdirs(cacheDir) - cacheDir - } - - protected def cacheName(path: Either[URI, HadoopPath])(implicit fs: FileSystem): HadoopPath = { - val (name, hash) = path match { - case Left(uri) ⇒ - (uri.getPath, MD5Hash.digest(uri.toASCIIString)) - case Right(p) ⇒ - (p.toString, MD5Hash.digest(p.toString)) - } - val basename = FilenameUtils.getBaseName(name) - val extension = FilenameUtils.getExtension(name) - val localFileName = s"$basename-$hash.$extension" - new HadoopPath(cacheDir, localFileName) - } - - protected def cachedURI(uri: URI)(implicit fs: FileSystem): Option[HadoopPath] = { - val dest = cacheName(Left(uri)) - dest.when(f ⇒ !expired(f)).orElse { - try { - val bytes = getBytes(uri) - withResource(fs.create(dest))(_.write(bytes)) - Some(dest) - } - catch { - case NonFatal(_) ⇒ - Try(fs.delete(dest, false)) - logger.warn(s"'$uri' not found") - None - } - } - } - - protected def cachedFile(fileName: HadoopPath)(implicit fs: FileSystem): Option[HadoopPath] = { - val dest = cacheName(Right(fileName)) - dest.when(f ⇒ !expired(f)) - } -} diff --git a/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/awspds/L8CatalogDataSource.scala b/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/awspds/L8CatalogDataSource.scala deleted file mode 100644 index 32c52bb59..000000000 --- a/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/awspds/L8CatalogDataSource.scala +++ /dev/null @@ -1,57 +0,0 @@ -/* - * This software is licensed under the Apache 2 license, quoted below. - * - * Copyright 2018 Astraea, Inc. - * - * Licensed under the Apache License, Version 2.0 (the "License"); you may not - * use this file except in compliance with the License. You may obtain a copy of - * the License at - * - * [http://www.apache.org/licenses/LICENSE-2.0] - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT - * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the - * License for the specific language governing permissions and limitations under - * the License. - * - * SPDX-License-Identifier: Apache-2.0 - * - */ - -package org.locationtech.rasterframes.experimental.datasource.awspds - -import java.io.FileNotFoundException -import java.net.URI - -import org.apache.hadoop.fs.FileSystem -import org.apache.spark.sql.SQLContext -import org.apache.spark.sql.sources.{BaseRelation, DataSourceRegister, RelationProvider} -import org.locationtech.rasterframes.experimental.datasource.ResourceCacheSupport - -/** - * Data source for querying AWS PDS catalog of L8 imagery. - * - * @since 9/28/17 - */ -class L8CatalogDataSource extends DataSourceRegister with RelationProvider { - def shortName = L8CatalogDataSource.SHORT_NAME - - def createRelation(sqlContext: SQLContext, parameters: Map[String, String]): BaseRelation = { - require(parameters.get("path").isEmpty, "L8CatalogDataSource doesn't support specifying a path. Please use `load()`.") - - val conf = sqlContext.sparkContext.hadoopConfiguration - implicit val fs = FileSystem.get(conf) - val path = L8CatalogDataSource.sceneListFile - L8CatalogRelation(sqlContext, path) - } -} - -object L8CatalogDataSource extends ResourceCacheSupport { - final val SHORT_NAME: String = "aws-pds-l8-catalog" - private val remoteSource = URI.create("http://landsat-pds.s3.amazonaws.com/c1/L8/scene_list.gz") - private def sceneListFile(implicit fs: FileSystem) = - cachedURI(remoteSource).getOrElse(throw new FileNotFoundException(remoteSource.toString)) -} - - diff --git a/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/awspds/L8CatalogRelation.scala b/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/awspds/L8CatalogRelation.scala deleted file mode 100644 index 9a14c86f3..000000000 --- a/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/awspds/L8CatalogRelation.scala +++ /dev/null @@ -1,121 +0,0 @@ -/* - * This software is licensed under the Apache 2 license, quoted below. - * - * Copyright 2018 Astraea, Inc. - * - * Licensed under the Apache License, Version 2.0 (the "License"); you may not - * use this file except in compliance with the License. You may obtain a copy of - * the License at - * - * [http://www.apache.org/licenses/LICENSE-2.0] - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT - * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the - * License for the specific language governing permissions and limitations under - * the License. - * - * SPDX-License-Identifier: Apache-2.0 - * - */ - -package org.locationtech.rasterframes.experimental.datasource.awspds - -import geotrellis.vector.Extent -import org.apache.hadoop.fs.{Path => HadoopPath} -import org.apache.spark.sql.functions._ -import org.apache.spark.sql.sources.{BaseRelation, TableScan} -import org.apache.spark.sql.types._ -import org.apache.spark.sql.{Dataset, Row, SQLContext, TypedColumn} -import org.locationtech.rasterframes.encoders.SparkBasicEncoders.stringEnc -import org.locationtech.rasterframes.experimental.datasource.CachedDatasetRelation -/** - * Schema definition and parser for AWS PDS L8 scene data. - * - * @author sfitch - * @since 9/28/17 - */ -case class L8CatalogRelation(sqlContext: SQLContext, sceneListPath: HadoopPath) - extends BaseRelation with TableScan with CachedDatasetRelation { - import L8CatalogRelation._ - - override def schema: StructType = L8CatalogRelation.schema - - protected def cacheFile: HadoopPath = sceneListPath.suffix(".parquet") - - protected def constructDataset: Dataset[Row] = { - import org.locationtech.rasterframes.encoders.StandardEncoders.extentEncoder - import sqlContext.implicits._ - logger.debug("Parsing " + sceneListPath) - - val bandCols = Bands.values.toSeq.map(b => l8_band_url(b) as b.toString) - - sqlContext.read - .schema(inputSchema) - .option("header", "true") - .csv(sceneListPath.toString) - .withColumn("__url", regexp_replace($"download_url", "index.html", "")) - .where(not($"${PRODUCT_ID.name}".endsWith("RT"))) - .drop("download_url") - .withColumn(BOUNDS_WGS84.name, struct( - $"min_lon" as "xmin", - $"min_lat" as "ymin", - $"max_lon" as "xmax", - $"max_lat" as "ymax" - ).as[Extent]) - .withColumnRenamed("__url", DOWNLOAD_URL.name) - .select(col("*") +: bandCols: _*) - .select(schema.map(f ⇒ col(f.name)): _*) - .orderBy(ACQUISITION_DATE.name, PATH.name, ROW.name) - .distinct() // The scene file contains duplicates. - .repartition(8, col(PATH.name), col(ROW.name)) - } -} - -object L8CatalogRelation extends PDSFields { - - /** - * Constructs link with the form: - * `https://s3-us-west-2.amazonaws.com/landsat-pds/c1/L8/149/039/LC08_L1TP_149039_20170411_20170415_01_T1/LC08_L1TP_149039_20170411_20170415_01_T1_{bandId].TIF` - * @param band Band identifier - * @return - */ - def l8_band_url(band: Bands.Band): TypedColumn[Any, String] = { - concat(col("download_url"), concat(col("product_id"), lit(s"_$band.TIF"))) - }.as(band.toString).as[String] - - private def inputSchema = StructType(Seq( - PRODUCT_ID, - ENTITY_ID, - ACQUISITION_DATE, - CLOUD_COVER, - PROC_LEVEL, - PATH, - ROW, - StructField("min_lat", DoubleType, false), - StructField("min_lon", DoubleType, false), - StructField("max_lat", DoubleType, false), - StructField("max_lon", DoubleType, false), - DOWNLOAD_URL - )) - - object Bands extends Enumeration { - type Band = Value - val B1, B2, B3, B4, B5, B6, B7, B8, B9, B10, B11, BQA = Value - val names: Seq[String] = values.toSeq.map(_.toString) - } - - - def schema = StructType(Seq( - PRODUCT_ID, - ENTITY_ID, - ACQUISITION_DATE, - CLOUD_COVER, - PROC_LEVEL, - PATH, - ROW, - BOUNDS_WGS84 - ) ++ Bands.names.map(n => StructField(n, StringType, true))) -} - - diff --git a/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/awspds/MODISCatalogDataSource.scala b/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/awspds/MODISCatalogDataSource.scala deleted file mode 100644 index ce2c552e3..000000000 --- a/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/awspds/MODISCatalogDataSource.scala +++ /dev/null @@ -1,165 +0,0 @@ -/* - * This software is licensed under the Apache 2 license, quoted below. - * - * Copyright 2018 Astraea, Inc. - * - * Licensed under the Apache License, Version 2.0 (the "License"); you may not - * use this file except in compliance with the License. You may obtain a copy of - * the License at - * - * [http://www.apache.org/licenses/LICENSE-2.0] - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT - * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the - * License for the specific language governing permissions and limitations under - * the License. - * - * SPDX-License-Identifier: Apache-2.0 - * - */ - -package org.locationtech.rasterframes.experimental.datasource.awspds - -import java.net.URI -import java.time.LocalDate -import java.time.temporal.ChronoUnit - -import com.typesafe.scalalogging.Logger -import org.apache.hadoop.fs.{FileSystem, Path => HadoopPath} -import org.apache.hadoop.io.IOUtils -import org.apache.spark.sql.SQLContext -import org.apache.spark.sql.sources.{BaseRelation, DataSourceRegister, RelationProvider} -import org.locationtech.rasterframes._ -import org.locationtech.rasterframes.experimental.datasource.ResourceCacheSupport -import org.locationtech.rasterframes.util.withResource -import org.slf4j.LoggerFactory - - -/** - * DataSource over the catalog of AWS PDS for MODIS MCD43A4 Surface Reflectance data product - * Param - * - * See https://docs.opendata.aws/modis-pds/readme.html for details - * - * @since 5/4/18 - */ -class MODISCatalogDataSource extends DataSourceRegister with RelationProvider { - @transient protected lazy val logger = Logger(LoggerFactory.getLogger(getClass.getName)) - - override def shortName(): String = MODISCatalogDataSource.SHORT_NAME - /** - * Create a MODIS catalog data source. - * @param sqlContext spark stuff - * @param parameters optional parameters are: - * `start`-start date for first scene files to fetch. default: "2013-01-01" - * `end`-end date for last scene file to fetch. default: today's date - 7 days - * `useBlacklist`-if false, ignore list of known missing scene files on AWS - */ - override def createRelation(sqlContext: SQLContext, parameters: Map[String, String]): BaseRelation = { - require(parameters.get("path").isEmpty, "MODISCatalogDataSource doesn't support specifying a path. Please use `load()`.") - - sqlContext.withRasterFrames - org.locationtech.rasterframes.experimental.datasource.register(sqlContext) - - val start = parameters.get("start").map(LocalDate.parse).getOrElse(LocalDate.of(2013, 1, 1)) - val end = parameters.get("end").map(LocalDate.parse).getOrElse(LocalDate.now().minusDays(7)) - val useBlacklist = parameters.get("useBlacklist").forall(_.toBoolean) - - val conf = sqlContext.sparkContext.hadoopConfiguration - implicit val fs = FileSystem.get(conf) - val path = MODISCatalogDataSource.sceneListFile(start, end, useBlacklist) - MODISCatalogRelation(sqlContext, path) - } -} - -object MODISCatalogDataSource extends ResourceCacheSupport { - final val SHORT_NAME = "aws-pds-modis-catalog" - final val MCD43A4_BASE = "https://modis-pds.s3.amazonaws.com/MCD43A4.006/" - override def maxCacheFileAgeHours: Int = Int.MaxValue - - // List of missing days in PDS - private val blacklist = Seq[String]( - "2018-02-27", - "2018-02-28", - "2018-03-01", - "2018-03-02", - "2018-03-03", - "2018-03-04", - "2018-03-05", - "2018-03-06", - "2018-03-07", - "2018-03-08", - "2018-03-09", - "2018-03-10", - "2018-03-11", - "2018-03-12", - "2018-03-13", - "2018-03-14", - "2018-03-15", - "2018-05-16", - "2018-05-17", - "2018-05-18", - "2018-05-19", - "2018-05-20", - "2018-05-21", - "2018-06-01", - "2018-06-04", - "2018-07-29", - "2018-08-03", - "2018-08-04", - "2018-08-05", - "2018-10-01", - "2018-10-02", - "2018-10-03", - "2018-10-22", - "2018-10-23", - "2018-11-12", - "2018-12-19", - "2018-12-20", - "2018-12-21", - "2018-12-22", - "2018-12-23", - "2018-12-24", - "2019-03-18" - ) - - private def sceneFiles(start: LocalDate, end: LocalDate, useBlacklist: Boolean) = { - val numDays = ChronoUnit.DAYS.between(start, end).toInt - for { - dayOffset <- 0 to numDays - currDay = start.plusDays(dayOffset) - if !useBlacklist || !blacklist.contains(currDay.toString) - } yield URI.create(s"$MCD43A4_BASE${currDay}_scenes.txt") - } - - private def sceneListFile(start: LocalDate, end: LocalDate, useBlacklist: Boolean)(implicit fs: FileSystem): HadoopPath = { - logger.info(s"Using '$cacheDir' for scene file cache") - val basename = new HadoopPath(s"$SHORT_NAME-$start-to-$end.csv") - cachedFile(basename).getOrElse { - val retval = cacheName(Right(basename)) - val inputs = sceneFiles(start, end, useBlacklist).par - .flatMap(cachedURI(_)) - .toArray - logger.debug(s"Concatinating scene files to '$retval':\n${inputs.mkString("\t" ,"\n\t", "\n")}") - try { - val dest = fs.create(retval) - dest.hflush() - dest.close() - fs.concat(retval, inputs) - } - catch { - case _ :UnsupportedOperationException ⇒ - // concat not supporty by RawLocalFileSystem - withResource(fs.create(retval)) { out ⇒ - inputs.foreach { p ⇒ - withResource(fs.open(p)) { in ⇒ - IOUtils.copyBytes(in, out, 1 << 15) - } - } - } - } - retval - } - } -} diff --git a/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/awspds/MODISCatalogRelation.scala b/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/awspds/MODISCatalogRelation.scala deleted file mode 100644 index 30b3ba234..000000000 --- a/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/awspds/MODISCatalogRelation.scala +++ /dev/null @@ -1,94 +0,0 @@ -/* - * This software is licensed under the Apache 2 license, quoted below. - * - * Copyright 2018 Astraea, Inc. - * - * Licensed under the Apache License, Version 2.0 (the "License"); you may not - * use this file except in compliance with the License. You may obtain a copy of - * the License at - * - * [http://www.apache.org/licenses/LICENSE-2.0] - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT - * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the - * License for the specific language governing permissions and limitations under - * the License. - * - * SPDX-License-Identifier: Apache-2.0 - * - */ - -package org.locationtech.rasterframes.experimental.datasource.awspds - -import org.apache.hadoop.fs.{Path => HadoopPath} -import org.apache.spark.sql._ -import org.apache.spark.sql.functions._ -import org.apache.spark.sql.sources._ -import org.apache.spark.sql.types._ -import org.locationtech.rasterframes.experimental.datasource.CachedDatasetRelation - -/** - * Constructs a dataframe from the available scenes - * - * @since 5/4/18 - */ -case class MODISCatalogRelation(sqlContext: SQLContext, sceneList: HadoopPath) - extends BaseRelation with TableScan with CachedDatasetRelation { - import MODISCatalogRelation._ - - protected def cacheFile: HadoopPath = sceneList.suffix(".parquet") - - override def schema: StructType = MODISCatalogRelation.schema - - protected def constructDataset: Dataset[Row] = { - import sqlContext.implicits._ - - logger.debug("Parsing " + sceneList) - val catalog = sqlContext.read - .option("header", "true") - .option("mode", "DROPMALFORMED") // <--- mainly for the fact that we have internal headers from the concat - .option("timestampFormat", "yyyy-MM-dd HH:mm:ss") - .schema(MODISCatalogRelation.inputSchema) - .csv(sceneList.toString) - - val bandCols = Bands.values.toSeq.map(b => MCD43A4_band_url(b) as b.toString) - - catalog - .withColumn("__split_gid", split($"gid", "\\.")) - .withColumn(DOWNLOAD_URL.name, regexp_replace(col(DOWNLOAD_URL.name), "index.html", "")) - .select(Seq( - $"__split_gid" (0) as PRODUCT_ID.name, - $"date" as ACQUISITION_DATE.name, - $"__split_gid" (2) as GRANULE_ID.name, - $"${GID.name}") ++ bandCols: _* - ) - .orderBy(ACQUISITION_DATE.name, GID.name) - .repartition(8, col(GRANULE_ID.name)) - } -} - -object MODISCatalogRelation extends PDSFields { - - def MCD43A4_band_url(suffix: Bands.Band) = - concat(col(DOWNLOAD_URL.name), concat(col(GID.name), lit(s"_${suffix}.TIF"))) - - object Bands extends Enumeration { - type Band = Value - val B01, B01qa, B02, B02qa, B03, B03aq, B04, B04qa, B05, B05qa, B06, B06qa, B07, B07qa = Value - val names: Seq[String] = values.toSeq.map(_.toString) - } - - def schema = StructType(Seq( - PRODUCT_ID, - ACQUISITION_DATE, - GRANULE_ID, - GID - ) ++ Bands.names.map(n => StructField(n, StringType, true))) - - private val inputSchema = StructType(Seq( - StructField("date", TimestampType, false), - DOWNLOAD_URL, - GID - )) -} diff --git a/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/awspds/PDSFields.scala b/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/awspds/PDSFields.scala deleted file mode 100644 index 40f8949f7..000000000 --- a/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/awspds/PDSFields.scala +++ /dev/null @@ -1,53 +0,0 @@ -/* - * This software is licensed under the Apache 2 license, quoted below. - * - * Copyright 2018 Astraea, Inc. - * - * Licensed under the Apache License, Version 2.0 (the "License"); you may not - * use this file except in compliance with the License. You may obtain a copy of - * the License at - * - * [http://www.apache.org/licenses/LICENSE-2.0] - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT - * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the - * License for the specific language governing permissions and limitations under - * the License. - * - * SPDX-License-Identifier: Apache-2.0 - * - */ - -package org.locationtech.rasterframes.experimental.datasource.awspds - -import org.locationtech.rasterframes.StandardColumns._ -import org.locationtech.rasterframes.util._ -import org.locationtech.rasterframes.encoders.StandardEncoders -import org.apache.spark.sql.jts.JTSTypes -import org.apache.spark.sql.types._ - -/** - * Standard column names - * - * @since 8/21/18 - */ -trait PDSFields { - final val PRODUCT_ID = StructField("product_id", StringType, false) - final val ENTITY_ID = StructField("entity_id", StringType, false) - final val ACQUISITION_DATE = StructField("acquisition_date", TimestampType, false) - final val TIMESTAMP = StructField(TIMESTAMP_COLUMN.columnName, TimestampType, false) - final val GRANULE_ID = StructField("granule_id", StringType, false) - final val DOWNLOAD_URL = StructField("download_url", StringType, false) - final val GID = StructField("gid", StringType, false) - final val CLOUD_COVER = StructField("cloud_cover_pct", FloatType, false) - final val PROC_LEVEL = StructField("processing_level", StringType, false) - final val PATH = StructField("path", ShortType, false) - final val ROW = StructField("row", ShortType, false) - final val BOUNDS = StructField(GEOMETRY_COLUMN.columnName, JTSTypes.GeometryTypeInstance, false) - final def BOUNDS_WGS84 = StructField( - "bounds_wgs84", StandardEncoders.extentEncoder.schema, false - ) -} - -object PDSFields extends PDSFields diff --git a/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/awspds/package.scala b/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/awspds/package.scala deleted file mode 100644 index bbd476528..000000000 --- a/experimental/src/main/scala/org/locationtech/rasterframes/experimental/datasource/awspds/package.scala +++ /dev/null @@ -1,46 +0,0 @@ -/* - * This software is licensed under the Apache 2 license, quoted below. - * - * Copyright 2018 Astraea, Inc. - * - * Licensed under the Apache License, Version 2.0 (the "License"); you may not - * use this file except in compliance with the License. You may obtain a copy of - * the License at - * - * [http://www.apache.org/licenses/LICENSE-2.0] - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT - * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the - * License for the specific language governing permissions and limitations under - * the License. - * - * SPDX-License-Identifier: Apache-2.0 - * - */ - -package org.locationtech.rasterframes.experimental.datasource - -import org.apache.spark.sql.DataFrameReader -import shapeless.tag -import shapeless.tag.@@ - -/** - * Module support. - * - * @since 5/4/18 - */ -package object awspds { - trait CatalogDataFrameReaderTag - type CatalogDataFrameReader = DataFrameReader @@ CatalogDataFrameReaderTag - - implicit class DataFrameReaderHasL8CatalogFormat(val reader: DataFrameReader) { - def l8Catalog: CatalogDataFrameReader = - tag[CatalogDataFrameReaderTag][DataFrameReader]( - reader.format(L8CatalogDataSource.SHORT_NAME)) - - def modisCatalog: CatalogDataFrameReader = - tag[CatalogDataFrameReaderTag][DataFrameReader]( - reader.format(MODISCatalogDataSource.SHORT_NAME)) - } -} diff --git a/poetry.lock b/poetry.lock new file mode 100644 index 000000000..1d5b490bf --- /dev/null +++ b/poetry.lock @@ -0,0 +1,2510 @@ +# This file is automatically @generated by Poetry 1.6.1 and should not be changed by hand. + +[[package]] +name = "affine" +version = "2.4.0" +description = "Matrices describing affine transformation of the plane" +optional = false +python-versions = ">=3.7" +files = [ + {file = "affine-2.4.0-py3-none-any.whl", hash = "sha256:8a3df80e2b2378aef598a83c1392efd47967afec4242021a0b06b4c7cbc61a92"}, + {file = "affine-2.4.0.tar.gz", hash = "sha256:a24d818d6a836c131976d22f8c27b8d3ca32d0af64c1d8d29deb7bafa4da1eea"}, +] + +[package.extras] +dev = ["coveralls", "flake8", "pydocstyle"] +test = ["pytest (>=4.6)", "pytest-cov"] + +[[package]] +name = "appnope" +version = "0.1.3" +description = "Disable App Nap on macOS >= 10.9" +optional = false +python-versions = "*" +files = [ + {file = "appnope-0.1.3-py2.py3-none-any.whl", hash = "sha256:265a455292d0bd8a72453494fa24df5a11eb18373a60c7c0430889f22548605e"}, + {file = "appnope-0.1.3.tar.gz", hash = "sha256:02bd91c4de869fbb1e1c50aafc4098827a7a54ab2f39d9dcba6c9547ed920e24"}, +] + +[[package]] +name = "asttokens" +version = "2.2.1" +description = "Annotate AST trees with source code positions" +optional = false +python-versions = "*" +files = [ + {file = "asttokens-2.2.1-py2.py3-none-any.whl", hash = "sha256:6b0ac9e93fb0335014d382b8fa9b3afa7df546984258005da0b9e7095b3deb1c"}, + {file = "asttokens-2.2.1.tar.gz", hash = "sha256:4622110b2a6f30b77e1473affaa97e711bc2f07d3f10848420ff1898edbe94f3"}, +] + +[package.dependencies] +six = "*" + +[package.extras] +test = ["astroid", "pytest"] + +[[package]] +name = "attrs" +version = "23.1.0" +description = "Classes Without Boilerplate" +optional = false +python-versions = ">=3.7" +files = [ + {file = "attrs-23.1.0-py3-none-any.whl", hash = "sha256:1f28b4522cdc2fb4256ac1a020c78acf9cba2c6b461ccd2c126f3aa8e8335d04"}, + {file = "attrs-23.1.0.tar.gz", hash = "sha256:6279836d581513a26f1bf235f9acd333bc9115683f14f7e8fae46c98fc50e015"}, +] + +[package.extras] +cov = ["attrs[tests]", "coverage[toml] (>=5.3)"] +dev = ["attrs[docs,tests]", "pre-commit"] +docs = ["furo", "myst-parser", "sphinx", "sphinx-notfound-page", "sphinxcontrib-towncrier", "towncrier", "zope-interface"] +tests = ["attrs[tests-no-zope]", "zope-interface"] +tests-no-zope = ["cloudpickle", "hypothesis", "mypy (>=1.1.1)", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins", "pytest-xdist[psutil]"] + +[[package]] +name = "backcall" +version = "0.2.0" +description = "Specifications for callback functions passed in to an API" +optional = false +python-versions = "*" +files = [ + {file = "backcall-0.2.0-py2.py3-none-any.whl", hash = "sha256:fbbce6a29f263178a1f7915c1940bde0ec2b2a967566fe1c65c1dfb7422bd255"}, + {file = "backcall-0.2.0.tar.gz", hash = "sha256:5cbdbf27be5e7cfadb448baf0aa95508f91f2bbc6c6437cd9cd06e2a4c215e1e"}, +] + +[[package]] +name = "beautifulsoup4" +version = "4.12.2" +description = "Screen-scraping library" +optional = false +python-versions = ">=3.6.0" +files = [ + {file = "beautifulsoup4-4.12.2-py3-none-any.whl", hash = "sha256:bd2520ca0d9d7d12694a53d44ac482d181b4ec1888909b035a3dbf40d0f57d4a"}, + {file = "beautifulsoup4-4.12.2.tar.gz", hash = "sha256:492bbc69dca35d12daac71c4db1bfff0c876c00ef4a2ffacce226d4638eb72da"}, +] + +[package.dependencies] +soupsieve = ">1.2" + +[package.extras] +html5lib = ["html5lib"] +lxml = ["lxml"] + +[[package]] +name = "black" +version = "22.12.0" +description = "The uncompromising code formatter." +optional = false +python-versions = ">=3.7" +files = [ + {file = "black-22.12.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9eedd20838bd5d75b80c9f5487dbcb06836a43833a37846cf1d8c1cc01cef59d"}, + {file = "black-22.12.0-cp310-cp310-win_amd64.whl", hash = "sha256:159a46a4947f73387b4d83e87ea006dbb2337eab6c879620a3ba52699b1f4351"}, + {file = "black-22.12.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d30b212bffeb1e252b31dd269dfae69dd17e06d92b87ad26e23890f3efea366f"}, + {file = "black-22.12.0-cp311-cp311-win_amd64.whl", hash = "sha256:7412e75863aa5c5411886804678b7d083c7c28421210180d67dfd8cf1221e1f4"}, + {file = "black-22.12.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c116eed0efb9ff870ded8b62fe9f28dd61ef6e9ddd28d83d7d264a38417dcee2"}, + {file = "black-22.12.0-cp37-cp37m-win_amd64.whl", hash = "sha256:1f58cbe16dfe8c12b7434e50ff889fa479072096d79f0a7f25e4ab8e94cd8350"}, + {file = "black-22.12.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:77d86c9f3db9b1bf6761244bc0b3572a546f5fe37917a044e02f3166d5aafa7d"}, + {file = "black-22.12.0-cp38-cp38-win_amd64.whl", hash = "sha256:82d9fe8fee3401e02e79767016b4907820a7dc28d70d137eb397b92ef3cc5bfc"}, + {file = "black-22.12.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:101c69b23df9b44247bd88e1d7e90154336ac4992502d4197bdac35dd7ee3320"}, + {file = "black-22.12.0-cp39-cp39-win_amd64.whl", hash = "sha256:559c7a1ba9a006226f09e4916060982fd27334ae1998e7a38b3f33a37f7a2148"}, + {file = "black-22.12.0-py3-none-any.whl", hash = "sha256:436cc9167dd28040ad90d3b404aec22cedf24a6e4d7de221bec2730ec0c97bcf"}, + {file = "black-22.12.0.tar.gz", hash = "sha256:229351e5a18ca30f447bf724d007f890f97e13af070bb6ad4c0a441cd7596a2f"}, +] + +[package.dependencies] +click = ">=8.0.0" +mypy-extensions = ">=0.4.3" +pathspec = ">=0.9.0" +platformdirs = ">=2" +tomli = {version = ">=1.1.0", markers = "python_full_version < \"3.11.0a7\""} +typing-extensions = {version = ">=3.10.0.0", markers = "python_version < \"3.10\""} + +[package.extras] +colorama = ["colorama (>=0.4.3)"] +d = ["aiohttp (>=3.7.4)"] +jupyter = ["ipython (>=7.8.0)", "tokenize-rt (>=3.2.0)"] +uvloop = ["uvloop (>=0.15.2)"] + +[[package]] +name = "bleach" +version = "6.0.0" +description = "An easy safelist-based HTML-sanitizing tool." +optional = false +python-versions = ">=3.7" +files = [ + {file = "bleach-6.0.0-py3-none-any.whl", hash = "sha256:33c16e3353dbd13028ab4799a0f89a83f113405c766e9c122df8a06f5b85b3f4"}, + {file = "bleach-6.0.0.tar.gz", hash = "sha256:1a1a85c1595e07d8db14c5f09f09e6433502c51c595970edc090551f0db99414"}, +] + +[package.dependencies] +six = ">=1.9.0" +webencodings = "*" + +[package.extras] +css = ["tinycss2 (>=1.1.0,<1.2)"] + +[[package]] +name = "boto3" +version = "1.26.118" +description = "The AWS SDK for Python" +optional = false +python-versions = ">= 3.7" +files = [ + {file = "boto3-1.26.118-py3-none-any.whl", hash = "sha256:1ff703152553f4d5fc9774071d114dbf06ec661eb1b29b6051f6b1f9d0c24873"}, + {file = "boto3-1.26.118.tar.gz", hash = "sha256:d0ed43228952b55c9f44d1c733f74656418c39c55dbe36bc37feeef6aa583ded"}, +] + +[package.dependencies] +botocore = ">=1.29.118,<1.30.0" +jmespath = ">=0.7.1,<2.0.0" +s3transfer = ">=0.6.0,<0.7.0" + +[package.extras] +crt = ["botocore[crt] (>=1.21.0,<2.0a0)"] + +[[package]] +name = "botocore" +version = "1.29.118" +description = "Low-level, data-driven core of boto 3." +optional = false +python-versions = ">= 3.7" +files = [ + {file = "botocore-1.29.118-py3-none-any.whl", hash = "sha256:44cb088a73b02dd716c5c5715143a64d5f10388957285246e11f3cc893eebf9d"}, + {file = "botocore-1.29.118.tar.gz", hash = "sha256:b51fc5d50cbc43edaf58b3ec4fa933b82755801c453bf8908c8d3e70ae1142c1"}, +] + +[package.dependencies] +jmespath = ">=0.7.1,<2.0.0" +python-dateutil = ">=2.1,<3.0.0" +urllib3 = ">=1.25.4,<1.27" + +[package.extras] +crt = ["awscrt (==0.16.9)"] + +[[package]] +name = "certifi" +version = "2022.12.7" +description = "Python package for providing Mozilla's CA Bundle." +optional = false +python-versions = ">=3.6" +files = [ + {file = "certifi-2022.12.7-py3-none-any.whl", hash = "sha256:4ad3232f5e926d6718ec31cfc1fcadfde020920e278684144551c91769c7bc18"}, + {file = "certifi-2022.12.7.tar.gz", hash = "sha256:35824b4c3a97115964b408844d64aa14db1cc518f6562e8d7261699d1350a9e3"}, +] + +[[package]] +name = "cffi" +version = "1.15.1" +description = "Foreign Function Interface for Python calling C code." +optional = false +python-versions = "*" +files = [ + {file = "cffi-1.15.1-cp27-cp27m-macosx_10_9_x86_64.whl", hash = "sha256:a66d3508133af6e8548451b25058d5812812ec3798c886bf38ed24a98216fab2"}, + {file = "cffi-1.15.1-cp27-cp27m-manylinux1_i686.whl", hash = "sha256:470c103ae716238bbe698d67ad020e1db9d9dba34fa5a899b5e21577e6d52ed2"}, + {file = "cffi-1.15.1-cp27-cp27m-manylinux1_x86_64.whl", hash = "sha256:9ad5db27f9cabae298d151c85cf2bad1d359a1b9c686a275df03385758e2f914"}, + {file = "cffi-1.15.1-cp27-cp27m-win32.whl", hash = "sha256:b3bbeb01c2b273cca1e1e0c5df57f12dce9a4dd331b4fa1635b8bec26350bde3"}, + {file = "cffi-1.15.1-cp27-cp27m-win_amd64.whl", hash = "sha256:e00b098126fd45523dd056d2efba6c5a63b71ffe9f2bbe1a4fe1716e1d0c331e"}, + {file = "cffi-1.15.1-cp27-cp27mu-manylinux1_i686.whl", hash = "sha256:d61f4695e6c866a23a21acab0509af1cdfd2c013cf256bbf5b6b5e2695827162"}, + {file = "cffi-1.15.1-cp27-cp27mu-manylinux1_x86_64.whl", hash = "sha256:ed9cb427ba5504c1dc15ede7d516b84757c3e3d7868ccc85121d9310d27eed0b"}, + {file = "cffi-1.15.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:39d39875251ca8f612b6f33e6b1195af86d1b3e60086068be9cc053aa4376e21"}, + {file = "cffi-1.15.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:285d29981935eb726a4399badae8f0ffdff4f5050eaa6d0cfc3f64b857b77185"}, + {file = "cffi-1.15.1-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:3eb6971dcff08619f8d91607cfc726518b6fa2a9eba42856be181c6d0d9515fd"}, + {file = "cffi-1.15.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:21157295583fe8943475029ed5abdcf71eb3911894724e360acff1d61c1d54bc"}, + {file = "cffi-1.15.1-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:5635bd9cb9731e6d4a1132a498dd34f764034a8ce60cef4f5319c0541159392f"}, + {file = "cffi-1.15.1-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:2012c72d854c2d03e45d06ae57f40d78e5770d252f195b93f581acf3ba44496e"}, + {file = "cffi-1.15.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:dd86c085fae2efd48ac91dd7ccffcfc0571387fe1193d33b6394db7ef31fe2a4"}, + {file = "cffi-1.15.1-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:fa6693661a4c91757f4412306191b6dc88c1703f780c8234035eac011922bc01"}, + {file = "cffi-1.15.1-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:59c0b02d0a6c384d453fece7566d1c7e6b7bae4fc5874ef2ef46d56776d61c9e"}, + {file = "cffi-1.15.1-cp310-cp310-win32.whl", hash = "sha256:cba9d6b9a7d64d4bd46167096fc9d2f835e25d7e4c121fb2ddfc6528fb0413b2"}, + {file = "cffi-1.15.1-cp310-cp310-win_amd64.whl", hash = "sha256:ce4bcc037df4fc5e3d184794f27bdaab018943698f4ca31630bc7f84a7b69c6d"}, + {file = "cffi-1.15.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:3d08afd128ddaa624a48cf2b859afef385b720bb4b43df214f85616922e6a5ac"}, + {file = "cffi-1.15.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:3799aecf2e17cf585d977b780ce79ff0dc9b78d799fc694221ce814c2c19db83"}, + {file = "cffi-1.15.1-cp311-cp311-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:a591fe9e525846e4d154205572a029f653ada1a78b93697f3b5a8f1f2bc055b9"}, + {file = "cffi-1.15.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3548db281cd7d2561c9ad9984681c95f7b0e38881201e157833a2342c30d5e8c"}, + {file = "cffi-1.15.1-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:91fc98adde3d7881af9b59ed0294046f3806221863722ba7d8d120c575314325"}, + {file = "cffi-1.15.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:94411f22c3985acaec6f83c6df553f2dbe17b698cc7f8ae751ff2237d96b9e3c"}, + {file = "cffi-1.15.1-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:03425bdae262c76aad70202debd780501fabeaca237cdfddc008987c0e0f59ef"}, + {file = "cffi-1.15.1-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:cc4d65aeeaa04136a12677d3dd0b1c0c94dc43abac5860ab33cceb42b801c1e8"}, + {file = "cffi-1.15.1-cp311-cp311-win32.whl", hash = "sha256:a0f100c8912c114ff53e1202d0078b425bee3649ae34d7b070e9697f93c5d52d"}, + {file = "cffi-1.15.1-cp311-cp311-win_amd64.whl", hash = "sha256:04ed324bda3cda42b9b695d51bb7d54b680b9719cfab04227cdd1e04e5de3104"}, + {file = "cffi-1.15.1-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:50a74364d85fd319352182ef59c5c790484a336f6db772c1a9231f1c3ed0cbd7"}, + {file = "cffi-1.15.1-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e263d77ee3dd201c3a142934a086a4450861778baaeeb45db4591ef65550b0a6"}, + {file = "cffi-1.15.1-cp36-cp36m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:cec7d9412a9102bdc577382c3929b337320c4c4c4849f2c5cdd14d7368c5562d"}, + {file = "cffi-1.15.1-cp36-cp36m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:4289fc34b2f5316fbb762d75362931e351941fa95fa18789191b33fc4cf9504a"}, + {file = "cffi-1.15.1-cp36-cp36m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:173379135477dc8cac4bc58f45db08ab45d228b3363adb7af79436135d028405"}, + {file = "cffi-1.15.1-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:6975a3fac6bc83c4a65c9f9fcab9e47019a11d3d2cf7f3c0d03431bf145a941e"}, + {file = "cffi-1.15.1-cp36-cp36m-win32.whl", hash = "sha256:2470043b93ff09bf8fb1d46d1cb756ce6132c54826661a32d4e4d132e1977adf"}, + {file = "cffi-1.15.1-cp36-cp36m-win_amd64.whl", hash = "sha256:30d78fbc8ebf9c92c9b7823ee18eb92f2e6ef79b45ac84db507f52fbe3ec4497"}, + {file = "cffi-1.15.1-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:198caafb44239b60e252492445da556afafc7d1e3ab7a1fb3f0584ef6d742375"}, + {file = "cffi-1.15.1-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:5ef34d190326c3b1f822a5b7a45f6c4535e2f47ed06fec77d3d799c450b2651e"}, + {file = "cffi-1.15.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8102eaf27e1e448db915d08afa8b41d6c7ca7a04b7d73af6514df10a3e74bd82"}, + {file = "cffi-1.15.1-cp37-cp37m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:5df2768244d19ab7f60546d0c7c63ce1581f7af8b5de3eb3004b9b6fc8a9f84b"}, + {file = "cffi-1.15.1-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:a8c4917bd7ad33e8eb21e9a5bbba979b49d9a97acb3a803092cbc1133e20343c"}, + {file = "cffi-1.15.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0e2642fe3142e4cc4af0799748233ad6da94c62a8bec3a6648bf8ee68b1c7426"}, + {file = "cffi-1.15.1-cp37-cp37m-win32.whl", hash = "sha256:e229a521186c75c8ad9490854fd8bbdd9a0c9aa3a524326b55be83b54d4e0ad9"}, + {file = "cffi-1.15.1-cp37-cp37m-win_amd64.whl", hash = "sha256:a0b71b1b8fbf2b96e41c4d990244165e2c9be83d54962a9a1d118fd8657d2045"}, + {file = "cffi-1.15.1-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:320dab6e7cb2eacdf0e658569d2575c4dad258c0fcc794f46215e1e39f90f2c3"}, + {file = "cffi-1.15.1-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:1e74c6b51a9ed6589199c787bf5f9875612ca4a8a0785fb2d4a84429badaf22a"}, + {file = "cffi-1.15.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a5c84c68147988265e60416b57fc83425a78058853509c1b0629c180094904a5"}, + {file = "cffi-1.15.1-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3b926aa83d1edb5aa5b427b4053dc420ec295a08e40911296b9eb1b6170f6cca"}, + {file = "cffi-1.15.1-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:87c450779d0914f2861b8526e035c5e6da0a3199d8f1add1a665e1cbc6fc6d02"}, + {file = "cffi-1.15.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4f2c9f67e9821cad2e5f480bc8d83b8742896f1242dba247911072d4fa94c192"}, + {file = "cffi-1.15.1-cp38-cp38-win32.whl", hash = "sha256:8b7ee99e510d7b66cdb6c593f21c043c248537a32e0bedf02e01e9553a172314"}, + {file = "cffi-1.15.1-cp38-cp38-win_amd64.whl", hash = "sha256:00a9ed42e88df81ffae7a8ab6d9356b371399b91dbdf0c3cb1e84c03a13aceb5"}, + {file = "cffi-1.15.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:54a2db7b78338edd780e7ef7f9f6c442500fb0d41a5a4ea24fff1c929d5af585"}, + {file = "cffi-1.15.1-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:fcd131dd944808b5bdb38e6f5b53013c5aa4f334c5cad0c72742f6eba4b73db0"}, + {file = "cffi-1.15.1-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:7473e861101c9e72452f9bf8acb984947aa1661a7704553a9f6e4baa5ba64415"}, + {file = "cffi-1.15.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:6c9a799e985904922a4d207a94eae35c78ebae90e128f0c4e521ce339396be9d"}, + {file = "cffi-1.15.1-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3bcde07039e586f91b45c88f8583ea7cf7a0770df3a1649627bf598332cb6984"}, + {file = "cffi-1.15.1-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:33ab79603146aace82c2427da5ca6e58f2b3f2fb5da893ceac0c42218a40be35"}, + {file = "cffi-1.15.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5d598b938678ebf3c67377cdd45e09d431369c3b1a5b331058c338e201f12b27"}, + {file = "cffi-1.15.1-cp39-cp39-musllinux_1_1_i686.whl", hash = "sha256:db0fbb9c62743ce59a9ff687eb5f4afbe77e5e8403d6697f7446e5f609976f76"}, + {file = "cffi-1.15.1-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:98d85c6a2bef81588d9227dde12db8a7f47f639f4a17c9ae08e773aa9c697bf3"}, + {file = "cffi-1.15.1-cp39-cp39-win32.whl", hash = "sha256:40f4774f5a9d4f5e344f31a32b5096977b5d48560c5592e2f3d2c4374bd543ee"}, + {file = "cffi-1.15.1-cp39-cp39-win_amd64.whl", hash = "sha256:70df4e3b545a17496c9b3f41f5115e69a4f2e77e94e1d2a8e1070bc0c38c8a3c"}, + {file = "cffi-1.15.1.tar.gz", hash = "sha256:d400bfb9a37b1351253cb402671cea7e89bdecc294e8016a707f6d1d8ac934f9"}, +] + +[package.dependencies] +pycparser = "*" + +[[package]] +name = "cfgv" +version = "3.3.1" +description = "Validate configuration and produce human readable error messages." +optional = false +python-versions = ">=3.6.1" +files = [ + {file = "cfgv-3.3.1-py2.py3-none-any.whl", hash = "sha256:c6a0883f3917a037485059700b9e75da2464e6c27051014ad85ba6aaa5884426"}, + {file = "cfgv-3.3.1.tar.gz", hash = "sha256:f5a830efb9ce7a445376bb66ec94c638a9787422f96264c98edc6bdeed8ab736"}, +] + +[[package]] +name = "click" +version = "8.1.3" +description = "Composable command line interface toolkit" +optional = false +python-versions = ">=3.7" +files = [ + {file = "click-8.1.3-py3-none-any.whl", hash = "sha256:bb4d8133cb15a609f44e8213d9b391b0809795062913b383c62be0ee95b1db48"}, + {file = "click-8.1.3.tar.gz", hash = "sha256:7682dc8afb30297001674575ea00d1814d808d6a36af415a82bd481d37ba7b8e"}, +] + +[package.dependencies] +colorama = {version = "*", markers = "platform_system == \"Windows\""} + +[[package]] +name = "click-plugins" +version = "1.1.1" +description = "An extension module for click to enable registering CLI commands via setuptools entry-points." +optional = false +python-versions = "*" +files = [ + {file = "click-plugins-1.1.1.tar.gz", hash = "sha256:46ab999744a9d831159c3411bb0c79346d94a444df9a3a3742e9ed63645f264b"}, + {file = "click_plugins-1.1.1-py2.py3-none-any.whl", hash = "sha256:5d262006d3222f5057fd81e1623d4443e41dcda5dc815c06b442aa3c02889fc8"}, +] + +[package.dependencies] +click = ">=4.0" + +[package.extras] +dev = ["coveralls", "pytest (>=3.6)", "pytest-cov", "wheel"] + +[[package]] +name = "cligj" +version = "0.7.2" +description = "Click params for commmand line interfaces to GeoJSON" +optional = false +python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, <4" +files = [ + {file = "cligj-0.7.2-py3-none-any.whl", hash = "sha256:c1ca117dbce1fe20a5809dc96f01e1c2840f6dcc939b3ddbb1111bf330ba82df"}, + {file = "cligj-0.7.2.tar.gz", hash = "sha256:a4bc13d623356b373c2c27c53dbd9c68cae5d526270bfa71f6c6fa69669c6b27"}, +] + +[package.dependencies] +click = ">=4.0" + +[package.extras] +test = ["pytest-cov"] + +[[package]] +name = "colorama" +version = "0.4.6" +description = "Cross-platform colored terminal text." +optional = false +python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,>=2.7" +files = [ + {file = "colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6"}, + {file = "colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44"}, +] + +[[package]] +name = "comm" +version = "0.1.3" +description = "Jupyter Python Comm implementation, for usage in ipykernel, xeus-python etc." +optional = false +python-versions = ">=3.6" +files = [ + {file = "comm-0.1.3-py3-none-any.whl", hash = "sha256:16613c6211e20223f215fc6d3b266a247b6e2641bf4e0a3ad34cb1aff2aa3f37"}, + {file = "comm-0.1.3.tar.gz", hash = "sha256:a61efa9daffcfbe66fd643ba966f846a624e4e6d6767eda9cf6e993aadaab93e"}, +] + +[package.dependencies] +traitlets = ">=5.3" + +[package.extras] +lint = ["black (>=22.6.0)", "mdformat (>0.7)", "mdformat-gfm (>=0.3.5)", "ruff (>=0.0.156)"] +test = ["pytest"] +typing = ["mypy (>=0.990)"] + +[[package]] +name = "contourpy" +version = "1.0.7" +description = "Python library for calculating contours of 2D quadrilateral grids" +optional = false +python-versions = ">=3.8" +files = [ + {file = "contourpy-1.0.7-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:95c3acddf921944f241b6773b767f1cbce71d03307270e2d769fd584d5d1092d"}, + {file = "contourpy-1.0.7-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:fc1464c97579da9f3ab16763c32e5c5d5bb5fa1ec7ce509a4ca6108b61b84fab"}, + {file = "contourpy-1.0.7-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:8acf74b5d383414401926c1598ed77825cd530ac7b463ebc2e4f46638f56cce6"}, + {file = "contourpy-1.0.7-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1c71fdd8f1c0f84ffd58fca37d00ca4ebaa9e502fb49825484da075ac0b0b803"}, + {file = "contourpy-1.0.7-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:f99e9486bf1bb979d95d5cffed40689cb595abb2b841f2991fc894b3452290e8"}, + {file = "contourpy-1.0.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:87f4d8941a9564cda3f7fa6a6cd9b32ec575830780677932abdec7bcb61717b0"}, + {file = "contourpy-1.0.7-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:9e20e5a1908e18aaa60d9077a6d8753090e3f85ca25da6e25d30dc0a9e84c2c6"}, + {file = "contourpy-1.0.7-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:a877ada905f7d69b2a31796c4b66e31a8068b37aa9b78832d41c82fc3e056ddd"}, + {file = "contourpy-1.0.7-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:6381fa66866b0ea35e15d197fc06ac3840a9b2643a6475c8fff267db8b9f1e69"}, + {file = "contourpy-1.0.7-cp310-cp310-win32.whl", hash = "sha256:3c184ad2433635f216645fdf0493011a4667e8d46b34082f5a3de702b6ec42e3"}, + {file = "contourpy-1.0.7-cp310-cp310-win_amd64.whl", hash = "sha256:3caea6365b13119626ee996711ab63e0c9d7496f65641f4459c60a009a1f3e80"}, + {file = "contourpy-1.0.7-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:ed33433fc3820263a6368e532f19ddb4c5990855e4886088ad84fd7c4e561c71"}, + {file = "contourpy-1.0.7-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:38e2e577f0f092b8e6774459317c05a69935a1755ecfb621c0a98f0e3c09c9a5"}, + {file = "contourpy-1.0.7-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:ae90d5a8590e5310c32a7630b4b8618cef7563cebf649011da80874d0aa8f414"}, + {file = "contourpy-1.0.7-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:130230b7e49825c98edf0b428b7aa1125503d91732735ef897786fe5452b1ec2"}, + {file = "contourpy-1.0.7-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:58569c491e7f7e874f11519ef46737cea1d6eda1b514e4eb5ac7dab6aa864d02"}, + {file = "contourpy-1.0.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:54d43960d809c4c12508a60b66cb936e7ed57d51fb5e30b513934a4a23874fae"}, + {file = "contourpy-1.0.7-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:152fd8f730c31fd67fe0ffebe1df38ab6a669403da93df218801a893645c6ccc"}, + {file = "contourpy-1.0.7-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:9056c5310eb1daa33fc234ef39ebfb8c8e2533f088bbf0bc7350f70a29bde1ac"}, + {file = "contourpy-1.0.7-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:a9d7587d2fdc820cc9177139b56795c39fb8560f540bba9ceea215f1f66e1566"}, + {file = "contourpy-1.0.7-cp311-cp311-win32.whl", hash = "sha256:4ee3ee247f795a69e53cd91d927146fb16c4e803c7ac86c84104940c7d2cabf0"}, + {file = "contourpy-1.0.7-cp311-cp311-win_amd64.whl", hash = "sha256:5caeacc68642e5f19d707471890f037a13007feba8427eb7f2a60811a1fc1350"}, + {file = "contourpy-1.0.7-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:fd7dc0e6812b799a34f6d12fcb1000539098c249c8da54f3566c6a6461d0dbad"}, + {file = "contourpy-1.0.7-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:0f9d350b639db6c2c233d92c7f213d94d2e444d8e8fc5ca44c9706cf72193772"}, + {file = "contourpy-1.0.7-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:e96a08b62bb8de960d3a6afbc5ed8421bf1a2d9c85cc4ea73f4bc81b4910500f"}, + {file = "contourpy-1.0.7-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:031154ed61f7328ad7f97662e48660a150ef84ee1bc8876b6472af88bf5a9b98"}, + {file = "contourpy-1.0.7-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:2e9ebb4425fc1b658e13bace354c48a933b842d53c458f02c86f371cecbedecc"}, + {file = "contourpy-1.0.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:efb8f6d08ca7998cf59eaf50c9d60717f29a1a0a09caa46460d33b2924839dbd"}, + {file = "contourpy-1.0.7-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:6c180d89a28787e4b73b07e9b0e2dac7741261dbdca95f2b489c4f8f887dd810"}, + {file = "contourpy-1.0.7-cp38-cp38-musllinux_1_1_i686.whl", hash = "sha256:b8d587cc39057d0afd4166083d289bdeff221ac6d3ee5046aef2d480dc4b503c"}, + {file = "contourpy-1.0.7-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:769eef00437edf115e24d87f8926955f00f7704bede656ce605097584f9966dc"}, + {file = "contourpy-1.0.7-cp38-cp38-win32.whl", hash = "sha256:62398c80ef57589bdbe1eb8537127321c1abcfdf8c5f14f479dbbe27d0322e66"}, + {file = "contourpy-1.0.7-cp38-cp38-win_amd64.whl", hash = "sha256:57119b0116e3f408acbdccf9eb6ef19d7fe7baf0d1e9aaa5381489bc1aa56556"}, + {file = "contourpy-1.0.7-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:30676ca45084ee61e9c3da589042c24a57592e375d4b138bd84d8709893a1ba4"}, + {file = "contourpy-1.0.7-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:3e927b3868bd1e12acee7cc8f3747d815b4ab3e445a28d2e5373a7f4a6e76ba1"}, + {file = "contourpy-1.0.7-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:366a0cf0fc079af5204801786ad7a1c007714ee3909e364dbac1729f5b0849e5"}, + {file = "contourpy-1.0.7-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:89ba9bb365446a22411f0673abf6ee1fea3b2cf47b37533b970904880ceb72f3"}, + {file = "contourpy-1.0.7-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:71b0bf0c30d432278793d2141362ac853859e87de0a7dee24a1cea35231f0d50"}, + {file = "contourpy-1.0.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e7281244c99fd7c6f27c1c6bfafba878517b0b62925a09b586d88ce750a016d2"}, + {file = "contourpy-1.0.7-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:b6d0f9e1d39dbfb3977f9dd79f156c86eb03e57a7face96f199e02b18e58d32a"}, + {file = "contourpy-1.0.7-cp39-cp39-musllinux_1_1_i686.whl", hash = "sha256:7f6979d20ee5693a1057ab53e043adffa1e7418d734c1532e2d9e915b08d8ec2"}, + {file = "contourpy-1.0.7-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:5dd34c1ae752515318224cba7fc62b53130c45ac6a1040c8b7c1a223c46e8967"}, + {file = "contourpy-1.0.7-cp39-cp39-win32.whl", hash = "sha256:c5210e5d5117e9aec8c47d9156d1d3835570dd909a899171b9535cb4a3f32693"}, + {file = "contourpy-1.0.7-cp39-cp39-win_amd64.whl", hash = "sha256:60835badb5ed5f4e194a6f21c09283dd6e007664a86101431bf870d9e86266c4"}, + {file = "contourpy-1.0.7-pp38-pypy38_pp73-macosx_10_9_x86_64.whl", hash = "sha256:ce41676b3d0dd16dbcfabcc1dc46090aaf4688fd6e819ef343dbda5a57ef0161"}, + {file = "contourpy-1.0.7-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5a011cf354107b47c58ea932d13b04d93c6d1d69b8b6dce885e642531f847566"}, + {file = "contourpy-1.0.7-pp38-pypy38_pp73-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:31a55dccc8426e71817e3fe09b37d6d48ae40aae4ecbc8c7ad59d6893569c436"}, + {file = "contourpy-1.0.7-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:69f8ff4db108815addd900a74df665e135dbbd6547a8a69333a68e1f6e368ac2"}, + {file = "contourpy-1.0.7-pp38-pypy38_pp73-win_amd64.whl", hash = "sha256:efe99298ba37e37787f6a2ea868265465410822f7bea163edcc1bd3903354ea9"}, + {file = "contourpy-1.0.7-pp39-pypy39_pp73-macosx_10_9_x86_64.whl", hash = "sha256:a1e97b86f73715e8670ef45292d7cc033548266f07d54e2183ecb3c87598888f"}, + {file = "contourpy-1.0.7-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cc331c13902d0f50845099434cd936d49d7a2ca76cb654b39691974cb1e4812d"}, + {file = "contourpy-1.0.7-pp39-pypy39_pp73-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:24847601071f740837aefb730e01bd169fbcaa610209779a78db7ebb6e6a7051"}, + {file = "contourpy-1.0.7-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:abf298af1e7ad44eeb93501e40eb5a67abbf93b5d90e468d01fc0c4451971afa"}, + {file = "contourpy-1.0.7-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:64757f6460fc55d7e16ed4f1de193f362104285c667c112b50a804d482777edd"}, + {file = "contourpy-1.0.7.tar.gz", hash = "sha256:d8165a088d31798b59e91117d1f5fc3df8168d8b48c4acc10fc0df0d0bdbcc5e"}, +] + +[package.dependencies] +numpy = ">=1.16" + +[package.extras] +bokeh = ["bokeh", "chromedriver", "selenium"] +docs = ["furo", "sphinx-copybutton"] +mypy = ["contourpy[bokeh]", "docutils-stubs", "mypy (==0.991)", "types-Pillow"] +test = ["Pillow", "matplotlib", "pytest"] +test-no-images = ["pytest"] + +[[package]] +name = "coverage" +version = "7.2.3" +description = "Code coverage measurement for Python" +optional = false +python-versions = ">=3.7" +files = [ + {file = "coverage-7.2.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:e58c0d41d336569d63d1b113bd573db8363bc4146f39444125b7f8060e4e04f5"}, + {file = "coverage-7.2.3-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:344e714bd0fe921fc72d97404ebbdbf9127bac0ca1ff66d7b79efc143cf7c0c4"}, + {file = "coverage-7.2.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:974bc90d6f6c1e59ceb1516ab00cf1cdfbb2e555795d49fa9571d611f449bcb2"}, + {file = "coverage-7.2.3-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:0743b0035d4b0e32bc1df5de70fba3059662ace5b9a2a86a9f894cfe66569013"}, + {file = "coverage-7.2.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5d0391fb4cfc171ce40437f67eb050a340fdbd0f9f49d6353a387f1b7f9dd4fa"}, + {file = "coverage-7.2.3-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:4a42e1eff0ca9a7cb7dc9ecda41dfc7cbc17cb1d02117214be0561bd1134772b"}, + {file = "coverage-7.2.3-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:be19931a8dcbe6ab464f3339966856996b12a00f9fe53f346ab3be872d03e257"}, + {file = "coverage-7.2.3-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:72fcae5bcac3333a4cf3b8f34eec99cea1187acd55af723bcbd559adfdcb5535"}, + {file = "coverage-7.2.3-cp310-cp310-win32.whl", hash = "sha256:aeae2aa38395b18106e552833f2a50c27ea0000122bde421c31d11ed7e6f9c91"}, + {file = "coverage-7.2.3-cp310-cp310-win_amd64.whl", hash = "sha256:83957d349838a636e768251c7e9979e899a569794b44c3728eaebd11d848e58e"}, + {file = "coverage-7.2.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:dfd393094cd82ceb9b40df4c77976015a314b267d498268a076e940fe7be6b79"}, + {file = "coverage-7.2.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:182eb9ac3f2b4874a1f41b78b87db20b66da6b9cdc32737fbbf4fea0c35b23fc"}, + {file = "coverage-7.2.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1bb1e77a9a311346294621be905ea8a2c30d3ad371fc15bb72e98bfcfae532df"}, + {file = "coverage-7.2.3-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:ca0f34363e2634deffd390a0fef1aa99168ae9ed2af01af4a1f5865e362f8623"}, + {file = "coverage-7.2.3-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:55416d7385774285b6e2a5feca0af9652f7f444a4fa3d29d8ab052fafef9d00d"}, + {file = "coverage-7.2.3-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:06ddd9c0249a0546997fdda5a30fbcb40f23926df0a874a60a8a185bc3a87d93"}, + {file = "coverage-7.2.3-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:fff5aaa6becf2c6a1699ae6a39e2e6fb0672c2d42eca8eb0cafa91cf2e9bd312"}, + {file = "coverage-7.2.3-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:ea53151d87c52e98133eb8ac78f1206498c015849662ca8dc246255265d9c3c4"}, + {file = "coverage-7.2.3-cp311-cp311-win32.whl", hash = "sha256:8f6c930fd70d91ddee53194e93029e3ef2aabe26725aa3c2753df057e296b925"}, + {file = "coverage-7.2.3-cp311-cp311-win_amd64.whl", hash = "sha256:fa546d66639d69aa967bf08156eb8c9d0cd6f6de84be9e8c9819f52ad499c910"}, + {file = "coverage-7.2.3-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:b2317d5ed777bf5a033e83d4f1389fd4ef045763141d8f10eb09a7035cee774c"}, + {file = "coverage-7.2.3-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:be9824c1c874b73b96288c6d3de793bf7f3a597770205068c6163ea1f326e8b9"}, + {file = "coverage-7.2.3-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:2c3b2803e730dc2797a017335827e9da6da0e84c745ce0f552e66400abdfb9a1"}, + {file = "coverage-7.2.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8f69770f5ca1994cb32c38965e95f57504d3aea96b6c024624fdd5bb1aa494a1"}, + {file = "coverage-7.2.3-cp37-cp37m-musllinux_1_1_aarch64.whl", hash = "sha256:1127b16220f7bfb3f1049ed4a62d26d81970a723544e8252db0efde853268e21"}, + {file = "coverage-7.2.3-cp37-cp37m-musllinux_1_1_i686.whl", hash = "sha256:aa784405f0c640940595fa0f14064d8e84aff0b0f762fa18393e2760a2cf5841"}, + {file = "coverage-7.2.3-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:3146b8e16fa60427e03884301bf8209221f5761ac754ee6b267642a2fd354c48"}, + {file = "coverage-7.2.3-cp37-cp37m-win32.whl", hash = "sha256:1fd78b911aea9cec3b7e1e2622c8018d51c0d2bbcf8faaf53c2497eb114911c1"}, + {file = "coverage-7.2.3-cp37-cp37m-win_amd64.whl", hash = "sha256:0f3736a5d34e091b0a611964c6262fd68ca4363df56185902528f0b75dbb9c1f"}, + {file = "coverage-7.2.3-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:981b4df72c93e3bc04478153df516d385317628bd9c10be699c93c26ddcca8ab"}, + {file = "coverage-7.2.3-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:c0045f8f23a5fb30b2eb3b8a83664d8dc4fb58faddf8155d7109166adb9f2040"}, + {file = "coverage-7.2.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f760073fcf8f3d6933178d67754f4f2d4e924e321f4bb0dcef0424ca0215eba1"}, + {file = "coverage-7.2.3-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:c86bd45d1659b1ae3d0ba1909326b03598affbc9ed71520e0ff8c31a993ad911"}, + {file = "coverage-7.2.3-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:172db976ae6327ed4728e2507daf8a4de73c7cc89796483e0a9198fd2e47b462"}, + {file = "coverage-7.2.3-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:d2a3a6146fe9319926e1d477842ca2a63fe99af5ae690b1f5c11e6af074a6b5c"}, + {file = "coverage-7.2.3-cp38-cp38-musllinux_1_1_i686.whl", hash = "sha256:f649dd53833b495c3ebd04d6eec58479454a1784987af8afb77540d6c1767abd"}, + {file = "coverage-7.2.3-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:7c4ed4e9f3b123aa403ab424430b426a1992e6f4c8fd3cb56ea520446e04d152"}, + {file = "coverage-7.2.3-cp38-cp38-win32.whl", hash = "sha256:eb0edc3ce9760d2f21637766c3aa04822030e7451981ce569a1b3456b7053f22"}, + {file = "coverage-7.2.3-cp38-cp38-win_amd64.whl", hash = "sha256:63cdeaac4ae85a179a8d6bc09b77b564c096250d759eed343a89d91bce8b6367"}, + {file = "coverage-7.2.3-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:20d1a2a76bb4eb00e4d36b9699f9b7aba93271c9c29220ad4c6a9581a0320235"}, + {file = "coverage-7.2.3-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:4ea748802cc0de4de92ef8244dd84ffd793bd2e7be784cd8394d557a3c751e21"}, + {file = "coverage-7.2.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:21b154aba06df42e4b96fc915512ab39595105f6c483991287021ed95776d934"}, + {file = "coverage-7.2.3-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:fd214917cabdd6f673a29d708574e9fbdb892cb77eb426d0eae3490d95ca7859"}, + {file = "coverage-7.2.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2c2e58e45fe53fab81f85474e5d4d226eeab0f27b45aa062856c89389da2f0d9"}, + {file = "coverage-7.2.3-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:87ecc7c9a1a9f912e306997ffee020297ccb5ea388421fe62a2a02747e4d5539"}, + {file = "coverage-7.2.3-cp39-cp39-musllinux_1_1_i686.whl", hash = "sha256:387065e420aed3c71b61af7e82c7b6bc1c592f7e3c7a66e9f78dd178699da4fe"}, + {file = "coverage-7.2.3-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:ea3f5bc91d7d457da7d48c7a732beaf79d0c8131df3ab278e6bba6297e23c6c4"}, + {file = "coverage-7.2.3-cp39-cp39-win32.whl", hash = "sha256:ae7863a1d8db6a014b6f2ff9c1582ab1aad55a6d25bac19710a8df68921b6e30"}, + {file = "coverage-7.2.3-cp39-cp39-win_amd64.whl", hash = "sha256:3f04becd4fcda03c0160d0da9c8f0c246bc78f2f7af0feea1ec0930e7c93fa4a"}, + {file = "coverage-7.2.3-pp37.pp38.pp39-none-any.whl", hash = "sha256:965ee3e782c7892befc25575fa171b521d33798132692df428a09efacaffe8d0"}, + {file = "coverage-7.2.3.tar.gz", hash = "sha256:d298c2815fa4891edd9abe5ad6e6cb4207104c7dd9fd13aea3fdebf6f9b91259"}, +] + +[package.dependencies] +tomli = {version = "*", optional = true, markers = "python_full_version <= \"3.11.0a6\" and extra == \"toml\""} + +[package.extras] +toml = ["tomli"] + +[[package]] +name = "cycler" +version = "0.11.0" +description = "Composable style cycles" +optional = false +python-versions = ">=3.6" +files = [ + {file = "cycler-0.11.0-py3-none-any.whl", hash = "sha256:3a27e95f763a428a739d2add979fa7494c912a32c17c4c38c4d5f082cad165a3"}, + {file = "cycler-0.11.0.tar.gz", hash = "sha256:9c87405839a19696e837b3b818fed3f5f69f16f1eec1a1ad77e043dcea9c772f"}, +] + +[[package]] +name = "debugpy" +version = "1.6.7" +description = "An implementation of the Debug Adapter Protocol for Python" +optional = false +python-versions = ">=3.7" +files = [ + {file = "debugpy-1.6.7-cp310-cp310-macosx_11_0_x86_64.whl", hash = "sha256:b3e7ac809b991006ad7f857f016fa92014445085711ef111fdc3f74f66144096"}, + {file = "debugpy-1.6.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e3876611d114a18aafef6383695dfc3f1217c98a9168c1aaf1a02b01ec7d8d1e"}, + {file = "debugpy-1.6.7-cp310-cp310-win32.whl", hash = "sha256:33edb4afa85c098c24cc361d72ba7c21bb92f501104514d4ffec1fb36e09c01a"}, + {file = "debugpy-1.6.7-cp310-cp310-win_amd64.whl", hash = "sha256:ed6d5413474e209ba50b1a75b2d9eecf64d41e6e4501977991cdc755dc83ab0f"}, + {file = "debugpy-1.6.7-cp37-cp37m-macosx_10_15_x86_64.whl", hash = "sha256:38ed626353e7c63f4b11efad659be04c23de2b0d15efff77b60e4740ea685d07"}, + {file = "debugpy-1.6.7-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:279d64c408c60431c8ee832dfd9ace7c396984fd7341fa3116aee414e7dcd88d"}, + {file = "debugpy-1.6.7-cp37-cp37m-win32.whl", hash = "sha256:dbe04e7568aa69361a5b4c47b4493d5680bfa3a911d1e105fbea1b1f23f3eb45"}, + {file = "debugpy-1.6.7-cp37-cp37m-win_amd64.whl", hash = "sha256:f90a2d4ad9a035cee7331c06a4cf2245e38bd7c89554fe3b616d90ab8aab89cc"}, + {file = "debugpy-1.6.7-cp38-cp38-macosx_10_15_x86_64.whl", hash = "sha256:5224eabbbeddcf1943d4e2821876f3e5d7d383f27390b82da5d9558fd4eb30a9"}, + {file = "debugpy-1.6.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bae1123dff5bfe548ba1683eb972329ba6d646c3a80e6b4c06cd1b1dd0205e9b"}, + {file = "debugpy-1.6.7-cp38-cp38-win32.whl", hash = "sha256:9cd10cf338e0907fdcf9eac9087faa30f150ef5445af5a545d307055141dd7a4"}, + {file = "debugpy-1.6.7-cp38-cp38-win_amd64.whl", hash = "sha256:aaf6da50377ff4056c8ed470da24632b42e4087bc826845daad7af211e00faad"}, + {file = "debugpy-1.6.7-cp39-cp39-macosx_11_0_x86_64.whl", hash = "sha256:0679b7e1e3523bd7d7869447ec67b59728675aadfc038550a63a362b63029d2c"}, + {file = "debugpy-1.6.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:de86029696e1b3b4d0d49076b9eba606c226e33ae312a57a46dca14ff370894d"}, + {file = "debugpy-1.6.7-cp39-cp39-win32.whl", hash = "sha256:d71b31117779d9a90b745720c0eab54ae1da76d5b38c8026c654f4a066b0130a"}, + {file = "debugpy-1.6.7-cp39-cp39-win_amd64.whl", hash = "sha256:c0ff93ae90a03b06d85b2c529eca51ab15457868a377c4cc40a23ab0e4e552a3"}, + {file = "debugpy-1.6.7-py2.py3-none-any.whl", hash = "sha256:53f7a456bc50706a0eaabecf2d3ce44c4d5010e46dfc65b6b81a518b42866267"}, + {file = "debugpy-1.6.7.zip", hash = "sha256:c4c2f0810fa25323abfdfa36cbbbb24e5c3b1a42cb762782de64439c575d67f2"}, +] + +[[package]] +name = "decorator" +version = "5.1.1" +description = "Decorators for Humans" +optional = false +python-versions = ">=3.5" +files = [ + {file = "decorator-5.1.1-py3-none-any.whl", hash = "sha256:b8c3f85900b9dc423225913c5aace94729fe1fa9763b38939a95226f02d37186"}, + {file = "decorator-5.1.1.tar.gz", hash = "sha256:637996211036b6385ef91435e4fae22989472f9d571faba8927ba8253acbc330"}, +] + +[[package]] +name = "defusedxml" +version = "0.7.1" +description = "XML bomb protection for Python stdlib modules" +optional = false +python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*" +files = [ + {file = "defusedxml-0.7.1-py2.py3-none-any.whl", hash = "sha256:a352e7e428770286cc899e2542b6cdaedb2b4953ff269a210103ec58f6198a61"}, + {file = "defusedxml-0.7.1.tar.gz", hash = "sha256:1bb3032db185915b62d7c6209c5a8792be6a32ab2fedacc84e01b52c51aa3e69"}, +] + +[[package]] +name = "deprecation" +version = "2.1.0" +description = "A library to handle automated deprecations" +optional = false +python-versions = "*" +files = [ + {file = "deprecation-2.1.0-py2.py3-none-any.whl", hash = "sha256:a10811591210e1fb0e768a8c25517cabeabcba6f0bf96564f8ff45189f90b14a"}, + {file = "deprecation-2.1.0.tar.gz", hash = "sha256:72b3bde64e5d778694b0cf68178aed03d15e15477116add3fb773e581f9518ff"}, +] + +[package.dependencies] +packaging = "*" + +[[package]] +name = "distlib" +version = "0.3.6" +description = "Distribution utilities" +optional = false +python-versions = "*" +files = [ + {file = "distlib-0.3.6-py2.py3-none-any.whl", hash = "sha256:f35c4b692542ca110de7ef0bea44d73981caeb34ca0b9b6b2e6d7790dda8f80e"}, + {file = "distlib-0.3.6.tar.gz", hash = "sha256:14bad2d9b04d3a36127ac97f30b12a19268f211063d8f8ee4f47108896e11b46"}, +] + +[[package]] +name = "exceptiongroup" +version = "1.1.1" +description = "Backport of PEP 654 (exception groups)" +optional = false +python-versions = ">=3.7" +files = [ + {file = "exceptiongroup-1.1.1-py3-none-any.whl", hash = "sha256:232c37c63e4f682982c8b6459f33a8981039e5fb8756b2074364e5055c498c9e"}, + {file = "exceptiongroup-1.1.1.tar.gz", hash = "sha256:d484c3090ba2889ae2928419117447a14daf3c1231d5e30d0aae34f354f01785"}, +] + +[package.extras] +test = ["pytest (>=6)"] + +[[package]] +name = "executing" +version = "1.2.0" +description = "Get the currently executing AST node of a frame, and other information" +optional = false +python-versions = "*" +files = [ + {file = "executing-1.2.0-py2.py3-none-any.whl", hash = "sha256:0314a69e37426e3608aada02473b4161d4caf5a4b244d1d0c48072b8fee7bacc"}, + {file = "executing-1.2.0.tar.gz", hash = "sha256:19da64c18d2d851112f09c287f8d3dbbdf725ab0e569077efb6cdcbd3497c107"}, +] + +[package.extras] +tests = ["asttokens", "littleutils", "pytest", "rich"] + +[[package]] +name = "fastjsonschema" +version = "2.16.3" +description = "Fastest Python implementation of JSON schema" +optional = false +python-versions = "*" +files = [ + {file = "fastjsonschema-2.16.3-py3-none-any.whl", hash = "sha256:04fbecc94300436f628517b05741b7ea009506ce8f946d40996567c669318490"}, + {file = "fastjsonschema-2.16.3.tar.gz", hash = "sha256:4a30d6315a68c253cfa8f963b9697246315aa3db89f98b97235e345dedfb0b8e"}, +] + +[package.extras] +devel = ["colorama", "json-spec", "jsonschema", "pylint", "pytest", "pytest-benchmark", "pytest-cache", "validictory"] + +[[package]] +name = "filelock" +version = "3.12.0" +description = "A platform independent file lock." +optional = false +python-versions = ">=3.7" +files = [ + {file = "filelock-3.12.0-py3-none-any.whl", hash = "sha256:ad98852315c2ab702aeb628412cbf7e95b7ce8c3bf9565670b4eaecf1db370a9"}, + {file = "filelock-3.12.0.tar.gz", hash = "sha256:fc03ae43288c013d2ea83c8597001b1129db351aad9c57fe2409327916b8e718"}, +] + +[package.extras] +docs = ["furo (>=2023.3.27)", "sphinx (>=6.1.3)", "sphinx-autodoc-typehints (>=1.23,!=1.23.4)"] +testing = ["covdefaults (>=2.3)", "coverage (>=7.2.3)", "diff-cover (>=7.5)", "pytest (>=7.3.1)", "pytest-cov (>=4)", "pytest-mock (>=3.10)", "pytest-timeout (>=2.1)"] + +[[package]] +name = "fiona" +version = "1.9.3" +description = "Fiona reads and writes spatial data files" +optional = false +python-versions = ">=3.7" +files = [ + {file = "Fiona-1.9.3-cp310-cp310-macosx_10_15_x86_64.whl", hash = "sha256:0e9141bdb8031419ed2f04c6da02ae12c3044a81987065e05ff40f39cc35e042"}, + {file = "Fiona-1.9.3-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:6c0251a57305e6bea3f0a8e8306c0bd05e2b0e30b8a294d7bdc429d5fceca68d"}, + {file = "Fiona-1.9.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:894127efde8141bb9383dc4dc890c732f3bfe4d601c3d1020a24fa3c24a8c4a8"}, + {file = "Fiona-1.9.3-cp310-cp310-win_amd64.whl", hash = "sha256:11ee3d3e6bb5d16f6f1643ffcde7ac4dfa5fbe98a26ce2af05c3c5426ce248d7"}, + {file = "Fiona-1.9.3-cp311-cp311-macosx_10_15_x86_64.whl", hash = "sha256:c99e9bca9e3d6be03a71e9b2f6ba66d446eae9b27df37c1f6b45483b2f215ca0"}, + {file = "Fiona-1.9.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:9a894362c1cf9f33ee931e96cfd4021d3a18f6ccf8c36b87df42a0a494e23545"}, + {file = "Fiona-1.9.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8b0761ff656d07aaef7a7274b74816e16485f0f15e77a962c107cd4a1cfb4757"}, + {file = "Fiona-1.9.3-cp311-cp311-win_amd64.whl", hash = "sha256:2e61caeabda88ab5fa45db373c2afd6913844b4452c0f2e3e9d924c60bc76fa3"}, + {file = "Fiona-1.9.3-cp37-cp37m-macosx_10_15_x86_64.whl", hash = "sha256:00628c5a3dd7e9bc037ba0487fc3b9f7163107e0a9794bd4c32c471ab65f3a45"}, + {file = "Fiona-1.9.3-cp37-cp37m-manylinux2014_x86_64.whl", hash = "sha256:95927ddd9afafdb0243bb83bf234557dcdb35bf0e888fd920ff82ffa80f6a53a"}, + {file = "Fiona-1.9.3-cp37-cp37m-win_amd64.whl", hash = "sha256:d1064e82a7fed73ce60ce9ce4f65b5a6558fb5b532a13130a17f132ed122ec75"}, + {file = "Fiona-1.9.3-cp38-cp38-macosx_10_15_x86_64.whl", hash = "sha256:65b096148bfe9a64d87d91ba8e7ff940a5aef8cbffc6738a70e289c6384e1cca"}, + {file = "Fiona-1.9.3-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:38d0d78d4e061592af3441c5962072b0456307246c9c6f412ad38ebef11d2903"}, + {file = "Fiona-1.9.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ee9b2ec9f0fb4b3798d607a94a5586b403fc27fea06e3e7ac2924c0785d4df61"}, + {file = "Fiona-1.9.3-cp38-cp38-win_amd64.whl", hash = "sha256:258151f26683a44ed715c09930a42e0b39b3b3444b438ec6e32633f7056740fa"}, + {file = "Fiona-1.9.3-cp39-cp39-macosx_10_15_x86_64.whl", hash = "sha256:f1fcadad17b00d342532dc51a47128005f8ced01a320fa6b72c8ef669edf3057"}, + {file = "Fiona-1.9.3-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:85b6694227ee4e00dfa52c6a9fcc89f1051aaf67df5fbd1faa33fb02c62a6203"}, + {file = "Fiona-1.9.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e661deb7a8722839bd27eae74f63f0e480559774cc755598dfa6c51bdf18be3d"}, + {file = "Fiona-1.9.3-cp39-cp39-win_amd64.whl", hash = "sha256:a57812a584b4a2fb4ffdfaa9135dc38312989f7cd2823ecbd23e11eade5eb7fe"}, + {file = "Fiona-1.9.3.tar.gz", hash = "sha256:60f3789ad9633c3a26acf7cbe39e82e3c7a12562c59af1d599fc3e4e8f7f8f25"}, +] + +[package.dependencies] +attrs = ">=19.2.0" +certifi = "*" +click = ">=8.0,<9.0" +click-plugins = ">=1.0" +cligj = ">=0.5" +importlib-metadata = {version = "*", markers = "python_version < \"3.10\""} +munch = ">=2.3.2" + +[package.extras] +all = ["Fiona[calc,s3,test]"] +calc = ["shapely"] +s3 = ["boto3 (>=1.3.1)"] +test = ["Fiona[s3]", "pytest (>=7)", "pytest-cov", "pytz"] + +[[package]] +name = "fonttools" +version = "4.39.3" +description = "Tools to manipulate font files" +optional = false +python-versions = ">=3.8" +files = [ + {file = "fonttools-4.39.3-py3-none-any.whl", hash = "sha256:64c0c05c337f826183637570ac5ab49ee220eec66cf50248e8df527edfa95aeb"}, + {file = "fonttools-4.39.3.zip", hash = "sha256:9234b9f57b74e31b192c3fc32ef1a40750a8fbc1cd9837a7b7bfc4ca4a5c51d7"}, +] + +[package.extras] +all = ["brotli (>=1.0.1)", "brotlicffi (>=0.8.0)", "fs (>=2.2.0,<3)", "lxml (>=4.0,<5)", "lz4 (>=1.7.4.2)", "matplotlib", "munkres", "scipy", "skia-pathops (>=0.5.0)", "sympy", "uharfbuzz (>=0.23.0)", "unicodedata2 (>=15.0.0)", "xattr", "zopfli (>=0.1.4)"] +graphite = ["lz4 (>=1.7.4.2)"] +interpolatable = ["munkres", "scipy"] +lxml = ["lxml (>=4.0,<5)"] +pathops = ["skia-pathops (>=0.5.0)"] +plot = ["matplotlib"] +repacker = ["uharfbuzz (>=0.23.0)"] +symfont = ["sympy"] +type1 = ["xattr"] +ufo = ["fs (>=2.2.0,<3)"] +unicode = ["unicodedata2 (>=15.0.0)"] +woff = ["brotli (>=1.0.1)", "brotlicffi (>=0.8.0)", "zopfli (>=0.1.4)"] + +[[package]] +name = "geopandas" +version = "0.12.2" +description = "Geographic pandas extensions" +optional = false +python-versions = ">=3.8" +files = [ + {file = "geopandas-0.12.2-py3-none-any.whl", hash = "sha256:0a470e4bf6f5367e6fd83ab6b40405e0b805c8174665bbcb7c4077ed90202912"}, + {file = "geopandas-0.12.2.tar.gz", hash = "sha256:0acdacddefa176525e4da6d9aeeece225da26055c4becdc6e97cf40fa97c27f4"}, +] + +[package.dependencies] +fiona = ">=1.8" +packaging = "*" +pandas = ">=1.0.0" +pyproj = ">=2.6.1.post1" +shapely = ">=1.7" + +[[package]] +name = "identify" +version = "2.5.22" +description = "File identification library for Python" +optional = false +python-versions = ">=3.7" +files = [ + {file = "identify-2.5.22-py2.py3-none-any.whl", hash = "sha256:f0faad595a4687053669c112004178149f6c326db71ee999ae4636685753ad2f"}, + {file = "identify-2.5.22.tar.gz", hash = "sha256:f7a93d6cf98e29bd07663c60728e7a4057615068d7a639d132dc883b2d54d31e"}, +] + +[package.extras] +license = ["ukkonen"] + +[[package]] +name = "importlib-metadata" +version = "6.6.0" +description = "Read metadata from Python packages" +optional = false +python-versions = ">=3.7" +files = [ + {file = "importlib_metadata-6.6.0-py3-none-any.whl", hash = "sha256:43dd286a2cd8995d5eaef7fee2066340423b818ed3fd70adf0bad5f1fac53fed"}, + {file = "importlib_metadata-6.6.0.tar.gz", hash = "sha256:92501cdf9cc66ebd3e612f1b4f0c0765dfa42f0fa38ffb319b6bd84dd675d705"}, +] + +[package.dependencies] +zipp = ">=0.5" + +[package.extras] +docs = ["furo", "jaraco.packaging (>=9)", "jaraco.tidelift (>=1.4)", "rst.linker (>=1.9)", "sphinx (>=3.5)", "sphinx-lint"] +perf = ["ipython"] +testing = ["flake8 (<5)", "flufl.flake8", "importlib-resources (>=1.3)", "packaging", "pyfakefs", "pytest (>=6)", "pytest-black (>=0.3.7)", "pytest-checkdocs (>=2.4)", "pytest-cov", "pytest-enabler (>=1.3)", "pytest-flake8", "pytest-mypy (>=0.9.1)", "pytest-perf (>=0.9.2)"] + +[[package]] +name = "importlib-resources" +version = "5.12.0" +description = "Read resources from Python packages" +optional = false +python-versions = ">=3.7" +files = [ + {file = "importlib_resources-5.12.0-py3-none-any.whl", hash = "sha256:7b1deeebbf351c7578e09bf2f63fa2ce8b5ffec296e0d349139d43cca061a81a"}, + {file = "importlib_resources-5.12.0.tar.gz", hash = "sha256:4be82589bf5c1d7999aedf2a45159d10cb3ca4f19b2271f8792bc8e6da7b22f6"}, +] + +[package.dependencies] +zipp = {version = ">=3.1.0", markers = "python_version < \"3.10\""} + +[package.extras] +docs = ["furo", "jaraco.packaging (>=9)", "jaraco.tidelift (>=1.4)", "rst.linker (>=1.9)", "sphinx (>=3.5)", "sphinx-lint"] +testing = ["flake8 (<5)", "pytest (>=6)", "pytest-black (>=0.3.7)", "pytest-checkdocs (>=2.4)", "pytest-cov", "pytest-enabler (>=1.3)", "pytest-flake8", "pytest-mypy (>=0.9.1)"] + +[[package]] +name = "iniconfig" +version = "2.0.0" +description = "brain-dead simple config-ini parsing" +optional = false +python-versions = ">=3.7" +files = [ + {file = "iniconfig-2.0.0-py3-none-any.whl", hash = "sha256:b6a85871a79d2e3b22d2d1b94ac2824226a63c6b741c88f7ae975f18b6778374"}, + {file = "iniconfig-2.0.0.tar.gz", hash = "sha256:2d91e135bf72d31a410b17c16da610a82cb55f6b0477d1a902134b24a455b8b3"}, +] + +[[package]] +name = "ipykernel" +version = "6.22.0" +description = "IPython Kernel for Jupyter" +optional = false +python-versions = ">=3.8" +files = [ + {file = "ipykernel-6.22.0-py3-none-any.whl", hash = "sha256:1ae6047c1277508933078163721bbb479c3e7292778a04b4bacf0874550977d6"}, + {file = "ipykernel-6.22.0.tar.gz", hash = "sha256:302558b81f1bc22dc259fb2a0c5c7cf2f4c0bdb21b50484348f7bafe7fb71421"}, +] + +[package.dependencies] +appnope = {version = "*", markers = "platform_system == \"Darwin\""} +comm = ">=0.1.1" +debugpy = ">=1.6.5" +ipython = ">=7.23.1" +jupyter-client = ">=6.1.12" +jupyter-core = ">=4.12,<5.0.dev0 || >=5.1.dev0" +matplotlib-inline = ">=0.1" +nest-asyncio = "*" +packaging = "*" +psutil = "*" +pyzmq = ">=20" +tornado = ">=6.1" +traitlets = ">=5.4.0" + +[package.extras] +cov = ["coverage[toml]", "curio", "matplotlib", "pytest-cov", "trio"] +docs = ["myst-parser", "pydata-sphinx-theme", "sphinx", "sphinx-autodoc-typehints", "sphinxcontrib-github-alt", "sphinxcontrib-spelling", "trio"] +pyqt5 = ["pyqt5"] +pyside6 = ["pyside6"] +test = ["flaky", "ipyparallel", "pre-commit", "pytest (>=7.0)", "pytest-asyncio", "pytest-cov", "pytest-timeout"] + +[[package]] +name = "ipython" +version = "8.12.0" +description = "IPython: Productive Interactive Computing" +optional = false +python-versions = ">=3.8" +files = [ + {file = "ipython-8.12.0-py3-none-any.whl", hash = "sha256:1c183bf61b148b00bcebfa5d9b39312733ae97f6dad90d7e9b4d86c8647f498c"}, + {file = "ipython-8.12.0.tar.gz", hash = "sha256:a950236df04ad75b5bc7f816f9af3d74dc118fd42f2ff7e80e8e60ca1f182e2d"}, +] + +[package.dependencies] +appnope = {version = "*", markers = "sys_platform == \"darwin\""} +backcall = "*" +colorama = {version = "*", markers = "sys_platform == \"win32\""} +decorator = "*" +jedi = ">=0.16" +matplotlib-inline = "*" +pexpect = {version = ">4.3", markers = "sys_platform != \"win32\""} +pickleshare = "*" +prompt-toolkit = ">=3.0.30,<3.0.37 || >3.0.37,<3.1.0" +pygments = ">=2.4.0" +stack-data = "*" +traitlets = ">=5" +typing-extensions = {version = "*", markers = "python_version < \"3.10\""} + +[package.extras] +all = ["black", "curio", "docrepr", "ipykernel", "ipyparallel", "ipywidgets", "matplotlib", "matplotlib (!=3.2.0)", "nbconvert", "nbformat", "notebook", "numpy (>=1.21)", "pandas", "pytest (<7)", "pytest (<7.1)", "pytest-asyncio", "qtconsole", "setuptools (>=18.5)", "sphinx (>=1.3)", "sphinx-rtd-theme", "stack-data", "testpath", "trio", "typing-extensions"] +black = ["black"] +doc = ["docrepr", "ipykernel", "matplotlib", "pytest (<7)", "pytest (<7.1)", "pytest-asyncio", "setuptools (>=18.5)", "sphinx (>=1.3)", "sphinx-rtd-theme", "stack-data", "testpath", "typing-extensions"] +kernel = ["ipykernel"] +nbconvert = ["nbconvert"] +nbformat = ["nbformat"] +notebook = ["ipywidgets", "notebook"] +parallel = ["ipyparallel"] +qtconsole = ["qtconsole"] +test = ["pytest (<7.1)", "pytest-asyncio", "testpath"] +test-extra = ["curio", "matplotlib (!=3.2.0)", "nbformat", "numpy (>=1.21)", "pandas", "pytest (<7.1)", "pytest-asyncio", "testpath", "trio"] + +[[package]] +name = "ipython-genutils" +version = "0.2.0" +description = "Vestigial utilities from IPython" +optional = false +python-versions = "*" +files = [ + {file = "ipython_genutils-0.2.0-py2.py3-none-any.whl", hash = "sha256:72dd37233799e619666c9f639a9da83c34013a73e8bbc79a7a6348d93c61fab8"}, + {file = "ipython_genutils-0.2.0.tar.gz", hash = "sha256:eb2e116e75ecef9d4d228fdc66af54269afa26ab4463042e33785b887c628ba8"}, +] + +[[package]] +name = "isort" +version = "5.12.0" +description = "A Python utility / library to sort Python imports." +optional = false +python-versions = ">=3.8.0" +files = [ + {file = "isort-5.12.0-py3-none-any.whl", hash = "sha256:f84c2818376e66cf843d497486ea8fed8700b340f308f076c6fb1229dff318b6"}, + {file = "isort-5.12.0.tar.gz", hash = "sha256:8bef7dde241278824a6d83f44a544709b065191b95b6e50894bdc722fcba0504"}, +] + +[package.extras] +colors = ["colorama (>=0.4.3)"] +pipfile-deprecated-finder = ["pip-shims (>=0.5.2)", "pipreqs", "requirementslib"] +plugins = ["setuptools"] +requirements-deprecated-finder = ["pip-api", "pipreqs"] + +[[package]] +name = "jedi" +version = "0.18.2" +description = "An autocompletion tool for Python that can be used for text editors." +optional = false +python-versions = ">=3.6" +files = [ + {file = "jedi-0.18.2-py2.py3-none-any.whl", hash = "sha256:203c1fd9d969ab8f2119ec0a3342e0b49910045abe6af0a3ae83a5764d54639e"}, + {file = "jedi-0.18.2.tar.gz", hash = "sha256:bae794c30d07f6d910d32a7048af09b5a39ed740918da923c6b780790ebac612"}, +] + +[package.dependencies] +parso = ">=0.8.0,<0.9.0" + +[package.extras] +docs = ["Jinja2 (==2.11.3)", "MarkupSafe (==1.1.1)", "Pygments (==2.8.1)", "alabaster (==0.7.12)", "babel (==2.9.1)", "chardet (==4.0.0)", "commonmark (==0.8.1)", "docutils (==0.17.1)", "future (==0.18.2)", "idna (==2.10)", "imagesize (==1.2.0)", "mock (==1.0.1)", "packaging (==20.9)", "pyparsing (==2.4.7)", "pytz (==2021.1)", "readthedocs-sphinx-ext (==2.1.4)", "recommonmark (==0.5.0)", "requests (==2.25.1)", "six (==1.15.0)", "snowballstemmer (==2.1.0)", "sphinx (==1.8.5)", "sphinx-rtd-theme (==0.4.3)", "sphinxcontrib-serializinghtml (==1.1.4)", "sphinxcontrib-websupport (==1.2.4)", "urllib3 (==1.26.4)"] +qa = ["flake8 (==3.8.3)", "mypy (==0.782)"] +testing = ["Django (<3.1)", "attrs", "colorama", "docopt", "pytest (<7.0.0)"] + +[[package]] +name = "jinja2" +version = "3.1.2" +description = "A very fast and expressive template engine." +optional = false +python-versions = ">=3.7" +files = [ + {file = "Jinja2-3.1.2-py3-none-any.whl", hash = "sha256:6088930bfe239f0e6710546ab9c19c9ef35e29792895fed6e6e31a023a182a61"}, + {file = "Jinja2-3.1.2.tar.gz", hash = "sha256:31351a702a408a9e7595a8fc6150fc3f43bb6bf7e319770cbc0db9df9437e852"}, +] + +[package.dependencies] +MarkupSafe = ">=2.0" + +[package.extras] +i18n = ["Babel (>=2.7)"] + +[[package]] +name = "jmespath" +version = "1.0.1" +description = "JSON Matching Expressions" +optional = false +python-versions = ">=3.7" +files = [ + {file = "jmespath-1.0.1-py3-none-any.whl", hash = "sha256:02e2e4cc71b5bcab88332eebf907519190dd9e6e82107fa7f83b1003a6252980"}, + {file = "jmespath-1.0.1.tar.gz", hash = "sha256:90261b206d6defd58fdd5e85f478bf633a2901798906be2ad389150c5c60edbe"}, +] + +[[package]] +name = "jsonschema" +version = "4.17.3" +description = "An implementation of JSON Schema validation for Python" +optional = false +python-versions = ">=3.7" +files = [ + {file = "jsonschema-4.17.3-py3-none-any.whl", hash = "sha256:a870ad254da1a8ca84b6a2905cac29d265f805acc57af304784962a2aa6508f6"}, + {file = "jsonschema-4.17.3.tar.gz", hash = "sha256:0f864437ab8b6076ba6707453ef8f98a6a0d512a80e93f8abdb676f737ecb60d"}, +] + +[package.dependencies] +attrs = ">=17.4.0" +importlib-resources = {version = ">=1.4.0", markers = "python_version < \"3.9\""} +pkgutil-resolve-name = {version = ">=1.3.10", markers = "python_version < \"3.9\""} +pyrsistent = ">=0.14.0,<0.17.0 || >0.17.0,<0.17.1 || >0.17.1,<0.17.2 || >0.17.2" + +[package.extras] +format = ["fqdn", "idna", "isoduration", "jsonpointer (>1.13)", "rfc3339-validator", "rfc3987", "uri-template", "webcolors (>=1.11)"] +format-nongpl = ["fqdn", "idna", "isoduration", "jsonpointer (>1.13)", "rfc3339-validator", "rfc3986-validator (>0.1.0)", "uri-template", "webcolors (>=1.11)"] + +[[package]] +name = "jupyter-client" +version = "8.2.0" +description = "Jupyter protocol implementation and client libraries" +optional = false +python-versions = ">=3.8" +files = [ + {file = "jupyter_client-8.2.0-py3-none-any.whl", hash = "sha256:b18219aa695d39e2ad570533e0d71fb7881d35a873051054a84ee2a17c4b7389"}, + {file = "jupyter_client-8.2.0.tar.gz", hash = "sha256:9fe233834edd0e6c0aa5f05ca2ab4bdea1842bfd2d8a932878212fc5301ddaf0"}, +] + +[package.dependencies] +importlib-metadata = {version = ">=4.8.3", markers = "python_version < \"3.10\""} +jupyter-core = ">=4.12,<5.0.dev0 || >=5.1.dev0" +python-dateutil = ">=2.8.2" +pyzmq = ">=23.0" +tornado = ">=6.2" +traitlets = ">=5.3" + +[package.extras] +docs = ["ipykernel", "myst-parser", "pydata-sphinx-theme", "sphinx (>=4)", "sphinx-autodoc-typehints", "sphinxcontrib-github-alt", "sphinxcontrib-spelling"] +test = ["coverage", "ipykernel (>=6.14)", "mypy", "paramiko", "pre-commit", "pytest", "pytest-cov", "pytest-jupyter[client] (>=0.4.1)", "pytest-timeout"] + +[[package]] +name = "jupyter-core" +version = "5.3.0" +description = "Jupyter core package. A base package on which Jupyter projects rely." +optional = false +python-versions = ">=3.8" +files = [ + {file = "jupyter_core-5.3.0-py3-none-any.whl", hash = "sha256:d4201af84559bc8c70cead287e1ab94aeef3c512848dde077b7684b54d67730d"}, + {file = "jupyter_core-5.3.0.tar.gz", hash = "sha256:6db75be0c83edbf1b7c9f91ec266a9a24ef945da630f3120e1a0046dc13713fc"}, +] + +[package.dependencies] +platformdirs = ">=2.5" +pywin32 = {version = ">=300", markers = "sys_platform == \"win32\" and platform_python_implementation != \"PyPy\""} +traitlets = ">=5.3" + +[package.extras] +docs = ["myst-parser", "sphinx-autodoc-typehints", "sphinxcontrib-github-alt", "sphinxcontrib-spelling", "traitlets"] +test = ["ipykernel", "pre-commit", "pytest", "pytest-cov", "pytest-timeout"] + +[[package]] +name = "jupyterlab-pygments" +version = "0.2.2" +description = "Pygments theme using JupyterLab CSS variables" +optional = false +python-versions = ">=3.7" +files = [ + {file = "jupyterlab_pygments-0.2.2-py2.py3-none-any.whl", hash = "sha256:2405800db07c9f770863bcf8049a529c3dd4d3e28536638bd7c1c01d2748309f"}, + {file = "jupyterlab_pygments-0.2.2.tar.gz", hash = "sha256:7405d7fde60819d905a9fa8ce89e4cd830e318cdad22a0030f7a901da705585d"}, +] + +[[package]] +name = "kiwisolver" +version = "1.4.4" +description = "A fast implementation of the Cassowary constraint solver" +optional = false +python-versions = ">=3.7" +files = [ + {file = "kiwisolver-1.4.4-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:2f5e60fabb7343a836360c4f0919b8cd0d6dbf08ad2ca6b9cf90bf0c76a3c4f6"}, + {file = "kiwisolver-1.4.4-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:10ee06759482c78bdb864f4109886dff7b8a56529bc1609d4f1112b93fe6423c"}, + {file = "kiwisolver-1.4.4-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:c79ebe8f3676a4c6630fd3f777f3cfecf9289666c84e775a67d1d358578dc2e3"}, + {file = "kiwisolver-1.4.4-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:abbe9fa13da955feb8202e215c4018f4bb57469b1b78c7a4c5c7b93001699938"}, + {file = "kiwisolver-1.4.4-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:7577c1987baa3adc4b3c62c33bd1118c3ef5c8ddef36f0f2c950ae0b199e100d"}, + {file = "kiwisolver-1.4.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f8ad8285b01b0d4695102546b342b493b3ccc6781fc28c8c6a1bb63e95d22f09"}, + {file = "kiwisolver-1.4.4-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:8ed58b8acf29798b036d347791141767ccf65eee7f26bde03a71c944449e53de"}, + {file = "kiwisolver-1.4.4-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:a68b62a02953b9841730db7797422f983935aeefceb1679f0fc85cbfbd311c32"}, + {file = "kiwisolver-1.4.4-cp310-cp310-win32.whl", hash = "sha256:e92a513161077b53447160b9bd8f522edfbed4bd9759e4c18ab05d7ef7e49408"}, + {file = "kiwisolver-1.4.4-cp310-cp310-win_amd64.whl", hash = "sha256:3fe20f63c9ecee44560d0e7f116b3a747a5d7203376abeea292ab3152334d004"}, + {file = "kiwisolver-1.4.4-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:e0ea21f66820452a3f5d1655f8704a60d66ba1191359b96541eaf457710a5fc6"}, + {file = "kiwisolver-1.4.4-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:bc9db8a3efb3e403e4ecc6cd9489ea2bac94244f80c78e27c31dcc00d2790ac2"}, + {file = "kiwisolver-1.4.4-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:d5b61785a9ce44e5a4b880272baa7cf6c8f48a5180c3e81c59553ba0cb0821ca"}, + {file = "kiwisolver-1.4.4-cp311-cp311-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:c2dbb44c3f7e6c4d3487b31037b1bdbf424d97687c1747ce4ff2895795c9bf69"}, + {file = "kiwisolver-1.4.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:6295ecd49304dcf3bfbfa45d9a081c96509e95f4b9d0eb7ee4ec0530c4a96514"}, + {file = "kiwisolver-1.4.4-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:4bd472dbe5e136f96a4b18f295d159d7f26fd399136f5b17b08c4e5f498cd494"}, + {file = "kiwisolver-1.4.4-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:bf7d9fce9bcc4752ca4a1b80aabd38f6d19009ea5cbda0e0856983cf6d0023f5"}, + {file = "kiwisolver-1.4.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:78d6601aed50c74e0ef02f4204da1816147a6d3fbdc8b3872d263338a9052c51"}, + {file = "kiwisolver-1.4.4-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:877272cf6b4b7e94c9614f9b10140e198d2186363728ed0f701c6eee1baec1da"}, + {file = "kiwisolver-1.4.4-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:db608a6757adabb32f1cfe6066e39b3706d8c3aa69bbc353a5b61edad36a5cb4"}, + {file = "kiwisolver-1.4.4-cp311-cp311-musllinux_1_1_ppc64le.whl", hash = "sha256:5853eb494c71e267912275e5586fe281444eb5e722de4e131cddf9d442615626"}, + {file = "kiwisolver-1.4.4-cp311-cp311-musllinux_1_1_s390x.whl", hash = "sha256:f0a1dbdb5ecbef0d34eb77e56fcb3e95bbd7e50835d9782a45df81cc46949750"}, + {file = "kiwisolver-1.4.4-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:283dffbf061a4ec60391d51e6155e372a1f7a4f5b15d59c8505339454f8989e4"}, + {file = "kiwisolver-1.4.4-cp311-cp311-win32.whl", hash = "sha256:d06adcfa62a4431d404c31216f0f8ac97397d799cd53800e9d3efc2fbb3cf14e"}, + {file = "kiwisolver-1.4.4-cp311-cp311-win_amd64.whl", hash = "sha256:e7da3fec7408813a7cebc9e4ec55afed2d0fd65c4754bc376bf03498d4e92686"}, + {file = "kiwisolver-1.4.4-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:62ac9cc684da4cf1778d07a89bf5f81b35834cb96ca523d3a7fb32509380cbf6"}, + {file = "kiwisolver-1.4.4-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:41dae968a94b1ef1897cb322b39360a0812661dba7c682aa45098eb8e193dbdf"}, + {file = "kiwisolver-1.4.4-cp37-cp37m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:02f79693ec433cb4b5f51694e8477ae83b3205768a6fb48ffba60549080e295b"}, + {file = "kiwisolver-1.4.4-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:d0611a0a2a518464c05ddd5a3a1a0e856ccc10e67079bb17f265ad19ab3c7597"}, + {file = "kiwisolver-1.4.4-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:db5283d90da4174865d520e7366801a93777201e91e79bacbac6e6927cbceede"}, + {file = "kiwisolver-1.4.4-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:1041feb4cda8708ce73bb4dcb9ce1ccf49d553bf87c3954bdfa46f0c3f77252c"}, + {file = "kiwisolver-1.4.4-cp37-cp37m-win32.whl", hash = "sha256:a553dadda40fef6bfa1456dc4be49b113aa92c2a9a9e8711e955618cd69622e3"}, + {file = "kiwisolver-1.4.4-cp37-cp37m-win_amd64.whl", hash = "sha256:03baab2d6b4a54ddbb43bba1a3a2d1627e82d205c5cf8f4c924dc49284b87166"}, + {file = "kiwisolver-1.4.4-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:841293b17ad704d70c578f1f0013c890e219952169ce8a24ebc063eecf775454"}, + {file = "kiwisolver-1.4.4-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:f4f270de01dd3e129a72efad823da90cc4d6aafb64c410c9033aba70db9f1ff0"}, + {file = "kiwisolver-1.4.4-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:f9f39e2f049db33a908319cf46624a569b36983c7c78318e9726a4cb8923b26c"}, + {file = "kiwisolver-1.4.4-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c97528e64cb9ebeff9701e7938653a9951922f2a38bd847787d4a8e498cc83ae"}, + {file = "kiwisolver-1.4.4-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:1d1573129aa0fd901076e2bfb4275a35f5b7aa60fbfb984499d661ec950320b0"}, + {file = "kiwisolver-1.4.4-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ad881edc7ccb9d65b0224f4e4d05a1e85cf62d73aab798943df6d48ab0cd79a1"}, + {file = "kiwisolver-1.4.4-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:b428ef021242344340460fa4c9185d0b1f66fbdbfecc6c63eff4b7c29fad429d"}, + {file = "kiwisolver-1.4.4-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:2e407cb4bd5a13984a6c2c0fe1845e4e41e96f183e5e5cd4d77a857d9693494c"}, + {file = "kiwisolver-1.4.4-cp38-cp38-win32.whl", hash = "sha256:75facbe9606748f43428fc91a43edb46c7ff68889b91fa31f53b58894503a191"}, + {file = "kiwisolver-1.4.4-cp38-cp38-win_amd64.whl", hash = "sha256:5bce61af018b0cb2055e0e72e7d65290d822d3feee430b7b8203d8a855e78766"}, + {file = "kiwisolver-1.4.4-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:8c808594c88a025d4e322d5bb549282c93c8e1ba71b790f539567932722d7bd8"}, + {file = "kiwisolver-1.4.4-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:f0a71d85ecdd570ded8ac3d1c0f480842f49a40beb423bb8014539a9f32a5897"}, + {file = "kiwisolver-1.4.4-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:b533558eae785e33e8c148a8d9921692a9fe5aa516efbdff8606e7d87b9d5824"}, + {file = "kiwisolver-1.4.4-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:efda5fc8cc1c61e4f639b8067d118e742b812c930f708e6667a5ce0d13499e29"}, + {file = "kiwisolver-1.4.4-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:7c43e1e1206cd421cd92e6b3280d4385d41d7166b3ed577ac20444b6995a445f"}, + {file = "kiwisolver-1.4.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:bc8d3bd6c72b2dd9decf16ce70e20abcb3274ba01b4e1c96031e0c4067d1e7cd"}, + {file = "kiwisolver-1.4.4-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:4ea39b0ccc4f5d803e3337dd46bcce60b702be4d86fd0b3d7531ef10fd99a1ac"}, + {file = "kiwisolver-1.4.4-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:968f44fdbf6dd757d12920d63b566eeb4d5b395fd2d00d29d7ef00a00582aac9"}, + {file = "kiwisolver-1.4.4-cp39-cp39-win32.whl", hash = "sha256:da7e547706e69e45d95e116e6939488d62174e033b763ab1496b4c29b76fabea"}, + {file = "kiwisolver-1.4.4-cp39-cp39-win_amd64.whl", hash = "sha256:ba59c92039ec0a66103b1d5fe588fa546373587a7d68f5c96f743c3396afc04b"}, + {file = "kiwisolver-1.4.4-pp37-pypy37_pp73-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:91672bacaa030f92fc2f43b620d7b337fd9a5af28b0d6ed3f77afc43c4a64b5a"}, + {file = "kiwisolver-1.4.4-pp37-pypy37_pp73-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:787518a6789009c159453da4d6b683f468ef7a65bbde796bcea803ccf191058d"}, + {file = "kiwisolver-1.4.4-pp37-pypy37_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:da152d8cdcab0e56e4f45eb08b9aea6455845ec83172092f09b0e077ece2cf7a"}, + {file = "kiwisolver-1.4.4-pp37-pypy37_pp73-win_amd64.whl", hash = "sha256:ecb1fa0db7bf4cff9dac752abb19505a233c7f16684c5826d1f11ebd9472b871"}, + {file = "kiwisolver-1.4.4-pp38-pypy38_pp73-macosx_10_9_x86_64.whl", hash = "sha256:28bc5b299f48150b5f822ce68624e445040595a4ac3d59251703779836eceff9"}, + {file = "kiwisolver-1.4.4-pp38-pypy38_pp73-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:81e38381b782cc7e1e46c4e14cd997ee6040768101aefc8fa3c24a4cc58e98f8"}, + {file = "kiwisolver-1.4.4-pp38-pypy38_pp73-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:2a66fdfb34e05b705620dd567f5a03f239a088d5a3f321e7b6ac3239d22aa286"}, + {file = "kiwisolver-1.4.4-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:872b8ca05c40d309ed13eb2e582cab0c5a05e81e987ab9c521bf05ad1d5cf5cb"}, + {file = "kiwisolver-1.4.4-pp38-pypy38_pp73-win_amd64.whl", hash = "sha256:70e7c2e7b750585569564e2e5ca9845acfaa5da56ac46df68414f29fea97be9f"}, + {file = "kiwisolver-1.4.4-pp39-pypy39_pp73-macosx_10_9_x86_64.whl", hash = "sha256:9f85003f5dfa867e86d53fac6f7e6f30c045673fa27b603c397753bebadc3008"}, + {file = "kiwisolver-1.4.4-pp39-pypy39_pp73-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:2e307eb9bd99801f82789b44bb45e9f541961831c7311521b13a6c85afc09767"}, + {file = "kiwisolver-1.4.4-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b1792d939ec70abe76f5054d3f36ed5656021dcad1322d1cc996d4e54165cef9"}, + {file = "kiwisolver-1.4.4-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f6cb459eea32a4e2cf18ba5fcece2dbdf496384413bc1bae15583f19e567f3b2"}, + {file = "kiwisolver-1.4.4-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:36dafec3d6d6088d34e2de6b85f9d8e2324eb734162fba59d2ba9ed7a2043d5b"}, + {file = "kiwisolver-1.4.4.tar.gz", hash = "sha256:d41997519fcba4a1e46eb4a2fe31bc12f0ff957b2b81bac28db24744f333e955"}, +] + +[[package]] +name = "markdown" +version = "3.4.3" +description = "Python implementation of John Gruber's Markdown." +optional = false +python-versions = ">=3.7" +files = [ + {file = "Markdown-3.4.3-py3-none-any.whl", hash = "sha256:065fd4df22da73a625f14890dd77eb8040edcbd68794bcd35943be14490608b2"}, + {file = "Markdown-3.4.3.tar.gz", hash = "sha256:8bf101198e004dc93e84a12a7395e31aac6a9c9942848ae1d99b9d72cf9b3520"}, +] + +[package.dependencies] +importlib-metadata = {version = ">=4.4", markers = "python_version < \"3.10\""} + +[package.extras] +testing = ["coverage", "pyyaml"] + +[[package]] +name = "markupsafe" +version = "2.1.2" +description = "Safely add untrusted strings to HTML/XML markup." +optional = false +python-versions = ">=3.7" +files = [ + {file = "MarkupSafe-2.1.2-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:665a36ae6f8f20a4676b53224e33d456a6f5a72657d9c83c2aa00765072f31f7"}, + {file = "MarkupSafe-2.1.2-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:340bea174e9761308703ae988e982005aedf427de816d1afe98147668cc03036"}, + {file = "MarkupSafe-2.1.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:22152d00bf4a9c7c83960521fc558f55a1adbc0631fbb00a9471e097b19d72e1"}, + {file = "MarkupSafe-2.1.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:28057e985dace2f478e042eaa15606c7efccb700797660629da387eb289b9323"}, + {file = "MarkupSafe-2.1.2-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:ca244fa73f50a800cf8c3ebf7fd93149ec37f5cb9596aa8873ae2c1d23498601"}, + {file = "MarkupSafe-2.1.2-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:d9d971ec1e79906046aa3ca266de79eac42f1dbf3612a05dc9368125952bd1a1"}, + {file = "MarkupSafe-2.1.2-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:7e007132af78ea9df29495dbf7b5824cb71648d7133cf7848a2a5dd00d36f9ff"}, + {file = "MarkupSafe-2.1.2-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:7313ce6a199651c4ed9d7e4cfb4aa56fe923b1adf9af3b420ee14e6d9a73df65"}, + {file = "MarkupSafe-2.1.2-cp310-cp310-win32.whl", hash = "sha256:c4a549890a45f57f1ebf99c067a4ad0cb423a05544accaf2b065246827ed9603"}, + {file = "MarkupSafe-2.1.2-cp310-cp310-win_amd64.whl", hash = "sha256:835fb5e38fd89328e9c81067fd642b3593c33e1e17e2fdbf77f5676abb14a156"}, + {file = "MarkupSafe-2.1.2-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:2ec4f2d48ae59bbb9d1f9d7efb9236ab81429a764dedca114f5fdabbc3788013"}, + {file = "MarkupSafe-2.1.2-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:608e7073dfa9e38a85d38474c082d4281f4ce276ac0010224eaba11e929dd53a"}, + {file = "MarkupSafe-2.1.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:65608c35bfb8a76763f37036547f7adfd09270fbdbf96608be2bead319728fcd"}, + {file = "MarkupSafe-2.1.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f2bfb563d0211ce16b63c7cb9395d2c682a23187f54c3d79bfec33e6705473c6"}, + {file = "MarkupSafe-2.1.2-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:da25303d91526aac3672ee6d49a2f3db2d9502a4a60b55519feb1a4c7714e07d"}, + {file = "MarkupSafe-2.1.2-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:9cad97ab29dfc3f0249b483412c85c8ef4766d96cdf9dcf5a1e3caa3f3661cf1"}, + {file = "MarkupSafe-2.1.2-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:085fd3201e7b12809f9e6e9bc1e5c96a368c8523fad5afb02afe3c051ae4afcc"}, + {file = "MarkupSafe-2.1.2-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:1bea30e9bf331f3fef67e0a3877b2288593c98a21ccb2cf29b74c581a4eb3af0"}, + {file = "MarkupSafe-2.1.2-cp311-cp311-win32.whl", hash = "sha256:7df70907e00c970c60b9ef2938d894a9381f38e6b9db73c5be35e59d92e06625"}, + {file = "MarkupSafe-2.1.2-cp311-cp311-win_amd64.whl", hash = "sha256:e55e40ff0cc8cc5c07996915ad367fa47da6b3fc091fdadca7f5403239c5fec3"}, + {file = "MarkupSafe-2.1.2-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:a6e40afa7f45939ca356f348c8e23048e02cb109ced1eb8420961b2f40fb373a"}, + {file = "MarkupSafe-2.1.2-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cf877ab4ed6e302ec1d04952ca358b381a882fbd9d1b07cccbfd61783561f98a"}, + {file = "MarkupSafe-2.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:63ba06c9941e46fa389d389644e2d8225e0e3e5ebcc4ff1ea8506dce646f8c8a"}, + {file = "MarkupSafe-2.1.2-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:f1cd098434e83e656abf198f103a8207a8187c0fc110306691a2e94a78d0abb2"}, + {file = "MarkupSafe-2.1.2-cp37-cp37m-musllinux_1_1_aarch64.whl", hash = "sha256:55f44b440d491028addb3b88f72207d71eeebfb7b5dbf0643f7c023ae1fba619"}, + {file = "MarkupSafe-2.1.2-cp37-cp37m-musllinux_1_1_i686.whl", hash = "sha256:a6f2fcca746e8d5910e18782f976489939d54a91f9411c32051b4aab2bd7c513"}, + {file = "MarkupSafe-2.1.2-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:0b462104ba25f1ac006fdab8b6a01ebbfbce9ed37fd37fd4acd70c67c973e460"}, + {file = "MarkupSafe-2.1.2-cp37-cp37m-win32.whl", hash = "sha256:7668b52e102d0ed87cb082380a7e2e1e78737ddecdde129acadb0eccc5423859"}, + {file = "MarkupSafe-2.1.2-cp37-cp37m-win_amd64.whl", hash = "sha256:6d6607f98fcf17e534162f0709aaad3ab7a96032723d8ac8750ffe17ae5a0666"}, + {file = "MarkupSafe-2.1.2-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:a806db027852538d2ad7555b203300173dd1b77ba116de92da9afbc3a3be3eed"}, + {file = "MarkupSafe-2.1.2-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:a4abaec6ca3ad8660690236d11bfe28dfd707778e2442b45addd2f086d6ef094"}, + {file = "MarkupSafe-2.1.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f03a532d7dee1bed20bc4884194a16160a2de9ffc6354b3878ec9682bb623c54"}, + {file = "MarkupSafe-2.1.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4cf06cdc1dda95223e9d2d3c58d3b178aa5dacb35ee7e3bbac10e4e1faacb419"}, + {file = "MarkupSafe-2.1.2-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:22731d79ed2eb25059ae3df1dfc9cb1546691cc41f4e3130fe6bfbc3ecbbecfa"}, + {file = "MarkupSafe-2.1.2-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:f8ffb705ffcf5ddd0e80b65ddf7bed7ee4f5a441ea7d3419e861a12eaf41af58"}, + {file = "MarkupSafe-2.1.2-cp38-cp38-musllinux_1_1_i686.whl", hash = "sha256:8db032bf0ce9022a8e41a22598eefc802314e81b879ae093f36ce9ddf39ab1ba"}, + {file = "MarkupSafe-2.1.2-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:2298c859cfc5463f1b64bd55cb3e602528db6fa0f3cfd568d3605c50678f8f03"}, + {file = "MarkupSafe-2.1.2-cp38-cp38-win32.whl", hash = "sha256:50c42830a633fa0cf9e7d27664637532791bfc31c731a87b202d2d8ac40c3ea2"}, + {file = "MarkupSafe-2.1.2-cp38-cp38-win_amd64.whl", hash = "sha256:bb06feb762bade6bf3c8b844462274db0c76acc95c52abe8dbed28ae3d44a147"}, + {file = "MarkupSafe-2.1.2-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:99625a92da8229df6d44335e6fcc558a5037dd0a760e11d84be2260e6f37002f"}, + {file = "MarkupSafe-2.1.2-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:8bca7e26c1dd751236cfb0c6c72d4ad61d986e9a41bbf76cb445f69488b2a2bd"}, + {file = "MarkupSafe-2.1.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:40627dcf047dadb22cd25ea7ecfe9cbf3bbbad0482ee5920b582f3809c97654f"}, + {file = "MarkupSafe-2.1.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:40dfd3fefbef579ee058f139733ac336312663c6706d1163b82b3003fb1925c4"}, + {file = "MarkupSafe-2.1.2-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:090376d812fb6ac5f171e5938e82e7f2d7adc2b629101cec0db8b267815c85e2"}, + {file = "MarkupSafe-2.1.2-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:2e7821bffe00aa6bd07a23913b7f4e01328c3d5cc0b40b36c0bd81d362faeb65"}, + {file = "MarkupSafe-2.1.2-cp39-cp39-musllinux_1_1_i686.whl", hash = "sha256:c0a33bc9f02c2b17c3ea382f91b4db0e6cde90b63b296422a939886a7a80de1c"}, + {file = "MarkupSafe-2.1.2-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:b8526c6d437855442cdd3d87eede9c425c4445ea011ca38d937db299382e6fa3"}, + {file = "MarkupSafe-2.1.2-cp39-cp39-win32.whl", hash = "sha256:137678c63c977754abe9086a3ec011e8fd985ab90631145dfb9294ad09c102a7"}, + {file = "MarkupSafe-2.1.2-cp39-cp39-win_amd64.whl", hash = "sha256:0576fe974b40a400449768941d5d0858cc624e3249dfd1e0c33674e5c7ca7aed"}, + {file = "MarkupSafe-2.1.2.tar.gz", hash = "sha256:abcabc8c2b26036d62d4c746381a6f7cf60aafcc653198ad678306986b09450d"}, +] + +[[package]] +name = "matplotlib" +version = "3.7.1" +description = "Python plotting package" +optional = false +python-versions = ">=3.8" +files = [ + {file = "matplotlib-3.7.1-cp310-cp310-macosx_10_12_universal2.whl", hash = "sha256:95cbc13c1fc6844ab8812a525bbc237fa1470863ff3dace7352e910519e194b1"}, + {file = "matplotlib-3.7.1-cp310-cp310-macosx_10_12_x86_64.whl", hash = "sha256:08308bae9e91aca1ec6fd6dda66237eef9f6294ddb17f0d0b3c863169bf82353"}, + {file = "matplotlib-3.7.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:544764ba51900da4639c0f983b323d288f94f65f4024dc40ecb1542d74dc0500"}, + {file = "matplotlib-3.7.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:56d94989191de3fcc4e002f93f7f1be5da476385dde410ddafbb70686acf00ea"}, + {file = "matplotlib-3.7.1-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:e99bc9e65901bb9a7ce5e7bb24af03675cbd7c70b30ac670aa263240635999a4"}, + {file = "matplotlib-3.7.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:eb7d248c34a341cd4c31a06fd34d64306624c8cd8d0def7abb08792a5abfd556"}, + {file = "matplotlib-3.7.1-cp310-cp310-win32.whl", hash = "sha256:ce463ce590f3825b52e9fe5c19a3c6a69fd7675a39d589e8b5fbe772272b3a24"}, + {file = "matplotlib-3.7.1-cp310-cp310-win_amd64.whl", hash = "sha256:3d7bc90727351fb841e4d8ae620d2d86d8ed92b50473cd2b42ce9186104ecbba"}, + {file = "matplotlib-3.7.1-cp311-cp311-macosx_10_12_universal2.whl", hash = "sha256:770a205966d641627fd5cf9d3cb4b6280a716522cd36b8b284a8eb1581310f61"}, + {file = "matplotlib-3.7.1-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:f67bfdb83a8232cb7a92b869f9355d677bce24485c460b19d01970b64b2ed476"}, + {file = "matplotlib-3.7.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:2bf092f9210e105f414a043b92af583c98f50050559616930d884387d0772aba"}, + {file = "matplotlib-3.7.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:89768d84187f31717349c6bfadc0e0d8c321e8eb34522acec8a67b1236a66332"}, + {file = "matplotlib-3.7.1-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:83111e6388dec67822e2534e13b243cc644c7494a4bb60584edbff91585a83c6"}, + {file = "matplotlib-3.7.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a867bf73a7eb808ef2afbca03bcdb785dae09595fbe550e1bab0cd023eba3de0"}, + {file = "matplotlib-3.7.1-cp311-cp311-win32.whl", hash = "sha256:fbdeeb58c0cf0595efe89c05c224e0a502d1aa6a8696e68a73c3efc6bc354304"}, + {file = "matplotlib-3.7.1-cp311-cp311-win_amd64.whl", hash = "sha256:c0bd19c72ae53e6ab979f0ac6a3fafceb02d2ecafa023c5cca47acd934d10be7"}, + {file = "matplotlib-3.7.1-cp38-cp38-macosx_10_12_universal2.whl", hash = "sha256:6eb88d87cb2c49af00d3bbc33a003f89fd9f78d318848da029383bfc08ecfbfb"}, + {file = "matplotlib-3.7.1-cp38-cp38-macosx_10_12_x86_64.whl", hash = "sha256:cf0e4f727534b7b1457898c4f4ae838af1ef87c359b76dcd5330fa31893a3ac7"}, + {file = "matplotlib-3.7.1-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:46a561d23b91f30bccfd25429c3c706afe7d73a5cc64ef2dfaf2b2ac47c1a5dc"}, + {file = "matplotlib-3.7.1-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:8704726d33e9aa8a6d5215044b8d00804561971163563e6e6591f9dcf64340cc"}, + {file = "matplotlib-3.7.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:4cf327e98ecf08fcbb82685acaf1939d3338548620ab8dfa02828706402c34de"}, + {file = "matplotlib-3.7.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:617f14ae9d53292ece33f45cba8503494ee199a75b44de7717964f70637a36aa"}, + {file = "matplotlib-3.7.1-cp38-cp38-win32.whl", hash = "sha256:7c9a4b2da6fac77bcc41b1ea95fadb314e92508bf5493ceff058e727e7ecf5b0"}, + {file = "matplotlib-3.7.1-cp38-cp38-win_amd64.whl", hash = "sha256:14645aad967684e92fc349493fa10c08a6da514b3d03a5931a1bac26e6792bd1"}, + {file = "matplotlib-3.7.1-cp39-cp39-macosx_10_12_universal2.whl", hash = "sha256:81a6b377ea444336538638d31fdb39af6be1a043ca5e343fe18d0f17e098770b"}, + {file = "matplotlib-3.7.1-cp39-cp39-macosx_10_12_x86_64.whl", hash = "sha256:28506a03bd7f3fe59cd3cd4ceb2a8d8a2b1db41afede01f66c42561b9be7b4b7"}, + {file = "matplotlib-3.7.1-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:8c587963b85ce41e0a8af53b9b2de8dddbf5ece4c34553f7bd9d066148dc719c"}, + {file = "matplotlib-3.7.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8bf26ade3ff0f27668989d98c8435ce9327d24cffb7f07d24ef609e33d582439"}, + {file = "matplotlib-3.7.1-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:def58098f96a05f90af7e92fd127d21a287068202aa43b2a93476170ebd99e87"}, + {file = "matplotlib-3.7.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f883a22a56a84dba3b588696a2b8a1ab0d2c3d41be53264115c71b0a942d8fdb"}, + {file = "matplotlib-3.7.1-cp39-cp39-win32.whl", hash = "sha256:4f99e1b234c30c1e9714610eb0c6d2f11809c9c78c984a613ae539ea2ad2eb4b"}, + {file = "matplotlib-3.7.1-cp39-cp39-win_amd64.whl", hash = "sha256:3ba2af245e36990facf67fde840a760128ddd71210b2ab6406e640188d69d136"}, + {file = "matplotlib-3.7.1-pp38-pypy38_pp73-macosx_10_12_x86_64.whl", hash = "sha256:3032884084f541163f295db8a6536e0abb0db464008fadca6c98aaf84ccf4717"}, + {file = "matplotlib-3.7.1-pp38-pypy38_pp73-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:3a2cb34336110e0ed8bb4f650e817eed61fa064acbefeb3591f1b33e3a84fd96"}, + {file = "matplotlib-3.7.1-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b867e2f952ed592237a1828f027d332d8ee219ad722345b79a001f49df0936eb"}, + {file = "matplotlib-3.7.1-pp38-pypy38_pp73-win_amd64.whl", hash = "sha256:57bfb8c8ea253be947ccb2bc2d1bb3862c2bccc662ad1b4626e1f5e004557042"}, + {file = "matplotlib-3.7.1-pp39-pypy39_pp73-macosx_10_12_x86_64.whl", hash = "sha256:438196cdf5dc8d39b50a45cb6e3f6274edbcf2254f85fa9b895bf85851c3a613"}, + {file = "matplotlib-3.7.1-pp39-pypy39_pp73-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:21e9cff1a58d42e74d01153360de92b326708fb205250150018a52c70f43c290"}, + {file = "matplotlib-3.7.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:75d4725d70b7c03e082bbb8a34639ede17f333d7247f56caceb3801cb6ff703d"}, + {file = "matplotlib-3.7.1-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:97cc368a7268141afb5690760921765ed34867ffb9655dd325ed207af85c7529"}, + {file = "matplotlib-3.7.1.tar.gz", hash = "sha256:7b73305f25eab4541bd7ee0b96d87e53ae9c9f1823be5659b806cd85786fe882"}, +] + +[package.dependencies] +contourpy = ">=1.0.1" +cycler = ">=0.10" +fonttools = ">=4.22.0" +importlib-resources = {version = ">=3.2.0", markers = "python_version < \"3.10\""} +kiwisolver = ">=1.0.1" +numpy = ">=1.20" +packaging = ">=20.0" +pillow = ">=6.2.0" +pyparsing = ">=2.3.1" +python-dateutil = ">=2.7" + +[[package]] +name = "matplotlib-inline" +version = "0.1.6" +description = "Inline Matplotlib backend for Jupyter" +optional = false +python-versions = ">=3.5" +files = [ + {file = "matplotlib-inline-0.1.6.tar.gz", hash = "sha256:f887e5f10ba98e8d2b150ddcf4702c1e5f8b3a20005eb0f74bfdbd360ee6f304"}, + {file = "matplotlib_inline-0.1.6-py3-none-any.whl", hash = "sha256:f1f41aab5328aa5aaea9b16d083b128102f8712542f819fe7e6a420ff581b311"}, +] + +[package.dependencies] +traitlets = "*" + +[[package]] +name = "mistune" +version = "2.0.5" +description = "A sane Markdown parser with useful plugins and renderers" +optional = false +python-versions = "*" +files = [ + {file = "mistune-2.0.5-py2.py3-none-any.whl", hash = "sha256:bad7f5d431886fcbaf5f758118ecff70d31f75231b34024a1341120340a65ce8"}, + {file = "mistune-2.0.5.tar.gz", hash = "sha256:0246113cb2492db875c6be56974a7c893333bf26cd92891c85f63151cee09d34"}, +] + +[[package]] +name = "munch" +version = "2.5.0" +description = "A dot-accessible dictionary (a la JavaScript objects)" +optional = false +python-versions = "*" +files = [ + {file = "munch-2.5.0-py2.py3-none-any.whl", hash = "sha256:6f44af89a2ce4ed04ff8de41f70b226b984db10a91dcc7b9ac2efc1c77022fdd"}, + {file = "munch-2.5.0.tar.gz", hash = "sha256:2d735f6f24d4dba3417fa448cae40c6e896ec1fdab6cdb5e6510999758a4dbd2"}, +] + +[package.dependencies] +six = "*" + +[package.extras] +testing = ["astroid (>=1.5.3,<1.6.0)", "astroid (>=2.0)", "coverage", "pylint (>=1.7.2,<1.8.0)", "pylint (>=2.3.1,<2.4.0)", "pytest"] +yaml = ["PyYAML (>=5.1.0)"] + +[[package]] +name = "mypy-extensions" +version = "1.0.0" +description = "Type system extensions for programs checked with the mypy type checker." +optional = false +python-versions = ">=3.5" +files = [ + {file = "mypy_extensions-1.0.0-py3-none-any.whl", hash = "sha256:4392f6c0eb8a5668a69e23d168ffa70f0be9ccfd32b5cc2d26a34ae5b844552d"}, + {file = "mypy_extensions-1.0.0.tar.gz", hash = "sha256:75dbf8955dc00442a438fc4d0666508a9a97b6bd41aa2f0ffe9d2f2725af0782"}, +] + +[[package]] +name = "nbclient" +version = "0.7.3" +description = "A client library for executing notebooks. Formerly nbconvert's ExecutePreprocessor." +optional = false +python-versions = ">=3.7.0" +files = [ + {file = "nbclient-0.7.3-py3-none-any.whl", hash = "sha256:8fa96f7e36693d5e83408f5e840f113c14a45c279befe609904dbe05dad646d1"}, + {file = "nbclient-0.7.3.tar.gz", hash = "sha256:26e41c6dca4d76701988bc34f64e1bfc2413ae6d368f13d7b5ac407efb08c755"}, +] + +[package.dependencies] +jupyter-client = ">=6.1.12" +jupyter-core = ">=4.12,<5.0.dev0 || >=5.1.dev0" +nbformat = ">=5.1" +traitlets = ">=5.3" + +[package.extras] +dev = ["pre-commit"] +docs = ["autodoc-traits", "mock", "moto", "myst-parser", "nbclient[test]", "sphinx (>=1.7)", "sphinx-book-theme", "sphinxcontrib-spelling"] +test = ["flaky", "ipykernel", "ipython", "ipywidgets", "nbconvert (>=7.0.0)", "pytest (>=7.0)", "pytest-asyncio", "pytest-cov (>=4.0)", "testpath", "xmltodict"] + +[[package]] +name = "nbconvert" +version = "7.3.1" +description = "Converting Jupyter Notebooks" +optional = false +python-versions = ">=3.7" +files = [ + {file = "nbconvert-7.3.1-py3-none-any.whl", hash = "sha256:d2e95904666f1ff77d36105b9de4e0801726f93b862d5b28f69e93d99ad3b19c"}, + {file = "nbconvert-7.3.1.tar.gz", hash = "sha256:78685362b11d2e8058e70196fe83b09abed8df22d3e599cf271f4d39fdc48b9e"}, +] + +[package.dependencies] +beautifulsoup4 = "*" +bleach = "*" +defusedxml = "*" +importlib-metadata = {version = ">=3.6", markers = "python_version < \"3.10\""} +jinja2 = ">=3.0" +jupyter-core = ">=4.7" +jupyterlab-pygments = "*" +markupsafe = ">=2.0" +mistune = ">=2.0.3,<3" +nbclient = ">=0.5.0" +nbformat = ">=5.1" +packaging = "*" +pandocfilters = ">=1.4.1" +pygments = ">=2.4.1" +tinycss2 = "*" +traitlets = ">=5.0" + +[package.extras] +all = ["nbconvert[docs,qtpdf,serve,test,webpdf]"] +docs = ["ipykernel", "ipython", "myst-parser", "nbsphinx (>=0.2.12)", "pydata-sphinx-theme", "sphinx (==5.0.2)", "sphinxcontrib-spelling"] +qtpdf = ["nbconvert[qtpng]"] +qtpng = ["pyqtwebengine (>=5.15)"] +serve = ["tornado (>=6.1)"] +test = ["ipykernel", "ipywidgets (>=7)", "pre-commit", "pytest", "pytest-dependency"] +webpdf = ["pyppeteer (>=1,<1.1)"] + +[[package]] +name = "nbformat" +version = "5.8.0" +description = "The Jupyter Notebook format" +optional = false +python-versions = ">=3.7" +files = [ + {file = "nbformat-5.8.0-py3-none-any.whl", hash = "sha256:d910082bd3e0bffcf07eabf3683ed7dda0727a326c446eeb2922abe102e65162"}, + {file = "nbformat-5.8.0.tar.gz", hash = "sha256:46dac64c781f1c34dfd8acba16547024110348f9fc7eab0f31981c2a3dc48d1f"}, +] + +[package.dependencies] +fastjsonschema = "*" +jsonschema = ">=2.6" +jupyter-core = "*" +traitlets = ">=5.1" + +[package.extras] +docs = ["myst-parser", "pydata-sphinx-theme", "sphinx", "sphinxcontrib-github-alt", "sphinxcontrib-spelling"] +test = ["pep440", "pre-commit", "pytest", "testpath"] + +[[package]] +name = "nest-asyncio" +version = "1.5.6" +description = "Patch asyncio to allow nested event loops" +optional = false +python-versions = ">=3.5" +files = [ + {file = "nest_asyncio-1.5.6-py3-none-any.whl", hash = "sha256:b9a953fb40dceaa587d109609098db21900182b16440652454a146cffb06e8b8"}, + {file = "nest_asyncio-1.5.6.tar.gz", hash = "sha256:d267cc1ff794403f7df692964d1d2a3fa9418ffea2a3f6859a439ff482fef290"}, +] + +[[package]] +name = "nodeenv" +version = "1.7.0" +description = "Node.js virtual environment builder" +optional = false +python-versions = ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*" +files = [ + {file = "nodeenv-1.7.0-py2.py3-none-any.whl", hash = "sha256:27083a7b96a25f2f5e1d8cb4b6317ee8aeda3bdd121394e5ac54e498028a042e"}, + {file = "nodeenv-1.7.0.tar.gz", hash = "sha256:e0e7f7dfb85fc5394c6fe1e8fa98131a2473e04311a45afb6508f7cf1836fa2b"}, +] + +[package.dependencies] +setuptools = "*" + +[[package]] +name = "numpy" +version = "1.22.4" +description = "NumPy is the fundamental package for array computing with Python." +optional = false +python-versions = ">=3.8" +files = [ + {file = "numpy-1.22.4-cp310-cp310-macosx_10_14_x86_64.whl", hash = "sha256:ba9ead61dfb5d971d77b6c131a9dbee62294a932bf6a356e48c75ae684e635b3"}, + {file = "numpy-1.22.4-cp310-cp310-macosx_10_15_x86_64.whl", hash = "sha256:1ce7ab2053e36c0a71e7a13a7475bd3b1f54750b4b433adc96313e127b870887"}, + {file = "numpy-1.22.4-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:7228ad13744f63575b3a972d7ee4fd61815b2879998e70930d4ccf9ec721dce0"}, + {file = "numpy-1.22.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:43a8ca7391b626b4c4fe20aefe79fec683279e31e7c79716863b4b25021e0e74"}, + {file = "numpy-1.22.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a911e317e8c826ea632205e63ed8507e0dc877dcdc49744584dfc363df9ca08c"}, + {file = "numpy-1.22.4-cp310-cp310-win32.whl", hash = "sha256:9ce7df0abeabe7fbd8ccbf343dc0db72f68549856b863ae3dd580255d009648e"}, + {file = "numpy-1.22.4-cp310-cp310-win_amd64.whl", hash = "sha256:3e1ffa4748168e1cc8d3cde93f006fe92b5421396221a02f2274aab6ac83b077"}, + {file = "numpy-1.22.4-cp38-cp38-macosx_10_15_x86_64.whl", hash = "sha256:59d55e634968b8f77d3fd674a3cf0b96e85147cd6556ec64ade018f27e9479e1"}, + {file = "numpy-1.22.4-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:c1d937820db6e43bec43e8d016b9b3165dcb42892ea9f106c70fb13d430ffe72"}, + {file = "numpy-1.22.4-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d4c5d5eb2ec8da0b4f50c9a843393971f31f1d60be87e0fb0917a49133d257d6"}, + {file = "numpy-1.22.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:64f56fc53a2d18b1924abd15745e30d82a5782b2cab3429aceecc6875bd5add0"}, + {file = "numpy-1.22.4-cp38-cp38-win32.whl", hash = "sha256:fb7a980c81dd932381f8228a426df8aeb70d59bbcda2af075b627bbc50207cba"}, + {file = "numpy-1.22.4-cp38-cp38-win_amd64.whl", hash = "sha256:e96d7f3096a36c8754207ab89d4b3282ba7b49ea140e4973591852c77d09eb76"}, + {file = "numpy-1.22.4-cp39-cp39-macosx_10_14_x86_64.whl", hash = "sha256:4c6036521f11a731ce0648f10c18ae66d7143865f19f7299943c985cdc95afb5"}, + {file = "numpy-1.22.4-cp39-cp39-macosx_10_15_x86_64.whl", hash = "sha256:b89bf9b94b3d624e7bb480344e91f68c1c6c75f026ed6755955117de00917a7c"}, + {file = "numpy-1.22.4-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:2d487e06ecbf1dc2f18e7efce82ded4f705f4bd0cd02677ffccfb39e5c284c7e"}, + {file = "numpy-1.22.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f3eb268dbd5cfaffd9448113539e44e2dd1c5ca9ce25576f7c04a5453edc26fa"}, + {file = "numpy-1.22.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:37431a77ceb9307c28382c9773da9f306435135fae6b80b62a11c53cfedd8802"}, + {file = "numpy-1.22.4-cp39-cp39-win32.whl", hash = "sha256:cc7f00008eb7d3f2489fca6f334ec19ca63e31371be28fd5dad955b16ec285bd"}, + {file = "numpy-1.22.4-cp39-cp39-win_amd64.whl", hash = "sha256:f0725df166cf4785c0bc4cbfb320203182b1ecd30fee6e541c8752a92df6aa32"}, + {file = "numpy-1.22.4-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0791fbd1e43bf74b3502133207e378901272f3c156c4df4954cad833b1380207"}, + {file = "numpy-1.22.4.zip", hash = "sha256:425b390e4619f58d8526b3dcf656dde069133ae5c240229821f01b5f44ea07af"}, +] + +[[package]] +name = "packaging" +version = "23.1" +description = "Core utilities for Python packages" +optional = false +python-versions = ">=3.7" +files = [ + {file = "packaging-23.1-py3-none-any.whl", hash = "sha256:994793af429502c4ea2ebf6bf664629d07c1a9fe974af92966e4b8d2df7edc61"}, + {file = "packaging-23.1.tar.gz", hash = "sha256:a392980d2b6cffa644431898be54b0045151319d1e7ec34f0cfed48767dd334f"}, +] + +[[package]] +name = "pandas" +version = "1.5.1" +description = "Powerful data structures for data analysis, time series, and statistics" +optional = false +python-versions = ">=3.8" +files = [ + {file = "pandas-1.5.1-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:0a78e05ec09731c5b3bd7a9805927ea631fe6f6cb06f0e7c63191a9a778d52b4"}, + {file = "pandas-1.5.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:5b0c970e2215572197b42f1cff58a908d734503ea54b326412c70d4692256391"}, + {file = "pandas-1.5.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:f340331a3f411910adfb4bbe46c2ed5872d9e473a783d7f14ecf49bc0869c594"}, + {file = "pandas-1.5.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d8c709f4700573deb2036d240d140934df7e852520f4a584b2a8d5443b71f54d"}, + {file = "pandas-1.5.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:32e3d9f65606b3f6e76555bfd1d0b68d94aff0929d82010b791b6254bf5a4b96"}, + {file = "pandas-1.5.1-cp310-cp310-win_amd64.whl", hash = "sha256:a52419d9ba5906db516109660b114faf791136c94c1a636ed6b29cbfff9187ee"}, + {file = "pandas-1.5.1-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:66a1ad667b56e679e06ba73bb88c7309b3f48a4c279bd3afea29f65a766e9036"}, + {file = "pandas-1.5.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:36aa1f8f680d7584e9b572c3203b20d22d697c31b71189322f16811d4ecfecd3"}, + {file = "pandas-1.5.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:bcf1a82b770b8f8c1e495b19a20d8296f875a796c4fe6e91da5ef107f18c5ecb"}, + {file = "pandas-1.5.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2c25e5c16ee5c0feb6cf9d982b869eec94a22ddfda9aa2fbed00842cbb697624"}, + {file = "pandas-1.5.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:932d2d7d3cab44cfa275601c982f30c2d874722ef6396bb539e41e4dc4618ed4"}, + {file = "pandas-1.5.1-cp311-cp311-win_amd64.whl", hash = "sha256:eb7e8cf2cf11a2580088009b43de84cabbf6f5dae94ceb489f28dba01a17cb77"}, + {file = "pandas-1.5.1-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:cb2a9cf1150302d69bb99861c5cddc9c25aceacb0a4ef5299785d0f5389a3209"}, + {file = "pandas-1.5.1-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:81f0674fa50b38b6793cd84fae5d67f58f74c2d974d2cb4e476d26eee33343d0"}, + {file = "pandas-1.5.1-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:17da7035d9e6f9ea9cdc3a513161f8739b8f8489d31dc932bc5a29a27243f93d"}, + {file = "pandas-1.5.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:669c8605dba6c798c1863157aefde959c1796671ffb342b80fcb80a4c0bc4c26"}, + {file = "pandas-1.5.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:683779e5728ac9138406c59a11e09cd98c7d2c12f0a5fc2b9c5eecdbb4a00075"}, + {file = "pandas-1.5.1-cp38-cp38-win32.whl", hash = "sha256:ddf46b940ef815af4e542697eaf071f0531449407a7607dd731bf23d156e20a7"}, + {file = "pandas-1.5.1-cp38-cp38-win_amd64.whl", hash = "sha256:db45b94885000981522fb92349e6b76f5aee0924cc5315881239c7859883117d"}, + {file = "pandas-1.5.1-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:927e59c694e039c75d7023465d311277a1fc29ed7236b5746e9dddf180393113"}, + {file = "pandas-1.5.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:e675f8fe9aa6c418dc8d3aac0087b5294c1a4527f1eacf9fe5ea671685285454"}, + {file = "pandas-1.5.1-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:04e51b01d5192499390c0015630975f57836cc95c7411415b499b599b05c0c96"}, + {file = "pandas-1.5.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5cee0c74e93ed4f9d39007e439debcaadc519d7ea5c0afc3d590a3a7b2edf060"}, + {file = "pandas-1.5.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b156a971bc451c68c9e1f97567c94fd44155f073e3bceb1b0d195fd98ed12048"}, + {file = "pandas-1.5.1-cp39-cp39-win32.whl", hash = "sha256:05c527c64ee02a47a24031c880ee0ded05af0623163494173204c5b72ddce658"}, + {file = "pandas-1.5.1-cp39-cp39-win_amd64.whl", hash = "sha256:6bb391659a747cf4f181a227c3e64b6d197100d53da98dcd766cc158bdd9ec68"}, + {file = "pandas-1.5.1.tar.gz", hash = "sha256:249cec5f2a5b22096440bd85c33106b6102e0672204abd2d5c014106459804ee"}, +] + +[package.dependencies] +numpy = [ + {version = ">=1.20.3", markers = "python_version < \"3.10\""}, + {version = ">=1.21.0", markers = "python_version >= \"3.10\""}, +] +python-dateutil = ">=2.8.1" +pytz = ">=2020.1" + +[package.extras] +test = ["hypothesis (>=5.5.3)", "pytest (>=6.0)", "pytest-xdist (>=1.31)"] + +[[package]] +name = "pandocfilters" +version = "1.5.0" +description = "Utilities for writing pandoc filters in python" +optional = false +python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*" +files = [ + {file = "pandocfilters-1.5.0-py2.py3-none-any.whl", hash = "sha256:33aae3f25fd1a026079f5d27bdd52496f0e0803b3469282162bafdcbdf6ef14f"}, + {file = "pandocfilters-1.5.0.tar.gz", hash = "sha256:0b679503337d233b4339a817bfc8c50064e2eff681314376a47cb582305a7a38"}, +] + +[[package]] +name = "parso" +version = "0.8.3" +description = "A Python Parser" +optional = false +python-versions = ">=3.6" +files = [ + {file = "parso-0.8.3-py2.py3-none-any.whl", hash = "sha256:c001d4636cd3aecdaf33cbb40aebb59b094be2a74c556778ef5576c175e19e75"}, + {file = "parso-0.8.3.tar.gz", hash = "sha256:8c07be290bb59f03588915921e29e8a50002acaf2cdc5fa0e0114f91709fafa0"}, +] + +[package.extras] +qa = ["flake8 (==3.8.3)", "mypy (==0.782)"] +testing = ["docopt", "pytest (<6.0.0)"] + +[[package]] +name = "pathspec" +version = "0.11.1" +description = "Utility library for gitignore style pattern matching of file paths." +optional = false +python-versions = ">=3.7" +files = [ + {file = "pathspec-0.11.1-py3-none-any.whl", hash = "sha256:d8af70af76652554bd134c22b3e8a1cc46ed7d91edcdd721ef1a0c51a84a5293"}, + {file = "pathspec-0.11.1.tar.gz", hash = "sha256:2798de800fa92780e33acca925945e9a19a133b715067cf165b8866c15a31687"}, +] + +[[package]] +name = "pexpect" +version = "4.8.0" +description = "Pexpect allows easy control of interactive console applications." +optional = false +python-versions = "*" +files = [ + {file = "pexpect-4.8.0-py2.py3-none-any.whl", hash = "sha256:0b48a55dcb3c05f3329815901ea4fc1537514d6ba867a152b581d69ae3710937"}, + {file = "pexpect-4.8.0.tar.gz", hash = "sha256:fc65a43959d153d0114afe13997d439c22823a27cefceb5ff35c2178c6784c0c"}, +] + +[package.dependencies] +ptyprocess = ">=0.5" + +[[package]] +name = "pickleshare" +version = "0.7.5" +description = "Tiny 'shelve'-like database with concurrency support" +optional = false +python-versions = "*" +files = [ + {file = "pickleshare-0.7.5-py2.py3-none-any.whl", hash = "sha256:9649af414d74d4df115d5d718f82acb59c9d418196b7b4290ed47a12ce62df56"}, + {file = "pickleshare-0.7.5.tar.gz", hash = "sha256:87683d47965c1da65cdacaf31c8441d12b8044cdec9aca500cd78fc2c683afca"}, +] + +[[package]] +name = "pillow" +version = "10.0.1" +description = "Python Imaging Library (Fork)" +optional = false +python-versions = ">=3.8" +files = [ + {file = "Pillow-10.0.1-cp310-cp310-macosx_10_10_x86_64.whl", hash = "sha256:8f06be50669087250f319b706decf69ca71fdecd829091a37cc89398ca4dc17a"}, + {file = "Pillow-10.0.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:50bd5f1ebafe9362ad622072a1d2f5850ecfa44303531ff14353a4059113b12d"}, + {file = "Pillow-10.0.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e6a90167bcca1216606223a05e2cf991bb25b14695c518bc65639463d7db722d"}, + {file = "Pillow-10.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f11c9102c56ffb9ca87134bd025a43d2aba3f1155f508eff88f694b33a9c6d19"}, + {file = "Pillow-10.0.1-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:186f7e04248103482ea6354af6d5bcedb62941ee08f7f788a1c7707bc720c66f"}, + {file = "Pillow-10.0.1-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:0462b1496505a3462d0f35dc1c4d7b54069747d65d00ef48e736acda2c8cbdff"}, + {file = "Pillow-10.0.1-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:d889b53ae2f030f756e61a7bff13684dcd77e9af8b10c6048fb2c559d6ed6eaf"}, + {file = "Pillow-10.0.1-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:552912dbca585b74d75279a7570dd29fa43b6d93594abb494ebb31ac19ace6bd"}, + {file = "Pillow-10.0.1-cp310-cp310-win_amd64.whl", hash = "sha256:787bb0169d2385a798888e1122c980c6eff26bf941a8ea79747d35d8f9210ca0"}, + {file = "Pillow-10.0.1-cp311-cp311-macosx_10_10_x86_64.whl", hash = "sha256:fd2a5403a75b54661182b75ec6132437a181209b901446ee5724b589af8edef1"}, + {file = "Pillow-10.0.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:2d7e91b4379f7a76b31c2dda84ab9e20c6220488e50f7822e59dac36b0cd92b1"}, + {file = "Pillow-10.0.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:19e9adb3f22d4c416e7cd79b01375b17159d6990003633ff1d8377e21b7f1b21"}, + {file = "Pillow-10.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:93139acd8109edcdeffd85e3af8ae7d88b258b3a1e13a038f542b79b6d255c54"}, + {file = "Pillow-10.0.1-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:92a23b0431941a33242b1f0ce6c88a952e09feeea9af4e8be48236a68ffe2205"}, + {file = "Pillow-10.0.1-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:cbe68deb8580462ca0d9eb56a81912f59eb4542e1ef8f987405e35a0179f4ea2"}, + {file = "Pillow-10.0.1-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:522ff4ac3aaf839242c6f4e5b406634bfea002469656ae8358644fc6c4856a3b"}, + {file = "Pillow-10.0.1-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:84efb46e8d881bb06b35d1d541aa87f574b58e87f781cbba8d200daa835b42e1"}, + {file = "Pillow-10.0.1-cp311-cp311-win_amd64.whl", hash = "sha256:898f1d306298ff40dc1b9ca24824f0488f6f039bc0e25cfb549d3195ffa17088"}, + {file = "Pillow-10.0.1-cp312-cp312-macosx_10_10_x86_64.whl", hash = "sha256:bcf1207e2f2385a576832af02702de104be71301c2696d0012b1b93fe34aaa5b"}, + {file = "Pillow-10.0.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:5d6c9049c6274c1bb565021367431ad04481ebb54872edecfcd6088d27edd6ed"}, + {file = "Pillow-10.0.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:28444cb6ad49726127d6b340217f0627abc8732f1194fd5352dec5e6a0105635"}, + {file = "Pillow-10.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:de596695a75496deb3b499c8c4f8e60376e0516e1a774e7bc046f0f48cd620ad"}, + {file = "Pillow-10.0.1-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:2872f2d7846cf39b3dbff64bc1104cc48c76145854256451d33c5faa55c04d1a"}, + {file = "Pillow-10.0.1-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:4ce90f8a24e1c15465048959f1e94309dfef93af272633e8f37361b824532e91"}, + {file = "Pillow-10.0.1-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:ee7810cf7c83fa227ba9125de6084e5e8b08c59038a7b2c9045ef4dde61663b4"}, + {file = "Pillow-10.0.1-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:b1be1c872b9b5fcc229adeadbeb51422a9633abd847c0ff87dc4ef9bb184ae08"}, + {file = "Pillow-10.0.1-cp312-cp312-win_amd64.whl", hash = "sha256:98533fd7fa764e5f85eebe56c8e4094db912ccbe6fbf3a58778d543cadd0db08"}, + {file = "Pillow-10.0.1-cp38-cp38-macosx_10_10_x86_64.whl", hash = "sha256:764d2c0daf9c4d40ad12fbc0abd5da3af7f8aa11daf87e4fa1b834000f4b6b0a"}, + {file = "Pillow-10.0.1-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:fcb59711009b0168d6ee0bd8fb5eb259c4ab1717b2f538bbf36bacf207ef7a68"}, + {file = "Pillow-10.0.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:697a06bdcedd473b35e50a7e7506b1d8ceb832dc238a336bd6f4f5aa91a4b500"}, + {file = "Pillow-10.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9f665d1e6474af9f9da5e86c2a3a2d2d6204e04d5af9c06b9d42afa6ebde3f21"}, + {file = "Pillow-10.0.1-cp38-cp38-manylinux_2_28_aarch64.whl", hash = "sha256:2fa6dd2661838c66f1a5473f3b49ab610c98a128fc08afbe81b91a1f0bf8c51d"}, + {file = "Pillow-10.0.1-cp38-cp38-manylinux_2_28_x86_64.whl", hash = "sha256:3a04359f308ebee571a3127fdb1bd01f88ba6f6fb6d087f8dd2e0d9bff43f2a7"}, + {file = "Pillow-10.0.1-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:723bd25051454cea9990203405fa6b74e043ea76d4968166dfd2569b0210886a"}, + {file = "Pillow-10.0.1-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:71671503e3015da1b50bd18951e2f9daf5b6ffe36d16f1eb2c45711a301521a7"}, + {file = "Pillow-10.0.1-cp38-cp38-win_amd64.whl", hash = "sha256:44e7e4587392953e5e251190a964675f61e4dae88d1e6edbe9f36d6243547ff3"}, + {file = "Pillow-10.0.1-cp39-cp39-macosx_10_10_x86_64.whl", hash = "sha256:3855447d98cced8670aaa63683808df905e956f00348732448b5a6df67ee5849"}, + {file = "Pillow-10.0.1-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:ed2d9c0704f2dc4fa980b99d565c0c9a543fe5101c25b3d60488b8ba80f0cce1"}, + {file = "Pillow-10.0.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f5bb289bb835f9fe1a1e9300d011eef4d69661bb9b34d5e196e5e82c4cb09b37"}, + {file = "Pillow-10.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3a0d3e54ab1df9df51b914b2233cf779a5a10dfd1ce339d0421748232cea9876"}, + {file = "Pillow-10.0.1-cp39-cp39-manylinux_2_28_aarch64.whl", hash = "sha256:2cc6b86ece42a11f16f55fe8903595eff2b25e0358dec635d0a701ac9586588f"}, + {file = "Pillow-10.0.1-cp39-cp39-manylinux_2_28_x86_64.whl", hash = "sha256:ca26ba5767888c84bf5a0c1a32f069e8204ce8c21d00a49c90dabeba00ce0145"}, + {file = "Pillow-10.0.1-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:f0b4b06da13275bc02adfeb82643c4a6385bd08d26f03068c2796f60d125f6f2"}, + {file = "Pillow-10.0.1-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:bc2e3069569ea9dbe88d6b8ea38f439a6aad8f6e7a6283a38edf61ddefb3a9bf"}, + {file = "Pillow-10.0.1-cp39-cp39-win_amd64.whl", hash = "sha256:8b451d6ead6e3500b6ce5c7916a43d8d8d25ad74b9102a629baccc0808c54971"}, + {file = "Pillow-10.0.1-pp310-pypy310_pp73-macosx_10_10_x86_64.whl", hash = "sha256:32bec7423cdf25c9038fef614a853c9d25c07590e1a870ed471f47fb80b244db"}, + {file = "Pillow-10.0.1-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b7cf63d2c6928b51d35dfdbda6f2c1fddbe51a6bc4a9d4ee6ea0e11670dd981e"}, + {file = "Pillow-10.0.1-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:f6d3d4c905e26354e8f9d82548475c46d8e0889538cb0657aa9c6f0872a37aa4"}, + {file = "Pillow-10.0.1-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:847e8d1017c741c735d3cd1883fa7b03ded4f825a6e5fcb9378fd813edee995f"}, + {file = "Pillow-10.0.1-pp39-pypy39_pp73-macosx_10_10_x86_64.whl", hash = "sha256:7f771e7219ff04b79e231d099c0a28ed83aa82af91fd5fa9fdb28f5b8d5addaf"}, + {file = "Pillow-10.0.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:459307cacdd4138edee3875bbe22a2492519e060660eaf378ba3b405d1c66317"}, + {file = "Pillow-10.0.1-pp39-pypy39_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:b059ac2c4c7a97daafa7dc850b43b2d3667def858a4f112d1aa082e5c3d6cf7d"}, + {file = "Pillow-10.0.1-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:d6caf3cd38449ec3cd8a68b375e0c6fe4b6fd04edb6c9766b55ef84a6e8ddf2d"}, + {file = "Pillow-10.0.1.tar.gz", hash = "sha256:d72967b06be9300fed5cfbc8b5bafceec48bf7cdc7dab66b1d2549035287191d"}, +] + +[package.extras] +docs = ["furo", "olefile", "sphinx (>=2.4)", "sphinx-copybutton", "sphinx-inline-tabs", "sphinx-removed-in", "sphinxext-opengraph"] +tests = ["check-manifest", "coverage", "defusedxml", "markdown2", "olefile", "packaging", "pyroma", "pytest", "pytest-cov", "pytest-timeout"] + +[[package]] +name = "pkgutil-resolve-name" +version = "1.3.10" +description = "Resolve a name to an object." +optional = false +python-versions = ">=3.6" +files = [ + {file = "pkgutil_resolve_name-1.3.10-py3-none-any.whl", hash = "sha256:ca27cc078d25c5ad71a9de0a7a330146c4e014c2462d9af19c6b828280649c5e"}, + {file = "pkgutil_resolve_name-1.3.10.tar.gz", hash = "sha256:357d6c9e6a755653cfd78893817c0853af365dd51ec97f3d358a819373bbd174"}, +] + +[[package]] +name = "platformdirs" +version = "3.2.0" +description = "A small Python package for determining appropriate platform-specific dirs, e.g. a \"user data dir\"." +optional = false +python-versions = ">=3.7" +files = [ + {file = "platformdirs-3.2.0-py3-none-any.whl", hash = "sha256:ebe11c0d7a805086e99506aa331612429a72ca7cd52a1f0d277dc4adc20cb10e"}, + {file = "platformdirs-3.2.0.tar.gz", hash = "sha256:d5b638ca397f25f979350ff789db335903d7ea010ab28903f57b27e1b16c2b08"}, +] + +[package.extras] +docs = ["furo (>=2022.12.7)", "proselint (>=0.13)", "sphinx (>=6.1.3)", "sphinx-autodoc-typehints (>=1.22,!=1.23.4)"] +test = ["appdirs (==1.4.4)", "covdefaults (>=2.3)", "pytest (>=7.2.2)", "pytest-cov (>=4)", "pytest-mock (>=3.10)"] + +[[package]] +name = "pluggy" +version = "1.0.0" +description = "plugin and hook calling mechanisms for python" +optional = false +python-versions = ">=3.6" +files = [ + {file = "pluggy-1.0.0-py2.py3-none-any.whl", hash = "sha256:74134bbf457f031a36d68416e1509f34bd5ccc019f0bcc952c7b909d06b37bd3"}, + {file = "pluggy-1.0.0.tar.gz", hash = "sha256:4224373bacce55f955a878bf9cfa763c1e360858e330072059e10bad68531159"}, +] + +[package.extras] +dev = ["pre-commit", "tox"] +testing = ["pytest", "pytest-benchmark"] + +[[package]] +name = "pre-commit" +version = "2.21.0" +description = "A framework for managing and maintaining multi-language pre-commit hooks." +optional = false +python-versions = ">=3.7" +files = [ + {file = "pre_commit-2.21.0-py2.py3-none-any.whl", hash = "sha256:e2f91727039fc39a92f58a588a25b87f936de6567eed4f0e673e0507edc75bad"}, + {file = "pre_commit-2.21.0.tar.gz", hash = "sha256:31ef31af7e474a8d8995027fefdfcf509b5c913ff31f2015b4ec4beb26a6f658"}, +] + +[package.dependencies] +cfgv = ">=2.0.0" +identify = ">=1.0.0" +nodeenv = ">=0.11.1" +pyyaml = ">=5.1" +virtualenv = ">=20.10.0" + +[[package]] +name = "prompt-toolkit" +version = "3.0.38" +description = "Library for building powerful interactive command lines in Python" +optional = false +python-versions = ">=3.7.0" +files = [ + {file = "prompt_toolkit-3.0.38-py3-none-any.whl", hash = "sha256:45ea77a2f7c60418850331366c81cf6b5b9cf4c7fd34616f733c5427e6abbb1f"}, + {file = "prompt_toolkit-3.0.38.tar.gz", hash = "sha256:23ac5d50538a9a38c8bde05fecb47d0b403ecd0662857a86f886f798563d5b9b"}, +] + +[package.dependencies] +wcwidth = "*" + +[[package]] +name = "psutil" +version = "5.9.5" +description = "Cross-platform lib for process and system monitoring in Python." +optional = false +python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*" +files = [ + {file = "psutil-5.9.5-cp27-cp27m-macosx_10_9_x86_64.whl", hash = "sha256:be8929ce4313f9f8146caad4272f6abb8bf99fc6cf59344a3167ecd74f4f203f"}, + {file = "psutil-5.9.5-cp27-cp27m-manylinux2010_i686.whl", hash = "sha256:ab8ed1a1d77c95453db1ae00a3f9c50227ebd955437bcf2a574ba8adbf6a74d5"}, + {file = "psutil-5.9.5-cp27-cp27m-manylinux2010_x86_64.whl", hash = "sha256:4aef137f3345082a3d3232187aeb4ac4ef959ba3d7c10c33dd73763fbc063da4"}, + {file = "psutil-5.9.5-cp27-cp27mu-manylinux2010_i686.whl", hash = "sha256:ea8518d152174e1249c4f2a1c89e3e6065941df2fa13a1ab45327716a23c2b48"}, + {file = "psutil-5.9.5-cp27-cp27mu-manylinux2010_x86_64.whl", hash = "sha256:acf2aef9391710afded549ff602b5887d7a2349831ae4c26be7c807c0a39fac4"}, + {file = "psutil-5.9.5-cp27-none-win32.whl", hash = "sha256:5b9b8cb93f507e8dbaf22af6a2fd0ccbe8244bf30b1baad6b3954e935157ae3f"}, + {file = "psutil-5.9.5-cp27-none-win_amd64.whl", hash = "sha256:8c5f7c5a052d1d567db4ddd231a9d27a74e8e4a9c3f44b1032762bd7b9fdcd42"}, + {file = "psutil-5.9.5-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:3c6f686f4225553615612f6d9bc21f1c0e305f75d7d8454f9b46e901778e7217"}, + {file = "psutil-5.9.5-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:7a7dd9997128a0d928ed4fb2c2d57e5102bb6089027939f3b722f3a210f9a8da"}, + {file = "psutil-5.9.5-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:89518112647f1276b03ca97b65cc7f64ca587b1eb0278383017c2a0dcc26cbe4"}, + {file = "psutil-5.9.5-cp36-abi3-win32.whl", hash = "sha256:104a5cc0e31baa2bcf67900be36acde157756b9c44017b86b2c049f11957887d"}, + {file = "psutil-5.9.5-cp36-abi3-win_amd64.whl", hash = "sha256:b258c0c1c9d145a1d5ceffab1134441c4c5113b2417fafff7315a917a026c3c9"}, + {file = "psutil-5.9.5-cp38-abi3-macosx_11_0_arm64.whl", hash = "sha256:c607bb3b57dc779d55e1554846352b4e358c10fff3abf3514a7a6601beebdb30"}, + {file = "psutil-5.9.5.tar.gz", hash = "sha256:5410638e4df39c54d957fc51ce03048acd8e6d60abc0f5107af51e5fb566eb3c"}, +] + +[package.extras] +test = ["enum34", "ipaddress", "mock", "pywin32", "wmi"] + +[[package]] +name = "ptyprocess" +version = "0.7.0" +description = "Run a subprocess in a pseudo terminal" +optional = false +python-versions = "*" +files = [ + {file = "ptyprocess-0.7.0-py2.py3-none-any.whl", hash = "sha256:4b41f3967fce3af57cc7e94b888626c18bf37a083e3651ca8feeb66d492fef35"}, + {file = "ptyprocess-0.7.0.tar.gz", hash = "sha256:5c5d0a3b48ceee0b48485e0c26037c0acd7d29765ca3fbb5cb3831d347423220"}, +] + +[[package]] +name = "pure-eval" +version = "0.2.2" +description = "Safely evaluate AST nodes without side effects" +optional = false +python-versions = "*" +files = [ + {file = "pure_eval-0.2.2-py3-none-any.whl", hash = "sha256:01eaab343580944bc56080ebe0a674b39ec44a945e6d09ba7db3cb8cec289350"}, + {file = "pure_eval-0.2.2.tar.gz", hash = "sha256:2b45320af6dfaa1750f543d714b6d1c520a1688dec6fd24d339063ce0aaa9ac3"}, +] + +[package.extras] +tests = ["pytest"] + +[[package]] +name = "pweave" +version = "0.30.3" +description = "Scientific reports with embedded python computations with reST, LaTeX or markdown" +optional = false +python-versions = "*" +files = [ + {file = "Pweave-0.30.3-py2.py3-none-any.whl", hash = "sha256:60cf8de680084b5423caa3a2131d4ff981c236f12f84f9d969a41f6632a44165"}, + {file = "Pweave-0.30.3.tar.gz", hash = "sha256:5e5298d90e06414a01f48e0d6aa4c36a70c5f223d929f2a9c7e2d388451c7357"}, +] + +[package.dependencies] +ipykernel = "*" +ipython = ">=6.0" +jupyter-client = "*" +markdown = "*" +nbconvert = "*" +nbformat = "*" +pygments = "*" + +[package.extras] +doc = ["sphinx", "sphinx-rtd-theme"] +test = ["coverage", "ipython", "matplotlib", "nose", "notebook", "scipy"] + +[[package]] +name = "py4j" +version = "0.10.9.7" +description = "Enables Python programs to dynamically access arbitrary Java objects" +optional = false +python-versions = "*" +files = [ + {file = "py4j-0.10.9.7-py2.py3-none-any.whl", hash = "sha256:85defdfd2b2376eb3abf5ca6474b51ab7e0de341c75a02f46dc9b5976f5a5c1b"}, + {file = "py4j-0.10.9.7.tar.gz", hash = "sha256:0b6e5315bb3ada5cf62ac651d107bb2ebc02def3dee9d9548e3baac644ea8dbb"}, +] + +[[package]] +name = "pycparser" +version = "2.21" +description = "C parser in Python" +optional = false +python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*" +files = [ + {file = "pycparser-2.21-py2.py3-none-any.whl", hash = "sha256:8ee45429555515e1f6b185e78100aea234072576aa43ab53aefcae078162fca9"}, + {file = "pycparser-2.21.tar.gz", hash = "sha256:e644fdec12f7872f86c58ff790da456218b10f863970249516d60a5eaca77206"}, +] + +[[package]] +name = "pygments" +version = "2.15.1" +description = "Pygments is a syntax highlighting package written in Python." +optional = false +python-versions = ">=3.7" +files = [ + {file = "Pygments-2.15.1-py3-none-any.whl", hash = "sha256:db2db3deb4b4179f399a09054b023b6a586b76499d36965813c71aa8ed7b5fd1"}, + {file = "Pygments-2.15.1.tar.gz", hash = "sha256:8ace4d3c1dd481894b2005f560ead0f9f19ee64fe983366be1a21e171d12775c"}, +] + +[package.extras] +plugins = ["importlib-metadata"] + +[[package]] +name = "pyparsing" +version = "3.0.9" +description = "pyparsing module - Classes and methods to define and execute parsing grammars" +optional = false +python-versions = ">=3.6.8" +files = [ + {file = "pyparsing-3.0.9-py3-none-any.whl", hash = "sha256:5026bae9a10eeaefb61dab2f09052b9f4307d44aee4eda64b309723d8d206bbc"}, + {file = "pyparsing-3.0.9.tar.gz", hash = "sha256:2b020ecf7d21b687f219b71ecad3631f644a47f01403fa1d1036b0c6416d70fb"}, +] + +[package.extras] +diagrams = ["jinja2", "railroad-diagrams"] + +[[package]] +name = "pyproj" +version = "3.4.1" +description = "Python interface to PROJ (cartographic projections and coordinate transformations library)" +optional = false +python-versions = ">=3.8" +files = [ + {file = "pyproj-3.4.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:e463c687007861a9949909211986850cfc2e72930deda0d06449ef2e315db534"}, + {file = "pyproj-3.4.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:2f87f16b902c8b2af007295c63a435f043db9e40bd45e6f96962c7b8cd08fdb5"}, + {file = "pyproj-3.4.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c60d112d8f1621a606b7f2adb0b1582f80498e663413d2ba9f5df1c93d99f432"}, + {file = "pyproj-3.4.1-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:f38dea459e22e86326b1c7d47718a3e10c7a27910cf5eb86ea2679b8084d0c4e"}, + {file = "pyproj-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5a53acbde511a7a9e1873c7f93c68f35b8c3653467b77195fe18e847555dcb7a"}, + {file = "pyproj-3.4.1-cp310-cp310-win32.whl", hash = "sha256:0c7b32382ae22a9bf5b690d24c7b4c0fb89ba313c3a91ef1a8c54b50baf10954"}, + {file = "pyproj-3.4.1-cp310-cp310-win_amd64.whl", hash = "sha256:6bdac3bc1899fcc4021be06d303b342923fb8311fe06f8d862c348a1a0e78b41"}, + {file = "pyproj-3.4.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:cd9f9c409f465834988ce0aa8c1ed496081c6957f2e5ef40ed28de04397d3c0b"}, + {file = "pyproj-3.4.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:0406f64ff59eb3342efb102c9f31536430aa5cde5ef0bfabd5aaccb73dd8cd5a"}, + {file = "pyproj-3.4.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a98fe3e53be428e67ae6a9ee9affff92346622e0e3ea0cbc15dce939b318d395"}, + {file = "pyproj-3.4.1-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:0189fdd7aa789542a7a623010dfff066c5849b24397f81f860ec3ee085cbf55c"}, + {file = "pyproj-3.4.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5f3f75b030cf811f040c90a8758a20115e8746063e4cad0d0e941a4954d1219b"}, + {file = "pyproj-3.4.1-cp311-cp311-win32.whl", hash = "sha256:ef8c30c62fe4e386e523e14e1e83bd460f745bd2c8dfd0d0c327f9460c4d3c0c"}, + {file = "pyproj-3.4.1-cp311-cp311-win_amd64.whl", hash = "sha256:2d1e7f42da205e0534831ae9aa9cee0353ab8c1aab2c369474adbb060294d98a"}, + {file = "pyproj-3.4.1-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:a5eada965e8ac24e783f2493d1d9bcd11c5c93959bd43558224dd31d9faebd1c"}, + {file = "pyproj-3.4.1-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:19f5de1a7c3b81b676d846350d4bdf2ae6af13b9a450d1881706f088ecad0e2c"}, + {file = "pyproj-3.4.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:57ec7d2b7f2773d877927abc72e2229ef8530c09181be0e28217742bae1bc4f5"}, + {file = "pyproj-3.4.1-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:a30d78e619dae5cd1bb69addae2f1e5f8ee1b4a8ab4f3d954e9eaf41948db506"}, + {file = "pyproj-3.4.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a32e1d12340ad93232b7ea4dc1a4f4b21fa9fa9efa4b293adad45be7af6b51ec"}, + {file = "pyproj-3.4.1-cp38-cp38-win32.whl", hash = "sha256:ce50126dad7cd4749ab86fc4c8b54ec0898149ce6710ab5c93c76a54a4afa249"}, + {file = "pyproj-3.4.1-cp38-cp38-win_amd64.whl", hash = "sha256:129234afa179c8293b010ea4f73655ff7b20b5afdf7fac170f223bcf0ed6defd"}, + {file = "pyproj-3.4.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:231c038c6b65395c41ae3362320f03ce8054cb54dc63556e605695e5d461a27e"}, + {file = "pyproj-3.4.1-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:e9d82df555cf19001bac40e1de0e40fb762dec785685b77edd6993286c01b7f7"}, + {file = "pyproj-3.4.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e8c0d1ac9ef5a4d2e6501a4b30136c55f1e1db049d1626cc313855c4f97d196d"}, + {file = "pyproj-3.4.1-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:97065fe82e80f7e2740e7897a0e36e8defc0a3614927f0276b4f1d1ea1ef66fa"}, + {file = "pyproj-3.4.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2bd633f3b8ca6eb09135dfaf06f09e2869deb139985aab26d728e8a60c9938b9"}, + {file = "pyproj-3.4.1-cp39-cp39-win32.whl", hash = "sha256:da96319b137cfd66f0bae0e300cdc77dd17af4785b9360a9bdddb1d7176a0bbb"}, + {file = "pyproj-3.4.1-cp39-cp39-win_amd64.whl", hash = "sha256:7aef19d5a0a3b2d6b17f7dc9a87af722e71139cd1eea7eb82ed062a8a4b0e272"}, + {file = "pyproj-3.4.1-pp38-pypy38_pp73-macosx_10_9_x86_64.whl", hash = "sha256:8078c90cea07d53e3406c7c84cbf76a2ac0ffc580c365f13801575486b9d558c"}, + {file = "pyproj-3.4.1-pp38-pypy38_pp73-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:321b82210dc5271558573d0874b9967c5a25872a28d0168049ddabe8bfecffce"}, + {file = "pyproj-3.4.1-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:25a5425cd2a0b16f5f944d49165196eebaa60b898a08c404a644c29e6a7a04b3"}, + {file = "pyproj-3.4.1-pp39-pypy39_pp73-macosx_10_9_x86_64.whl", hash = "sha256:3d70ca5933cddbe6f51396006fb9fc78bc2b1f9d28775922453c4b04625a7efb"}, + {file = "pyproj-3.4.1-pp39-pypy39_pp73-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:c240fe6bcb5c325b50fc967d5458d708412633f4f05fefc7fb14c14254ebf421"}, + {file = "pyproj-3.4.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ef76abfee1a0676ef973470abe11e22998750f2bd944afaf76d44ad70b538c06"}, + {file = "pyproj-3.4.1.tar.gz", hash = "sha256:261eb29b1d55b1eb7f336127344d9b31284d950a9446d1e0d1c2411f7dd8e3ac"}, +] + +[package.dependencies] +certifi = "*" + +[[package]] +name = "pyrsistent" +version = "0.19.3" +description = "Persistent/Functional/Immutable data structures" +optional = false +python-versions = ">=3.7" +files = [ + {file = "pyrsistent-0.19.3-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:20460ac0ea439a3e79caa1dbd560344b64ed75e85d8703943e0b66c2a6150e4a"}, + {file = "pyrsistent-0.19.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4c18264cb84b5e68e7085a43723f9e4c1fd1d935ab240ce02c0324a8e01ccb64"}, + {file = "pyrsistent-0.19.3-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:4b774f9288dda8d425adb6544e5903f1fb6c273ab3128a355c6b972b7df39dcf"}, + {file = "pyrsistent-0.19.3-cp310-cp310-win32.whl", hash = "sha256:5a474fb80f5e0d6c9394d8db0fc19e90fa540b82ee52dba7d246a7791712f74a"}, + {file = "pyrsistent-0.19.3-cp310-cp310-win_amd64.whl", hash = "sha256:49c32f216c17148695ca0e02a5c521e28a4ee6c5089f97e34fe24163113722da"}, + {file = "pyrsistent-0.19.3-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:f0774bf48631f3a20471dd7c5989657b639fd2d285b861237ea9e82c36a415a9"}, + {file = "pyrsistent-0.19.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3ab2204234c0ecd8b9368dbd6a53e83c3d4f3cab10ecaf6d0e772f456c442393"}, + {file = "pyrsistent-0.19.3-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:e42296a09e83028b3476f7073fcb69ffebac0e66dbbfd1bd847d61f74db30f19"}, + {file = "pyrsistent-0.19.3-cp311-cp311-win32.whl", hash = "sha256:64220c429e42a7150f4bfd280f6f4bb2850f95956bde93c6fda1b70507af6ef3"}, + {file = "pyrsistent-0.19.3-cp311-cp311-win_amd64.whl", hash = "sha256:016ad1afadf318eb7911baa24b049909f7f3bb2c5b1ed7b6a8f21db21ea3faa8"}, + {file = "pyrsistent-0.19.3-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:c4db1bd596fefd66b296a3d5d943c94f4fac5bcd13e99bffe2ba6a759d959a28"}, + {file = "pyrsistent-0.19.3-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:aeda827381f5e5d65cced3024126529ddc4289d944f75e090572c77ceb19adbf"}, + {file = "pyrsistent-0.19.3-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:42ac0b2f44607eb92ae88609eda931a4f0dfa03038c44c772e07f43e738bcac9"}, + {file = "pyrsistent-0.19.3-cp37-cp37m-win32.whl", hash = "sha256:e8f2b814a3dc6225964fa03d8582c6e0b6650d68a232df41e3cc1b66a5d2f8d1"}, + {file = "pyrsistent-0.19.3-cp37-cp37m-win_amd64.whl", hash = "sha256:c9bb60a40a0ab9aba40a59f68214eed5a29c6274c83b2cc206a359c4a89fa41b"}, + {file = "pyrsistent-0.19.3-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:a2471f3f8693101975b1ff85ffd19bb7ca7dd7c38f8a81701f67d6b4f97b87d8"}, + {file = "pyrsistent-0.19.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cc5d149f31706762c1f8bda2e8c4f8fead6e80312e3692619a75301d3dbb819a"}, + {file = "pyrsistent-0.19.3-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:3311cb4237a341aa52ab8448c27e3a9931e2ee09561ad150ba94e4cfd3fc888c"}, + {file = "pyrsistent-0.19.3-cp38-cp38-win32.whl", hash = "sha256:f0e7c4b2f77593871e918be000b96c8107da48444d57005b6a6bc61fb4331b2c"}, + {file = "pyrsistent-0.19.3-cp38-cp38-win_amd64.whl", hash = "sha256:c147257a92374fde8498491f53ffa8f4822cd70c0d85037e09028e478cababb7"}, + {file = "pyrsistent-0.19.3-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:b735e538f74ec31378f5a1e3886a26d2ca6351106b4dfde376a26fc32a044edc"}, + {file = "pyrsistent-0.19.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:99abb85579e2165bd8522f0c0138864da97847875ecbd45f3e7e2af569bfc6f2"}, + {file = "pyrsistent-0.19.3-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:3a8cb235fa6d3fd7aae6a4f1429bbb1fec1577d978098da1252f0489937786f3"}, + {file = "pyrsistent-0.19.3-cp39-cp39-win32.whl", hash = "sha256:c74bed51f9b41c48366a286395c67f4e894374306b197e62810e0fdaf2364da2"}, + {file = "pyrsistent-0.19.3-cp39-cp39-win_amd64.whl", hash = "sha256:878433581fc23e906d947a6814336eee031a00e6defba224234169ae3d3d6a98"}, + {file = "pyrsistent-0.19.3-py3-none-any.whl", hash = "sha256:ccf0d6bd208f8111179f0c26fdf84ed7c3891982f2edaeae7422575f47e66b64"}, + {file = "pyrsistent-0.19.3.tar.gz", hash = "sha256:1a2994773706bbb4995c31a97bc94f1418314923bd1048c6d964837040376440"}, +] + +[[package]] +name = "pyspark" +version = "3.4.0" +description = "Apache Spark Python API" +optional = false +python-versions = ">=3.7" +files = [ + {file = "pyspark-3.4.0.tar.gz", hash = "sha256:167a23e11854adb37f8602de6fcc3a4f96fd5f1e323b9bb83325f38408c5aafd"}, +] + +[package.dependencies] +py4j = "0.10.9.7" + +[package.extras] +connect = ["googleapis-common-protos (>=1.56.4)", "grpcio (>=1.48.1)", "grpcio-status (>=1.48.1)", "numpy (>=1.15)", "pandas (>=1.0.5)", "pyarrow (>=1.0.0)"] +ml = ["numpy (>=1.15)"] +mllib = ["numpy (>=1.15)"] +pandas-on-spark = ["numpy (>=1.15)", "pandas (>=1.0.5)", "pyarrow (>=1.0.0)"] +sql = ["numpy (>=1.15)", "pandas (>=1.0.5)", "pyarrow (>=1.0.0)"] + +[[package]] +name = "pytest" +version = "7.3.1" +description = "pytest: simple powerful testing with Python" +optional = false +python-versions = ">=3.7" +files = [ + {file = "pytest-7.3.1-py3-none-any.whl", hash = "sha256:3799fa815351fea3a5e96ac7e503a96fa51cc9942c3753cda7651b93c1cfa362"}, + {file = "pytest-7.3.1.tar.gz", hash = "sha256:434afafd78b1d78ed0addf160ad2b77a30d35d4bdf8af234fe621919d9ed15e3"}, +] + +[package.dependencies] +colorama = {version = "*", markers = "sys_platform == \"win32\""} +exceptiongroup = {version = ">=1.0.0rc8", markers = "python_version < \"3.11\""} +iniconfig = "*" +packaging = "*" +pluggy = ">=0.12,<2.0" +tomli = {version = ">=1.0.0", markers = "python_version < \"3.11\""} + +[package.extras] +testing = ["argcomplete", "attrs (>=19.2.0)", "hypothesis (>=3.56)", "mock", "nose", "pygments (>=2.7.2)", "requests", "xmlschema"] + +[[package]] +name = "pytest-cov" +version = "4.0.0" +description = "Pytest plugin for measuring coverage." +optional = false +python-versions = ">=3.6" +files = [ + {file = "pytest-cov-4.0.0.tar.gz", hash = "sha256:996b79efde6433cdbd0088872dbc5fb3ed7fe1578b68cdbba634f14bb8dd0470"}, + {file = "pytest_cov-4.0.0-py3-none-any.whl", hash = "sha256:2feb1b751d66a8bd934e5edfa2e961d11309dc37b73b0eabe73b5945fee20f6b"}, +] + +[package.dependencies] +coverage = {version = ">=5.2.1", extras = ["toml"]} +pytest = ">=4.6" + +[package.extras] +testing = ["fields", "hunter", "process-tests", "pytest-xdist", "six", "virtualenv"] + +[[package]] +name = "python-dateutil" +version = "2.8.2" +description = "Extensions to the standard Python datetime module" +optional = false +python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,>=2.7" +files = [ + {file = "python-dateutil-2.8.2.tar.gz", hash = "sha256:0123cacc1627ae19ddf3c27a5de5bd67ee4586fbdd6440d9748f8abb483d3e86"}, + {file = "python_dateutil-2.8.2-py2.py3-none-any.whl", hash = "sha256:961d03dc3453ebbc59dbdea9e4e11c5651520a876d0f4db161e8674aae935da9"}, +] + +[package.dependencies] +six = ">=1.5" + +[[package]] +name = "pytz" +version = "2023.3" +description = "World timezone definitions, modern and historical" +optional = false +python-versions = "*" +files = [ + {file = "pytz-2023.3-py2.py3-none-any.whl", hash = "sha256:a151b3abb88eda1d4e34a9814df37de2a80e301e68ba0fd856fb9b46bfbbbffb"}, + {file = "pytz-2023.3.tar.gz", hash = "sha256:1d8ce29db189191fb55338ee6d0387d82ab59f3d00eac103412d64e0ebd0c588"}, +] + +[[package]] +name = "pywin32" +version = "306" +description = "Python for Window Extensions" +optional = false +python-versions = "*" +files = [ + {file = "pywin32-306-cp310-cp310-win32.whl", hash = "sha256:06d3420a5155ba65f0b72f2699b5bacf3109f36acbe8923765c22938a69dfc8d"}, + {file = "pywin32-306-cp310-cp310-win_amd64.whl", hash = "sha256:84f4471dbca1887ea3803d8848a1616429ac94a4a8d05f4bc9c5dcfd42ca99c8"}, + {file = "pywin32-306-cp311-cp311-win32.whl", hash = "sha256:e65028133d15b64d2ed8f06dd9fbc268352478d4f9289e69c190ecd6818b6407"}, + {file = "pywin32-306-cp311-cp311-win_amd64.whl", hash = "sha256:a7639f51c184c0272e93f244eb24dafca9b1855707d94c192d4a0b4c01e1100e"}, + {file = "pywin32-306-cp311-cp311-win_arm64.whl", hash = "sha256:70dba0c913d19f942a2db25217d9a1b726c278f483a919f1abfed79c9cf64d3a"}, + {file = "pywin32-306-cp312-cp312-win32.whl", hash = "sha256:383229d515657f4e3ed1343da8be101000562bf514591ff383ae940cad65458b"}, + {file = "pywin32-306-cp312-cp312-win_amd64.whl", hash = "sha256:37257794c1ad39ee9be652da0462dc2e394c8159dfd913a8a4e8eb6fd346da0e"}, + {file = "pywin32-306-cp312-cp312-win_arm64.whl", hash = "sha256:5821ec52f6d321aa59e2db7e0a35b997de60c201943557d108af9d4ae1ec7040"}, + {file = "pywin32-306-cp37-cp37m-win32.whl", hash = "sha256:1c73ea9a0d2283d889001998059f5eaaba3b6238f767c9cf2833b13e6a685f65"}, + {file = "pywin32-306-cp37-cp37m-win_amd64.whl", hash = "sha256:72c5f621542d7bdd4fdb716227be0dd3f8565c11b280be6315b06ace35487d36"}, + {file = "pywin32-306-cp38-cp38-win32.whl", hash = "sha256:e4c092e2589b5cf0d365849e73e02c391c1349958c5ac3e9d5ccb9a28e017b3a"}, + {file = "pywin32-306-cp38-cp38-win_amd64.whl", hash = "sha256:e8ac1ae3601bee6ca9f7cb4b5363bf1c0badb935ef243c4733ff9a393b1690c0"}, + {file = "pywin32-306-cp39-cp39-win32.whl", hash = "sha256:e25fd5b485b55ac9c057f67d94bc203f3f6595078d1fb3b458c9c28b7153a802"}, + {file = "pywin32-306-cp39-cp39-win_amd64.whl", hash = "sha256:39b61c15272833b5c329a2989999dcae836b1eed650252ab1b7bfbe1d59f30f4"}, +] + +[[package]] +name = "pyyaml" +version = "6.0" +description = "YAML parser and emitter for Python" +optional = false +python-versions = ">=3.6" +files = [ + {file = "PyYAML-6.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:d4db7c7aef085872ef65a8fd7d6d09a14ae91f691dec3e87ee5ee0539d516f53"}, + {file = "PyYAML-6.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:9df7ed3b3d2e0ecfe09e14741b857df43adb5a3ddadc919a2d94fbdf78fea53c"}, + {file = "PyYAML-6.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:77f396e6ef4c73fdc33a9157446466f1cff553d979bd00ecb64385760c6babdc"}, + {file = "PyYAML-6.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:a80a78046a72361de73f8f395f1f1e49f956c6be882eed58505a15f3e430962b"}, + {file = "PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:f84fbc98b019fef2ee9a1cb3ce93e3187a6df0b2538a651bfb890254ba9f90b5"}, + {file = "PyYAML-6.0-cp310-cp310-win32.whl", hash = "sha256:2cd5df3de48857ed0544b34e2d40e9fac445930039f3cfe4bcc592a1f836d513"}, + {file = "PyYAML-6.0-cp310-cp310-win_amd64.whl", hash = "sha256:daf496c58a8c52083df09b80c860005194014c3698698d1a57cbcfa182142a3a"}, + {file = "PyYAML-6.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:d4b0ba9512519522b118090257be113b9468d804b19d63c71dbcf4a48fa32358"}, + {file = "PyYAML-6.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:81957921f441d50af23654aa6c5e5eaf9b06aba7f0a19c18a538dc7ef291c5a1"}, + {file = "PyYAML-6.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:afa17f5bc4d1b10afd4466fd3a44dc0e245382deca5b3c353d8b757f9e3ecb8d"}, + {file = "PyYAML-6.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:dbad0e9d368bb989f4515da330b88a057617d16b6a8245084f1b05400f24609f"}, + {file = "PyYAML-6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:432557aa2c09802be39460360ddffd48156e30721f5e8d917f01d31694216782"}, + {file = "PyYAML-6.0-cp311-cp311-win32.whl", hash = "sha256:bfaef573a63ba8923503d27530362590ff4f576c626d86a9fed95822a8255fd7"}, + {file = "PyYAML-6.0-cp311-cp311-win_amd64.whl", hash = "sha256:01b45c0191e6d66c470b6cf1b9531a771a83c1c4208272ead47a3ae4f2f603bf"}, + {file = "PyYAML-6.0-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:897b80890765f037df3403d22bab41627ca8811ae55e9a722fd0392850ec4d86"}, + {file = "PyYAML-6.0-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:50602afada6d6cbfad699b0c7bb50d5ccffa7e46a3d738092afddc1f9758427f"}, + {file = "PyYAML-6.0-cp36-cp36m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:48c346915c114f5fdb3ead70312bd042a953a8ce5c7106d5bfb1a5254e47da92"}, + {file = "PyYAML-6.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:98c4d36e99714e55cfbaaee6dd5badbc9a1ec339ebfc3b1f52e293aee6bb71a4"}, + {file = "PyYAML-6.0-cp36-cp36m-win32.whl", hash = "sha256:0283c35a6a9fbf047493e3a0ce8d79ef5030852c51e9d911a27badfde0605293"}, + {file = "PyYAML-6.0-cp36-cp36m-win_amd64.whl", hash = "sha256:07751360502caac1c067a8132d150cf3d61339af5691fe9e87803040dbc5db57"}, + {file = "PyYAML-6.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:819b3830a1543db06c4d4b865e70ded25be52a2e0631ccd2f6a47a2822f2fd7c"}, + {file = "PyYAML-6.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:473f9edb243cb1935ab5a084eb238d842fb8f404ed2193a915d1784b5a6b5fc0"}, + {file = "PyYAML-6.0-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:0ce82d761c532fe4ec3f87fc45688bdd3a4c1dc5e0b4a19814b9009a29baefd4"}, + {file = "PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:231710d57adfd809ef5d34183b8ed1eeae3f76459c18fb4a0b373ad56bedcdd9"}, + {file = "PyYAML-6.0-cp37-cp37m-win32.whl", hash = "sha256:c5687b8d43cf58545ade1fe3e055f70eac7a5a1a0bf42824308d868289a95737"}, + {file = "PyYAML-6.0-cp37-cp37m-win_amd64.whl", hash = "sha256:d15a181d1ecd0d4270dc32edb46f7cb7733c7c508857278d3d378d14d606db2d"}, + {file = "PyYAML-6.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:0b4624f379dab24d3725ffde76559cff63d9ec94e1736b556dacdfebe5ab6d4b"}, + {file = "PyYAML-6.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:213c60cd50106436cc818accf5baa1aba61c0189ff610f64f4a3e8c6726218ba"}, + {file = "PyYAML-6.0-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:9fa600030013c4de8165339db93d182b9431076eb98eb40ee068700c9c813e34"}, + {file = "PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:277a0ef2981ca40581a47093e9e2d13b3f1fbbeffae064c1d21bfceba2030287"}, + {file = "PyYAML-6.0-cp38-cp38-win32.whl", hash = "sha256:d4eccecf9adf6fbcc6861a38015c2a64f38b9d94838ac1810a9023a0609e1b78"}, + {file = "PyYAML-6.0-cp38-cp38-win_amd64.whl", hash = "sha256:1e4747bc279b4f613a09eb64bba2ba602d8a6664c6ce6396a4d0cd413a50ce07"}, + {file = "PyYAML-6.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:055d937d65826939cb044fc8c9b08889e8c743fdc6a32b33e2390f66013e449b"}, + {file = "PyYAML-6.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:e61ceaab6f49fb8bdfaa0f92c4b57bcfbea54c09277b1b4f7ac376bfb7a7c174"}, + {file = "PyYAML-6.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d67d839ede4ed1b28a4e8909735fc992a923cdb84e618544973d7dfc71540803"}, + {file = "PyYAML-6.0-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:cba8c411ef271aa037d7357a2bc8f9ee8b58b9965831d9e51baf703280dc73d3"}, + {file = "PyYAML-6.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:40527857252b61eacd1d9af500c3337ba8deb8fc298940291486c465c8b46ec0"}, + {file = "PyYAML-6.0-cp39-cp39-win32.whl", hash = "sha256:b5b9eccad747aabaaffbc6064800670f0c297e52c12754eb1d976c57e4f74dcb"}, + {file = "PyYAML-6.0-cp39-cp39-win_amd64.whl", hash = "sha256:b3d267842bf12586ba6c734f89d1f5b871df0273157918b0ccefa29deb05c21c"}, + {file = "PyYAML-6.0.tar.gz", hash = "sha256:68fb519c14306fec9720a2a5b45bc9f0c8d1b9c72adf45c37baedfcd949c35a2"}, +] + +[[package]] +name = "pyzmq" +version = "25.0.2" +description = "Python bindings for 0MQ" +optional = false +python-versions = ">=3.6" +files = [ + {file = "pyzmq-25.0.2-cp310-cp310-macosx_10_15_universal2.whl", hash = "sha256:ac178e666c097c8d3deb5097b58cd1316092fc43e8ef5b5fdb259b51da7e7315"}, + {file = "pyzmq-25.0.2-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:659e62e1cbb063151c52f5b01a38e1df6b54feccfa3e2509d44c35ca6d7962ee"}, + {file = "pyzmq-25.0.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8280ada89010735a12b968ec3ea9a468ac2e04fddcc1cede59cb7f5178783b9c"}, + {file = "pyzmq-25.0.2-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:a9b5eeb5278a8a636bb0abdd9ff5076bcbb836cd2302565df53ff1fa7d106d54"}, + {file = "pyzmq-25.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9a2e5fe42dfe6b73ca120b97ac9f34bfa8414feb15e00e37415dbd51cf227ef6"}, + {file = "pyzmq-25.0.2-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:827bf60e749e78acb408a6c5af6688efbc9993e44ecc792b036ec2f4b4acf485"}, + {file = "pyzmq-25.0.2-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:7b504ae43d37e282301da586529e2ded8b36d4ee2cd5e6db4386724ddeaa6bbc"}, + {file = "pyzmq-25.0.2-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:cb1f69a0a2a2b1aae8412979dd6293cc6bcddd4439bf07e4758d864ddb112354"}, + {file = "pyzmq-25.0.2-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:2b9c9cc965cdf28381e36da525dcb89fc1571d9c54800fdcd73e3f73a2fc29bd"}, + {file = "pyzmq-25.0.2-cp310-cp310-win32.whl", hash = "sha256:24abbfdbb75ac5039205e72d6c75f10fc39d925f2df8ff21ebc74179488ebfca"}, + {file = "pyzmq-25.0.2-cp310-cp310-win_amd64.whl", hash = "sha256:6a821a506822fac55d2df2085a52530f68ab15ceed12d63539adc32bd4410f6e"}, + {file = "pyzmq-25.0.2-cp311-cp311-macosx_10_15_universal2.whl", hash = "sha256:9af0bb0277e92f41af35e991c242c9c71920169d6aa53ade7e444f338f4c8128"}, + {file = "pyzmq-25.0.2-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:54a96cf77684a3a537b76acfa7237b1e79a8f8d14e7f00e0171a94b346c5293e"}, + {file = "pyzmq-25.0.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:88649b19ede1cab03b96b66c364cbbf17c953615cdbc844f7f6e5f14c5e5261c"}, + {file = "pyzmq-25.0.2-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:715cff7644a80a7795953c11b067a75f16eb9fc695a5a53316891ebee7f3c9d5"}, + {file = "pyzmq-25.0.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:312b3f0f066b4f1d17383aae509bacf833ccaf591184a1f3c7a1661c085063ae"}, + {file = "pyzmq-25.0.2-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:d488c5c8630f7e782e800869f82744c3aca4aca62c63232e5d8c490d3d66956a"}, + {file = "pyzmq-25.0.2-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:38d9f78d69bcdeec0c11e0feb3bc70f36f9b8c44fc06e5d06d91dc0a21b453c7"}, + {file = "pyzmq-25.0.2-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:3059a6a534c910e1d5d068df42f60d434f79e6cc6285aa469b384fa921f78cf8"}, + {file = "pyzmq-25.0.2-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:6526d097b75192f228c09d48420854d53dfbc7abbb41b0e26f363ccb26fbc177"}, + {file = "pyzmq-25.0.2-cp311-cp311-win32.whl", hash = "sha256:5c5fbb229e40a89a2fe73d0c1181916f31e30f253cb2d6d91bea7927c2e18413"}, + {file = "pyzmq-25.0.2-cp311-cp311-win_amd64.whl", hash = "sha256:ed15e3a2c3c2398e6ae5ce86d6a31b452dfd6ad4cd5d312596b30929c4b6e182"}, + {file = "pyzmq-25.0.2-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:032f5c8483c85bf9c9ca0593a11c7c749d734ce68d435e38c3f72e759b98b3c9"}, + {file = "pyzmq-25.0.2-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:374b55516393bfd4d7a7daa6c3b36d6dd6a31ff9d2adad0838cd6a203125e714"}, + {file = "pyzmq-25.0.2-cp36-cp36m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:08bfcc21b5997a9be4fefa405341320d8e7f19b4d684fb9c0580255c5bd6d695"}, + {file = "pyzmq-25.0.2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:1a843d26a8da1b752c74bc019c7b20e6791ee813cd6877449e6a1415589d22ff"}, + {file = "pyzmq-25.0.2-cp36-cp36m-musllinux_1_1_aarch64.whl", hash = "sha256:b48616a09d7df9dbae2f45a0256eee7b794b903ddc6d8657a9948669b345f220"}, + {file = "pyzmq-25.0.2-cp36-cp36m-musllinux_1_1_i686.whl", hash = "sha256:d4427b4a136e3b7f85516c76dd2e0756c22eec4026afb76ca1397152b0ca8145"}, + {file = "pyzmq-25.0.2-cp36-cp36m-musllinux_1_1_x86_64.whl", hash = "sha256:26b0358e8933990502f4513c991c9935b6c06af01787a36d133b7c39b1df37fa"}, + {file = "pyzmq-25.0.2-cp36-cp36m-win32.whl", hash = "sha256:c8fedc3ccd62c6b77dfe6f43802057a803a411ee96f14e946f4a76ec4ed0e117"}, + {file = "pyzmq-25.0.2-cp36-cp36m-win_amd64.whl", hash = "sha256:2da6813b7995b6b1d1307329c73d3e3be2fd2d78e19acfc4eff2e27262732388"}, + {file = "pyzmq-25.0.2-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:a35960c8b2f63e4ef67fd6731851030df68e4b617a6715dd11b4b10312d19fef"}, + {file = "pyzmq-25.0.2-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:eef2a0b880ab40aca5a878933376cb6c1ec483fba72f7f34e015c0f675c90b20"}, + {file = "pyzmq-25.0.2-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:85762712b74c7bd18e340c3639d1bf2f23735a998d63f46bb6584d904b5e401d"}, + {file = "pyzmq-25.0.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:64812f29d6eee565e129ca14b0c785744bfff679a4727137484101b34602d1a7"}, + {file = "pyzmq-25.0.2-cp37-cp37m-musllinux_1_1_aarch64.whl", hash = "sha256:510d8e55b3a7cd13f8d3e9121edf0a8730b87d925d25298bace29a7e7bc82810"}, + {file = "pyzmq-25.0.2-cp37-cp37m-musllinux_1_1_i686.whl", hash = "sha256:b164cc3c8acb3d102e311f2eb6f3c305865ecb377e56adc015cb51f721f1dda6"}, + {file = "pyzmq-25.0.2-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:28fdb9224a258134784a9cf009b59265a9dde79582fb750d4e88a6bcbc6fa3dc"}, + {file = "pyzmq-25.0.2-cp37-cp37m-win32.whl", hash = "sha256:dd771a440effa1c36d3523bc6ba4e54ff5d2e54b4adcc1e060d8f3ca3721d228"}, + {file = "pyzmq-25.0.2-cp37-cp37m-win_amd64.whl", hash = "sha256:9bdc40efb679b9dcc39c06d25629e55581e4c4f7870a5e88db4f1c51ce25e20d"}, + {file = "pyzmq-25.0.2-cp38-cp38-macosx_10_15_universal2.whl", hash = "sha256:1f82906a2d8e4ee310f30487b165e7cc8ed09c009e4502da67178b03083c4ce0"}, + {file = "pyzmq-25.0.2-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:21ec0bf4831988af43c8d66ba3ccd81af2c5e793e1bf6790eb2d50e27b3c570a"}, + {file = "pyzmq-25.0.2-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:abbce982a17c88d2312ec2cf7673985d444f1beaac6e8189424e0a0e0448dbb3"}, + {file = "pyzmq-25.0.2-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:9e1d2f2d86fc75ed7f8845a992c5f6f1ab5db99747fb0d78b5e4046d041164d2"}, + {file = "pyzmq-25.0.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a2e92ff20ad5d13266bc999a29ed29a3b5b101c21fdf4b2cf420c09db9fb690e"}, + {file = "pyzmq-25.0.2-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:edbbf06cc2719889470a8d2bf5072bb00f423e12de0eb9ffec946c2c9748e149"}, + {file = "pyzmq-25.0.2-cp38-cp38-musllinux_1_1_i686.whl", hash = "sha256:77942243ff4d14d90c11b2afd8ee6c039b45a0be4e53fb6fa7f5e4fd0b59da39"}, + {file = "pyzmq-25.0.2-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:ab046e9cb902d1f62c9cc0eca055b1d11108bdc271caf7c2171487298f229b56"}, + {file = "pyzmq-25.0.2-cp38-cp38-win32.whl", hash = "sha256:ad761cfbe477236802a7ab2c080d268c95e784fe30cafa7e055aacd1ca877eb0"}, + {file = "pyzmq-25.0.2-cp38-cp38-win_amd64.whl", hash = "sha256:8560756318ec7c4c49d2c341012167e704b5a46d9034905853c3d1ade4f55bee"}, + {file = "pyzmq-25.0.2-cp39-cp39-macosx_10_15_universal2.whl", hash = "sha256:ab2c056ac503f25a63f6c8c6771373e2a711b98b304614151dfb552d3d6c81f6"}, + {file = "pyzmq-25.0.2-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:cca8524b61c0eaaa3505382dc9b9a3bc8165f1d6c010fdd1452c224225a26689"}, + {file = "pyzmq-25.0.2-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:cfb9f7eae02d3ac42fbedad30006b7407c984a0eb4189a1322241a20944d61e5"}, + {file = "pyzmq-25.0.2-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:5eaeae038c68748082137d6896d5c4db7927e9349237ded08ee1bbd94f7361c9"}, + {file = "pyzmq-25.0.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4a31992a8f8d51663ebf79df0df6a04ffb905063083d682d4380ab8d2c67257c"}, + {file = "pyzmq-25.0.2-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:6a979e59d2184a0c8f2ede4b0810cbdd86b64d99d9cc8a023929e40dce7c86cc"}, + {file = "pyzmq-25.0.2-cp39-cp39-musllinux_1_1_i686.whl", hash = "sha256:1f124cb73f1aa6654d31b183810febc8505fd0c597afa127c4f40076be4574e0"}, + {file = "pyzmq-25.0.2-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:65c19a63b4a83ae45d62178b70223adeee5f12f3032726b897431b6553aa25af"}, + {file = "pyzmq-25.0.2-cp39-cp39-win32.whl", hash = "sha256:83d822e8687621bed87404afc1c03d83fa2ce39733d54c2fd52d8829edb8a7ff"}, + {file = "pyzmq-25.0.2-cp39-cp39-win_amd64.whl", hash = "sha256:24683285cc6b7bf18ad37d75b9db0e0fefe58404e7001f1d82bf9e721806daa7"}, + {file = "pyzmq-25.0.2-pp37-pypy37_pp73-macosx_10_9_x86_64.whl", hash = "sha256:4a4b4261eb8f9ed71f63b9eb0198dd7c934aa3b3972dac586d0ef502ba9ab08b"}, + {file = "pyzmq-25.0.2-pp37-pypy37_pp73-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:62ec8d979f56c0053a92b2b6a10ff54b9ec8a4f187db2b6ec31ee3dd6d3ca6e2"}, + {file = "pyzmq-25.0.2-pp37-pypy37_pp73-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:affec1470351178e892121b3414c8ef7803269f207bf9bef85f9a6dd11cde264"}, + {file = "pyzmq-25.0.2-pp37-pypy37_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ffc71111433bd6ec8607a37b9211f4ef42e3d3b271c6d76c813669834764b248"}, + {file = "pyzmq-25.0.2-pp37-pypy37_pp73-win_amd64.whl", hash = "sha256:6fadc60970714d86eff27821f8fb01f8328dd36bebd496b0564a500fe4a9e354"}, + {file = "pyzmq-25.0.2-pp38-pypy38_pp73-macosx_10_9_x86_64.whl", hash = "sha256:269968f2a76c0513490aeb3ba0dc3c77b7c7a11daa894f9d1da88d4a0db09835"}, + {file = "pyzmq-25.0.2-pp38-pypy38_pp73-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:f7c8b8368e84381ae7c57f1f5283b029c888504aaf4949c32e6e6fb256ec9bf0"}, + {file = "pyzmq-25.0.2-pp38-pypy38_pp73-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:25e6873a70ad5aa31e4a7c41e5e8c709296edef4a92313e1cd5fc87bbd1874e2"}, + {file = "pyzmq-25.0.2-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b733076ff46e7db5504c5e7284f04a9852c63214c74688bdb6135808531755a3"}, + {file = "pyzmq-25.0.2-pp38-pypy38_pp73-win_amd64.whl", hash = "sha256:a6f6ae12478fdc26a6d5fdb21f806b08fa5403cd02fd312e4cb5f72df078f96f"}, + {file = "pyzmq-25.0.2-pp39-pypy39_pp73-macosx_10_9_x86_64.whl", hash = "sha256:67da1c213fbd208906ab3470cfff1ee0048838365135a9bddc7b40b11e6d6c89"}, + {file = "pyzmq-25.0.2-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:531e36d9fcd66f18de27434a25b51d137eb546931033f392e85674c7a7cea853"}, + {file = "pyzmq-25.0.2-pp39-pypy39_pp73-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:34a6fddd159ff38aa9497b2e342a559f142ab365576284bc8f77cb3ead1f79c5"}, + {file = "pyzmq-25.0.2-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b491998ef886662c1f3d49ea2198055a9a536ddf7430b051b21054f2a5831800"}, + {file = "pyzmq-25.0.2-pp39-pypy39_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:5d496815074e3e3d183fe2c7fcea2109ad67b74084c254481f87b64e04e9a471"}, + {file = "pyzmq-25.0.2-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:56a94ab1d12af982b55ca96c6853db6ac85505e820d9458ac76364c1998972f4"}, + {file = "pyzmq-25.0.2.tar.gz", hash = "sha256:6b8c1bbb70e868dc88801aa532cae6bd4e3b5233784692b786f17ad2962e5149"}, +] + +[package.dependencies] +cffi = {version = "*", markers = "implementation_name == \"pypy\""} + +[[package]] +name = "rasterio" +version = "1.3.6" +description = "Fast and direct raster I/O for use with Numpy and SciPy" +optional = false +python-versions = ">=3.8" +files = [ + {file = "rasterio-1.3.6-cp310-cp310-macosx_10_15_x86_64.whl", hash = "sha256:23a8d10ba17301029962a5667915381a8b4711ed80b712eb71cf68834cb5f946"}, + {file = "rasterio-1.3.6-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:76b6bd4b566cd733f0ddd05ba88bea3f96705ff74e2e5fab73ead2a26cbc5979"}, + {file = "rasterio-1.3.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:50785004d7adf66cf96c9c3498cf530ec91292e9349e66e8d1f1183085ee93b1"}, + {file = "rasterio-1.3.6-cp310-cp310-win_amd64.whl", hash = "sha256:9f3f901097c3f306f1143d6fdc503440596c66a2c39054e25604bdf3f4eaaff3"}, + {file = "rasterio-1.3.6-cp311-cp311-macosx_10_15_x86_64.whl", hash = "sha256:a732f8d314b7d9cb532b1969e968d08bf208886f04309662a5d16884af39bb4a"}, + {file = "rasterio-1.3.6-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:d03e2fcd8f3aafb0ea1fa27a021fecc385655630a46c70d6ba693675c6cc3830"}, + {file = "rasterio-1.3.6-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:69fdc712e9c79e82d00d783d23034bb16ca8faa18856e83e297bb7e4d7e3e277"}, + {file = "rasterio-1.3.6-cp311-cp311-win_amd64.whl", hash = "sha256:83f764c2b30e3d07bea5626392f1ce5481e61d5583256ab66f3a610a2f40dec7"}, + {file = "rasterio-1.3.6-cp38-cp38-macosx_10_15_x86_64.whl", hash = "sha256:1321372c653a36928b4e5e11cbe7f851903fb76608b8e48a860168b248d5f8e6"}, + {file = "rasterio-1.3.6-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:8a584fedd92953a0580e8de3f41ce9f33a3205ba79ea58fff8f90ba5d14a0c04"}, + {file = "rasterio-1.3.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:92f0f92254fcce57d25d5f60ef2cf649297f8a1e1fa279b32795bde20f11ff41"}, + {file = "rasterio-1.3.6-cp38-cp38-win_amd64.whl", hash = "sha256:e73339e8fb9b9091a4a0ffd9f84725b2d1f118cf51c35fb0d03b94e82e1736a3"}, + {file = "rasterio-1.3.6-cp39-cp39-macosx_10_15_x86_64.whl", hash = "sha256:eaaeb2e661d1ffc07a7ae4fd997bb326d3561f641178126102842d608a010cc3"}, + {file = "rasterio-1.3.6-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:0883a38bd32e6a3d8d85bac67e3b75a2f04f7de265803585516883223ddbb8d1"}, + {file = "rasterio-1.3.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b72fc032ddca55d73de87ef3872530b7384989378a1bc66d77c69cedafe7feaf"}, + {file = "rasterio-1.3.6-cp39-cp39-win_amd64.whl", hash = "sha256:cb3288add5d55248f5d48815f9d509819ba8985cd0302d2e8dd743f83c5ec96d"}, + {file = "rasterio-1.3.6.tar.gz", hash = "sha256:c8b90eb10e16102d1ab0334a7436185f295de1c07f0d197e206d1c005fc33905"}, +] + +[package.dependencies] +affine = "*" +attrs = "*" +boto3 = {version = ">=1.2.4", optional = true, markers = "extra == \"s3\""} +certifi = "*" +click = ">=4.0" +click-plugins = "*" +cligj = ">=0.5" +numpy = ">=1.18" +setuptools = "*" +snuggs = ">=1.4.1" + +[package.extras] +all = ["boto3 (>=1.2.4)", "ghp-import", "hypothesis", "ipython (>=2.0)", "matplotlib", "numpydoc", "packaging", "pytest (>=2.8.2)", "pytest-cov (>=2.2.0)", "shapely", "sphinx", "sphinx-rtd-theme"] +docs = ["ghp-import", "numpydoc", "sphinx", "sphinx-rtd-theme"] +ipython = ["ipython (>=2.0)"] +plot = ["matplotlib"] +s3 = ["boto3 (>=1.2.4)"] +test = ["boto3 (>=1.2.4)", "hypothesis", "packaging", "pytest (>=2.8.2)", "pytest-cov (>=2.2.0)", "shapely"] + +[[package]] +name = "s3transfer" +version = "0.6.0" +description = "An Amazon S3 Transfer Manager" +optional = false +python-versions = ">= 3.7" +files = [ + {file = "s3transfer-0.6.0-py3-none-any.whl", hash = "sha256:06176b74f3a15f61f1b4f25a1fc29a4429040b7647133a463da8fa5bd28d5ecd"}, + {file = "s3transfer-0.6.0.tar.gz", hash = "sha256:2ed07d3866f523cc561bf4a00fc5535827981b117dd7876f036b0c1aca42c947"}, +] + +[package.dependencies] +botocore = ">=1.12.36,<2.0a.0" + +[package.extras] +crt = ["botocore[crt] (>=1.20.29,<2.0a.0)"] + +[[package]] +name = "setuptools" +version = "67.7.2" +description = "Easily download, build, install, upgrade, and uninstall Python packages" +optional = false +python-versions = ">=3.7" +files = [ + {file = "setuptools-67.7.2-py3-none-any.whl", hash = "sha256:23aaf86b85ca52ceb801d32703f12d77517b2556af839621c641fca11287952b"}, + {file = "setuptools-67.7.2.tar.gz", hash = "sha256:f104fa03692a2602fa0fec6c6a9e63b6c8a968de13e17c026957dd1f53d80990"}, +] + +[package.extras] +docs = ["furo", "jaraco.packaging (>=9)", "jaraco.tidelift (>=1.4)", "pygments-github-lexers (==0.0.5)", "rst.linker (>=1.9)", "sphinx (>=3.5)", "sphinx-favicon", "sphinx-hoverxref (<2)", "sphinx-inline-tabs", "sphinx-lint", "sphinx-notfound-page (==0.8.3)", "sphinx-reredirects", "sphinxcontrib-towncrier"] +testing = ["build[virtualenv]", "filelock (>=3.4.0)", "flake8 (<5)", "flake8-2020", "ini2toml[lite] (>=0.9)", "jaraco.envs (>=2.2)", "jaraco.path (>=3.2.0)", "pip (>=19.1)", "pip-run (>=8.8)", "pytest (>=6)", "pytest-black (>=0.3.7)", "pytest-checkdocs (>=2.4)", "pytest-cov", "pytest-enabler (>=1.3)", "pytest-flake8", "pytest-mypy (>=0.9.1)", "pytest-perf", "pytest-timeout", "pytest-xdist", "tomli-w (>=1.0.0)", "virtualenv (>=13.0.0)", "wheel"] +testing-integration = ["build[virtualenv]", "filelock (>=3.4.0)", "jaraco.envs (>=2.2)", "jaraco.path (>=3.2.0)", "pytest", "pytest-enabler", "pytest-xdist", "tomli", "virtualenv (>=13.0.0)", "wheel"] + +[[package]] +name = "shapely" +version = "2.0.1" +description = "Manipulation and analysis of geometric objects" +optional = false +python-versions = ">=3.7" +files = [ + {file = "shapely-2.0.1-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:b06d031bc64149e340448fea25eee01360a58936c89985cf584134171e05863f"}, + {file = "shapely-2.0.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:9a6ac34c16f4d5d3c174c76c9d7614ec8fe735f8f82b6cc97a46b54f386a86bf"}, + {file = "shapely-2.0.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:865bc3d7cc0ea63189d11a0b1120d1307ed7a64720a8bfa5be2fde5fc6d0d33f"}, + {file = "shapely-2.0.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:45b4833235b90bc87ee26c6537438fa77559d994d2d3be5190dd2e54d31b2820"}, + {file = "shapely-2.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ce88ec79df55430e37178a191ad8df45cae90b0f6972d46d867bf6ebbb58cc4d"}, + {file = "shapely-2.0.1-cp310-cp310-win32.whl", hash = "sha256:01224899ff692a62929ef1a3f5fe389043e262698a708ab7569f43a99a48ae82"}, + {file = "shapely-2.0.1-cp310-cp310-win_amd64.whl", hash = "sha256:da71de5bf552d83dcc21b78cc0020e86f8d0feea43e202110973987ffa781c21"}, + {file = "shapely-2.0.1-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:502e0a607f1dcc6dee0125aeee886379be5242c854500ea5fd2e7ac076b9ce6d"}, + {file = "shapely-2.0.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:7d3bbeefd8a6a1a1017265d2d36f8ff2d79d0162d8c141aa0d37a87063525656"}, + {file = "shapely-2.0.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:f470a130d6ddb05b810fc1776d918659407f8d025b7f56d2742a596b6dffa6c7"}, + {file = "shapely-2.0.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4641325e065fd3e07d55677849c9ddfd0cf3ee98f96475126942e746d55b17c8"}, + {file = "shapely-2.0.1-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:90cfa4144ff189a3c3de62e2f3669283c98fb760cfa2e82ff70df40f11cadb39"}, + {file = "shapely-2.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:70a18fc7d6418e5aea76ac55dce33f98e75bd413c6eb39cfed6a1ba36469d7d4"}, + {file = "shapely-2.0.1-cp311-cp311-win32.whl", hash = "sha256:09d6c7763b1bee0d0a2b84bb32a4c25c6359ad1ac582a62d8b211e89de986154"}, + {file = "shapely-2.0.1-cp311-cp311-win_amd64.whl", hash = "sha256:d8f55f355be7821dade839df785a49dc9f16d1af363134d07eb11e9207e0b189"}, + {file = "shapely-2.0.1-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:83a8ec0ee0192b6e3feee9f6a499d1377e9c295af74d7f81ecba5a42a6b195b7"}, + {file = "shapely-2.0.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a529218e72a3dbdc83676198e610485fdfa31178f4be5b519a8ae12ea688db14"}, + {file = "shapely-2.0.1-cp37-cp37m-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:91575d97fd67391b85686573d758896ed2fc7476321c9d2e2b0c398b628b961c"}, + {file = "shapely-2.0.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c8b0d834b11be97d5ab2b4dceada20ae8e07bcccbc0f55d71df6729965f406ad"}, + {file = "shapely-2.0.1-cp37-cp37m-win32.whl", hash = "sha256:b4f0711cc83734c6fad94fc8d4ec30f3d52c1787b17d9dca261dc841d4731c64"}, + {file = "shapely-2.0.1-cp37-cp37m-win_amd64.whl", hash = "sha256:05c51a29336e604c084fb43ae5dbbfa2c0ef9bd6fedeae0a0d02c7b57a56ba46"}, + {file = "shapely-2.0.1-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:b519cf3726ddb6c67f6a951d1bb1d29691111eaa67ea19ddca4d454fbe35949c"}, + {file = "shapely-2.0.1-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:193a398d81c97a62fc3634a1a33798a58fd1dcf4aead254d080b273efbb7e3ff"}, + {file = "shapely-2.0.1-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:e55698e0ed95a70fe9ff9a23c763acfe0bf335b02df12142f74e4543095e9a9b"}, + {file = "shapely-2.0.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f32a748703e7bf6e92dfa3d2936b2fbfe76f8ce5f756e24f49ef72d17d26ad02"}, + {file = "shapely-2.0.1-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:1a34a23d6266ca162499e4a22b79159dc0052f4973d16f16f990baa4d29e58b6"}, + {file = "shapely-2.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d173d24e85e51510e658fb108513d5bc11e3fd2820db6b1bd0522266ddd11f51"}, + {file = "shapely-2.0.1-cp38-cp38-win32.whl", hash = "sha256:3cb256ae0c01b17f7bc68ee2ffdd45aebf42af8992484ea55c29a6151abe4386"}, + {file = "shapely-2.0.1-cp38-cp38-win_amd64.whl", hash = "sha256:c7eed1fb3008a8a4a56425334b7eb82651a51f9e9a9c2f72844a2fb394f38a6c"}, + {file = "shapely-2.0.1-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:ac1dfc397475d1de485e76de0c3c91cc9d79bd39012a84bb0f5e8a199fc17bef"}, + {file = "shapely-2.0.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:33403b8896e1d98aaa3a52110d828b18985d740cc9f34f198922018b1e0f8afe"}, + {file = "shapely-2.0.1-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:2569a4b91caeef54dd5ae9091ae6f63526d8ca0b376b5bb9fd1a3195d047d7d4"}, + {file = "shapely-2.0.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a70a614791ff65f5e283feed747e1cc3d9e6c6ba91556e640636bbb0a1e32a71"}, + {file = "shapely-2.0.1-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:c43755d2c46b75a7b74ac6226d2cc9fa2a76c3263c5ae70c195c6fb4e7b08e79"}, + {file = "shapely-2.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3ad81f292fffbd568ae71828e6c387da7eb5384a79db9b4fde14dd9fdeffca9a"}, + {file = "shapely-2.0.1-cp39-cp39-win32.whl", hash = "sha256:b50c401b64883e61556a90b89948297f1714dbac29243d17ed9284a47e6dd731"}, + {file = "shapely-2.0.1-cp39-cp39-win_amd64.whl", hash = "sha256:bca57b683e3d94d0919e2f31e4d70fdfbb7059650ef1b431d9f4e045690edcd5"}, + {file = "shapely-2.0.1.tar.gz", hash = "sha256:66a6b1a3e72ece97fc85536a281476f9b7794de2e646ca8a4517e2e3c1446893"}, +] + +[package.dependencies] +numpy = ">=1.14" + +[package.extras] +docs = ["matplotlib", "numpydoc (==1.1.*)", "sphinx", "sphinx-book-theme", "sphinx-remove-toctrees"] +test = ["pytest", "pytest-cov"] + +[[package]] +name = "six" +version = "1.16.0" +description = "Python 2 and 3 compatibility utilities" +optional = false +python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*" +files = [ + {file = "six-1.16.0-py2.py3-none-any.whl", hash = "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"}, + {file = "six-1.16.0.tar.gz", hash = "sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926"}, +] + +[[package]] +name = "snuggs" +version = "1.4.7" +description = "Snuggs are s-expressions for Numpy" +optional = false +python-versions = "*" +files = [ + {file = "snuggs-1.4.7-py3-none-any.whl", hash = "sha256:988dde5d4db88e9d71c99457404773dabcc7a1c45971bfbe81900999942d9f07"}, + {file = "snuggs-1.4.7.tar.gz", hash = "sha256:501cf113fe3892e14e2fee76da5cd0606b7e149c411c271898e6259ebde2617b"}, +] + +[package.dependencies] +numpy = "*" +pyparsing = ">=2.1.6" + +[package.extras] +test = ["hypothesis", "pytest"] + +[[package]] +name = "soupsieve" +version = "2.4.1" +description = "A modern CSS selector implementation for Beautiful Soup." +optional = false +python-versions = ">=3.7" +files = [ + {file = "soupsieve-2.4.1-py3-none-any.whl", hash = "sha256:1c1bfee6819544a3447586c889157365a27e10d88cde3ad3da0cf0ddf646feb8"}, + {file = "soupsieve-2.4.1.tar.gz", hash = "sha256:89d12b2d5dfcd2c9e8c22326da9d9aa9cb3dfab0a83a024f05704076ee8d35ea"}, +] + +[[package]] +name = "stack-data" +version = "0.6.2" +description = "Extract data from python stack frames and tracebacks for informative displays" +optional = false +python-versions = "*" +files = [ + {file = "stack_data-0.6.2-py3-none-any.whl", hash = "sha256:cbb2a53eb64e5785878201a97ed7c7b94883f48b87bfb0bbe8b623c74679e4a8"}, + {file = "stack_data-0.6.2.tar.gz", hash = "sha256:32d2dd0376772d01b6cb9fc996f3c8b57a357089dec328ed4b6553d037eaf815"}, +] + +[package.dependencies] +asttokens = ">=2.1.0" +executing = ">=1.2.0" +pure-eval = "*" + +[package.extras] +tests = ["cython", "littleutils", "pygments", "pytest", "typeguard"] + +[[package]] +name = "tinycss2" +version = "1.2.1" +description = "A tiny CSS parser" +optional = false +python-versions = ">=3.7" +files = [ + {file = "tinycss2-1.2.1-py3-none-any.whl", hash = "sha256:2b80a96d41e7c3914b8cda8bc7f705a4d9c49275616e886103dd839dfc847847"}, + {file = "tinycss2-1.2.1.tar.gz", hash = "sha256:8cff3a8f066c2ec677c06dbc7b45619804a6938478d9d73c284b29d14ecb0627"}, +] + +[package.dependencies] +webencodings = ">=0.4" + +[package.extras] +doc = ["sphinx", "sphinx_rtd_theme"] +test = ["flake8", "isort", "pytest"] + +[[package]] +name = "tomli" +version = "2.0.1" +description = "A lil' TOML parser" +optional = false +python-versions = ">=3.7" +files = [ + {file = "tomli-2.0.1-py3-none-any.whl", hash = "sha256:939de3e7a6161af0c887ef91b7d41a53e7c5a1ca976325f429cb46ea9bc30ecc"}, + {file = "tomli-2.0.1.tar.gz", hash = "sha256:de526c12914f0c550d15924c62d72abc48d6fe7364aa87328337a31007fe8a4f"}, +] + +[[package]] +name = "tornado" +version = "6.3.1" +description = "Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed." +optional = false +python-versions = ">= 3.8" +files = [ + {file = "tornado-6.3.1-cp38-abi3-macosx_10_9_universal2.whl", hash = "sha256:db181eb3df8738613ff0a26f49e1b394aade05034b01200a63e9662f347d4415"}, + {file = "tornado-6.3.1-cp38-abi3-macosx_10_9_x86_64.whl", hash = "sha256:b4e7b956f9b5e6f9feb643ea04f07e7c6b49301e03e0023eedb01fa8cf52f579"}, + {file = "tornado-6.3.1-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9661aa8bc0e9d83d757cd95b6f6d1ece8ca9fd1ccdd34db2de381e25bf818233"}, + {file = "tornado-6.3.1-cp38-abi3-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:81c17e0cc396908a5e25dc8e9c5e4936e6dfd544c9290be48bd054c79bcad51e"}, + {file = "tornado-6.3.1-cp38-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a27a1cfa9997923f80bdd962b3aab048ac486ad8cfb2f237964f8ab7f7eb824b"}, + {file = "tornado-6.3.1-cp38-abi3-musllinux_1_1_aarch64.whl", hash = "sha256:d7117f3c7ba5d05813b17a1f04efc8e108a1b811ccfddd9134cc68553c414864"}, + {file = "tornado-6.3.1-cp38-abi3-musllinux_1_1_i686.whl", hash = "sha256:ffdce65a281fd708da5a9def3bfb8f364766847fa7ed806821a69094c9629e8a"}, + {file = "tornado-6.3.1-cp38-abi3-musllinux_1_1_x86_64.whl", hash = "sha256:90f569a35a8ec19bde53aa596952071f445da678ec8596af763b9b9ce07605e6"}, + {file = "tornado-6.3.1-cp38-abi3-win32.whl", hash = "sha256:3455133b9ff262fd0a75630af0a8ee13564f25fb4fd3d9ce239b8a7d3d027bf8"}, + {file = "tornado-6.3.1-cp38-abi3-win_amd64.whl", hash = "sha256:1285f0691143f7ab97150831455d4db17a267b59649f7bd9700282cba3d5e771"}, + {file = "tornado-6.3.1.tar.gz", hash = "sha256:5e2f49ad371595957c50e42dd7e5c14d64a6843a3cf27352b69c706d1b5918af"}, +] + +[[package]] +name = "traitlets" +version = "5.9.0" +description = "Traitlets Python configuration system" +optional = false +python-versions = ">=3.7" +files = [ + {file = "traitlets-5.9.0-py3-none-any.whl", hash = "sha256:9e6ec080259b9a5940c797d58b613b5e31441c2257b87c2e795c5228ae80d2d8"}, + {file = "traitlets-5.9.0.tar.gz", hash = "sha256:f6cde21a9c68cf756af02035f72d5a723bf607e862e7be33ece505abf4a3bad9"}, +] + +[package.extras] +docs = ["myst-parser", "pydata-sphinx-theme", "sphinx"] +test = ["argcomplete (>=2.0)", "pre-commit", "pytest", "pytest-mock"] + +[[package]] +name = "typer" +version = "0.7.0" +description = "Typer, build great CLIs. Easy to code. Based on Python type hints." +optional = false +python-versions = ">=3.6" +files = [ + {file = "typer-0.7.0-py3-none-any.whl", hash = "sha256:b5e704f4e48ec263de1c0b3a2387cd405a13767d2f907f44c1a08cbad96f606d"}, + {file = "typer-0.7.0.tar.gz", hash = "sha256:ff797846578a9f2a201b53442aedeb543319466870fbe1c701eab66dd7681165"}, +] + +[package.dependencies] +click = ">=7.1.1,<9.0.0" + +[package.extras] +all = ["colorama (>=0.4.3,<0.5.0)", "rich (>=10.11.0,<13.0.0)", "shellingham (>=1.3.0,<2.0.0)"] +dev = ["autoflake (>=1.3.1,<2.0.0)", "flake8 (>=3.8.3,<4.0.0)", "pre-commit (>=2.17.0,<3.0.0)"] +doc = ["cairosvg (>=2.5.2,<3.0.0)", "mdx-include (>=1.4.1,<2.0.0)", "mkdocs (>=1.1.2,<2.0.0)", "mkdocs-material (>=8.1.4,<9.0.0)", "pillow (>=9.3.0,<10.0.0)"] +test = ["black (>=22.3.0,<23.0.0)", "coverage (>=6.2,<7.0)", "isort (>=5.0.6,<6.0.0)", "mypy (==0.910)", "pytest (>=4.4.0,<8.0.0)", "pytest-cov (>=2.10.0,<5.0.0)", "pytest-sugar (>=0.9.4,<0.10.0)", "pytest-xdist (>=1.32.0,<4.0.0)", "rich (>=10.11.0,<13.0.0)", "shellingham (>=1.3.0,<2.0.0)"] + +[[package]] +name = "typing-extensions" +version = "4.5.0" +description = "Backported and Experimental Type Hints for Python 3.7+" +optional = false +python-versions = ">=3.7" +files = [ + {file = "typing_extensions-4.5.0-py3-none-any.whl", hash = "sha256:fb33085c39dd998ac16d1431ebc293a8b3eedd00fd4a32de0ff79002c19511b4"}, + {file = "typing_extensions-4.5.0.tar.gz", hash = "sha256:5cb5f4a79139d699607b3ef622a1dedafa84e115ab0024e0d9c044a9479ca7cb"}, +] + +[[package]] +name = "urllib3" +version = "1.26.15" +description = "HTTP library with thread-safe connection pooling, file post, and more." +optional = false +python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*" +files = [ + {file = "urllib3-1.26.15-py2.py3-none-any.whl", hash = "sha256:aa751d169e23c7479ce47a0cb0da579e3ede798f994f5816a74e4f4500dcea42"}, + {file = "urllib3-1.26.15.tar.gz", hash = "sha256:8a388717b9476f934a21484e8c8e61875ab60644d29b9b39e11e4b9dc1c6b305"}, +] + +[package.extras] +brotli = ["brotli (>=1.0.9)", "brotlicffi (>=0.8.0)", "brotlipy (>=0.6.0)"] +secure = ["certifi", "cryptography (>=1.3.4)", "idna (>=2.0.0)", "ipaddress", "pyOpenSSL (>=0.14)", "urllib3-secure-extra"] +socks = ["PySocks (>=1.5.6,!=1.5.7,<2.0)"] + +[[package]] +name = "virtualenv" +version = "20.22.0" +description = "Virtual Python Environment builder" +optional = false +python-versions = ">=3.7" +files = [ + {file = "virtualenv-20.22.0-py3-none-any.whl", hash = "sha256:48fd3b907b5149c5aab7c23d9790bea4cac6bc6b150af8635febc4cfeab1275a"}, + {file = "virtualenv-20.22.0.tar.gz", hash = "sha256:278753c47aaef1a0f14e6db8a4c5e1e040e90aea654d0fc1dc7e0d8a42616cc3"}, +] + +[package.dependencies] +distlib = ">=0.3.6,<1" +filelock = ">=3.11,<4" +platformdirs = ">=3.2,<4" + +[package.extras] +docs = ["furo (>=2023.3.27)", "proselint (>=0.13)", "sphinx (>=6.1.3)", "sphinx-argparse (>=0.4)", "sphinxcontrib-towncrier (>=0.2.1a0)", "towncrier (>=22.12)"] +test = ["covdefaults (>=2.3)", "coverage (>=7.2.3)", "coverage-enable-subprocess (>=1)", "flaky (>=3.7)", "packaging (>=23.1)", "pytest (>=7.3.1)", "pytest-env (>=0.8.1)", "pytest-freezegun (>=0.4.2)", "pytest-mock (>=3.10)", "pytest-randomly (>=3.12)", "pytest-timeout (>=2.1)"] + +[[package]] +name = "wcwidth" +version = "0.2.6" +description = "Measures the displayed width of unicode strings in a terminal" +optional = false +python-versions = "*" +files = [ + {file = "wcwidth-0.2.6-py2.py3-none-any.whl", hash = "sha256:795b138f6875577cd91bba52baf9e445cd5118fd32723b460e30a0af30ea230e"}, + {file = "wcwidth-0.2.6.tar.gz", hash = "sha256:a5220780a404dbe3353789870978e472cfe477761f06ee55077256e509b156d0"}, +] + +[[package]] +name = "webencodings" +version = "0.5.1" +description = "Character encoding aliases for legacy web content" +optional = false +python-versions = "*" +files = [ + {file = "webencodings-0.5.1-py2.py3-none-any.whl", hash = "sha256:a0af1213f3c2226497a97e2b3aa01a7e4bee4f403f95be16fc9acd2947514a78"}, + {file = "webencodings-0.5.1.tar.gz", hash = "sha256:b36a1c245f2d304965eb4e0a82848379241dc04b865afcc4aab16748587e1923"}, +] + +[[package]] +name = "wheel" +version = "0.38.4" +description = "A built-package format for Python" +optional = false +python-versions = ">=3.7" +files = [ + {file = "wheel-0.38.4-py3-none-any.whl", hash = "sha256:b60533f3f5d530e971d6737ca6d58681ee434818fab630c83a734bb10c083ce8"}, + {file = "wheel-0.38.4.tar.gz", hash = "sha256:965f5259b566725405b05e7cf774052044b1ed30119b5d586b2703aafe8719ac"}, +] + +[package.extras] +test = ["pytest (>=3.0.0)"] + +[[package]] +name = "zipp" +version = "3.15.0" +description = "Backport of pathlib-compatible object wrapper for zip files" +optional = false +python-versions = ">=3.7" +files = [ + {file = "zipp-3.15.0-py3-none-any.whl", hash = "sha256:48904fc76a60e542af151aded95726c1a5c34ed43ab4134b597665c86d7ad556"}, + {file = "zipp-3.15.0.tar.gz", hash = "sha256:112929ad649da941c23de50f356a2b5570c954b65150642bccdd66bf194d224b"}, +] + +[package.extras] +docs = ["furo", "jaraco.packaging (>=9)", "jaraco.tidelift (>=1.4)", "rst.linker (>=1.9)", "sphinx (>=3.5)", "sphinx-lint"] +testing = ["big-O", "flake8 (<5)", "jaraco.functools", "jaraco.itertools", "more-itertools", "pytest (>=6)", "pytest-black (>=0.3.7)", "pytest-checkdocs (>=2.4)", "pytest-cov", "pytest-enabler (>=1.3)", "pytest-flake8", "pytest-mypy (>=0.9.1)"] + +[metadata] +lock-version = "2.0" +python-versions = ">=3.8,<4" +content-hash = "023d394ad6160c28f61014d2848f046476e893cef9e48f1c971c14792c71235b" diff --git a/project/BenchmarkPlugin.scala b/project/BenchmarkPlugin.scala index e4f1d1acc..1d2e365c3 100644 --- a/project/BenchmarkPlugin.scala +++ b/project/BenchmarkPlugin.scala @@ -91,7 +91,7 @@ object BenchmarkPlugin extends AutoPlugin { val args = s" $t $f $i $wi $tu -rf $rf -rff $rff $extra $pat" state.value.log.debug("Starting: jmh:run " + args) - (run in Jmh).toTask(args) + (Jmh / run).toTask(args) } val benchFilesParser: Def.Initialize[State => Parser[File]] = Def.setting { state: State => @@ -101,12 +101,12 @@ object BenchmarkPlugin extends AutoPlugin { ) val dirs = Seq( - extracted.get(scalaSource in Compile), - extracted.get(scalaSource in Test) + extracted.get(Compile / scalaSource), + extracted.get(Test / scalaSource) ) def benchFileParser(dir: File) = fileParser(dir) - .filter(f ⇒ pat.accept(f.name), s ⇒ "Not a benchmark file: " + s) + .filter(f => pat.accept(f.name), s => "Not a benchmark file: " + s) val parsers = dirs.map(benchFileParser) diff --git a/project/PythonBuildPlugin.scala b/project/PythonBuildPlugin.scala index 37404ddae..d5cecd123 100644 --- a/project/PythonBuildPlugin.scala +++ b/project/PythonBuildPlugin.scala @@ -37,6 +37,34 @@ object PythonBuildPlugin extends AutoPlugin { val pythonCommand = settingKey[String]("Python command. Defaults to 'python'") val pySetup = inputKey[Int]("Run 'python setup.py
- -## Related Links - -* [Gitter Channel](https://gitter.im/locationtech/rasterframes) -* [Scala API Documentation](latest/api/index.html) -* [GitHub Repository](https://github.com/locationtech/rasterframes) -* [Astraea, Inc.](http://www.astraea.earth/), the company behind RasterFrames - -## Detailed Contents - -@@ toc { depth=4 } - -@@@ index -* [Overview](description.md) -* [Getting Started](getting-started.md) -* [Concepts](concepts.md) -* [Raster Data I/O](raster-io.md) -* [Vector Data](vector-data.md) -* [Raster Processing](raster-processing.md) -* [Numpy and Pandas](numpy-pandas.md) -* [Scala and SQL](languages.md) -* [Function Reference](reference.md) -* [Release Notes](release-notes.md) -@@@ - diff --git a/pyrasterframes/src/main/python/docs/raster-write.pymd b/pyrasterframes/src/main/python/docs/raster-write.pymd deleted file mode 100644 index 6328b2c31..000000000 --- a/pyrasterframes/src/main/python/docs/raster-write.pymd +++ /dev/null @@ -1,104 +0,0 @@ -# Writing Raster Data - -RasterFrames is oriented toward large scale analyses of spatial data. The primary output of these analyses could be a @ref:[statistical summary](aggregation.md), a @ref:[machine learning model](machine-learning.md), or some other result that is generally much smaller than the input dataset. - -However, there are times in any analysis where writing a representative sample of the work in progress provides valuable feedback on the current state of the process and results. - -```python imports, echo=False -import pyrasterframes -from pyrasterframes.utils import create_rf_spark_session -from pyrasterframes.rasterfunctions import * -from IPython.display import display -import os.path - -spark = create_rf_spark_session() -``` - -## Tile Samples - -We have some convenience methods to quickly visualize _tile_s (see discussion of the RasterFrame @ref:[schema](raster-read.md#single-raster) for orientation to the concept) when inspecting a subset of the data in a Notebook. - -In an IPython or Jupyter interpreter, a `Tile` object will be displayed as an image with limited metadata. - -```python tile_sample -def scene(band): - b = str(band).zfill(2) # converts int 2 to '02' - return 'https://modis-pds.s3.amazonaws.com/MCD43A4.006/11/08/2019059/' \ - 'MCD43A4.A2019059.h11v08.006.2019072203257_B{}.TIF'.format(b) -spark_df = spark.read.raster(scene(2), tile_dimensions=(128, 128)) -tile = spark_df.select(rf_tile('proj_raster').alias('tile')).first()['tile'] -tile -``` - -```python display_tile, echo=False, output=True -display(tile) # IPython.display function -``` - -## DataFrame Samples - -Within an IPython or Jupyter interpreter, a Spark and Pandas DataFrames containing a column of _tiles_ will be rendered as the samples discussed above. Simply import the `rf_ipython` submodule to enable enhanced HTML rendering of these DataFrame types. - -```python to_samples, evaluate=True -import pyrasterframes.rf_ipython - -samples = spark_df \ - .select( - rf_extent('proj_raster').alias('extent'), - rf_tile('proj_raster').alias('tile'), - )\ - .select('extent.*', 'tile') \ - .limit(3) -samples -``` - - -## GeoTIFFs - -GeoTIFF is one of the most common file formats for spatial data, providing flexibility in data encoding, representation, and storage. RasterFrames provides a specialized Spark DataFrame writer for rendering a RasterFrame to a GeoTIFF. - -One downside to GeoTIFF is that it is not a big data native format. To create a GeoTIFF, all the data to be encoded has to be in the memory of one computer (in Spark parlance, this is a "collect"), limiting it's maximum size substantially compared to that of a full cluster environment. When rendering GeoTIFFs in RasterFrames, you must either specify the dimensions of the output raster, or deliberately limit the size of the collected data. - -Fortunately, we can use the cluster computing capability to downsample the data into a more manageable size. For sake of example, let's render an overview of a scene's red band as a small raster, reprojecting it to latitude and longitude coordinates on the [WGS84](https://en.wikipedia.org/wiki/World_Geodetic_System) reference ellipsoid (aka [EPSG:4326](https://spatialreference.org/ref/epsg/4326/)). - -```python write_geotiff -outfile = os.path.join('/tmp', 'geotiff-overview.tif') -spark_df.write.geotiff(outfile, crs='EPSG:4326', raster_dimensions=(256, 256)) -``` - -We can view the written file with `rasterio`: - -```python view_geotiff -import rasterio -from rasterio.plot import show, show_hist - -with rasterio.open(outfile) as src: - # View raster - show(src, adjust='linear') - # View data distribution - show_hist(src, bins=50, lw=0.0, stacked=False, alpha=0.6, - histtype='stepfilled', title="Overview Histogram") -``` - -If there are many _tile_ or projected raster columns in the DataFrame, the GeoTIFF writer will write each one as a separate band in the file. Each band in the output will be tagged the input column names for reference. - -```python, echo=False -os.remove(outfile) -``` - -## GeoTrellis Layers - -[GeoTrellis][GeoTrellis] is one of the key libraries upon which RasterFrames is built. It provides a Scala language API for working with geospatial raster data. GeoTrellis defines a [tile layer storage](https://geotrellis.readthedocs.io/en/latest/guide/tile-backends.html) format for persisting imagery mosaics. RasterFrames can write data from a `RasterFrameLayer` into a [GeoTrellis Layer](https://geotrellis.readthedocs.io/en/latest/guide/tile-backends.html). RasterFrames provides a `geotrellis` DataSource that supports both @ref:[reading](raster-read.md#geotrellis-layers) and @ref:[writing](raster-write.md#geotrellis-layers) GeoTrellis layers. - -> An example is forthcoming. In the mean time referencing the [`GeoTrellisDataSourceSpec` test code](https://github.com/locationtech/rasterframes/blob/develop/datasource/src/test/scala/org/locationtech/rasterframes/datasource/geotrellis/GeoTrellisDataSourceSpec.scala) may help. - -## Parquet - -You can write a RasterFrame to the [Apache Parquet][Parquet] format. This format is designed to efficiently persist and query columnar data in distributed file system, such as HDFS. It also provides benefits when working in single node (or "local") mode, such as tailoring organization for defined query patterns. - -```python write_parquet, evaluate=False -spark_df.withColumn('exp', rf_expm1('proj_raster')) \ - .write.mode('append').parquet('hdfs:///rf-user/sample.pq') -``` - -[GeoTrellis]: https://geotrellis.readthedocs.io/en/latest/ -[Parquet]: https://spark.apache.org/docs/latest/sql-data-sources-parquet.html diff --git a/pyrasterframes/src/main/python/pyrasterframes/rasterfunctions.py b/pyrasterframes/src/main/python/pyrasterframes/rasterfunctions.py deleted file mode 100644 index 9c0e52f09..000000000 --- a/pyrasterframes/src/main/python/pyrasterframes/rasterfunctions.py +++ /dev/null @@ -1,932 +0,0 @@ -# -# This software is licensed under the Apache 2 license, quoted below. -# -# Copyright 2019 Astraea, Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); you may not -# use this file except in compliance with the License. You may obtain a copy of -# the License at -# -# [http://www.apache.org/licenses/LICENSE-2.0] -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT -# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the -# License for the specific language governing permissions and limitations under -# the License. -# -# SPDX-License-Identifier: Apache-2.0 -# - -""" -This module creates explicit Python functions that map back to the existing Scala -implementations. Most functions are standard Column functions, but those with unique -signatures are handled here as well. -""" -from __future__ import absolute_import -from pyspark.sql.column import Column, _to_java_column -from .rf_context import RFContext -from .rf_types import CellType - -THIS_MODULE = 'pyrasterframes' - - -def _context_call(name, *args): - f = RFContext.active().lookup(name) - return f(*args) - - -def _parse_cell_type(cell_type_arg): - """ Convert the cell type representation to the expected JVM CellType object.""" - - def to_jvm(ct): - return _context_call('_parse_cell_type', ct) - - if isinstance(cell_type_arg, str): - return to_jvm(cell_type_arg) - elif isinstance(cell_type_arg, CellType): - return to_jvm(cell_type_arg.cell_type_name) - - -def rf_cell_types(): - """Return a list of standard cell types""" - return [CellType(str(ct)) for ct in _context_call('rf_cell_types')] - - -def rf_assemble_tile(col_index, row_index, cell_data_col, num_cols, num_rows, cell_type=None): - """Create a Tile from a column of cell data with location indices""" - jfcn = RFContext.active().lookup('rf_assemble_tile') - - if isinstance(num_cols, Column): - num_cols = _to_java_column(num_cols) - - if isinstance(num_rows, Column): - num_rows = _to_java_column(num_rows) - - if cell_type is None: - return Column(jfcn( - _to_java_column(col_index), _to_java_column(row_index), _to_java_column(cell_data_col), - num_cols, num_rows - )) - - else: - return Column(jfcn( - _to_java_column(col_index), _to_java_column(row_index), _to_java_column(cell_data_col), - num_cols, num_rows, _parse_cell_type(cell_type) - )) - -def rf_array_to_tile(array_col, num_cols, num_rows): - """Convert array in `array_col` into a Tile of dimensions `num_cols` and `num_rows'""" - jfcn = RFContext.active().lookup('rf_array_to_tile') - return Column(jfcn(_to_java_column(array_col), num_cols, num_rows)) - - -def rf_convert_cell_type(tile_col, cell_type): - """Convert the numeric type of the Tiles in `tileCol`""" - jfcn = RFContext.active().lookup('rf_convert_cell_type') - return Column(jfcn(_to_java_column(tile_col), _parse_cell_type(cell_type))) - -def rf_interpret_cell_type_as(tile_col, cell_type): - """Change the interpretation of the tile_col's cell values according to specified cell_type""" - jfcn = RFContext.active().lookup('rf_interpret_cell_type_as') - return Column(jfcn(_to_java_column(tile_col), _parse_cell_type(cell_type))) - - -def rf_make_constant_tile(scalar_value, num_cols, num_rows, cell_type=CellType.float64()): - """Constructor for constant tile column""" - jfcn = RFContext.active().lookup('rf_make_constant_tile') - return Column(jfcn(scalar_value, num_cols, num_rows, _parse_cell_type(cell_type))) - - -def rf_make_zeros_tile(num_cols, num_rows, cell_type=CellType.float64()): - """Create column of constant tiles of zero""" - jfcn = RFContext.active().lookup('rf_make_zeros_tile') - return Column(jfcn(num_cols, num_rows, _parse_cell_type(cell_type))) - - -def rf_make_ones_tile(num_cols, num_rows, cell_type=CellType.float64()): - """Create column of constant tiles of one""" - jfcn = RFContext.active().lookup('rf_make_ones_tile') - return Column(jfcn(num_cols, num_rows, _parse_cell_type(cell_type))) - - -def rf_rasterize(geometry_col, bounds_col, value_col, num_cols_col, num_rows_col): - """Create a tile where cells in the grid defined by cols, rows, and bounds are filled with the given value.""" - jfcn = RFContext.active().lookup('rf_rasterize') - return Column(jfcn(_to_java_column(geometry_col), _to_java_column(bounds_col), - _to_java_column(value_col), _to_java_column(num_cols_col), _to_java_column(num_rows_col))) - - -def st_reproject(geometry_col, src_crs, dst_crs): - """Reproject a column of geometry given the CRSs of the source and destination.""" - jfcn = RFContext.active().lookup('st_reproject') - return Column(jfcn(_to_java_column(geometry_col), _to_java_column(src_crs), _to_java_column(dst_crs))) - - -def rf_explode_tiles(*tile_cols): - """Create a row for each cell in Tile.""" - jfcn = RFContext.active().lookup('rf_explode_tiles') - jcols = [_to_java_column(arg) for arg in tile_cols] - return Column(jfcn(RFContext.active().list_to_seq(jcols))) - - -def rf_explode_tiles_sample(sample_frac, seed, *tile_cols): - """Create a row for a sample of cells in Tile columns.""" - jfcn = RFContext.active().lookup('rf_explode_tiles_sample') - jcols = [_to_java_column(arg) for arg in tile_cols] - return Column(jfcn(sample_frac, seed, RFContext.active().list_to_seq(jcols))) - - -def rf_mask_by_value(data_tile, mask_tile, mask_value): - """Generate a tile with the values from the data tile, but where cells in the masking tile contain the masking - value, replace the data value with NODATA. """ - jfcn = RFContext.active().lookup('rf_mask_by_value') - return Column(jfcn(_to_java_column(data_tile), _to_java_column(mask_tile), _to_java_column(mask_value))) - - -def rf_inverse_mask_by_value(data_tile, mask_tile, mask_value): - """Generate a tile with the values from the data tile, but where cells in the masking tile do not contain the - masking value, replace the data value with NODATA. """ - jfcn = RFContext.active().lookup('rf_inverse_mask_by_value') - return Column(jfcn(_to_java_column(data_tile), _to_java_column(mask_tile), _to_java_column(mask_value))) - - -def _apply_scalar_to_tile(name, tile_col, scalar): - jfcn = RFContext.active().lookup(name) - return Column(jfcn(_to_java_column(tile_col), scalar)) - - -def rf_with_no_data(tile_col, scalar): - """Assign a `NoData` value to the Tiles in the given Column.""" - return _apply_scalar_to_tile('rf_with_no_data', tile_col, scalar) - - -def rf_local_add_double(tile_col, scalar): - """Add a floating point scalar to a Tile""" - return _apply_scalar_to_tile('rf_local_add_double', tile_col, scalar) - - -def rf_local_add_int(tile_col, scalar): - """Add an integral scalar to a Tile""" - return _apply_scalar_to_tile('rf_local_add_int', tile_col, scalar) - - -def rf_local_subtract_double(tile_col, scalar): - """Subtract a floating point scalar from a Tile""" - return _apply_scalar_to_tile('rf_local_subtract_double', tile_col, scalar) - - -def rf_local_subtract_int(tile_col, scalar): - """Subtract an integral scalar from a Tile""" - return _apply_scalar_to_tile('rf_local_subtract_int', tile_col, scalar) - - -def rf_local_multiply_double(tile_col, scalar): - """Multiply a Tile by a float point scalar""" - return _apply_scalar_to_tile('rf_local_multiply_double', tile_col, scalar) - - -def rf_local_multiply_int(tile_col, scalar): - """Multiply a Tile by an integral scalar""" - return _apply_scalar_to_tile('rf_local_multiply_int', tile_col, scalar) - - -def rf_local_divide_double(tile_col, scalar): - """Divide a Tile by a floating point scalar""" - return _apply_scalar_to_tile('rf_local_divide_double', tile_col, scalar) - - -def rf_local_divide_int(tile_col, scalar): - """Divide a Tile by an integral scalar""" - return _apply_scalar_to_tile('rf_local_divide_int', tile_col, scalar) - - -def rf_local_less_double(tile_col, scalar): - """Return a Tile with values equal 1 if the cell is less than a scalar, otherwise 0""" - return _apply_scalar_to_tile('foo', tile_col, scalar) - - -def rf_local_less_int(tile_col, scalar): - """Return a Tile with values equal 1 if the cell is less than a scalar, otherwise 0""" - return _apply_scalar_to_tile('rf_local_less_double', tile_col, scalar) - - -def rf_local_less_equal_double(tile_col, scalar): - """Return a Tile with values equal 1 if the cell is less than or equal to a scalar, otherwise 0""" - return _apply_scalar_to_tile('rf_local_less_equal_double', tile_col, scalar) - - -def rf_local_less_equal_int(tile_col, scalar): - """Return a Tile with values equal 1 if the cell is less than or equal to a scalar, otherwise 0""" - return _apply_scalar_to_tile('rf_local_less_equal_int', tile_col, scalar) - - -def rf_local_greater_double(tile_col, scalar): - """Return a Tile with values equal 1 if the cell is greater than a scalar, otherwise 0""" - return _apply_scalar_to_tile('rf_local_greater_double', tile_col, scalar) - - -def rf_local_greater_int(tile_col, scalar): - """Return a Tile with values equal 1 if the cell is greater than a scalar, otherwise 0""" - return _apply_scalar_to_tile('rf_local_greater_int', tile_col, scalar) - - -def rf_local_greater_equal_double(tile_col, scalar): - """Return a Tile with values equal 1 if the cell is greater than or equal to a scalar, otherwise 0""" - return _apply_scalar_to_tile('rf_local_greater_equal_double', tile_col, scalar) - - -def rf_local_greater_equal_int(tile_col, scalar): - """Return a Tile with values equal 1 if the cell is greater than or equal to a scalar, otherwise 0""" - return _apply_scalar_to_tile('rf_local_greater_equal_int', tile_col, scalar) - - -def rf_local_equal_double(tile_col, scalar): - """Return a Tile with values equal 1 if the cell is equal to a scalar, otherwise 0""" - return _apply_scalar_to_tile('rf_local_equal_double', tile_col, scalar) - - -def rf_local_equal_int(tile_col, scalar): - """Return a Tile with values equal 1 if the cell is equal to a scalar, otherwise 0""" - return _apply_scalar_to_tile('rf_local_equal_int', tile_col, scalar) - - -def rf_local_unequal_double(tile_col, scalar): - """Return a Tile with values equal 1 if the cell is not equal to a scalar, otherwise 0""" - return _apply_scalar_to_tile('rf_local_unequal_double', tile_col, scalar) - - -def rf_local_unequal_int(tile_col, scalar): - """Return a Tile with values equal 1 if the cell is not equal to a scalar, otherwise 0""" - return _apply_scalar_to_tile('rf_local_unequal_int', tile_col, scalar) - -def rf_local_no_data(tile_col): - """Return a tile with ones where the input is NoData, otherwise zero.""" - return _apply_column_function('rf_local_no_data', tile_col) - -def rf_local_data(tile_col): - """Return a tile with zeros where the input is NoData, otherwise one.""" - return _apply_column_function('rf_local_data', tile_col) - -def _apply_column_function(name, *args): - jfcn = RFContext.active().lookup(name) - jcols = [_to_java_column(arg) for arg in args] - return Column(jfcn(*jcols)) - - -def rf_dimensions(tile_col): - """Query the number of (cols, rows) in a Tile.""" - return _apply_column_function('rf_dimensions', tile_col) - - -def rf_tile_to_array_int(tile_col): - """Flattens Tile into an array of integers.""" - return _apply_column_function('rf_tile_to_array_int', tile_col) - - -def rf_tile_to_array_double(tile_col): - """Flattens Tile into an array of doubles.""" - return _apply_column_function('rf_tile_to_array_double', tile_col) - - -def rf_cell_type(tile_col): - """Extract the Tile's cell type""" - return _apply_column_function('rf_cell_type', tile_col) - - -def rf_is_no_data_tile(tile_col): - """Report if the Tile is entirely NODDATA cells""" - return _apply_column_function('rf_is_no_data_tile', tile_col) - - -def rf_exists(tile_col): - """Returns true if any cells in the tile are true (non-zero and not NoData)""" - return _apply_column_function('rf_exists', tile_col) - - -def rf_for_all(tile_col): - """Returns true if all cells in the tile are true (non-zero and not NoData).""" - return _apply_column_function('rf_for_all', tile_col) - - -def rf_agg_approx_histogram(tile_col): - """Compute the full column aggregate floating point histogram""" - return _apply_column_function('rf_agg_approx_histogram', tile_col) - - -def rf_agg_stats(tile_col): - """Compute the full column aggregate floating point statistics""" - return _apply_column_function('rf_agg_stats', tile_col) - - -def rf_agg_mean(tile_col): - """Computes the column aggregate mean""" - return _apply_column_function('rf_agg_mean', tile_col) - - -def rf_agg_data_cells(tile_col): - """Computes the number of non-NoData cells in a column""" - return _apply_column_function('rf_agg_data_cells', tile_col) - - -def rf_agg_no_data_cells(tile_col): - """Computes the number of NoData cells in a column""" - return _apply_column_function('rf_agg_no_data_cells', tile_col) - - -def rf_tile_histogram(tile_col): - """Compute the Tile-wise histogram""" - return _apply_column_function('rf_tile_histogram', tile_col) - - -def rf_tile_mean(tile_col): - """Compute the Tile-wise mean""" - return _apply_column_function('rf_tile_mean', tile_col) - - -def rf_tile_sum(tile_col): - """Compute the Tile-wise sum""" - return _apply_column_function('rf_tile_sum', tile_col) - - -def rf_tile_min(tile_col): - """Compute the Tile-wise minimum""" - return _apply_column_function('rf_tile_min', tile_col) - - -def rf_tile_max(tile_col): - """Compute the Tile-wise maximum""" - return _apply_column_function('rf_tile_max', tile_col) - - -def rf_tile_stats(tile_col): - """Compute the Tile-wise floating point statistics""" - return _apply_column_function('rf_tile_stats', tile_col) - - -def rf_render_ascii(tile_col): - """Render ASCII art of tile""" - return _apply_column_function('rf_render_ascii', tile_col) - - -def rf_render_matrix(tile_col): - """Render Tile cell values as numeric values, for debugging purposes""" - return _apply_column_function('rf_render_matrix', tile_col) - - -def rf_render_png(red_tile_col, green_tile_col, blue_tile_col): - """Converts columns of tiles representing RGB channels into a PNG encoded byte array.""" - return _apply_column_function('rf_render_png', red_tile_col, green_tile_col, blue_tile_col) - - -def rf_rgb_composite(red_tile_col, green_tile_col, blue_tile_col): - """Converts columns of tiles representing RGB channels into a single RGB packaged tile.""" - return _apply_column_function('rf_rgb_composite', red_tile_col, green_tile_col, blue_tile_col) - - -def rf_no_data_cells(tile_col): - """Count of NODATA cells""" - return _apply_column_function('rf_no_data_cells', tile_col) - - -def rf_data_cells(tile_col): - """Count of cells with valid data""" - return _apply_column_function('rf_data_cells', tile_col) - - -def rf_local_add(left_tile_col, right_tile_col): - """Add two Tiles""" - return _apply_column_function('rf_local_add', left_tile_col, right_tile_col) - - -def rf_local_subtract(left_tile_col, right_tile_col): - """Subtract two Tiles""" - return _apply_column_function('rf_local_subtract', left_tile_col, right_tile_col) - - -def rf_local_multiply(left_tile_col, right_tile_col): - """Multiply two Tiles""" - return _apply_column_function('rf_local_multiply', left_tile_col, right_tile_col) - - -def rf_local_divide(left_tile_col, right_tile_col): - """Divide two Tiles""" - return _apply_column_function('rf_local_divide', left_tile_col, right_tile_col) - - -def rf_normalized_difference(left_tile_col, right_tile_col): - """Compute the normalized difference of two tiles""" - return _apply_column_function('rf_normalized_difference', left_tile_col, right_tile_col) - - -def rf_agg_local_max(tile_col): - """Compute the cell-wise/local max operation between Tiles in a column.""" - return _apply_column_function('rf_agg_local_max', tile_col) - - -def rf_agg_local_min(tile_col): - """Compute the cellwise/local min operation between Tiles in a column.""" - return _apply_column_function('rf_agg_local_min', tile_col) - - -def rf_agg_local_mean(tile_col): - """Compute the cellwise/local mean operation between Tiles in a column.""" - return _apply_column_function('rf_agg_local_mean', tile_col) - - -def rf_agg_local_data_cells(tile_col): - """Compute the cellwise/local count of non-NoData cells for all Tiles in a column.""" - return _apply_column_function('rf_agg_local_data_cells', tile_col) - - -def rf_agg_local_no_data_cells(tile_col): - """Compute the cellwise/local count of NoData cells for all Tiles in a column.""" - return _apply_column_function('rf_agg_local_no_data_cells', tile_col) - - -def rf_agg_local_stats(tile_col): - """Compute cell-local aggregate descriptive statistics for a column of Tiles.""" - return _apply_column_function('rf_agg_local_stats', tile_col) - - -def rf_mask(src_tile_col, mask_tile_col): - """Where the rf_mask (second) tile contains NODATA, replace values in the source (first) tile with NODATA.""" - return _apply_column_function('rf_mask', src_tile_col, mask_tile_col) - - -def rf_inverse_mask(src_tile_col, mask_tile_col): - """Where the rf_mask (second) tile DOES NOT contain NODATA, replace values in the source (first) tile with NODATA.""" - return _apply_column_function('rf_inverse_mask', src_tile_col, mask_tile_col) - - -def rf_local_less(left_tile_col, right_tile_col): - """Cellwise less than comparison between two tiles""" - return _apply_column_function('rf_local_less', left_tile_col, right_tile_col) - - -def rf_local_less_equal(left_tile_col, right_tile_col): - """Cellwise less than or equal to comparison between two tiles""" - return _apply_column_function('rf_local_less_equal', left_tile_col, right_tile_col) - - -def rf_local_greater(left_tile_col, right_tile_col): - """Cellwise greater than comparison between two tiles""" - return _apply_column_function('rf_local_greater', left_tile_col, right_tile_col) - - -def rf_local_greater_equal(left_tile_col, right_tile_col): - """Cellwise greater than or equal to comparison between two tiles""" - return _apply_column_function('rf_local_greater_equal', left_tile_col, right_tile_col) - - -def rf_local_equal(left_tile_col, right_tile_col): - """Cellwise equality comparison between two tiles""" - return _apply_column_function('rf_local_equal', left_tile_col, right_tile_col) - - -def rf_local_unequal(left_tile_col, right_tile_col): - """Cellwise inequality comparison between two tiles""" - return _apply_column_function('rf_local_unequal', left_tile_col, right_tile_col) - - -def rf_round(tile_col): - """Round cell values to the nearest integer without changing the cell type""" - return _apply_column_function('rf_round', tile_col) - - -def rf_abs(tile_col): - """Compute the absolute value of each cell""" - return _apply_column_function('rf_abs', tile_col) - - -def rf_log(tile_col): - """Performs cell-wise natural logarithm""" - return _apply_column_function('rf_log', tile_col) - - -def rf_log10(tile_col): - """Performs cell-wise logartithm with base 10""" - return _apply_column_function('rf_log10', tile_col) - - -def rf_log2(tile_col): - """Performs cell-wise logartithm with base 2""" - return _apply_column_function('rf_log2', tile_col) - - -def rf_log1p(tile_col): - """Performs natural logarithm of cell values plus one""" - return _apply_column_function('rf_log1p', tile_col) - - -def rf_exp(tile_col): - """Performs cell-wise exponential""" - return _apply_column_function('rf_exp', tile_col) - - -def rf_exp2(tile_col): - """Compute 2 to the power of cell values""" - return _apply_column_function('rf_exp2', tile_col) - - -def rf_exp10(tile_col): - """Compute 10 to the power of cell values""" - return _apply_column_function('rf_exp10', tile_col) - - -def rf_expm1(tile_col): - """Performs cell-wise exponential, then subtract one""" - return _apply_column_function('rf_expm1', tile_col) - - -def rf_identity(tile_col): - """Pass tile through unchanged""" - return _apply_column_function('rf_identity', tile_col) - - -def rf_resample(tile_col, scale_factor_col): - """Resample tile to different size based on scalar factor or tile whose dimension to match - Scalar less than one will downsample tile; greater than one will upsample. Uses nearest-neighbor.""" - return _apply_column_function('rf_resample', tile_col, scale_factor_col) - - -def rf_crs(tile_col): - """Get the CRS of a RasterSource or ProjectedRasterTile""" - return _apply_column_function('rf_crs', tile_col) - - -def rf_mk_crs(crs_text): - """Resolve CRS from text identifier. Supported registries are EPSG, ESRI, WORLD, NAD83, & NAD27. - An example of a valid CRS name is EPSG:3005.""" - return Column(_context_call('_make_crs_literal', crs_text)) - - -def st_extent(geom_col): - """Compute the extent/bbox of a Geometry (a tile with embedded extent and CRS)""" - return _apply_column_function('st_extent', geom_col) - - -def rf_extent(proj_raster_col): - """Get the extent of a RasterSource or ProjectedRasterTile (a tile with embedded extent and CRS)""" - return _apply_column_function('rf_extent', proj_raster_col) - - -def rf_tile(proj_raster_col): - """Extracts the Tile component of a ProjectedRasterTile (or Tile).""" - return _apply_column_function('rf_tile', proj_raster_col) - - -def st_geometry(geom_col): - """Convert the given extent/bbox to a polygon""" - return _apply_column_function('st_geometry', geom_col) - - -def rf_geometry(proj_raster_col): - """Get the extent of a RasterSource or ProjectdRasterTile as a Geometry""" - return _apply_column_function('rf_geometry', proj_raster_col) - -# ------ GeoMesa Functions ------ - -def st_geomFromGeoHash(*args): - """""" - return _apply_column_function('st_geomFromGeoHash', *args) - - -def st_geomFromWKT(*args): - """""" - return _apply_column_function('st_geomFromWKT', *args) - - -def st_geomFromWKB(*args): - """""" - return _apply_column_function('st_geomFromWKB', *args) - - -def st_lineFromText(*args): - """""" - return _apply_column_function('st_lineFromText', *args) - - -def st_makeBox2D(*args): - """""" - return _apply_column_function('st_makeBox2D', *args) - - -def st_makeBBox(*args): - """""" - return _apply_column_function('st_makeBBox', *args) - - -def st_makePolygon(*args): - """""" - return _apply_column_function('st_makePolygon', *args) - - -def st_makePoint(*args): - """""" - return _apply_column_function('st_makePoint', *args) - - -def st_makeLine(*args): - """""" - return _apply_column_function('st_makeLine', *args) - - -def st_makePointM(*args): - """""" - return _apply_column_function('st_makePointM', *args) - - -def st_mLineFromText(*args): - """""" - return _apply_column_function('st_mLineFromText', *args) - - -def st_mPointFromText(*args): - """""" - return _apply_column_function('st_mPointFromText', *args) - - -def st_mPolyFromText(*args): - """""" - return _apply_column_function('st_mPolyFromText', *args) - - -def st_point(*args): - """""" - return _apply_column_function('st_point', *args) - - -def st_pointFromGeoHash(*args): - """""" - return _apply_column_function('st_pointFromGeoHash', *args) - - -def st_pointFromText(*args): - """""" - return _apply_column_function('st_pointFromText', *args) - - -def st_pointFromWKB(*args): - """""" - return _apply_column_function('st_pointFromWKB', *args) - - -def st_polygon(*args): - """""" - return _apply_column_function('st_polygon', *args) - - -def st_polygonFromText(*args): - """""" - return _apply_column_function('st_polygonFromText', *args) - - -def st_castToPoint(*args): - """""" - return _apply_column_function('st_castToPoint', *args) - - -def st_castToPolygon(*args): - """""" - return _apply_column_function('st_castToPolygon', *args) - - -def st_castToLineString(*args): - """""" - return _apply_column_function('st_castToLineString', *args) - - -def st_byteArray(*args): - """""" - return _apply_column_function('st_byteArray', *args) - - -def st_boundary(*args): - """""" - return _apply_column_function('st_boundary', *args) - - -def st_coordDim(*args): - """""" - return _apply_column_function('st_coordDim', *args) - - -def st_dimension(*args): - """""" - return _apply_column_function('st_dimension', *args) - - -def st_envelope(*args): - """""" - return _apply_column_function('st_envelope', *args) - - -def st_exteriorRing(*args): - """""" - return _apply_column_function('st_exteriorRing', *args) - - -def st_geometryN(*args): - """""" - return _apply_column_function('st_geometryN', *args) - - -def st_geometryType(*args): - """""" - return _apply_column_function('st_geometryType', *args) - - -def st_interiorRingN(*args): - """""" - return _apply_column_function('st_interiorRingN', *args) - - -def st_isClosed(*args): - """""" - return _apply_column_function('st_isClosed', *args) - - -def st_isCollection(*args): - """""" - return _apply_column_function('st_isCollection', *args) - - -def st_isEmpty(*args): - """""" - return _apply_column_function('st_isEmpty', *args) - - -def st_isRing(*args): - """""" - return _apply_column_function('st_isRing', *args) - - -def st_isSimple(*args): - """""" - return _apply_column_function('st_isSimple', *args) - - -def st_isValid(*args): - """""" - return _apply_column_function('st_isValid', *args) - - -def st_numGeometries(*args): - """""" - return _apply_column_function('st_numGeometries', *args) - - -def st_numPoints(*args): - """""" - return _apply_column_function('st_numPoints', *args) - - -def st_pointN(*args): - """""" - return _apply_column_function('st_pointN', *args) - - -def st_x(*args): - """""" - return _apply_column_function('st_x', *args) - - -def st_y(*args): - """""" - return _apply_column_function('st_y', *args) - - -def st_asBinary(*args): - """""" - return _apply_column_function('st_asBinary', *args) - - -def st_asGeoJSON(*args): - """""" - return _apply_column_function('st_asGeoJSON', *args) - - -def st_asLatLonText(*args): - """""" - return _apply_column_function('st_asLatLonText', *args) - - -def st_asText(*args): - """""" - return _apply_column_function('st_asText', *args) - - -def st_geoHash(*args): - """""" - return _apply_column_function('st_geoHash', *args) - - -def st_bufferPoint(*args): - """""" - return _apply_column_function('st_bufferPoint', *args) - - -def st_antimeridianSafeGeom(*args): - """""" - return _apply_column_function('st_antimeridianSafeGeom', *args) - - -def st_translate(*args): - """""" - return _apply_column_function('st_translate', *args) - - -def st_contains(*args): - """""" - return _apply_column_function('st_contains', *args) - - -def st_covers(*args): - """""" - return _apply_column_function('st_covers', *args) - - -def st_crosses(*args): - """""" - return _apply_column_function('st_crosses', *args) - - -def st_disjoint(*args): - """""" - return _apply_column_function('st_disjoint', *args) - - -def st_equals(*args): - """""" - return _apply_column_function('st_equals', *args) - - -def st_intersects(*args): - """""" - return _apply_column_function('st_intersects', *args) - - -def st_overlaps(*args): - """""" - return _apply_column_function('st_overlaps', *args) - - -def st_touches(*args): - """""" - return _apply_column_function('st_touches', *args) - - -def st_within(*args): - """""" - return _apply_column_function('st_within', *args) - - -def st_relate(*args): - """""" - return _apply_column_function('st_relate', *args) - - -def st_relateBool(*args): - """""" - return _apply_column_function('st_relateBool', *args) - - -def st_area(*args): - """""" - return _apply_column_function('st_area', *args) - - -def st_closestPoint(*args): - """""" - return _apply_column_function('st_closestPoint', *args) - - -def st_centroid(*args): - """""" - return _apply_column_function('st_centroid', *args) - - -def st_distance(*args): - """""" - return _apply_column_function('st_distance', *args) - - -def st_distanceSphere(*args): - """""" - return _apply_column_function('st_distanceSphere', *args) - - -def st_length(*args): - """""" - return _apply_column_function('st_length', *args) - - -def st_aggregateDistanceSphere(*args): - """""" - return _apply_column_function('st_aggregateDistanceSphere', *args) - - -def st_lengthSphere(*args): - """""" - return _apply_column_function('st_lengthSphere', *args) diff --git a/pyrasterframes/src/main/python/pyrasterframes/rf_ipython.py b/pyrasterframes/src/main/python/pyrasterframes/rf_ipython.py deleted file mode 100644 index 0066e7dd7..000000000 --- a/pyrasterframes/src/main/python/pyrasterframes/rf_ipython.py +++ /dev/null @@ -1,219 +0,0 @@ -# -# This software is licensed under the Apache 2 license, quoted below. -# -# Copyright 2019 Astraea, Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); you may not -# use this file except in compliance with the License. You may obtain a copy of -# the License at -# -# [http://www.apache.org/licenses/LICENSE-2.0] -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT -# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the -# License for the specific language governing permissions and limitations under -# the License. -# -# SPDX-License-Identifier: Apache-2.0 -# - -import pyrasterframes.rf_types -import numpy as np - - -def plot_tile(tile, normalize=True, lower_percentile=1, upper_percentile=99, axis=None, **imshow_args): - """ - Display an image of the tile - - Parameters - ---------- - normalize: if True, will normalize the data between using - lower_percentile and upper_percentile as bounds - lower_percentile: between 0 and 100 inclusive. - Specifies to clip values below this percentile - upper_percentile: between 0 and 100 inclusive. - Specifies to clip values above this percentile - axis : matplotlib axis object to plot onto. Creates new axis if None - imshow_args : parameters to pass into matplotlib.pyplot.imshow - see https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.imshow.html - Returns - ------- - created or modified axis object - """ - - if axis is None: - import matplotlib.pyplot as plt - axis = plt.gca() - - arr = tile.cells - - def normalize_cells(cells): - assert upper_percentile > lower_percentile, 'invalid upper and lower percentiles {}, {}'.format(lower_percentile, upper_percentile) - sans_mask = np.array(cells) - lower = np.nanpercentile(sans_mask, lower_percentile) - upper = np.nanpercentile(sans_mask, upper_percentile) - cells_clipped = np.clip(cells, lower, upper) - return (cells_clipped - lower) / (upper - lower) - - axis.set_aspect('equal') - axis.xaxis.set_ticks([]) - axis.yaxis.set_ticks([]) - - if normalize: - cells = normalize_cells(arr) - else: - cells = arr - - axis.imshow(cells, **imshow_args) - - return axis - - -def tile_to_png(tile, lower_percentile=1, upper_percentile=99, title=None, fig_size=None): - """ Provide image of Tile.""" - if tile.cells is None: - return None - - import io - from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas - from matplotlib.figure import Figure - - # Set up matplotlib objects - nominal_size = 3 # approx full size for a 256x256 tile - if fig_size is None: - fig_size = (nominal_size, nominal_size) - - fig = Figure(figsize=fig_size) - canvas = FigureCanvas(fig) - axis = fig.add_subplot(1, 1, 1) - - plot_tile(tile, True, lower_percentile, upper_percentile, axis=axis) - axis.set_aspect('equal') - axis.xaxis.set_ticks([]) - axis.yaxis.set_ticks([]) - - if title is None: - axis.set_title('{}, {}'.format(tile.dimensions(), tile.cell_type.__repr__()), - fontsize=fig_size[0]*4) # compact metadata as title - else: - axis.set_title(title, fontsize=fig_size[0]*4) # compact metadata as title - - with io.BytesIO() as output: - canvas.print_png(output) - return output.getvalue() - - -def tile_to_html(tile, fig_size=None): - """ Provide HTML string representation of Tile image.""" - import base64 - b64_img_html = '
'
- png_bits = tile_to_png(tile, fig_size=fig_size)
- b64_png = base64.b64encode(png_bits).decode('utf-8').replace('\n', '')
- return b64_img_html.format(b64_png)
-
-
-def pandas_df_to_html(df):
- """Provide HTML formatting for pandas.DataFrame with rf_types.Tile in the columns. """
- import pandas as pd
- # honor the existing options on display
- if not pd.get_option("display.notebook_repr_html"):
- return None
-
- if len(df) == 0:
- return df._repr_html_()
-
- tile_cols = []
- for c in df.columns:
- if isinstance(df.iloc[0][c], pyrasterframes.rf_types.Tile): # if the first is a Tile try formatting
- tile_cols.append(c)
-
- def _safe_tile_to_html(t):
- if isinstance(t, pyrasterframes.rf_types.Tile):
- return tile_to_html(t, fig_size=(2, 2))
- else:
- # handles case where objects in a column are not all Tile type
- return t.__repr__()
-
- # dict keyed by column with custom rendering function
- formatter = {c: _safe_tile_to_html for c in tile_cols}
-
- # This is needed to avoid our tile being rendered as `\n",
- "
\n",
- " "
- ],
- "text/plain": [
- "SparkSession - in-memory
\n", - " \n", - "\n",
- "
\n",
- " \n",
- " SparkContext
\n", - "\n", - " \n", - "\n", - "- \n",
- "
- Version \n", - "
v2.3.4\n",
- " - Master \n", - "
local[*]\n",
- " - AppName \n", - "
pyspark-shell\n",
- "
\n",
- "\n",
- "\n",
- " \n",
- "
\n",
- "
"
- ],
- "text/plain": [
- " eod_collection_display_name eod_collection_family \\\n",
- "0 Landsat 8 landsat8 \n",
- "\n",
- " eod_collection_family_display_name eod_grid_id \\\n",
- "0 Landsat 8 WRS2-030027 \n",
- "\n",
- " created datetime eo_cloud_cover \\\n",
- "0 2019-08-19T20:54:33.413548Z 2018-07-17T17:15:57.1536740Z 1.49 \n",
- "\n",
- " eo_constellation eo_epsg eo_gsd ... \\\n",
- "0 landsat-8 32614 30.0 ... \n",
- "\n",
- " B2 \\\n",
- "0 https://landsat-pds.s3.us-west-2.amazonaws.com... \n",
- "\n",
- " BQA \\\n",
- "0 https://landsat-pds.s3.us-west-2.amazonaws.com... \n",
- "\n",
- " B4 \\\n",
- "0 https://landsat-pds.s3.us-west-2.amazonaws.com... \n",
- "\n",
- " B1 \\\n",
- "0 https://landsat-pds.s3.us-west-2.amazonaws.com... \n",
- "\n",
- " B8 \\\n",
- "0 https://landsat-pds.s3.us-west-2.amazonaws.com... \n",
- "\n",
- " B11 collection \\\n",
- "0 https://landsat-pds.s3.us-west-2.amazonaws.com... landsat8_l1tp \n",
- "\n",
- " geometry \\\n",
- "0 (POLYGON ((-98.62404379679178 46.4012557977134... \n",
- "\n",
- " id \\\n",
- "0 LC08_L1TP_030027_20180717_20180730_01_T1_L1TP \n",
- "\n",
- " target \n",
- "0 file:///tmp/scene_30_27_target_utm.tif \n",
- "\n",
- "[1 rows x 33 columns]"
- ]
- },
- "execution_count": 13,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "scene_df['target'] = 'file:///tmp/scene_30_27_target_utm.tif'\n",
- "scene_df"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "metadata": {},
- "outputs": [],
- "source": [
- "features_rf = spark.read.raster( \n",
- " catalog=scene_df,\n",
- " catalog_col_names=['B1', 'B2', 'B3', 'B4', 'B5', 'B6', 'B7', 'BQA', 'target'],\n",
- " tile_dimensions=(256, 256)\n",
- ").repartition(200)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Feature creation"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 16,
- "metadata": {},
- "outputs": [],
- "source": [
- "features_rf = features_rf.withColumn('ndvi', rf_normalized_difference(features_rf.B5, features_rf.B4)) \\\n",
- " .withColumn('ndwi1', rf_normalized_difference(features_rf.B5, features_rf.B6)) \\\n",
- " .withColumn('ndwi2', rf_normalized_difference(features_rf.B5, features_rf.B7)) \\\n",
- " .withColumn('ndwi3', rf_normalized_difference(features_rf.B3, features_rf.B5)) \\\n",
- " .withColumn('evi', rf_local_multiply(rf_local_divide(rf_local_subtract(features_rf.B5, features_rf.B4), rf_local_add(rf_local_subtract(rf_local_add(features_rf.B5, rf_local_multiply(features_rf.B4, lit(6.0))), rf_local_multiply(features_rf.B2, lit(7.5))), lit(1.0))), lit(2.5))) \\\n",
- " .withColumn('savi', rf_local_multiply(rf_local_divide(rf_local_subtract(features_rf.B5, features_rf.B4), rf_local_add(rf_local_add(features_rf.B5, features_rf.B4), lit(0.5))), lit(1.5))) \\\n",
- " .withColumn('osavi', rf_local_divide(rf_local_subtract(features_rf.B5, features_rf.B4), rf_local_add(rf_local_add(features_rf.B5, features_rf.B4), lit(0.16)))) \\\n",
- " .withColumn('satvi', rf_local_subtract(rf_local_multiply(rf_local_divide(rf_local_subtract(features_rf.B6, features_rf.B4),rf_local_add(rf_local_add(features_rf.B6, features_rf.B4), lit(0.5))), lit(1.5)), rf_local_divide(features_rf.B7, lit(2.0)))) \\\n",
- " .withColumn('mean_swir', rf_local_divide(rf_local_add(features_rf.B6, features_rf.B7), lit(2.0))) \\\n",
- " .withColumn('vli', rf_local_divide(rf_local_add(rf_local_add(rf_local_add(features_rf.B1, features_rf.B2), features_rf.B3), features_rf.B4), lit(4.0))) \\\n",
- " .withColumn('dbsi', rf_local_subtract(rf_normalized_difference(features_rf.B6, features_rf.B3), rf_normalized_difference(features_rf.B5, features_rf.B4)))\n",
- "\n",
- "features_rf = features_rf.select(\n",
- " features_rf.target,\n",
- " rf_crs(features_rf.B1).alias('crs'),\n",
- " rf_extent(features_rf.B1).alias('extent'),\n",
- " rf_tile(features_rf.B1).alias('coastal'),\n",
- " rf_tile(features_rf.B2).alias('blue'),\n",
- " rf_tile(features_rf.B3).alias('green'),\n",
- " rf_tile(features_rf.B4).alias('red'),\n",
- " rf_tile(features_rf.B5).alias('nir'),\n",
- " rf_tile(features_rf.B6).alias('swir1'),\n",
- " rf_tile(features_rf.B7).alias('swir2'),\n",
- " rf_tile(features_rf.ndvi).alias('ndvi'),\n",
- " rf_tile(features_rf.ndwi1).alias('ndwi1'),\n",
- " rf_tile(features_rf.ndwi2).alias('ndwi2'),\n",
- " rf_tile(features_rf.ndwi3).alias('ndwi3'),\n",
- " rf_tile(features_rf.evi).alias('evi'),\n",
- " rf_tile(features_rf.savi).alias('savi'),\n",
- " rf_tile(features_rf.osavi).alias('osavi'),\n",
- " rf_tile(features_rf.satvi).alias('satvi'),\n",
- " rf_tile(features_rf.mean_swir).alias('mean_swir'),\n",
- " rf_tile(features_rf.vli).alias('vli'),\n",
- " rf_tile(features_rf.dbsi).alias('dbsi'),\n",
- " rf_tile(features_rf.BQA).alias('qa')\n",
- ")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 17,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "root\n",
- " |-- target: struct (nullable = true)\n",
- " | |-- tile_context: struct (nullable = false)\n",
- " | | |-- extent: struct (nullable = false)\n",
- " | | | |-- xmin: double (nullable = false)\n",
- " | | | |-- ymin: double (nullable = false)\n",
- " | | | |-- xmax: double (nullable = false)\n",
- " | | | |-- ymax: double (nullable = false)\n",
- " | | |-- crs: struct (nullable = false)\n",
- " | | | |-- crsProj4: string (nullable = false)\n",
- " | |-- tile: tile (nullable = false)\n",
- " |-- crs: struct (nullable = true)\n",
- " | |-- crsProj4: string (nullable = false)\n",
- " |-- extent: struct (nullable = true)\n",
- " | |-- xmin: double (nullable = false)\n",
- " | |-- ymin: double (nullable = false)\n",
- " | |-- xmax: double (nullable = false)\n",
- " | |-- ymax: double (nullable = false)\n",
- " |-- coastal: tile (nullable = true)\n",
- " |-- blue: tile (nullable = true)\n",
- " |-- green: tile (nullable = true)\n",
- " |-- red: tile (nullable = true)\n",
- " |-- nir: tile (nullable = true)\n",
- " |-- swir1: tile (nullable = true)\n",
- " |-- swir2: tile (nullable = true)\n",
- " |-- ndvi: tile (nullable = true)\n",
- " |-- ndwi1: tile (nullable = true)\n",
- " |-- ndwi2: tile (nullable = true)\n",
- " |-- ndwi3: tile (nullable = true)\n",
- " |-- evi: tile (nullable = true)\n",
- " |-- savi: tile (nullable = true)\n",
- " |-- osavi: tile (nullable = true)\n",
- " |-- satvi: tile (nullable = true)\n",
- " |-- mean_swir: tile (nullable = true)\n",
- " |-- vli: tile (nullable = true)\n",
- " |-- dbsi: tile (nullable = true)\n",
- " |-- mask: tile (nullable = true)\n",
- "\n"
- ]
- }
- ],
- "source": [
- "# Values of qa band indicating cloudy conditions\n",
- "cloud = [2800, 2804, 2808, 2812, 6896, 6900, 6904, 6908]\n",
- "\n",
- "mask_part = features_rf \\\n",
- " .withColumn('cloud1', rf_local_equal('qa', lit(2800))) \\\n",
- " .withColumn('cloud2', rf_local_equal('qa', lit(2804))) \\\n",
- " .withColumn('cloud3', rf_local_equal('qa', lit(2808))) \\\n",
- " .withColumn('cloud4', rf_local_equal('qa', lit(2812))) \\\n",
- " .withColumn('cloud5', rf_local_equal('qa', lit(6896))) \\\n",
- " .withColumn('cloud6', rf_local_equal('qa', lit(6900))) \\\n",
- " .withColumn('cloud7', rf_local_equal('qa', lit(6904))) \\\n",
- " .withColumn('cloud8', rf_local_equal('qa', lit(6908))) \n",
- "\n",
- "df_mask_inv = mask_part \\\n",
- " .withColumn('mask', rf_local_add('cloud1', 'cloud2')) \\\n",
- " .withColumn('mask', rf_local_add('mask', 'cloud3')) \\\n",
- " .withColumn('mask', rf_local_add('mask', 'cloud4')) \\\n",
- " .withColumn('mask', rf_local_add('mask', 'cloud5')) \\\n",
- " .withColumn('mask', rf_local_add('mask', 'cloud6')) \\\n",
- " .withColumn('mask', rf_local_add('mask', 'cloud7')) \\\n",
- " .withColumn('mask', rf_local_add('mask', 'cloud8')) \\\n",
- " .drop('cloud1', 'cloud2', 'cloud3', 'cloud4', 'cloud5', 'cloud6', 'cloud7', 'cloud8', 'qa')\n",
- " \n",
- "# at this point the mask contains 0 for good cells and 1 for defect, etc\n",
- "# convert cell type and set value 1 to NoData\n",
- "# also set the value of 100 to nodata in the target. #darkarts\n",
- "mask_rf = df_mask_inv.withColumn('mask', rf_with_no_data(rf_convert_cell_type('mask', 'uint8'), 1.0)) \\\n",
- " .withColumn('target', rf_with_no_data('target', 100))\n",
- "\n",
- "mask_rf.printSchema()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Train/test split"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 19,
- "metadata": {},
- "outputs": [],
- "source": [
- "rf = mask_rf.withColumn('train_set', when(rand(seed=1234) > 0.3, 1).otherwise(0))\n",
- "train_df = rf.filter(rf.train_set == 1)\n",
- "test_df = rf.filter(rf.train_set == 0)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Create ML Pipeline"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "metadata": {},
- "outputs": [],
- "source": [
- "# exploded_df = train_df.select(rf_explode_tiles(array('coastal','blue','green','red','nir','swir1','swir2','ndvi','ndwi1','ndwi2','ndwi3','evi','savi','osavi','satvi','mean_swir','vli','dbsi','target', 'mask')))\n",
- "# exploded_df.show()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 20,
- "metadata": {},
- "outputs": [],
- "source": [
- "from pyrasterframes import TileExploder\n",
- "from pyrasterframes.rf_types import NoDataFilter\n",
- "\n",
- "from pyspark.ml.feature import VectorAssembler\n",
- "from pyspark.ml.classification import DecisionTreeClassifier\n",
- "from pyspark.ml.evaluation import MulticlassClassificationEvaluator\n",
- "from pyspark.ml import Pipeline"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 21,
- "metadata": {},
- "outputs": [],
- "source": [
- "exploder = TileExploder()\n",
- "\n",
- "noDataFilter = NoDataFilter() \\\n",
- " .setInputCols(['target', 'mask'])\n",
- "\n",
- "assembler = VectorAssembler() \\\n",
- " .setInputCols(['coastal','blue','green','red','nir','swir1','swir2','ndvi','ndwi1','ndwi2','ndwi3','evi','savi','osavi','satvi','mean_swir','vli','dbsi']) \\\n",
- " .setOutputCol(\"features\")\n",
- "\n",
- "classifier = DecisionTreeClassifier() \\\n",
- " .setLabelCol('target') \\\n",
- " .setMaxDepth(10) \\\n",
- " .setFeaturesCol(assembler.getOutputCol())\n",
- "\n",
- "pipeline = Pipeline() \\\n",
- " .setStages([exploder, noDataFilter, assembler, classifier])"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Train the model"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 22,
- "metadata": {},
- "outputs": [],
- "source": [
- "model = pipeline.fit(train_df)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "#model.transform(train_df).show()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": []
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "prediction_df = model.transform(test_df) \\\n",
- " .drop(assembler.getOutputCol()).cache()\n",
- "prediction_df.printSchema()\n",
- "\n",
- "eval = MulticlassClassificationEvaluator(\n",
- " predictionCol=classifier.getPredictionCol(),\n",
- " labelCol=classifier.getLabelCol(),\n",
- " metricName='fMeasureByThreshold'\n",
- ")\n",
- "\n",
- "f1score = eval.evaluate(prediction_df)\n",
- "print(\"\\nF1 Score:\", f1score)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "cnf_mtrx = prediction_df.groupBy(classifier.getPredictionCol()) \\\n",
- " .pivot(classifier.getLabelCol()) \\\n",
- " .count() \\\n",
- " .sort(classifier.getPredictionCol())\n",
- "cnf_mtrx"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.7.4"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/pyrasterframes/src/main/python/setup.cfg b/pyrasterframes/src/main/python/setup.cfg
deleted file mode 100644
index 4d9369ec4..000000000
--- a/pyrasterframes/src/main/python/setup.cfg
+++ /dev/null
@@ -1,13 +0,0 @@
-[metadata]
-license_files = LICENSE.txt
-
-[bdist_wheel]
-universal = 0
-
-[aliases]
-test = pytest
-
-[tool:pytest]
-addopts = --verbose
-testpaths = tests
-python_files = *.py
diff --git a/pyrasterframes/src/main/python/setup.py b/pyrasterframes/src/main/python/setup.py
deleted file mode 100644
index 0538768c8..000000000
--- a/pyrasterframes/src/main/python/setup.py
+++ /dev/null
@@ -1,232 +0,0 @@
-#
-# This software is licensed under the Apache 2 license, quoted below.
-#
-# Copyright 2019 Astraea, Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License"); you may not
-# use this file except in compliance with the License. You may obtain a copy of
-# the License at
-#
-# [http://www.apache.org/licenses/LICENSE-2.0]
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
-# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
-# License for the specific language governing permissions and limitations under
-# the License.
-#
-# SPDX-License-Identifier: Apache-2.0
-#
-
-# Always prefer setuptools over distutils
-from setuptools import setup
-from os import path
-import sys
-from glob import glob
-from io import open
-import distutils.cmd
-
-try:
- exec(open('pyrasterframes/version.py').read()) # executable python script contains __version__; credit pyspark
-except IOError:
- print("Run setup via `sbt 'pySetup arg1 arg2'` to ensure correct access to all source files and binaries.")
- sys.exit(-1)
-
-
-VERSION = __version__
-
-here = path.abspath(path.dirname(__file__))
-
-# Get the long description from the README file
-with open(path.join(here, 'README.md'), encoding='utf-8') as f:
- readme = f.read()
-
-
-def _divided(msg):
- divider = ('-' * 50)
- return divider + '\n' + msg + '\n' + divider
-
-
-class PweaveDocs(distutils.cmd.Command):
- """A custom command to run documentation scripts through pweave."""
- description = 'Pweave PyRasterFrames documentation scripts'
- user_options = [
- # The format is (long option, short option, description).
- ('files=', 's', 'Specific files to pweave. Defaults to all in `docs` directory.'),
- ('format=', 'f', 'Output format type. Defaults to `markdown`'),
- ('quick=', 'q', 'Check to see if the source file is newer than existing output before building. Defaults to `False`.')
- ]
-
-
- def initialize_options(self):
- """Set default values for options."""
- # Each user option must be listed here with their default value.
- self.files = filter(
- lambda x: not path.basename(x)[:1] == '_',
- glob(path.join(here, 'docs', '*.pymd'))
- )
- self.format = 'markdown'
- self.quick = False
-
- def finalize_options(self):
- """Post-process options."""
- import re
- if isinstance(self.files, str):
- self.files = filter(lambda s: len(s) > 0, re.split(',', self.files))
- # `html` doesn't do quite what one expects... only replaces code blocks, leaving markdown in place
- print("format.....", self.format)
- if self.format.strip() == 'html':
- self.format = 'pandoc2html'
- if isinstance(self.quick, str):
- self.quick = self.quick == 'True' or self.quick == 'true'
-
- def dest_file(self, src_file):
- return path.splitext(src_file)[0] + '.md'
-
- def run(self):
- """Run pweave."""
- import traceback
- import pweave
- from docs import PegdownMarkdownFormatter
-
- bad_words = ["Error"]
- pweave.rcParams["chunk"]["defaultoptions"].update({'wrap': False, 'dpi': 175})
- if self.format == 'markdown':
- pweave.PwebFormats.formats['markdown'] = {
- 'class': PegdownMarkdownFormatter,
- 'description': 'Pegdown compatible markdown'
- }
-
- for file in sorted(self.files, reverse=True):
- name = path.splitext(path.basename(file))[0]
- dest = self.dest_file(file)
-
- if (not self.quick) or (not path.exists(dest)) or (path.getmtime(dest) < path.getmtime(file)):
- print(_divided('Running %s' % name))
- try:
- pweave.weave(file=str(file), doctype=self.format)
- if self.format == 'markdown':
- if not path.exists(dest):
- raise FileNotFoundError("Markdown file '%s' didn't get created as expected" % dest)
- with open(dest, "r") as result:
- for (n, line) in enumerate(result):
- for word in bad_words:
- if word in line:
- raise ChildProcessError("Error detected on line %s in %s:\n%s" % (n + 1, dest, line))
-
- except Exception:
- print(_divided('%s Failed:' % file))
- print(traceback.format_exc())
- exit(1)
- else:
- print(_divided('Skipping %s' % name))
-
-
-class PweaveNotebooks(PweaveDocs):
- def initialize_options(self):
- super().initialize_options()
- self.format = 'notebook'
-
- def dest_file(self, src_file):
- return path.splitext(src_file)[0] + '.ipynb'
-
-pytz = 'pytz'
-shapely = 'Shapely>=1.6.0'
-pyspark ='pyspark==2.4.4'
-numpy = 'numpy>=1.12.0'
-matplotlib ='matplotlib'
-pandas = 'pandas>=0.24.2'
-geopandas = 'geopandas'
-requests = 'requests'
-pytest_runner = 'pytest-runner'
-setuptools = 'setuptools>=0.8'
-ipython = 'ipython==6.2.1'
-ipykernel = 'ipykernel==4.8.0'
-pweave = 'Pweave==0.30.3'
-fiona = 'fiona==1.8.6'
-rasterio = 'rasterio>=1.0.0'
-folium = 'folium'
-pytest = 'pytest>4.0.0,<5.0.0'
-pypandoc = 'pypandoc'
-boto3 = 'boto3'
-
-setup(
- name='pyrasterframes',
- description='Access and process geospatial raster data in PySpark DataFrames',
- long_description=readme,
- long_description_content_type='text/markdown',
- version=VERSION,
- author='Astraea, Inc.',
- author_email='info@astraea.earth',
- license='Apache 2',
- url='https://rasterframes.io',
- project_urls={
- 'Bug Reports': 'https://github.com/locationtech/rasterframes/issues',
- 'Source': 'https://github.com/locationtech/rasterframes',
- },
- python_requires=">=3.5",
- install_requires=[
- pytz,
- shapely,
- pyspark,
- numpy,
- pandas
- ],
- setup_requires=[
- pytz,
- shapely,
- pyspark,
- numpy,
- matplotlib,
- pandas,
- geopandas,
- requests,
- pytest_runner,
- setuptools,
- ipython,
- ipykernel,
- pweave,
- fiona,
- rasterio,
- folium
- ],
- tests_require=[
- pytest,
- pypandoc,
- numpy,
- shapely,
- pandas,
- rasterio,
- boto3,
- pweave
- ],
- packages=[
- 'pyrasterframes',
- 'geomesa_pyspark',
- 'pyrasterframes.jars',
- ],
- package_dir={
- 'pyrasterframes.jars': 'deps/jars'
- },
- package_data={
- 'pyrasterframes.jars': ['*.jar']
- },
- include_package_data=True,
- classifiers=[
- 'Development Status :: 4 - Beta',
- 'Environment :: Other Environment',
- 'License :: OSI Approved :: Apache Software License',
- 'Natural Language :: English',
- 'Operating System :: Unix',
- 'Programming Language :: Python',
- 'Topic :: Software Development :: Libraries',
- 'Topic :: Scientific/Engineering :: GIS',
- 'Topic :: Multimedia :: Graphics :: Graphics Conversion',
- ],
- zip_safe=False,
- test_suite="pytest-runner",
- cmdclass={
- 'pweave': PweaveDocs,
- 'notebooks': PweaveNotebooks
- }
-)
diff --git a/pyrasterframes/src/main/python/tests/ExploderTests.py b/pyrasterframes/src/main/python/tests/ExploderTests.py
deleted file mode 100644
index f7635ad7a..000000000
--- a/pyrasterframes/src/main/python/tests/ExploderTests.py
+++ /dev/null
@@ -1,58 +0,0 @@
-#
-# This software is licensed under the Apache 2 license, quoted below.
-#
-# Copyright 2019 Astraea, Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License"); you may not
-# use this file except in compliance with the License. You may obtain a copy of
-# the License at
-#
-# [http://www.apache.org/licenses/LICENSE-2.0]
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
-# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
-# License for the specific language governing permissions and limitations under
-# the License.
-#
-# SPDX-License-Identifier: Apache-2.0
-#
-
-from . import TestEnvironment
-
-from pyrasterframes.rasterfunctions import *
-from pyrasterframes.rf_types import *
-from pyrasterframes import TileExploder
-
-from pyspark.ml.feature import VectorAssembler
-from pyspark.ml import Pipeline
-from pyspark.sql.functions import *
-
-import unittest
-
-
-class ExploderTests(TestEnvironment):
-
- def test_tile_exploder_pipeline_for_prt(self):
- # NB the tile is a Projected Raster Tile
- df = self.spark.read.raster(self.img_uri)
- t_col = 'proj_raster'
- self.assertTrue(t_col in df.columns)
-
- assembler = VectorAssembler().setInputCols([t_col])
- pipe = Pipeline().setStages([TileExploder(), assembler])
- pipe_model = pipe.fit(df)
- tranformed_df = pipe_model.transform(df)
- self.assertTrue(tranformed_df.count() > df.count())
-
- def test_tile_exploder_pipeline_for_tile(self):
- t_col = 'tile'
- df = self.spark.read.raster(self.img_uri) \
- .withColumn(t_col, rf_tile('proj_raster')) \
- .drop('proj_raster')
-
- assembler = VectorAssembler().setInputCols([t_col])
- pipe = Pipeline().setStages([TileExploder(), assembler])
- pipe_model = pipe.fit(df)
- tranformed_df = pipe_model.transform(df)
- self.assertTrue(tranformed_df.count() > df.count())
diff --git a/pyrasterframes/src/main/python/tests/GeoTiffWriterTests.py b/pyrasterframes/src/main/python/tests/GeoTiffWriterTests.py
deleted file mode 100644
index ef28c6562..000000000
--- a/pyrasterframes/src/main/python/tests/GeoTiffWriterTests.py
+++ /dev/null
@@ -1,89 +0,0 @@
-#
-# This software is licensed under the Apache 2 license, quoted below.
-#
-# Copyright 2019 Astraea, Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License"); you may not
-# use this file except in compliance with the License. You may obtain a copy of
-# the License at
-#
-# [http://www.apache.org/licenses/LICENSE-2.0]
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
-# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
-# License for the specific language governing permissions and limitations under
-# the License.
-#
-# SPDX-License-Identifier: Apache-2.0
-#
-
-import os
-import tempfile
-
-from . import TestEnvironment
-import rasterio
-
-
-class GeoTiffWriter(TestEnvironment):
-
- @staticmethod
- def _tmpfile():
- return os.path.join(tempfile.gettempdir(), "pyrf-test.tif")
-
- def test_identity_write(self):
- rf = self.spark.read.geotiff(self.img_uri)
- rf_count = rf.count()
- self.assertTrue(rf_count > 0)
-
- dest = self._tmpfile()
- rf.write.geotiff(dest)
-
- rf2 = self.spark.read.geotiff(dest)
-
- self.assertEqual(rf2.count(), rf.count())
-
- os.remove(dest)
-
- def test_unstructured_write(self):
- rf = self.spark.read.raster(self.img_uri)
- dest_file = self._tmpfile()
- rf.write.geotiff(dest_file, crs='EPSG:32616')
-
- rf2 = self.spark.read.raster(dest_file)
- self.assertEqual(rf2.count(), rf.count())
-
- with rasterio.open(self.img_uri) as source:
- with rasterio.open(dest_file) as dest:
- self.assertEqual((dest.width, dest.height), (source.width, source.height))
- self.assertEqual(dest.bounds, source.bounds)
- self.assertEqual(dest.crs, source.crs)
-
- os.remove(dest_file)
-
- def test_unstructured_write_schemaless(self):
- # should be able to write a projected raster tile column to path like '/data/foo/file.tif'
- from pyrasterframes.rasterfunctions import rf_agg_stats, rf_crs
- rf = self.spark.read.raster(self.img_uri)
- max = rf.agg(rf_agg_stats('proj_raster').max.alias('max')).first()['max']
- crs = rf.select(rf_crs('proj_raster').crsProj4.alias('c')).first()['c']
-
- dest_file = self._tmpfile()
- self.assertTrue(not dest_file.startswith('file://'))
- rf.write.geotiff(dest_file, crs=crs)
-
- with rasterio.open(dest_file) as src:
- self.assertEqual(src.read().max(), max)
-
- os.remove(dest_file)
-
- def test_downsampled_write(self):
- rf = self.spark.read.raster(self.img_uri)
- dest = self._tmpfile()
- rf.write.geotiff(dest, crs='EPSG:32616', raster_dimensions=(128, 128))
-
- with rasterio.open(dest) as f:
- self.assertEqual((f.width, f.height), (128, 128))
-
- os.remove(dest)
-
diff --git a/pyrasterframes/src/main/python/tests/PyRasterFramesTests.py b/pyrasterframes/src/main/python/tests/PyRasterFramesTests.py
deleted file mode 100644
index 7cda3b997..000000000
--- a/pyrasterframes/src/main/python/tests/PyRasterFramesTests.py
+++ /dev/null
@@ -1,426 +0,0 @@
-#
-# This software is licensed under the Apache 2 license, quoted below.
-#
-# Copyright 2019 Astraea, Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License"); you may not
-# use this file except in compliance with the License. You may obtain a copy of
-# the License at
-#
-# [http://www.apache.org/licenses/LICENSE-2.0]
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
-# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
-# License for the specific language governing permissions and limitations under
-# the License.
-#
-# SPDX-License-Identifier: Apache-2.0
-#
-
-import unittest
-
-import numpy as np
-from pyrasterframes.rasterfunctions import *
-from pyrasterframes.rf_types import *
-from pyspark.sql import SQLContext
-from pyspark.sql.functions import *
-from . import TestEnvironment
-
-
-class UtilTest(TestEnvironment):
-
- def test_spark_confs(self):
- from . import app_name
- self.assertEqual(self.spark.conf.get('spark.app.name'), app_name)
- self.assertEqual(self.spark.conf.get('spark.ui.enabled'), 'false')
-
-
-class CellTypeHandling(unittest.TestCase):
-
- def test_is_raw(self):
- self.assertTrue(CellType("float32raw").is_raw())
- self.assertFalse(CellType("float64ud1234").is_raw())
- self.assertFalse(CellType("float32").is_raw())
- self.assertTrue(CellType("int8raw").is_raw())
- self.assertFalse(CellType("uint16d12").is_raw())
- self.assertFalse(CellType("int32").is_raw())
-
- def test_is_floating_point(self):
- self.assertTrue(CellType("float32raw").is_floating_point())
- self.assertTrue(CellType("float64ud1234").is_floating_point())
- self.assertTrue(CellType("float32").is_floating_point())
- self.assertFalse(CellType("int8raw").is_floating_point())
- self.assertFalse(CellType("uint16d12").is_floating_point())
- self.assertFalse(CellType("int32").is_floating_point())
-
- def test_cell_type_no_data(self):
- import math
- self.assertIsNone(CellType.bool().no_data_value())
-
- self.assertTrue(CellType.int8().has_no_data())
- self.assertEqual(CellType.int8().no_data_value(), -128)
-
- self.assertTrue(CellType.uint8().has_no_data())
- self.assertEqual(CellType.uint8().no_data_value(), 0)
-
- self.assertTrue(CellType.int16().has_no_data())
- self.assertEqual(CellType.int16().no_data_value(), -32768)
-
- self.assertTrue(CellType.uint16().has_no_data())
- self.assertEqual(CellType.uint16().no_data_value(), 0)
-
- self.assertTrue(CellType.float32().has_no_data())
- self.assertTrue(np.isnan(CellType.float32().no_data_value()))
-
- self.assertEqual(CellType("float32ud-98").no_data_value(), -98.0)
- self.assertEqual(CellType("float32ud-98").no_data_value(), -98)
- self.assertEqual(CellType("int32ud-98").no_data_value(), -98.0)
- self.assertEqual(CellType("int32ud-98").no_data_value(), -98)
-
- self.assertTrue(math.isnan(CellType.float64().no_data_value()))
- self.assertEqual(CellType.uint8().no_data_value(), 0)
-
- def test_cell_type_conversion(self):
- for ct in rf_cell_types():
- self.assertEqual(ct.to_numpy_dtype(),
- CellType.from_numpy_dtype(ct.to_numpy_dtype()).to_numpy_dtype(),
- "dtype comparison for " + str(ct))
- if not ct.is_raw():
- self.assertEqual(ct,
- CellType.from_numpy_dtype(ct.to_numpy_dtype()),
- "GTCellType comparison for " + str(ct))
- else:
- ct_ud = ct.with_no_data_value(99)
- self.assertEqual(ct_ud.base_cell_type_name(),
- repr(CellType.from_numpy_dtype(ct_ud.to_numpy_dtype())),
- "GTCellType comparison for " + str(ct_ud)
- )
-
-
-class UDT(TestEnvironment):
-
- def setUp(self):
- self.create_layer()
-
- def test_mask_no_data(self):
- t1 = Tile(np.array([[1, 2], [3, 4]]), CellType("int8ud3"))
- self.assertTrue(t1.cells.mask[1][0])
- self.assertIsNotNone(t1.cells[1][1])
- self.assertEqual(len(t1.cells.compressed()), 3)
-
- t2 = Tile(np.array([[1.0, 2.0], [float('nan'), 4.0]]), CellType.float32())
- self.assertEqual(len(t2.cells.compressed()), 3)
- self.assertTrue(t2.cells.mask[1][0])
- self.assertIsNotNone(t2.cells[1][1])
-
- def test_tile_udt_serialization(self):
- from pyspark.sql.types import StructType, StructField
-
- udt = TileUDT()
- cell_types = (ct for ct in rf_cell_types() if not (ct.is_raw() or ("bool" in ct.base_cell_type_name())))
-
- for ct in cell_types:
- cells = (100 + np.random.randn(3, 3) * 100).astype(ct.to_numpy_dtype())
-
- if ct.is_floating_point():
- nd = 33.0
- else:
- nd = 33
-
- cells[1][1] = nd
- a_tile = Tile(cells, ct.with_no_data_value(nd))
- round_trip = udt.fromInternal(udt.toInternal(a_tile))
- self.assertEquals(a_tile, round_trip, "round-trip serialization for " + str(ct))
-
- schema = StructType([StructField("tile", TileUDT(), False)])
- df = self.spark.createDataFrame([{"tile": a_tile}], schema)
-
- long_trip = df.first()["tile"]
- self.assertEqual(long_trip, a_tile)
-
- def test_udf_on_tile_type_input(self):
- import numpy.testing
- df = self.spark.read.raster(self.img_uri)
- rf = self.rf
-
- # create trivial UDF that does something we already do with raster_Functions
- @udf('integer')
- def my_udf(t):
- a = t.cells
- return a.size # same as rf_dimensions.cols * rf_dimensions.rows
-
- rf_result = rf.select(
- (rf_dimensions('tile').cols.cast('int') * rf_dimensions('tile').rows.cast('int')).alias('expected'),
- my_udf('tile').alias('result')).toPandas()
-
- numpy.testing.assert_array_equal(
- rf_result.expected.tolist(),
- rf_result.result.tolist()
- )
-
- df_result = df.select(
- (rf_dimensions(df.proj_raster).cols.cast('int') * rf_dimensions(df.proj_raster).rows.cast('int') -
- my_udf(rf_tile(df.proj_raster))).alias('result')
- ).toPandas()
-
- numpy.testing.assert_array_equal(
- np.zeros(len(df_result)),
- df_result.result.tolist()
- )
-
- def test_udf_on_tile_type_output(self):
- import numpy.testing
-
- rf = self.rf
-
- # create a trivial UDF that does something we already do with a raster_functions
- @udf(TileUDT())
- def my_udf(t):
- import numpy as np
- return Tile(np.log1p(t.cells))
-
- rf_result = rf.select(
- rf_tile_max(
- rf_local_subtract(
- my_udf(rf.tile),
- rf_log1p(rf.tile)
- )
- ).alias('expect_zeros')
- ).collect()
-
- # almost equal because of different implemenations under the hoods: C (numpy) versus Java (rf_)
- numpy.testing.assert_almost_equal(
- [r['expect_zeros'] for r in rf_result],
- [0.0 for _ in rf_result],
- decimal=6
- )
-
- def test_no_data_udf_handling(self):
- from pyspark.sql.types import StructType, StructField
-
- t1 = Tile(np.array([[1, 2], [0, 4]]), CellType.uint8())
- self.assertEqual(t1.cell_type.to_numpy_dtype(), np.dtype("uint8"))
- e1 = Tile(np.array([[2, 3], [0, 5]]), CellType.uint8())
- schema = StructType([StructField("tile", TileUDT(), False)])
- df = self.spark.createDataFrame([{"tile": t1}], schema)
-
- @udf(TileUDT())
- def increment(t):
- return t + 1
-
- r1 = df.select(increment(df.tile).alias("inc")).first()["inc"]
- self.assertEqual(r1, e1)
-
- def test_udf_np_implicit_type_conversion(self):
- import math
- import pandas
-
- a1 = np.array([[1, 2], [0, 4]])
- t1 = Tile(a1, CellType.uint8())
- exp_array = a1.astype('>f8')
-
- @udf(TileUDT())
- def times_pi(t):
- return t * math.pi
-
- @udf(TileUDT())
- def divide_pi(t):
- return t / math.pi
-
- @udf(TileUDT())
- def plus_pi(t):
- return t + math.pi
-
- @udf(TileUDT())
- def less_pi(t):
- return t - math.pi
-
- df = self.spark.createDataFrame(pandas.DataFrame([{"tile": t1}]))
- r1 = df.select(
- less_pi(divide_pi(times_pi(plus_pi(df.tile))))
- ).first()[0]
-
- self.assertTrue(np.all(r1.cells == exp_array))
- self.assertEqual(r1.cells.dtype, exp_array.dtype)
-
-
-class TileOps(TestEnvironment):
-
- def setUp(self):
- from pyspark.sql import Row
- # convenience so we can assert around Tile() == Tile()
- self.t1 = Tile(np.array([[1, 2],
- [3, 4]]), CellType.int8().with_no_data_value(3))
- self.t2 = Tile(np.array([[1, 2],
- [3, 4]]), CellType.int8().with_no_data_value(1))
- self.t3 = Tile(np.array([[1, 2],
- [-3, 4]]), CellType.int8().with_no_data_value(3))
-
- self.df = self.spark.createDataFrame([Row(t1=self.t1, t2=self.t2, t3=self.t3)])
-
- def test_addition(self):
- e1 = np.ma.masked_equal(np.array([[5, 6],
- [7, 8]]), 7)
- self.assertTrue(np.array_equal((self.t1 + 4).cells, e1))
-
- e2 = np.ma.masked_equal(np.array([[3, 4],
- [3, 8]]), 3)
- r2 = (self.t1 + self.t2).cells
- self.assertTrue(np.ma.allequal(r2, e2))
-
- col_result = self.df.select(rf_local_add('t1', 't3').alias('sum')).first()
- self.assertEqual(col_result.sum, self.t1 + self.t3)
-
- def test_multiplication(self):
- e1 = np.ma.masked_equal(np.array([[4, 8],
- [12, 16]]), 12)
-
- self.assertTrue(np.array_equal((self.t1 * 4).cells, e1))
-
- e2 = np.ma.masked_equal(np.array([[3, 4], [3, 16]]), 3)
- r2 = (self.t1 * self.t2).cells
- self.assertTrue(np.ma.allequal(r2, e2))
-
- r3 = self.df.select(rf_local_multiply('t1', 't3').alias('r3')).first().r3
- self.assertEqual(r3, self.t1 * self.t3)
-
- def test_subtraction(self):
- t3 = self.t1 * 4
- r1 = t3 - self.t1
- # note careful construction of mask value and dtype above
- e1 = Tile(np.ma.masked_equal(np.array([[4 - 1, 8 - 2],
- [3, 16 - 4]], dtype='int8'),
- 3, )
- )
- self.assertTrue(r1 == e1,
- "{} does not equal {}".format(r1, e1))
- # put another way
- self.assertTrue(r1 == self.t1 * 3,
- "{} does not equal {}".format(r1, self.t1 * 3))
-
- def test_division(self):
- t3 = self.t1 * 9
- r1 = t3 / 9
- self.assertTrue(np.array_equal(r1.cells, self.t1.cells),
- "{} does not equal {}".format(r1, self.t1))
-
- r2 = (self.t1 / self.t1).cells
- self.assertTrue(np.array_equal(r2, np.array([[1,1], [1, 1]], dtype=r2.dtype)))
-
- def test_matmul(self):
- # if sys.version >= '3.5': # per https://docs.python.org/3.7/library/operator.html#operator.matmul new in 3.5
- # r1 = self.t1 @ self.t2
- r1 = self.t1.__matmul__(self.t2)
-
- # The behavior of np.matmul with masked arrays is not well documented
- # it seems to treat the 2nd arg as if not a MaskedArray
- e1 = Tile(np.matmul(self.t1.cells, self.t2.cells), r1.cell_type)
-
- self.assertTrue(r1 == e1, "{} was not equal to {}".format(r1, e1))
- self.assertEqual(r1, e1)
-
-
-class PandasInterop(TestEnvironment):
-
- def setUp(self):
- self.create_layer()
-
- def test_pandas_conversion(self):
- import pandas as pd
- # pd.options.display.max_colwidth = 256
- cell_types = (ct for ct in rf_cell_types() if not (ct.is_raw() or ("bool" in ct.base_cell_type_name())))
- tiles = [Tile(np.random.randn(5, 5) * 100, ct) for ct in cell_types]
- in_pandas = pd.DataFrame({
- 'tile': tiles
- })
-
- in_spark = self.spark.createDataFrame(in_pandas)
- out_pandas = in_spark.select(rf_identity('tile').alias('tile')).toPandas()
- self.assertTrue(out_pandas.equals(in_pandas), str(in_pandas) + "\n\n" + str(out_pandas))
-
- def test_extended_pandas_ops(self):
- import pandas as pd
-
- self.assertIsInstance(self.rf.sql_ctx, SQLContext)
-
- # Try to collect self.rf which is read from a geotiff
- rf_collect = self.rf.take(2)
- self.assertTrue(
- all([isinstance(row.tile.cells, np.ndarray) for row in rf_collect]))
-
- # Try to create a tile from numpy.
- self.assertEqual(Tile(np.random.randn(10, 10), CellType.int8()).dimensions(), [10, 10])
-
- tiles = [Tile(np.random.randn(10, 12), CellType.float64()) for _ in range(3)]
- to_spark = pd.DataFrame({
- 't': tiles,
- 'b': ['a', 'b', 'c'],
- 'c': [1, 2, 4],
- })
- rf_maybe = self.spark.createDataFrame(to_spark)
-
- # rf_maybe.select(rf_render_matrix(rf_maybe.t)).show(truncate=False)
-
- # Try to do something with it.
- sums = to_spark.t.apply(lambda a: a.cells.sum()).tolist()
- maybe_sums = rf_maybe.select(rf_tile_sum(rf_maybe.t).alias('tsum'))
- maybe_sums = [r.tsum for r in maybe_sums.collect()]
- np.testing.assert_almost_equal(maybe_sums, sums, 12)
-
- # Test round trip for an array
- simple_array = Tile(np.array([[1, 2], [3, 4]]), CellType.float64())
- to_spark_2 = pd.DataFrame({
- 't': [simple_array]
- })
-
- rf_maybe_2 = self.spark.createDataFrame(to_spark_2)
- #print("RasterFrameLayer `show`:")
- #rf_maybe_2.select(rf_render_matrix(rf_maybe_2.t).alias('t')).show(truncate=False)
-
- pd_2 = rf_maybe_2.toPandas()
- array_back_2 = pd_2.iloc[0].t
- #print("Array collected from toPandas output\n", array_back_2)
-
- self.assertIsInstance(array_back_2, Tile)
- np.testing.assert_equal(array_back_2.cells, simple_array.cells)
-
-
-class RasterJoin(TestEnvironment):
-
- def setUp(self):
- self.create_layer()
-
- def test_raster_join(self):
- # re-read the same source
- rf_prime = self.spark.read.geotiff(self.img_uri) \
- .withColumnRenamed('tile', 'tile2').alias('rf_prime')
-
- rf_joined = self.rf.raster_join(rf_prime)
-
- self.assertTrue(rf_joined.count(), self.rf.count())
- self.assertTrue(len(rf_joined.columns) == len(self.rf.columns) + len(rf_prime.columns) - 2)
-
- rf_joined_2 = self.rf.raster_join(rf_prime, self.rf.extent, self.rf.crs, rf_prime.extent, rf_prime.crs)
- self.assertTrue(rf_joined_2.count(), self.rf.count())
- self.assertTrue(len(rf_joined_2.columns) == len(self.rf.columns) + len(rf_prime.columns) - 2)
-
- # this will bring arbitrary additional data into join; garbage result
- join_expression = self.rf.extent.xmin == rf_prime.extent.xmin
- rf_joined_3 = self.rf.raster_join(rf_prime, self.rf.extent, self.rf.crs,
- rf_prime.extent, rf_prime.crs,
- join_expression)
- self.assertTrue(rf_joined_3.count(), self.rf.count())
- self.assertTrue(len(rf_joined_3.columns) == len(self.rf.columns) + len(rf_prime.columns) - 2)
-
- # throws if you don't pass in all expected columns
- with self.assertRaises(AssertionError):
- self.rf.raster_join(rf_prime, join_exprs=self.rf.extent)
-
-
-def suite():
- function_tests = unittest.TestSuite()
- return function_tests
-
-
-unittest.TextTestRunner().run(suite())
diff --git a/pyrasterframes/src/main/python/tests/RasterFunctionsTests.py b/pyrasterframes/src/main/python/tests/RasterFunctionsTests.py
deleted file mode 100644
index ca17dc325..000000000
--- a/pyrasterframes/src/main/python/tests/RasterFunctionsTests.py
+++ /dev/null
@@ -1,361 +0,0 @@
-#
-# This software is licensed under the Apache 2 license, quoted below.
-#
-# Copyright 2019 Astraea, Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License"); you may not
-# use this file except in compliance with the License. You may obtain a copy of
-# the License at
-#
-# [http://www.apache.org/licenses/LICENSE-2.0]
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
-# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
-# License for the specific language governing permissions and limitations under
-# the License.
-#
-# SPDX-License-Identifier: Apache-2.0
-#
-
-from pyrasterframes.rasterfunctions import *
-from pyrasterframes.utils import gdal_version
-from pyrasterframes.rf_types import Tile
-from pyspark import Row
-from pyspark.sql.functions import *
-
-
-from . import TestEnvironment
-
-
-class RasterFunctions(TestEnvironment):
-
- def setUp(self):
- self.create_layer()
-
- def test_setup(self):
- self.assertEqual(self.spark.sparkContext.getConf().get("spark.serializer"),
- "org.apache.spark.serializer.KryoSerializer")
- print("GDAL version", gdal_version())
-
- def test_identify_columns(self):
- cols = self.rf.tile_columns()
- self.assertEqual(len(cols), 1, '`tileColumns` did not find the proper number of columns.')
- print("Tile columns: ", cols)
- col = self.rf.spatial_key_column()
- self.assertIsInstance(col, Column, '`spatialKeyColumn` was not found')
- print("Spatial key column: ", col)
- col = self.rf.temporal_key_column()
- self.assertIsNone(col, '`temporalKeyColumn` should be `None`')
- print("Temporal key column: ", col)
-
- def test_tile_creation(self):
- from pyrasterframes.rf_types import CellType
-
- base = self.spark.createDataFrame([1, 2, 3, 4], 'integer')
- tiles = base.select(rf_make_constant_tile(3, 3, 3, "int32"), rf_make_zeros_tile(3, 3, "int32"),
- rf_make_ones_tile(3, 3, CellType.int32()))
- tiles.show()
- self.assertEqual(tiles.count(), 4)
-
- def test_multi_column_operations(self):
- df1 = self.rf.withColumnRenamed('tile', 't1').as_layer()
- df2 = self.rf.withColumnRenamed('tile', 't2').as_layer()
- df3 = df1.spatial_join(df2).as_layer()
- df3 = df3.withColumn('norm_diff', rf_normalized_difference('t1', 't2'))
- # df3.printSchema()
-
- aggs = df3.agg(
- rf_agg_mean('norm_diff'),
- )
- aggs.show()
- row = aggs.first()
-
- self.assertTrue(self.rounded_compare(row['rf_agg_mean(norm_diff)'], 0))
-
- def test_general(self):
- meta = self.rf.tile_layer_metadata()
- self.assertIsNotNone(meta['bounds'])
- df = self.rf.withColumn('dims', rf_dimensions('tile')) \
- .withColumn('type', rf_cell_type('tile')) \
- .withColumn('dCells', rf_data_cells('tile')) \
- .withColumn('ndCells', rf_no_data_cells('tile')) \
- .withColumn('min', rf_tile_min('tile')) \
- .withColumn('max', rf_tile_max('tile')) \
- .withColumn('mean', rf_tile_mean('tile')) \
- .withColumn('sum', rf_tile_sum('tile')) \
- .withColumn('stats', rf_tile_stats('tile')) \
- .withColumn('extent', st_extent('geometry')) \
- .withColumn('extent_geom1', st_geometry('extent')) \
- .withColumn('ascii', rf_render_ascii('tile')) \
- .withColumn('log', rf_log('tile')) \
- .withColumn('exp', rf_exp('tile')) \
- .withColumn('expm1', rf_expm1('tile')) \
- .withColumn('round', rf_round('tile')) \
- .withColumn('abs', rf_abs('tile'))
-
- df.first()
-
- def test_agg_mean(self):
- mean = self.rf.agg(rf_agg_mean('tile')).first()['rf_agg_mean(tile)']
- self.assertTrue(self.rounded_compare(mean, 10160))
-
- def test_agg_local_mean(self):
- from pyspark.sql import Row
- from pyrasterframes.rf_types import Tile
- import numpy as np
-
- # this is really testing the nodata propagation in the agg local summation
- ct = CellType.int8().with_no_data_value(4)
- df = self.spark.createDataFrame([
- Row(tile=Tile(np.array([[1, 2, 3, 4, 5, 6]]), ct)),
- Row(tile=Tile(np.array([[1, 2, 4, 3, 5, 6]]), ct)),
- ])
-
- result = df.agg(rf_agg_local_mean('tile').alias('mean')).first().mean
-
- expected = Tile(np.array([[1.0, 2.0, 3.0, 3.0, 5.0, 6.0]]), CellType.float64())
- self.assertEqual(result, expected)
-
- def test_aggregations(self):
- aggs = self.rf.agg(
- rf_agg_data_cells('tile'),
- rf_agg_no_data_cells('tile'),
- rf_agg_stats('tile'),
- rf_agg_approx_histogram('tile')
- )
- row = aggs.first()
-
- # print(row['rf_agg_data_cells(tile)'])
- self.assertEqual(row['rf_agg_data_cells(tile)'], 387000)
- self.assertEqual(row['rf_agg_no_data_cells(tile)'], 1000)
- self.assertEqual(row['rf_agg_stats(tile)'].data_cells, row['rf_agg_data_cells(tile)'])
-
- def test_sql(self):
-
- self.rf.createOrReplaceTempView("rf_test_sql")
-
- arith = self.spark.sql("""SELECT tile,
- rf_local_add(tile, 1) AS add_one,
- rf_local_subtract(tile, 1) AS less_one,
- rf_local_multiply(tile, 2) AS times_two,
- rf_local_divide(
- rf_convert_cell_type(tile, "float32"),
- 2) AS over_two
- FROM rf_test_sql""")
-
- arith.createOrReplaceTempView('rf_test_sql_1')
- arith.show(truncate=False)
- stats = self.spark.sql("""
- SELECT rf_tile_mean(tile) as base,
- rf_tile_mean(add_one) as plus_one,
- rf_tile_mean(less_one) as minus_one,
- rf_tile_mean(times_two) as double,
- rf_tile_mean(over_two) as half,
- rf_no_data_cells(tile) as nd
-
- FROM rf_test_sql_1
- ORDER BY rf_no_data_cells(tile)
- """)
- stats.show(truncate=False)
- stats.createOrReplaceTempView('rf_test_sql_stats')
-
- compare = self.spark.sql("""
- SELECT
- plus_one - 1.0 = base as add,
- minus_one + 1.0 = base as subtract,
- double / 2.0 = base as multiply,
- half * 2.0 = base as divide,
- nd
- FROM rf_test_sql_stats
- """)
-
- expect_row1 = compare.orderBy('nd').first()
-
- self.assertTrue(expect_row1.subtract)
- self.assertTrue(expect_row1.multiply)
- self.assertTrue(expect_row1.divide)
- self.assertEqual(expect_row1.nd, 0)
- self.assertTrue(expect_row1.add)
-
- expect_row2 = compare.orderBy('nd', ascending=False).first()
-
- self.assertTrue(expect_row2.subtract)
- self.assertTrue(expect_row2.multiply)
- self.assertTrue(expect_row2.divide)
- self.assertTrue(expect_row2.nd > 0)
- self.assertTrue(expect_row2.add) # <-- Would fail in a case where ND + 1 = 1
-
- def test_explode(self):
- import pyspark.sql.functions as F
- self.rf.select('spatial_key', rf_explode_tiles('tile')).show()
- # +-----------+------------+---------+-------+
- # |spatial_key|column_index|row_index|tile |
- # +-----------+------------+---------+-------+
- # |[2,1] |4 |0 |10150.0|
- cell = self.rf.select(self.rf.spatial_key_column(), rf_explode_tiles(self.rf.tile)) \
- .where(F.col("spatial_key.col") == 2) \
- .where(F.col("spatial_key.row") == 1) \
- .where(F.col("column_index") == 4) \
- .where(F.col("row_index") == 0) \
- .select(F.col("tile")) \
- .collect()[0][0]
- self.assertEqual(cell, 10150.0)
-
- # Test the sample version
- frac = 0.01
- sample_count = self.rf.select(rf_explode_tiles_sample(frac, 1872, 'tile')).count()
- print('Sample count is {}'.format(sample_count))
- self.assertTrue(sample_count > 0)
- self.assertTrue(sample_count < (frac * 1.1) * 387000) # give some wiggle room
-
- def test_mask_by_value(self):
- from pyspark.sql.functions import lit
-
- # create an artificial mask for values > 25000; masking value will be 4
- mask_value = 4
-
- rf1 = self.rf.select(self.rf.tile,
- rf_local_multiply(
- rf_convert_cell_type(
- rf_local_greater_int(self.rf.tile, 25000),
- "uint8"),
- lit(mask_value)).alias('mask'))
- rf2 = rf1.select(rf1.tile, rf_mask_by_value(rf1.tile, rf1.mask, lit(mask_value)).alias('masked'))
- result = rf2.agg(rf_agg_no_data_cells(rf2.tile) < rf_agg_no_data_cells(rf2.masked)) \
- .collect()[0][0]
- self.assertTrue(result)
-
- rf3 = rf1.select(rf1.tile, rf_inverse_mask_by_value(rf1.tile, rf1.mask, lit(mask_value)).alias('masked'))
- result = rf3.agg(rf_agg_no_data_cells(rf3.tile) < rf_agg_no_data_cells(rf3.masked)) \
- .collect()[0][0]
- self.assertTrue(result)
-
- def test_mask(self):
- from pyspark.sql import Row
- from pyrasterframes.rf_types import Tile, CellType
- import numpy as np
-
- np.random.seed(999)
- ma = np.ma.array(np.random.randint(0, 10, (5, 5), dtype='int8'), mask=np.random.rand(5, 5) > 0.7)
- expected_data_values = ma.compressed().size
- expected_no_data_values = ma.size - expected_data_values
- self.assertTrue(expected_data_values > 0, "Make sure random seed is cooperative ")
- self.assertTrue(expected_no_data_values > 0, "Make sure random seed is cooperative ")
-
- df = self.spark.createDataFrame([
- Row(t=Tile(np.ones(ma.shape, ma.dtype)), m=Tile(ma))
- ])
-
- df = df.withColumn('masked_t', rf_mask('t', 'm'))
- result = df.select(rf_data_cells('masked_t')).first()[0]
- self.assertEqual(result, expected_data_values)
-
- nd_result = df.select(rf_no_data_cells('masked_t')).first()[0]
- self.assertEqual(nd_result, expected_no_data_values)
-
- def test_resample(self):
- from pyspark.sql.functions import lit
- result = self.rf.select(
- rf_tile_min(rf_local_equal(
- rf_resample(rf_resample(self.rf.tile, lit(2)), lit(0.5)),
- self.rf.tile))
- ).collect()[0][0]
-
- self.assertTrue(result == 1) # short hand for all values are true
-
- def test_exists_for_all(self):
- df = self.rf.withColumn('should_exist', rf_make_ones_tile(5, 5, 'int8')) \
- .withColumn('should_not_exist', rf_make_zeros_tile(5, 5, 'int8'))
-
- should_exist = df.select(rf_exists(df.should_exist).alias('se')).take(1)[0].se
- self.assertTrue(should_exist)
-
- should_not_exist = df.select(rf_exists(df.should_not_exist).alias('se')).take(1)[0].se
- self.assertTrue(not should_not_exist)
-
- self.assertTrue(df.select(rf_for_all(df.should_exist).alias('se')).take(1)[0].se)
- self.assertTrue(not df.select(rf_for_all(df.should_not_exist).alias('se')).take(1)[0].se)
-
- def test_cell_type_in_functions(self):
- from pyrasterframes.rf_types import CellType
- ct = CellType.float32().with_no_data_value(-999)
-
- df = self.rf.withColumn('ct_str', rf_convert_cell_type('tile', ct.cell_type_name)) \
- .withColumn('ct', rf_convert_cell_type('tile', ct)) \
- .withColumn('make', rf_make_constant_tile(99, 3, 4, CellType.int8())) \
- .withColumn('make2', rf_with_no_data('make', 99))
-
- result = df.select('ct', 'ct_str', 'make', 'make2').first()
-
- self.assertEqual(result['ct'].cell_type, ct)
- self.assertEqual(result['ct_str'].cell_type, ct)
- self.assertEqual(result['make'].cell_type, CellType.int8())
-
- counts = df.select(
- rf_no_data_cells('make').alias("nodata1"),
- rf_data_cells('make').alias("data1"),
- rf_no_data_cells('make2').alias("nodata2"),
- rf_data_cells('make2').alias("data2")
- ).first()
-
- self.assertEqual(counts["data1"], 3 * 4)
- self.assertEqual(counts["nodata1"], 0)
- self.assertEqual(counts["data2"], 0)
- self.assertEqual(counts["nodata2"], 3 * 4)
- self.assertEqual(result['make2'].cell_type, CellType.int8().with_no_data_value(99))
-
- def test_render_composite(self):
- cat = self.spark.createDataFrame([
- Row(red=self.l8band_uri(4), green=self.l8band_uri(3), blue=self.l8band_uri(2))
- ])
- rf = self.spark.read.raster(cat, catalog_col_names=cat.columns)
-
- # Test composite construction
- rgb = rf.select(rf_tile(rf_rgb_composite('red', 'green', 'blue')).alias('rgb')).first()['rgb']
-
- # TODO: how to better test this?
- self.assertIsInstance(rgb, Tile)
- self.assertEqual(rgb.dimensions(), [186, 169])
-
- ## Test PNG generation
- png_bytes = rf.select(rf_render_png('red', 'green', 'blue').alias('png')).first()['png']
- # Look for the PNG magic cookie
- self.assertEqual(png_bytes[0:8], bytearray([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]))
-
-
-
-
- def test_rf_interpret_cell_type_as(self):
- from pyspark.sql import Row
- from pyrasterframes.rf_types import Tile
- import numpy as np
-
- df = self.spark.createDataFrame([
- Row(t=Tile(np.array([[1, 3, 4], [5, 0, 3]]), CellType.uint8().with_no_data_value(5)))
- ])
- df = df.withColumn('tile', rf_interpret_cell_type_as('t', 'uint8ud3')) # threes become ND
- result = df.select(rf_tile_sum(rf_local_equal('t', lit(3))).alias('threes')).first()['threes']
- self.assertEqual(result, 2)
-
- result_5 = df.select(rf_tile_sum(rf_local_equal('t', lit(5))).alias('fives')).first()['fives']
- self.assertEqual(result_5, 0)
-
- def test_rf_local_data_and_no_data(self):
- from pyspark.sql import Row
- from pyrasterframes.rf_types import Tile
- import numpy as np
- from numpy.testing import assert_equal
-
- t = Tile(np.array([[1, 3, 4], [5, 0, 3]]), CellType.uint8().with_no_data_value(5))
- #note the convert is due to issue #188
- df = self.spark.createDataFrame([Row(t=t)])\
- .withColumn('lnd', rf_convert_cell_type(rf_local_no_data('t'), 'uint8')) \
- .withColumn('ld', rf_convert_cell_type(rf_local_data('t'), 'uint8'))
-
- result = df.first()
- result_nd = result['lnd']
- assert_equal(result_nd.cells, t.cells.mask)
-
- result_d = result['ld']
- assert_equal(result_d.cells, np.invert(t.cells.mask))
diff --git a/pyrasterframes/src/main/python/tests/RasterSourceTest.py b/pyrasterframes/src/main/python/tests/RasterSourceTest.py
deleted file mode 100644
index c4c8e64f7..000000000
--- a/pyrasterframes/src/main/python/tests/RasterSourceTest.py
+++ /dev/null
@@ -1,213 +0,0 @@
-#
-# This software is licensed under the Apache 2 license, quoted below.
-#
-# Copyright 2019 Astraea, Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License"); you may not
-# use this file except in compliance with the License. You may obtain a copy of
-# the License at
-#
-# [http://www.apache.org/licenses/LICENSE-2.0]
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
-# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
-# License for the specific language governing permissions and limitations under
-# the License.
-#
-# SPDX-License-Identifier: Apache-2.0
-#
-
-from pyrasterframes.rasterfunctions import *
-from pyrasterframes.rf_types import *
-from pyspark.sql.functions import *
-import pandas as pd
-from shapely.geometry import Point
-import os.path
-from unittest import skip
-from . import TestEnvironment
-
-
-class RasterSourceTest(TestEnvironment):
-
- @staticmethod
- def path(scene, band):
- scene_dict = {
- 1: 'https://landsat-pds.s3.amazonaws.com/c1/L8/015/041/LC08_L1TP_015041_20190305_20190309_01_T1/LC08_L1TP_015041_20190305_20190309_01_T1_B{}.TIF',
- 2: 'https://landsat-pds.s3.amazonaws.com/c1/L8/015/042/LC08_L1TP_015042_20190305_20190309_01_T1/LC08_L1TP_015042_20190305_20190309_01_T1_B{}.TIF',
- 3: 'https://landsat-pds.s3.amazonaws.com/c1/L8/016/041/LC08_L1TP_016041_20190224_20190309_01_T1/LC08_L1TP_016041_20190224_20190309_01_T1_B{}.TIF',
- }
-
- assert band in range(1, 12)
- assert scene in scene_dict.keys()
- p = scene_dict[scene]
- return p.format(band)
-
- def path_pandas_df(self):
- return pd.DataFrame([
- {'b1': self.path(1, 1), 'b2': self.path(1, 2), 'b3': self.path(1, 3), 'geo': Point(1, 1)},
- {'b1': self.path(2, 1), 'b2': self.path(2, 2), 'b3': self.path(2, 3), 'geo': Point(2, 2)},
- {'b1': self.path(3, 1), 'b2': self.path(3, 2), 'b3': self.path(3, 3), 'geo': Point(3, 3)},
- ])
-
-
- def test_handle_lazy_eval(self):
- df = self.spark.read.raster(self.path(1, 1))
- ltdf = df.select('proj_raster')
- self.assertGreater(ltdf.count(), 0)
- self.assertIsNotNone(ltdf.first().proj_raster)
-
- tdf = df.select(rf_tile('proj_raster').alias('pr'))
- self.assertGreater(tdf.count(), 0)
- self.assertIsNotNone(tdf.first().pr)
-
- def test_strict_eval(self):
- df_lazy = self.spark.read.raster(self.img_uri, lazy_tiles=True)
- # when doing Show on a lazy tile we will see something like RasterRefTile(RasterRef(JVMGeoTiffRasterSource(...
- # use this trick to get the `show` string
- show_str_lazy = df_lazy.select('proj_raster')._jdf.showString(1, -1, False)
- self.assertTrue('RasterRef' in show_str_lazy)
-
- # again for strict
- df_strict = self.spark.read.raster(self.img_uri, lazy_tiles=False)
- show_str_strict = df_strict.select('proj_raster')._jdf.showString(1, -1, False)
- self.assertTrue('RasterRef' not in show_str_strict)
-
- def test_prt_functions(self):
- df = self.spark.read.raster(self.img_uri) \
- .withColumn('crs', rf_crs('proj_raster')) \
- .withColumn('ext', rf_extent('proj_raster')) \
- .withColumn('geom', rf_geometry('proj_raster'))
- df.select('crs', 'ext', 'geom').first()
-
- def test_list_of_str(self):
- # much the same as RasterSourceDataSourceSpec here; but using https PDS. Takes about 30s to run
-
- def l8path(b):
- assert b in range(1, 12)
- base = "https://s3-us-west-2.amazonaws.com/landsat-pds/c1/L8/199/026/LC08_L1TP_199026_20180919_20180928_01_T1/LC08_L1TP_199026_20180919_20180928_01_T1_B{}.TIF"
- return base.format(b)
-
- path_param = [l8path(b) for b in [1, 2, 3]]
- tile_size = 512
-
- df = self.spark.read.raster(
- path_param,
- tile_dimensions=(tile_size, tile_size),
- lazy_tiles=True,
- ).cache()
-
- print(df.take(3))
-
- # schema is tile_path and tile
- # df.printSchema()
- self.assertTrue(len(df.columns) == 2 and 'proj_raster_path' in df.columns and 'proj_raster' in df.columns)
-
- # the most common tile dimensions should be as passed to `options`, showing that options are correctly applied
- tile_size_df = df.select(rf_dimensions(df.proj_raster).rows.alias('r'), rf_dimensions(df.proj_raster).cols.alias('c')) \
- .groupby(['r', 'c']).count().toPandas()
- most_common_size = tile_size_df.loc[tile_size_df['count'].idxmax()]
- self.assertTrue(most_common_size.r == tile_size and most_common_size.c == tile_size)
-
- # all rows are from a single source URI
- path_count = df.groupby(df.proj_raster_path).count()
- print(path_count.collect())
- self.assertTrue(path_count.count() == 3)
-
- def test_list_of_list_of_str(self):
- lol = [
- [self.path(1, 1), self.path(1, 2)],
- [self.path(2, 1), self.path(2, 2)],
- [self.path(3, 1), self.path(3, 2)]
- ]
- df = self.spark.read.raster(lol)
- self.assertTrue(len(df.columns) == 4) # 2 cols of uris plus 2 cols of proj_rasters
- self.assertEqual(sorted(df.columns), sorted(['proj_raster_0_path', 'proj_raster_1_path',
- 'proj_raster_0', 'proj_raster_1']))
- uri_df = df.select('proj_raster_0_path', 'proj_raster_1_path').distinct()
-
- # check that various uri's are in the dataframe
- self.assertEqual(
- uri_df.filter(col('proj_raster_0_path') == lit(self.path(1, 1))).count(),
- 1)
-
- self.assertEqual(
- uri_df \
- .filter(col('proj_raster_0_path') == lit(self.path(1, 1))) \
- .filter(col('proj_raster_1_path') == lit(self.path(1, 2))) \
- .count(),
- 1)
-
- self.assertEqual(
- uri_df \
- .filter(col('proj_raster_0_path') == lit(self.path(3, 1))) \
- .filter(col('proj_raster_1_path') == lit(self.path(3, 2))) \
- .count(),
- 1)
-
- def test_schemeless_string(self):
- import os.path
- path = os.path.join(self.resource_dir, "L8-B8-Robinson-IL.tiff")
- self.assertTrue(not path.startswith('file://'))
- self.assertTrue(os.path.exists(path))
- df = self.spark.read.raster(path)
- self.assertTrue(df.count() > 0)
-
- def test_spark_df_source(self):
- catalog_columns = ['b1', 'b2', 'b3']
- catalog = self.spark.createDataFrame(self.path_pandas_df())
-
- df = self.spark.read.raster(
- catalog,
- tile_dimensions=(512, 512),
- catalog_col_names=catalog_columns,
- lazy_tiles=True # We'll get an OOM error if we try to read 9 scenes all at once!
- )
-
- self.assertTrue(len(df.columns) == 7) # three bands times {path, tile} plus geo
- self.assertTrue(df.select('b1_path').distinct().count() == 3) # as per scene_dict
- b1_paths_maybe = df.select('b1_path').distinct().collect()
- b1_paths = [self.path(s, 1) for s in [1, 2, 3]]
- self.assertTrue(all([row.b1_path in b1_paths for row in b1_paths_maybe]))
-
- def test_pandas_source(self):
-
- df = self.spark.read.raster(
- self.path_pandas_df(),
- catalog_col_names=['b1', 'b2', 'b3']
- )
- self.assertEqual(len(df.columns), 7) # three path cols, three tile cols, and geo
- self.assertTrue('geo' in df.columns)
- self.assertTrue(df.select('b1_path').distinct().count() == 3)
-
- def test_geopandas_source(self):
- from geopandas import GeoDataFrame
- # Same test as test_pandas_source with geopandas
- geo_df = GeoDataFrame(self.path_pandas_df(), crs={'init': 'EPSG:4326'}, geometry='geo')
- df = self.spark.read.raster(geo_df, ['b1', 'b2', 'b3'])
-
- self.assertEqual(len(df.columns), 7) # three path cols, three tile cols, and geo
- self.assertTrue('geo' in df.columns)
- self.assertTrue(df.select('b1_path').distinct().count() == 3)
-
- def test_csv_string(self):
-
- s = """metadata,b1,b2
- a,{},{}
- b,{},{}
- c,{},{}
- """.format(
- self.path(1, 1), self.path(1, 2),
- self.path(2, 1), self.path(2, 2),
- self.path(3, 1), self.path(3, 2),
- )
-
- df = self.spark.read.raster(s, ['b1', 'b2'])
- self.assertEqual(len(df.columns), 3 + 2) # number of columns in original DF plus cardinality of catalog_col_names
- self.assertTrue(len(df.take(1))) # non-empty check
-
- def test_catalog_named_arg(self):
- # through version 0.8.1 reading a catalog was via named argument only.
- df = self.spark.read.raster(catalog=self.path_pandas_df(), catalog_col_names=['b1', 'b2', 'b3'])
- self.assertEqual(len(df.columns), 7) # three path cols, three tile cols, and geo
- self.assertTrue(df.select('b1_path').distinct().count() == 3)
diff --git a/pyrasterframes/src/main/python/tests/VectorTypesTests.py b/pyrasterframes/src/main/python/tests/VectorTypesTests.py
deleted file mode 100644
index e31f26b43..000000000
--- a/pyrasterframes/src/main/python/tests/VectorTypesTests.py
+++ /dev/null
@@ -1,158 +0,0 @@
-#
-# This software is licensed under the Apache 2 license, quoted below.
-#
-# Copyright 2019 Astraea, Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License"); you may not
-# use this file except in compliance with the License. You may obtain a copy of
-# the License at
-#
-# [http://www.apache.org/licenses/LICENSE-2.0]
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
-# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
-# License for the specific language governing permissions and limitations under
-# the License.
-#
-# SPDX-License-Identifier: Apache-2.0
-#
-
-from pyrasterframes.rasterfunctions import *
-from pyspark.sql import Row
-from pyspark.sql.functions import *
-
-from . import TestEnvironment
-
-
-class VectorTypes(TestEnvironment):
-
- def setUp(self):
- self.create_layer()
- import pandas as pd
- self.pandas_df = pd.DataFrame({
- 'eye': ['a', 'b', 'c', 'd'],
- 'x': [0.0, 1.0, 2.0, 3.0],
- 'y': [-4.0, -3.0, -2.0, -1.0],
- })
- df = self.spark.createDataFrame(self.pandas_df)
- df = df.withColumn("point_geom",
- st_point(df.x, df.y)
- )
- self.df = df.withColumn("poly_geom", st_bufferPoint(df.point_geom, lit(1250.0)))
-
- def test_spatial_relations(self):
- from pyspark.sql.functions import udf, sum
- from geomesa_pyspark.types import PointUDT
- import shapely
- import numpy.testing
-
- # Use python shapely UDT in a UDF
- @udf("double")
- def area_fn(g):
- return g.area
-
- @udf("double")
- def length_fn(g):
- return g.length
-
- df = self.df.withColumn("poly_area", area_fn(self.df.poly_geom))
- df = df.withColumn("poly_len", length_fn(df.poly_geom))
-
- # Return UDT in a UDF!
- def some_point(g):
- return g.representative_point()
-
- some_point_udf = udf(some_point, PointUDT())
-
- df = df.withColumn("any_point", some_point_udf(df.poly_geom))
- # spark-side UDF/UDT are correct
- intersect_total = df.agg(sum(
- st_intersects(df.poly_geom, df.any_point).astype('double')
- ).alias('s')).collect()[0].s
- self.assertTrue(intersect_total == df.count())
-
- # Collect to python driver in shapely UDT
- pandas_df_out = df.toPandas()
-
- # Confirm we get a shapely type back from st_* function and UDF
- self.assertIsInstance(pandas_df_out.poly_geom.iloc[0], shapely.geometry.Polygon)
- self.assertIsInstance(pandas_df_out.any_point.iloc[0], shapely.geometry.Point)
-
- # And our spark-side manipulations were correct
- xs_correct = pandas_df_out.point_geom.apply(lambda g: g.coords[0][0]) == self.pandas_df.x
- self.assertTrue(all(xs_correct))
-
- centroid_ys = pandas_df_out.poly_geom.apply(lambda g:
- g.centroid.coords[0][1]).tolist()
- numpy.testing.assert_almost_equal(centroid_ys, self.pandas_df.y.tolist())
-
- # Including from UDF's
- numpy.testing.assert_almost_equal(
- pandas_df_out.poly_geom.apply(lambda g: g.area).values,
- pandas_df_out.poly_area.values
- )
- numpy.testing.assert_almost_equal(
- pandas_df_out.poly_geom.apply(lambda g: g.length).values,
- pandas_df_out.poly_len.values
- )
-
- def test_geometry_udf(self):
- from geomesa_pyspark.types import PolygonUDT
- # simple test that raster contents are not invalid
-
- # create a udf to buffer (the bounds) polygon
- def _buffer(g, d):
- return g.buffer(d)
-
- @udf("double")
- def area(g):
- return g.area
-
- buffer_udf = udf(_buffer, PolygonUDT())
-
- buf_cells = 10
- with_poly = self.rf.withColumn('poly', buffer_udf(self.rf.geometry, lit(-15 * buf_cells))) # cell res is 15x15
- area = with_poly.select(area('poly') < area('geometry'))
- area_result = area.collect()
- self.assertTrue(all([r[0] for r in area_result]))
-
- def test_rasterize(self):
- from geomesa_pyspark.types import PolygonUDT
-
- @udf(PolygonUDT())
- def buffer(g, d):
- return g.buffer(d)
-
- # start with known polygon, the tile extents, **negative buffered** by 10 cells
- buf_cells = 10
- with_poly = self.rf.withColumn('poly', buffer(self.rf.geometry, lit(-15 * buf_cells))) # cell res is 15x15
-
- # rasterize value 16 into buffer shape.
- cols = 194 # from dims of tile
- rows = 250 # from dims of tile
- with_raster = with_poly.withColumn('rasterized',
- rf_rasterize('poly', 'geometry', lit(16), lit(cols), lit(rows)))
- result = with_raster.select(rf_tile_sum(rf_local_equal_int(with_raster.rasterized, 16)),
- rf_tile_sum(with_raster.rasterized))
- #
- expected_burned_in_cells = (cols - 2 * buf_cells) * (rows - 2 * buf_cells)
- self.assertEqual(result.first()[0], float(expected_burned_in_cells))
- self.assertEqual(result.first()[1], 16. * expected_burned_in_cells)
-
- def test_parse_crs(self):
- df = self.spark.createDataFrame([Row(id=1)])
- self.assertEqual(df.select(rf_mk_crs('EPSG:4326')).count(), 1)
-
- def test_reproject(self):
- reprojected = self.rf.withColumn('reprojected',
- st_reproject('center', rf_mk_crs('EPSG:4326'), rf_mk_crs('EPSG:3857')))
- reprojected.show()
- self.assertEqual(reprojected.count(), 8)
-
- def test_geojson(self):
- import os
- sample = 'file://' + os.path.join(self.resource_dir, 'buildings.geojson')
- geo = self.spark.read.geojson(sample)
- geo.show()
- self.assertEqual(geo.select('geometry').count(), 8)
diff --git a/pyrasterframes/src/main/python/tests/__init__.py b/pyrasterframes/src/main/python/tests/__init__.py
deleted file mode 100644
index bea51f58b..000000000
--- a/pyrasterframes/src/main/python/tests/__init__.py
+++ /dev/null
@@ -1,96 +0,0 @@
-#
-# This software is licensed under the Apache 2 license, quoted below.
-#
-# Copyright 2019 Astraea, Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License"); you may not
-# use this file except in compliance with the License. You may obtain a copy of
-# the License at
-#
-# [http://www.apache.org/licenses/LICENSE-2.0]
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
-# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
-# License for the specific language governing permissions and limitations under
-# the License.
-#
-# SPDX-License-Identifier: Apache-2.0
-#
-
-import glob
-import os
-import unittest
-
-from pyrasterframes.utils import create_rf_spark_session
-
-import sys
-
-if sys.version_info[0] > 2:
- import builtins
-else:
- import __builtin__ as builtins
-
-app_name = 'pyrasterframes test suite'
-
-def resource_dir():
- def pdir(curr):
- return os.path.dirname(curr)
-
- here = os.path.dirname(os.path.realpath(__file__))
- scala_target = os.path.realpath(os.path.join(pdir(pdir(here)), 'scala-2.11'))
- rez_dir = os.path.realpath(os.path.join(scala_target, 'test-classes'))
- # If not running in build mode, try source dirs.
- if not os.path.exists(rez_dir):
- rez_dir = os.path.realpath(os.path.join(pdir(pdir(pdir(here))), 'test', 'resources'))
- return rez_dir
-
-
-def spark_test_session():
- spark = create_rf_spark_session(**{
- 'spark.ui.enabled': 'false',
- 'spark.app.name': app_name
- })
- spark.sparkContext.setLogLevel('ERROR')
-
- print("Spark Version: " + spark.version)
- print("Spark Config: " + str(spark.sparkContext._conf.getAll()))
-
- return spark
-
-
-class TestEnvironment(unittest.TestCase):
- """
- Base class for tests.
- """
-
- def rounded_compare(self, val1, val2):
- print('Comparing {} and {} using round()'.format(val1, val2))
- return builtins.round(val1) == builtins.round(val2)
-
- @classmethod
- def setUpClass(cls):
- # hard-coded relative path for resources
- cls.resource_dir = resource_dir()
-
- cls.spark = spark_test_session()
-
- cls.img_path = os.path.join(cls.resource_dir, 'L8-B8-Robinson-IL.tiff')
-
- cls.img_uri = 'file://' + cls.img_path
-
- @classmethod
- def l8band_uri(cls, band_index):
- return 'file://' + os.path.join(cls.resource_dir, 'L8-B{}-Elkton-VA.tiff'.format(band_index))
-
- def create_layer(self):
- from pyrasterframes.rasterfunctions import rf_convert_cell_type
- # load something into a rasterframe
- rf = self.spark.read.geotiff(self.img_uri) \
- .with_bounds() \
- .with_center()
-
- # convert the tile cell type to provide for other operations
- self.rf = rf.withColumn('tile2', rf_convert_cell_type('tile', 'float32')) \
- .drop('tile') \
- .withColumnRenamed('tile2', 'tile').as_layer()
diff --git a/pyrasterframes/src/main/python/tests/coverage-report.sh b/pyrasterframes/src/main/python/tests/coverage-report.sh
deleted file mode 100755
index 6b547e026..000000000
--- a/pyrasterframes/src/main/python/tests/coverage-report.sh
+++ /dev/null
@@ -1,9 +0,0 @@
-#!/usr/bin/env bash -e
-
-# If `coverage` tool isn't installed: `{pip|conda} install coverage`
-
-DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
-
-cd "$( dirname "${BASH_SOURCE[0]}" )"/..
-
-coverage run setup.py test && coverage html --omit='.eggs/*,setup.py' && open htmlcov/index.html
\ No newline at end of file
diff --git a/pyrasterframes/src/main/scala/org/locationtech/rasterframes/py/PyRFContext.scala b/pyrasterframes/src/main/scala/org/locationtech/rasterframes/py/PyRFContext.scala
index c31dccd38..2e5bdd8f0 100644
--- a/pyrasterframes/src/main/scala/org/locationtech/rasterframes/py/PyRFContext.scala
+++ b/pyrasterframes/src/main/scala/org/locationtech/rasterframes/py/PyRFContext.scala
@@ -21,20 +21,20 @@
package org.locationtech.rasterframes.py
import java.nio.ByteBuffer
-
import geotrellis.proj4.CRS
import geotrellis.raster.{CellType, MultibandTile}
-import geotrellis.spark.io._
-import geotrellis.spark.{ContextRDD, MultibandTileLayerRDD, SpaceTimeKey, SpatialKey, TileLayerMetadata}
+import geotrellis.spark._
+import geotrellis.layer._
import geotrellis.vector.Extent
import org.apache.spark.sql._
import org.locationtech.rasterframes
+import org.locationtech.rasterframes.util.{KryoSupport, ResampleMethod}
import org.locationtech.rasterframes.extensions.RasterJoin
import org.locationtech.rasterframes.model.LazyCRS
-import org.locationtech.rasterframes.ref.{GDALRasterSource, RasterRef, RasterSource}
-import org.locationtech.rasterframes.util.KryoSupport
-import org.locationtech.rasterframes.{RasterFunctions, _}
+import org.locationtech.rasterframes.ref.{GDALRasterSource, RFRasterSource, RasterRef}
+import org.locationtech.rasterframes._
import spray.json._
+import org.locationtech.rasterframes.util.JsonCodecs._
import scala.collection.JavaConverters._
@@ -109,20 +109,36 @@ class PyRFContext(implicit sparkSession: SparkSession) extends RasterFunctions
/**
* Left spatial join managing reprojection and merging of `other`
*/
- def rasterJoin(df: DataFrame, other: DataFrame): DataFrame = RasterJoin(df, other)
+ def rasterJoin(df: DataFrame, other: DataFrame, resamplingMethod: String): DataFrame = {
+ val m = resamplingMethod match {
+ case ResampleMethod(mm) => mm
+ case _ => throw new IllegalArgumentException(s"Incorrect resampling method passed: ${resamplingMethod}")
+ }
+ RasterJoin(df, other, m, None)
+ }
/**
* Left spatial join managing reprojection and merging of `other`; uses extent and CRS columns to determine if rows intersect
*/
- def rasterJoin(df: DataFrame, other: DataFrame, leftExtent: Column, leftCRS: Column, rightExtent: Column, rightCRS: Column): DataFrame =
- RasterJoin(df, other, leftExtent, leftCRS, rightExtent, rightCRS)
+ def rasterJoin(df: DataFrame, other: DataFrame, leftExtent: Column, leftCRS: Column, rightExtent: Column, rightCRS: Column, resamplingMethod: String): DataFrame = {
+ val m = resamplingMethod match {
+ case ResampleMethod(mm) => mm
+ case _ => throw new IllegalArgumentException(s"Incorrect resampling method passed: ${resamplingMethod}")
+ }
+
+ RasterJoin(df, other, leftExtent, leftCRS, rightExtent, rightCRS, m, None)
+ }
/**
* Left spatial join managing reprojection and merging of `other`; uses joinExprs to conduct initial join then extent and CRS columns to determine if rows intersect
*/
- def rasterJoin(df: DataFrame, other: DataFrame, joinExprs: Column, leftExtent: Column, leftCRS: Column, rightExtent: Column, rightCRS: Column): DataFrame =
- RasterJoin(df, other, joinExprs, leftExtent, leftCRS, rightExtent, rightCRS)
-
+ def rasterJoin(df: DataFrame, other: DataFrame, joinExprs: Column, leftExtent: Column, leftCRS: Column, rightExtent: Column, rightCRS: Column, resamplingMethod: String): DataFrame = {
+ val m = resamplingMethod match {
+ case ResampleMethod(mm) => mm
+ case _ => throw new IllegalArgumentException(s"Incorrect resampling method passed: ${resamplingMethod}")
+ }
+ RasterJoin(df, other, joinExprs, leftExtent, leftCRS, rightExtent, rightCRS, m, None)
+ }
/**
* Convenience functions for use in Python
@@ -191,6 +207,13 @@ class PyRFContext(implicit sparkSession: SparkSession) extends RasterFunctions
def rf_local_unequal_int(col: Column, scalar: Int): Column = rf_local_unequal[Int](col, scalar)
+ // other function support
+ /** py4j friendly version of this function */
+ def rf_agg_approx_quantiles(tile: Column, probabilities: java.util.List[Double], relativeError: Double): TypedColumn[Any, Seq[Double]] = {
+ import scala.collection.JavaConverters._
+ rf_agg_approx_quantiles(tile, probabilities.asScala, relativeError)
+ }
+
def _make_crs_literal(crsText: String): Column = {
rasterframes.encoders.serialized_literal[CRS](LazyCRS(crsText))
}
@@ -226,7 +249,7 @@ class PyRFContext(implicit sparkSession: SparkSession) extends RasterFunctions
type jDouble = java.lang.Double
// NB: Tightly coupled to the `RFContext.resolve_raster_ref` method in `pyrasterframes.rf_context`. */
def _resolveRasterRef(srcBin: Array[Byte], bandIndex: jInt, xmin: jDouble, ymin: jDouble, xmax: jDouble, ymax: jDouble): AnyRef = {
- val src = KryoSupport.deserialize[RasterSource](ByteBuffer.wrap(srcBin))
+ val src = KryoSupport.deserialize[RFRasterSource](ByteBuffer.wrap(srcBin))
val extent = Extent(xmin, ymin, xmax, ymax)
RasterRef(src, bandIndex, Some(extent), None)
}
@@ -240,4 +263,7 @@ class PyRFContext(implicit sparkSession: SparkSession) extends RasterFunctions
import rasterframes.util.DFWithPrettyPrint
df.toHTML(numRows, truncate, renderTiles = true)
}
+
+ def _reprojectExtent(extent: Extent, srcCRS: String, destCRS: String): Extent =
+ extent.reproject(LazyCRS(srcCRS), LazyCRS(destCRS))
}
diff --git a/pyrasterframes/src/test/resources/L8-B4-Elkton-VA-4326.tiff b/pyrasterframes/src/test/resources/L8-B4-Elkton-VA-4326.tiff
new file mode 100644
index 000000000..2bc57e255
Binary files /dev/null and b/pyrasterframes/src/test/resources/L8-B4-Elkton-VA-4326.tiff differ
diff --git a/pyrasterframes/src/test/resources/L8-B4_3_2-Elkton-VA.tiff b/pyrasterframes/src/test/resources/L8-B4_3_2-Elkton-VA.tiff
new file mode 100644
index 000000000..c351f5887
Binary files /dev/null and b/pyrasterframes/src/test/resources/L8-B4_3_2-Elkton-VA.tiff differ
diff --git a/pyrasterframes/src/main/python/LICENSE.txt b/python/LICENSE.txt
similarity index 100%
rename from pyrasterframes/src/main/python/LICENSE.txt
rename to python/LICENSE.txt
diff --git a/pyrasterframes/src/main/python/README.md b/python/README.md
similarity index 79%
rename from pyrasterframes/src/main/python/README.md
rename to python/README.md
index 00a915387..e71f564d1 100644
--- a/pyrasterframes/src/main/python/README.md
+++ b/python/README.md
@@ -32,12 +32,21 @@ df.select(rf_local_add(df.tile, lit(3))).show(5, False)
Reach out to us on [gitter][gitter]!
-Issue tracking is through [github](https://github.com/locationtech/rasterframes/issues).
+Issue tracking is through [github](https://github.com/locationtech/rasterframes/issues).
## Contributing
Community contributions are always welcome. To get started, please review our [contribution guidelines](https://github.com/locationtech/rasterframes/blob/develop/CONTRIBUTING.md), [code of conduct](https://github.com/locationtech/rasterframes/blob/develop/CODE_OF_CONDUCT.md), and [developer's guide](../../../README.md). Reach out to us on [gitter][gitter] so the community can help you get started!
+## Development environment setup
+For best results, we suggest using `conda` and the `conda-forge` channel to install the compiled dependencies before installing the packages in `setup.py`. Assuming you're in the same directory as this file:
+
+ conda create -n rasterframes python==3.7
+ conda install --file ./requirements-condaforge.txt
+
+Then you can install the source dependencies:
+
+ pip install -e .
[gitter]: https://gitter.im/locationtech/rasterframes
diff --git a/pyrasterframes/src/main/python/geomesa_pyspark/__init__.py b/python/docs/__init__.py
similarity index 100%
rename from pyrasterframes/src/main/python/geomesa_pyspark/__init__.py
rename to python/docs/__init__.py
diff --git a/pyrasterframes/src/main/python/docs/aggregation.pymd b/python/docs/aggregation.pymd
similarity index 94%
rename from pyrasterframes/src/main/python/docs/aggregation.pymd
rename to python/docs/aggregation.pymd
index 2243e5b37..e32df5393 100644
--- a/pyrasterframes/src/main/python/docs/aggregation.pymd
+++ b/python/docs/aggregation.pymd
@@ -2,7 +2,6 @@
```python, setup, echo=False
from pyrasterframes import rf_ipython
-from docs import *
from pyrasterframes.utils import create_rf_spark_session
from pyrasterframes.rasterfunctions import *
from pyspark.sql import *
@@ -71,7 +70,7 @@ rf.agg(rf_agg_local_mean('tile')) \
We can also count the total number of data and NoData cells over all the _tiles_ in a DataFrame using @ref:[`rf_agg_data_cells`](reference.md#rf-agg-data-cells) and @ref:[`rf_agg_no_data_cells`](reference.md#rf-agg-no-data-cells). There are ~3.8 million data cells and ~1.9 million NoData cells in this DataFrame. See the section on @ref:["NoData" handling](nodata-handling.md) for additional discussion on handling missing data.
```python, cell_counts
-rf = spark.read.raster('https://s22s-test-geotiffs.s3.amazonaws.com/MCD43A4.006/11/05/2018233/MCD43A4.A2018233.h11v05.006.2018242035530_B02.TIF')
+rf = spark.read.raster('https://rasterframes.s3.amazonaws.com/samples/MCD43A4.006/11/05/2018233/MCD43A4.A2018233.h11v05.006.2018242035530_B02.TIF')
stats = rf.agg(rf_agg_data_cells('proj_raster'), rf_agg_no_data_cells('proj_raster'))
stats
```
@@ -83,7 +82,7 @@ The statistical summary functions return a summary of cell values: number of dat
The @ref:[`rf_tile_stats`](reference.md#rf-tile-stats) function computes summary statistics separately for each row in a _tile_ column as shown below.
```python, tile_stats
-rf = spark.read.raster('https://s22s-test-geotiffs.s3.amazonaws.com/luray_snp/B02.tif')
+rf = spark.read.raster('https://rasterframes.s3.amazonaws.com/samples/luray_snp/B02.tif')
stats = rf.select(rf_tile_stats('proj_raster').alias('stats'))
stats.printSchema()
@@ -98,7 +97,7 @@ The @ref:[`rf_agg_stats`](reference.md#rf-agg-stats) function aggregates over al
```python, agg_stats
stats = rf.agg(rf_agg_stats('proj_raster').alias('stats')) \
.select('stats.min', 'stats.max', 'stats.mean', 'stats.variance')
-stats
+stats
```
The @ref:[`rf_agg_local_stats`](reference.md#rf-agg-local-stats) function computes the element-wise local aggregate statistical summary as shown below. The DataFrame used in the previous two code blocks has unequal _tile_ dimensions, so a different DataFrame is used in this code block to avoid a runtime error.
@@ -109,7 +108,7 @@ rf = spark.createDataFrame([
Row(id=3, tile=t1 * 3),
Row(id=5, tile=t1 * 5)
]).agg(rf_agg_local_stats('tile').alias('stats'))
-
+
agg_local_stats = rf.select('stats.min', 'stats.max', 'stats.mean', 'stats.variance').collect()
for r in agg_local_stats:
@@ -125,7 +124,7 @@ The @ref:[`rf_tile_histogram`](reference.md#rf-tile-histogram) function computes
```python, tile_histogram
import matplotlib.pyplot as plt
-rf = spark.read.raster('https://s22s-test-geotiffs.s3.amazonaws.com/MCD43A4.006/11/05/2018233/MCD43A4.A2018233.h11v05.006.2018242035530_B02.TIF')
+rf = spark.read.raster('https://rasterframes.s3.amazonaws.com/samples/MCD43A4.006/11/05/2018233/MCD43A4.A2018233.h11v05.006.2018242035530_B02.TIF')
hist_df = rf.select(rf_tile_histogram('proj_raster')['bins'].alias('bins'))
hist_df.printSchema()
diff --git a/python/docs/build_docs.py b/python/docs/build_docs.py
new file mode 100644
index 000000000..ff88ec651
--- /dev/null
+++ b/python/docs/build_docs.py
@@ -0,0 +1,151 @@
+#
+# This software is licensed under the Apache 2 license, quoted below.
+#
+# Copyright 2019 Astraea, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may not
+# use this file except in compliance with the License. You may obtain a copy of
+# the License at
+#
+# [http://www.apache.org/licenses/LICENSE-2.0]
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations under
+# the License.
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+
+
+import traceback
+from enum import Enum
+from glob import glob
+from os import path
+from typing import List
+
+import pweave
+import typer
+from pweave import PwebPandocFormatter
+
+
+# Setuptools/easy_install doesn't properly set the execute bit on the Spark scripts,
+# So this preemptively attempts to do it.
+def _chmodit():
+ try:
+ import os
+ from importlib.util import find_spec
+
+ module_home = find_spec("pyspark").origin
+ print(module_home)
+ bin_dir = os.path.join(os.path.dirname(module_home), "bin")
+ for filename in os.listdir(bin_dir):
+ try:
+ os.chmod(os.path.join(bin_dir, filename), mode=0o555, follow_symlinks=True)
+ except OSError:
+ pass
+ except ImportError:
+ pass
+
+
+_chmodit()
+
+
+class PegdownMarkdownFormatter(PwebPandocFormatter):
+ def __init__(self, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+
+ # Pegdown doesn't support the width and label options.
+ def make_figure_string(self, figname, width, label, caption=""):
+ return "" % (caption, figname)
+
+
+app = typer.Typer()
+
+
+def _dest_file(src_file, ext):
+ return path.splitext(src_file)[0] + ext
+
+
+def _divided(msg):
+ divider = "-" * 50
+ return divider + "\n" + msg + "\n" + divider
+
+
+def _get_files():
+ here = path.abspath(path.dirname(__file__))
+ return list(filter(lambda x: not path.basename(x)[:1] == "_", glob(path.join(here, "*.pymd"))))
+
+
+class Format(str, Enum):
+ html = "html"
+ markdown = "markdown"
+ notebook = "notebook"
+ pandoc2html = "pandoc2html"
+
+
+@app.command()
+def pweave_docs(
+ files: List[str] = typer.Option(
+ _get_files(), help="Specific files to pweave. Defaults to all in `docs` directory."
+ ),
+ format: Format = typer.Option(
+ Format.markdown, help="Output format type. Defaults to `markdown`"
+ ),
+ quick: bool = typer.Option(
+ False,
+ help="Check to see if the source file is newer than existing output before building. Defaults to `False`.",
+ ),
+):
+
+ """Pweave PyRasterFrames documentation scripts"""
+
+ ext = ".md"
+ bad_words = ["Error"]
+ pweave.rcParams["chunk"]["defaultoptions"].update({"wrap": False, "dpi": 175})
+
+ if format == Format.markdown:
+ pweave.PwebFormats.formats["markdown"] = {
+ "class": PegdownMarkdownFormatter,
+ "description": "Pegdown compatible markdown",
+ }
+ elif format == Format.notebook:
+ # Just convert to an unevaluated notebook.
+ pweave.rcParams["chunk"]["defaultoptions"].update({"evaluate": False})
+ ext = ".ipynb"
+ elif format == Format.html:
+ # `html` doesn't do quite what one expects... only replaces code blocks, leaving markdown in place
+ format = Format.pandoc2html
+
+ for file in sorted(files, reverse=False):
+ name = path.splitext(path.basename(file))[0]
+ dest = _dest_file(file, ext)
+
+ if (not quick) or (not path.exists(dest)) or (path.getmtime(dest) < path.getmtime(file)):
+ print(_divided("Running %s" % name))
+ try:
+ pweave.weave(file=str(file), doctype=format)
+ if format == Format.markdown:
+ if not path.exists(dest):
+ raise FileNotFoundError(
+ "Markdown file '%s' didn't get created as expected" % dest
+ )
+ with open(dest, "r") as result:
+ for (n, line) in enumerate(result):
+ for word in bad_words:
+ if word in line:
+ raise ChildProcessError(
+ "Error detected on line %s in %s:\n%s" % (n + 1, dest, line)
+ )
+
+ except Exception:
+ print(_divided("%s Failed:" % file))
+ print(traceback.format_exc())
+ # raise typer.Exit(code=1)
+ else:
+ print(_divided("Skipping %s" % name))
+
+
+if __name__ == "__main__":
+ app()
diff --git a/pyrasterframes/src/main/python/docs/description.pymd b/python/docs/description.pymd
similarity index 100%
rename from pyrasterframes/src/main/python/docs/description.pymd
rename to python/docs/description.pymd
diff --git a/pyrasterframes/src/main/python/docs/getting-started.pymd b/python/docs/getting-started.pymd
similarity index 97%
rename from pyrasterframes/src/main/python/docs/getting-started.pymd
rename to python/docs/getting-started.pymd
index 748070eee..2ae114c2b 100644
--- a/pyrasterframes/src/main/python/docs/getting-started.pymd
+++ b/python/docs/getting-started.pymd
@@ -116,8 +116,8 @@ If you would like to use RasterFrames in Scala, you'll need to add the following
```scala
resolvers ++= Seq(
- "locationtech-releases" at "https://repo.locationtech.org/content/groups/releases",
- "Azavea Public Builds" at "https://dl.bintray.com/azavea/geotrellis"
+ "Azavea Public Builds" at "https://dl.bintray.com/azavea/geotrellis",
+ "locationtech-releases" at "https://repo.locationtech.org/content/groups/releases"
)
libraryDependencies ++= Seq(
"org.locationtech.rasterframes" %% "rasterframes" % ${VERSION},
@@ -125,7 +125,9 @@ libraryDependencies ++= Seq(
// This is optional. Provides access to AWS PDS catalogs.
"org.locationtech.rasterframes" %% "rasterframes-experimental" % ${VERSION}
)
-```
+```
+
+RasterFrames is compatible with Spark 2.4.x.
## Installing GDAL Support
diff --git a/python/docs/ipython.pymd b/python/docs/ipython.pymd
new file mode 100644
index 000000000..263a0c44f
--- /dev/null
+++ b/python/docs/ipython.pymd
@@ -0,0 +1,94 @@
+# IPython/Jupyter Extensions
+
+The `pyrasterframes.rf_ipython` module injects a number of visualization extensions into the IPython environment, enhancing visualization of `DataFrame`s and `Tile`s.
+
+By default, the last expression's result in a IPython cell is passed to the `IPython.display.display` function. This function in turn looks for a [`DisplayFormatter`](https://ipython.readthedocs.io/en/stable/api/generated/IPython.core.formatters.html#IPython.core.formatters.DisplayFormatter) associated with the type, which in turn converts the instance to a display-appropriate representation, based on MIME type. For example, each `DisplayFormatter` may `plain/text` version for the IPython shell, and a `text/html` version for a Jupyter Notebook.
+
+This will be our setup for the following examples:
+
+```python setup
+from pyrasterframes import *
+from pyrasterframes.rasterfunctions import *
+from pyrasterframes.utils import create_rf_spark_session
+import pyrasterframes.rf_ipython
+from IPython.display import display
+import os.path
+spark = create_rf_spark_session()
+def scene(band):
+ b = str(band).zfill(2) # converts int 2 to '02'
+ return 'https://modis-pds.s3.amazonaws.com/MCD43A4.006/11/08/2019059/' \
+ 'MCD43A4.A2019059.h11v08.006.2019072203257_B{}.TIF'.format(b)
+rf = spark.read.raster(scene(2), tile_dimensions=(256, 256))
+```
+
+## Tile Samples
+
+We have some convenience methods to quickly visualize tiles (see discussion of the RasterFrame @ref:[schema](raster-read.md#single-raster) for orientation to the concept) when inspecting a subset of the data in a Notebook.
+
+In an IPython or Jupyter interpreter, a `Tile` object will be displayed as an image with limited metadata.
+
+```python, sample_tile
+sample_tile = rf.select(rf_tile('proj_raster').alias('tile')).first()['tile']
+sample_tile # or `display(sample_tile)`
+```
+
+## DataFrame Samples
+
+Within an IPython or Jupyter interpreter, a Spark and Pandas DataFrames containing a column of _tiles_ will be rendered as the samples discussed above. Simply import the `rf_ipython` submodule to enable enhanced HTML rendering of these DataFrame types.
+
+```python display_samples
+rf # or `display(rf)`, or `rf.display()`
+```
+
+### Changing Number of Rows
+
+By default the RasterFrame sample display renders 5 rows. Because the `IPython.display.display` function doesn't pass parameters to the underlying rendering functions, we have to provide a different means of passing parameters to the rendering code. Pandas approach to this is to use global settings via `set_option`/`get_option`. We take a more functional approach and have the user invoke an explicit `display` method:
+
+```python custom_display, evaluate=False
+rf.display(num_rows=1, truncate=True)
+```
+
+```python custom_display_mime, echo=False
+rf.display(num_rows=1, truncate=True, mimetype='text/markdown')
+```
+
+### Pandas
+
+There is similar rendering support injected into the Pandas by the `rf_ipython` module, for Pandas Dataframes having Tiles in them:
+
+```python pandas_dataframe
+# Limit copy of data from Spark to a few tiles.
+pandas_df = rf.select(rf_tile('proj_raster'), rf_extent('proj_raster')).limit(4).toPandas()
+pandas_df # or `display(pandas_df)`
+```
+
+## Sample Colorization
+
+RasterFrames uses the "Viridis" color ramp as the default color profile for tile column. There are other options for reasoning about how color should be applied in the results.
+
+### Color Composite
+
+As shown in @ref:[Writing Raster Data section](raster-write.md) section, composites can be constructed for visualization:
+
+```python, png_color_composite
+from IPython.display import Image # For telling IPython how to interpret the PNG byte array
+# Select red, green, and blue, respectively
+three_band_rf = spark.read.raster(source=[[scene(1), scene(4), scene(3)]])
+composite_rf = three_band_rf.withColumn('png',
+ rf_render_png('proj_raster_0', 'proj_raster_1', 'proj_raster_2'))
+png_bytes = composite_rf.select('png').first()['png']
+Image(png_bytes)
+```
+
+```python, png_render, echo=False
+from IPython.display import display_markdown
+display_markdown(pyrasterframes.rf_ipython.binary_to_html(png_bytes), raw=True)
+```
+
+### Custom Color Ramp
+
+You can also apply a different color ramp to a single-channel Tile using the @ref[`rf_render_color_ramp_png`](reference.md#rf-render-color-ramp-png) function. See the function documentation for information about the available color maps.
+
+```python, color_map
+rf.select(rf_render_color_ramp_png('proj_raster', 'Magma'))
+```
diff --git a/pyrasterframes/src/main/python/docs/languages.pymd b/python/docs/languages.pymd
similarity index 96%
rename from pyrasterframes/src/main/python/docs/languages.pymd
rename to python/docs/languages.pymd
index b4d189fbe..fca39f5f2 100644
--- a/pyrasterframes/src/main/python/docs/languages.pymd
+++ b/python/docs/languages.pymd
@@ -1,6 +1,6 @@
# Scala and SQL
-One of the great powers of RasterFrames is the ability to express computation in multiple programming languages. The content in this manual focuses on Python because it is the most commonly used language in data science and GIS analytics. However, Scala (the implementation language of RasterFrames) and SQL (commonly used in many domains) are also fully supported. Examples in Python can be mechanically translated into the other two languages without much difficulty once the naming conventions are understood.
+One of the great powers of RasterFrames is the ability to express computation in multiple programming languages. The content in this manual focuses on Python because it is the most commonly used language in data science and GIS analytics. However, Scala (the implementation language of RasterFrames) and SQL (commonly used in many domains) are also fully supported. Examples in Python can be mechanically translated into the other two languages without much difficulty once the naming conventions are understood.
In the sections below we will show the same example program in each language. To do so we will compute the average NDVI per month for a single _tile_ in Tanzania.
@@ -33,11 +33,11 @@ red_nir_monthly_2017 = modis \
col('B02').alias('nir')
) \
.where(
- (year('acquisition_date') == 2017) &
- (dayofmonth('acquisition_date') == 15) &
+ (year('acquisition_date') == 2017) &
+ (dayofmonth('acquisition_date') == 15) &
(col('granule_id') == 'h21v09')
)
-red_nir_monthly_2017.printSchema()
+red_nir_monthly_2017.printSchema()
```
### Step 3: Read tiles
@@ -125,7 +125,7 @@ grouped
The latest Scala API documentation is available here:
-* [Scala API Documentation](https://rasterframes.io/latest/api/index.html)
+* [Scala API Documentation](https://rasterframes.io/latest/api/index.html)
### Step 1: Load the catalog
@@ -178,6 +178,6 @@ val result = red_nir_tiles_monthly_2017
.agg(rf_agg_stats(rf_normalized_difference($"nir", $"red")) as "ndvi_stats")
.orderBy("month")
.select("month", "ndvi_stats.*")
-
-result.show()
+
+result.show()
```
diff --git a/pyrasterframes/src/main/python/docs/local-algebra.pymd b/python/docs/local-algebra.pymd
similarity index 97%
rename from pyrasterframes/src/main/python/docs/local-algebra.pymd
rename to python/docs/local-algebra.pymd
index fc83ae2d2..0e1f13d06 100644
--- a/pyrasterframes/src/main/python/docs/local-algebra.pymd
+++ b/python/docs/local-algebra.pymd
@@ -25,7 +25,7 @@ Here is an example of computing the Normalized Differential Vegetation Index (ND
> NDVI is often used worldwide to monitor drought, monitor and predict agricultural production, assist in predicting hazardous fire zones, and map desert encroachment. The NDVI is preferred for global vegetation monitoring because it helps to compensate for changing illumination conditions, surface slope, aspect, and other extraneous factors (Lillesand. _Remote sensing and image interpretation_. 2004)
-We will apply the @ref:[catalog pattern](raster-catalogs.md) for defining the data we wish to process. To compute NDVI we need to compute local algebra on the *red* and *near infrared* (nir) bands:
+We will apply the @ref:[catalog pattern](raster-catalogs.md) for defining the data we wish to process. To compute NDVI we need to compute local algebra on the *red* and *near infrared* (nir) bands:
nir - red
NDVI = ---------
@@ -35,7 +35,7 @@ This form of `(x - y) / (x + y)` is common in remote sensing and is called a nor
```python, read_rasters
from pyspark.sql import Row
-uri_pattern = 'https://s22s-test-geotiffs.s3.amazonaws.com/luray_snp/B0{}.tif'
+uri_pattern = 'https://rasterframes.s3.amazonaws.com/samples/luray_snp/B0{}.tif'
catalog_df = spark.createDataFrame([
Row(red=uri_pattern.format(4), nir=uri_pattern.format(8))
])
diff --git a/python/docs/masking.pymd b/python/docs/masking.pymd
new file mode 100644
index 000000000..0949fc315
--- /dev/null
+++ b/python/docs/masking.pymd
@@ -0,0 +1,203 @@
+# Masking
+
+```python setup, echo=False
+import pyrasterframes
+from pyrasterframes.utils import create_rf_spark_session
+from pyrasterframes.rasterfunctions import *
+import pyrasterframes.rf_ipython
+from IPython.display import display
+import pandas as pd
+import numpy as np
+from pyrasterframes.rf_types import Tile
+
+spark = create_rf_spark_session()
+```
+
+Masking is a common operation in raster processing. It is setting certain cells to the @ref:[NoData value](nodata-handling.md). This is usually done to remove low-quality observations from the raster processing. Another related use case is to @ref:["clip"](masking.md#clipping) a raster to a given polygon.
+
+In this section we will demonstrate two common schemes for masking. In Sentinel 2, there is a separate classification raster that defines low quality areas. In Landsat 8, several quality factors are measured and the indications are packed into a single integer, which we have to unpack.
+
+## Masking Sentinel 2
+
+Let's demonstrate masking with a pair of bands of Sentinel-2 data. The measurement bands we will use, blue and green, have no defined NoData. They share quality information from a separate file called the scene classification (SCL), which delineates areas of missing data and probable clouds. For more information on this, see the [Sentinel-2 algorithm overview](https://earth.esa.int/web/sentinel/technical-guides/sentinel-2-msi/level-2a/algorithm). Figure 3 tells us how to interpret the scene classification. For this example, we will exclude NoData, defective pixels, probable clouds, and cirrus clouds: values 0, 1, 8, 9, and 10.
+
+
+
+Credit: [Sentinel-2 algorithm overview](https://earth.esa.int/web/sentinel/technical-guides/sentinel-2-msi/level-2a/algorithm)
+
+The first step is to create a catalog with our band of interest and the SCL band. We read the data from the catalog, so all _tiles_ are aligned across rows.
+
+```python, blue_scl_cat
+from pyspark.sql import Row
+
+blue_uri = 'https://rasterframes.s3.amazonaws.com/samples/luray_snp/B02.tif'
+green_uri = 'https://rasterframes.s3.amazonaws.com/samples/luray_snp/B03.tif'
+scl_uri = 'https://rasterframes.s3.amazonaws.com/samples/luray_snp/SCL.tif'
+cat = spark.createDataFrame([Row(blue=blue_uri, green=green_uri, scl=scl_uri),])
+unmasked = spark.read.raster(cat, catalog_col_names=['blue', 'green', 'scl'])
+unmasked.printSchema()
+```
+
+```python, show_cell_types
+unmasked.select(rf_cell_type('blue'), rf_cell_type('scl')).distinct()
+```
+
+### Define CellType for Masked Tile
+
+Because there is not a NoData already defined for the blue band, we must choose one. If we try to apply a masking function to a tile whose cell type has no NoData defined, an error will be thrown.
+
+In this particular example, the minimum value of all cells in all tiles in the column is greater than zero, so we can use 0 as the NoData value. We will construct a new `CellType` object to represent this.
+
+```python, pick_nd
+blue_min = unmasked.agg(rf_agg_stats('blue').min.alias('blue_min'))
+print('Nonzero minimum value in the blue band:', blue_min.first())
+
+blue_ct = unmasked.select(rf_cell_type('blue')).distinct().first()[0][0]
+masked_blue_ct = CellType(blue_ct).with_no_data_value(0)
+masked_blue_ct.cell_type_name
+```
+
+We next convert the blue band to this cell type.
+
+```python, convert_blue
+converted = unmasked.select('scl', 'green', rf_convert_cell_type('blue', masked_blue_ct).alias('blue'))
+```
+
+### Apply Mask from Quality Band
+
+Now we set cells of our `blue` column to NoData for all locations where the `scl` tile is in our set of undesirable values. This is the actual _masking_ operation.
+
+```python, apply_mask_blue
+from pyspark.sql.functions import lit
+
+masked = converted.withColumn('blue_masked', rf_mask_by_values('blue', 'scl', [0, 1, 8, 9, 10]))
+masked
+```
+
+We can verify that the number of NoData cells in the resulting `blue_masked` column matches the total of the boolean `mask` _tile_ to ensure our logic is correct.
+
+```python, show_masked_counts
+masked.select(rf_no_data_cells('blue_masked'), rf_tile_sum(rf_local_is_in('scl', [0, 1, 8, 9, 10])))
+```
+
+It's also nice to view a sample. The white regions are areas of NoData.
+
+```python, display_blu, caption='Blue band masked against selected SCL values'
+sample = masked.orderBy(-rf_no_data_cells('blue_masked')).select(rf_tile('blue_masked'), rf_tile('scl')).first()
+display(sample[0])
+```
+
+And the original SCL data. The bright yellow is a cloudy region in the original image.
+
+```python, display_scl, caption='SCL tile for above'
+display(sample[1])
+```
+
+### Transferring Mask
+
+We can now apply the same mask from the blue column to the green column. Note here we have supressed the step of explicitly checking what a "safe" NoData value for the green band should be.
+
+```python, mask_green
+masked.withColumn('green_masked', rf_mask(rf_convert_cell_type('green', masked_blue_ct), 'blue_masked')) \
+ .orderBy(-rf_no_data_cells('blue_masked'))
+```
+
+## Masking Landsat 8
+
+
+We will work with the Landsat scene [here](https://landsat-pds.s3.us-west-2.amazonaws.com/c1/L8/153/075/LC08_L1TP_153075_20190718_20190731_01_T1/index.html). For simplicity, we will just use two of the seven 30m bands. The quality mask for all bands is all contained in the `BQA` band.
+
+
+```python, build_l8_df
+base_url = 'https://landsat-pds.s3.us-west-2.amazonaws.com/c1/L8/153/075/LC08_L1TP_153075_20190718_20190731_01_T1/LC08_L1TP_153075_20190718_20190731_01_T1_'
+data4 = base_url + 'B4.TIF'
+data2 = base_url + 'B2.TIF'
+mask = base_url + 'BQA.TIF'
+l8_df = spark.read.raster([[data4, data2, mask]]) \
+ .withColumnRenamed('proj_raster_0', 'data') \
+ .withColumnRenamed('proj_raster_1', 'data2') \
+ .withColumnRenamed('proj_raster_2', 'mask')
+```
+
+Masking is described [on the Landsat Missions page](https://www.usgs.gov/land-resources/nli/landsat/landsat-collection-1-level-1-quality-assessment-band). It is pretty dense. Focus for this data set is the Collection 1 Level-1 for Landsat 8.
+
+There are several inter-related factors to consider. In this exercise we will mask away the following.
+
+ * Designated Fill = yes
+ * Cloud = yes
+ * Cloud Shadow Confidence = Medium or High
+ * Cirrus Confidence = Medium or High
+
+Note that you should consider your application and do your own exploratory analysis to determine the most appropriate mask!
+
+According to the information on the Landsat site this translates to masking by bit values in the BQA according to the following table.
+
+| Description | Value | Bits | Bit values |
+|-------------------- |---------- |------- |---------------- |
+| Designated fill | yes | 0 | 1 |
+| Cloud | yes | 4 | 1 |
+| Cloud shadow conf. | med / hi | 7-8 | 10, 11 (2, 3) |
+| Cirrus conf. | med / hi | 11-12 | 10, 11 (2, 3) |
+
+
+In this case, we will use the value of 0 as the NoData in the band data. Inspecting the associated [MTL txt file](https://landsat-pds.s3.us-west-2.amazonaws.com/c1/L8/153/075/LC08_L1TP_153075_20190718_20190731_01_T1/LC08_L1TP_153075_20190718_20190731_01_T1_MTL.txt) By inspection, we can discover that the minimum value in the band will be 1, thus allowing our use of 0 for NoData.
+
+The code chunk below works through each of the rows in the table above. The first expression sets the cell type to have the selected NoData. The @ref:[`rf_mask_by_bit`](reference.md#rf-mask-by-bit) and @ref:[`rf_mask_by_bits`](reference.md#rf-mask-by-bits) functions extract the selected bit or bits from the `mask` cells and compare them to the provided values.
+
+```python, build_l8_mask
+l8_df = l8_df.withColumn('data_masked', # set to cell type that has a nodata
+ rf_convert_cell_type('data', CellType.uint16())) \
+ .withColumn('data_masked', # fill yes
+ rf_mask_by_bit('data_masked', 'mask', 0, 1)) \
+ .withColumn('data_masked', # cloud yes
+ rf_mask_by_bit('data_masked', 'mask', 4, 1)) \
+ .withColumn('data_masked', # cloud shadow conf is medium or high
+ rf_mask_by_bits('data_masked', 'mask', 7, 2, [2, 3])) \
+ .withColumn('data_masked', # cloud shadow conf is medium or high
+ rf_mask_by_bits('data_masked', 'mask', 11, 2, [2, 3])) \
+ .withColumn('data2', # mask other data col against the other band
+ rf_mask(rf_convert_cell_type('data2',CellType.uint16()), 'data_masked')) \
+ .filter(rf_data_cells('data_masked') > 0) # remove any entirely ND rows
+
+# Inspect a sample
+l8_df.select('data', 'mask', 'data_masked', 'data2', rf_extent('data_masked')) \
+ .filter(rf_data_cells('data_masked') > 32000)
+```
+
+
+## Clipping
+
+Clipping is the use of a polygon to determine the areas to mask in a raster. Typically the areas inside a polygon are retained and the cells outside are set to NoData. Given a geometry column on our DataFrame, we have to carry out three basic steps. First we have to ensure the vector geometry is correctly projected to the same @ref:[CRS](concepts.md#coordinate-reference-system-crs) as the raster. We'll continue with our Sentinel 2 example, creating a simple polygon. Buffering a point will create an approximate circle.
+
+
+```python, reproject_geom
+to_rasterize = masked.withColumn('geom_4326',
+ st_bufferPoint(
+ st_point(lit(-78.0783132), lit(38.3184340)),
+ lit(15000))) \
+ .withColumn('geom_native', st_reproject('geom_4326', rf_mk_crs('epsg:4326'), rf_crs('blue_masked')))
+```
+
+Second, we will rasterize the geometry, or burn-in the geometry into the same grid as the raster.
+
+```python, rasterize
+to_clip = to_rasterize.withColumn('clip_raster',
+ rf_rasterize('geom_native', rf_geometry('blue_masked'), lit(1), rf_dimensions('blue_masked').cols, rf_dimensions('blue_masked').rows))
+
+# visualize some of the edges of our circle
+to_clip.select('blue_masked', 'clip_raster') \
+ .filter(rf_data_cells('clip_raster') > 20) \
+ .orderBy(rf_data_cells('clip_raster'))
+```
+
+Finally, we create a new _tile_ column with the blue band clipped to our circle. Again we will use the `rf_mask` function to pass the NoData regions along from the rasterized geometry.
+
+```python, clip
+to_clip.select('blue_masked',
+ 'clip_raster',
+ rf_mask('blue_masked', 'clip_raster').alias('blue_clipped')) \
+ .filter(rf_data_cells('clip_raster') > 20) \
+ .orderBy(rf_data_cells('clip_raster'))
+```
+
+This kind of clipping technique is further used in @ref:[zonal statistics](zonal-algebra.md).
diff --git a/pyrasterframes/src/main/python/docs/nodata-handling.pymd b/python/docs/nodata-handling.pymd
similarity index 68%
rename from pyrasterframes/src/main/python/docs/nodata-handling.pymd
rename to python/docs/nodata-handling.pymd
index c9fffe390..915553c21 100644
--- a/pyrasterframes/src/main/python/docs/nodata-handling.pymd
+++ b/python/docs/nodata-handling.pymd
@@ -2,7 +2,7 @@
## What is NoData?
-In raster operations, the preservation and correct processing of missing observations is very important. In [most DataFrames and in scientific computing](https://www.oreilly.com/learning/handling-missing-data), the idea of missing data is expressed as a `null` or `NaN` value. However, a great deal of raster data is stored for space efficiency, which typically leads to use of integral values with a ["sentinel" value](https://en.wikipedia.org/wiki/Sentinel_value) designated to represent missing observations. This sentinel value varies across data products and is usually called the "NoData" value.
+In raster operations, the preservation and correct processing of missing observations is very important. In [most DataFrames and in scientific computing](https://www.oreilly.com/learning/handling-missing-data), the idea of missing data is expressed as a `null` or `NaN` value. However, a great deal of raster data is stored for space efficiency, which typically leads to use of integral values with a ["sentinel" value](https://en.wikipedia.org/wiki/Sentinel_value) designated to represent missing observations. This sentinel value varies across data products and bands. In a generic sense, it is usually called the "NoData" value.
RasterFrames provides a variety of functions to inspect and manage NoData within _tiles_.
@@ -40,9 +40,9 @@ CellType.float64()
We can also inspect the cell type of a given _tile_ or `proj_raster` column.
```python, ct_from_sen
-cell_types = spark.read.raster('https://s22s-test-geotiffs.s3.amazonaws.com/luray_snp/B02.tif') \
+cell_types = spark.read.raster('https://rasterframes.s3.amazonaws.com/samples/luray_snp/B02.tif') \
.select(rf_cell_type('proj_raster')).distinct()
-cell_types
+cell_types
```
### Understanding Cell Types and NoData
@@ -75,95 +75,6 @@ print(CellType.float32().no_data_value())
print(CellType.float32().with_no_data_value(-99.9).no_data_value())
```
-## Masking
-
-Let's continue the example above with Sentinel-2 data. Band 2 is blue and has no defined NoData. The quality information is in a separate file called the scene classification (SCL), which delineates areas of missing data and probable clouds. For more information on this, see the [Sentinel-2 algorithm overview](https://earth.esa.int/web/sentinel/technical-guides/sentinel-2-msi/level-2a/algorithm). Figure 3 tells us how to interpret the scene classification. For this example, we will exclude NoData, defective pixels, probable clouds, and cirrus clouds: values 0, 1, 8, 9, and 10.
-
-
-
-Credit: [Sentinel-2 algorithm overview](https://earth.esa.int/web/sentinel/technical-guides/sentinel-2-msi/level-2a/algorithm)
-
-The first step is to create a catalog with our band of interest and the SCL band. We read the data from the catalog, so the blue band and SCL _tiles_ are aligned across rows.
-
-```python, blue_scl_cat
-from pyspark.sql import Row
-
-blue_uri = 'https://s22s-test-geotiffs.s3.amazonaws.com/luray_snp/B02.tif'
-scl_uri = 'https://s22s-test-geotiffs.s3.amazonaws.com/luray_snp/SCL.tif'
-cat = spark.createDataFrame([Row(blue=blue_uri, scl=scl_uri),])
-unmasked = spark.read.raster(cat, catalog_col_names=['blue', 'scl'])
-unmasked.printSchema()
-```
-
-```python, show_cell_types
-cell_types = unmasked.select(rf_cell_type('blue'), rf_cell_type('scl')).distinct()
-cell_types
-```
-
-Drawing on @ref:[local map algebra](local-algebra.md) techniques, we will create new _tile_ columns that are indicators of unwanted pixels, as defined above. Since the mask column is an integer type, the addition is equivalent to a logical or, so the boolean true values are 1.
-
-```python, def_mask
-from pyspark.sql.functions import lit
-
-mask_part = unmasked.withColumn('nodata', rf_local_equal('scl', lit(0))) \
- .withColumn('defect', rf_local_equal('scl', lit(1))) \
- .withColumn('cloud8', rf_local_equal('scl', lit(8))) \
- .withColumn('cloud9', rf_local_equal('scl', lit(9))) \
- .withColumn('cirrus', rf_local_equal('scl', lit(10)))
-
-one_mask = mask_part.withColumn('mask', rf_local_add('nodata', 'defect')) \
- .withColumn('mask', rf_local_add('mask', 'cloud8')) \
- .withColumn('mask', rf_local_add('mask', 'cloud9')) \
- .withColumn('mask', rf_local_add('mask', 'cirrus'))
-
-cell_types = one_mask.select(rf_cell_type('mask')).distinct()
-cell_types
-```
-
-Because there is not a NoData already defined, we will choose one. In this particular example, the minimum value is greater than zero, so we can use 0 as the NoData value.
-
-```python, pick_nd
-blue_min = one_mask.agg(rf_agg_stats('blue').min.alias('blue_min'))
-blue_min
-```
-
-We can now construct the cell type string for our blue band's cell type, designating 0 as NoData.
-
-```python, get_ct_string
-blue_ct = one_mask.select(rf_cell_type('blue')).distinct().first()[0][0]
-masked_blue_ct = CellType(blue_ct).with_no_data_value(0)
-masked_blue_ct.cell_type_name
-```
-
-Now we will use the @ref:[`rf_mask_by_value`](reference.md#rf-mask-by-value) to designate the cloudy and other unwanted pixels as NoData in the blue column by converting the cell type and applying the mask.
-
-```python, mask_blu
-with_nd = rf_convert_cell_type('blue', masked_blue_ct)
-masked = one_mask.withColumn('blue_masked',
- rf_mask_by_value(with_nd, 'mask', lit(1))) \
- .drop('nodata', 'defect', 'cloud8', 'cloud9', 'cirrus', 'blue')
-```
-
-We can verify that the number of NoData cells in the resulting `blue_masked` column matches the total of the boolean `mask` _tile_ to ensure our logic is correct.
-
-```python, show_masked
-counts = masked.select(rf_no_data_cells('blue_masked'), rf_tile_sum('mask'))
-counts
-```
-
-It's also nice to view a sample. The white regions are areas of NoData.
-
-```python, display_blu, caption='Blue band masked against selected SCL values'
-sample = masked.orderBy(-rf_no_data_cells('blue_masked')).select(rf_tile('blue_masked'), rf_tile('scl')).first()
-display(sample[0])
-```
-
-And the original SCL data. The bright yellow is a cloudy region in the original image.
-
-```python, display_scl, caption='SCL tile for above'
-display(sample[1])
-```
-
## NoData and Local Arithmetic
Let's now explore how the presence of NoData affects @ref:[local map algebra](local-algebra.md) operations. To demonstrate the behavior, lets create two _tiles_. One _tile_ will have values of 0 and 1, and the other will have values of just 0.
@@ -270,7 +181,7 @@ sums = rf.select(
rf_cell_type('y'),
rf_cell_type(rf_local_add('x', 'y')).alias('xy_sum'),
)
-sums
+sums
```
Combining _tile_ columns of different cell types gets a little trickier when user defined NoData cell types are involved. Let's create two _tile_ columns: one with a NoData value of 1, and one with a NoData value of 2 (using our previously defined `get_nodata_ct` function).
diff --git a/pyrasterframes/src/main/python/docs/numpy-pandas.pymd b/python/docs/numpy-pandas.pymd
similarity index 100%
rename from pyrasterframes/src/main/python/docs/numpy-pandas.pymd
rename to python/docs/numpy-pandas.pymd
diff --git a/pyrasterframes/src/main/python/docs/raster-catalogs.pymd b/python/docs/raster-catalogs.pymd
similarity index 99%
rename from pyrasterframes/src/main/python/docs/raster-catalogs.pymd
rename to python/docs/raster-catalogs.pymd
index 3b634b767..1af68c2c6 100644
--- a/pyrasterframes/src/main/python/docs/raster-catalogs.pymd
+++ b/python/docs/raster-catalogs.pymd
@@ -72,14 +72,14 @@ scene2_B02 = "https://modis-pds.s3.amazonaws.com/MCD43A4.006/04/09/2018188/MCD43
two_d_cat_pd = pd.DataFrame([
{'B01': [scene1_B01], 'B02': [scene1_B02]},
{'B01': [scene2_B01], 'B02': [scene2_B02]}
-])
+])
# or
two_d_cat_df = spark.createDataFrame([
Row(B01=scene1_B01, B02=scene1_B02),
Row(B01=scene2_B01, B02=scene2_B02)
])
-
+
# As CSV string
tow_d_cat_csv = '\n'.join(['B01,B02', scene1_B01 + "," + scene1_B02, scene2_B01 + "," + scene2_B02])
```
diff --git a/python/docs/raster-join.pymd b/python/docs/raster-join.pymd
new file mode 100644
index 000000000..29bd35b4a
--- /dev/null
+++ b/python/docs/raster-join.pymd
@@ -0,0 +1,80 @@
+# Raster Join
+
+```python, init, echo=False
+from IPython.display import display
+import pyrasterframes.rf_ipython
+import pandas as pd
+from pyrasterframes.utils import create_rf_spark_session
+from pyrasterframes.rasterfunctions import *
+from pyspark.sql.functions import *
+spark = create_rf_spark_session(**{
+ 'spark.driver.memory': '4G',
+ 'spark.ui.enabled': 'false'
+})
+
+```
+
+## Description
+
+A common operation for raster data is reprojecting or warping the data to a different @ref:[CRS][CRS] with a specific @link:[transform](https://gdal.org/user/raster_data_model.html#affine-geotransform) { open=new }. In many use cases, the particulars of the warp operation depend on another set of raster data. Furthermore, the warp is done to put both sets of raster data to a common set of grid to enable manipulation of the datasets together.
+
+In RasterFrames, you can perform a **Raster Join** on two DataFrames containing raster data.
+The operation will perform a _spatial join_ based on the [CRS][CRS] and [extent][extent] data in each DataFrame. By default it is a left join and uses an intersection operator.
+For each candidate row, all _tile_ columns on the right hand side are warped to match the left hand side's [CRS][CRS], [extent][extent], and dimensions. Warping relies on GeoTrellis library code. You can specify the resampling method to be applied as one of: nearest_neighbor, bilinear, cubic_convolution, cubic_spline, lanczos, average, mode, median, max, min, or sum.
+The operation is also an aggregate, with multiple intersecting right-hand side tiles `merge`d into the result. There is no guarantee about the ordering of tiles used to select cell values in the case of overlapping tiles.
+When using the @ref:[`raster` DataSource](raster-join.md) you will automatically get the @ref:[CRS][CRS] and @ref:[extent][extent] information needed to do this operation.
+
+
+## Example Code
+
+Because the raster join is a distributed spatial join, indexing of both DataFrames using the [spatial index][spatial-index] is crucial for performance.
+
+```python, example_raster_join
+# Southern Mozambique December 29, 2016
+modis = spark.read.raster('s3://astraea-opendata/MCD43A4.006/21/11/2016297/MCD43A4.A2016297.h21v11.006.2016306075821_B01.TIF',
+ spatial_index_partitions=True) \
+ .withColumnRenamed('proj_raster', 'modis')
+
+landsat8 = spark.read.raster('https://landsat-pds.s3.us-west-2.amazonaws.com/c1/L8/167/077/LC08_L1TP_167077_20161015_20170319_01_T1/LC08_L1TP_167077_20161015_20170319_01_T1_B4.TIF',
+ spatial_index_partitions=True) \
+ .withColumnRenamed('proj_raster', 'landsat')
+
+rj = landsat8.raster_join(modis, resampling_method="cubic_convolution")
+
+# Show some non-empty tiles
+rj.select('landsat', 'modis', 'crs', 'extent') \
+ .filter(rf_data_cells('modis') > 0) \
+ .filter(rf_tile_max('landsat') > 0)
+```
+
+## Additional Options
+
+The following optional arguments are allowed:
+
+ * `left_extent` - the column on the left-hand DataFrame giving the [extent][extent] of the tile columns
+ * `left_crs` - the column on the left-hand DataFrame giving the [CRS][CRS] of the tile columns
+ * `right_extent` - the column on the right-hand DataFrame giving the [extent][extent] of the tile columns
+ * `right_crs` - the column on the right-hand DataFrame giving the [CRS][CRS] of the tile columns
+ * `join_exprs` - a single column expression as would be used in the [`on` parameter of `join`](https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame.join)
+ * `resampling_method` - resampling algorithm to use in reprojection of right-hand tile column
+
+
+
+ Note that the `join_exprs` will override the join behavior described above. By default the expression is equivalent to:
+
+```python, join_expr, evaluate=False
+st_intersects(
+ st_geometry(left[left_extent]),
+ st_reproject(st_geometry(right[right_extent]), right[right_crs], left[left_crs])
+)
+```
+
+Resampling method to use can be specified by passing one of the following strings into `resampling_method` parameter.
+The point resampling methods are: `"nearest_neighbor"`, `"bilinear"`, `"cubic_convolution"`, `"cubic_spline"`, and `"lanczos"`.
+The aggregating resampling methods are: `"average"`, `"mode"`, `"median"`, `"max"`, "`min`", or `"sum"`.
+Note the aggregating methods are intended for downsampling. For example a 0.25 factor and `max` method returns the maximum value in a 4x4 neighborhood.
+
+
+[CRS]: concepts.md#coordinate-reference-system--crs
+[extent]: concepts.md#extent
+[spatial-index]:raster-read.md#spatial-indexing-and-partitioning
diff --git a/pyrasterframes/src/main/python/docs/raster-read.pymd b/python/docs/raster-read.pymd
similarity index 62%
rename from pyrasterframes/src/main/python/docs/raster-read.pymd
rename to python/docs/raster-read.pymd
index 53f3a96e6..6b7e1445b 100644
--- a/pyrasterframes/src/main/python/docs/raster-read.pymd
+++ b/python/docs/raster-read.pymd
@@ -7,19 +7,22 @@ import pandas as pd
from pyrasterframes.utils import create_rf_spark_session
from pyrasterframes.rasterfunctions import *
from pyspark.sql.functions import *
-spark = create_rf_spark_session()
+spark = create_rf_spark_session(**{
+ 'spark.driver.memory': '4G',
+ 'spark.ui.enabled': 'false'
+})
```
RasterFrames registers a DataSource named `raster` that enables reading of GeoTIFFs (and other formats when @ref:[GDAL is installed](getting-started.md#installing-gdal)) from arbitrary URIs. The `raster` DataSource operates on either a single raster file location or another DataFrame, called a _catalog_, containing pointers to many raster file locations.
RasterFrames can also read from @ref:[GeoTrellis catalogs and layers](raster-read.md#geotrellis).
-## Single Raster
+## Single Rasters
The simplest way to use the `raster` reader is with a single raster from a single URI or file. In the examples that follow we'll be reading from a Sentinel-2 scene stored in an AWS S3 bucket.
```python, read_one_uri
-rf = spark.read.raster('https://s22s-test-geotiffs.s3.amazonaws.com/luray_snp/B02.tif')
+rf = spark.read.raster('https://rasterframes.s3.amazonaws.com/samples/luray_snp/B02.tif')
rf.printSchema()
```
@@ -33,21 +36,89 @@ print("CRS", crs.value.crsProj4)
```
```python, raster_parts
-parts = rf.select(
+rf.select(
rf_extent("proj_raster").alias("extent"),
rf_tile("proj_raster").alias("tile")
)
-parts
```
-
-You can also see that the single raster has been broken out into many arbitrary non-overlapping regions. Doing so takes advantage of parallel in-memory reads from the cloud hosted data source and allows Spark to work on manageable amounts of data per task. The following code fragment shows us how many subtiles were created from a single source image.
-
-```python, count_by_uri
-counts = rf.groupby(rf.proj_raster_path).count()
-counts
+You can also see that the single raster has been broken out into many rows containing arbitrary non-overlapping regions. Doing so takes advantage of parallel in-memory reads from the cloud hosted data source and allows Spark to work on manageable amounts of data per row.
+The map below shows downsampled imagery with the bounds of the individual tiles.
+
+@@@ note
+
+The image contains visible "seams" between the tile extents due to reprojection and downsampling used to create the image.
+The native imagery in the DataFrame does not contain any gaps in the source raster's coverage.
+
+@@@
+
+```python, folium_map_of_tile_extents, echo=False
+from pyrasterframes.rf_types import Extent
+import folium
+import pyproj
+from functools import partial
+from shapely.ops import transform as shtransform
+from shapely.geometry import box
+import geopandas
+import numpy
+
+wm_crs = 'EPSG:3857'
+crs84 = 'urn:ogc:def:crs:OGC:1.3:CRS84'
+
+# Generate overview image
+wm_extent = rf.agg(
+ rf_agg_reprojected_extent(rf_extent('proj_raster'), rf_crs('proj_raster'), wm_crs)
+ ).first()[0]
+aoi = Extent.from_row(wm_extent)
+
+aspect = aoi.width / aoi.height
+ov_size = 1024
+ov = rf.agg(
+ rf_agg_overview_raster('proj_raster', int(ov_size * aspect), ov_size, aoi)
+ ).first()[0]
+
+# Reproject the web mercator extent to WGS84
+project = partial(
+ pyproj.transform,
+ pyproj.Proj(wm_crs),
+ pyproj.Proj(crs84)
+ )
+crs84_extent = shtransform(project, box(*wm_extent))
+
+# Individual tile WGS84 extents in a dataframe
+tile_extents_df = rf.select(
+ st_reproject(
+ rf_geometry('proj_raster'),
+ rf_crs('proj_raster'),
+ rf_mk_crs('epsg:4326')
+ ).alias('geometry')
+).toPandas()
+
+ntiles = numpy.nanquantile(ov.cells, [0.03, 0.97])
+
+# use `filled` because folium doesn't know how to maskedArray
+a = numpy.clip(ov.cells.filled(0), ntiles[0], ntiles[1])
+
+m = folium.Map([crs84_extent.centroid.y, crs84_extent.centroid.x],
+ zoom_start=9) \
+ .add_child(
+ folium.raster_layers.ImageOverlay(
+ a,
+ [[crs84_extent.bounds[1], crs84_extent.bounds[0]],
+ [crs84_extent.bounds[3], crs84_extent.bounds[2]]],
+ name='rf.proj_raster.tile'
+ )
+ ) \
+ .add_child(folium.GeoJson(
+ geopandas.GeoDataFrame(tile_extents_df, crs=crs84),
+ name='rf.proj_raster.extent',
+ style_function=lambda _: {'fillOpacity':0}
+ )) \
+ .add_child(folium.LayerControl(collapsed=False))
+m
```
+
Let's select a single _tile_ and view it. The _tile_ preview image as well as the string representation provide some basic information about the _tile_: its dimensions as numbers of columns and rows and the cell type, or data type of all the cells in the _tile_. For more about cell types, refer to @ref:[this discussion](nodata-handling.md#cell-types).
```python, show_tile_sample
@@ -55,11 +126,74 @@ tile = rf.select(rf_tile("proj_raster")).first()[0]
display(tile)
```
+## Multiple Singleband Rasters
+
+In this example, we show the reading @ref:[two bands](concepts.md#band) of [Landsat 8](https://landsat.gsfc.nasa.gov/landsat-8/) imagery (red and near-infrared), combining them with `rf_normalized_difference` to compute [NDVI](https://en.wikipedia.org/wiki/Normalized_difference_vegetation_index), a common measure of vegetation health. As described in the section on @ref:[catalogs](raster-catalogs.md), image URIs in a single row are assumed to be from the same scene/granule, and therefore compatible. This pattern is commonly used when multiple bands are stored in separate files.
+
+```python, multi_singleband
+bands = [f'B{b}' for b in [4, 5]]
+uris = [f'https://landsat-pds.s3.us-west-2.amazonaws.com/c1/L8/014/032/LC08_L1TP_014032_20190720_20190731_01_T1/LC08_L1TP_014032_20190720_20190731_01_T1_{b}.TIF' for b in bands]
+catalog = ','.join(bands) + '\n' + ','.join(uris)
+
+rf = (spark.read.raster(catalog, bands)
+ # Adding semantic names
+ .withColumnRenamed('B4', 'red').withColumnRenamed('B5', 'NIR')
+ # Adding tile center point for reference
+ .withColumn('longitude_latitude', st_reproject(st_centroid(rf_geometry('red')), rf_crs('red'), lit('EPSG:4326')))
+ # Compute NDVI
+ .withColumn('NDVI', rf_normalized_difference('NIR', 'red'))
+ # For the purposes of inspection, filter out rows where there's not much vegetation
+ .where(rf_tile_sum('NDVI') > 10000)
+ # Order output
+ .select('longitude_latitude', 'red', 'NIR', 'NDVI'))
+display(rf)
+```
+
+## Multiband Rasters
+
+A multiband raster is represented by a three dimensional numeric array stored in a single file. The first two dimensions are spatial, and the third dimension is typically designated for different spectral @ref:[bands](concepts.md#band). The bands could represent intensity of different wavelengths of light (or other electromagnetic radiation), or they could measure other phenomena such as time, quality indications, or additional gas concentrations, etc.
+
+Multiband rasters files have a strictly ordered set of bands, which are typically indexed from 1. Some files have metadata tags associated with each band. Some files have a color interpetation metadata tag indicating how to interpret the bands.
+
+When reading a multiband raster or a @ref:[_catalog_](#raster-catalogs) describing multiband rasters, you will need to know ahead of time which bands you want to read. You will specify the bands to read, **indexed from zero**, as a list of integers into the `band_indexes` parameter of the `raster` reader.
+
+For example, we can read a four-band (red, green, blue, and near-infrared) image as follows. The individual rows of the resulting DataFrame still represent distinct spatial extents, with a projected raster column for each band specified by `band_indexes`.
+
+```python, multiband
+mb = spark.read.raster(
+ 'https://rasterframes.s3.amazonaws.com/samples/naip/m_3807863_nw_17_1_20160620.tif',
+ band_indexes=[0, 1, 2, 3],
+)
+display(mb)
+```
+
+If a band is passed into `band_indexes` that exceeds the number of bands in the raster, a projected raster column will still be generated in the schema but the column will be full of `null` values.
+
+You can also pass a _catalog_ and `band_indexes` together into the `raster` reader. This will create a projected raster column for the combination of all items in `catalog_col_names` and `band_indexes`. Again if a band in `band_indexes` exceeds the number of bands in a raster, it will have a `null` value for the corresponding column.
+
+Here is a trivial example with a _catalog_ over multiband rasters. We specify two columns containing URIs and two bands, resulting in four projected raster columns.
+
+```python, multiband_catalog
+import pandas as pd
+mb_cat = pd.DataFrame([
+ {'foo': 'https://rasterframes.s3.amazonaws.com/samples/naip/m_3807863_nw_17_1_20160620.tif',
+ 'bar': 'https://rasterframes.s3.amazonaws.com/samples/naip/m_3807863_nw_17_1_20160620.tif'
+ },
+])
+mb2 = spark.read.raster(
+ spark.createDataFrame(mb_cat),
+ catalog_col_names=['foo', 'bar'],
+ band_indexes=[0, 1],
+ tile_dimensions=(64,64)
+)
+mb2.printSchema()
+```
+
## URI Formats
RasterFrames relies on three different I/O drivers, selected based on a combination of scheme, file extentions, and library availability. GDAL is used by default if a compatible version of GDAL (>= 2.4) is installed, and if GDAL supports the specified scheme. If GDAL is not available, either the _Java I/O_ or _Hadoop_ driver will be selected, depending on scheme.
-Note: The GDAL driver is the only one that can read non-GeoTIFF files.
+Note: The GDAL driver is the only one that can read non-GeoTIFF files.
| Prefix | GDAL | Java I/O | Hadoop |
@@ -115,7 +249,7 @@ MODIS data products are delivered on a regular, consistent grid, making identifi
For example, MODIS data right above the equator is all grid coordinates with `v07`.
```python, catalog_filtering
-equator = modis_catalog.where(F.col('gid').like('%v07%'))
+equator = modis_catalog.where(F.col('gid').like('%v07%'))
equator.select('date', 'gid')
```
@@ -132,7 +266,7 @@ Observe the schema of the resulting DataFrame has a projected raster struct for
sample = rf \
.select('gid', rf_extent('red'), rf_extent('nir'), rf_tile('red'), rf_tile('nir')) \
.where(~rf_is_no_data_tile('red'))
-sample.limit(3)
+sample.limit(3)
```
## Lazy Raster Reading
@@ -142,7 +276,7 @@ By default, reading raster pixel values is delayed until it is absolutely needed
Consider the following two reads of the same data source. In the first, the lazy case, there is a pointer to the URI, extent and band to read. This will not be evaluated until the cell values are absolutely required. The second case shows the option to force the raster to be fully loaded right away.
```python, lazy_demo_1
-uri = 'https://s22s-test-geotiffs.s3.amazonaws.com/luray_snp/B02.tif'
+uri = 'https://rasterframes.s3.amazonaws.com/samples/luray_snp/B02.tif'
lazy = spark.read.raster(uri).select(col('proj_raster.tile').cast('string'))
lazy
```
@@ -154,44 +288,24 @@ non_lazy
In the initial examples on this page, you may have noticed that the realized (non-lazy) _tiles_ are shown, but we did not change `lazy_tiles`. Instead, we used @ref:[`rf_tile`](reference.md#rf-tile) to explicitly request the realized _tile_ from the lazy representation.
-## Multiband Rasters
-
-A multiband raster represents a three dimensional numeric array. The first two dimensions are spatial, and the third dimension is typically designated for different spectral @ref:[bands](concepts.md#band). The bands could represent intensity of different wavelengths of light (or other electromagnetic radiation), or they could measure other phenomena such as time, quality indications, or additional gas concentrations, etc.
+## Spatial Indexing and Partitioning
-Multiband rasters files have a strictly ordered set of bands, which are typically indexed from 1. Some files have metadata tags associated with each band. Some files have a color interpetation metadata tag indicating how to interpret the bands.
+@@@ warning
+This is an experimental feature, and may be removed.
+@@@
-When reading a multiband raster or a _catalog_ describing multiband rasters, you will need to know ahead of time which bands you want to read. You will specify the bands to read, **indexed from zero**, as a list of integers into the `band_indexes` parameter of the `raster` reader.
-For example, we can read a four-band (red, green, blue, and near-infrared) image as follows. The individual rows of the resulting DataFrame still represent distinct spatial extents, with a projected raster column for each band specified by `band_indexes`.
+It's often desirable to take extra steps in ensuring your data is effectively distributed over your computing resources. One way of doing that is using something called a ["space filling curve"](https://en.wikipedia.org/wiki/Space-filling_curve), which turns an N-dimensional value into a one dimensional value, with properties that favor keeping entities near each other in N-space near each other in index space. In particular RasterFrames support space-filling curves mapping the geographic location of _tiles_ to a one-dimensional index space called [`xz2`](https://www.geomesa.org/documentation/user/datastores/index_overview.html). To have RasterFrames add a spatial index based partitioning on a raster reads, use the `spatial_index_partitions` parameter. By default it will use the same number of partitions as configured in [`spark.sql.shuffle.partitions`](https://spark.apache.org/docs/latest/sql-performance-tuning.html#other-configuration-options).
-```python, multiband
-mb = spark.read.raster(
- 's3://s22s-test-geotiffs/naip/m_3807863_nw_17_1_20160620.tif',
- band_indexes=[0, 1, 2, 3],
-)
-mb.printSchema()
+```python, spatial_indexing
+df = spark.read.raster(uri, spatial_index_partitions=True)
+df
```
-If a band is passed into `band_indexes` that exceeds the number of bands in the raster, a projected raster column will still be generated in the schema but the column will be full of `null` values.
-
-You can also pass a _catalog_ and `band_indexes` together into the `raster` reader. This will create a projected raster column for the combination of all items in `catalog_col_names` and `band_indexes`. Again if a band in `band_indexes` exceeds the number of bands in a raster, it will have a `null` value for the corresponding column.
-
-Here is a trivial example with a _catalog_ over multiband rasters. We specify two columns containing URIs and two bands, resulting in four projected raster columns.
+You can also pass a positive integer to the parameter to specify the number of desired partitions.
-```python, multiband_catalog
-import pandas as pd
-mb_cat = pd.DataFrame([
- {'foo': 's3://s22s-test-geotiffs/naip/m_3807863_nw_17_1_20160620.tif',
- 'bar': 's3://s22s-test-geotiffs/naip/m_3807863_nw_17_1_20160620.tif'
- },
-])
-mb2 = spark.read.raster(
- spark.createDataFrame(mb_cat),
- catalog_col_names=['foo', 'bar'],
- band_indexes=[0, 1],
- tile_dimensions=(64,64)
-)
-mb2.printSchema()
+```python, spatial_indexing
+df = spark.read.raster(uri, spatial_index_partitions=800)
```
## GeoTrellis
diff --git a/python/docs/raster-write.pymd b/python/docs/raster-write.pymd
new file mode 100644
index 000000000..befc6329c
--- /dev/null
+++ b/python/docs/raster-write.pymd
@@ -0,0 +1,168 @@
+# Writing Raster Data
+
+RasterFrames is oriented toward large scale analyses of spatial data. The primary output of these analyses could be a @ref:[statistical summary](aggregation.md), a @ref:[machine learning model](machine-learning.md), or some other result that is generally much smaller than the input dataset.
+
+However, there are times in any analysis where writing a representative sample of the work in progress provides valuable feedback on the current state of the process and results, or you are constructing a new dataset to be used in other analyses.
+
+
+This will be our setup for the following examples:
+
+```python setup
+from pyrasterframes import *
+from pyrasterframes.rasterfunctions import *
+from pyrasterframes.utils import create_rf_spark_session
+import pyrasterframes.rf_ipython
+from IPython.display import display
+import os.path
+spark = create_rf_spark_session(**{
+ 'spark.driver.memory': '4G',
+ 'spark.ui.enabled': 'false'
+})
+def scene(band):
+ b = str(band).zfill(2) # converts int 2 to '02'
+ return 'https://modis-pds.s3.amazonaws.com/MCD43A4.006/11/08/2019059/' \
+ 'MCD43A4.A2019059.h11v08.006.2019072203257_B{}.TIF'.format(b)
+rf = spark.read.raster(scene(2), tile_dimensions=(256, 256))
+```
+
+
+## IPython/Jupyter
+
+@ref:[This section](ipython.md) provides details on how Tiles and DataFrames with Tiles in them can be viewed in the IPython/Jupyter.
+
+## Overview Rasters
+
+In cases where writing and reading to/from a GeoTIFF isn't convenient, RasterFrames provides the @ref:[`rf_agg_overview_raster`](reference.md#rf-agg-overview-raster) aggregate function, where you can construct a single raster (rendered as a tile) downsampled from all or a subset of the DataFrame. This allows you to effectively construct the same operations the GeoTIFF writer performs, but without the file I/O.
+
+The `rf_agg_overview_raster` function will reproject data to the commonly used ["web mercator"](https://en.wikipedia.org/wiki/Web_Mercator_projection) CRS. You must specify an "Area of Interest" (AOI) in web mercator. You can use @ref:[`rf_agg_reprojected_extent`](reference.md#rf-agg-reprojected-extent) to compute the extent of a DataFrame in any CRS or mix of CRSs.
+
+```python, overview
+wm_extent = rf.agg(
+ rf_agg_reprojected_extent(rf_extent('proj_raster'), rf_crs('proj_raster'), 'EPSG:3857')
+ ).first()[0]
+aoi = Extent.from_row(wm_extent)
+print(aoi)
+aspect = aoi.width / aoi.height
+
+ov = rf.agg(
+ rf_agg_overview_raster('proj_raster', int(512 * aspect), 512, aoi)
+).first()[0]
+print("`ov` is of type", type(ov))
+ov
+```
+
+## GeoTIFFs
+
+GeoTIFF is one of the most common file formats for spatial data, providing flexibility in data encoding, representation, and storage. RasterFrames provides a specialized Spark DataFrame writer for rendering a RasterFrame to a GeoTIFF. It is accessed by calling `dataframe.write.geotiff`.
+
+### Limitations and mitigations
+
+One downside to GeoTIFF is that it is not a big-data native format. To create a GeoTIFF, all the data to be written must be `collect`ed in the memory of the Spark driver. This means you must actively limit the size of the data to be written. It is trivial to lazily read a set of inputs that cannot feasibly be written to GeoTIFF in the same environment.
+
+When writing GeoTIFFs in RasterFrames, you should limit the size of the collected data. Consider filtering the dataframe by time or @ref:[spatial filters](vector-data.md#geomesa-functions-and-spatial-relations).
+
+You can also specify the dimensions of the GeoTIFF file to be written using the `raster_dimensions` parameter as described below.
+
+### Parameters
+
+If there are many _tile_ or projected raster columns in the DataFrame, the GeoTIFF writer will write each one as a separate band in the file. Each band in the output will be tagged the input column names for reference.
+
+* `path`: the path local to the driver where the file will be written
+* `crs`: the PROJ4 string of the CRS the GeoTIFF is to be written in
+* `raster_dimensions`: optional, a tuple of two ints giving the size of the resulting file. If specified, RasterFrames will downsample the data in distributed fashion using bilinear resampling. If not specified, the default is to write the dataframe at full resolution, which can result in an out of memory error.
+
+### Example
+
+See also the example in the @ref:[unsupervised learning page](unsupervised-learning.md).
+
+Let's render an overview of a scene's red band as a small raster, reprojecting it to latitude and longitude coordinates on the [WGS84](https://en.wikipedia.org/wiki/World_Geodetic_System) reference ellipsoid (aka [EPSG:4326](https://spatialreference.org/ref/epsg/4326/)).
+
+```python write_geotiff
+outfile = os.path.join('/tmp', 'geotiff-overview.tif')
+rf.write.geotiff(outfile, crs='EPSG:4326', raster_dimensions=(256, 256))
+```
+
+We can view the written file with `rasterio`:
+
+```python view_geotiff
+import rasterio
+from rasterio.plot import show, show_hist
+
+with rasterio.open(outfile) as src:
+ # View raster
+ show(src, adjust='linear')
+ # View data distribution
+ show_hist(src, bins=50, lw=0.0, stacked=False, alpha=0.6,
+ histtype='stepfilled', title="Overview Histogram")
+```
+
+
+@@@ warning
+Attempting to write a full resolution GeoTIFF constructed from multiple scenes is likely to result in an out of memory error. Consider filtering the dataframe more aggressively and using a smaller value for the `raster_dimensions` parameter.
+@@@
+
+### Color Composites
+
+If the DataFrame has three or four tile columns, the GeoTIFF is written with the `ColorInterp` tags on the [bands](https://gdal.org/user/raster_data_model.html?highlight=color%20interpretation#raster-band) to indicate red, green, blue, and optionally alpha. Use a `select` statement to ensure the bands are in the desired order. If the bands chosen are red, green, and blue, the composite is called a true-color composite. Otherwise it is a false-color composite. If the number of tile columns is not 3 or 4, the `ColorInterp` tag will indicate greyscale.
+
+Also see [Color Composite](ipython.md#color-composite) in the IPython/Juptyer Extensions.
+
+### PNG
+
+In this example we will use the @ref:[`rf_rgb_composite`](reference.md#rf-rgb-composite) function, we will compute a three band PNG image as a `bytearray`. The resulting `bytearray` will be displayed as an image in either a Spark or pandas DataFrame display if `rf_ipython` has been imported.
+
+```python, png_composite
+# Select red, green, and blue, respectively
+composite_df = spark.read.raster([[scene(1), scene(4), scene(3)]])
+
+composite_df = composite_df.withColumn('png',
+ rf_render_png('proj_raster_0', 'proj_raster_1', 'proj_raster_2')).cache()
+composite_df.select('png').limit(1)
+```
+
+Alternatively the `bytearray` result can be displayed with [`pillow`](https://pillow.readthedocs.io/en/stable/).
+
+```python, single_tile_pil
+import io
+from PIL.Image import open as PIL_open
+png_bytearray = composite_df.first()['png']
+pil_image = PIL_open(io.BytesIO(png_bytearray))
+pil_image
+```
+
+### GeoTIFF
+
+In this example we will write a false-color composite as a GeoTIFF
+
+```python, geotiff_composite
+outfile = os.path.join('/tmp', 'geotiff-composite.tif')
+composite_df = spark.read.raster([[scene(3), scene(1), scene(4)]])
+composite_df.write.geotiff(outfile, crs='EPSG:4326', raster_dimensions=(256, 256))
+```
+
+```python, show_geotiff
+with rasterio.open(outfile) as src:
+ show(src)
+```
+
+## GeoTrellis Layers
+
+[GeoTrellis][GeoTrellis] is one of the key libraries upon which RasterFrames is built. It provides a Scala language API for working with geospatial raster data. GeoTrellis defines a [tile layer storage](https://geotrellis.readthedocs.io/en/latest/guide/tile-backends.html) format for persisting imagery mosaics. RasterFrames can write data from a `RasterFrameLayer` into a [GeoTrellis Layer](https://geotrellis.readthedocs.io/en/latest/guide/tile-backends.html). RasterFrames provides a `geotrellis` DataSource that supports both @ref:[reading](raster-read.md#geotrellis-layers) and @ref:[writing](raster-write.md#geotrellis-layers) GeoTrellis layers.
+
+> An example is forthcoming. In the mean time referencing the [`GeoTrellisDataSourceSpec` test code](https://github.com/locationtech/rasterframes/blob/develop/datasource/src/test/scala/org/locationtech/rasterframes/datasource/geotrellis/GeoTrellisDataSourceSpec.scala) may help.
+
+## Parquet
+
+You can write a RasterFrame to the [Apache Parquet][Parquet] format. This format is designed to efficiently persist and query columnar data in distributed file system, such as HDFS. It also provides benefits when working in single node (or "local") mode, such as tailoring organization for defined query patterns.
+
+```python write_parquet, evaluate=False
+rf.withColumn('exp', rf_expm1('proj_raster')) \
+ .write.mode('append').parquet('hdfs:///rf-user/sample.pq')
+```
+
+```python, cleanup, echo=False
+spark.stop()
+```
+
+[GeoTrellis]: https://geotrellis.readthedocs.io/en/latest/
+[Parquet]: https://spark.apache.org/docs/latest/sql-data-sources-parquet.html
diff --git a/pyrasterframes/src/main/python/docs/static/rasterframe-anatomy.png b/python/docs/static/rasterframe-anatomy.png
similarity index 100%
rename from pyrasterframes/src/main/python/docs/static/rasterframe-anatomy.png
rename to python/docs/static/rasterframe-anatomy.png
diff --git a/pyrasterframes/src/main/python/docs/static/rasterframes-data-sources.png b/python/docs/static/rasterframes-data-sources.png
similarity index 100%
rename from pyrasterframes/src/main/python/docs/static/rasterframes-data-sources.png
rename to python/docs/static/rasterframes-data-sources.png
diff --git a/pyrasterframes/src/main/python/docs/static/rasterframes-locationtech-stack.png b/python/docs/static/rasterframes-locationtech-stack.png
similarity index 100%
rename from pyrasterframes/src/main/python/docs/static/rasterframes-locationtech-stack.png
rename to python/docs/static/rasterframes-locationtech-stack.png
diff --git a/python/docs/static/rasterframes-pipeline-nologo.png b/python/docs/static/rasterframes-pipeline-nologo.png
new file mode 100644
index 000000000..caa011d70
Binary files /dev/null and b/python/docs/static/rasterframes-pipeline-nologo.png differ
diff --git a/pyrasterframes/src/main/python/docs/static/rasterframes-pipeline.png b/python/docs/static/rasterframes-pipeline.png
similarity index 100%
rename from pyrasterframes/src/main/python/docs/static/rasterframes-pipeline.png
rename to python/docs/static/rasterframes-pipeline.png
diff --git a/pyrasterframes/src/main/python/docs/static/sentinel-2-scene-classification-labels.png b/python/docs/static/sentinel-2-scene-classification-labels.png
similarity index 100%
rename from pyrasterframes/src/main/python/docs/static/sentinel-2-scene-classification-labels.png
rename to python/docs/static/sentinel-2-scene-classification-labels.png
diff --git a/pyrasterframes/src/main/python/docs/supervised-learning.pymd b/python/docs/supervised-learning.pymd
similarity index 82%
rename from pyrasterframes/src/main/python/docs/supervised-learning.pymd
rename to python/docs/supervised-learning.pymd
index c66697032..756c344b9 100644
--- a/pyrasterframes/src/main/python/docs/supervised-learning.pymd
+++ b/python/docs/supervised-learning.pymd
@@ -24,7 +24,7 @@ The first step is to create a Spark DataFrame containing our imagery data. To ac
The imagery for feature data will come from [eleven bands of 60 meter resolution Sentinel-2](https://earth.esa.int/web/sentinel/user-guides/sentinel-2-msi/resolutions/spatial) imagery. We also will use the [scene classification (SCL)](https://earth.esa.int/web/sentinel/technical-guides/sentinel-2-msi/level-2a/algorithm) data to identify high quality, non-cloudy pixels.
```python, read_bands
-uri_base = 's3://s22s-test-geotiffs/luray_snp/{}.tif'
+uri_base = 'https://rasterframes.s3.amazonaws.com/samples/luray_snp/{}.tif'
bands = ['B01', 'B02', 'B03', 'B04', 'B05', 'B06', 'B07', 'B08', 'B09', 'B11', 'B12']
cols = ['SCL'] + bands
@@ -32,7 +32,8 @@ catalog_df = pd.DataFrame([
{b: uri_base.format(b) for b in cols}
])
-df = spark.read.raster(catalog_df, catalog_col_names=cols, tile_dimensions=(128, 128)) \
+tile_size = 256
+df = spark.read.raster(catalog_df, catalog_col_names=cols, tile_dimensions=(tile_size, tile_size)) \
.repartition(100)
df = df.select(
@@ -70,7 +71,7 @@ print('Found ', len(crses), 'distinct CRS.')
crs = crses[0][0]
from pyspark import SparkFiles
-spark.sparkContext.addFile('https://github.com/locationtech/rasterframes/raw/develop/pyrasterframes/src/test/resources/luray-labels.geojson')
+spark.sparkContext.addFile('https://rasterframes.s3.amazonaws.com/samples/luray_snp/luray-labels.geojson')
label_df = spark.read.geojson(SparkFiles.get('luray-labels.geojson')) \
.select('id', st_reproject('geometry', lit('EPSG:4326'), lit(crs)).alias('geometry')) \
@@ -79,7 +80,7 @@ label_df = spark.read.geojson(SparkFiles.get('luray-labels.geojson')) \
df_joined = df.join(label_df, st_intersects(st_geometry('extent'), 'geometry')) \
.withColumn('dims', rf_dimensions('B01'))
-df_labeled = df_joined.withColumn('label',
+df_labeled = df_joined.withColumn('label',
rf_rasterize('geometry', st_geometry('extent'), 'id', 'dims.cols', 'dims.rows')
)
```
@@ -91,23 +92,12 @@ To filter only for good quality pixels, we follow roughly the same procedure as
```python, make_mask
from pyspark.sql.functions import lit
-mask_part = df_labeled \
- .withColumn('nodata', rf_local_equal('scl', lit(0))) \
- .withColumn('defect', rf_local_equal('scl', lit(1))) \
- .withColumn('cloud8', rf_local_equal('scl', lit(8))) \
- .withColumn('cloud9', rf_local_equal('scl', lit(9))) \
- .withColumn('cirrus', rf_local_equal('scl', lit(10)))
-
-df_mask_inv = mask_part \
- .withColumn('mask', rf_local_add('nodata', 'defect')) \
- .withColumn('mask', rf_local_add('mask', 'cloud8')) \
- .withColumn('mask', rf_local_add('mask', 'cloud9')) \
- .withColumn('mask', rf_local_add('mask', 'cirrus')) \
- .drop('nodata', 'defect', 'cloud8', 'cloud9', 'cirrus')
-
+df_labeled = df_labeled \
+ .withColumn('mask', rf_local_is_in('scl', [0, 1, 8, 9, 10]))
+
# at this point the mask contains 0 for good cells and 1 for defect, etc
# convert cell type and set value 1 to NoData
-df_mask = df_mask_inv.withColumn('mask',
+df_mask = df_labeled.withColumn('mask',
rf_with_no_data(rf_convert_cell_type('mask', 'uint8'), 1.0)
)
@@ -184,14 +174,14 @@ accuracy = eval.evaluate(prediction_df)
print("\nAccuracy:", accuracy)
```
-As an example of using the flexibility provided by DataFrames, the code below computes and displays the confusion matrix.
+As an example of using the flexibility provided by DataFrames, the code below computes and displays the confusion matrix.
```python, confusion_mtrx
cnf_mtrx = prediction_df.groupBy(classifier.getPredictionCol()) \
.pivot(classifier.getLabelCol()) \
.count() \
.sort(classifier.getPredictionCol())
-cnf_mtrx
+cnf_mtrx
```
## Visualize Prediction
@@ -204,29 +194,29 @@ scored = model.transform(df_mask.drop('label'))
retiled = scored \
.groupBy('extent', 'crs') \
.agg(
- rf_assemble_tile('column_index', 'row_index', 'prediction', 128, 128).alias('prediction'),
- rf_assemble_tile('column_index', 'row_index', 'B04', 128, 128).alias('red'),
- rf_assemble_tile('column_index', 'row_index', 'B03', 128, 128).alias('grn'),
- rf_assemble_tile('column_index', 'row_index', 'B02', 128, 128).alias('blu')
+ rf_assemble_tile('column_index', 'row_index', 'prediction', tile_size, tile_size).alias('prediction'),
+ rf_assemble_tile('column_index', 'row_index', 'B04', tile_size, tile_size).alias('red'),
+ rf_assemble_tile('column_index', 'row_index', 'B03', tile_size, tile_size).alias('grn'),
+ rf_assemble_tile('column_index', 'row_index', 'B02', tile_size, tile_size).alias('blu')
)
retiled.printSchema()
```
-Take a look at a sample of the resulting output and the corresponding area's red-green-blue composite image.
-
-```python, display_rgb
-sample = retiled \
- .select('prediction', rf_rgb_composite('red', 'grn', 'blu').alias('rgb')) \
- .sort(-rf_tile_sum(rf_local_equal('prediction', lit(3.0)))) \
- .first()
-
-sample_rgb = sample['rgb']
-mins = np.nanmin(sample_rgb.cells, axis=(0,1))
-plt.imshow((sample_rgb.cells - mins) / (np.nanmax(sample_rgb.cells, axis=(0,1)) - mins))
-```
-
-Recall the label coding: 1 is forest (purple), 2 is cropland (green) and 3 is developed areas(yellow).
-
-```python, display_prediction
-display(sample['prediction'])
+Take a look at a sample of the resulting prediction and the corresponding area's red-green-blue composite image. Note that because each `prediction` tile is rendered independently, the colors may not have the same meaning across rows.
+
+```python
+scaling_quantiles = retiled.agg(
+ rf_agg_approx_quantiles('red', [0.03, 0.97]).alias('red_q'),
+ rf_agg_approx_quantiles('grn', [0.03, 0.97]).alias('grn_q'),
+ rf_agg_approx_quantiles('blu', [0.03, 0.97]).alias('blu_q')
+ ).first()
+
+retiled.select(
+ rf_render_png(
+ rf_local_clamp('red', *scaling_quantiles['red_q']).alias('red'),
+ rf_local_clamp('grn', *scaling_quantiles['grn_q']).alias('grn'),
+ rf_local_clamp('blu', *scaling_quantiles['blu_q']).alias('blu')
+ ).alias('tci'),
+ rf_render_color_ramp_png('prediction', 'ClassificationBoldLandUse').alias('prediction')
+ )
```
diff --git a/pyrasterframes/src/main/python/docs/time-series.pymd b/python/docs/time-series.pymd
similarity index 91%
rename from pyrasterframes/src/main/python/docs/time-series.pymd
rename to python/docs/time-series.pymd
index c1f7c6675..832ebcb93 100644
--- a/pyrasterframes/src/main/python/docs/time-series.pymd
+++ b/python/docs/time-series.pymd
@@ -19,7 +19,7 @@ spark = create_rf_spark_session("local[4]")
In this example, we will show how the flexibility of the DataFrame concept for raster data allows a simple and intuitive way to extract a time series from Earth observation data. We will continue our example from the @ref:[Zonal Map Algebra page](zonal-algebra.md).
-We will summarize the change in @ref:[NDVI](local-algebra.md#computing-ndvi) over the spring and early summer of 2018 in the Cuyahoga Valley National Park in Ohio, USA.
+We will summarize the change in @ref:[NDVI](local-algebra.md#computing-ndvi) over the spring and early summer of 2018 in the Cuyahoga Valley National Park in Ohio, USA.
```python vector, echo=False, results='hidden'
cat = spark.read.format('aws-pds-modis-catalog').load().repartition(200)
@@ -41,23 +41,21 @@ def simplify(g, tol):
park_vector = park_vector.withColumn('geo_simp', simplify('geometry', lit(0.001))) \
.select('geo_simp') \
.hint('broadcast')
-
-
```
## Catalog Read
-As in our other example, we will query for a single known MODIS granule directly. We limit the vector data to the single park of interest. The longer time period selected should show the change in plant vigor as leaves emerge over the spring and into early summer. The definitions of `cat` and `park_vector` are as in the @ref:[Zonal Map Algebra page](zonal-algebra.md).
+As in our other example, we will query for a single known MODIS granule directly. We limit the vector data to the single park of interest. The longer time period selected should show the change in plant vigor as leaves emerge over the spring and into early summer. The definitions of `cat` and `park_vector` are as in the @ref:[Zonal Map Algebra page](zonal-algebra.md).
```python query_catalog
park_cat = cat \
.filter(
(cat.granule_id == 'h11v04') &
(cat.acquisition_date > lit('2018-02-19')) &
- (cat.acquisition_date < lit('2018-07-01'))
+ (cat.acquisition_date < lit('2018-07-01'))
) \
- .crossJoin(park_vector.filter('OBJECTID == 380')) #only coyahuga
-
+ .crossJoin(park_vector.filter('UNIT_CODE == "CUVA"')) #only coyahuga
+
```
## Vector and Raster Data Interaction
@@ -80,16 +78,16 @@ rf_park_tile = spark.read.raster(
## Create Time Series
-We next aggregate across the cell values to arrive at an average NDVI for each week of the year. We use `pyspark`'s built in `groupby` and time functions with a RasterFrames @ref:[aggregate function](aggregation.md) to do this. Note that the computation is creating a weighted average, which is weighted by the number of valid observations per week.
+We next aggregate across the cell values to arrive at an average NDVI for each week of the year. We use `pyspark`'s built in `groupby` and time functions with a RasterFrames @ref:[aggregate function](aggregation.md) to do this. Note that the computation is creating a weighted average, which is weighted by the number of valid observations per week.
```python ndvi_time_series
from pyspark.sql.functions import col, year, weekofyear, month
time_series = rf_park_tile \
.groupby(
- year('acquisition_date').alias('year'),
+ year('acquisition_date').alias('year'),
weekofyear('acquisition_date').alias('week')) \
- .agg(rf_agg_mean('ndvi_masked').alias('ndvi'))
+ .agg(rf_agg_mean('ndvi_masked').alias('ndvi'))
```
Finally, we will take a look at the NDVI over time.
diff --git a/pyrasterframes/src/main/python/docs/unsupervised-learning.pymd b/python/docs/unsupervised-learning.pymd
similarity index 69%
rename from pyrasterframes/src/main/python/docs/unsupervised-learning.pymd
rename to python/docs/unsupervised-learning.pymd
index 494eb9bea..f4db8c04f 100644
--- a/pyrasterframes/src/main/python/docs/unsupervised-learning.pymd
+++ b/python/docs/unsupervised-learning.pymd
@@ -20,7 +20,7 @@ We import various Spark components needed to construct our `Pipeline`.
```python, imports, echo=True
import pandas as pd
from pyrasterframes import TileExploder
-from pyrasterframes.rasterfunctions import rf_assemble_tile, rf_crs, rf_extent, rf_tile, rf_dimensions
+from pyrasterframes.rasterfunctions import *
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.clustering import KMeans
@@ -30,20 +30,28 @@ from pyspark.ml import Pipeline
The first step is to create a Spark DataFrame of our imagery data. To achieve that we will create a catalog DataFrame using the pattern from [the I/O page](raster-io.html#Single-Scene--Multiple-Bands). In the catalog, each row represents a distinct area and time, and each column is the URI to a band's image product. The resulting Spark DataFrame may have many rows per URI, with a column corresponding to each band.
-
```python, catalog
-filenamePattern = "https://github.com/locationtech/rasterframes/" \
- "raw/develop/core/src/test/resources/L8-B{}-Elkton-VA.tiff"
+filenamePattern = "https://rasterframes.s3.amazonaws.com/samples/elkton/L8-B{}-Elkton-VA.tiff"
catalog_df = pd.DataFrame([
{'b' + str(b): filenamePattern.format(b) for b in range(1, 8)}
])
-df = spark.read.raster(catalog_df, catalog_col_names=catalog_df.columns)
+tile_size = 256
+df = spark.read.raster(catalog_df, catalog_col_names=catalog_df.columns, tile_size=tile_size)
df = df.withColumn('crs', rf_crs(df.b1)) \
- .withColumn('extent', rf_crs(df.b1))
+ .withColumn('extent', rf_extent(df.b1))
df.printSchema()
```
+In this small example, all the images in our `catalog_df` have the same @ref:[CRS](concepts.md#coordinate-reference-system-crs-), which we verify in the code snippet below. The `crs` object will be useful for visualization later.
+
+```python, crses
+crses = df.select('crs.crsProj4').distinct().collect()
+print('Found ', len(crses), 'distinct CRS: ', crses)
+assert len(crses) == 1
+crs = crses[0]['crsProj4']
+```
+
## Create ML Pipeline
SparkML requires that each observation be in its own row, and features for each observation be packed into a single `Vector`. For this unsupervised learning problem, we will treat each _pixel_ as an observation and each band as a feature. The first step is to "explode" the _tiles_ into a single row per pixel. In RasterFrames, generally a pixel is called a @ref:[`cell`](concepts.md#cell).
@@ -52,7 +60,7 @@ SparkML requires that each observation be in its own row, and features for each
exploder = TileExploder()
```
-To "vectorize" the the band columns, we use the SparkML `VectorAssembler`. Each of the seven bands is a different feature.
+To "vectorize" the band columns, we use the SparkML `VectorAssembler`. Each of the seven bands is a different feature.
```python, assembler
assembler = VectorAssembler() \
@@ -112,22 +120,23 @@ We can recreate the tiled data structure using the metadata added by the `TileEx
```python, assemble
from pyrasterframes.rf_types import CellType
-tile_dims = df.select(rf_dimensions(df.b1).alias('dims')).first()['dims']
retiled = clustered.groupBy('extent', 'crs') \
.agg(
rf_assemble_tile('column_index', 'row_index', 'prediction',
- tile_dims['cols'], tile_dims['rows'], CellType.int8()).alias('prediction')
+ tile_size, tile_size, CellType.int8())
)
-
-retiled.printSchema()
```
-```python, display
-retiled
-```
-
-The resulting output is shown below.
+Next we will @ref:[write the output to a GeoTiff file](raster-write.md#geotiffs). Doing so in this case works quickly and well for a few specific reasons that may not hold in all cases. We can write the data at full resolution, by omitting the `raster_dimensions` argument, because we know the input raster dimensions are small. Also, the data is all in a single CRS, as we demonstrated above. Because the `catalog_df` is only a single row, we know the output GeoTIFF value at a given location corresponds to a single input. Finally, the `retiled` `DataFrame` only has a single `Tile` column, so the band interpretation is trivial.
```python, viz
-display(retiled.select('prediction').first()['prediction'])
+import rasterio
+output_tif = 'unsupervised.tif'
+
+retiled.write.geotiff(output_tif, crs=crs)
+
+with rasterio.open(output_tif) as src:
+ for b in range(1, src.count + 1):
+ print("Tags on band", b, src.tags(b))
+ display(src)
```
diff --git a/pyrasterframes/src/main/python/docs/vector-data.pymd b/python/docs/vector-data.pymd
similarity index 98%
rename from pyrasterframes/src/main/python/docs/vector-data.pymd
rename to python/docs/vector-data.pymd
index 31a450f6b..7226cb822 100644
--- a/pyrasterframes/src/main/python/docs/vector-data.pymd
+++ b/python/docs/vector-data.pymd
@@ -1,6 +1,6 @@
# Vector Data
-RasterFrames provides a variety of ways to work with spatial vector data (points, lines, and polygons) alongside raster data.
+RasterFrames provides a variety of ways to work with spatial vector data (points, lines, and polygons) alongside raster data.
* DataSource for GeoJSON format
* Ability to convert between from [GeoPandas][GeoPandas] and Spark DataFrames
diff --git a/pyrasterframes/src/main/python/docs/zonal-algebra.pymd b/python/docs/zonal-algebra.pymd
similarity index 90%
rename from pyrasterframes/src/main/python/docs/zonal-algebra.pymd
rename to python/docs/zonal-algebra.pymd
index 9869e6b36..556b5c0f4 100644
--- a/pyrasterframes/src/main/python/docs/zonal-algebra.pymd
+++ b/python/docs/zonal-algebra.pymd
@@ -64,7 +64,7 @@ park_vector = park_vector.withColumn('geo_simp', simplify('geometry', lit(0.005)
## Catalog Read
-Both parks are entirely contained in MODIS granule h11 v04. We will simply filter on this granule, rather than using a @ref:[spatial relation](vector-data.md#geomesa-functions-and-spatial-relations).
+Both parks are entirely contained in MODIS granule h11 v04. We will simply filter on this granule, rather than using a @ref:[spatial relation](vector-data.md#geomesa-functions-and-spatial-relations).
```python query_catalog
cat = spark.read.format('aws-pds-modis-catalog').load().repartition(50)
@@ -72,10 +72,10 @@ park_cat = cat \
.filter(
(cat.granule_id == 'h11v04') &
(cat.acquisition_date >= lit('2018-05-01')) &
- (cat.acquisition_date < lit('2018-06-01'))
+ (cat.acquisition_date < lit('2018-06-01'))
) \
.crossJoin(park_vector)
-
+
park_cat.printSchema()
```
@@ -89,14 +89,14 @@ park_rf = spark.read.raster(
park_cat.select(['acquisition_date', 'granule_id'] + raster_cols + park_vector.columns),
catalog_col_names=raster_cols) \
.withColumn('park_native', st_reproject('geo_simp', lit('EPSG:4326'), rf_crs('B01'))) \
- .filter(st_intersects('park_native', rf_geometry('B01')))
+ .filter(st_intersects('park_native', rf_geometry('B01')))
park_rf.printSchema()
```
## Define Zone Tiles
-Now we have the vector representation of the park boundary alongside the _tiles_ of red and near infrared bands. Next, we need to create a _tile_ representation of the park to allow us to limit the raster analysis to pixels within the park _zone_. This is similar to the masking operation demonstrated in @ref:[NoData handling](nodata-handling.md#masking). We rasterize the geometries using @ref:[`rf_rasterize`](reference.md#rf-rasterize): this creates a new _tile_ column aligned with the imagery, and containing the park's OBJECTID attribute for cells intersecting the _zone_. Cells outside the park _zones_ have a NoData value.
+Now we have the vector representation of the park boundary alongside the _tiles_ of red and near infrared bands. Next, we need to create a _tile_ representation of the park to allow us to limit the raster analysis to pixels within the park _zone_. This is similar to the masking operation demonstrated in @ref:[Masking](masking.md#masking). We rasterize the geometries using @ref:[`rf_rasterize`](reference.md#rf-rasterize): this creates a new _tile_ column aligned with the imagery, and containing the park's OBJECTID attribute for cells intersecting the _zone_. Cells outside the park _zones_ have a NoData value.
```python burn_in
rf_park_tile = park_rf \
@@ -121,7 +121,7 @@ rf_ndvi = rf_park_tile \
zonal_mean = rf_ndvi \
.groupby('OBJECTID', 'UNIT_NAME') \
- .agg(rf_agg_mean('ndvi'))
+ .agg(rf_agg_mean('ndvi_masked'))
zonal_mean
```
diff --git a/python/geomesa_pyspark/__init__.py b/python/geomesa_pyspark/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/pyrasterframes/src/main/python/geomesa_pyspark/types.py b/python/geomesa_pyspark/types.py
similarity index 92%
rename from pyrasterframes/src/main/python/geomesa_pyspark/types.py
rename to python/geomesa_pyspark/types.py
index 5f1d0a110..bbc402718 100644
--- a/pyrasterframes/src/main/python/geomesa_pyspark/types.py
+++ b/python/geomesa_pyspark/types.py
@@ -9,7 +9,7 @@
http://www.opensource.org/licenses/apache2.0.php.
+ ***********************************************************************/"""
-from pyspark.sql.types import UserDefinedType, StructField, BinaryType, StructType
+from pyspark.sql.types import BinaryType, StructField, StructType, UserDefinedType
from shapely import wkb
from shapely.geometry import LineString, MultiLineString, MultiPoint, MultiPolygon, Point, Polygon
from shapely.geometry.base import BaseGeometry
@@ -17,18 +17,17 @@
class ShapelyGeometryUDT(UserDefinedType):
-
@classmethod
def sqlType(cls):
return StructType([StructField("wkb", BinaryType(), True)])
@classmethod
def module(cls):
- return 'geomesa_pyspark.types'
+ return "geomesa_pyspark.types"
@classmethod
def scalaUDT(cls):
- return 'org.apache.spark.sql.jts.' + cls.__name__
+ return "org.apache.spark.sql.jts." + cls.__name__
def serialize(self, obj):
return [_serialize_to_wkb(obj)]
diff --git a/pyrasterframes/src/main/python/pyrasterframes/__init__.py b/python/pyrasterframes/__init__.py
similarity index 51%
rename from pyrasterframes/src/main/python/pyrasterframes/__init__.py
rename to python/pyrasterframes/__init__.py
index 7915af34e..8e569447e 100644
--- a/pyrasterframes/src/main/python/pyrasterframes/__init__.py
+++ b/python/pyrasterframes/__init__.py
@@ -23,23 +23,23 @@
appended to PySpark classes.
"""
-from __future__ import absolute_import
+from typing import List, Optional, Tuple
+
+import geomesa_pyspark.types # enable vector integrations
from pyspark import SparkContext
-from pyspark.sql import SparkSession, DataFrame, DataFrameReader, DataFrameWriter
+from pyspark.sql import DataFrame, DataFrameReader, DataFrameWriter, SparkSession
from pyspark.sql.column import _to_java_column
-from geomesa_pyspark import types # <-- required to ensure Shapely UDTs get registered.
# Import RasterFrameLayer types and functions
from .rf_context import RFContext
+from .rf_types import RasterFrameLayer, RasterSourceUDT, TileExploder, TileUDT
from .version import __version__
-from .rf_types import RasterFrameLayer, TileExploder, TileUDT, RasterSourceUDT
-import geomesa_pyspark.types # enable vector integrations
-__all__ = ['RasterFrameLayer', 'TileExploder']
+__all__ = ["RasterFrameLayer", "TileExploder"]
-def _rf_init(spark_session):
- """ Adds RasterFrames functionality to PySpark session."""
+def _rf_init(spark_session: SparkSession) -> SparkSession:
+ """Adds RasterFrames functionality to PySpark session."""
if not hasattr(spark_session, "rasterframes"):
spark_session.rasterframes = RFContext(spark_session)
spark_session.sparkContext._rf_context = spark_session.rasterframes
@@ -47,69 +47,112 @@ def _rf_init(spark_session):
return spark_session
-def _kryo_init(builder):
+def _kryo_init(builder: SparkSession.Builder) -> SparkSession.Builder:
"""Registers Kryo Serializers for better performance."""
# NB: These methods need to be kept up-to-date wit those in `org.locationtech.rasterframes.extensions.KryoMethods`
- builder \
- .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") \
- .config("spark.kryo.registrator", "org.locationtech.rasterframes.util.RFKryoRegistrator") \
- .config("spark.kryoserializer.buffer.max", "500m")
+ builder.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer").config(
+ "spark.kryo.registrator", "org.locationtech.rasterframes.util.RFKryoRegistrator"
+ ).config("spark.kryoserializer.buffer.max", "500m")
return builder
-def _convert_df(df, sp_key=None, metadata=None):
+
+def _convert_df(df: DataFrame, sp_key=None, metadata=None) -> RasterFrameLayer:
+ """Internal function to convert a DataFrame to a RasterFrameLayer."""
ctx = SparkContext._active_spark_context._rf_context
if sp_key is None:
return RasterFrameLayer(ctx._jrfctx.asLayer(df._jdf), ctx._spark_session)
else:
import json
- return RasterFrameLayer(ctx._jrfctx.asLayer(
- df._jdf, _to_java_column(sp_key), json.dumps(metadata)), ctx._spark_session)
-
-def _raster_join(df, other, left_extent=None, left_crs=None, right_extent=None, right_crs=None, join_exprs=None):
+ return RasterFrameLayer(
+ ctx._jrfctx.asLayer(df._jdf, _to_java_column(sp_key), json.dumps(metadata)),
+ ctx._spark_session,
+ )
+
+
+def _raster_join(
+ df: DataFrame,
+ other: DataFrame,
+ left_extent=None,
+ left_crs=None,
+ right_extent=None,
+ right_crs=None,
+ join_exprs=None,
+ resampling_method="nearest_neighbor",
+) -> DataFrame:
ctx = SparkContext._active_spark_context._rf_context
+ resampling_method = resampling_method.lower().strip().replace("_", "")
+ assert resampling_method in [
+ "nearestneighbor",
+ "bilinear",
+ "cubicconvolution",
+ "cubicspline",
+ "lanczos",
+ "average",
+ "mode",
+ "median",
+ "max",
+ "min",
+ "sum",
+ ]
if join_exprs is not None:
- assert left_extent is not None and left_crs is not None and right_extent is not None and right_crs is not None
+ assert (
+ left_extent is not None
+ and left_crs is not None
+ and right_extent is not None
+ and right_crs is not None
+ )
# Note the order of arguments here.
cols = [join_exprs, left_extent, left_crs, right_extent, right_crs]
- jdf = ctx._jrfctx.rasterJoin(df._jdf, other._jdf, *[_to_java_column(c) for c in cols])
+ args = [_to_java_column(c) for c in cols] + [resampling_method]
+ jdf = ctx._jrfctx.rasterJoin(df._jdf, other._jdf, *args)
elif left_extent is not None:
assert left_crs is not None and right_extent is not None and right_crs is not None
cols = [left_extent, left_crs, right_extent, right_crs]
- jdf = ctx._jrfctx.rasterJoin(df._jdf, other._jdf, *[_to_java_column(c) for c in cols])
+ args = [_to_java_column(c) for c in cols] + [resampling_method]
+ jdf = ctx._jrfctx.rasterJoin(df._jdf, other._jdf, *args)
else:
- jdf = ctx._jrfctx.rasterJoin(df._jdf, other._jdf)
+ jdf = ctx._jrfctx.rasterJoin(df._jdf, other._jdf, resampling_method)
- return RasterFrameLayer(jdf, ctx._spark_session)
+ return DataFrame(jdf, ctx._spark_session)
-def _layer_reader(df_reader, format_key, path, **options):
- """ Loads the file of the given type at the given path."""
+def _layer_reader(
+ df_reader: DataFrameReader, format_key: str, path: Optional[str], **options: str
+) -> RasterFrameLayer:
+ """Loads the file of the given type at the given path."""
df = df_reader.format(format_key).load(path, **options)
return _convert_df(df)
-def _aliased_reader(df_reader, format_key, path, **options):
- """ Loads the file of the given type at the given path."""
+def _aliased_reader(
+ df_reader: DataFrameReader, format_key: str, path: Optional[str], **options: str
+) -> DataFrame:
+ """Loads the file of the given type at the given path."""
return df_reader.format(format_key).load(path, **options)
-def _aliased_writer(df_writer, format_key, path, **options):
- """ Saves the dataframe to a file of the given type at the given path."""
+def _aliased_writer(
+ df_writer: DataFrameWriter, format_key: str, path: Optional[str], **options: str
+):
+ """Saves the dataframe to a file of the given type at the given path."""
return df_writer.format(format_key).save(path, **options)
def _raster_reader(
- df_reader,
- source=None,
- catalog_col_names=None,
- band_indexes=None,
- tile_dimensions=(256, 256),
- lazy_tiles=True,
- **options):
+ df_reader: DataFrameReader,
+ source=None,
+ catalog_col_names: Optional[List[str]] = None,
+ band_indexes: Optional[List[int]] = None,
+ buffer_size: int = 0,
+ tile_dimensions: Tuple[int] = (256, 256),
+ lazy_tiles: bool = True,
+ spatial_index_partitions=None,
+ **options: str,
+) -> DataFrame:
"""
Returns a Spark DataFrame from raster data files specified by URIs.
Each row in the returned DataFrame will contain a column with struct of (CRS, Extent, Tile) for each item in
@@ -123,34 +166,55 @@ def _raster_reader(
:param catalog_col_names: required if `source` is a DataFrame or CSV string. It is a list of strings giving the names of columns containing URIs to read.
:param band_indexes: list of integers indicating which bands, zero-based, to read from the raster files specified; default is to read only the first band.
:param tile_dimensions: tuple or list of two indicating the default tile dimension as (columns, rows).
+ :param buffer_size: buffer each tile read by this many cells on all sides.
:param lazy_tiles: If true (default) only generate minimal references to tile contents; if false, fetch tile cell values.
+ :param spatial_index_partitions: If true, partitions read tiles by a Z2 spatial index using the default shuffle partitioning.
+ If a values > 0, the given number of partitions are created instead of the default.
:param options: Additional keyword arguments to pass to the Spark DataSource.
"""
from pandas import DataFrame as PdDataFrame
- if 'catalog' in options:
- source = options['catalog'] # maintain back compatibility with 0.8.0
+ if "catalog" in options:
+ source = options["catalog"] # maintain back compatibility with 0.8.0
def to_csv(comp):
if isinstance(comp, str):
return comp
else:
- return ','.join(str(v) for v in comp)
+ return ",".join(str(v) for v in comp)
def temp_name():
- """ Create a random name for a temporary view """
+ """Create a random name for a temporary view"""
import uuid
- return str(uuid.uuid4()).replace('-', '')
+
+ return str(uuid.uuid4()).replace("-", "")
if band_indexes is None:
band_indexes = [0]
- options.update({
- "band_indexes": to_csv(band_indexes),
- "tile_dimensions": to_csv(tile_dimensions),
- "lazy_tiles": lazy_tiles
- })
+ if spatial_index_partitions:
+ num = int(spatial_index_partitions)
+ if num < 0:
+ spatial_index_partitions = "-1"
+ elif num == 0:
+ spatial_index_partitions = None
+
+ if spatial_index_partitions:
+ if spatial_index_partitions == True:
+ spatial_index_partitions = "-1"
+ else:
+ spatial_index_partitions = str(spatial_index_partitions)
+ options.update({"spatial_index_partitions": spatial_index_partitions})
+
+ options.update(
+ {
+ "band_indexes": to_csv(band_indexes),
+ "tile_dimensions": to_csv(tile_dimensions),
+ "lazy_tiles": str(lazy_tiles),
+ "buffer_size": int(buffer_size),
+ }
+ )
# Parse the `source` argument
path = None # to pass into `path` param
@@ -158,19 +222,24 @@ def temp_name():
if all([isinstance(i, str) for i in source]):
path = None
catalog = None
- options.update(dict(paths='\n'.join([str(i) for i in source]))) # pass in "uri1\nuri2\nuri3\n..."
+ options.update(
+ dict(paths="\n".join([str(i) for i in source]))
+ ) # pass in "uri1\nuri2\nuri3\n..."
if all([isinstance(i, list) for i in source]):
# list of lists; we will rely on pandas to:
# - coerce all data to str (possibly using objects' __str__ or __repr__)
# - ensure data is not "ragged": all sublists are same len
path = None
- catalog_col_names = ['proj_raster_{}'.format(i) for i in range(len(source[0]))] # assign these names
- catalog = PdDataFrame(source,
- columns=catalog_col_names,
- dtype=str,
- )
+ catalog_col_names = [
+ "proj_raster_{}".format(i) for i in range(len(source[0]))
+ ] # assign these names
+ catalog = PdDataFrame(
+ source,
+ columns=catalog_col_names,
+ dtype=str,
+ )
elif isinstance(source, str):
- if '\n' in source or '\r' in source:
+ if "\n" in source or "\r" in source:
# then the `source` string is a catalog as a CSV (header is required)
path = None
catalog = source
@@ -187,25 +256,23 @@ def temp_name():
raise Exception("'catalog_col_names' required when DataFrame 'catalog' specified")
if isinstance(catalog, str):
- options.update({
- "catalog_csv": catalog,
- "catalog_col_names": to_csv(catalog_col_names)
- })
+ options.update({"catalog_csv": catalog, "catalog_col_names": to_csv(catalog_col_names)})
elif isinstance(catalog, DataFrame):
# check catalog_col_names
- assert all([c in catalog.columns for c in catalog_col_names]), \
- "All items in catalog_col_names must be the name of a column in the catalog DataFrame."
+ assert all(
+ [c in catalog.columns for c in catalog_col_names]
+ ), "All items in catalog_col_names must be the name of a column in the catalog DataFrame."
# Create a random view name
tmp_name = temp_name()
catalog.createOrReplaceTempView(tmp_name)
- options.update({
- "catalog_table": tmp_name,
- "catalog_col_names": to_csv(catalog_col_names)
- })
+ options.update(
+ {"catalog_table": tmp_name, "catalog_col_names": to_csv(catalog_col_names)}
+ )
elif isinstance(catalog, PdDataFrame):
# check catalog_col_names
- assert all([c in catalog.columns for c in catalog_col_names]), \
- "All items in catalog_col_names must be the name of a column in the catalog DataFrame."
+ assert all(
+ [c in catalog.columns for c in catalog_col_names]
+ ), "All items in catalog_col_names must be the name of a column in the catalog DataFrame."
# Handle to active spark session
session = SparkContext._active_spark_context._rf_context._spark_session
@@ -213,44 +280,53 @@ def temp_name():
tmp_name = temp_name()
spark_catalog = session.createDataFrame(catalog)
spark_catalog.createOrReplaceTempView(tmp_name)
- options.update({
- "catalog_table": tmp_name,
- "catalog_col_names": to_csv(catalog_col_names)
- })
+ options.update(
+ {"catalog_table": tmp_name, "catalog_col_names": to_csv(catalog_col_names)}
+ )
- return df_reader \
- .format("raster") \
- .load(path, **options)
+ return df_reader.format("raster").load(path, **options)
-def _geotiff_writer(
- df_writer,
- path=None,
- crs=None,
- raster_dimensions=None,
- **options):
+def _stac_api_reader(df_reader: DataFrameReader, uri: str, filters: dict = None) -> DataFrame:
+ """
+ :param uri: STAC API uri
+ :param filters: STAC API Search filters dict (bbox, datetime, intersects, collections, items, limit, query, next), see the STAC API Spec for more details https://github.com/radiantearth/stac-api-spec
+ """
+ import json
+ return (
+ df_reader.format("stac-api")
+ .option("uri", uri)
+ .option("search-filters", json.dumps(filters))
+ .load()
+ )
+
+
+def _geotiff_writer(
+ df_writer: DataFrameWriter,
+ path: str,
+ crs: Optional[str] = None,
+ raster_dimensions: Tuple[int] = None,
+ **options: str,
+):
def set_dims(parts):
parts = [int(p) for p in parts]
assert len(parts) == 2, "Expected dimensions specification to have exactly two components"
- assert all([p > 0 for p in parts]), "Expected all components in dimensions to be positive integers"
- options.update({
- "imageWidth": parts[0],
- "imageHeight": parts[1]
- })
+ assert all(
+ [p > 0 for p in parts]
+ ), "Expected all components in dimensions to be positive integers"
+ options.update({"imageWidth": str(parts[0]), "imageHeight": str(parts[1])})
parts = [int(p) for p in parts]
- assert all([p > 0 for p in parts]), 'nice message'
+ assert all([p > 0 for p in parts]), "nice message"
if raster_dimensions is not None:
if isinstance(raster_dimensions, (list, tuple)):
set_dims(raster_dimensions)
elif isinstance(raster_dimensions, str):
- set_dims(raster_dimensions.split(','))
+ set_dims(raster_dimensions.split(","))
if crs is not None:
- options.update({
- "crs": crs
- })
+ options.update({"crs": crs})
return _aliased_writer(df_writer, "geotiff", path, **options)
@@ -273,5 +349,8 @@ def set_dims(parts):
DataFrameReader.geotiff = lambda df_reader, path: _layer_reader(df_reader, "geotiff", path)
DataFrameWriter.geotiff = _geotiff_writer
DataFrameReader.geotrellis = lambda df_reader, path: _layer_reader(df_reader, "geotrellis", path)
-DataFrameReader.geotrellis_catalog = lambda df_reader, path: _aliased_reader(df_reader, "geotrellis-catalog", path)
+DataFrameReader.geotrellis_catalog = lambda df_reader, path: _aliased_reader(
+ df_reader, "geotrellis-catalog", path
+)
DataFrameWriter.geotrellis = lambda df_writer, path: _aliased_writer(df_writer, "geotrellis", path)
+DataFrameReader.stacapi = _stac_api_reader
diff --git a/python/pyrasterframes/rasterfunctions.py b/python/pyrasterframes/rasterfunctions.py
new file mode 100644
index 000000000..83a01011b
--- /dev/null
+++ b/python/pyrasterframes/rasterfunctions.py
@@ -0,0 +1,1434 @@
+#
+# This software is licensed under the Apache 2 license, quoted below.
+#
+# Copyright 2019 Astraea, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may not
+# use this file except in compliance with the License. You may obtain a copy of
+# the License at
+#
+# [http://www.apache.org/licenses/LICENSE-2.0]
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations under
+# the License.
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+
+"""
+This module creates explicit Python functions that map back to the existing Scala
+implementations. Most functions are standard Column functions, but those with unique
+signatures are handled here as well.
+"""
+from typing import Iterable, List, Optional, Union
+
+from deprecation import deprecated
+from py4j.java_gateway import JavaObject
+from pyspark.sql.column import Column, _to_java_column
+from pyspark.sql.functions import lit
+
+from .rf_context import RFContext
+from .rf_types import CRS, CellType, Extent
+from .version import __version__
+
+THIS_MODULE = "pyrasterframes"
+
+Column_type = Union[str, Column]
+
+
+def _context_call(name: str, *args):
+ f = RFContext.active().lookup(name)
+ return f(*args)
+
+
+def _apply_column_function(name: str, *args: Column_type) -> Column:
+ jfcn = RFContext.active().lookup(name)
+ jcols = [_to_java_column(arg) for arg in args]
+ return Column(jfcn(*jcols))
+
+
+def _apply_scalar_to_tile(name: str, tile_col: Column_type, scalar: Union[int, float]) -> Column:
+ jfcn = RFContext.active().lookup(name)
+ return Column(jfcn(_to_java_column(tile_col), scalar))
+
+
+def _parse_cell_type(cell_type_arg: Union[str, CellType]) -> JavaObject:
+ """Convert the cell type representation to the expected JVM CellType object."""
+
+ def to_jvm(ct):
+ return _context_call("_parse_cell_type", ct)
+
+ if isinstance(cell_type_arg, str):
+ return to_jvm(cell_type_arg)
+ elif isinstance(cell_type_arg, CellType):
+ return to_jvm(cell_type_arg.cell_type_name)
+
+
+def rf_cell_types() -> List[CellType]:
+ """Return a list of standard cell types"""
+ return [CellType(str(ct)) for ct in _context_call("rf_cell_types")]
+
+
+def rf_assemble_tile(
+ col_index: Column_type,
+ row_index: Column_type,
+ cell_data_col: Column_type,
+ num_cols: Union[int, Column_type],
+ num_rows: Union[int, Column_type],
+ cell_type: Optional[Union[str, CellType]] = None,
+) -> Column:
+ """Create a Tile from a column of cell data with location indices"""
+ jfcn = RFContext.active().lookup("rf_assemble_tile")
+
+ if isinstance(num_cols, Column):
+ num_cols = _to_java_column(num_cols)
+
+ if isinstance(num_rows, Column):
+ num_rows = _to_java_column(num_rows)
+
+ if cell_type is None:
+ return Column(
+ jfcn(
+ _to_java_column(col_index),
+ _to_java_column(row_index),
+ _to_java_column(cell_data_col),
+ num_cols,
+ num_rows,
+ )
+ )
+
+ else:
+ return Column(
+ jfcn(
+ _to_java_column(col_index),
+ _to_java_column(row_index),
+ _to_java_column(cell_data_col),
+ num_cols,
+ num_rows,
+ _parse_cell_type(cell_type),
+ )
+ )
+
+
+def rf_array_to_tile(array_col: Column_type, num_cols: int, num_rows: int) -> Column:
+ """Convert array in `array_col` into a Tile of dimensions `num_cols` and `num_rows'"""
+ jfcn = RFContext.active().lookup("rf_array_to_tile")
+ return Column(jfcn(_to_java_column(array_col), num_cols, num_rows))
+
+
+def rf_convert_cell_type(tile_col: Column_type, cell_type: Union[str, CellType]) -> Column:
+ """Convert the numeric type of the Tiles in `tileCol`"""
+ jfcn = RFContext.active().lookup("rf_convert_cell_type")
+ return Column(jfcn(_to_java_column(tile_col), _parse_cell_type(cell_type)))
+
+
+def rf_interpret_cell_type_as(tile_col: Column_type, cell_type: Union[str, CellType]) -> Column:
+ """Change the interpretation of the tile_col's cell values according to specified cell_type"""
+ jfcn = RFContext.active().lookup("rf_interpret_cell_type_as")
+ return Column(jfcn(_to_java_column(tile_col), _parse_cell_type(cell_type)))
+
+
+def rf_make_constant_tile(
+ scalar_value: Union[int, float],
+ num_cols: int,
+ num_rows: int,
+ cell_type: Union[str, CellType] = CellType.float64(),
+) -> Column:
+ """Constructor for constant tile column"""
+ jfcn = RFContext.active().lookup("rf_make_constant_tile")
+ return Column(jfcn(scalar_value, num_cols, num_rows, _parse_cell_type(cell_type)))
+
+
+def rf_make_zeros_tile(
+ num_cols: int, num_rows: int, cell_type: Union[str, CellType] = CellType.float64()
+) -> Column:
+ """Create column of constant tiles of zero"""
+ jfcn = RFContext.active().lookup("rf_make_zeros_tile")
+ return Column(jfcn(num_cols, num_rows, _parse_cell_type(cell_type)))
+
+
+def rf_make_ones_tile(
+ num_cols: int, num_rows: int, cell_type: Union[str, CellType] = CellType.float64()
+) -> Column:
+ """Create column of constant tiles of one"""
+ jfcn = RFContext.active().lookup("rf_make_ones_tile")
+ return Column(jfcn(num_cols, num_rows, _parse_cell_type(cell_type)))
+
+
+def rf_rasterize(
+ geometry_col: Column_type,
+ bounds_col: Column_type,
+ value_col: Column_type,
+ num_cols_col: Column_type,
+ num_rows_col: Column_type,
+) -> Column:
+ """Create a tile where cells in the grid defined by cols, rows, and bounds are filled with the given value."""
+ return _apply_column_function(
+ "rf_rasterize", geometry_col, bounds_col, value_col, num_cols_col, num_rows_col
+ )
+
+
+def st_reproject(geometry_col: Column_type, src_crs: Column_type, dst_crs: Column_type) -> Column:
+ """Reproject a column of geometry given the CRSs of the source and destination."""
+ return _apply_column_function("st_reproject", geometry_col, src_crs, dst_crs)
+
+
+def rf_explode_tiles(*tile_cols: Column_type) -> Column:
+ """Create a row for each cell in Tile."""
+ jfcn = RFContext.active().lookup("rf_explode_tiles")
+ jcols = [_to_java_column(arg) for arg in tile_cols]
+ return Column(jfcn(RFContext.active().list_to_seq(jcols)))
+
+
+def rf_explode_tiles_sample(sample_frac: float, seed: int, *tile_cols: Column_type) -> Column:
+ """Create a row for a sample of cells in Tile columns."""
+ jfcn = RFContext.active().lookup("rf_explode_tiles_sample")
+ jcols = [_to_java_column(arg) for arg in tile_cols]
+ return Column(jfcn(sample_frac, seed, RFContext.active().list_to_seq(jcols)))
+
+
+def rf_with_no_data(tile_col: Column_type, scalar: Union[int, float]) -> Column:
+ """Assign a `NoData` value to the Tiles in the given Column."""
+ return _apply_scalar_to_tile("rf_with_no_data", tile_col, scalar)
+
+
+def rf_local_add(left_tile_col: Column_type, rhs: Union[float, int, Column_type]) -> Column:
+ """Add two Tiles, or add a scalar to a Tile"""
+ if isinstance(rhs, (float, int)):
+ rhs = lit(rhs)
+ return _apply_column_function("rf_local_add", left_tile_col, rhs)
+
+
+@deprecated(deprecated_in="0.9.0", removed_in="1.0.0", current_version=__version__)
+def rf_local_add_double(tile_col: Column_type, scalar: float) -> Column:
+ """Add a floating point scalar to a Tile"""
+ return _apply_scalar_to_tile("rf_local_add_double", tile_col, scalar)
+
+
+@deprecated(deprecated_in="0.9.0", removed_in="1.0.0", current_version=__version__)
+def rf_local_add_int(tile_col, scalar) -> Column:
+ """Add an integral scalar to a Tile"""
+ return _apply_scalar_to_tile("rf_local_add_int", tile_col, scalar)
+
+
+def rf_local_subtract(left_tile_col: Column_type, rhs: Union[float, int, Column_type]) -> Column:
+ """Subtract two Tiles, or subtract a scalar from a Tile"""
+ if isinstance(rhs, (float, int)):
+ rhs = lit(rhs)
+ return _apply_column_function("rf_local_subtract", left_tile_col, rhs)
+
+
+@deprecated(deprecated_in="0.9.0", removed_in="1.0.0", current_version=__version__)
+def rf_local_subtract_double(tile_col, scalar):
+ """Subtract a floating point scalar from a Tile"""
+ return _apply_scalar_to_tile("rf_local_subtract_double", tile_col, scalar)
+
+
+@deprecated(deprecated_in="0.9.0", removed_in="1.0.0", current_version=__version__)
+def rf_local_subtract_int(tile_col, scalar):
+ """Subtract an integral scalar from a Tile"""
+ return _apply_scalar_to_tile("rf_local_subtract_int", tile_col, scalar)
+
+
+def rf_local_multiply(left_tile_col: Column_type, rhs: Union[float, int, Column_type]) -> Column:
+ """Multiply two Tiles cell-wise, or multiply Tile cells by a scalar"""
+ if isinstance(rhs, (float, int)):
+ rhs = lit(rhs)
+ return _apply_column_function("rf_local_multiply", left_tile_col, rhs)
+
+
+@deprecated(deprecated_in="0.9.0", removed_in="1.0.0", current_version=__version__)
+def rf_local_multiply_double(tile_col, scalar):
+ """Multiply a Tile by a float point scalar"""
+ return _apply_scalar_to_tile("rf_local_multiply_double", tile_col, scalar)
+
+
+@deprecated(deprecated_in="0.9.0", removed_in="1.0.0", current_version=__version__)
+def rf_local_multiply_int(tile_col, scalar):
+ """Multiply a Tile by an integral scalar"""
+ return _apply_scalar_to_tile("rf_local_multiply_int", tile_col, scalar)
+
+
+def rf_local_divide(left_tile_col: Column_type, rhs: Union[float, int, Column_type]) -> Column:
+ """Divide two Tiles cell-wise, or divide a Tile's cell values by a scalar"""
+ if isinstance(rhs, (float, int)):
+ rhs = lit(rhs)
+ return _apply_column_function("rf_local_divide", left_tile_col, rhs)
+
+
+@deprecated(deprecated_in="0.9.0", removed_in="1.0.0", current_version=__version__)
+def rf_local_divide_double(tile_col, scalar):
+ """Divide a Tile by a floating point scalar"""
+ return _apply_scalar_to_tile("rf_local_divide_double", tile_col, scalar)
+
+
+@deprecated(deprecated_in="0.9.0", removed_in="1.0.0", current_version=__version__)
+def rf_local_divide_int(tile_col, scalar):
+ """Divide a Tile by an integral scalar"""
+ return _apply_scalar_to_tile("rf_local_divide_int", tile_col, scalar)
+
+
+def rf_local_less(left_tile_col: Column_type, rhs: Union[float, int, Column_type]) -> Column:
+ """Cellwise less than comparison between two tiles, or with a scalar value"""
+ if isinstance(rhs, (float, int)):
+ rhs = lit(rhs)
+ return _apply_column_function("rf_local_less", left_tile_col, rhs)
+
+
+@deprecated(deprecated_in="0.9.0", removed_in="1.0.0", current_version=__version__)
+def rf_local_less_double(tile_col, scalar):
+ """Return a Tile with values equal 1 if the cell is less than a scalar, otherwise 0"""
+ return _apply_scalar_to_tile("foo", tile_col, scalar)
+
+
+@deprecated(deprecated_in="0.9.0", removed_in="1.0.0", current_version=__version__)
+def rf_local_less_int(tile_col, scalar):
+ """Return a Tile with values equal 1 if the cell is less than a scalar, otherwise 0"""
+ return _apply_scalar_to_tile("rf_local_less_double", tile_col, scalar)
+
+
+def rf_local_less_equal(left_tile_col: Column_type, rhs: Union[float, int, Column_type]) -> Column:
+ """Cellwise less than or equal to comparison between two tiles, or with a scalar value"""
+ if isinstance(rhs, (float, int)):
+ rhs = lit(rhs)
+ return _apply_column_function("rf_local_less_equal", left_tile_col, rhs)
+
+
+@deprecated(deprecated_in="0.9.0", removed_in="1.0.0", current_version=__version__)
+def rf_local_less_equal_double(tile_col, scalar):
+ """Return a Tile with values equal 1 if the cell is less than or equal to a scalar, otherwise 0"""
+ return _apply_scalar_to_tile("rf_local_less_equal_double", tile_col, scalar)
+
+
+@deprecated(deprecated_in="0.9.0", removed_in="1.0.0", current_version=__version__)
+def rf_local_less_equal_int(tile_col, scalar):
+ """Return a Tile with values equal 1 if the cell is less than or equal to a scalar, otherwise 0"""
+ return _apply_scalar_to_tile("rf_local_less_equal_int", tile_col, scalar)
+
+
+def rf_local_greater(left_tile_col: Column, rhs: Union[float, int, Column_type]) -> Column:
+ """Cellwise greater than comparison between two tiles, or with a scalar value"""
+ if isinstance(rhs, (float, int)):
+ rhs = lit(rhs)
+ return _apply_column_function("rf_local_greater", left_tile_col, rhs)
+
+
+@deprecated(deprecated_in="0.9.0", removed_in="1.0.0", current_version=__version__)
+def rf_local_greater_double(tile_col, scalar):
+ """Return a Tile with values equal 1 if the cell is greater than a scalar, otherwise 0"""
+ return _apply_scalar_to_tile("rf_local_greater_double", tile_col, scalar)
+
+
+@deprecated(deprecated_in="0.9.0", removed_in="1.0.0", current_version=__version__)
+def rf_local_greater_int(tile_col, scalar):
+ """Return a Tile with values equal 1 if the cell is greater than a scalar, otherwise 0"""
+ return _apply_scalar_to_tile("rf_local_greater_int", tile_col, scalar)
+
+
+def rf_local_greater_equal(left_tile_col: Column, rhs: Union[float, int, Column_type]) -> Column:
+ """Cellwise greater than or equal to comparison between two tiles, or with a scalar value"""
+ if isinstance(rhs, (float, int)):
+ rhs = lit(rhs)
+ return _apply_column_function("rf_local_greater_equal", left_tile_col, rhs)
+
+
+@deprecated(deprecated_in="0.9.0", removed_in="1.0.0", current_version=__version__)
+def rf_local_greater_equal_double(tile_col, scalar):
+ """Return a Tile with values equal 1 if the cell is greater than or equal to a scalar, otherwise 0"""
+ return _apply_scalar_to_tile("rf_local_greater_equal_double", tile_col, scalar)
+
+
+@deprecated(deprecated_in="0.9.0", removed_in="1.0.0", current_version=__version__)
+def rf_local_greater_equal_int(tile_col, scalar):
+ """Return a Tile with values equal 1 if the cell is greater than or equal to a scalar, otherwise 0"""
+ return _apply_scalar_to_tile("rf_local_greater_equal_int", tile_col, scalar)
+
+
+def rf_local_equal(left_tile_col, rhs: Union[float, int, Column_type]) -> Column:
+ """Cellwise equality comparison between two tiles, or with a scalar value"""
+ if isinstance(rhs, (float, int)):
+ rhs = lit(rhs)
+ return _apply_column_function("rf_local_equal", left_tile_col, rhs)
+
+
+@deprecated(deprecated_in="0.9.0", removed_in="1.0.0", current_version=__version__)
+def rf_local_equal_double(tile_col, scalar):
+ """Return a Tile with values equal 1 if the cell is equal to a scalar, otherwise 0"""
+ return _apply_scalar_to_tile("rf_local_equal_double", tile_col, scalar)
+
+
+@deprecated(deprecated_in="0.9.0", removed_in="1.0.0", current_version=__version__)
+def rf_local_equal_int(tile_col, scalar):
+ """Return a Tile with values equal 1 if the cell is equal to a scalar, otherwise 0"""
+ return _apply_scalar_to_tile("rf_local_equal_int", tile_col, scalar)
+
+
+def rf_local_unequal(left_tile_col, rhs: Union[float, int, Column_type]) -> Column:
+ """Cellwise inequality comparison between two tiles, or with a scalar value"""
+ if isinstance(rhs, (float, int)):
+ rhs = lit(rhs)
+ return _apply_column_function("rf_local_unequal", left_tile_col, rhs)
+
+
+@deprecated(deprecated_in="0.9.0", removed_in="1.0.0", current_version=__version__)
+def rf_local_unequal_double(tile_col, scalar):
+ """Return a Tile with values equal 1 if the cell is not equal to a scalar, otherwise 0"""
+ return _apply_scalar_to_tile("rf_local_unequal_double", tile_col, scalar)
+
+
+@deprecated(deprecated_in="0.9.0", removed_in="1.0.0", current_version=__version__)
+def rf_local_unequal_int(tile_col, scalar):
+ """Return a Tile with values equal 1 if the cell is not equal to a scalar, otherwise 0"""
+ return _apply_scalar_to_tile("rf_local_unequal_int", tile_col, scalar)
+
+
+def rf_local_no_data(tile_col: Column_type) -> Column:
+ """Return a tile with ones where the input is NoData, otherwise zero."""
+ return _apply_column_function("rf_local_no_data", tile_col)
+
+
+def rf_local_data(tile_col: Column_type) -> Column:
+ """Return a tile with zeros where the input is NoData, otherwise one."""
+ return _apply_column_function("rf_local_data", tile_col)
+
+
+def rf_local_is_in(tile_col: Column_type, array: Union[Column_type, List]) -> Column:
+ """Return a tile with cell values of 1 where the `tile_col` cell is in the provided array."""
+ from pyspark.sql.functions import array as sql_array
+
+ if isinstance(array, list):
+ array = sql_array([lit(v) for v in array])
+
+ return _apply_column_function("rf_local_is_in", tile_col, array)
+
+
+def rf_dimensions(tile_col: Column_type) -> Column:
+ """Query the number of (cols, rows) in a Tile."""
+ return _apply_column_function("rf_dimensions", tile_col)
+
+
+def rf_tile_to_array_int(tile_col: Column_type) -> Column:
+ """Flattens Tile into an array of integers."""
+ return _apply_column_function("rf_tile_to_array_int", tile_col)
+
+
+def rf_tile_to_array_double(tile_col: Column_type) -> Column:
+ """Flattens Tile into an array of doubles."""
+ return _apply_column_function("rf_tile_to_array_double", tile_col)
+
+
+def rf_cell_type(tile_col: Column_type) -> Column:
+ """Extract the Tile's cell type"""
+ return _apply_column_function("rf_cell_type", tile_col)
+
+
+def rf_is_no_data_tile(tile_col: Column_type) -> Column:
+ """Report if the Tile is entirely NODDATA cells"""
+ return _apply_column_function("rf_is_no_data_tile", tile_col)
+
+
+def rf_exists(tile_col: Column_type) -> Column:
+ """Returns true if any cells in the tile are true (non-zero and not NoData)"""
+ return _apply_column_function("rf_exists", tile_col)
+
+
+def rf_for_all(tile_col: Column_type) -> Column:
+ """Returns true if all cells in the tile are true (non-zero and not NoData)."""
+ return _apply_column_function("rf_for_all", tile_col)
+
+
+def rf_agg_approx_histogram(tile_col: Column_type) -> Column:
+ """Compute the full column aggregate floating point histogram"""
+ return _apply_column_function("rf_agg_approx_histogram", tile_col)
+
+
+def rf_agg_approx_quantiles(tile_col, probabilities, relative_error=0.00001):
+ """
+ Calculates the approximate quantiles of a tile column of a DataFrame.
+
+ :param tile_col: column to extract cells from.
+ :param probabilities: a list of quantile probabilities. Each number must belong to [0, 1].
+ For example 0 is the minimum, 0.5 is the median, 1 is the maximum.
+ :param relative_error: The relative target precision to achieve (greater than or equal to 0). Default is 0.00001
+ :return: An array of values approximately at the specified `probabilities`
+ """
+
+ _jfn = RFContext.active().lookup("rf_agg_approx_quantiles")
+ _tile_col = _to_java_column(tile_col)
+ return Column(_jfn(_tile_col, probabilities, relative_error))
+
+
+def rf_agg_stats(tile_col: Column_type) -> Column:
+ """Compute the full column aggregate floating point statistics"""
+ return _apply_column_function("rf_agg_stats", tile_col)
+
+
+def rf_agg_mean(tile_col: Column_type) -> Column:
+ """Computes the column aggregate mean"""
+ return _apply_column_function("rf_agg_mean", tile_col)
+
+
+def rf_agg_data_cells(tile_col: Column_type) -> Column:
+ """Computes the number of non-NoData cells in a column"""
+ return _apply_column_function("rf_agg_data_cells", tile_col)
+
+
+def rf_agg_no_data_cells(tile_col: Column_type) -> Column:
+ """Computes the number of NoData cells in a column"""
+ return _apply_column_function("rf_agg_no_data_cells", tile_col)
+
+
+def rf_agg_extent(extent_col):
+ """Compute the aggregate extent over a column"""
+ return _apply_column_function("rf_agg_extent", extent_col)
+
+
+def rf_agg_reprojected_extent(extent_col, src_crs_col, dest_crs):
+ """Compute the aggregate extent over a column, first projecting from the row CRS to the destination CRS."""
+ return Column(
+ RFContext.call(
+ "rf_agg_reprojected_extent",
+ _to_java_column(extent_col),
+ _to_java_column(src_crs_col),
+ CRS(dest_crs).__jvm__,
+ )
+ )
+
+
+def rf_agg_overview_raster(
+ tile_col: Column,
+ cols: int,
+ rows: int,
+ aoi: Extent,
+ tile_extent_col: Column = None,
+ tile_crs_col: Column = None,
+):
+ """Construct an overview raster of size `cols`x`rows` where data in `proj_raster` intersects the
+ `aoi` bound box in web-mercator. Uses bi-linear sampling method."""
+ ctx = RFContext.active()
+ jfcn = ctx.lookup("rf_agg_overview_raster")
+
+ if tile_extent_col is None or tile_crs_col is None:
+ return Column(jfcn(_to_java_column(tile_col), cols, rows, aoi.__jvm__))
+ else:
+ return Column(
+ jfcn(
+ _to_java_column(tile_col),
+ _to_java_column(tile_extent_col),
+ _to_java_column(tile_crs_col),
+ cols,
+ rows,
+ aoi.__jvm__,
+ )
+ )
+
+
+def rf_tile_histogram(tile_col: Column_type) -> Column:
+ """Compute the Tile-wise histogram"""
+ return _apply_column_function("rf_tile_histogram", tile_col)
+
+
+def rf_tile_mean(tile_col: Column_type) -> Column:
+ """Compute the Tile-wise mean"""
+ return _apply_column_function("rf_tile_mean", tile_col)
+
+
+def rf_tile_sum(tile_col: Column_type) -> Column:
+ """Compute the Tile-wise sum"""
+ return _apply_column_function("rf_tile_sum", tile_col)
+
+
+def rf_tile_min(tile_col: Column_type) -> Column:
+ """Compute the Tile-wise minimum"""
+ return _apply_column_function("rf_tile_min", tile_col)
+
+
+def rf_tile_max(tile_col: Column_type) -> Column:
+ """Compute the Tile-wise maximum"""
+ return _apply_column_function("rf_tile_max", tile_col)
+
+
+def rf_tile_stats(tile_col: Column_type) -> Column:
+ """Compute the Tile-wise floating point statistics"""
+ return _apply_column_function("rf_tile_stats", tile_col)
+
+
+def rf_render_ascii(tile_col: Column_type) -> Column:
+ """Render ASCII art of tile"""
+ return _apply_column_function("rf_render_ascii", tile_col)
+
+
+def rf_render_matrix(tile_col: Column_type) -> Column:
+ """Render Tile cell values as numeric values, for debugging purposes"""
+ return _apply_column_function("rf_render_matrix", tile_col)
+
+
+def rf_render_png(
+ red_tile_col: Column_type, green_tile_col: Column_type, blue_tile_col: Column_type
+) -> Column:
+ """Converts columns of tiles representing RGB channels into a PNG encoded byte array."""
+ return _apply_column_function("rf_render_png", red_tile_col, green_tile_col, blue_tile_col)
+
+
+def rf_render_color_ramp_png(tile_col, color_ramp_name):
+ """Converts columns of tiles representing RGB channels into a PNG encoded byte array."""
+ return Column(RFContext.call("rf_render_png", _to_java_column(tile_col), color_ramp_name))
+
+
+def rf_rgb_composite(
+ red_tile_col: Column_type, green_tile_col: Column_type, blue_tile_col: Column_type
+) -> Column:
+ """Converts columns of tiles representing RGB channels into a single RGB packaged tile."""
+ return _apply_column_function("rf_rgb_composite", red_tile_col, green_tile_col, blue_tile_col)
+
+
+def rf_no_data_cells(tile_col: Column_type) -> Column:
+ """Count of NODATA cells"""
+ return _apply_column_function("rf_no_data_cells", tile_col)
+
+
+def rf_data_cells(tile_col: Column_type) -> Column:
+ """Count of cells with valid data"""
+ return _apply_column_function("rf_data_cells", tile_col)
+
+
+def rf_normalized_difference(left_tile_col: Column_type, right_tile_col: Column_type) -> Column:
+ """Compute the normalized difference of two tiles"""
+ return _apply_column_function("rf_normalized_difference", left_tile_col, right_tile_col)
+
+
+def rf_agg_local_max(tile_col: Column_type) -> Column:
+ """Compute the cell-wise/local max operation between Tiles in a column."""
+ return _apply_column_function("rf_agg_local_max", tile_col)
+
+
+def rf_agg_local_min(tile_col: Column_type) -> Column:
+ """Compute the cellwise/local min operation between Tiles in a column."""
+ return _apply_column_function("rf_agg_local_min", tile_col)
+
+
+def rf_agg_local_mean(tile_col: Column_type) -> Column:
+ """Compute the cellwise/local mean operation between Tiles in a column."""
+ return _apply_column_function("rf_agg_local_mean", tile_col)
+
+
+def rf_agg_local_data_cells(tile_col: Column_type) -> Column:
+ """Compute the cellwise/local count of non-NoData cells for all Tiles in a column."""
+ return _apply_column_function("rf_agg_local_data_cells", tile_col)
+
+
+def rf_agg_local_no_data_cells(tile_col: Column_type) -> Column:
+ """Compute the cellwise/local count of NoData cells for all Tiles in a column."""
+ return _apply_column_function("rf_agg_local_no_data_cells", tile_col)
+
+
+def rf_agg_local_stats(tile_col: Column_type) -> Column:
+ """Compute cell-local aggregate descriptive statistics for a column of Tiles."""
+ return _apply_column_function("rf_agg_local_stats", tile_col)
+
+
+def rf_mask(src_tile_col: Column_type, mask_tile_col: Column_type, inverse: bool = False) -> Column:
+ """Where the rf_mask (second) tile contains NODATA, replace values in the source (first) tile with NODATA.
+ If `inverse` is true, replaces values in the source tile with NODATA where the mask tile contains valid data.
+ """
+ if not inverse:
+ return _apply_column_function("rf_mask", src_tile_col, mask_tile_col)
+ else:
+ rf_inverse_mask(src_tile_col, mask_tile_col)
+
+
+def rf_inverse_mask(src_tile_col: Column_type, mask_tile_col: Column_type) -> Column:
+ """Where the rf_mask (second) tile DOES NOT contain NODATA, replace values in the source
+ (first) tile with NODATA."""
+ return _apply_column_function("rf_inverse_mask", src_tile_col, mask_tile_col)
+
+
+def rf_mask_by_value(
+ data_tile: Column_type,
+ mask_tile: Column_type,
+ mask_value: Union[int, float, Column_type],
+ inverse: bool = False,
+) -> Column:
+ """Generate a tile with the values from the data tile, but where cells in the masking tile contain the masking
+ value, replace the data value with NODATA."""
+ if isinstance(mask_value, (int, float)):
+ mask_value = lit(mask_value)
+ jfcn = RFContext.active().lookup("rf_mask_by_value")
+
+ return Column(
+ jfcn(
+ _to_java_column(data_tile),
+ _to_java_column(mask_tile),
+ _to_java_column(mask_value),
+ inverse,
+ )
+ )
+
+
+def rf_mask_by_values(
+ data_tile: Column_type,
+ mask_tile: Column_type,
+ mask_values: Union[List[Union[int, float]], Column_type],
+) -> Column:
+ """Generate a tile with the values from `data_tile`, but where cells in the `mask_tile` are in the `mask_values`
+ list, replace the value with NODATA.
+ """
+ from pyspark.sql.functions import array as sql_array
+
+ if isinstance(mask_values, list):
+ mask_values = sql_array([lit(v) for v in mask_values])
+
+ jfcn = RFContext.active().lookup("rf_mask_by_values")
+ col_args = [_to_java_column(c) for c in [data_tile, mask_tile, mask_values]]
+ return Column(jfcn(*col_args))
+
+
+def rf_inverse_mask_by_value(
+ data_tile: Column_type, mask_tile: Column_type, mask_value: Union[int, float, Column_type]
+) -> Column:
+ """Generate a tile with the values from the data tile, but where cells in the masking tile do not contain the
+ masking value, replace the data value with NODATA."""
+ if isinstance(mask_value, (int, float)):
+ mask_value = lit(mask_value)
+ return _apply_column_function("rf_inverse_mask_by_value", data_tile, mask_tile, mask_value)
+
+
+def rf_mask_by_bit(
+ data_tile: Column_type,
+ mask_tile: Column_type,
+ bit_position: Union[int, Column_type],
+ value_to_mask: Union[int, float, bool, Column_type],
+) -> Column:
+ """Applies a mask using bit values in the `mask_tile`. Working from the right, extract the bit at `bitPosition` from the `maskTile`. In all locations where these are equal to the `valueToMask`, the returned tile is set to NoData, else the original `dataTile` cell value."""
+ if isinstance(bit_position, int):
+ bit_position = lit(bit_position)
+ if isinstance(value_to_mask, (int, float, bool)):
+ value_to_mask = lit(bool(value_to_mask))
+ return _apply_column_function(
+ "rf_mask_by_bit", data_tile, mask_tile, bit_position, value_to_mask
+ )
+
+
+def rf_mask_by_bits(
+ data_tile: Column_type,
+ mask_tile: Column_type,
+ start_bit: Union[int, Column_type],
+ num_bits: Union[int, Column_type],
+ values_to_mask: Union[Iterable[Union[int, float]], Column_type],
+) -> Column:
+ """Applies a mask from blacklisted bit values in the `mask_tile`. Working from the right, the bits from `start_bit` to `start_bit + num_bits` are @ref:[extracted](reference.md#rf_local_extract_bits) from cell values of the `mask_tile`. In all locations where these are in the `mask_values`, the returned tile is set to NoData; otherwise the original `tile` cell value is returned."""
+ if isinstance(start_bit, int):
+ start_bit = lit(start_bit)
+ if isinstance(num_bits, int):
+ num_bits = lit(num_bits)
+ if isinstance(values_to_mask, (tuple, list)):
+ from pyspark.sql.functions import array
+
+ values_to_mask = array([lit(v) for v in values_to_mask])
+
+ return _apply_column_function(
+ "rf_mask_by_bits", data_tile, mask_tile, start_bit, num_bits, values_to_mask
+ )
+
+
+def rf_local_extract_bits(
+ tile: Column_type, start_bit: Union[int, Column_type], num_bits: Union[int, Column_type] = 1
+) -> Column:
+ """Extract value from specified bits of the cells' underlying binary data.
+ * `startBit` is the first bit to consider, working from the right. It is zero indexed.
+ * `numBits` is the number of bits to take moving further to the left."""
+ if isinstance(start_bit, int):
+ start_bit = lit(start_bit)
+ if isinstance(num_bits, int):
+ num_bits = lit(num_bits)
+ return _apply_column_function("rf_local_extract_bits", tile, start_bit, num_bits)
+
+
+def rf_round(tile_col: Column_type) -> Column:
+ """Round cell values to the nearest integer without changing the cell type"""
+ return _apply_column_function("rf_round", tile_col)
+
+
+def rf_local_min(tile_col, min):
+ """Performs cell-wise minimum two tiles or a tile and a scalar."""
+ if isinstance(min, (int, float)):
+ min = lit(min)
+ return _apply_column_function("rf_local_min", tile_col, min)
+
+
+def rf_local_max(tile_col, max):
+ """Performs cell-wise maximum two tiles or a tile and a scalar."""
+ if isinstance(max, (int, float)):
+ max = lit(max)
+ return _apply_column_function("rf_local_max", tile_col, max)
+
+
+def rf_local_clamp(tile_col, min, max):
+ """Return the tile with its values limited to a range defined by min and max, inclusive."""
+ if isinstance(min, (int, float)):
+ min = lit(min)
+ if isinstance(max, (int, float)):
+ max = lit(max)
+ return _apply_column_function("rf_local_clamp", tile_col, min, max)
+
+
+def rf_where(condition, x, y):
+ """Return a tile with cell values chosen from `x` or `y` depending on `condition`.
+ Operates cell-wise in a similar fashion to Spark SQL `when` and `otherwise`."""
+ return _apply_column_function("rf_where", condition, x, y)
+
+
+def rf_standardize(tile, mean=None, stddev=None):
+ """
+ Standardize cell values such that the mean is zero and the standard deviation is one.
+ If specified, the `mean` and `stddev` are applied to all tiles in the column.
+ If not specified, each tile will be standardized according to the statistics of its cell values;
+ this can result in inconsistent values across rows in a tile column.
+ """
+ if isinstance(mean, (int, float)):
+ mean = lit(mean)
+ if isinstance(stddev, (int, float)):
+ stddev = lit(stddev)
+ if mean is None and stddev is None:
+ return _apply_column_function("rf_standardize", tile)
+ if mean is not None and stddev is not None:
+ return _apply_column_function("rf_standardize", tile, mean, stddev)
+ raise ValueError(
+ "Either `mean` or `stddev` should both be specified or omitted in call to rf_standardize."
+ )
+
+
+def rf_rescale(tile, min=None, max=None):
+ """
+ Rescale cell values such that the minimum is zero and the maximum is one. Other values will be linearly interpolated into the range.
+ If specified, the `min` parameter will become the zero value and the `max` parameter will become 1. See @ref:[`rf_agg_stats`](reference.md#rf_agg_stats).
+ Values outside the range will be set to 0 or 1.
+ If `min` and `max` are not specified, the __tile-wise__ minimum and maximum are used; this can result in inconsistent values across rows in a tile column.
+ """
+ if isinstance(min, (int, float)):
+ min = lit(float(min))
+ if isinstance(max, (int, float)):
+ max = lit(float(max))
+ if min is None and max is None:
+ return _apply_column_function("rf_rescale", tile)
+ if min is not None and max is not None:
+ return _apply_column_function("rf_rescale", tile, min, max)
+ raise ValueError(
+ "Either `min` or `max` should both be specified or omitted in call to rf_rescale."
+ )
+
+
+def rf_abs(tile_col: Column_type) -> Column:
+ """Compute the absolute value of each cell"""
+ return _apply_column_function("rf_abs", tile_col)
+
+
+def rf_log(tile_col: Column_type) -> Column:
+ """Performs cell-wise natural logarithm"""
+ return _apply_column_function("rf_log", tile_col)
+
+
+def rf_log10(tile_col: Column_type) -> Column:
+ """Performs cell-wise logartithm with base 10"""
+ return _apply_column_function("rf_log10", tile_col)
+
+
+def rf_log2(tile_col: Column_type) -> Column:
+ """Performs cell-wise logartithm with base 2"""
+ return _apply_column_function("rf_log2", tile_col)
+
+
+def rf_log1p(tile_col: Column_type) -> Column:
+ """Performs natural logarithm of cell values plus one"""
+ return _apply_column_function("rf_log1p", tile_col)
+
+
+def rf_exp(tile_col: Column_type) -> Column:
+ """Performs cell-wise exponential"""
+ return _apply_column_function("rf_exp", tile_col)
+
+
+def rf_exp2(tile_col: Column_type) -> Column:
+ """Compute 2 to the power of cell values"""
+ return _apply_column_function("rf_exp2", tile_col)
+
+
+def rf_exp10(tile_col: Column_type) -> Column:
+ """Compute 10 to the power of cell values"""
+ return _apply_column_function("rf_exp10", tile_col)
+
+
+def rf_expm1(tile_col: Column_type) -> Column:
+ """Performs cell-wise exponential, then subtract one"""
+ return _apply_column_function("rf_expm1", tile_col)
+
+
+def rf_sqrt(tile_col: Column_type) -> Column:
+ """Performs cell-wise square root"""
+ return _apply_column_function("rf_sqrt", tile_col)
+
+
+def rf_identity(tile_col: Column_type) -> Column:
+ """Pass tile through unchanged"""
+ return _apply_column_function("rf_identity", tile_col)
+
+
+def rf_focal_max(
+ tile_col: Column_type,
+ neighborhood: Union[str, Column_type],
+ target: Union[str, Column_type] = "all",
+) -> Column:
+ """Compute the max value in its neighborhood of each cell"""
+ if isinstance(neighborhood, str):
+ neighborhood = lit(neighborhood)
+ if isinstance(target, str):
+ target = lit(target)
+ return _apply_column_function("rf_focal_max", tile_col, neighborhood, target)
+
+
+def rf_focal_mean(
+ tile_col: Column_type,
+ neighborhood: Union[str, Column_type],
+ target: Union[str, Column_type] = "all",
+) -> Column:
+ """Compute the mean value in its neighborhood of each cell"""
+ if isinstance(neighborhood, str):
+ neighborhood = lit(neighborhood)
+ if isinstance(target, str):
+ target = lit(target)
+ return _apply_column_function("rf_focal_mean", tile_col, neighborhood, target)
+
+
+def rf_focal_median(
+ tile_col: Column_type,
+ neighborhood: Union[str, Column_type],
+ target: Union[str, Column_type] = "all",
+) -> Column:
+ """Compute the max in its neighborhood value of each cell"""
+ if isinstance(neighborhood, str):
+ neighborhood = lit(neighborhood)
+ if isinstance(target, str):
+ target = lit(target)
+ return _apply_column_function("rf_focal_median", tile_col, neighborhood, target)
+
+
+def rf_focal_min(
+ tile_col: Column_type,
+ neighborhood: Union[str, Column_type],
+ target: Union[str, Column_type] = "all",
+) -> Column:
+ """Compute the min value in its neighborhood of each cell"""
+ if isinstance(neighborhood, str):
+ neighborhood = lit(neighborhood)
+ if isinstance(target, str):
+ target = lit(target)
+ return _apply_column_function("rf_focal_min", tile_col, neighborhood, target)
+
+
+def rf_focal_mode(
+ tile_col: Column_type,
+ neighborhood: Union[str, Column_type],
+ target: Union[str, Column_type] = "all",
+) -> Column:
+ """Compute the mode value in its neighborhood of each cell"""
+ if isinstance(neighborhood, str):
+ neighborhood = lit(neighborhood)
+ if isinstance(target, str):
+ target = lit(target)
+ return _apply_column_function("rf_focal_mode", tile_col, neighborhood, target)
+
+
+def rf_focal_std_dev(
+ tile_col: Column_type,
+ neighborhood: Union[str, Column_type],
+ target: Union[str, Column_type] = "all",
+) -> Column:
+ """Compute the standard deviation value in its neighborhood of each cell"""
+ if isinstance(neighborhood, str):
+ neighborhood = lit(neighborhood)
+ if isinstance(target, str):
+ target = lit(target)
+ return _apply_column_function("rf_focal_std_dev", tile_col, neighborhood, target)
+
+
+def rf_moransI(
+ tile_col: Column_type,
+ neighborhood: Union[str, Column_type],
+ target: Union[str, Column_type] = "all",
+) -> Column:
+ """Compute moransI in its neighborhood value of each cell"""
+ if isinstance(neighborhood, str):
+ neighborhood = lit(neighborhood)
+ if isinstance(target, str):
+ target = lit(target)
+ return _apply_column_function("rf_focal_moransi", tile_col, neighborhood, target)
+
+
+def rf_aspect(tile_col: Column_type, target: Union[str, Column_type] = "all") -> Column:
+ """Calculates the aspect of each cell in an elevation raster"""
+ if isinstance(target, str):
+ target = lit(target)
+ return _apply_column_function("rf_aspect", tile_col, target)
+
+
+def rf_slope(
+ tile_col: Column_type,
+ z_factor: Union[int, float, Column_type],
+ target: Union[str, Column_type] = "all",
+) -> Column:
+ """Calculates slope of each cell in an elevation raster"""
+ if isinstance(z_factor, (int, float)):
+ z_factor = lit(z_factor)
+ if isinstance(target, str):
+ target = lit(target)
+ return _apply_column_function("rf_slope", tile_col, z_factor, target)
+
+
+def rf_hillshade(
+ tile_col: Column_type,
+ azimuth: Union[int, float, Column_type],
+ altitude: Union[int, float, Column_type],
+ z_factor: Union[int, float, Column_type],
+ target: Union[str, Column_type] = "all",
+) -> Column:
+ """Calculates the hillshade of each cell in an elevation raster"""
+ if isinstance(azimuth, (int, float)):
+ azimuth = lit(azimuth)
+ if isinstance(altitude, (int, float)):
+ altitude = lit(altitude)
+ if isinstance(z_factor, (int, float)):
+ z_factor = lit(z_factor)
+ if isinstance(target, str):
+ target = lit(target)
+ return _apply_column_function("rf_hillshade", tile_col, azimuth, altitude, z_factor, target)
+
+
+def rf_resample(tile_col: Column_type, scale_factor: Union[int, float, Column_type]) -> Column:
+ """Resample tile to different size based on scalar factor or tile whose dimension to match
+ Scalar less than one will downsample tile; greater than one will upsample. Uses nearest-neighbor."""
+ if isinstance(scale_factor, (int, float)):
+ scale_factor = lit(scale_factor)
+ return _apply_column_function("rf_resample", tile_col, scale_factor)
+
+
+def rf_crs(tile_col: Column_type) -> Column:
+ """Get the CRS of a RasterSource or ProjectedRasterTile"""
+ return _apply_column_function("rf_crs", tile_col)
+
+
+def rf_mk_crs(crs_text: str) -> Column:
+ """Resolve CRS from text identifier. Supported registries are EPSG, ESRI, WORLD, NAD83, & NAD27.
+ An example of a valid CRS name is EPSG:3005."""
+ return Column(_context_call("_make_crs_literal", crs_text))
+
+
+def st_extent(geom_col: Column_type) -> Column:
+ """Compute the extent/bbox of a Geometry (a tile with embedded extent and CRS)"""
+ return _apply_column_function("st_extent", geom_col)
+
+
+def rf_extent(proj_raster_col: Column_type) -> Column:
+ """Get the extent of a RasterSource or ProjectedRasterTile (a tile with embedded extent and CRS)"""
+ return _apply_column_function("rf_extent", proj_raster_col)
+
+
+def rf_tile(proj_raster_col: Column_type) -> Column:
+ """Extracts the Tile component of a ProjectedRasterTile (or Tile)."""
+ return _apply_column_function("rf_tile", proj_raster_col)
+
+
+def rf_proj_raster(tile, extent, crs):
+ """
+ Construct a `proj_raster` structure from individual CRS, Extent, and Tile columns
+ """
+ return _apply_column_function("rf_proj_raster", tile, extent, crs)
+
+
+def st_geometry(extent_col: Column_type) -> Column:
+ """Convert the given extent/bbox to a polygon"""
+ return _apply_column_function("st_geometry", extent_col)
+
+
+def rf_geometry(proj_raster_col: Column_type) -> Column:
+ """Get the extent of a RasterSource or ProjectdRasterTile as a Geometry"""
+ return _apply_column_function("rf_geometry", proj_raster_col)
+
+
+def rf_xz2_index(
+ geom_col: Column_type, crs_col: Optional[Column_type] = None, index_resolution: int = 18
+) -> Column:
+ """Constructs a XZ2 index in WGS84 from either a Geometry, Extent, ProjectedRasterTile, or RasterSource and its CRS.
+ For details: https://www.geomesa.org/documentation/user/datastores/index_overview.html"""
+
+ jfcn = RFContext.active().lookup("rf_xz2_index")
+
+ if crs_col is not None:
+ return Column(jfcn(_to_java_column(geom_col), _to_java_column(crs_col), index_resolution))
+ else:
+ return Column(jfcn(_to_java_column(geom_col), index_resolution))
+
+
+def rf_z2_index(
+ geom_col: Column_type, crs_col: Optional[Column_type] = None, index_resolution: int = 18
+) -> Column:
+ """Constructs a Z2 index in WGS84 from either a Geometry, Extent, ProjectedRasterTile, or RasterSource and its CRS.
+ First the native extent is extracted or computed, and then center is used as the indexing location.
+ For details: https://www.geomesa.org/documentation/user/datastores/index_overview.html"""
+
+ jfcn = RFContext.active().lookup("rf_z2_index")
+
+ if crs_col is not None:
+ return Column(jfcn(_to_java_column(geom_col), _to_java_column(crs_col), index_resolution))
+ else:
+ return Column(jfcn(_to_java_column(geom_col), index_resolution))
+
+
+# ------ GeoMesa Functions ------
+
+
+def st_geomFromGeoHash(*args):
+ """"""
+ return _apply_column_function("st_geomFromGeoHash", *args)
+
+
+def st_geomFromWKT(*args):
+ """"""
+ return _apply_column_function("st_geomFromWKT", *args)
+
+
+def st_geomFromWKB(*args):
+ """"""
+ return _apply_column_function("st_geomFromWKB", *args)
+
+
+def st_lineFromText(*args):
+ """"""
+ return _apply_column_function("st_lineFromText", *args)
+
+
+def st_makeBox2D(*args):
+ """"""
+ return _apply_column_function("st_makeBox2D", *args)
+
+
+def st_makeBBox(*args):
+ """"""
+ return _apply_column_function("st_makeBBox", *args)
+
+
+def st_makePolygon(*args):
+ """"""
+ return _apply_column_function("st_makePolygon", *args)
+
+
+def st_makePoint(*args):
+ """"""
+ return _apply_column_function("st_makePoint", *args)
+
+
+def st_makeLine(*args):
+ """"""
+ return _apply_column_function("st_makeLine", *args)
+
+
+def st_makePointM(*args):
+ """"""
+ return _apply_column_function("st_makePointM", *args)
+
+
+def st_mLineFromText(*args):
+ """"""
+ return _apply_column_function("st_mLineFromText", *args)
+
+
+def st_mPointFromText(*args):
+ """"""
+ return _apply_column_function("st_mPointFromText", *args)
+
+
+def st_mPolyFromText(*args):
+ """"""
+ return _apply_column_function("st_mPolyFromText", *args)
+
+
+def st_point(*args):
+ """"""
+ return _apply_column_function("st_point", *args)
+
+
+def st_pointFromGeoHash(*args):
+ """"""
+ return _apply_column_function("st_pointFromGeoHash", *args)
+
+
+def st_pointFromText(*args):
+ """"""
+ return _apply_column_function("st_pointFromText", *args)
+
+
+def st_pointFromWKB(*args):
+ """"""
+ return _apply_column_function("st_pointFromWKB", *args)
+
+
+def st_polygon(*args):
+ """"""
+ return _apply_column_function("st_polygon", *args)
+
+
+def st_polygonFromText(*args):
+ """"""
+ return _apply_column_function("st_polygonFromText", *args)
+
+
+def st_castToPoint(*args):
+ """"""
+ return _apply_column_function("st_castToPoint", *args)
+
+
+def st_castToPolygon(*args):
+ """"""
+ return _apply_column_function("st_castToPolygon", *args)
+
+
+def st_castToLineString(*args):
+ """"""
+ return _apply_column_function("st_castToLineString", *args)
+
+
+def st_byteArray(*args):
+ """"""
+ return _apply_column_function("st_byteArray", *args)
+
+
+def st_boundary(*args):
+ """"""
+ return _apply_column_function("st_boundary", *args)
+
+
+def st_coordDim(*args):
+ """"""
+ return _apply_column_function("st_coordDim", *args)
+
+
+def st_dimension(*args):
+ """"""
+ return _apply_column_function("st_dimension", *args)
+
+
+def st_envelope(*args):
+ """"""
+ return _apply_column_function("st_envelope", *args)
+
+
+def st_exteriorRing(*args):
+ """"""
+ return _apply_column_function("st_exteriorRing", *args)
+
+
+def st_geometryN(*args):
+ """"""
+ return _apply_column_function("st_geometryN", *args)
+
+
+def st_geometryType(*args):
+ """"""
+ return _apply_column_function("st_geometryType", *args)
+
+
+def st_interiorRingN(*args):
+ """"""
+ return _apply_column_function("st_interiorRingN", *args)
+
+
+def st_isClosed(*args):
+ """"""
+ return _apply_column_function("st_isClosed", *args)
+
+
+def st_isCollection(*args):
+ """"""
+ return _apply_column_function("st_isCollection", *args)
+
+
+def st_isEmpty(*args):
+ """"""
+ return _apply_column_function("st_isEmpty", *args)
+
+
+def st_isRing(*args):
+ """"""
+ return _apply_column_function("st_isRing", *args)
+
+
+def st_isSimple(*args):
+ """"""
+ return _apply_column_function("st_isSimple", *args)
+
+
+def st_isValid(*args):
+ """"""
+ return _apply_column_function("st_isValid", *args)
+
+
+def st_numGeometries(*args):
+ """"""
+ return _apply_column_function("st_numGeometries", *args)
+
+
+def st_numPoints(*args):
+ """"""
+ return _apply_column_function("st_numPoints", *args)
+
+
+def st_pointN(*args):
+ """"""
+ return _apply_column_function("st_pointN", *args)
+
+
+def st_x(*args):
+ """"""
+ return _apply_column_function("st_x", *args)
+
+
+def st_y(*args):
+ """"""
+ return _apply_column_function("st_y", *args)
+
+
+def st_asBinary(*args):
+ """"""
+ return _apply_column_function("st_asBinary", *args)
+
+
+def st_asGeoJSON(*args):
+ """"""
+ return _apply_column_function("st_asGeoJSON", *args)
+
+
+def st_asLatLonText(*args):
+ """"""
+ return _apply_column_function("st_asLatLonText", *args)
+
+
+def st_asText(*args):
+ """"""
+ return _apply_column_function("st_asText", *args)
+
+
+def st_geoHash(*args):
+ """"""
+ return _apply_column_function("st_geoHash", *args)
+
+
+def st_bufferPoint(*args):
+ """"""
+ return _apply_column_function("st_bufferPoint", *args)
+
+
+def st_antimeridianSafeGeom(*args):
+ """"""
+ return _apply_column_function("st_antimeridianSafeGeom", *args)
+
+
+def st_translate(*args):
+ """"""
+ return _apply_column_function("st_translate", *args)
+
+
+def st_contains(*args):
+ """"""
+ return _apply_column_function("st_contains", *args)
+
+
+def st_covers(*args):
+ """"""
+ return _apply_column_function("st_covers", *args)
+
+
+def st_crosses(*args):
+ """"""
+ return _apply_column_function("st_crosses", *args)
+
+
+def st_disjoint(*args):
+ """"""
+ return _apply_column_function("st_disjoint", *args)
+
+
+def st_equals(*args):
+ """"""
+ return _apply_column_function("st_equals", *args)
+
+
+def st_intersects(*args):
+ """"""
+ return _apply_column_function("st_intersects", *args)
+
+
+def st_overlaps(*args):
+ """"""
+ return _apply_column_function("st_overlaps", *args)
+
+
+def st_touches(*args):
+ """"""
+ return _apply_column_function("st_touches", *args)
+
+
+def st_within(*args):
+ """"""
+ return _apply_column_function("st_within", *args)
+
+
+def st_relate(*args):
+ """"""
+ return _apply_column_function("st_relate", *args)
+
+
+def st_relateBool(*args):
+ """"""
+ return _apply_column_function("st_relateBool", *args)
+
+
+def st_area(*args):
+ """"""
+ return _apply_column_function("st_area", *args)
+
+
+def st_closestPoint(*args):
+ """"""
+ return _apply_column_function("st_closestPoint", *args)
+
+
+def st_centroid(*args):
+ """"""
+ return _apply_column_function("st_centroid", *args)
+
+
+def st_distance(*args):
+ """"""
+ return _apply_column_function("st_distance", *args)
+
+
+def st_distanceSphere(*args):
+ """"""
+ return _apply_column_function("st_distanceSphere", *args)
+
+
+def st_length(*args):
+ """"""
+ return _apply_column_function("st_length", *args)
+
+
+def st_aggregateDistanceSphere(*args):
+ """"""
+ return _apply_column_function("st_aggregateDistanceSphere", *args)
+
+
+def st_lengthSphere(*args):
+ """"""
+ return _apply_column_function("st_lengthSphere", *args)
diff --git a/pyrasterframes/src/main/python/pyrasterframes/rf_context.py b/python/pyrasterframes/rf_context.py
similarity index 73%
rename from pyrasterframes/src/main/python/pyrasterframes/rf_context.py
rename to python/pyrasterframes/rf_context.py
index 39a470697..0a4703428 100644
--- a/pyrasterframes/src/main/python/pyrasterframes/rf_context.py
+++ b/python/pyrasterframes/rf_context.py
@@ -22,32 +22,45 @@
This module contains access to the jvm SparkContext with RasterFrameLayer support.
"""
+from typing import Any, List, Tuple
+
+from py4j.java_collections import JavaList, JavaMap
+from py4j.java_gateway import JavaMember
from pyspark import SparkContext
+from pyspark.sql import SparkSession
-__all__ = ['RFContext']
+__all__ = ["RFContext"]
class RFContext(object):
"""
Entrypoint to RasterFrames services
"""
- def __init__(self, spark_session):
+
+ def __init__(self, spark_session: SparkSession):
self._spark_session = spark_session
self._gateway = spark_session.sparkContext._gateway
self._jvm = self._gateway.jvm
jsess = self._spark_session._jsparkSession
self._jrfctx = self._jvm.org.locationtech.rasterframes.py.PyRFContext(jsess)
- def list_to_seq(self, py_list):
- conv = self.lookup('_listToSeq')
+ def list_to_seq(self, py_list: List[Any]) -> JavaList:
+ conv = self.lookup("_listToSeq")
return conv(py_list)
- def lookup(self, function_name):
+ def lookup(self, function_name: str) -> JavaMember:
return getattr(self._jrfctx, function_name)
- def build_info(self):
+ def build_info(self) -> JavaMap:
return self._jrfctx.buildInfo()
+ def companion_of(self, classname: str):
+ if not classname.endswith("$"):
+ classname = classname + "$"
+ companion_module = getattr(self._jvm, classname)
+ singleton = getattr(companion_module, "MODULE$")
+ return singleton
+
# NB: Tightly coupled to `org.locationtech.rasterframes.py.PyRFContext._resolveRasterRef`
def _resolve_raster_ref(self, ref_struct):
f = self.lookup("_resolveRasterRef")
@@ -66,9 +79,10 @@ def active():
Get the active Python RFContext and throw an error if it is not enabled for RasterFrames.
"""
sc = SparkContext._active_spark_context
- if not hasattr(sc, '_rf_context'):
+ if not hasattr(sc, "_rf_context"):
raise AttributeError(
- "RasterFrames have not been enabled for the active session. Call 'SparkSession.withRasterFrames()'.")
+ "RasterFrames have not been enabled for the active session. Call 'SparkSession.withRasterFrames()'."
+ )
return sc._rf_context
@staticmethod
@@ -77,9 +91,8 @@ def call(name, *args):
return f(*args)
@staticmethod
- def _jvm_mirror():
+ def jvm():
"""
Get the active Scala PyRFContext and throw an error if it is not enabled for RasterFrames.
"""
- return RFContext.active()._jrfctx
-
+ return RFContext.active()._jvm
diff --git a/python/pyrasterframes/rf_ipython.py b/python/pyrasterframes/rf_ipython.py
new file mode 100644
index 000000000..0f4a4e09a
--- /dev/null
+++ b/python/pyrasterframes/rf_ipython.py
@@ -0,0 +1,315 @@
+#
+# This software is licensed under the Apache 2 license, quoted below.
+#
+# Copyright 2019 Astraea, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may not
+# use this file except in compliance with the License. You may obtain a copy of
+# the License at
+#
+# [http://www.apache.org/licenses/LICENSE-2.0]
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations under
+# the License.
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+from functools import partial
+from typing import Optional, Tuple, Union
+
+import numpy as np
+import pyrasterframes.rf_types
+from matplotlib.axes import Axes
+from pandas import DataFrame
+from pyrasterframes.rf_types import Tile
+from shapely.geometry.base import BaseGeometry
+
+_png_header = bytearray([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A])
+
+
+def plot_tile(
+ tile: Tile,
+ normalize: bool = True,
+ lower_percentile: float = 1.0,
+ upper_percentile: float = 99.0,
+ axis: Optional[Axes] = None,
+ **imshow_args,
+):
+ """
+ Display an image of the tile
+
+ Parameters
+ ----------
+ tile: item to plot
+ normalize: if True, will normalize the data between using
+ lower_percentile and upper_percentile as bounds
+ lower_percentile: between 0 and 100 inclusive.
+ Specifies to clip values below this percentile
+ upper_percentile: between 0 and 100 inclusive.
+ Specifies to clip values above this percentile
+ axis : matplotlib axis object to plot onto. Creates new axis if None
+ imshow_args : parameters to pass into matplotlib.pyplot.imshow
+ see https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.imshow.html
+ Returns
+ -------
+ created or modified axis object
+ """
+ if axis is None:
+ import matplotlib.pyplot as plt
+
+ axis = plt.gca()
+
+ arr = tile.cells
+
+ def normalize_cells(cells: np.ndarray) -> np.ndarray:
+ assert (
+ upper_percentile > lower_percentile
+ ), "invalid upper and lower percentiles {}, {}".format(lower_percentile, upper_percentile)
+ sans_mask = np.array(cells)
+ lower = np.nanpercentile(sans_mask, lower_percentile)
+ upper = np.nanpercentile(sans_mask, upper_percentile)
+ cells_clipped = np.clip(cells, lower, upper)
+ return (cells_clipped - lower) / (upper - lower)
+
+ axis.set_aspect("equal")
+ axis.xaxis.set_ticks([])
+ axis.yaxis.set_ticks([])
+
+ if normalize:
+ cells = normalize_cells(arr)
+ else:
+ cells = arr
+
+ axis.imshow(cells, **imshow_args)
+
+ return axis
+
+
+def tile_to_png(
+ tile: Tile,
+ lower_percentile: float = 1.0,
+ upper_percentile: float = 99.0,
+ title: Optional[str] = None,
+ fig_size: Optional[Tuple[int, int]] = None,
+) -> bytes:
+ """Provide image of Tile."""
+ if tile.cells is None:
+ return None
+
+ import io
+
+ from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
+ from matplotlib.figure import Figure
+
+ # Set up matplotlib objects
+ nominal_size = 2
+ if fig_size is None:
+ fig_size = (nominal_size, nominal_size)
+
+ fig = Figure(figsize=fig_size)
+ canvas = FigureCanvas(fig)
+ axis = fig.add_subplot(1, 1, 1)
+
+ plot_tile(tile, True, lower_percentile, upper_percentile, axis=axis)
+ axis.set_aspect("equal")
+ axis.xaxis.set_ticks([])
+ axis.yaxis.set_ticks([])
+
+ if title is None:
+ axis.set_title(
+ "{}, {}".format(tile.dimensions(), tile.cell_type.__repr__()), fontsize=fig_size[0] * 4
+ ) # compact metadata as title
+ else:
+ axis.set_title(title, fontsize=fig_size[0] * 4) # compact metadata as title
+
+ with io.BytesIO() as output:
+ canvas.print_png(output)
+ return output.getvalue()
+
+
+def tile_to_html(tile: Tile, fig_size: Optional[Tuple[int, int]] = None) -> str:
+ """Provide HTML string representation of Tile image."""
+ import base64
+
+ b64_img_html = '| \n", - " | eod_collection_display_name | \n", - "eod_collection_family | \n", - "eod_collection_family_display_name | \n", - "eod_grid_id | \n", - "created | \n", - "datetime | \n", - "eo_cloud_cover | \n", - "eo_constellation | \n", - "eo_epsg | \n", - "eo_gsd | \n", - "... | \n", - "B2 | \n", - "BQA | \n", - "B4 | \n", - "B1 | \n", - "B8 | \n", - "B11 | \n", - "collection | \n", - "geometry | \n", - "id | \n", - "target | \n", - "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", - "Landsat 8 | \n", - "landsat8 | \n", - "Landsat 8 | \n", - "WRS2-030027 | \n", - "2019-08-19T20:54:33.413548Z | \n", - "2018-07-17T17:15:57.1536740Z | \n", - "1.49 | \n", - "landsat-8 | \n", - "32614 | \n", - "30.0 | \n", - "... | \n", - "https://landsat-pds.s3.us-west-2.amazonaws.com... | \n", - "https://landsat-pds.s3.us-west-2.amazonaws.com... | \n", - "https://landsat-pds.s3.us-west-2.amazonaws.com... | \n", - "https://landsat-pds.s3.us-west-2.amazonaws.com... | \n", - "https://landsat-pds.s3.us-west-2.amazonaws.com... | \n", - "https://landsat-pds.s3.us-west-2.amazonaws.com... | \n", - "landsat8_l1tp | \n", - "(POLYGON ((-98.62404379679178 46.4012557977134... | \n", - "LC08_L1TP_030027_20180717_20180730_01_T1_L1TP | \n", - "file:///tmp/scene_30_27_target_utm.tif | \n", - "
1 rows × 33 columns
\n", - "'
+ png_bits = tile_to_png(tile, fig_size=fig_size)
+ b64_png = base64.b64encode(png_bits).decode("utf-8").replace("\n", "")
+ return b64_img_html.format(b64_png)
+
+
+def binary_to_html(blob) -> Union[str, bytearray]:
+ """When using rf_render_png, the result from the JVM is a byte string with special PNG header
+ Look for this header and return base64 encoded HTML for Jupyter display
+ """
+ import base64
+
+ if blob[:8] == _png_header:
+ b64_img_html = ''
+ b64_png = base64.b64encode(blob).decode("utf-8").replace("\n", "")
+ return b64_img_html.format(b64_png)
+ else:
+ return blob
+
+
+def pandas_df_to_html(df: DataFrame) -> Optional[str]:
+ """Provide HTML formatting for pandas.DataFrame with rf_types.Tile in the columns."""
+ import pandas as pd
+
+ # honor the existing options on display
+ if not pd.get_option("display.notebook_repr_html"):
+ return None
+
+ default_max_colwidth = pd.get_option(
+ "display.max_colwidth"
+ ) # we'll try to politely put it back
+
+ if len(df) == 0:
+ return df._repr_html_()
+
+ tile_cols = []
+ geom_cols = []
+ bytearray_cols = []
+ for c in df.columns:
+ if isinstance(
+ df.iloc[0][c], pyrasterframes.rf_types.Tile
+ ): # if the first is a Tile try formatting
+ tile_cols.append(c)
+ elif isinstance(df.iloc[0][c], BaseGeometry): # if the first is a Geometry try formatting
+ geom_cols.append(c)
+ elif isinstance(df.iloc[0][c], bytearray):
+ bytearray_cols.append(c)
+
+ def _safe_tile_to_html(t):
+ if isinstance(t, pyrasterframes.rf_types.Tile):
+ return tile_to_html(t, fig_size=(2, 2))
+ else:
+ # handles case where objects in a column are not all Tile type
+ return t.__repr__()
+
+ def _safe_geom_to_html(g):
+ if isinstance(g, BaseGeometry):
+ wkt = g.wkt
+ if len(wkt) > default_max_colwidth:
+ return wkt[: default_max_colwidth - 3] + "..."
+ else:
+ return wkt
+ else:
+ return g.__repr__()
+
+ def _safe_bytearray_to_html(b):
+ if isinstance(b, bytearray):
+ return binary_to_html(b)
+ else:
+ return b.__repr__()
+
+ # dict keyed by column with custom rendering function
+ formatter = {c: _safe_tile_to_html for c in tile_cols}
+ formatter.update({c: _safe_geom_to_html for c in geom_cols})
+ formatter.update({c: _safe_bytearray_to_html for c in bytearray_cols})
+
+ # This is needed to avoid our tile being rendered as `| rf_crs(proj_raster) | rf_extent(proj_raster) | rf_aspect(proj_raster) | rf_slope(proj_raster, 1) | rf_hillshade(proj_raster, 315, 45, 1) |
|---|---|---|---|---|
| utm-CS | {240929.2154, 4398599.0319, 256289.2154, 4401599.0319} | |||
| utm-CS | {210209.2154, 4432319.0319, 225569.2154, 4447679.0319} | |||
| utm-CS | {256289.2154, 4416959.0319, 271649.2154, 4432319.0319} | |||
| utm-CS | {271649.2154, 4509119.0319, 287009.2154, 4524479.0319} | |||
| utm-CS | {333089.2154, 4398599.0319, 341969.2154, 4401599.0319} |
| rf_local_add(proj_raster, 3) |
|---|
| proj_raster_path | footprint |
|---|---|
| https://modis-pds.s3.amazonaws.com/MCD43A4.006/11/08/2019059/MCD43A4.A2019059.h11v08.006.2019072203257_B02.TIF | POLYGON ((-70.85954815687087 8.933333332533772, -71.07986282542622 9.999999999104968, -69.99674110618135 9.999999999104968, -69.77978361352781 8.933333332533772, -70.85954815687087 8.933333332533772)) |
| https://modis-pds.s3.amazonaws.com/MCD43A4.006/11/08/2019059/MCD43A4.A2019059.h11v08.006.2019072203257_B02.TIF | POLYGON ((-69.77978361352781 8.933333332533772, -69.99674110618135 9.999999999104968, -68.91361938693649 9.999999999104968, -68.70001907018472 8.933333332533772, -69.77978361352781 8.933333332533772)) |
| https://modis-pds.s3.amazonaws.com/MCD43A4.006/11/08/2019059/MCD43A4.A2019059.h11v08.006.2019072203257_B02.TIF | POLYGON ((-68.70001907018474 8.933333332533772, -68.9136193869365 9.999999999104968, -67.83049766769162 9.999999999104968, -67.62025452684165 8.933333332533772, -68.70001907018474 8.933333332533772)) |
| https://modis-pds.s3.amazonaws.com/MCD43A4.006/11/08/2019059/MCD43A4.A2019059.h11v08.006.2019072203257_B02.TIF | POLYGON ((-67.62025452684165 8.933333332533772, -67.83049766769162 9.999999999104968, -66.74737594844675 9.999999999104968, -66.54048998349857 8.933333332533772, -67.62025452684165 8.933333332533772)) |
| https://modis-pds.s3.amazonaws.com/MCD43A4.006/11/08/2019059/MCD43A4.A2019059.h11v08.006.2019072203257_B02.TIF | POLYGON ((-66.54048998349859 8.933333332533772, -66.74737594844676 9.999999999104968, -65.66425422920187 9.999999999104968, -65.4607254401555 8.933333332533772, -66.54048998349859 8.933333332533772)) |
"
- ],
- "text/plain": [
- "
| id | collection | geometry |
|---|---|---|
| LC08_L1TP_232093_20210716_20210717_01_T1 | landsat-8-l1-c1 | POLYGON ((-74.64766964714028 -46.3435154... |
| LC08_L1TP_232092_20210716_20210717_01_T1 | landsat-8-l1-c1 | POLYGON ((-74.07682865409966 -44.9166888... |
| LC08_L1TP_232091_20210716_20210717_01_T1 | landsat-8-l1-c1 | POLYGON ((-73.54155930424828 -43.4885910... |
| LC08_L1TP_232090_20210716_20210717_01_T1 | landsat-8-l1-c1 | POLYGON ((-73.02667875381594 -42.0589406... |
| LC08_L1TP_232089_20210716_20210717_01_T1 | landsat-8-l1-c1 | POLYGON ((-72.67424121162182 -40.6804236... |
| rf_crs(band) | rf_extent(band) | rf_aspect(band, all) | rf_slope(band, 1, all) | rf_hillshade(band, 315, 45, 1, all) |
|---|---|---|---|---|
| utm-CS | {488445.0, -5335365.0, 503805.0, -5320005.0} | |||
| utm-CS | {657405.0, -5335365.0, 672765.0, -5320005.0} | |||
| utm-CS | {688125.0, -5335365.0, 703485.0, -5320005.0} | |||
| utm-CS | {642045.0, -5197125.0, 657405.0, -5181765.0} | |||
| utm-CS | {549885.0, -5366085.0, 565245.0, -5350725.0} |
| rf_crs(band) | rf_extent(band) | rf_aspect(band, data) | rf_slope(band, 1, data) | rf_hillshade(band, 315, 45, 1, data) |
|---|---|---|---|---|
| utm-CS | {488445.0, -5335365.0, 503805.0, -5320005.0} | |||
| utm-CS | {657405.0, -5335365.0, 672765.0, -5320005.0} | |||
| utm-CS | {688125.0, -5335365.0, 703485.0, -5320005.0} | |||
| utm-CS | {642045.0, -5197125.0, 657405.0, -5181765.0} | |||
| utm-CS | {549885.0, -5366085.0, 565245.0, -5350725.0} |
| proj_raster_path | tile | crs | ext |
|---|---|---|---|
| https://modis-pds.s3.amazonaws.com/MCD43A4.006/31/11/2017158/MCD43A4.A2017158.h31v11.006.2017171203421_B01.TIF | [int16ud32767, (256,256), [1225,1244,1247,1222,1189,1216,1206,1185,1132,1040,...,1575,1489,1281,1189,1202,1145,1171,1189,1297,1382]] | [+proj=sinu +lon_0=0 +x_0=0 +y_0=0 +a=6371007.181 +b=6371007.181 +units=m +no_defs ] | [1.4455356755667E7, -2342509.0947640934, 1.4573964811098093E7, -2223901.039333] |
| https://modis-pds.s3.amazonaws.com/MCD43A4.006/31/11/2017158/MCD43A4.A2017158.h31v11.006.2017171203421_B01.TIF | [int16ud32767, (256,256), [1140,1227,1147,1106,1026,994,1047,1020,1174,1348,...,1793,1743,1685,1688,1706,1727,1766,1689,1561,1515]] | [+proj=sinu +lon_0=0 +x_0=0 +y_0=0 +a=6371007.181 +b=6371007.181 +units=m +no_defs ] | [1.4573964811098093E7, -2342509.0947640934, 1.4692572866529187E7, -2223901.039333] |
| https://modis-pds.s3.amazonaws.com/MCD43A4.006/31/11/2017158/MCD43A4.A2017158.h31v11.006.2017171203421_B01.TIF | [int16ud32767, (256,256), [1546,1445,1329,1539,1653,1576,1533,1603,1610,1584,...,1399,1434,1330,1429,1470,1451,1422,1407,1369,1310]] | [+proj=sinu +lon_0=0 +x_0=0 +y_0=0 +a=6371007.181 +b=6371007.181 +units=m +no_defs ] | [1.4692572866529185E7, -2342509.0947640934, 1.4811180921960281E7, -2223901.039333] |
| https://modis-pds.s3.amazonaws.com/MCD43A4.006/31/11/2017158/MCD43A4.A2017158.h31v11.006.2017171203421_B01.TIF | [int16ud32767, (256,256), [1765,1675,1704,1674,1665,1685,1551,1556,1576,1626,...,1814,1768,1771,1812,1825,1773,1737,1728,1734,1684]] | [+proj=sinu +lon_0=0 +x_0=0 +y_0=0 +a=6371007.181 +b=6371007.181 +units=m +no_defs ] | [1.481118092196028E7, -2342509.0947640934, 1.4929788977391373E7, -2223901.039333] |
| https://modis-pds.s3.amazonaws.com/MCD43A4.006/31/11/2017158/MCD43A4.A2017158.h31v11.006.2017171203421_B01.TIF | [int16ud32767, (256,256), [1171,1272,1306,1294,1202,1065,998,971,976,1188,...,1455,1481,1458,1469,1449,1392,1227,1085,1102,1091]] | [+proj=sinu +lon_0=0 +x_0=0 +y_0=0 +a=6371007.181 +b=6371007.181 +units=m +no_defs ] | [1.4929788977391373E7, -2342509.0947640934, 1.5048397032822467E7, -2223901.039333] |
\n",
- "\n",
- "\n",
- " \n",
- "
\n",
- "
"
- ],
- "text/plain": [
- " proj_raster_path \\\n",
- "0 https://modis-pds.s3.amazonaws.com/MCD43A4.006... \n",
- "1 https://modis-pds.s3.amazonaws.com/MCD43A4.006... \n",
- "2 https://modis-pds.s3.amazonaws.com/MCD43A4.006... \n",
- "3 https://modis-pds.s3.amazonaws.com/MCD43A4.006... \n",
- "\n",
- " tile \\\n",
- "0 Tile(dimensions=[256, 256], cell_type=CellType... \n",
- "1 Tile(dimensions=[256, 256], cell_type=CellType... \n",
- "2 Tile(dimensions=[256, 256], cell_type=CellType... \n",
- "3 Tile(dimensions=[256, 256], cell_type=CellType... \n",
- "\n",
- " crs \\\n",
- "0 (+proj=sinu +lon_0=0 +x_0=0 +y_0=0 +a=6371007.... \n",
- "1 (+proj=sinu +lon_0=0 +x_0=0 +y_0=0 +a=6371007.... \n",
- "2 (+proj=sinu +lon_0=0 +x_0=0 +y_0=0 +a=6371007.... \n",
- "3 (+proj=sinu +lon_0=0 +x_0=0 +y_0=0 +a=6371007.... \n",
- "\n",
- " ext \n",
- "0 (14455356.755667, -2342509.0947640934, 1457396... \n",
- "1 (14573964.811098093, -2342509.0947640934, 1469... \n",
- "2 (14692572.866529185, -2342509.0947640934, 1481... \n",
- "3 (14811180.92196028, -2342509.0947640934, 14929... "
- ]
- },
- "execution_count": 9,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "pandas_df = df.limit(10).toPandas()\n",
- "pandas_df.head(4)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "You still get the default string representatation of a `pandas.Series`"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {
- "scrolled": true
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "proj_raster_path https://modis-pds.s3.amazonaws.com/MCD43A4.006...\n",
- "tile Tile(dimensions=[256, 256], cell_type=CellType...\n",
- "crs (+proj=sinu +lon_0=0 +x_0=0 +y_0=0 +a=6371007....\n",
- "ext (15404221.199115746, -2342509.0947640934, 1552...\n",
- "Name: 8, dtype: object"
- ]
- },
- "execution_count": 10,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "pandas_df.iloc[8]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "metadata": {
- "scrolled": true
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "0 Tile(dimensions=[256, 256], cell_type=CellType...\n",
- "1 Tile(dimensions=[256, 256], cell_type=CellType...\n",
- "2 Tile(dimensions=[256, 256], cell_type=CellType...\n",
- "3 Tile(dimensions=[256, 256], cell_type=CellType...\n",
- "4 Tile(dimensions=[256, 256], cell_type=CellType...\n",
- "5 Tile(dimensions=[256, 256], cell_type=CellType...\n",
- "6 Tile(dimensions=[256, 256], cell_type=CellType...\n",
- "7 Tile(dimensions=[256, 256], cell_type=CellType...\n",
- "8 Tile(dimensions=[256, 256], cell_type=CellType...\n",
- "9 Tile(dimensions=[96, 256], cell_type=CellType(...\n",
- "Name: tile, dtype: object"
- ]
- },
- "execution_count": 11,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "pandas_df.tile"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "And nothing different happens for a `pandas.DataFrame` that doesn't have a `Tile` in it."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/html": [
- "| \n", - " | proj_raster_path | \n", - "tile | \n", - "crs | \n", - "ext | \n", - "
|---|---|---|---|---|
| 0 | \n", - "https://modis-pds.s3.amazonaws.com/MCD43A4.006/31/11/2017158/MCD43A4.A2017158.h31v11.006.2017171203421_B01.TIF | \n", - "(+proj=sinu +lon_0=0 +x_0=0 +y_0=0 +a=6371007.181 +b=6371007.181 +units=m +no_defs ,) | \n", - "(14455356.755667, -2342509.0947640934, 14573964.811098093, -2223901.039333) | \n", - "|
| 1 | \n", - "https://modis-pds.s3.amazonaws.com/MCD43A4.006/31/11/2017158/MCD43A4.A2017158.h31v11.006.2017171203421_B01.TIF | \n", - "(+proj=sinu +lon_0=0 +x_0=0 +y_0=0 +a=6371007.181 +b=6371007.181 +units=m +no_defs ,) | \n", - "(14573964.811098093, -2342509.0947640934, 14692572.866529187, -2223901.039333) | \n", - "|
| 2 | \n", - "https://modis-pds.s3.amazonaws.com/MCD43A4.006/31/11/2017158/MCD43A4.A2017158.h31v11.006.2017171203421_B01.TIF | \n", - "(+proj=sinu +lon_0=0 +x_0=0 +y_0=0 +a=6371007.181 +b=6371007.181 +units=m +no_defs ,) | \n", - "(14692572.866529185, -2342509.0947640934, 14811180.921960281, -2223901.039333) | \n", - "|
| 3 | \n", - "https://modis-pds.s3.amazonaws.com/MCD43A4.006/31/11/2017158/MCD43A4.A2017158.h31v11.006.2017171203421_B01.TIF | \n", - "(+proj=sinu +lon_0=0 +x_0=0 +y_0=0 +a=6371007.181 +b=6371007.181 +units=m +no_defs ,) | \n", - "(14811180.92196028, -2342509.0947640934, 14929788.977391373, -2223901.039333) | \n", - "
\n",
- "\n",
- "\n",
- " \n",
- "
\n",
- "
"
- ],
- "text/plain": [
- " iata airport city state country \\\n",
- "0 ORD Chicago O'Hare International Chicago IL USA \n",
- "1 ATL William B Hartsfield-Atlanta Intl Atlanta GA USA \n",
- "2 DFW Dallas-Fort Worth International Dallas-Fort Worth TX USA \n",
- "3 PHX Phoenix Sky Harbor International Phoenix AZ USA \n",
- "4 DEN Denver Intl Denver CO USA \n",
- "5 IAH George Bush Intercontinental Houston TX USA \n",
- "6 SFO San Francisco International San Francisco CA USA \n",
- "7 LAX Los Angeles International Los Angeles CA USA \n",
- "8 MCO Orlando International Orlando FL USA \n",
- "9 CLT Charlotte/Douglas International Charlotte NC USA \n",
- "\n",
- " lat long cnt \n",
- "0 41.979595 -87.904464 25129 \n",
- "1 33.640444 -84.426944 21925 \n",
- "2 32.895951 -97.037200 20662 \n",
- "3 33.434167 -112.008056 17290 \n",
- "4 39.858408 -104.667002 13781 \n",
- "5 29.980472 -95.339722 13223 \n",
- "6 37.619002 -122.374843 12016 \n",
- "7 33.942536 -118.408074 11797 \n",
- "8 28.428889 -81.316028 10536 \n",
- "9 35.214011 -80.943126 10490 "
- ]
- },
- "execution_count": 12,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "import pandas\n",
- "pandas.read_csv('https://raw.githubusercontent.com/plotly/datasets/master/2011_february_us_airport_traffic.csv').head(10)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.7.3"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/version.sbt b/version.sbt
deleted file mode 100644
index 58771512b..000000000
--- a/version.sbt
+++ /dev/null
@@ -1 +0,0 @@
-version in ThisBuild := "0.8.4-SNAPSHOT"
| \n", - " | iata | \n", - "airport | \n", - "city | \n", - "state | \n", - "country | \n", - "lat | \n", - "long | \n", - "cnt | \n", - "
|---|---|---|---|---|---|---|---|---|
| 0 | \n", - "ORD | \n", - "Chicago O'Hare International | \n", - "Chicago | \n", - "IL | \n", - "USA | \n", - "41.979595 | \n", - "-87.904464 | \n", - "25129 | \n", - "
| 1 | \n", - "ATL | \n", - "William B Hartsfield-Atlanta Intl | \n", - "Atlanta | \n", - "GA | \n", - "USA | \n", - "33.640444 | \n", - "-84.426944 | \n", - "21925 | \n", - "
| 2 | \n", - "DFW | \n", - "Dallas-Fort Worth International | \n", - "Dallas-Fort Worth | \n", - "TX | \n", - "USA | \n", - "32.895951 | \n", - "-97.037200 | \n", - "20662 | \n", - "
| 3 | \n", - "PHX | \n", - "Phoenix Sky Harbor International | \n", - "Phoenix | \n", - "AZ | \n", - "USA | \n", - "33.434167 | \n", - "-112.008056 | \n", - "17290 | \n", - "
| 4 | \n", - "DEN | \n", - "Denver Intl | \n", - "Denver | \n", - "CO | \n", - "USA | \n", - "39.858408 | \n", - "-104.667002 | \n", - "13781 | \n", - "
| 5 | \n", - "IAH | \n", - "George Bush Intercontinental | \n", - "Houston | \n", - "TX | \n", - "USA | \n", - "29.980472 | \n", - "-95.339722 | \n", - "13223 | \n", - "
| 6 | \n", - "SFO | \n", - "San Francisco International | \n", - "San Francisco | \n", - "CA | \n", - "USA | \n", - "37.619002 | \n", - "-122.374843 | \n", - "12016 | \n", - "
| 7 | \n", - "LAX | \n", - "Los Angeles International | \n", - "Los Angeles | \n", - "CA | \n", - "USA | \n", - "33.942536 | \n", - "-118.408074 | \n", - "11797 | \n", - "
| 8 | \n", - "MCO | \n", - "Orlando International | \n", - "Orlando | \n", - "FL | \n", - "USA | \n", - "28.428889 | \n", - "-81.316028 | \n", - "10536 | \n", - "
| 9 | \n", - "CLT | \n", - "Charlotte/Douglas International | \n", - "Charlotte | \n", - "NC | \n", - "USA | \n", - "35.214011 | \n", - "-80.943126 | \n", - "10490 | \n", - "