Spark 3+ support #1167

Expected Behavior

Can use spark 3+ clusters to run jobs.

Current Behavior

If used together with spark 3+, there is an exception

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/sources/v2/StreamWriteSupport

which is caused by old (2+) spark dependencies for IngestionJob.

Steps to reproduce

I guess pip install pyspark>=3 and start any job will be enough

Specifications

• Version: 0.8.0
• Platform: linux
• Subsystem:

Possible Solution

Bump spark.version in dependencies and update job code accordingly.