Expected Behavior
The Dataflow runner in Feast Core should start Ingestion Jobs without encountering any errors.
Current Behavior
Currently, the Dataflow runner encounters this error when starting Ingestion Jobs encounters this exception when starting Dataflow Jobs:
020-04-22 11:27:16.266 ERROR Zhu-Zhanyan --- [ol-453-thread-1] f.c.j.d.DataflowJobManager : Error submitting job
java.lang.IllegalStateException: no translator registered for primitive transform WriteFailureMetricsTransform at node WriteFailureMetrics
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState(Preconditions.java:823)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.visitPrimitiveTransform(DataflowPipelineTranslator.java:468)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:665)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:317)
at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:251)
at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:460)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.translate(DataflowPipelineTranslator.java:414)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translate(DataflowPipelineTranslator.java:173)
at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:763)
at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:186)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:315)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:301)
at feast.ingestion.ImportJob.runPipeline(ImportJob.java:171)
at feast.core.job.dataflow.DataflowJobManager.runPipeline(DataflowJobManager.java:298)
at feast.core.job.dataflow.DataflowJobManager.submitDataflowJob(DataflowJobManager.java:248)
at feast.core.job.dataflow.DataflowJobManager.startJob(DataflowJobManager.java:123)
at feast.core.job.JobUpdateTask.startJob(JobUpdateTask.java:160)
at feast.core.job.JobUpdateTask.lambda$call$2(JobUpdateTask.java:113)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:830)
2020-04-22 11:27:16.267 ERROR Zhu-Zhanyan --- [ol-453-thread-1] f.c.j.JobUpdateTask : Error running ingestion job: java.lang.IllegalStateException: no translator registered for primitive transform WriteFailureMetricsTransform at node WriteFailureMetrics
Steps to reproduce
- Configure Feast Core to use the Dataflow runner in
application.yml
- Start Feast Core, Feast Serving and their dependencies.
- Use
ApplyFeatureSet to create a FeatureSet in Feast Core.
- Wait for
JobCoordinatorService to pick up the new FeatureSet and create an ingestion job for it.
- Error occurs when attempting to create Dataflow Job
Specifications
This error occurs in Feast Core built on the current master branch.
Expected Behavior
The Dataflow runner in Feast Core should start Ingestion Jobs without encountering any errors.
Current Behavior
Currently, the Dataflow runner encounters this error when starting Ingestion Jobs encounters this exception when starting Dataflow Jobs:
Steps to reproduce
application.ymlApplyFeatureSetto create aFeatureSetin Feast Core.JobCoordinatorServiceto pick up the newFeatureSetand create an ingestion job for it.Specifications
This error occurs in Feast Core built on the current master branch.