Skip to content

[FlinkRunner] Ingestion job tries to connect to every store available in core #20

Description

@zhilingc

Expected Behavior

Should only connect to the relevant stores specified in the FeatureSpec.

Current Behavior

Will pull and attempt to connect to all the stores once the job is sent to the remote Flink cluster. If there are stores it cannot connect to, the entire job will fail.

Steps to reproduce

  • Register a storage that cannot be reached (e.g. 1.2.3.4:6379)
  • Run a job (not using that store) on a remote flink cluster

Possible Solution

Since we only pull specs at the graph building stage now, ingestion shouldn't have to retrieve and cache all available stores, only the stores it needs.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions