Skip to content

fix: Redundant feature materialization and premature incremental materialization timestamp updates #3789

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
achals merged 10 commits into from
Oct 25, 2023
Merged

fix: Redundant feature materialization and premature incremental materialization timestamp updates #3789

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
948c7fb
SAASMLOPS-767 wait for jobs to complete
james-crabtree-sp Sep 12, 2023
0ec030f
SAASMLOPS-805 Stopgap change to fix duplicate materialization of data
james-crabtree-sp Sep 19, 2023
71c02f0
SAASMLOPS-805 save BYTEWAX_REPLICAS=1
james-crabtree-sp Sep 19, 2023
2313c1d
SAASMLOPS-809 fix bytewax workers so they only process a single file …
james-crabtree-sp Sep 20, 2023
5f97ed0
SAASMLOPS-833 add configurable job timeout (#7)
james-crabtree-sp Sep 29, 2023
65c039a
develop Run large materializations in batches of pods
james-crabtree-sp Oct 6, 2023
cbda2bf
master Set job_batch_size at least equal to max_parallelism
james-crabtree-sp Oct 12, 2023
c35b0c1
master clarity max_parallelism description
james-crabtree-sp Oct 12, 2023
5a8018d
master resolve bug that causes materialization to continue after job …
james-crabtree-sp Oct 25, 2023
938b903
master resolve bug causing pod logs to not be printed
james-crabtree-sp Oct 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
master resolve bug causing pod logs to not be printed 
Signed-off-by: James Crabtree <james.crabtree@sailpoint.com>
  • Loading branch information
@james-crabtree-sp
james-crabtree-sp committed Oct 25, 2023
commit 938b9030e3e78078e587c0261cde01600a146495
9 changes: 6 additions & 3 deletions sdk/python/feast/infra/materialization/contrib/bytewax/bytewax_materialization_engine.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -248,9 +248,6 @@ def _await_path_materialization(
f"(of {total_pods}) complete with status {job.status()}"
)
except BaseException as e:
if self.batch_engine_config.print_pod_logs_on_failure:
self._print_pod_logs(job.job_id(), feature_view, batch_start)

logger.info(f"Deleting job {job.job_id()}")
try:
self.batch_v1.delete_namespaced_job(job.job_id(), self.namespace)
Expand All @@ -268,6 +265,12 @@ def _await_path_materialization(
f"Could not delete configmap due to API Error: {ae.body}"
)

if (
job.status() == MaterializationJobStatus.ERROR
and self.batch_engine_config.print_pod_logs_on_failure
):
self._print_pod_logs(job.job_id(), feature_view, batch_start)

return job

def _print_pod_logs(self, job_id, feature_view, offset=0):
Expand Down