Skip to content

fix: Redundant feature materialization and premature incremental materialization timestamp updates #3789

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
achals merged 10 commits into from
Oct 25, 2023
Merged

fix: Redundant feature materialization and premature incremental materialization timestamp updates #3789

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
948c7fb
SAASMLOPS-767 wait for jobs to complete
james-crabtree-sp Sep 12, 2023
0ec030f
SAASMLOPS-805 Stopgap change to fix duplicate materialization of data
james-crabtree-sp Sep 19, 2023
71c02f0
SAASMLOPS-805 save BYTEWAX_REPLICAS=1
james-crabtree-sp Sep 19, 2023
2313c1d
SAASMLOPS-809 fix bytewax workers so they only process a single file …
james-crabtree-sp Sep 20, 2023
5f97ed0
SAASMLOPS-833 add configurable job timeout (#7)
james-crabtree-sp Sep 29, 2023
65c039a
develop Run large materializations in batches of pods
james-crabtree-sp Oct 6, 2023
cbda2bf
master Set job_batch_size at least equal to max_parallelism
james-crabtree-sp Oct 12, 2023
c35b0c1
master clarity max_parallelism description
james-crabtree-sp Oct 12, 2023
5a8018d
master resolve bug that causes materialization to continue after job …
james-crabtree-sp Oct 25, 2023
938b903
master resolve bug causing pod logs to not be printed
james-crabtree-sp Oct 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
master Set job_batch_size at least equal to max_parallelism 
Signed-off-by: James Crabtree <james.crabtree@sailpoint.com>
  • Loading branch information
@james-crabtree-sp
james-crabtree-sp committed Oct 23, 2023
commit cbda2bfbefb5b02a39f1340ee3629d77b1c2b315
5 changes: 5 additions & 0 deletions sdk/python/feast/infra/materialization/contrib/bytewax/bytewax_materialization_engine.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,6 +200,11 @@ def _materialize_one(
batch_size = self.batch_engine_config.job_batch_size
if batch_size < 1:
raise ValueError("job_batch_size must be a value greater than 0")
if batch_size < self.batch_engine_config.max_parallelism:
logger.warning(
"job_batch_size is less than max_parallelism. Setting job_batch_size = max_parallelism"
)
batch_size = self.batch_engine_config.max_parallelism

while True:
next_offset = min(offset + batch_size, total_pods)
Expand Down