Skip to content

Optimize memory footprint for Spark Ingestion Job - #1265

Merged
pyalex merged 1 commit into
feast-dev:masterfrom
pyalex:spark-memory-optimize
Jan 14, 2021
Merged

Optimize memory footprint for Spark Ingestion Job#1265
pyalex merged 1 commit into
feast-dev:masterfrom
pyalex:spark-memory-optimize

Conversation

@pyalex

@pyalex pyalex commented Jan 14, 2021

Copy link
Copy Markdown
Collaborator

Signed-off-by: Oleksii Moskalenko moskalenko.alexey@gmail.com

What this PR does / why we need it:

  1. Avoid expensive partition materialization in metrics calculation
  2. Checkpoint data in case of repartitioning (trigger materialization)

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

NONE

Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>

@khorshuheng khorshuheng left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@feast-ci-bot

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: khorshuheng, pyalex

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@khorshuheng

Copy link
Copy Markdown
Collaborator

/test test-end-to-end-aws

@feast-ci-bot

Copy link
Copy Markdown
Collaborator

@pyalex: The following tests failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
test-end-to-end-azure 2020234 link /test test-end-to-end-azure
test-end-to-end-aws 2020234 link /test test-end-to-end-aws

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@pyalex
pyalex merged commit eaaf233 into feast-dev:master Jan 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants