Skip to content

dataflow: workaround for beam base image bug#4899

Merged
leahecole merged 4 commits into
GoogleCloudPlatform:masterfrom
davidcavazos:dataflow-fix-flex-templates-version
Oct 26, 2020
Merged

dataflow: workaround for beam base image bug#4899
leahecole merged 4 commits into
GoogleCloudPlatform:masterfrom
davidcavazos:dataflow-fix-flex-templates-version

Conversation

@davidcavazos

@davidcavazos davidcavazos commented Oct 23, 2020

Copy link
Copy Markdown

Description

Fixes #4891

Comment from Melody:

I did couple of tests with different beam versions and base images.

The problem seemed to be introduced after we upgraded to apache beam 2.24.0. I got the same error with 2.24.0 for latest two base images and a slightly different error (but was still caused by missing libffi-dev) for even earlier images.

Apache beam 2.23.0 and previous versions all worked fine for recent base images.
I would suggest using beam 2.23.0 for an immediate fix and we will work on the image side to deal with this dependency.

Checklist

@davidcavazos davidcavazos requested a review from a team as a code owner October 23, 2020 17:35
@google-cla google-cla Bot added the cla: yes This human has signed the Contributor License Agreement. label Oct 23, 2020
@tmatsuo

tmatsuo commented Oct 23, 2020

Copy link
Copy Markdown
Contributor

I think renovate bot will try to bump the version, no?

@tmatsuo

tmatsuo commented Oct 23, 2020

Copy link
Copy Markdown
Contributor

Also, I don't think this is an ultimate fix for the problem. Why can't we tweak the Docker image and support the newer version of apache beam?

@product-auto-label product-auto-label Bot added api: dataflow Issues related to the Dataflow API. samples Issues that are directly related to samples. labels Oct 24, 2020
@davidcavazos

davidcavazos commented Oct 26, 2020

Copy link
Copy Markdown
Author

Yes, I could also do that. I'll add a comment to the dependency installation in the Dockerfile and link to the GitHub issue.

@davidcavazos davidcavazos force-pushed the dataflow-fix-flex-templates-version branch from 11b015f to e066e8a Compare October 26, 2020 17:58
@davidcavazos davidcavazos requested a review from tmatsuo October 26, 2020 18:04
@leahecole leahecole merged commit 872ded2 into GoogleCloudPlatform:master Oct 26, 2020
@leahecole leahecole mentioned this pull request Oct 26, 2020
Merged
1 task
@davidcavazos davidcavazos deleted the dataflow-fix-flex-templates-version branch October 27, 2020 15:40
craiglabenz pushed a commit that referenced this pull request Nov 2, 2020
@leahecole @craiglabenz
dataflow: workaround for beam base image bug (#4899)
b6ce32a 
* dataflow: install libffi-dev explicitly

* Mention region explicitly in cleanup

Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

2 more reviewers

@tmatsuo tmatsuo tmatsuo approved these changes

@busunkim96 busunkim96 busunkim96 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@tmatsuo tmatsuo

Labels

api: dataflow Issues related to the Dataflow API. cla: yes This human has signed the Contributor License Agreement. samples Issues that are directly related to samples.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Flex template sample not working (pip install error)

4 participants

@davidcavazos @tmatsuo @busunkim96 @leahecole