Changed connection driver for ease of use. by pranaynanda · Pull Request #2194 · GoogleCloudPlatform/python-docs-samples

pranaynanda · 2019-06-05T11:47:14Z

The current code does not work if tried to implement as is. There are bugs as described in the first comment of this StackOverflow question. https://stackoverflow.com/a/53784015/5056741. To be precise, it raises 'raise InterfaceError("communication error", e)' error which appears to be related to a bug in postgres+pg8000 driver

The current code does not work if tried to implement as is. There are bugs as described in the first comment of this StackOverflow question. https://stackoverflow.com/a/53784015/5056741

googlebot · 2019-06-05T11:47:17Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here (e.g. I signed it!) and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

pranaynanda · 2019-06-05T11:49:23Z

I signed it!

googlebot · 2019-06-05T11:49:26Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

cloud-sql/postgres/sqlalchemy/main.py

gguuss

Looks like you need to push a commit removing the trailing whitespace for your PR to pass lint.

nox > flake8 --show-source --builtin gettext --max-complexity 20 --import-order-style google --exclude .nox,.cache,env,lib,generated_pb2,*_pb2.py,*_pb2_grpc.py --ignore=E121,E123,E126,E226,E24,E704,W503,W504,I100,I201,I202 --application-import-names main,templates .
./main.py:45:25: W291 trailing whitespace
        database=db_name 
                        ^
nox > Command flake8 --show-source --builtin gettext --max-complexity 20 --import-order-style google --exclude .nox,.cache,env,lib,generated_pb2,*_pb2.py,*_pb2_grpc.py --ignore=E121,E123,E126,E226,E24,E704,W503,W504,I100,I201,I202 --application-import-names main,templates . failed with exit code 1
nox > Session lint(sample='./cloud-sql/postgres/sqlalchemy') failed.
The command "./scripts/travis.sh" exited with 1.

gguuss

LGTM

gguuss

I'm less familiar with the psycop connector, will have to re-review

gguuss · 2019-07-23T21:34:36Z

@kurtisvg Can you comment on whether we should be using pg8000 or psycopg2?

gguuss · 2019-07-23T21:21:45Z

cloud-sql/postgres/sqlalchemy/requirements.txt

 Flask==1.0.2
 SQLAlchemy==1.2.17
-pg8000==1.13.1
+psycopg2==2.8.2


Update versions here.

Flask==1.1.1 SQLAlchemy==1.3.6 pg8000==1.13.2

gguuss · 2019-07-23T21:35:44Z

cloud-sql/postgres/sqlalchemy/main.py

 db = sqlalchemy.create_engine(
    # Equivalent URL:
-    # postgres+pg8000://<db_user>:<db_pass>@/<db_name>?unix_socket=/cloudsql/<cloud_sql_instance_name>/.s.PGSQL.5432
+    # postgres+psycopg2://<db_user>:<db_pass>@/<db_name>?unix_socket=/cloudsql/<cloud_sql_instance_name>/.s.PGSQL.5432


Are you sure the parameter is unix_socket? I believe I've seen it as unix_sock in some adapters.

I'd prefer to keep the pg8000 adapter and just show the correct parameter. If you are using the psycopg2 connector, it seems that you will need to install additional native dependencies for your platform.

It's cool if it works with the pg8000 adapter. I only changed it because the pg8000 driver wasn't working as per demo and was raising eroors as mentioned at the top. I tested it with psycopg2 instead and it worked. Because the PR is over a month old, I can't find the exact piece of working code that I tested with but I double checked again and this piece works:

db = sqlalchemy.create_engine( # Equivalent URL: 'postgres+psycopg2://{}:{}@/{}?host=/cloudsql/{}'.format(db_user, db_pass, db_name, cloud_sql_instance_name) )

Also StackOverflow says that 'unix_sock' should be 'host'.

I'll investigate deeper.

And found this in SQLAlchemy's documentation.

Moreover it appears that pg8000 does not support connecting over Cloud SQL proxy which is a recommended best practice.

I'm testing now, thank you btw for taking the time to make the PR :)

I think you are right, just will confirm.

The parameters get passed directly to the driver, and thus change depend on which driver you are using. pg8000 requires unix_sock: http://pybrary.net/pg8000/dbapi.html

pranaynanda · 2019-07-23T21:41:41Z

@kurtisvg Can you comment on whether we should be using pg8000 or psycopg2?

I tried using pg8000 as is initially. It returned 'raise InterfaceError("communication error", e)'. Changing it to psycopg2 worked. That's when I opened the pull request.

kurtisvg · 2019-07-23T22:56:58Z

We use pg8000 specifically because it's a pure python driver and thus it works in many situations where psycopg2 doesn't. The issue you are experiencing is because pg8000 requires the user to specify the suffix .s.PGSQL.5432, where as psycopg2 automatically adds this suffix for you.

The suffix was added to the sample in #2235. It looks like the comment is still incorrect however.

kurtisvg · 2019-07-23T23:04:55Z

It looks like the parameters are backwards between mysql and postgres.

kurtisvg

Please update the comments in the mysql and postgres files to correctly reflect the parameters below.

I would prefer to leave the driver as pg8000 since it is a pure python driver.

pranaynanda · 2019-07-23T23:24:11Z

I've changed both main.py and requirements.txt to use pg8000 by default. I've also changed main.py to the correct query statements.

gguuss · 2019-07-24T00:35:58Z

@pranaynanda Thanks again for the PR, I'm thinking it may make sense to leave as pg8000 to simplify getting the requirements installed.

Anecdotally, the sample as configured in Master is working for me and I'm unable to get psycopg2 installed on my dev machines as the sources for postgresql are missing. What's weird though is that I'm unable to install the dependency pg8000==1.13.2, only pg8000==1.12.5, potentially because I'm testing with Python2. When I first ran the sample, I had the wrong DB name which caused the same exception you were seeing.

pranaynanda · 2019-07-24T13:10:03Z

Sure. Makes sense. The code might have changed against another PR after I created this PR. The one in master branch works now. I had an older copy that did not have the /.s.PGSQL.5432 suffix for the pg8000 driver.

kurtisvg

Thanks for taking the time to PR this. Apologies it sat unattended for so long.

changes made

Changed connection driver for ease of use.

32136b8

The current code does not work if tried to implement as is. There are bugs as described in the first comment of this StackOverflow question. https://stackoverflow.com/a/53784015/5056741

googlebot added the cla: no This human has *not* signed the Contributor License Agreement. label Jun 5, 2019

googlebot added cla: yes This human has signed the Contributor License Agreement. and removed cla: no This human has *not* signed the Contributor License Agreement. labels Jun 5, 2019

pranaynanda added 2 commits June 5, 2019 17:21

Updated driver in requirements file as well

91d1ef1

Merge branch 'master' into master

8bf2915

pranaynanda commented Jun 22, 2019

View reviewed changes

cloud-sql/postgres/sqlalchemy/main.py Outdated Show resolved Hide resolved

gguuss suggested changes Jul 23, 2019

View reviewed changes

Removed trailing whitespace

3ce4ef1

gguuss approved these changes Jul 23, 2019

View reviewed changes

gguuss suggested changes Jul 23, 2019

View reviewed changes

gguuss assigned gguuss and kurtisvg Jul 23, 2019

gguuss previously requested changes Jul 23, 2019

View reviewed changes

Updated versions as per suggestion

8fcf39c

Updated to use pg8000

e86d518

kurtisvg suggested changes Jul 23, 2019

View reviewed changes

pranaynanda added 4 commits July 24, 2019 04:40

Updated formatting.

fb45efa

Removed host parameter after adding correct query

73a3b6f

Updated to use pg8000

ec4c33f

Fixed indentation

ad977dc

Fix comment in both samples.

481894a

kurtisvg approved these changes Jul 24, 2019

View reviewed changes

kurtisvg merged commit 599d096 into GoogleCloudPlatform:master Jul 24, 2019

kurtisvg mentioned this pull request Jul 24, 2019

Update instructions for pgsql connect #2259

Closed

Conversation

pranaynanda commented Jun 5, 2019

Uh oh!

googlebot commented Jun 5, 2019

What to do if you already signed the CLA

Individual signers

Corporate signers

Uh oh!

pranaynanda commented Jun 5, 2019

Uh oh!

googlebot commented Jun 5, 2019

Uh oh!

Uh oh!

gguuss left a comment

Choose a reason for hiding this comment

Uh oh!

gguuss left a comment

Choose a reason for hiding this comment

Uh oh!

gguuss left a comment

Choose a reason for hiding this comment

Uh oh!

gguuss commented Jul 23, 2019

Uh oh!

gguuss Jul 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gguuss Jul 23, 2019

Choose a reason for hiding this comment

Uh oh!

gguuss Jul 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pranaynanda Jul 23, 2019

Choose a reason for hiding this comment

Uh oh!

gguuss Jul 23, 2019

Choose a reason for hiding this comment

Uh oh!

pranaynanda Jul 23, 2019

Choose a reason for hiding this comment

Uh oh!

gguuss Jul 23, 2019

Choose a reason for hiding this comment

Uh oh!

gguuss Jul 23, 2019

Choose a reason for hiding this comment

Uh oh!

kurtisvg Jul 23, 2019

Choose a reason for hiding this comment

Uh oh!

pranaynanda commented Jul 23, 2019

Uh oh!

kurtisvg commented Jul 23, 2019

Uh oh!

kurtisvg commented Jul 23, 2019

Uh oh!

kurtisvg left a comment

Choose a reason for hiding this comment

Uh oh!

pranaynanda commented Jul 23, 2019

Uh oh!

gguuss commented Jul 24, 2019

Uh oh!

pranaynanda commented Jul 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kurtisvg left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gguuss Jul 23, 2019 •

edited

Loading

gguuss Jul 23, 2019 •

edited

Loading

pranaynanda commented Jul 24, 2019 •

edited

Loading