Skip to content

fix(bigquery): close GAPIC storage transport and auth sessions to prevent socket leaks#17508

Open
chalmerlowe wants to merge 2 commits into
mainfrom
fix/bigquery-socket-leak
Open

fix(bigquery): close GAPIC storage transport and auth sessions to prevent socket leaks#17508
chalmerlowe wants to merge 2 commits into
mainfrom
fix/bigquery-socket-leak

Conversation

@chalmerlowe

Copy link
Copy Markdown
Contributor

This PR resolves resource leaks (specifically open sockets left in the ESTABLISHED state) that occur during client lifecycle operations and credential refreshing in system/unit tests.

The Problem

  1. Transport Lifecycle: When closing the BigQuery Storage client, calling _transport.grpc_channel.close() was insufficient for releasing all network resources. The full _transport.close() method needs to be invoked to tear down the underlying transport channel correctly.

  2. Dynamic Auth Sessions: Under certain authentication environments (like Workload Identity/GCE Metadata server inside Kokoro CI), the google-auth library dynamically instantiates helper HTTP sessions to fetch access tokens. These sessions are not owned by the BigQuery client and are not closed automatically, leading to leaked sockets.

  3. Flaky Test Assertions: Socket count assertions in system tests were flaky because Python's garbage collection is non-deterministic, meaning sockets remained open in the operating system even after client close calls until a garbage collection cycle swept them.

Changes

  • Client & Transport Lifecycle:

    • Updated connection.py, magics.py, and table.py to close the BigQuery Storage transport using _transport.close() instead of _transport.grpc_channel.close().
  • Testing Improvements:

    • Added patch_tracked_requests interceptor to system/unit tests to track and explicitly close all dynamically spawned credential-refreshing HTTP sessions when the test context exits.
    • Added explicit gc.collect() calls to socket leak verification tests to force synchronous sweeping of unreferenced socket objects before asserting final socket counts.
  • Code Coverage:

    • Appended # pragma: NO COVER to Python version checks in __init__.py for code paths that render a deprecation warning code if attempted to run on Python runtimes <3.10 test matrix (these paths do not run in our CI/CD since we never execute code with the older runtimes).

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the BigQuery client to close the transport directly via _transport.close() instead of _transport.grpc_channel.close(), updates corresponding unit and system tests, and adds a new TestPropertyGraphReference test class. The review feedback suggests improving the patch_tracked_requests context manager in system tests to avoid closing pre-existing sessions, and refactoring manual debug prints to use idiomatic assertion messages instead.

Comment thread packages/google-cloud-bigquery/tests/system/test_client.py Outdated
Comment thread packages/google-cloud-bigquery/tests/system/test_client.py Outdated
Comment thread packages/google-cloud-bigquery/tests/system/test_client.py Outdated
Comment thread packages/google-cloud-bigquery/tests/system/test_magics.py Outdated
Comment thread packages/google-cloud-bigquery/tests/system/test_magics.py Outdated
@chalmerlowe chalmerlowe force-pushed the fix/bigquery-socket-leak branch from 77adf99 to f2ec799 Compare June 18, 2026 15:24
@chalmerlowe chalmerlowe force-pushed the fix/bigquery-socket-leak branch from f2ec799 to 43e6eec Compare June 18, 2026 15:43
@chalmerlowe chalmerlowe marked this pull request as ready for review June 18, 2026 16:10
@chalmerlowe chalmerlowe requested review from a team as code owners June 18, 2026 16:10
@chalmerlowe chalmerlowe requested review from tswast and removed request for a team June 18, 2026 16:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant