Skip to content

chore(bigquery): fix timeout flakiness in job tests#17826

Open
chalmerlowe wants to merge 1 commit into
mainfrom
fix/bigquery-job-timeout
Open

chore(bigquery): fix timeout flakiness in job tests#17826
chalmerlowe wants to merge 1 commit into
mainfrom
fix/bigquery-job-timeout

Conversation

@chalmerlowe

Copy link
Copy Markdown
Contributor

The Problem

Two unit tests in google-cloud-bigquery (test_result_w_retry_wo_state and test_result_w_custom_retry) were failing intermittently with a TimeoutError or RetryError. This flakiness occurs because the tests use a very short inner retry deadline of 0.1 seconds. Under heavy load (such as during parallel testing or on constrained continuous integration runners), this deadline can be exceeded, causing the test to fail confusingly.

The Solution

Increased the deadline parameter for the custom retry objects in both tests from 0.1 seconds to 1.0 seconds. This provides enough buffer for the test assertions to complete without timing out, aligning with other similar tests in the suite.

Notes to Reviewers

Here is a snippet of the failure:

    def test_result_w_retry_wo_state(global_time_lock):
        from google.cloud.bigquery.retry import DEFAULT_GET_JOB_TIMEOUT
    
        begun_job_resource = helpers._make_job_resource(
            job_id=JOB_ID, project_id=PROJECT, location="EU", started=True
        )
        done_job_resource = helpers._make_job_resource(
            job_id=JOB_ID,
            project_id=PROJECT,
            location="EU",
            started=True,
            ended=True,
        )
        conn = helpers.make_connection(
            exceptions.NotFound("not normally retriable"),
            begun_job_resource,
            exceptions.NotFound("not normally retriable"),
            done_job_resource,
        )
        client = helpers._make_client(project=PROJECT, connection=conn)
        job = google.cloud.bigquery.job._AsyncJob(
            google.cloud.bigquery.job._JobReference(JOB_ID, PROJECT, "EU"), client
        )
        custom_predicate = mock.Mock()
        custom_predicate.return_value = True
        custom_retry = google.api_core.retry.Retry(
            predicate=custom_predicate,
            initial=0.001,
            maximum=0.001,
            deadline=0.1,
        )
>       assert job.result(retry=custom_retry) is job
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

tests/unit/job/test_async_job_retry.py:115: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
google/cloud/bigquery/job/base.py:1047: in result
    return super(_AsyncJob, self).result(timeout=timeout, retry=retry)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.nox/unit-3-12-test_type-unit/lib/python3.12/site-packages/google/api_core/future/polling.py:256: in result
    self._blocking_poll(timeout=timeout, retry=retry, polling=polling)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = _AsyncJob<project=test-project, location=EU, id=test-job-id>
timeout = None
retry = <google.api_core.retry.retry_unary.Retry object at 0x7ff2beda32c0>
polling = <google.api_core.retry.retry_unary.Retry object at 0x7ff2beda01d0>

    def _blocking_poll(self, timeout=_DEFAULT_VALUE, retry=None, polling=None):
        """Poll and wait for the Future to be resolved."""
    
        if self._result_set:
            return
    
        polling = polling or self._polling
        if timeout is not PollingFuture._DEFAULT_VALUE:
            polling = polling.with_timeout(timeout)
    
        try:
            polling(self._done_or_raise)(retry=retry)
        except exceptions.RetryError:
>           raise concurrent.futures.TimeoutError(
                f"Operation did not complete within the designated timeout of "
                f"{polling.timeout} seconds."
            )
E           TimeoutError: Operation did not complete within the designated timeout of None seconds.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request reorganizes imports and increases the retry deadline from 0.1 to 1.0 seconds in the unit tests test_async_job_retry.py and test_query_job_retry.py for Google Cloud BigQuery jobs. There are no review comments, so I have no feedback to provide.

@chalmerlowe chalmerlowe self-assigned this Jul 22, 2026
@chalmerlowe
chalmerlowe marked this pull request as ready for review July 22, 2026 17:59
@chalmerlowe
chalmerlowe requested review from a team as code owners July 22, 2026 17:59
@chalmerlowe
chalmerlowe requested review from tswast and removed request for a team July 22, 2026 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant