fix(gax): use fake clock in ScheduledRetryingExecutorTest for robust and fast execution#13553
Draft
lqiu96 wants to merge 2 commits into
Draft
fix(gax): use fake clock in ScheduledRetryingExecutorTest for robust and fast execution#13553lqiu96 wants to merge 2 commits into
lqiu96 wants to merge 2 commits into
Conversation
…and fast execution Introduced FakeApiClock and RecordingScheduler to ScheduledRetryingExecutorTest to run most tests (including the flaky testSuccessWithFailuresGetAttempt) instantly without real-time delays. Isolated tests that rely on asynchronous polling or cancellation to use local real executors and clocks, with increased timeout (10s) to avoid flakiness on slow VMs. Made RecordingScheduler stubbings lenient to avoid UnnecessaryStubbingException in tests that do not query all recorded metrics. TAG=agy CONV=71af906a-b7dc-4dda-a2fa-ebe24b5d8b91 BUG=13535
Contributor
There was a problem hiding this comment.
Code Review
This pull request refactors the test suite in ScheduledRetryingExecutorTest to use a fake clock and introduces local scheduled executors for isolated test runs. It also applies Mockito.lenient() to mock configurations in RecordingScheduler to prevent strict stubbing errors. The review feedback suggests moving the creation and shutdown of the local executors inside the test loops for testSuccessWithFailuresPeekAttempt and testCancelOuterFutureAfterStart to ensure complete isolation between iterations and avoid potential test flakiness.
Comment on lines
106
to
157
| ScheduledExecutorService localExecutor = Executors.newSingleThreadScheduledExecutor(); | ||
| try { | ||
| for (int executionsCount = 0; executionsCount < EXECUTIONS_COUNT; executionsCount++) { | ||
|
|
||
| FailingCallable callable = new FailingCallable(15, "request", "SUCCESS", tracer); | ||
|
|
||
| RetryingExecutorWithContext<String> executor = | ||
| getRetryingExecutor( | ||
| getAlgorithm(retrySettings, 0, null, NanoClock.getDefaultClock()), localExecutor); | ||
| RetryingFuture<String> future = | ||
| executor.createFuture( | ||
| callable, | ||
| FakeCallContext.createDefault().withTracer(tracer).withRetrySettings(retrySettings)); | ||
| callable.setExternalFuture(future); | ||
|
|
||
| assertNull(future.peekAttemptResult()); | ||
| assertSame(future.peekAttemptResult(), future.peekAttemptResult()); | ||
| assertFalse(future.getAttemptResult().isDone()); | ||
| assertFalse(future.getAttemptResult().isCancelled()); | ||
|
|
||
| future.setAttemptFuture(executor.submit(future)); | ||
|
|
||
| final AtomicInteger failedAttempts = new AtomicInteger(0); | ||
| final AtomicReference<ApiFuture<String>> lastSeenAttempt = new AtomicReference<>(); | ||
| await() | ||
| .pollInterval(Duration.ofMillis(2)) | ||
| .atMost(Duration.ofSeconds(5)) | ||
| .until( | ||
| () -> { | ||
| ApiFuture<String> attemptResult = future.peekAttemptResult(); | ||
| if (attemptResult != null && attemptResult != lastSeenAttempt.get()) { | ||
| lastSeenAttempt.set(attemptResult); | ||
| assertTrue(attemptResult.isDone()); | ||
| assertFalse(attemptResult.isCancelled()); | ||
| try { | ||
| attemptResult.get(); | ||
| } catch (ExecutionException e) { | ||
| if (e.getCause() instanceof CustomException) { | ||
| failedAttempts.incrementAndGet(); | ||
| } | ||
| } | ||
| } | ||
| } | ||
| return future.isDone(); | ||
| }); | ||
| return future.isDone(); | ||
| }); | ||
|
|
||
| assertFutureSuccess(future); | ||
| assertEquals(15, future.getAttemptSettings().getAttemptCount()); | ||
| assertTrue(failedAttempts.get() > 0); | ||
| assertFutureSuccess(future); | ||
| assertEquals(15, future.getAttemptSettings().getAttemptCount()); | ||
| assertTrue(failedAttempts.get() > 0); | ||
| } | ||
| } finally { | ||
| localExecutor.shutdownNow(); | ||
| } |
Contributor
There was a problem hiding this comment.
Creating the localExecutor outside the loop means it is shared across all iterations of the test. If any tasks from a previous iteration are still executing or scheduled, they could interfere with subsequent iterations, leading to test flakiness or race conditions. Creating and shutting down the executor inside the loop ensures complete isolation between iterations.
for (int executionsCount = 0; executionsCount < EXECUTIONS_COUNT; executionsCount++) {
ScheduledExecutorService localExecutor = Executors.newSingleThreadScheduledExecutor();
try {
FailingCallable callable = new FailingCallable(15, "request", "SUCCESS", tracer);
RetryingExecutorWithContext<String> executor =
getRetryingExecutor(
getAlgorithm(retrySettings, 0, null, NanoClock.getDefaultClock()), localExecutor);
RetryingFuture<String> future =
executor.createFuture(
callable,
FakeCallContext.createDefault().withTracer(tracer).withRetrySettings(retrySettings));
callable.setExternalFuture(future);
assertNull(future.peekAttemptResult());
assertSame(future.peekAttemptResult(), future.peekAttemptResult());
assertFalse(future.getAttemptResult().isDone());
assertFalse(future.getAttemptResult().isCancelled());
future.setAttemptFuture(executor.submit(future));
final AtomicInteger failedAttempts = new AtomicInteger(0);
final AtomicReference<ApiFuture<String>> lastSeenAttempt = new AtomicReference<>();
await()
.pollInterval(Duration.ofMillis(2))
.atMost(Duration.ofSeconds(5))
.until(
() -> {
ApiFuture<String> attemptResult = future.peekAttemptResult();
if (attemptResult != null && attemptResult != lastSeenAttempt.get()) {
lastSeenAttempt.set(attemptResult);
assertTrue(attemptResult.isDone());
assertFalse(attemptResult.isCancelled());
try {
attemptResult.get();
} catch (ExecutionException e) {
if (e.getCause() instanceof CustomException) {
failedAttempts.incrementAndGet();
}
}
}
return future.isDone();
});
assertFutureSuccess(future);
assertEquals(15, future.getAttemptSettings().getAttemptCount());
assertTrue(failedAttempts.get() > 0);
} finally {
localExecutor.shutdownNow();
}
}
Comment on lines
287
to
323
| ScheduledExecutorService localExecutor = Executors.newSingleThreadScheduledExecutor(); | ||
| try { | ||
| for (int executionsCount = 0; executionsCount < EXECUTIONS_COUNT; executionsCount++) { | ||
| FailingCallable callable = new FailingCallable(4, "request", "SUCCESS", tracer); | ||
| RetryingExecutorWithContext<String> executor = | ||
| getRetryingExecutor( | ||
| getAlgorithm(retrySettings, 0, null, NanoClock.getDefaultClock()), localExecutor); | ||
| RetryingFuture<String> future = | ||
| executor.createFuture( | ||
| callable, | ||
| FakeCallContext.createDefault().withTracer(tracer).withRetrySettings(retrySettings)); | ||
| callable.setExternalFuture(future); | ||
| future.setAttemptFuture(executor.submit(future)); | ||
|
|
||
| await() | ||
| .atMost(Duration.ofSeconds(5)) | ||
| .until( | ||
| () -> | ||
| future.getAttemptSettings() != null | ||
| && future.getAttemptSettings().getAttemptCount() > 0); | ||
|
|
||
| boolean res = future.cancel(false); | ||
| assertTrue(res); | ||
| assertFutureCancel(future); | ||
|
|
||
| // Verify that the cancelled future is traced. Every attempt increases the number | ||
| // of cancellation attempts from the tracer. | ||
| Mockito.verify(tracer, Mockito.times(executionsCount + 1)).attemptCancelled(); | ||
|
|
||
| // Assert that future has at least been attempted once | ||
| // i.e. The future from executor.submit() has been run by the ScheduledExecutor | ||
| assertTrue(future.getAttemptSettings().getAttemptCount() > 0); | ||
| assertTrue(future.getAttemptSettings().getAttemptCount() < 4); | ||
| } | ||
| } finally { | ||
| localExecutor.shutdownNow(); | ||
| } |
Contributor
There was a problem hiding this comment.
Similar to the other test, creating the localExecutor outside the loop can lead to cross-iteration interference. Since this test verifies the number of cancellation attempts on the shared tracer, any delayed task from a previous iteration that runs late and invokes the tracer could cause the verification to fail. Creating and shutting down the executor inside the loop ensures complete isolation.
for (int executionsCount = 0; executionsCount < EXECUTIONS_COUNT; executionsCount++) {
ScheduledExecutorService localExecutor = Executors.newSingleThreadScheduledExecutor();
try {
FailingCallable callable = new FailingCallable(4, "request", "SUCCESS", tracer);
RetryingExecutorWithContext<String> executor =
getRetryingExecutor(
getAlgorithm(retrySettings, 0, null, NanoClock.getDefaultClock()), localExecutor);
RetryingFuture<String> future =
executor.createFuture(
callable,
FakeCallContext.createDefault().withTracer(tracer).withRetrySettings(retrySettings));
callable.setExternalFuture(future);
future.setAttemptFuture(executor.submit(future));
await()
.atMost(Duration.ofSeconds(5))
.until(
() ->
future.getAttemptSettings() != null
&& future.getAttemptSettings().getAttemptCount() > 0);
boolean res = future.cancel(false);
assertTrue(res);
assertFutureCancel(future);
// Verify that the cancelled future is traced. Every attempt increases the number
// of cancellation attempts from the tracer.
Mockito.verify(tracer, Mockito.times(executionsCount + 1)).attemptCancelled();
// Assert that future has at least been attempted once
// i.e. The future from executor.submit() has been run by the ScheduledExecutor
assertTrue(future.getAttemptSettings().getAttemptCount() > 0);
assertTrue(future.getAttemptSettings().getAttemptCount() < 4);
} finally {
localExecutor.shutdownNow();
}
}…xecutorTest Addressed PR comments by moving the creation and shutdown of local executors inside the test loops for testSuccessWithFailuresPeekAttempt and testCancelOuterFutureAfterStart to ensure complete isolation between iterations and avoid potential flakiness. TAG=agy CONV=71af906a-b7dc-4dda-a2fa-ebe24b5d8b91 BUG=13535
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduced FakeApiClock and RecordingScheduler to ScheduledRetryingExecutorTest to run most tests (including the flaky testSuccessWithFailuresGetAttempt) instantly without real-time delays.
Isolated tests that rely on asynchronous polling or cancellation to use local real executors and clocks, with increased timeout (10s) to avoid flakiness on slow VMs.
Made RecordingScheduler stubbings lenient to avoid UnnecessaryStubbingException in tests that do not query all recorded metrics.
Fixes: #13535