Skip to content

fix(localstack): copy the starter script before the container starts - #11963

Open
uhla wants to merge 1 commit into
testcontainers:mainfrom
uhla:fix/localstack-starter-script-race
Open

fix(localstack): copy the starter script before the container starts#11963
uhla wants to merge 1 commit into
testcontainers:mainfrom
uhla:fix/localstack-starter-script-race

Conversation

@uhla

@uhla uhla commented Aug 9, 2026

Copy link
Copy Markdown

Problem

LocalStackContainer starts the container with an entrypoint that waits for
/testcontainers_start.sh to exist and then executes it:

cmd.withEntrypoint("sh", "-c",
    "while [ ! -f " + STARTER_SCRIPT + " ]; do sleep 0.1; done; " + STARTER_SCRIPT);

but the script is copied from containerIsStarting, i.e. once the container is already running. The
copy publishes the filename before the contents, so a poll landing mid-copy executes an unfinished
file. The container exits with code 126, never logs Ready., and the failure surfaces 60s later as a
wait-strategy timeout that points away from the cause:

Caused by: java.lang.IllegalStateException: Wait strategy failed. Container exited with code 126
Caused by: ContainerLaunchException: Timed out waiting for log output matching '.*Ready\.'

The container's own stderr says what really happened, but nothing surfaces it:

sh: 1: /testcontainers_start.sh: Text file busy

Existence is not readiness: during extraction the path exists while the file is still incomplete and
open for writing, so execve fails with ETXTBSY, or with EACCES on runtimes that apply the mode
only after the write. Both make a POSIX shell exit 126.

We hit this on CI at roughly 1% of builds (about 0.33% per container start) on Docker 20.10.21.

Fix

Move the copy to containerIsCreated, which GenericContainer.doStart calls between
createCommand.exec() and startContainerCmd, the same window withCopyToContainer entries already
use. Nothing is running in the container there, so an incomplete script cannot be executed. This
removes the race rather than narrowing it, and does not depend on whether the runtime applies the file
mode before or after the write.

internalMarkerLabels() read getContainerInfo(), which is not populated that early, so the labels
now come from an explicit inspectContainerCmd(containerId).

Applied to both org.testcontainers.localstack.LocalStackContainer and the deprecated
org.testcontainers.containers.localstack.LocalStackContainer.

Note that containerIsStarting also ran on the reuse path, so a reused container had the script
re-copied over itself while running. containerIsCreated does not, which is both correct (the script
is already there, same container, same labels) and safer.

Testing

  • StarterScriptTest reads the script back out of the created but not yet started container and
    asserts it is complete. Without this change it fails with NotFoundException: no such file or directory, so it guards the ordering rather than passing alongside it.
  • Full :testcontainers-localstack:test green (35 tests), including
    LambdaContainerLabels.shouldLabelLambdaContainers, which covers the refactored label path.
  • A/B through the library with the same padded starter script in both arms: copied from
    containerIsStarting it fails with Container exited with code 126, copied from
    containerIsCreated it starts cleanly.

The entrypoint waits only for the script to exist and then executes it, but the
copy happened after the container was already running, so a poll landing
mid-copy could execute an incomplete file and the container exited with code 126.
containerIsCreated runs before start; labels come from an explicit inspect
because getContainerInfo() is not populated yet.
@uhla
uhla requested a review from a team as a code owner August 9, 2026 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant