Repository navigation
Investigate possible solutions for "Text file busy" errors during builds #14726
Description
This is unfortunately an issue with Terraform (and largely just filesystems in general). Some background:
text file busycan occur in Linux when a process tries to write or modify an executable that's currently running. In TF's case this typically happens when multiple processes try to write and use plugins from the same cache directory.- In Coder's case we use a separate cache directory per runner. e.g. if you're running 3 runners they should all be writing their plugins to their own exclusive directories avoiding this problem.
- Despite this I'm guessing that some terraform or plugin process is not exiting on a run which causes subsequent runs to fail.
Regardless we should do our best to circumvent this when we detect this during a build. It's unknown to me how long the process lingers but it could potentially be indefinitely which would render that particular runner dead for subsequent runs, requiring the end user to rebuild a number of times before getting a successful build which is not acceptable.
e.g.
Initializing the backend...
09:45:57.672Initializing provider plugins...
09:45:57.672- Finding coder/coder versions matching "~> 0.23.0"...
09:45:57.761- Finding hashicorp/kubernetes versions matching "~> 2.30.0"...
09:45:58.031- Installing coder/coder v0.23.0...
09:45:58.330- Installing hashicorp/kubernetes v2.30.0...
09:45:59.405- Installed hashicorp/kubernetes v2.30.0 (signed by HashiCorp)
09:45:59.406Error: Failed to install provider
09:45:59.406Error while installing coder/coder v0.23.0: open
09:45:59.407/tmp/coder/provisioner-1/tf/registry.terraform.io/coder/coder/0.23.0/linux_amd64/terraform-provider-coder_v0.23.0:
09:45:59.408text file busy
Activity
spikecurtis commentedon Sep 20, 2024
Terraform plugins use https://github.com/hashicorp/go-plugin to communicate with Terraform. Plugins themselves are gRPC services that listen on a Unix domain socket in a temp directory (or on localhost on Windows). Terraform is the "client" of the gRPC service, but manages the lifecycle of the "service".
At the end of the day, Terraform is supposed to call a special API to tell the provider to shut down, and/or forcibly kill the child process. However, it seems that sometimes that's not happening, and the provider process can linger, just listening on its domain socket.
spikecurtis commentedon Sep 20, 2024
Some ideas:
If we get "text file busy" error:
psor the/procsystem) the offending process and kill itShutdowncommand --- might be complicated by authentication protocols (go-plugin supports an "auto mTLS" setting that ensures only the original client can connect).spikecurtis commentedon Sep 20, 2024
We could fix Terraform and/or OpenTofu such that they don't reinstall the provider binary if it already exists (possibly including hashing contents). This would sidestep the issue, since we don't write to the file. However, the underlying issue of leaking provider processes would remain.
spikecurtis commentedon Sep 20, 2024
I'll also check whether Terraform is setting Pdeathsignal in https://pkg.go.dev/syscall#SysProcAttr
UPDATE: Terraform doesn't set any special
SysProcAttr, so I think that just means the child getsSIGHUPby default if the parent dies without sending it a signal.spikecurtis commentedon Sep 23, 2024
I don't think I have the full story yet, but I've just confirmed that if you specify a
provider_installation->filesystem_mirrorin your.terraformrc, then when "installing" providers, Terraform just creates symlinks to your mirror. That means that when running multiple provisionerds on a single host, they're not actually being isolated at the filesystem level, even though we take some care to give them unique cache directories.https://developer.hashicorp.com/terraform/cli/config/config-file#filesystem_mirror
Pretty sure at least one customer who is seeing this issue is using a
filesystem_mirror, we should confirm with others.spikecurtis commentedon Sep 24, 2024
One of the customers seeing this is using the zipped layout for their filesystem mirror. That explains why the provider gets opened for writing (unzipping the provider package), but doesn't explain why the provider is still executing, since the symlink stuff I mentioned only applies to unpacked layout.
I've confirmed that the output of
lsofon Busybox which the customer is using lists the real path of the executable, and it shows that the cache directory is not symlinked to the filesystem mirror. So symlinking to the mirror doesn't explain how they sawtext file busy.I'm following up with the other customers who have seen the issue, so see what kind of layout they are using.
I'm back to thinking that our provider must be still running from a previous build on the same provisionerd. Not sure how this could happen.
spikecurtis commentedon Sep 30, 2024
Another customer has said they aren't using a
filesystem_mirror, so that discredits the symlinking theory even more.spikecurtis commentedon Oct 3, 2024
Well, I guess that means coder/terraform-provider-coder#290 didn't help, since that was released in v1.0.3
spikecurtis commentedon Oct 3, 2024
@datapedd what was the workspace ID of the build that failed with "text file busy"?
Is there any chance you could exec into the coderd pod and see if
/home/coder/.cache/coder/provisioner-0/tf/registry.terraform.io/coder/coder/1.0.3/linux_amd64/terraform-provider-coder_v1.0.3is still running, and if so, get a core dump?spikecurtis commentedon Oct 3, 2024
@datapedd what was the exact time the build that failed started? You should be able to see this by clicking the builds icon on the left:
Then, click the failed build.
chore: log provider stack traces on text file busy (#15078)
chore: log provider stack traces on text file busy (#15078)
chore: log provider stack traces on text file busy (#15249)
spikecurtis commentedon Jun 9, 2025
Everyone has gone quiet about this and we should now log stack-traces if it recurs. Closing.