Skip to content

fix: handle potential DB conflict due to concurrent upload requests in postFile#19005

Merged
cstyan merged 4 commits into
mainfrom
callum-solve-template-multi-file
Jul 30, 2025
Merged

fix: handle potential DB conflict due to concurrent upload requests in postFile#19005
cstyan merged 4 commits into
mainfrom
callum-solve-template-multi-file

Conversation

@cstyan
Copy link
Copy Markdown
Contributor

@cstyan cstyan commented Jul 22, 2025

This issue manifests when users have multiple templates which rely on the same files, for example see: #17442

In the files table we have a constraint to enforce that there can only be one entry per hash, created_by combo. When running terraform apply to update templates, this can mean that the file upload can fail for some of the templates as they hit a race condition all trying to insert at the same time.

The fix here is to detect presence of the conflict error and run another GetFileByHashAndCreator. This should happen infrequently enough to not cause significant extra load on the DB. If in the future we notice that it does, we could change the underlying SQL for file insertion to run a CAS like call via the ON CONFLICT syntax.

Note that the test added here will fail without the change from the first commit. I also tested this manually via deploy.sh in my own workspace.

Apply diff example without code changes 
terraform apply --parallelism=10
var.coder_url
  Coder deployment URL

  Enter a value: ***

coderd_template.test_templates["template-a"]: Refreshing state... [id=5d457fb1-4697-4567-97e9-856627f12b1a]
coderd_template.test_templates["template-b"]: Refreshing state... [id=898f2d5f-f22b-4741-abbb-ac5c3e36eb5f]
coderd_template.test_templates["template-c"]: Refreshing state... [id=acd53214-8bbd-402d-ab71-83fc9b5c0822]

Terraform used the selected providers to generate the following execution plan. Resource actions are
indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # coderd_template.test_templates["template-a"] will be updated in-place
  ~ resource "coderd_template" "test_templates" {
      ~ display_name                      = "template-a" -> (known after apply)
        id                                = "5d457fb1-4697-4567-97e9-856627f12b1a"
      ~ max_port_share_level              = "public" -> (known after apply)
        name                              = "template-a"
      ~ organization_id                   = "8ff5f083-aec6-4405-8df7-a16d9362c8f4" -> (known after apply)
      ~ versions                          = [
          ~ {
              ~ directory_hash = "0de194f2590374f4d9f579712ff6eed00a71bf60a37c5df40d03a9640ecc5a00" -> "29ff9a13dd5f3fbf1c2c5cf1a33ac650492c3d266f77e65bb569f9bfa711ac3d"
              ~ id             = "7161d56a-1de3-43f1-afe0-f559e70712aa" -> (known after apply)
              ~ name           = "1.0.5" -> "1.0.7"
                # (4 unchanged attributes hidden)
            },
        ]
        # (14 unchanged attributes hidden)
    }

  # coderd_template.test_templates["template-b"] will be updated in-place
  ~ resource "coderd_template" "test_templates" {
      ~ display_name                      = "template-b" -> (known after apply)
        id                                = "898f2d5f-f22b-4741-abbb-ac5c3e36eb5f"
      ~ max_port_share_level              = "public" -> (known after apply)
        name                              = "template-b"
      ~ organization_id                   = "8ff5f083-aec6-4405-8df7-a16d9362c8f4" -> (known after apply)
      ~ versions                          = [
          ~ {
              ~ directory_hash = "22e2692a6d58da8010c8e48d2e4a1965cad34aac0a10a718cdb935d0ea625c14" -> "29ff9a13dd5f3fbf1c2c5cf1a33ac650492c3d266f77e65bb569f9bfa711ac3d"
              ~ id             = "c431e295-1026-4ca4-a467-0d875eb89f4f" -> (known after apply)
              ~ name           = "1.0.6" -> "1.0.7"
                # (4 unchanged attributes hidden)
            },
        ]
        # (14 unchanged attributes hidden)
    }

  # coderd_template.test_templates["template-c"] will be updated in-place
  ~ resource "coderd_template" "test_templates" {
      ~ display_name                      = "template-c" -> (known after apply)
        id                                = "acd53214-8bbd-402d-ab71-83fc9b5c0822"
      ~ max_port_share_level              = "public" -> (known after apply)
        name                              = "template-c"
      ~ organization_id                   = "8ff5f083-aec6-4405-8df7-a16d9362c8f4" -> (known after apply)
      ~ versions                          = [
          ~ {
              ~ directory_hash = "22e2692a6d58da8010c8e48d2e4a1965cad34aac0a10a718cdb935d0ea625c14" -> "29ff9a13dd5f3fbf1c2c5cf1a33ac650492c3d266f77e65bb569f9bfa711ac3d"
              ~ id             = "304215dc-bbde-4d41-8a5a-c9dd0926a96b" -> (known after apply)
              ~ name           = "1.0.6" -> "1.0.7"
                # (4 unchanged attributes hidden)
            },
        ]
        # (14 unchanged attributes hidden)
    }

Plan: 0 to add, 3 to change, 0 to destroy.
Error from apply 
Plan: 0 to add, 3 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

coderd_template.test_templates["template-b"]: Modifying... [id=898f2d5f-f22b-4741-abbb-ac5c3e36eb5f]
coderd_template.test_templates["template-a"]: Modifying... [id=5d457fb1-4697-4567-97e9-856627f12b1a]
coderd_template.test_templates["template-c"]: Modifying... [id=acd53214-8bbd-402d-ab71-83fc9b5c0822]
coderd_template.test_templates["template-a"]: Modifications complete after 4s [id=5d457fb1-4697-4567-97e9-856627f12b1a]
coderd_template.test_templates["template-b"]: Modifications complete after 6s [id=898f2d5f-f22b-4741-abbb-ac5c3e36eb5f]
╷
│ Error: Provisioner Error
│ 
│   with coderd_template.test_templates["template-c"],
│   on main.tf line 19, in resource "coderd_template" "test_templates":
│   19: resource "coderd_template" "test_templates" {
│ 
│ failed to upload directory: POST ***/api/v2/files: unexpected
│ status code 500: Internal error saving file.
│       Error: pq: duplicate key value violates unique constraint "files_hash_created_by_key"
│
Running the apply again succeeds 
Terraform used the selected providers to generate the following execution plan. Resource actions are
indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # coderd_template.test_templates["template-c"] will be updated in-place
  ~ resource "coderd_template" "test_templates" {
      ~ display_name                      = "template-c" -> (known after apply)
        id                                = "acd53214-8bbd-402d-ab71-83fc9b5c0822"
      ~ max_port_share_level              = "public" -> (known after apply)
        name                              = "template-c"
      ~ organization_id                   = "8ff5f083-aec6-4405-8df7-a16d9362c8f4" -> (known after apply)
      ~ versions                          = [
          ~ {
              ~ directory_hash = "22e2692a6d58da8010c8e48d2e4a1965cad34aac0a10a718cdb935d0ea625c14" -> "29ff9a13dd5f3fbf1c2c5cf1a33ac650492c3d266f77e65bb569f9bfa711ac3d"
              ~ id             = "304215dc-bbde-4d41-8a5a-c9dd0926a96b" -> (known after apply)
              ~ name           = "1.0.6" -> "1.0.7"
                # (4 unchanged attributes hidden)
            },
        ]
        # (14 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

coderd_template.test_templates["template-c"]: Modifying... [id=acd53214-8bbd-402d-ab71-83fc9b5c0822]
coderd_template.test_templates["template-c"]: Modifications complete after 3s [id=acd53214-8bbd-402d-ab71-83fc9b5c0822]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
Apply diff example with code changes 
coderd_template.test_templates["template-c"]: Refreshing state... [id=acd53214-8bbd-402d-ab71-83fc9b5c0822]
coderd_template.test_templates["template-b"]: Refreshing state... [id=898f2d5f-f22b-4741-abbb-ac5c3e36eb5f]
coderd_template.test_templates["template-a"]: Refreshing state... [id=5d457fb1-4697-4567-97e9-856627f12b1a]

Terraform used the selected providers to generate the following execution plan. Resource actions are
indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # coderd_template.test_templates["template-a"] will be updated in-place
  ~ resource "coderd_template" "test_templates" {
      ~ display_name                      = "template-a" -> (known after apply)
        id                                = "5d457fb1-4697-4567-97e9-856627f12b1a"
      ~ max_port_share_level              = "public" -> (known after apply)
        name                              = "template-a"
      ~ organization_id                   = "8ff5f083-aec6-4405-8df7-a16d9362c8f4" -> (known after apply)
      ~ versions                          = [
          ~ {
              ~ directory_hash = "29ff9a13dd5f3fbf1c2c5cf1a33ac650492c3d266f77e65bb569f9bfa711ac3d" -> "d4a509e4af5f96f5b3970098233d742105f75c997aafa8a6da2ee4953c864a54"
              ~ id             = "e130f32f-2c3e-4332-818f-fd6de980128f" -> (known after apply)
              ~ name           = "1.0.7" -> "1.0.8"
                # (4 unchanged attributes hidden)
            },
        ]
        # (14 unchanged attributes hidden)
    }

  # coderd_template.test_templates["template-b"] will be updated in-place
  ~ resource "coderd_template" "test_templates" {
      ~ display_name                      = "template-b" -> (known after apply)
        id                                = "898f2d5f-f22b-4741-abbb-ac5c3e36eb5f"
      ~ max_port_share_level              = "public" -> (known after apply)
        name                              = "template-b"
      ~ organization_id                   = "8ff5f083-aec6-4405-8df7-a16d9362c8f4" -> (known after apply)
      ~ versions                          = [
          ~ {
              ~ directory_hash = "29ff9a13dd5f3fbf1c2c5cf1a33ac650492c3d266f77e65bb569f9bfa711ac3d" -> "d4a509e4af5f96f5b3970098233d742105f75c997aafa8a6da2ee4953c864a54"
              ~ id             = "67207027-a3a0-4679-834e-1f8ac69ad31e" -> (known after apply)
              ~ name           = "1.0.7" -> "1.0.8"
                # (4 unchanged attributes hidden)
            },
        ]
        # (14 unchanged attributes hidden)
    }

  # coderd_template.test_templates["template-c"] will be updated in-place
  ~ resource "coderd_template" "test_templates" {
      ~ display_name                      = "template-c" -> (known after apply)
        id                                = "acd53214-8bbd-402d-ab71-83fc9b5c0822"
      ~ max_port_share_level              = "public" -> (known after apply)
        name                              = "template-c"
      ~ organization_id                   = "8ff5f083-aec6-4405-8df7-a16d9362c8f4" -> (known after apply)
      ~ versions                          = [
          ~ {
              ~ directory_hash = "29ff9a13dd5f3fbf1c2c5cf1a33ac650492c3d266f77e65bb569f9bfa711ac3d" -> "d4a509e4af5f96f5b3970098233d742105f75c997aafa8a6da2ee4953c864a54"
              ~ id             = "8771b486-16a2-47d8-bcc0-08d7aeae5e15" -> (known after apply)
              ~ name           = "1.0.7" -> "1.0.8"
                # (4 unchanged attributes hidden)
            },
        ]
        # (14 unchanged attributes hidden)
    }

Plan: 0 to add, 3 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.
apply works on the first try 
Plan: 0 to add, 3 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

coderd_template.test_templates["template-a"]: Modifying... [id=5d457fb1-4697-4567-97e9-856627f12b1a]
coderd_template.test_templates["template-c"]: Modifying... [id=acd53214-8bbd-402d-ab71-83fc9b5c0822]
coderd_template.test_templates["template-b"]: Modifying... [id=898f2d5f-f22b-4741-abbb-ac5c3e36eb5f]
coderd_template.test_templates["template-a"]: Modifications complete after 2s [id=5d457fb1-4697-4567-97e9-856627f12b1a]
coderd_template.test_templates["template-c"]: Modifications complete after 5s [id=acd53214-8bbd-402d-ab71-83fc9b5c0822]
coderd_template.test_templates["template-b"]: Modifications complete after 8s [id=898f2d5f-f22b-4741-abbb-ac5c3e36eb5f]

The template files are as follows:

Top level main.tf 
terraform {
  required_providers {
    coderd = {
      source = "coder/coderd"
    }
  }
}

provider "coderd" {
  url = var.coder_url
}

variable "coder_url" {
  description = "Coder deployment URL"
  type        = string
}

# Create 3 templates that use identical files
resource "coderd_template" "test_templates" {
  for_each = {
    "template-a" = "Template A"
    "template-b" = "Template B" 
    "template-c" = "Template C"
  }
  
  name        = each.key
  description = each.value
  
  versions = [{
    name      = "1.0.0"
    directory = "./template"  # Same directory = same files = same hash
    active    = true
    tf_vars = [
      {
        name  = "template_name"
        value = each.key  # Only difference between templates
      }
    ]
  }]
}
template/main.tf 
terraform {
  required_providers {
    coder = {
      source = "coder/coder"
    }
  }
}

variable "template_name" {
  description = "Name of the template"
  type        = string
  default     = "test"
}

data "coder_workspace" "me" {}

resource "coder_agent" "main" {
  arch = "amd64"
  os   = "linux"
}

resource "null_resource" "example" {
  provisioner "local-exec" {
    command = "echo 'Template name is: ${var.template_name}'"
  }
}
template/README.md 
# Workspace Template

This is a test template for reproducing the file upload race condition.

## Features
- Basic workspace setup
- Agent configuration
- Multiple files to increase upload size

To test, simply modify the template files (adding a comment in each is all you need, plus updating the version #) and then terraform apply with the --parallelism flag.

cstyan added 2 commits July 22, 2025 18:48
@cstyan
handle conflict in postFile
a009ba7 
Signed-off-by: Callum Styan <callumstyan@gmail.com>
@cstyan
add test to ensure that concurrent file uploads don't result in an error
df947ef 
Signed-off-by: Callum Styan <callumstyan@gmail.com>
@cstyan
fix linting
f6ae297 
Signed-off-by: Callum Styan <callumstyan@gmail.com>
brettkolodny
brettkolodny approved these changes Jul 24, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@brettkolodny brettkolodny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thought abut logging but otherwise lgtm 👍

Comment thread coderd/files.go
Comment on lines +121 to +127
if database.IsUniqueViolation(err, database.UniqueFilesHashCreatedByKey) {
// The file was uploaded by some concurrent process since the last time we checked for it, fetch it again.
file, err = api.Database.GetFileByHashAndCreator(ctx, database.GetFileByHashAndCreatorParams{
Hash: hash,
CreatedBy: apiKey.UserID,
})
}
Copy link
Copy Markdown
Contributor

@brettkolodny brettkolodny Jul 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even though we suspect that this won't happen often enough to be an issue, would it be worth adding a log here so that we can verify how often this happens later, or would that be needless clutter to the logs?

Copy link
Copy Markdown
Contributor Author

@cstyan cstyan Jul 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absolutely, added one 👍

@cstyan
add log line if we hit the unique violation error case
b71c58a 
Signed-off-by: Callum Styan <callumstyan@gmail.com>
@cstyan cstyan merged commit d736af1 into main Jul 30, 2025
47 checks passed
@cstyan cstyan deleted the callum-solve-template-multi-file branch July 30, 2025 20:55
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 30, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

1 more reviewer

@brettkolodny brettkolodny brettkolodny approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@cstyan cstyan

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@cstyan @brettkolodny