Skip to content

Issue with sysbox: "output pipes were not closed after 10s" #16473

Closed

Description

@jplu

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

When I create a workspace from a template with sysbox activated, the newly created workspace is 'unhealthy' mode. Because of a Warning saying that stdout and/or stderr are used.

Relevant Log Output

time="2025-02-06T11:08:50.506065573Z" level=info msg="loading plugin \"io.containerd.nri.v1.nri\"..." type=io.containerd.nri.v1
time="2025-02-06T11:08:50.506073231Z" level=info msg="NRI interface is disabled by configuration."
time="2025-02-06T11:08:50.506676736Z" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock
time="2025-02-06T11:08:50.506752092Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
time="2025-02-06T11:08:50.506805190Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
time="2025-02-06T11:08:50.506830289Z" level=info msg="containerd successfully booted in 0.028811s"
time="2025-02-06T11:08:54.446894750Z" level=info msg="Loading containers: start."
time="2025-02-06T11:08:54.727642697Z" level=info msg="Loading containers: done."
time="2025-02-06T11:08:54.758994289Z" level=warning msg="WARNING: bridge-nf-call-iptables is disabled"
time="2025-02-06T11:08:54.759047675Z" level=warning msg="WARNING: bridge-nf-call-ip6tables is disabled"
time="2025-02-06T11:08:54.759077543Z" level=info msg="Docker daemon" commit=92a8393 containerd-snapshotter=false storage-driver=overlay2 version=27.4.0
time="2025-02-06T11:08:54.759178723Z" level=info msg="Daemon has completed initialization"
time="2025-02-06T11:08:54.802571696Z" level=info msg="API listen on /var/run/docker.sock"
WARNING: script exited successfully, but output pipes were not closed after 10s. This usually means a child process was started with references to stdout or stderr. As a result, this process may now have been terminated. Consider redirecting the output or using a separate "coder_script" for the process, see https://coder.com/docs/templates/troubleshooting#startup-script-issues for more information.

Expected Behavior

Not having the warning at the end of the log and having a "healthy" workspace

Steps to Reproduce

Here my template:

terraform {
  required_providers {
    coder = {
      source = "coder/coder"
    }
    docker = {
      source = "kreuzwerker/docker"
    }
  }
}

locals {
  username = data.coder_workspace_owner.me.name
}

variable "docker_socket" {
  default     = ""
  description = "(Optional) Docker socket URI"
  type        = string
}

provider "docker" {
  # Defaulting to null if the variable is an empty string lets us have an optional variable without having to set our own default
  host = var.docker_socket != "" ? var.docker_socket : null
}

data "coder_provisioner" "me" {}
data "coder_workspace" "me" {}
data "coder_workspace_owner" "me" {}

resource "coder_agent" "main" {
  arch           = data.coder_provisioner.me.arch
  os             = "linux"
  startup_script = <<-EOT
    set -e

    # Prepare user home with default files on first start.
    if [ ! -f ~/.init_done ]; then
      cp -rT /etc/skel ~
      touch ~/.init_done
    fi

    # Install the latest code-server.
    # Append "--version x.x.x" to install a specific version of code-server.
    curl -fsSL https://code-server.dev/install.sh | sh -s -- --method=standalone --prefix=/tmp/code-server

    # Start code-server in the background.
    /tmp/code-server/bin/code-server --auth none --port 13337 >/tmp/code-server.log 2>&1 &

    sudo apt update
    sudo apt install -y gpg-agent keychain direnv

    # Start Docker
    sudo dockerd 2>&1 &
  EOT

  # These environment variables allow you to make Git commits right away after creating a
  # workspace. Note that they take precedence over configuration defined in ~/.gitconfig!
  # You can remove this block if you'd prefer to configure Git manually or using
  # dotfiles. (see docs/dotfiles.md)
  #env = {
  #  GIT_AUTHOR_NAME     = coalesce(data.coder_workspace_owner.me.full_name, data.coder_workspace_owner.me.name)
  #  GIT_AUTHOR_EMAIL    = "${data.coder_workspace_owner.me.email}"
  #  GIT_COMMITTER_NAME  = coalesce(data.coder_workspace_owner.me.full_name, data.coder_workspace_owner.me.name)
  #  GIT_COMMITTER_EMAIL = "${data.coder_workspace_owner.me.email}"
  #}

  # The following metadata blocks are optional. They are used to display
  # information about your workspace in the dashboard. You can remove them
  # if you don't want to display any information.
  # For basic resources, you can use the `coder stat` command.
  # If you need more control, you can write your own script.
  metadata {
    display_name = "CPU Usage"
    key          = "0_cpu_usage"
    script       = "coder stat cpu"
    interval     = 10
    timeout      = 1
  }

  metadata {
    display_name = "RAM Usage"
    key          = "1_ram_usage"
    script       = "coder stat mem"
    interval     = 10
    timeout      = 1
  }

  metadata {
    display_name = "Home Disk"
    key          = "3_home_disk"
    script       = "coder stat disk --path $${HOME}"
    interval     = 60
    timeout      = 1
  }

  metadata {
    display_name = "CPU Usage (Host)"
    key          = "4_cpu_usage_host"
    script       = "coder stat cpu --host"
    interval     = 10
    timeout      = 1
  }

  metadata {
    display_name = "Memory Usage (Host)"
    key          = "5_mem_usage_host"
    script       = "coder stat mem --host"
    interval     = 10
    timeout      = 1
  }

  metadata {
    display_name = "Load Average (Host)"
    key          = "6_load_host"
    # get load avg scaled by number of cores
    script   = <<EOT
      echo "`cat /proc/loadavg | awk '{ print $1 }'` `nproc`" | awk '{ printf "%0.2f", $1/$2 }'
    EOT
    interval = 60
    timeout  = 1
  }

  metadata {
    display_name = "Swap Usage (Host)"
    key          = "7_swap_host"
    script       = <<EOT
      free -b | awk '/^Swap/ { printf("%.1f/%.1f", $3/1024.0/1024.0/1024.0, $2/1024.0/1024.0/1024.0) }'
    EOT
    interval     = 10
    timeout      = 1
  }
}

resource "coder_app" "code-server" {
  agent_id     = coder_agent.main.id
  slug         = "code-server"
  display_name = "code-server"
  url          = "http://localhost:13337/?folder=/home/coder"
  icon         = "/icon/code.svg"
  subdomain    = false
  share        = "owner"

  healthcheck {
    url       = "http://localhost:13337/healthz"
    interval  = 5
    threshold = 6
  }
}

resource "docker_volume" "home_volume" {
  name = "coder-${data.coder_workspace.me.id}-home"
  # Protect the volume from being deleted due to changes in attributes.
  lifecycle {
    ignore_changes = all
  }
  # Add labels in Docker to keep track of orphan resources.
  labels {
    label = "coder.owner"
    value = data.coder_workspace_owner.me.name
  }
  labels {
    label = "coder.owner_id"
    value = data.coder_workspace_owner.me.id
  }
  labels {
    label = "coder.workspace_id"
    value = data.coder_workspace.me.id
  }
  # This field becomes outdated if the workspace is renamed but can
  # be useful for debugging or cleaning out dangling volumes.
  labels {
    label = "coder.workspace_name_at_creation"
    value = data.coder_workspace.me.name
  }
}

resource "docker_container" "workspace" {
  count = data.coder_workspace.me.start_count
  image = "codercom/enterprise-base:ubuntu"
  # Uses lower() to avoid Docker restriction on container names.
  name = "coder-${data.coder_workspace_owner.me.name}-${lower(data.coder_workspace.me.name)}"
  command = ["sh", "-c", coder_agent.main.init_script]
  # Hostname makes the shell more user friendly: coder@my-workspace:~$
  hostname = data.coder_workspace.me.name
  runtime = "sysbox-runc"
  env        = [
    "CODER_AGENT_TOKEN=${coder_agent.main.token}",
  ]
  host {
    host = "host.docker.internal"
    ip   = "host-gateway"
  }
  volumes {
    container_path = "/home/coder"
    volume_name    = docker_volume.home_volume.name
    read_only      = false
  }

  # Add labels in Docker to keep track of orphan resources.
  labels {
    label = "coder.owner"
    value = data.coder_workspace_owner.me.name
  }
  labels {
    label = "coder.owner_id"
    value = data.coder_workspace_owner.me.id
  }
  labels {
    label = "coder.workspace_id"
    value = data.coder_workspace.me.id
  }
  labels {
    label = "coder.workspace_name"
    value = data.coder_workspace.me.name
  }
}

Environment

  • Host OS: Ubuntu 24.04.1
  • Coder version: 2.19.0

Additional Context

No response

Activity

  1. added
    s2Broken use cases or features (with a workaround). Only humans may set this.
    and removed
    needs-triageIssue that require triage
    on Feb 6, 2025
  2. matifali commented on Feb 6, 2025

    @matifali
    Member

    Thanks for the report. Could you update the issue to also share your Docker, containerd and sysbox-ce version?

  3. jplu commented on Feb 6, 2025

    @jplu
    Author

    Here the versions I have on my host:

    • Docker version 27.4.1, build b9d17ea
    • containerd containerd.io 1.7.24 88bf19b2105c8b17560993bee28a01ddc2f97182
    • sysbox-ce 0.6.6 1b440ff266841f3d2d296e664122a9e29ceb9fd8
  4. matifali commented on Feb 6, 2025

    @matifali
    Member

    Could you also share a screenshot of the workspace page? And what message do you see there?

  5. jplu commented on Feb 6, 2025

    @jplu
    Author

    Is-it what you asked for?

    Image

    For the message you can see it in full length in my first message.

  6. matifali commented on Feb 6, 2025

    @matifali
    Member

    Yes. It looks like the agent is connected, and you can connect via VSCode and the web terminal. Can you confirm that?

    We are also working on redesigning the page, which would better show these errors. See #15423

  7. jplu commented on Feb 6, 2025

    @jplu
    Author

    Yes I confirm that I can connect to coder-server and terminal, and the containers inside are properly working. My issue was more how to fix the warning, because I don't know what would be the cause in my init script. Any idea?

  8. matifali commented on Feb 6, 2025

    @matifali
    Member

    @mafredri do you have an idea what could be causing that warning?

  9. mafredri commented on Feb 7, 2025

    @mafredri
    Member

    Sure, this happens because the dockerd command it keeping references to stdout:

    sudo dockerd 2>&1 &
    

    This only redirects stderr to stdout, to prevent the error we should redirect stdout too (note order is important here):

    sudo dockerd >/tmp/dockerd.log 2>&1 &
    

    We used to have better docs on writing startup scripts but I think those disappeared in a refactor as I can't find any reference to them in our current docs.

  10. removed
    s2Broken use cases or features (with a workaround). Only humans may set this.
    on Feb 7, 2025
  11. jplu commented on Feb 7, 2025

    @jplu
    Author

    Thanks very much. It perfectly works.

  12. locked and limited conversation to collaborators on Feb 7, 2025
  13. converted this issue into a discussion #16487 on Feb 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions