Skip to content

ol worker init Fails During Initialization with Exit Status 1 #260

Open
Open
ol worker init Fails During Initialization with Exit Status 1#260

Description

@Arun0A
Arun0A
opened on Nov 29, 2024

Environment Details

  • Operating System: MX Linux (VirtualBox VM)
  • Docker Version: Docker version 27.3.1, build ce12230
  • OpenLambda Version: Latest cloned from GitHub

System Resources

  • Disk Space: 16 GB (available)
  • Memory: 4 GB

Steps to Reproduce

1. Clone OpenLambda Repository

git clone https://github.com/open-lambda/open-lambda.git
cd open-lambda

2. Build Docker Images

make imgs/ol-wasm

3. Verify Docker Images

$ docker images
REPOSITORY    TAG       IMAGE ID       CREATED             SIZE
ol-wasm       latest    60caa6e8c79f   16 minutes ago      2.68GB
ol-min        latest    22c15eb4e98f   About an hour ago   520MB
ubuntu        22.04     97271d29cb79   2 months ago        77.9MB
hello-world   latest    d2c94e258dcb   19 months ago       13.3kB

4. Initialize OpenLambda

./ol worker init

Observed Behavior

The command fails with the following error:

Init OL directory at /home/***/projects/open-lambda/default-ol
    Extract 'ol-wasm' Docker image to /home/***/projects/open-lambda/default-ol/lambda (may take several minutes).
    Create handler/host/packages/resolve.conf over base image.
    Create /dev/(null,random,urandom) over base image.
exit status 1

Expected Behavior

The ol worker init command should successfully initialize the worker environment and prepare it for execution.

Troubleshooting Attempts

Docker Verification

  • Docker service is active, and other images work fine.

Image Rebuild

make imgs/ol-wasm

Directory Cleanup

rm -rf /home/***/projects/open-lambda/default-ol

Manual Image Extraction

docker save ol-wasm | tar -x -C /home/***/projects/open-lambda/default-ol/lambda
  • This completed successfully but did not resolve the issue.

Debugging Attempts

Debug Mode

DEBUG=1 ./ol worker init
  • Produced no additional information beyond the exit status 1 error.
  • No Docker logs indicating an issue with the container.

Request for Assistance

Could you help investigate why the initialization fails with exit status 1? Any guidance on additional debugging steps or fixes would be appreciated.

Activity

  1. Arun0A commented on Nov 30, 2024

    @Arun0A
    Arun0A
    on Nov 30, 2024
    Author

    Update:

    I am now able to initialise the worker but fails when trying to start up. I think it might be an issue with the cgroups configuration in the system. OpenLambda is attempting to write to a 'cgroup.subtree_control' file that doesn't exist. This is often due to a mismatch in the cgroups version (v1 vs. v2) or improper setup of the cgroup hierarchy.

    I noticed that both docker and ol worker are operated with cgroup2. How should I be handling this?

    Console:

    arb@armx:~/projects/open-lambda
$ sudo ./ol worker init worker1
Previous deployment found at /home/arb/projects/open-lambda/default-ol.
No OpenLambda instance is running. No further actions are needed.
Clean previous files in /home/arb/projects/open-lambda/default-ol
	Remove /home/arb/projects/open-lambda/default-ol/config.json
	Remove /home/arb/projects/open-lambda/default-ol/default-zygotes-40.json
	Keep /home/arb/projects/open-lambda/default-ol/lambda
	Remove /home/arb/projects/open-lambda/default-ol/ol.init
	Remove /home/arb/projects/open-lambda/default-ol/registry
	Remove /home/arb/projects/open-lambda/default-ol/worker
Init OL directory at /home/arb/projects/open-lambda/default-ol
	Reusing prior base at /home/arb/projects/open-lambda/default-ol/lambda (pass -b to reconstruct this)

You may optionally modify the defaults here: /home/arb/projects/open-lambda/default-ol/config.json

Next start a worker using the "ol worker up" command.
arb@armx:~/projects/open-lambda
$ sudo ./ol worker up worker1
No OpenLambda instance is running. No further actions are needed.
2024/11/30 16:03:34.134094 Saved PID 7015 to file /home/arb/projects/open-lambda/default-ol/worker/worker.pid
2024/11/30 16:03:34.134551 Starting new lambda server
2024/11/30 16:03:34.134809 Storage dir at /home/arb/projects/open-lambda/default-ol/worker/code
2024/11/30 16:03:34.134852 Storage dir at /home/arb/projects/open-lambda/default-ol/worker/scratch
2024/11/30 16:03:34.135004 Creating SandboxPool
2024/11/30 16:03:34.135373 create /sys/fs/cgroup/default-ol-sandboxes [CGROUP POOL default-ol-sandboxes]
panic: Error writing to /sys/fs/cgroup/default-ol-sandboxes/cgroup.subtree_control: write /sys/fs/cgroup/default-ol-sandboxes/cgroup.subtree_control: no such file or directory

goroutine 1 [running]:
github.com/open-lambda/open-lambda/ol/worker/sandbox/cgroups.NewCgroupPool({0x9d38e1, 0x9})
	/home/arb/projects/open-lambda/src/worker/sandbox/cgroups/pool.go:48 +0x499
github.com/open-lambda/open-lambda/ol/worker/sandbox.NewSOCKPool({0x9d38e1, 0x9}, 0xc0001709c0)
	/home/arb/projects/open-lambda/src/worker/sandbox/sockPool.go:37 +0x26
github.com/open-lambda/open-lambda/ol/worker/sandbox.SandboxPoolFromConfig({0x9d38e1, 0x9}, 0x0?)
	/home/arb/projects/open-lambda/src/worker/sandbox/sandbox.go:15 +0x97
github.com/open-lambda/open-lambda/ol/worker/lambda.NewLambdaMgr()
	/home/arb/projects/open-lambda/src/worker/lambda/lambdaManager.go:71 +0x215
github.com/open-lambda/open-lambda/ol/worker/event.NewLambdaServer()
	/home/arb/projects/open-lambda/src/worker/event/lambdaServer.go:84 +0x85
github.com/open-lambda/open-lambda/ol/worker/event.Main()
	/home/arb/projects/open-lambda/src/worker/event/server.go:234 +0x417
github.com/open-lambda/open-lambda/ol/worker.upCmd(0xc000125860?)
	/home/arb/projects/open-lambda/src/worker/commands.go:178 +0x4e5
github.com/urfave/cli/v2.(*Command).Run(0xc0000ca2c0, 0xc000024d80, {0xc000068ce0, 0x2, 0x2})
	/home/arb/projects/open-lambda/src/vendor/github.com/urfave/cli/v2/command.go:274 +0x998
github.com/urfave/cli/v2.(*Command).Run(0xc0000cbce0, 0xc000024c40, {0xc000170030, 0x3, 0x3})
	/home/arb/projects/open-lambda/src/vendor/github.com/urfave/cli/v2/command.go:267 +0xbe5
github.com/urfave/cli/v2.(*Command).Run(0xc0001646e0, 0xc000024b80, {0xc000024080, 0x4, 0x4})
	/home/arb/projects/open-lambda/src/vendor/github.com/urfave/cli/v2/command.go:267 +0xbe5
github.com/urfave/cli/v2.(*App).RunContext(0xc000160000, {0xaa89f0?, 0xe4c140}, {0xc000024080, 0x4, 0x4})
	/home/arb/projects/open-lambda/src/vendor/github.com/urfave/cli/v2/app.go:332 +0x5b7
github.com/urfave/cli/v2.(*App).Run(...)
	/home/arb/projects/open-lambda/src/vendor/github.com/urfave/cli/v2/app.go:309
main.main()
	/home/arb/projects/open-lambda/src/main.go:393 +0x985

  2. kaimast commented on Nov 30, 2024

    @kaimast
    kaimast
    on Nov 30, 2024
    Collaborator

    I noticed that both docker and ol worker are operated with cgroup2. How should I be handling this?

    I am not familiar with MX Linux. Is it still using old (v1) cgroups? That is not something we support anymore.

    You could try starting your system with systemd.unified_cgroup_hierarchy=1 as a kernel parameter, but, again, I am unclear if MX Linux supports this.

  3. Arun0A commented on Dec 3, 2024

    @Arun0A
    Arun0A
    on Dec 3, 2024
    Author

    @kaimast, Yes you are right, MX uses cgroup1. However, I mounted a cgroup2 type and also tried changing the kernel parameter(s) as you mentioned.

    Maybe you are right, MX doesn't support it anymore. I actually gave up on it and went for a virtual machine, and made it work on Ubuntu 22.04. :)

    Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions