Description

This feature request emerged from an issue being faced by EVE-OS Project. EVE is about to reach the maximum size allowed for rootfs keeping backward compatibility with older versions regarding the upgrade process. Also, some EVE variants produce larger rootfs images in size, which make impossible to upgrade from a "smaller size" variant to a "larger size" variant.

Goals

The main goal is to be able to produce a rootfs image that doesn't contain all images from the yml file inside it. Instead, part of the images (configurable) can be pulled during runtime and/or be stored outside the main rootfs.

Example

Let's consider a hypothetical example where three images are part of the rootfs acting as service containers:

services:
  - name: newlogd
      image: lfedge/eve-newlog:952cf852ea4825ec365f67c3a4eae84ee43bebb0-amd64
      cgroupsPath: /eve/services/newlogd
      oomScoreAdj: -999
  - name: edgeview
      image: lfedge/eve-edgeview:303439c3398ac3e19108caa87439685f0898ec85-amd64
      cgroupsPath: /eve/services/eve-edgeview
      oomScoreAdj: -800
  - name: debug
      image: lfedge/eve-debug:e300a196d807cfaf05c3ed78e10262dd85206145-amd64
      cgroupsPath: /eve/services/debug
      oomScoreAdj: -999

In this case, linuxkit will fetch and include the images newlogd, edgeview and debug inside the final rootfs image file.

Let's say edgeview and debug images are not critical to the system boot and might be instantiated (pulled, deployed, etc) during runtime. Thus, the contents of these images would not be part of the final rootfs image, reducing the file size. Instead, they would be pulled and instantiated by another service (e.g. linuxkit/init) during boot/runtime process.

High level Requirements

1. Images from yml file might not be fully included to the final rootfs image (only manifest for pull and validation during runtime). Images not included in the final rootfs image will be called as of now as "external images"
2. Linuxkit must be able to pull and deploy external images during runtime
3. Since network access is never guaranteed, it must provide a mechanism to install the external images in advance
4. In some use cases, rootfs it's a immutable file (e.g. SquashFS), so the metadata to validate the external images must still be present in the primary rootfs image
5. External images storage must be unique to the primary rootfs image, so there will be no clashes on a system with two different versions of a primary rootfs image

Brainstorm

One possible approach would be introduce a new option to the yml file to mark external images. For instance:

services:
  - name: newlogd
      image: lfedge/eve-newlog:952cf852ea4825ec365f67c3a4eae84ee43bebb0-amd64
      cgroupsPath: /eve/services/newlogd
      oomScoreAdj: -999
  - name: edgeview
      image: lfedge/eve-edgeview:303439c3398ac3e19108caa87439685f0898ec85-amd64
      cgroupsPath: /eve/services/eve-edgeview
      oomScoreAdj: -800
      ondisk: false
  - name: debug
      image: lfedge/eve-debug:e300a196d807cfaf05c3ed78e10262dd85206145-amd64
      cgroupsPath: /eve/services/debug
      oomScoreAdj: -999
      ondisk: false

In this case, edgeview and debug would not be included to the final rootfs image, only their manifests. During system's initialization, linuxkit/init will check if external images are already present on disk or not. If so, it will proceed with the regular container instantiation. Otherwise it will pull them (from internet) and proceed with the deployment, save these images on disk, so next time they don't need get pulled again.

Open questions

1. Even though the idea is to pull/deploy external images during runtime, the initial goal is to allow these images to live outside the primary rootfs image, which means that linuxkit must provide a way to deploy them in advance, along with the primary rootfs image. What would be a good approach for that? Produce 2 rootfs images? Produce a tarball with the file system structure (containerd) for external images?
2. Let's try to not reinvent the wheel, so the whole infrastructure provided by containerd can be used to fetch/validate/install these images. However, what would be the best format to release external images? This relates to question 1.