We use skaffold to make local dev on kubernetes a lot easier. Skaffold allows us to use a single almost-identical toolchain to both do "docker-compose" style local dev (a little slower because our massive source tree is baked into the docker image), as well as deploy real production/test/etc instances to a k8s cluster.

This makes debugging your production infra setup really easy: its just like what you're running locally.

We recommend adding export SKAFFOLD_CACHE_ARTIFACTS=false to your ~/.zprofile or ~/.zshrc. Alternatively you can pass the --cache-artifacts=false flag to every skaffold invocation.

Skaffold's caching feature doesn't work well with many small files, and actually adds time to builds instead of saving them.

We recommend starting with Docker Desktop to run Kubernetes. If at some point you'd like to switch to a fully open source toolchain, try (minikube)[https://minikube.sigs.k8s.io/].

1. Install Docker Desktop
   1. Alternatively: brew install --cask docker
2. Docker Desktop Settings:
   1. Turn on Kubernetes
   2. If you installed docker before 2025: enable the containerd image store
   3. Set Resources->Disk Space to at least 100GB (!!!)
3. (Optional) Download Headlamp to watch as k8s resources are created, view logs, etc.

From code-dot-org/ run:

skaffold dev --trigger=manual -p setup-db -p setup-s3

NOTE: this command will not exit automatically when complete, and will take 30+ minutes. When you see the output: [setup-db] setup-db COMPLETE, press Ctrl-C to exit, and move on to the next step.

Alternatively, you can try dashboard before you ctrl-c: http://localhost-studio.code.org:13000

Once setup is done, the command to start dashboard is:

skaffold dev

Then open: http://localhost-studio.code.org:13000

We create a number of kubernetes resources in a stock dashboard dev setup, including mysql and redis deployments, a dashboard deployment (that can scale up if you set it to), configurationmaps, storage volumes, etc.

To understand what's going on, the easiest way is to explore: poke around to view logs, watch statuses change, etc. While this can all be done using kubectl, the stock k8s tool which peeks and pokes yaml, it can be really really useful to use an interactive k8s browser.

Recommendations:

1. Headlamp is a cross-platform open source k8s browser application in relatively early dev days. I recommend starting here for k8s exploration purposes.
2. The VSCode Kubernetes Extension is top-notch.
3. k9s is great if you want a keyboard navigable curses-style CLI.

Normally, you'll not need to create a new cluster, you'll deploy to an existing cluster or a local kubernetes cluster. But if you're starting from scratch (e.g., new org, disaster, curiosity, etc), see: k8s-gitops/bootstrap/codeai-k8s/README.md in the k8s-gitops repo.

Conventionally, the k8s-gitops repo will be checked out at ../k8s-gitops.

Commits to the main branch of k8s-gitops will be automatically synced to the live cluster by ArgoCD in approximately 3 minutes. See: k8s-gitops/README.md to learn more.

Skaffold is configured by skaffold.yaml (api docs). A particularly useful page is skaffold's CLI docs.

Here's a quick overview of the 4 most useful commands, run from the repo root:

• skaffold build to build the docker images, this will also happen automatically when you run other skaffold commands, if needed.
• skaffold dev run dashboard in "dev mode", this will tail log output to the console, and when you ctrl-c to quit, the app will be uninstalled from the cluster automatically.
• skaffold run deploy dashboard to the k8s cluster as a standalone app.
• skaffold dev -p ________: -p activates a skaffold profile, which can modify skaffold behavior (like running a job, or switching to a more production-like configuration), see profiles: in skaffold.yaml for some of the profiles.

From the repo root:

• ./k8s/bin/shell will shell you into the current dashboard kubernetes container, if you are running one (i.e. skaffold dev)
• ./k8s/bin/bundle_exec lets you run one-liners under a bundle exec, e.g. k8s/bin/bundle_exec irb
• ./k8s/bin/rake lets you run rake commands, e.g. k8s/bin/rake build
• ./skaffold run -p development or skaffold run -p production: deploy dashboard to the k8s cluster, unlike skaffold dev these will be deployed permanently, and you'll have to watch the cluster for log output, etc.
• You can stack skaffold profiles, so e.g. you could do skaffold run -p production,seed to do a seed against the production configuration.

We use helm as our kubernetes packaging approach, helm charts can be found in k8s/helm.

Skaffold is configured by skaffold.yaml in the root folder.

This setup builds "full docker images", including all the source code inside the image. This setup is good because it means production === development: you're running exactly the same image, and the image is all you need to spin up an instance. The downside is that it can get a little slow because we have such a giant repo. As a result, this docker container currently removes a huge amount of stuff, including pegasus, i18n locales, etc.

Dockerfiles are found in k8s/docker/*.dockerfile, note the .dockerignore files that are used to shape what's included in each image. The key issue is not to be sending 20GBs to the docker daemon when you build, that takes foooooorever and then each skaffold dev suddenly takes 2minutes.

To work on our k8s system, including dockerfiles, skaffold and helm charts, it can be a challenge because of our 10GB+ docker image size. Mimic allows you to deploy our full chart and test dockerfiles without including the actual contents of our app. Everything is stubbed.

For details, see: k8s/mimic/README.md