About privileged workload admission in Autopilot mode

You can control which privileged workloads can run in Google Kubernetes Engine (GKE) Autopilot mode by using allowlists and policies. This document describes the default constraints that Autopilot enforces on workloads, the types of privileged workloads that you can run in Autopilot mode, and the methods of controlling privileged workload admission.

This document is for the following people:

• Platform admins and operators who want to run privileged GKE workloads from various sources in Autopilot mode.
• Identity and account admins who want to improve the security posture of your organization by blocking all privileged workloads except for specific workloads.

About privileged workloads in Autopilot mode

To improve your security posture, Autopilot mode clusters and nodes enforce specific security constraints by default. For example, Autopilot rejects most Pods that set the spec.securityContext.privileged field to true in a container. Privileged workloads in Autopilot are workloads that don't meet these default constraints. By default, Autopilot nodes in a Google Cloud organization reject privileged workloads.

Specific privileged workloads can run in Autopilot mode only if the cluster has allowlists that match those workloads. When a cluster admin creates or modifies a cluster, they can specify paths to allowlists that the cluster can run. By default, every Google Cloud organization supports allowlists from Autopilot partners and verified open source projects.

Organization administrators can modify this default behavior in the following ways by using Organization Policy Service:

• Allow all workloads from GKE Autopilot partners and verified open source workloads. This is the default behavior in all Google Cloud organizations.
• Allow specific partner workloads or verified open source workloads.
• Allow only specific customer-owned privileged workloads.
• Deny all privileged workloads.

This deny all, and then allow some approach is a security best practice that lets you control exactly what runs with elevated privileges in your GKE clusters.

You might want to control privileged workloads in Autopilot mode in situations like the following:

• You run workloads in Autopilot mode in Standard clusters and you want privileged workloads to run on those nodes.
• You want to allow only a specific set of privileged workloads to run in your organization, folder, or project.

Sources of privileged Autopilot workloads

The following table describes the types of privileged workloads that you can run in Autopilot:

How privileged workload admission control works

To allow specific privileged workloads to run in your Autopilot nodes, you install allowlists that correspond to each workload. These allowlists are WorkloadAllowlist Kubernetes custom resources. GKE validates the workload details against any WorkloadAllowlists in the cluster, and allows admission only if the details match. By default, every Autopilot or Standard cluster in a Google Cloud organization lets you install WorkloadAllowlists for Autopilot partner workloads and for open source workloads.

At a high level, running privileged workloads in Autopilot nodes involves these steps:

1. The organization administrator verifies that the container.managed.autopilotPrivilegedAdmission managed Organization Policy constraint allows the installation of allowlists from a specific source. For more information, see the Organization Policy managed constraint for allowlists section.
2. For customer-owned privileged workloads, the platform administrator creates allowlists that match the privileged workloads. For more information, see the Customer-owned privileged workloads section.
3. The cluster admin configures an Autopilot or Standard cluster to allow installation of allowlists from a specific source. For more information, see the Cluster configuration section.
4. The cluster admin creates an AllowlistSynchronizer that references the path to the allowlist. The AllowlistSynchronizer installs the allowlist and keeps the allowlist up to date. For more information, see the Allowlist installation section.

After these steps are completed, an app operator can deploy the privileged workload in the cluster.

Organization Policy managed constraint for allowlists

By default, every Google Cloud organization enforces an organization policy that's based on the container.managed.autopilotPrivilegedAdmission managed constraint. This constraint defines a set of file or directory paths from which GKE clusters can install allowlists. The cluster configuration must use allowlist paths from the parameters of this constraint.

By default, the autopilotPrivilegedAdmission constraint allows the use of all allowlists that are owned by approved GKE partners and open source projects. This setting matches the default configuration of GKE clusters, which can install any of those approved allowlists.

As an organization policy administrator, you can modify the set of allowlist paths that can be applied to GKE clusters in your organization, folder, or project. For example, to allow the use of a custom allowlist that your organization owns, you add that allowlist path to the managed constraint.

Managed constraint parameters

The autopilotPrivilegedAdmission managed constraint has the following specification:

spec:
  rules:
  - enforce: true
    parameters:
      allowAnyGKEPath: ALLOW_GKE_PATHS
      allowPaths:
      - PATH1
      - PATH2
      - PATH3

The following fields in the specification control the set of allowlists that clusters can use:

• allowAnyGKEPath: a boolean value that controls whether clusters can use the default set of GKE-approved allowlist paths. This parameter supports the following values:

  • True: cluster admins can configure clusters without modifying the allowlist configuration, with one or more allowlist paths that start with gke://, or with an empty string to disable all allowlists for the cluster. True is the default value for this parameter.

  • False: cluster admins must configure clusters with an empty string for allowlist sources or with any allowlist paths from the allowPaths parameter.

• allowPaths: a list of approved allowlist sources that cluster admins can specify values from when they configure a cluster. You can specify paths to specific allowlist files or to entire directories.

The following table describes the results of configuring these parameters in different ways:

For more information about controlling privileged workload admission in organization policies, see Restrict privileged GKE workloads in organizations.

Customer-owned privileged workloads

As a GKE customer, you might have your own specialized workloads that require more privileges in the cluster than the default Autopilot security constraints allow. In GKE version 1.35 and later, you can exempt these workloads from the default Autopilot constraints by creating allowlists.

An allowlist is a YAML file that defines a WorkloadAllowlist Kubernetes custom resource. The specification of a WorkloadAllowlist matches various fields in the Kubernetes Pod specification. When you define a WorkloadAllowlist, you match the values in the WorkloadAllowlist to corresponding fields in the specification of your privileged workload. You store the YAML files in a Cloud Storage bucket.

To install these customer-owned allowlists in a cluster, the following steps must happen:

1. Your organization administrator adds the Cloud Storage paths to an organization policy. For more information, see the About the Organization Policy managed constraint section.
2. You add the Cloud Storage paths to a cluster configuration. For more information, see the Cluster configuration section.

For more information about creating your own WorkloadAllowlists, see Create allowlists for privileged Autopilot workloads.

Cluster configuration

To run a privileged workload in Autopilot mode, a cluster admin adds one or more approved allowlist paths to the cluster configuration. After you add the paths to the cluster, app operators can install the corresponding allowlists and privileged workloads in the cluster.

By default, you can install allowlists from Autopilot partners and approved open source projects. This default set of approved sources for allowlists might change because of an organization policy, as described in the preceding section. If an organization policy controls the approved allowlist paths, you must coordinate with your organization administrator to identify those paths. You can then specify one or more of the approved paths in the cluster configuration.

For more information about configuring clusters, see Run privileged workloads in Autopilot.

Allowlist installation

To install an allowlist from an approved source in your cluster, you use a Kubernetes custom resource named the AllowlistSynchronizer. You define and create AllowlistSynchronizers similarly to how you create other Kubernetes resources like Pods and Deployments. Each AllowlistSynchronizer has a list of paths to allowlists that you want to install in the cluster.

When you create an AllowlistSynchronizer, GKE installs allowlists from these paths and keeps the allowlists up-to-date. Every 10 minutes, GKE checks for changes to allowlist files. If an update exists, the synchronizer installs the updated allowlist in the cluster.

After GKE installs an allowlist based on an AllowlistSynchronizer, you can create the corresponding privileged workload in the cluster.

For more information about installing allowlists in a cluster, see Run privileged workloads in Autopilot.

Allowlist paths

When you configure allowlist support in your organization policy or clusters, you specify paths to one or more WorkloadAllowlists that you've approved for installation. The syntax for these paths depends on who created the WorkloadAllowlist, as follows:

• Specific Autopilot partners and approved open source projects:

  gke://REPOSITORY_PATH/SELECTOR
  

  Replace the following:

  • REPOSITORY_PATH: the path to the directory that contains allowlists for the workload, such as Grafana/alloy. For more information about the allowlist paths for these workloads, see the following pages:
    • Autopilot partner workloads
    • Privileged open source workloads with Autopilot support
  • SELECTOR: the name of a specific file or the * character, which selects every file in that directory.

  For example, the following path selects any allowlist in the Grafana/alloy directory:

  gke://Grafana/alloy/*
  

• Customer-owned privileged workloads:

  gs://DIRECTORY_PATH/SELECTOR
  

  Replace DIRECTORY_PATH with the path to the directory in Cloud Storage that contains your allowlist files. You can specify multiple paths.

You can specify one or more of the preceding paths in your organization policy or cluster configuration. For example, you can add GKE partner workloads and customer-owned workloads to the same configuration. You can also use the * character to select all allowlists. For example, gke://* selects any GKE-approved allowlist from any source.

The following table describes the configuration that's needed in organization policies or clusters to support certain objectives:

What's next

• Restrict privileged GKE workloads in organizations.
• Run privileged workloads in Autopilot.