View instance distribution logs

When you create a regional managed instance group (MIG) or increase its target size, Compute Engine evaluates real-time resource availability to distribute new compute instances across zones and machine types. During this process, Compute Engine records an instance distribution log entry in Cloud Logging.

This document describes how to view and interpret these instance distribution logs. It also explains how to use these logs to investigate pending allocations and configure alerts.

To see the actual distribution of your running instances, see Monitor instance distribution in MIGs.

About instance distribution logs

Instance distribution logs provide a one-time snapshot of the MIG's planned allocations for compute instances. In these logs, an allocated instance means that the MIG found capacity and plans to create the compute instance on that machine type and in that zone. An allocated size doesn't mean that the compute instances are successfully created yet.

Because the logs provide a snapshot, Compute Engine doesn't update these logs when the MIG creates the remaining compute instances later. The log entry lets you view the following details:

• Machine type distribution: the number of compute instances that the MIG has allocated per machine type. This information is present only for regional MIGs that use instance flexibility.

• Zonal distribution: the number of compute instances that the MIG has allocated per zone.

• Instances awaiting capacity: the number of instances for which the MIG couldn't allocate capacity immediately and is actively seeking capacity.

Limitations

• Instance distribution logs are available only for regional MIGs.

• For regional MIGs that use the EVEN target distribution shape without autoscaling, the log entry includes only the oldSize, newSize, deltaSize, and targetShape fields, without any instance distribution details.

• Compute Engine logs instance distribution details only when you create a MIG or increase its target size. Compute Engine doesn't log these details when the target size decreases, or during instance repair, recreate, and update operations.

View instance distribution logs of a specific MIG

To view the instance distribution logs of a specific MIG, use one of the following options:

gcloud logging read "logName=\"projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event\" AND \
  protoPayload.methodName=\"compute.instanceGroupManagers.instanceDistribution\" AND \
  resource.labels.instance_group_manager_name=\"MIG_NAME\""

POST https://logging.googleapis.com/v2/entries:list
{
"resourceNames": [
  "projects/PROJECT_ID"
],
"filter": "logName=\"projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event\" AND protoPayload.methodName=\"compute.instanceGroupManagers.instanceDistribution\" AND resource.labels.instance_group_manager_name=\"MIG_NAME\""
}

For more information about each field in a log, see Instance distribution log entry.

View all instance distribution logs in a project

To view all instance distribution logs in a project, select one of the following options:

gcloud logging read "logName=\"projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event\" AND \
  protoPayload.methodName=\"compute.instanceGroupManagers.instanceDistribution\""

gcloud logging read "logName=\"projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event\" AND \
    protoPayload.methodName=\"compute.instanceGroupManagers.instanceDistribution\"" \
    --freshness=1d

gcloud logging read "logName=\"projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event\" AND \
    protoPayload.methodName=\"compute.instanceGroupManagers.instanceDistribution\" AND \
    protoPayload.metadata.details.data.activeSeekDeltaSize > 0"

POST https://logging.googleapis.com/v2/entries:list
{
  "resourceNames": [
    "projects/PROJECT_ID"
  ],
  "filter": "logName=\"projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event\" AND protoPayload.methodName=\"compute.instanceGroupManagers.instanceDistribution\""
}

POST https://logging.googleapis.com/v2/entries:list
{
  "resourceNames": [
    "projects/PROJECT_ID"
  ],
  "filter": "logName=\"projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event\" AND protoPayload.methodName=\"compute.instanceGroupManagers.instanceDistribution\" AND timestamp >= \"YYYY-MM-DDThh:mm:ssZ\""
}

POST https://logging.googleapis.com/v2/entries:list
{
  "resourceNames": [
    "projects/PROJECT_ID"
  ],
  "filter": "logName=\"projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event\" AND protoPayload.methodName=\"compute.instanceGroupManagers.instanceDistribution\" AND protoPayload.metadata.details.data.activeSeekDeltaSize > 0"
}

For more information about each field in a log, see Instance distribution log entry.

Instance distribution log entry

The following is a sample log entry (protoPayload.metadata.details). Because this event is a one-time snapshot, the log entry isn't updated when the remaining instances are created later. The log entry lets you view the details as follows:

• details.data: shows the structured details about the instance distribution.

• details.message: shows the summary of the instance distribution.

In this example, a regional MIG with instance flexibility is scaling out from 0 to 100. The MIG immediately allocates 80 of the 100 new instances across two ranked instance selections (20 c2-standard-4 instances in rank2-c2 selection and 60 n2d-standard-4 instances in rank3-n2d selection). The entry also shows that 40 compute instances are allocated in us-central1-b and us-central1-c each. For the remaining 20 instances (activeSeekDeltaSize), the MIG continues to seek capacity across all configured instance selections and zones. For regional MIGs without instance flexibility, the log entry omits the instanceSelections field and shows only the zonal distribution.

"metadata": {
  "details": {
    "data": {
      "oldSize": 0,
      "newSize": 100,
      "deltaSize": 100,
      "targetShape": "BALANCED",
      "activeSeekDeltaSize": 20,
      "instanceSelections": {
        "rank1-n2": {
          "rank": 1,
          "machineSelections": [
            {
              "machineType": "n2-standard-4",
              "allocatedSize": 0
            }
          ]
        },
        "rank2-c2": {
          "rank": 2,
          "machineSelections": [
            {
              "machineType": "c2-standard-4",
              "allocatedSize": 20
            }
          ]
        },
        "rank3-n2d": {
          "rank": 3,
          "machineSelections": [
            {
              "machineType": "n2d-standard-4",
              "allocatedSize": 60
            }
          ]
        }
      },
      "zonalDistribution": {
        "us-central1-a": {
          "allocatedSize": 0
        },
        "us-central1-b": {
          "allocatedSize": 40
        },
        "us-central1-c": {
          "allocatedSize": 40
        }
      }
    },
    "message": "Increasing MIG target size from 0 to 100 instances (+100).
    Allocated 80 of 100 new instances. Allocated instances across instance
    selections: rank2-c2 (rank 2; c2-standard-4: 20); rank3-n2d (rank 3;
    n2d-standard-4: 60). Allocated instances across zones based on target
    distribution shape BALANCED: us-central1-a: 0, us-central1-b: 40,
    us-central1-c: 40. MIG continues to allocate the remaining 20 new instances
    across all instance selections and zones. The final distribution across zones
    and machine types may differ as instances are created. View the group's
    instances for the current distribution."
  }
}

The following table describes each field in a log entry.

Interpret instance distribution logs

The instance distribution log provides a snapshot of the MIG's plan for compute instance creation. Because the log indicates that the MIG allocates compute instances before they are successfully created, you should use the log data alongside compute instance status and error messages to understand your MIG's actual behavior.

Use the following table to interpret common log outputs and determine if you need to take action:

Check pending instance allocations

If your MIG is still waiting for capacity (activeSeekDeltaSize > 0), then you can check the following to understand the reasons behind the pending allocations:

• Check instance creation errors. To see if instance creation failed after the MIG allocated capacity, or if compute instances are still waiting for capacity, check the instance errors. For more information, see List instance errors.

• Check instance status. To see the status and current action of each instance, list the instances in the MIG and check their instanceStatus field. For more information, see Check the status of managed instances.

• Review your configuration. If you've configured an instance flexibility policy in the MIG, then review your regional quotas for the machine families in the policy. Also, verify that your selected zones support the requested machine types. For more information, see View and manage quotas and Available regions and zones.

Create alerting policies based on logs

To get notified when MIG resize operations wait for capacity, you can create a log-based alerting policy in Logs Explorer. For example, you can trigger an alert when activeSeekDeltaSize is greater than 0.

To create this alert, do the following:

1. In the Google Cloud console, go to the Logs Explorer page.

   Go to Logs Explorer

2. Enter the following query:

   logName="projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event"
   protoPayload.methodName="compute.instanceGroupManagers.instanceDistribution"
   protoPayload.metadata.details.data.activeSeekDeltaSize > 0
   

   Replace PROJECT_ID with the ID of your project.

3. Click Run query to validate the query.

4. Create the alerting policy by following the instructions in the create a log-based alerting policy section of the Cloud Logging documentation.

What's next

• Learn how regional MIGs distribute compute instances across zones.
• Learn how multiple machine type selection works in MIGs.
• View instance flexibility configuration in a MIG.