This example shows how to use Cloud Scheduler, Pub/Sub and Cloud Functions to configure a schedule that creates Cloud Bigtable backups periodically.

The idea is to have a Cloud Scheduler job that invokes a Cloud function by sending a message to the Pub/Sub topic which contains information about the Cloud Bigtable backup creation request. Then the Cloud function initiates a backup using Cloud Bigtable Java API.

Before proceeding with the tutorial, ensure the following:

• A Cloud Bigtable table exists in the same Google Cloud project. Please check Cloud Bigtable documentation if needed.
• Google Cloud SDK is installed

The diagram below focuses on the actions flow between human roles and APIs.

The administrator should be granted specific roles to deploy the services needed for the solution.

You also need a custom role (ie. backups-admin) with below permissions * appengine.applications.create - for Cloud Scheduler to create an App Engine app * serviceusage.services.use - for Cloud Scheduler to use the App Engine app

Cloud Functions calls Cloud Bigtable API to create a backup, it gets triggered when a message arrives on the Pub/Sub topic. For successful execution of the cloud function, it should be able to consume from the Pub/Sub topic and should have permissions to create Cloud Bigtable backups. To accomplish this, perform the following steps:

1. Create a Service Account (e.g. cbt-scheduled-backups@iam.gserviceaccount.com).
2. Create a custom role (e.g. backups-admin) with the permissions:
   • bigtable.backups.create
   • bigtable.backups.delete
   • bigtable.backups.get
   • bigtable.backups.list
   • bigtable.backups.restore
   • bigtable.backups.update
   • bigtable.instances.get
   • bigtable.tables.create
   • bigtable.tables.readRows
3. Assign the custom role and roles/pubsub.subscriber to the service account. This allows Cloud Functions to read messages from the Pub/Sub topic and initiate a create backup request.
4. Add the administrator as a service account user by adding the user as a member of the service account with role roles/iam.serviceAccountUser. This allows the administrator to deploy Cloud Functions.

1. Clone this directory and make changes to ./config/scheduled-backups.properties file to match your configuration. Fields that need to be updated start with "replace".

2. Create a Cloud Pub/Sub topic cloud-bigtable-scheduled-backups that serves as the target of the Cloud Scheduler job and triggers the Cloud function. For example:

gcloud pubsub topics create cloud-bigtable-scheduled-backups --project <project-id>

1. Create and deploy a Cloud Function cbt-create-backup-function which is called whenever a Pub/Sub message arrives in cloud-bigtable-scheduled-backups topic:

./scripts/scheduled_backups.sh deploy-backup-function

1. Deploy the scheduled backup configuration to Cloud Scheduler:

./scripts/scheduled_backups.sh create-schedule

To get email notifications on backup creation failures, follow these steps:

1. Follow this guide to add your email address as a notification channel.

2. Create and deploy a custom metrics configuration file to filter logs generated by Cloud Functions, Cloud Scheduler and Cloud Bigtable. We use Deployment Manager to create custom metrics. The example file can be found in ./config/metrics.yaml. Deploy the custom metrics in Cloud Logging:

./scripts/scheduled_backups.sh add-metrics

After this, you should see two user-defined metrics under Logs-based Metrics in Cloud Logging.

1. Go to Logs-based Metrics in Cloud Logging and select Create alert from metric option for each of the two metrics created in the above step. From there, you can choose Aggregrator, such as sum or mean, for the target metric, and define what the condition of triggering an alert is, e.g., any time series violates that the value is above 0 for 1 minute.

2. Add notification channels you just created to alerting policies.

After you've finished this tutorial, you can clean up the resources created during this tutorial and you won't be billed for them in the future. Including:

• Delete the Pub/Sub topic.
• Delete the Cloud Scheduler job.
• Delete the Cloud function.

Or you can simply delete the project. Please be cautions because you will also delete any other work you've done in the project.

To use Cloud Scheduler, you must create an App Engine app. Once you set a zone for the App Engine app, you cannot change it. Your Cloud Scheduler job will be running in the same zone as your App Engine app.