Scheduling backups

Create client service accounts for Cloud Scheduler and Cloud Run functions

1. If you haven't already done so, from the Google Cloud console, click Activate Cloud Shell.

2. Create a client service account that Cloud Scheduler runs as to invoke a Cloud Run functions function. For this example, use the iam service-accounts create command to name the account schedulerunner and set the display name to "Service Account for FS Backups-Scheduler":

   gcloud iam service-accounts create schedulerunner \
       --display-name="Service Account for FS Backups-Scheduler"
   

3. Create a client service account that Cloud Run functions runs as to call the Filestore endpoint. For this example, we name the account backupagent and set the display name to "Service Account for FS Backups-GCF":

   gcloud iam service-accounts create backupagent \
       --display-name="Service Account for FS Backups-GCF"
   

   You can check whether the service account is created by running the iam service-accounts list command:

   gcloud iam service-accounts list
   

   The command returns something like this:

   NAME                                         EMAIL                                                   DISABLED
   Service Account for FS Backups-GCF           backupagent@$PROJECT_ID.         False
   Service Account for FS Backups-Scheduler     schedulerunner@$PROJECT_ID.      False
   

Set up environment variables

Set up the following environment variables in your local environment:

• Google Cloud project ID and project:

  export PROJECT_ID=`gcloud config get-value core/project`
  export PROJECT_NUMBER=`gcloud projects describe $PROJECT_ID --format="value(projectNumber)"`
  

• The Cloud Scheduler service agent and the client service accounts for Cloud Scheduler and Cloud Run functions:

  export SCHEDULER_SA=service-$PROJECT_NUMBER@gcp-sa-cloudscheduler.
  export SCHEDULER_CLIENT_SA=schedulerunner@$PROJECT_ID.
  export GCF_CLIENT_SA=backupagent@$PROJECT_ID.
  

• Your Filestore instance:

  export SOURCE_INSTANCE_LOCATION=fs-location
  export SOURCE_INSTANCE_NAME=instance-id
  export SHARE_NAME=file-share-name
  

  Replace the following:

  • fs-location with the zone or region where the source Filestore instance resides.
  • instance-id with the instance ID of the source Filestore instance.
  • file-share-name with the name you specify for the NFS file share that is served from the instance.

• Set up environment variables for your Filestore backup:

  export BACKUP_REGION=backup-region
  

  Replace backup-region with the region where you want to store the backup.

Create a function that creates a backup

1. In the Google Cloud console, go to the Cloud Run functions page.

   Go to the Cloud Run functions page

2. Click Write a function and configure the function as follows:

   • Configure:
     • Service Name: For this example, we name the function fsbackup.
     • Region: For this example, select us-central1.
     • Runtime: Select any supported Python 3 Runtime fully supported by Cloud Run functions from the menu.
   • Trigger:
     • Trigger doesn't need to be set for this example.
   • Authentication: Select Require authentication.
   • Ingress: Select All.
   • Container(s), Volumes, Networking, Security
     • Go to Security tab and select Service Account for FS Backups-GCF (backupagent@$PROJECT_ID.) from the menu.

3. Click Create and continue the configuration as follows:

   • Function entry point: Enter create_backup.

   • Add the following dependencies to your requirements.txt file:

     functions-framework==3.*
     google-auth==2.29.0
     requests==2.31.0
     

     Depending on your use case, you may need to specify other dependencies along with their corresponding version numbers. For more information, see Pre-installed packages.

   • Copy the following Python code sample into the main.py file using the inline editor:

     Create a backup

     This code sample creates a backup named mybackup- appended with the creation time.

     PROJECT_ID = 'project-id'
     SOURCE_INSTANCE_LOCATION = 'fs-location'
     SOURCE_INSTANCE_NAME = 'instance-id'
     SOURCE_FILE_SHARE_NAME = 'file-share-name'
     BACKUP_REGION = 'backup-region'
     
     import functions_framework
     import google.auth
     import google.auth.transport.requests
     from google.auth.transport.requests import AuthorizedSession
     import time
     import requests
     import json
     
     credentials, project = google.auth.default()
     request = google.auth.transport.requests.Request()
     credentials.refresh(request)
     authed_session = AuthorizedSession(credentials)
     
     def get_backup_id():
         return "mybackup-" + time.strftime("%Y%m%d-%H%M%S")
     
     @functions_framework.http
     def create_backup(request):
         trigger_run_url = "https://file.googleapis.com/v1/projects/{}/locations/{}/backups?backupId={}".format(PROJECT_ID, BACKUP_REGION, get_backup_id())
         headers = {
           'Content-Type': 'application/json'
         }
         post_data = {
           "description": "my new backup",
           "source_instance": "projects/{}/locations/{}/instances/{}".format(PROJECT_ID, SOURCE_INSTANCE_LOCATION, SOURCE_INSTANCE_NAME),
           "source_file_share": "{}".format(SOURCE_FILE_SHARE_NAME)
         }
         print("Making a request to " + trigger_run_url)
         r = authed_session.post(url=trigger_run_url, headers=headers, data=json.dumps(post_data))
         data = r.json()
         print(data)
         if r.status_code == requests.codes.ok:
           print(str(r.status_code) + ": The backup is uploading in the background.")
         else:
           raise RuntimeError(data['error'])
         return "Backup creation has begun!"
     

     Replace the following:

     • project-id with the Google Cloud project ID of the source Filestore instance.
     • fs-location with the zone or region of the source Filestore instance.
     • instance-id with the name of the source Filestore instance.
     • file-share-name with the name of the file share.
     • backup-region with the region to store the backup.

     1. Click Test

        A new tab session opens in Cloud Shell. In it, the following message is returned if successful:

        Backup creation has begun!
        

     2. Click Save and redeploy and wait for the deployment to finish.

     3. Switch back to the previous Cloud Shell tab.

     Delete a backup

     This code sample deletes backups older than a predefined period.

     You can only delete one backup per source instance at a time. For more information, see Backups.

     Configure this function in the same way as the function you used to create a backup, using the following modifications:

     • Function name: deletefsbackup.
     • Entry point: delete_backup.

     PROJECT_ID = 'project-id'
     BACKUP_REGION = 'region'
     BACKUP_RETENTION_TIME_HRS = hours
     
     import functions_framework
     import google.auth
     import google.auth.transport.requests
     from google.auth.transport.requests import AuthorizedSession
     import time
     import requests
     import json
     
     credentials, project = google.auth.default()
     request = google.auth.transport.requests.Request()
     credentials.refresh(request)
     authed_session = AuthorizedSession(credentials)
     
     retention_seconds = BACKUP_RETENTION_TIME_HRS * 60 * 60
     
     @functions_framework.http
     def delete_backup(request):
         now = time.time()
         backup_list = []
         trigger_run_url = "https://file.googleapis.com/v1/projects/{}/locations/{}/backups".format(PROJECT_ID, BACKUP_REGION)
         r = authed_session.get(trigger_run_url)
         data = r.json()
         if not data:
             print("No backups to delete.")
             return "No backups to delete."
         else:
             backup_list.extend(data['backups'])
             while "nextPageToken" in data.keys():
                 nextPageToken = data['nextPageToken']
                 trigger_run_url_next = "https://file.googleapis.com/v1/projects/{}/locations/{}/backups?pageToken={}".format(PROJECT_ID, BACKUP_REGION, nextPageToken)
                 r = authed_session.get(trigger_run_url_next)
                 data = r.json()
                 backup_list.extend(data['backups'])
         for i in backup_list:
             backup_time = i['createTime']
             backup_time = backup_time[:-4]
             backup_time = float(time.mktime(time.strptime(backup_time, "%Y-%m-%dT%H:%M:%S.%f")))
             i['backup_timestamp'] = backup_time
         sorted_backup_list = sorted(backup_list, key=lambda d: d['backup_timestamp'])
         oldest_backup = sorted_backup_list[0]
         if now - oldest_backup['backup_timestamp'] > retention_seconds:
             print(oldest_backup['name'] + " is older than the indicated retention time.")
             r = authed_session.delete("https://file.googleapis.com/v1/{}".format(oldest_backup['name']))
             data = r.json()
             print(data)
             if r.status_code == requests.codes.ok:
                 print(str(r.status_code) + ": Deleting " + oldest_backup['name'] + " in the background.")
             else:
                 raise RuntimeError(data['error'])
             return "Backup deletion has begun!"
         return "All backups are within the indicated retention period."
     

     Replace the following:

     • project-id with the Google Cloud project ID of the backup.
     • region with the region where the backup resides. The backup, scheduler job, and function should all reside in the same location.
     • hours with the number of hours to retain backups. For example, if you want to retain backups for 10 days, input 240.

Assign IAM roles to the client service accounts

1. Add the Cloud Scheduler service agent to the IAM policy of the Cloud Scheduler client service account with the role of roles/cloudscheduler.serviceAgent. This allows the service agent to impersonate the client service account in order to invoke the function that creates a backup. Run the iam service-accounts add-iam-policy-binding command:

   gcloud iam service-accounts add-iam-policy-binding $SCHEDULER_CLIENT_SA \
       --member=serviceAccount:$SCHEDULER_SA \
       --role=roles/cloudscheduler.serviceAgent
   

2. Give the client service account of Cloud Run functions the roles/file.editor role so that it can make calls to the Filestore endpoint. Run the projects add-iam-policy-binding command:

   gcloud projects add-iam-policy-binding $PROJECT_ID \
       --member=serviceAccount:$GCF_CLIENT_SA \
       --role=roles/file.editor
   

3. Grant the client service account of Cloud Scheduler the role of roles/run.invoker for the function you want to use. Run the following run services add-iam-policy-binding command:

   Create a backup

   gcloud run services add-iam-policy-binding fsbackup \
       --member serviceAccount:$SCHEDULER_CLIENT_SA \
       --role roles/run.invoker \
       --region=us-central1
   

   Now, only the client service account of Cloud Scheduler can invoke fsbackup.

   Delete a backup

   gcloud run services add-iam-policy-binding deletefsbackup \
       --member serviceAccount:$SCHEDULER_CLIENT_SA \
       --role roles/run.invoker
   

   Now, only the client service account of Cloud Scheduler can invoke deletefsbackup.

Create a Cloud Scheduler job that triggers the function on a specified schedule

Low quota alerts for backups

If your implementation of scheduling backups puts you at risk of running out of backups quota, we recommend that you set up low backups quota alerts. This way, you are notified when backups quota runs low.