Backup and Restore for External Blobstores

This topic explains how to back up and restore external blobstores with BOSH Backup and Restore (BBR).

BBR supports Amazon S3 buckets, S3-compatible storage solutions, and Azure storage containers.

The necessary configuration and supported restore scenarios differ depending on the type of blobstore you use. For details, see the sections below.

S3-Compatible Unversioned Blobstores

BBR backs up external blobstores by copying blobs from live buckets to specified backup buckets. BBR uses the native copy functionality of your blobstore to transfer blobs between the live and backup buckets, without transferring the blobs to your BBR instance.

For resiliency and safety, store your backup buckets and live buckets in different regions.

Enable Backup and Restore of Your Unversioned S3-Compatible Blobstore

To back up a Cloud Foundry deployment that uses an unversioned S3-compatible external blobstore, you must colocate the s3-unversioned-blobstore-backup-restorer job from backup-and-restore-sdk-release as part of your deployment.

  1. Verify that your blobstore is one of the following:

    • Amazon S3 buckets
    • S3-compatible buckets with support for AWS Signature Version 4.

    Note: If your S3-compatible blobstore uses a custom CA certificate, refer to Configuring Trusted Certificates in the BOSH documentation. BBR automatically makes use of your configured trusted certificates.

  2. Create backup buckets for droplets, packages, and buildpacks. We recommend that either the backup buckets or copies of them should be in a different region to the live buckets.

  3. Include the enable-backup-restore-s3-unversioned.yml ops file in your deployment.

    Set the variables specified in vars-enable-backup-restore-s3-unversioned.yml to configure your backup bucket locations.

    Note: Apply enable-backup-restore.yml and use-s3-blobstore.yml before enable-backup-restore-s3-unversioned.yml. See Apply Ops Files in the Correct Order.

    If you do not use cf-deployment and ops files, you can still back up and restore external blobstores with BBR. Refer to the contents of the enable-backup-restore-s3-unversioned.yml ops file as an example and see the backup-and-restore-sdk-release documentation in GitHub for details.

WARNING: We strongly recommend creating a lifecycle rule on your backup buckets to minimize storage costs by permanently deleting old backups. For more information, see How Do I Create a Lifecycle Policy for an S3 Bucket? in the Amazon S3 documentation.

Supported Restore Scenarios for Your Unversioned S3-Compatible Blobstore

The restore process will copy blobs from a directory in the backup bucket to the live buckets (droplets, packages and buildpacks) in use by the CF deployment you are restoring into.

If your original backup buckets are lost but you are restoring from copies, ensure that the variables listed in vars-enable-backup-restore-s3-unversioned.yml point to the copies.

WARNING: Restoring to an unversioned bucket will overwite blobs with the backed up copies of those blobs. BBR will only modify the blobs that were copied to the backup bucket. Any other blobs will not be affected.

S3-Compatible Versioned Blobstores

BBR only supports S3-compatible buckets that are versioned and support AWS Signature Version 4. For more details about enabling versioning on your blobstore, see enable versioning.

External blobstores are backed up by storing the current version of each blob, not the actual files. Those versions will be set to be the current versions at restore time. This makes backups and restores faster, but also means that restores only work if the original bucket still exists.

If the bucket gets deleted, all its versions are deleted with it and you cannot restore it unless you have a replica. For how to restore from a replica, see Restore from Replicas below.

WARNING: The s3-versioned-blobstore-backup-restorer in backup-and-restore-sdk v1.5.1 and earlier does not support the backup and restore of an AWS S3 blobstore with individual blobs greater than 5GB.

Enable Backup and Restore of Your Versioned S3-Compatible Blobstore

To back up a Cloud Foundry deployment that uses an external blobstore, you must colocate the s3-versioned-blobstore-backup-restorer job from backup-and-restore-sdk-release as part of your deployment.

  1. Verify that your blobstore is one of the following:

    Note: If your S3-compatible blobstore uses a custom CA certificate, refer to Configuring Trusted Certificates in the BOSH documentation. BBR automatically makes use of your configured trusted certificates.

  2. Include the enable-backup-restore-s3-versioned.yml ops file in your deployment.

    Note: Apply enable-backup-restore.yml and use-s3-blobstore.yml before enable-backup-restore-s3-versioned.yml. See Apply Ops Files in the Correct Order.

    If you do not use cf-deployment and ops files, you can still back up and restore external blobstores with BBR. Refer to the contents of the enable-backup-restore-s3-versioned.yml ops file as an example and see the backup-and-restore-sdk-release documentation in GitHub for details.

Enable Versioning on Your S3-Compatible Blobstore

BBR only supports the backup and restore of blobstores stored in versioned, Amazon S3 buckets and in S3-compatible buckets that are versioned and support AWS Signature Version 4.

Three Cloud Foundry buckets are backed up by BBR, so you only need to enable versioning of:

  • droplets
  • packages
  • buildpacks

Enabling versioning of the resource_pool bucket is not required.

To enable versioning of Amazon S3 buckets, follow the instructions in the Amazon S3 documentation, How Do I Enable or Suspend Versioning for an S3 Bucket?. If you prefer to use the aws CLI, you can make use of the put-bucket-versioning command.

  1. If your blobstore buckets are not empty, run the following command on each one of them, using the AWS CLI:

    aws s3 cp s3://BUCKET-NAME/ s3://BUCKET-NAME/ --recursive --metadata bump=true

    For example,

    $ aws s3 cp s3://my-bucket/ s3://my-bucket/ --recursive --metadata bump=true
    

    This ensures that each file in your buckets has a valid version ID.

  2. After enabling versioning, you have a current version and zero or more non-current versions for each object.

    WARNING: We strongly recommend setting an expiration policy on your buckets to delete old non-current versions and minimize storage costs. For more information, see Example 6: Specifying a Lifecycle Rule for a Versioning-Enabled Bucket in the Amazon Simple Storage Service documentation.

Note: If your blobstore uses S3-compatible buckets not from Amazon, see the documentation for that storage provider regarding enabling versioning and setting an expiration policy.

Enable Replication on Your Versioned S3-Compatible Blobstore

BBR does not download your blobs to the backup artifact when performing a backup. Instead, BBR notes the current version identifier of each blob and stores the identifiers in the artifact.

When restoring, BBR reverts your blobs to the original versions using the version identifiers. Since the backup artifact contains only identifiers, not blobs, BBR can only restore the blobs if the original bucket containing the blob versions still exists.

If the bucket is deleted, all its versions are deleted with it, and you cannot restore using it. To prevent this from happening, set up Cross-Region Replication. See the put-bucket-replication command if you prefer to use the AWS CLI.

Replication results in buckets that are identical to the original ones, including their version identifiers. This allows you to perform restores from the replicas, in case your original buckets are lost. To restore from a replica, see Restore from Replicas below.

Supported Restore Scenarios for Your Versioned S3-Compatible Blobstore

Determine which of the following scenarios applies to your deployment before restoring it:

When restoring to a bucket, BBR will only modify blobs that are recorded in the backup artifact. Any other blobs will not be affected.

In-Place Restore

When backing up an external blobstore, the backup consists of a snapshot of the objects’ IDs and versions. If your destination Cloud Foundry for a restore uses the original buckets of the backed-up Cloud Foundry, that is, you are doing an in-place restore, then, during a restore, those versions are retrieved and set to be the current versions in the buckets.

Restore to New Buckets

If your destination Cloud Foundry for a restore uses different buckets, then you can also restore into those buckets, if the original buckets still exist. During restore, the original versions are copied from the original buckets to the destination buckets.

The destination buckets must be versioned. For how to version an Amazon S3 blobstore, see Enable Versioning on Your S3-Compatible Blobstore above.

Before restoring, ensure that the s3-versioned-blobstore-backup-restorer job is configured to point to the destination buckets.

Restore from Replicas

To protect yourself from losing your blobstore buckets, you should enable replication. This will allow you to perform restores from the replicas, in case your original buckets are lost.

If you need to restore from replicas, you must modify the backup artifacts to point to the replicas before beginning a restore.

To modify the backup artifacts, do the following:

  1. Change into the backup artifact directory.

  2. Run the following command to extract the blobstore backup archive:

    tar xvf BLOBSTORE-ARTIFACT.tar

    For example,

    $ tar xvf backup-restore-0-s3-versioned-blobstore-backup-restorer.tar
  3. Modify the blobstore.json to point to the replicated buckets.

  4. Run the following command to recalculate the shasum of blobstore.json:

    shasum -a 256 blobstore.json

  5. Update the metadata file entry for that blobstore.json with the new checksum.

  6. Run the following command to recreate the archive:

    tar cvf BLOBSTORE-ARTIFACT.tar ./blobstore.json

    For example,

    $ tar cvf backup-restore-0-s3-versioned-blobstore-backup-restorer.tar ./blobstore.json

The backup artifact is now ready to be restored from the replicated buckets.

Azure Blobstores

BBR backs up Azure storage containers by storing the ETags of each blob, not the actual blobs, which makes backups and restores faster. However, this means that restores work only if the original containers still exist.

Enable Soft Delete in Your Azure Storage Account

BBR requires that you enable soft delete in your Azure storage account. With soft delete, you can recover your data when blobs or blob snapshots are deleted. To enable soft delete in your Azure storage account, follow the instructions in the Azure documentation. You should set a reasonable retention policy to minimize storage costs.

Enable Replication for Your Azure Blobstore

BBR does not download your blobs to the backup artifact when it performs a backup. Instead, BBR records the current ETag of each blob and stores the identifiers in the artifact.

When restoring, BBR reverts your blobs to the original versions using the ETags. Because the backup artifact stores identifiers, not blobs, BBR can restore the blobs only if the original container with the blob versions still exists.

An Azure data center failure can render your original container unavailable. To mitigate this threat, configure replication for your container. For more information, see Azure Storage replication in the Azure documentation.

Enable Backup and Restore of Your Azure Blobstore

To back up a Cloud Foundry deployment that uses an Azure external blobstore, you must include the enable-backup-restore-azure.yml ops file in your deployment. This colocates the azure-blobstore-backup-restorer job from backup-and-restore-sdk-release as part of your deployment.

Note: Apply enable-backup-restore.yml and use-azure-storage-blobstore.yml before enable-backup-restore-azure.yml. For more information, see Apply Ops Files in the Correct Order.

If you do not use cf-deployment and ops files, you can still back up and restore external blobstores with BBR. Refer to the contents of the enable-backup-restore-azure.yml ops file as an example and see the backup-and-restore-sdk-release documentation in GitHub for details.

Supported Restore Scenarios for Your Azure Blobstore

Determine which of the following scenarios applies to your deployment before restoring it:

When restoring to a container, BBR modifies only blobs that are recorded in the backup artifact. Other blobs are not affected.

In-Place Restore

When BBR backs up an Azure blobstore, the backup consists of a snapshot with the IDs and ETags of the objects.

In an in-place restore, you reuse the same containers. BBR replaces the current blob versions in the container with those corresponding to the ETags recorded in the backup artifact.

Restore to a New Container in the Same Storage Account

When you restore to different containers, BBR copies the versions recorded in the backup artifact from the backed-up containers to the new containers. In this scenario, the backed-up containers must exist during the restore.

Restore to a New Container in a Different Storage Account

You can restore blobs to a container in a different Azure storage account from the original. In this scenario, BBR copies the blobs from the backed-up containers in the source Azure storage account to the new containers in the storage account used for restore. You must use the enable-restore-azure-clone.yml ops file in your deployment.

Note: Apply enable-backup-restore.yml and use-azure-storage-blobstore.yml before enable-restore-azure-clone.yml. For more information, see Apply Ops Files in the Correct Order.

After restoring, replace the enable-restore-azure-clone.yml ops file in your deployment with the enable-backup-restore-azure.yml ops file and redeploy.

Create a pull request or raise an issue on the source for this page in GitHub