Backup and Restore with External Blobstores

This topic describes configuring an external blobstore so that it can be backed up and restored by BOSH Backup and Restore (BBR).

This topic also describes the different restore scenarios for external blobstores and, if necessary, how to prepare the backup artifact for restore.

BBR supports the backup and restore of blobstores stored in Amazon S3 buckets and in S3-compatible storage solutions.

The necessary configuration and supported restore scenarios differ depending on whether your blobstore is versioned or not - for specific details see the sections below.

S3-Compatible Unversioned Blobstores

External blobstores are backed up by copying blobs from live buckets to specified backup buckets. The unversioned backup-restorer uses the blobstore’s copy functionality to transfer blobs between the buckets to avoid transferring and storing the blobs on your instances.

Restore only works if the backup buckets still exist. For this reason, we recommend that either the backup buckets or copies of them should be in a different region to the live buckets.

Enable Backup and Restore of Your Unversioned S3-Compatible Blobstore

To back up a Cloud Foundry deployment that uses an unversioned S3-compatible external blobstore, you must colocate the s3-unversioned-blobstore-backup-restorer job from backup-and-restore-sdk-release as part of your deployment.

  1. Verify that your blobstore is one of the following:

    • Amazon S3 buckets
    • S3-compatible buckets with support for AWS Signature Version 4.

    Note: If your S3-compatible blobstore uses a custom CA certificate, refer to Configuring Trusted Certificates in the BOSH documentation. BBR automatically makes use of your configured trusted certificates.

  2. Create backup buckets for droplets, packages and buildpacks. We recommend that either the backup buckets or copies of them should be in a different region to the live buckets.

  3. Include the enable-backup-restore-s3-unversioned.yml ops file in your deployment.

    Set the variables specified in vars-enable-backup-restore-s3-unversioned.yml to configure your backup bucket locations.

    Note: Apply enable-backup-restore.yml and use-s3-blobstore.yml before enable-backup-restore-s3-unversioned.yml. See Apply Ops Files in the Correct Order.

    If you do not use cf-deployment and ops files, you can still set up your Cloud Foundry deployment to use BBR. Examine the contents of the ops file, and use it as a guide to customize your deployment manifest directly.

Supported Restore Scenarios For Your Unversioned S3-Compatible Blobstore

The restore process will copy blobs from a directory in the backup bucket to the live buckets (droplets, packages and buildpacks) in use by the CF deployment you are restoring into.

If your original backup buckets are lost but you are restoring from copies, ensure that the variables listed in vars-enable-backup-restore-s3-unversioned.yml point to the copies.

WARNING: Restoring to an unversioned bucket will overwite blobs with the backed up copies of those blobs. BBR will only modify the blobs that were copied to the backup bucket. Any other blobs will not be affected.

S3-Compatible Versioned Blobstores

The versioned backup-restorer only supports S3-compatible buckets that are versioned and support AWS Signature Version 4. For more details about enabling versioning on your blobstore, see enable versioning.

External blobstores are backed up by storing the current version of each blob, not the actual files. Those versions will be set to be the current versions at restore time. This makes backups and restores faster, but also means that restores only work if the original bucket still exists.

If the bucket gets deleted, all its versions are deleted with it and you cannot restore it unless you have a replica. For how to restore from a replica, see Restore from Replicas below.

WARNING: The s3-versioned-blobstore-backup-restorer in backup-and-restore-sdk v1.5.1 and earlier does not support the backup and restore of an AWS S3 blobstore with individual blobs greater than 5GB.

Enable Backup and Restore of Your Versioned S3-Compatible Blobstore

To back up a Cloud Foundry deployment that uses an external blobstore, you must colocate the s3-versioned-blobstore-backup-restorer job from backup-and-restore-sdk-release as part of your deployment.

  1. Verify that your blobstore is one of the following:

    Note: If your S3-compatible blobstore uses a custom CA certificate, refer to Configuring Trusted Certificates in the BOSH documentation. BBR automatically makes use of your configured trusted certificates.

  2. Include the enable-backup-restore-s3-versioned.yml ops file in your deployment.

    Note: Apply enable-backup-restore.yml and use-s3-blobstore.yml before enable-backup-restore-s3-versioned.yml. See Apply Ops Files in the Correct Order.

    If you do not use cf-deployment and ops files, you can still set up your Cloud Foundry deployment to use BBR. Examine the contents of the ops file, and use it as a guide to customize your deployment manifest directly.

Enable Versioning on Your S3-Compatible Blobstore

BBR only supports the backup and restore of blobstores stored in versioned, Amazon S3 buckets and in S3-compatible buckets that are versioned and support AWS Signature Version 4.

To enable versioning of Amazon S3 buckets, do the following:

Follow the instructions in the Amazon S3 documentation, How Do I Enable or Suspend Versioning for an S3 Bucket? to enable versioning. If you prefer to use the aws CLI, you can make use of the put-bucket-versioning command.

  1. If your blobstore buckets are not empty, run the following command on each one of them, using the AWS CLI:

    aws s3 cp s3://BUCKET-NAME/ s3://BUCKET-NAME/ --recursive --metadata bump=true

    For example,

    $ aws s3 cp s3://my-bucket/ s3://my-bucket/ --recursive --metadata bump=true
    

    This ensures that each file in your buckets has a valid version ID.

  2. After enabling versioning, you have a current version and zero or more non-current versions for each object.

    WARNING: We strongly recommend setting an expiration policy on your buckets to delete old non-current versions and minimize storage costs. For more information, see Example 6: Specifying a Lifecycle Rule for a Versioning-Enabled Bucket in the Amazon Simple Storage Service documentation.

Note: If your blobstore uses S3-compatible buckets not from Amazon, see the documentation for that storage provider regarding enabling versioning and setting an expiration policy.

Enable Replication on Your Versioned S3-Compatible Blobstore

BBR does not download your blobs to the backup artifact when performing a backup. Instead, BBR notes the current version identifier of each blob and stores the identifiers in the artifact.

When restoring, BBR reverts your blobs to the original versions using the version identifiers. Since the backup artifact contains only identifiers, not blobs, BBR can only restore the blobs if the original bucket containing the blob versions still exists.

If the bucket is deleted, all its versions are deleted with it, and you cannot restore using it. To prevent this from happening, set up Cross-Region Replication. See the put-bucket-replication command if you prefer to use the AWS CLI.

Replication results in buckets that are identical to the original ones, including their version identifiers. This allows you to perform restores from the replicas, in case your original buckets are lost. To restore from a replica, see Restore from Replicas below.

Supported Restore Scenarios For Your Versioned S3-Compatible Blobstore

There are three distinct restore scenarios:

Make sure that you understand which scenario applies to you before beginning a restore.

When restoring to a bucket, BBR will only modify blobs that are recorded in the backup artifact. Any other blobs will not be affected.

In-Place Restore

When backing up an external blobstore, the backup consists of a snapshot of the objects’ IDs and versions. If your destination Cloud Foundry for a restore uses the original buckets of the backed-up Cloud Foundry, that is, you are doing an in-place restore, then, during a restore, those versions are retrieved and set to be the current versions in the buckets.

Restore to New Buckets

If your destination Cloud Foundry for a restore uses different buckets, then you can also restore into those buckets, if the original buckets still exist. During restore, the original versions are copied from the original buckets to the destination buckets.

The destination buckets must be versioned. For how to version an Amazon S3 blobstore, see Enable Versioning on Your S3-Compatible Blobstore above.

Before restoring, ensure that the s3-versioned-blobstore-backup-restorer job is configured to point to the destination buckets.

Restore from Replicas

To protect yourself from losing your blobstore buckets, you should enable replication. This will allow you to perform restores from the replicas, in case your original buckets are lost.

If you need to restore from replicas, you must modify the backup artifacts to point to the replicas before beginning a restore.

To modify the backup artifacts, do the following:

  1. Change into the backup artifact directory.

  2. Run the following command to extract the blobstore backup archive:

    tar xvf BLOBSTORE-ARTIFACT.tar

    For example,

    $ tar xvf backup-restore-0-s3-versioned-blobstore-backup-restorer.tar
  3. Modify the blobstore.json to point to the replicated buckets.

  4. Run the following command to recalculate the shasum of blobstore.json:

    shasum -a 256 blobstore.json

  5. Update the metadata file entry for that blobstore.json with the new checksum.

  6. Run the following command to recreate the archive:

    tar cvf BLOBSTORE-ARTIFACT.tar blobstore.json

    For example,

    $ tar cvf backup-restore-0-s3-versioned-blobstore-backup-restorer.tar blobstore.json

The backup artifact is now ready to be restored from the replicated buckets.

Create a pull request or raise an issue on the source for this page in GitHub