Backup and Restore with External Blobstores

This topic describes how to configure an external blobstore so that it can be backed up and restored by BOSH Backup and Restore (BBR).

If your external blobstore is hosted on Amazon S3 or a compatible storage solution that supports S3 versioning and AWS Signature Version 4, it can be backed up and restored by BBR.

This topic also decribes the different restore scenarios for external blobstores and, if necessary, how to prepare the backup artifact for restore.

The backup artifact only contains references to versions of blobs, not the actual files. As a consequence, restores only work if the original bucket still exists. If the bucket gets deleted, all its versions are deleted with it and you cannot restore it unless you have a replica. For how to restore from a replica, see Restore from Replicas below.

Enable Versioning on Your External Blobstore

BBR only supports the backup and restore of blobstores stored in versioned, Amazon S3 buckets and in S3-compatible buckets that are versioned and support AWS Signature Version 4.

To enable versioning of Amazon S3 buckets, do the following:

  1. Follow the instructions in the Amazon S3 documentation, How Do I Enable or Suspend Versioning for an S3 Bucket? to enable versioning.

  2. If your blobstore buckets are not empty, run the following command on each one of them, using the AWS CLI:

    aws s3 cp s3://BUCKET-NAME/ s3://BUCKET-NAME/ --recursive --metadata bump=true

    For example,

    $ aws s3 cp s3://my-bucket/ s3://my-bucket/ --recursive --metadata bump=true
    

    This ensures that each file in your buckets has a valid version ID.

Note: If your blobstore uses S3-compatible buckets not from Amazon, see the documentation for that storage provider to enable versioning.

Enable Backup and Restore of External Blobstores

To back up a Cloud Foundry deployment that uses an external blobstore, you must colocate the s3-versioned-blobstore-backup-restorer job from backup-and-restore-sdk-release as part of your deployment.

This procedure describes how to do this if you are using ops files to customize your deployment:

  1. Verify that your blobstore is one of the following:

  2. Include the enable-backup-restore-s3.yml ops file in your deployment.

    Note: Apply enable-backup-restore.yml and use-s3-blobstore.yml before enable-backup-restore-s3.yml. See Apply Ops Files in the Correct Order.

If you do not use cf-deployment and ops files, you can still set up your Cloud Foundry deployment to use BBR. Examine the contents of the ops file, and use it as a guide to customize your deployment manifest directly.

Supported Restore Scenarios

There are three distinct restore scenarios:

Make sure that you understand which scenario applies to you before beginning a restore.

In-Place Restore

When backing up an external blobstore, the backup consists of a snapshot of the objects’ IDs and versions. If your destination Cloud Foundry for a restore uses the original buckets of the backed-up Cloud Foundry, that is, you are doing an in-place restore), then, during a restore, those versions are retrieved and set to be the current versions in the buckets.

Restore to New Buckets

If your destination Cloud Foundry for a restore uses different buckets, then you can also restore into those buckets, if the original buckets still exist. During restore, the original versions are copied from the original buckets to the destination buckets.

The destination buckets must be versioned. For how to version an Amazon S3 blobstore, see Enable Versioning on Your External Blobstore above.

Before restoring, ensure that the s3-versioned-blobstore-backup-restorer job is configured to point to the destination buckets.

Restore from Replicas

To protect yourself from losing your blobstore buckets, set up Cross-Region Replication. Replication results in buckets that are identical to the original ones, including their version IDs. This allows you to perform restores from the replicas, in case your original buckets are lost.

If you need to restore from replicas, you must modify the backup artifacts to point to the replicas before beginning a restore.

To modify the backup artifacts, do the following:

  1. Change into the backup artifact directory.

  2. Run the following command to extract the blobstore backup archive:

    tar xvf BLOBSTORE-ARTIFACT.tar

    For example,

    $ tar xvf backup-restore-0-s3-versioned-blobstore-backup-restorer.tar
  3. Modify the blobstore.json to point to the replicated buckets.

  4. Run the following command to recalculate the shasum of blobstore.json:

    shasum -a 256 blobstore.json

  5. Update the metadata file entry for that blobstore.json with the new checksum.

  6. Run the following command to recreate the archive:

    tar cvf BLOBSTORE-ARTIFACT.tar blobstore.json

    For example,

    $ tar cvf backup-restore-0-s3-versioned-blobstore-backup-restorer.tar blobstore.json

The backup artifact is now ready to be restored from the replicated buckets.

Create a pull request or raise an issue on the source for this page in GitHub