Backup and Restore with External Blobstores
This topic describes how to configure an external blobstore so that it can be backed up and restored by BOSH Backup and Restore (BBR).
If your external blobstore is hosted on Amazon S3 or a compatible storage solution that supports S3 versioning and AWS Signature Version 4, it can be backed up and restored by BBR.
This topic also decribes the different restore scenarios for external blobstores and, if necessary, how to prepare the backup artifact for restore.
The backup artifact only contains references to versions of blobs, not the actual files. As a consequence, restores only work if the original bucket still exists. If the bucket gets deleted, all its versions are deleted with it and you cannot restore it unless you have a replica. For how to restore from a replica, see Restore from Replicas below.
BBR only supports the backup and restore of blobstores stored in versioned, Amazon S3 buckets and in S3-compatible buckets that are versioned and support AWS Signature Version 4.
To enable versioning of Amazon S3 buckets, do the following:
Follow the instructions in the Amazon S3 documentation, How Do I Enable or Suspend Versioning for an S3 Bucket? to enable versioning.
If your blobstore buckets are not empty, run the following command on each one of them, using the AWS CLI:
aws s3 cp s3://BUCKET-NAME/ s3://BUCKET-NAME/ --recursive --metadata bump=true
$ aws s3 cp s3://my-bucket/ s3://my-bucket/ --recursive --metadata bump=true
This ensures that each file in your buckets has a valid version ID.
Note: If your blobstore uses S3-compatible buckets not from Amazon, see the documentation for that storage provider to enable versioning.
To back up a Cloud Foundry deployment that uses an external blobstore,
you must colocate the
backup-and-restore-sdk-release as part of your deployment.
This procedure describes how to do this if you are using ops files to customize your deployment:
Verify that your blobstore is one of the following:
- Amazon S3 buckets with versioning enabled. See Enable Versioning on Your External Blobstore above.
- S3-compatible buckets with versioning enabled and support for AWS Signature Version 4.
enable-backup-restore-s3.ymlops file in your deployment.
enable-backup-restore-s3.yml. See Apply Ops Files in the Correct Order.
If you do not use cf-deployment and ops files, you can still set up your Cloud Foundry deployment to use BBR. Examine the contents of the ops file, and use it as a guide to customize your deployment manifest directly.
There are three distinct restore scenarios:
Make sure that you understand which scenario applies to you before beginning a restore.
When backing up an external blobstore, the backup consists of a snapshot of the objects’ IDs and versions. If your destination Cloud Foundry for a restore uses the original buckets of the backed-up Cloud Foundry, that is, you are doing an in-place restore), then, during a restore, those versions are retrieved and set to be the current versions in the buckets.
If your destination Cloud Foundry for a restore uses different buckets, then you can also restore into those buckets, if the original buckets still exist. During restore, the original versions are copied from the original buckets to the destination buckets.
The destination buckets must be versioned. For how to version an Amazon S3 blobstore, see Enable Versioning on Your External Blobstore above.
Before restoring, ensure that the
s3-versioned-blobstore-backup-restorer job is configured to point to the
To protect yourself from losing your blobstore buckets, set up Cross-Region Replication. Replication results in buckets that are identical to the original ones, including their version IDs. This allows you to perform restores from the replicas, in case your original buckets are lost.
If you need to restore from replicas, you must modify the backup artifacts to point to the replicas before beginning a restore.
To modify the backup artifacts, do the following:
Change into the backup artifact directory.
Run the following command to extract the blobstore backup archive:
tar xvf BLOBSTORE-ARTIFACT.tar
$ tar xvf backup-restore-0-s3-versioned-blobstore-backup-restorer.tar
blobstore.jsonto point to the replicated buckets.
Run the following command to recalculate the shasum of
shasum -a 256 blobstore.json
Update the metadata file entry for that
blobstore.jsonwith the new checksum.
Run the following command to recreate the archive:
tar cvf BLOBSTORE-ARTIFACT.tar blobstore.json
$ tar cvf backup-restore-0-s3-versioned-blobstore-backup-restorer.tar blobstore.json
The backup artifact is now ready to be restored from the replicated buckets.Create a pull request or raise an issue on the source for this page in GitHub