Restoring with BOSH Backup and Restore

This topic describes how to use BOSH Backup and Restore (BBR) to restore a BOSH deployment or BOSH Director.

To back up a BOSH deployment or BOSH Director with BBR, see the Backing up with BOSH Backup and Restore topic.

The steps in this topic allow you to restore a BOSH Director, a BOSH deployment, or both.

Compatibility of Restore

This section describes the restrictions for a backup artifact to be restorable to another environment. This section is for guidance only, and we recommend that operators validate their backups by using the backup artifacts in a restore.

The restrictions for a backup artifact to be restorable are the following:

  • Topology: BBR requires the BOSH topology of a deployment to be the same in the restore environment as it was in the backup environment.
  • Naming of instance groups and jobs: For any deployment that implements the backup and restore scripts, the instance groups and jobs must have the same names.
  • Number of instance groups and jobs: For instance groups and jobs that have backup and restore scripts, there must be the same number of instances.
  • Limited validation: BBR puts the backed up data into the corresponding instance groups and jobs in the restored environment, but cannot validate the restore beyond that. For example, if the MySQL encryption key is different in the restore environment, the BBR restore might succeed although the restored MySQL database is unusable.

Note: A change in VM size or underlying hardware should not affect BBR’s ability to restore data, as long as there is adequate storage space to restore the data.

Step 1: Recreate VMs

Before restoring a BOSH deployment or Director, you must create the VMs that constitute that deployment or Director.

In a disaster recovery scenario, you can re-create the deployment with your BOSH deployment manifest. If you used the --with-manifest flag when running the BBR backup command, your backup artifact includes a copy of your manifest.

Alternatively, if you are restoring a deployment and you have already restored the Director which contains knowledge of this deployment, as long as the BOSH Resurrector is enabled your VMs for the deployment will be brought back by the Director.

Step 2: Transfer Artifacts to Jumpbox

Move your BBR backup artifact from your safe storage location to the jumpbox.

For instance, you could run the following command to SCP the backup artifact to your jumpbox:

scp LOCAL-PATH-TO-BACKUP-ARTIFACT JUMPBOX-USER/JUMPBOX-ADDRESS

If this artifact is encrypted, you must decrypt it.

Step 3: Restore

Note: The BBR restore command can take a long time to complete. You can run it independently of the SSH session so that the process can continue running even if your connection to the jumpbox fails. The command above uses nohup, but you run the command in a screen or tmux session instead.

Use the optional --debug flag to enable debug logs. See the BBR Logging topic for more information.

Restore a BOSH Director

Perform the following steps to restore a BOSH Director:

  1. Ensure the BOSH Director backup artifact is in the folder you will run BBR from.
  2. Run the BBR restore command from your jumpbox to restore the BOSH Director:

    nohup bbr director \  
    --private-key-path PATH-TO-PRIVATE-KEY \
    --username USER-NAME \
    --host HOST \
    restore \
    --artifact-path PATH-TO-DIRECTOR-BACKUP
    

    Replace the placeholder values as follows:

    • PATH-TO-PRIVATE-KEY: This is the path to the SSH private key used to connect to the BOSH Director.
    • USER-NAME: This is the SSH username of the BOSH Director.
    • HOST: This is the address of the BOSH Director with an optional port that defaults to 22. If the BOSH Director is public, this is a URL, such as https://my-bosh.xxx.cf-app.com. Otherwise, this is the BOSH Director IP address.
    • PATH-TO-DIRECTOR-BACKUP: This is the path to the BOSH Director backup you want to restore.

If the command fails, follow the steps in Recover from a Failing Command.

Restore a BOSH Deployment

Perform the following steps to restore a BOSH deployment:

  1. Ensure the BOSH deployment backup artifact is in the folder you will run BBR from.
  2. Run the BBR restore:

    BOSH_CLIENT_SECRET=BOSH-CLIENT-SECRET \
    nohup bbr deployment \
    --target BOSH-TARGET \
    --username BOSH-CLIENT \
    --deployment DEPLOYMENT-NAME \
    --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE \
    restore \
    --artifact-path PATH-TO-DEPLOYMENT-BACKUP
    

    Replace the placeholder values as follows:

    • BOSH-CLIENT, BOSH-CLIENT-SECRET: If you have a BOSH Director with User Account and Authentication (UAA) as the authentication provider, use a UAA client as the username and a UAA client secret as the password. If you have a BOSH Director with basic auth configured, use your username and password.
    • BOSH-TARGET: This is the FQDN or IP address of your BOSH Director.
    • DEPLOYMENT-NAME: This is the name of the deployment you want to restore.
    • PATH-TO-BOSH-CA-CERTIFICATE: This is the path to the BOSH Director’s Certificate Authority (CA) certificate if the certificate is not verifiable by the local machine’s certificate chain.
    • PATH-TO-DEPLOYMENT-BACKUP: This is the path to the BOSH deployment backup you want to restore.

If the command fails, follow the steps in Recover from a Failing Command.

Recover from a Failing Command

  1. Ensure all the parameters in the command are set.
  2. Ensure the BOSH Director credentials are valid.
  3. Ensure the specified BOSH deployment exists.
  4. Ensure that the jumpbox can reach the BOSH Director.
  5. Ensure the source BOSH deployment is compatible with the target BOSH deployment.
  6. If you see the error message Directory /var/vcap/store/bbr-backup already exists on instance, run the relevant cleanup command. See Clean up After Failed Restore.

Cancel a Restore

If you need to cancel a restore, perform the following steps:

  1. Terminate the BBR process by pressing Ctrl-C and typing yes to confirm.
  2. Stopping a restore can leave the system in an unusable state and prevent future restores. Perform the procedures in the Clean up After Failed Restore section to enable future restores.

Clean up After Failed Restore

If your restore process fails, then the process may leave the BBR restore folder on the instance. As a result, any subsequent restore attempts may also fail. In addition, BBR may not have run the post-restore scripts, which can leave the instance in a locked state.

In order to resolve these issues, run the BBR cleanup script.

To clean up after a failed BOSH Director restore, run the following command:

bbr director \
--private-key-path PATH-TO-PRIVATE-KEY \
--username USER-NAME \
--host HOST
restore-cleanup

To clean up after a failed BOSH deployment restore, run the following command:

BOSH_CLIENT_SECRET=BOSH-CLIENT-SECRET \
bbr deployment \
--target BOSH-TARGET \
--username BOSH-CLIENT \
--deployment DEPLOYMENT-NAME \
--ca-cert PATH-TO-BOSH-CA-CERTIFICATE \
restore-cleanup
Create a pull request or raise an issue on the source for this page in GitHub