Restoring with BOSH Backup and Restore

This topic describes how to use BOSH Backup and Restore (BBR) to restore a BOSH deployment or BOSH Director.

To back up a BOSH deployment or BOSH Director with BBR, see the Backing Up with BOSH Backup and Restore topic.

The steps in this topic allow you to restore a BOSH Director, a BOSH deployment, or both.

Compatibility of Restore

This section describes the restrictions for a backup artifact to be restorable to another environment. This section is for guidance only, and Pivotal highly recommends that operators validate their backups by using the backup artifacts in a restore.

The restrictions for a backup artifact to be restorable are the following:

  • Topology: BBR requires the BOSH topology of a deployment to be the same in the restore environment as it was in the backup environment.
  • Naming of instance groups and jobs: For any deployment that implements the backup and restore scripts, the instance groups and jobs must have the same names.
  • Number of instance groups and jobs: For instance groups and jobs that have backup and restore scripts, there must be the same number of instances.
  • Limited validation: BBR puts the backed up data into the corresponding instance groups and jobs in the restored environment, but can’t validate the restore beyond that. For example, if the MySQL encryption key is different in the restore environment, the BBR restore might succeed although the restored MySQL database is unusable.

Note: A change in VM size or underlying hardware should not affect BBR’s ability to restore data, as long as there is adequate storage space to restore the data.

Step 1: Recreate VMs

Before restoring a BOSH Deployment or director, you must create the VMs that constitute the deployment.

In a disaster recovery scenario, re-create the deployment with your BOSH deployment manifest. If you used the --with-manifest flag when running the BBR backup command, your backup artifact includes a copy of your manifest.

Step 2: Transfer Artifacts to Jumpbox

In the Backing Up with BOSH Backup and Restore topic, you moved the TAR and metadata files of the backup artifact off your jumpbox to your preferred storage space. Now you must transfer those files back to your jumpbox.

For instance, you could SCP the backup artifact to your jumpbox:

$ scp LOCAL_PATH_TO_BACKUP_ARTIFACT JUMPBOX_USER/JUMPBOX_ADDRESS

Step 3: Restore

Restore a BOSH Director

Perform the following steps to restore a BOSH Director:

  1. Ensure the BOSH Director backup artifact is in the folder you will run BBR from.
  2. Run the BBR restore command from your jumpbox to restore the BOSH Director:

    $ nohup bbr director \
      --private-key-path PATH_TO_PRIVATE_KEY \
      --username USER_NAME \
      --host HOST \
      restore \
        --artifact-path PATH_TO_DIRECTOR_BACKUP
    

    Use the optional --debug flag to enable debug logs. See the Exit Codes and Logging topic for more information.

    Replace the placeholder values as follows:

    • PATH_TO_PRIVATE_KEY: This is the path to the SSH private key used to connect to the BOSH Director.
    • USER_NAME: This is the SSH username of the BOSH Director.
    • HOST: This is the address of the BOSH Director with an optional port that defaults to 22. If the BOSH Director is public, this is a URL, such as https://my-bosh.xxx.cf-app.com. Otherwise, this is the BOSH Director IP address.
    • PATH_TO_DIRECTOR_BACKUP: This is the path to the BOSH Director backup you want to restore.

Note: The BBR restore command can take a long time to complete. You can run it independently of the SSH session so that the process can continue running even if your connection to the jumpbox fails. The command above uses nohup, but you could also run the command in a screen or tmux session.

If the command fails, do the following:

  1. Ensure all the parameters in the command are set.
  2. Ensure the BOSH Director credentials are valid.
  3. Ensure the specified BOSH Director exists.
  4. Ensure that the jumpbox can reach the BOSH Director.
  5. Ensure the source BOSH Director is compatible with the target BOSH Director.
  6. Consult the Exit Codes and Logging topic.

Restore a BOSH Deployment

Perform the following steps to restore a BOSH deployment:

  1. Ensure the BOSH deployment backup artifact is in the folder you will run BBR from.
  2. Run the BBR restore:

    $ BOSH_CLIENT_SECRET=BOSH_CLIENT_SECRET \
      nohup bbr deployment \
      --target BOSH_TARGET \
      --username BOSH_CLIENT \
      --deployment DEPLOYMENT_NAME \
      --ca-cert PATH_TO_BOSH_SERVER_CERT \
      restore \
      --artifact-path PATH_TO_DEPLOYMENT_BACKUP
    

    Use the optional --debug flag to enable debug logs. See the Exit Codes and Logging topic for more information.

    Replace the placeholder values as follows:

    • BOSH_CLIENT, BOSH_CLIENT_SECRET: If you have a BOSH Director with User Account and Authentication (UAA) as the authentication provider, use a UAA client as the username and a UAA client secret as the password. If you have a BOSH Director with basic auth configured, use your username and password.
    • BOSH_TARGET: This is the FQDN or IP address of your BOSH Director.
    • DEPLOYMENT-NAME: This is the name of the deployment you want to restore.
    • PATH_TO_BOSH_SERVER_CERT: This is the path to the BOSH Director’s Certificate Authority (CA) certificate if the certificate is not verifiable by the local machine’s certificate chain.
    • PATH_TO_DEPLOYMENT_BACKUP: This is the path to the BOSH deployment backup you want to restore.

      Note: The BBR restore command can take a long time to complete. You can run it independently of the SSH session so that the process can continue running even if your connection to the jumpbox fails. The command above uses nohup, but you could also run the command in a screen or tmux session.

If the command fails, do the following:

  1. Ensure all the parameters in the command are set.
  2. Ensure the BOSH Director credentials are valid.
  3. Ensure the specified BOSH deployment exists.
  4. Ensure that the jumpbox can reach the BOSH Director.
  5. Ensure the source BOSH deployment is compatible with the target BOSH deployment.
  6. Consult the Exit Codes and Logging topic.

Cancel a Restore

If you need to cancel a restore, perform the following steps:

  1. Terminate the BBR process by pressing Ctrl-C and typing yes to confirm.
  2. Stopping a restore can leave the system in an unusable state and prevent future restores. Perform the procedures in the Clean Up After Failed Restore section to enable future restores.

Clean Up After Failed Restore

If your restore process fails, then the process may leave the BBR restore folder on the instance. As a result, any subsequent restore attempts may also fail. In addition, BBR may not have run the post-restore scripts, which can leave the instance in a locked state.

In order to resolve these issues, run the BBR cleanup script.

To clean up after a failed BOSH Director restore, run the following command:

$ bbr director \
    --private-key-path PATH_TO_PRIVATE_KEY \
    --username USER_NAME \
    --host HOST
    restore-cleanup

To clean up after a failed BOSH deployment restore, run the following command:

$ BOSH_CLIENT_SECRET=BOSH_CLIENT_SECRET \
    bbr deployment \
    --target BOSH_TARGET \
    --username BOSH_CLIENT \
    --deployment DEPLOYMENT_NAME \
    --ca-cert PATH_TO_BOSH_SERVER_CERT \
    restore-cleanup

If the cleanup script fails, consult the following table to match the exit codes to an error message.

Value Error
0 Success
1 General failure
8 The post-restore unlock failed. Your BOSH deployment or BOSH Director may be in a bad state and require attention.
16 The cleanup failed. This is a non-fatal error indicating that the utility has been unable to clean up open BOSH SSH connections to the deployment VMs. Manual cleanup may be required to clear any hanging BOSH users and connections.

For more information about how to interpret the exit code, see the Exit Codes section of the Exit Codes and Logging topic.

Create a pull request or raise an issue on the source for this page in GitHub