Restoring with BOSH Backup and Restore
This topic describes how to use BOSH Backup and Restore (BBR) to restore a BOSH deployment or BOSH Director.
To back up a BOSH deployment or BOSH Director with BBR, see the Backing Up with BOSH Backup and Restore topic.
The steps in this topic allow you to restore a BOSH Director, a BOSH deployment, or both.
This section describes the restrictions for a backup artifact to be restorable to another environment. This section is for guidance only, and Pivotal highly recommends that operators validate their backups by using the backup artifacts in a restore.
The restrictions for a backup artifact to be restorable are the following:
- Topology: BBR requires the BOSH topology of a deployment to be the same in the restore environment as it was in the backup environment.
- Naming of instance groups and jobs: For any deployment that implements the backup and restore scripts, the instance groups and jobs must have the same names.
- Number of instance groups and jobs: For instance groups and jobs that have backup and restore scripts, there must be the same number of instances.
- Limited validation: BBR puts the backed up data into the corresponding instance groups and jobs in the restored environment, but cannot validate the restore beyond that. For example, if the MySQL encryption key is different in the restore environment, the BBR restore might succeed although the restored MySQL database is unusable.
Note: A change in VM size or underlying hardware should not affect BBR’s ability to restore data, as long as there is adequate storage space to restore the data.
Before restoring a BOSH Deployment or director, you must create the VMs that constitute that deployment/director.
In a disaster recovery scenario, you can re-create the deployment with your BOSH deployment manifest.
If you used the
--with-manifest flag when running the BBR backup command, your backup artifact includes a copy of your manifest.
Alternatively, if you are restoring a deployment and you have already restored the director which contains knowledge of this deployment, then so long as the BOSH Resurrector is on, your VMs for the deployment will be brought back by the director.
Move your BBR backup artifact from your safe storage location to the jumpbox.
For instance, you could SCP the backup artifact to your jumpbox:
$ scp LOCAL_PATH_TO_BACKUP_ARTIFACT JUMPBOX_USER/JUMPBOX_ADDRESS
If it is encrypted, decrypt it.
Note: The BBR restore command can take a long time to complete. You can
run it independently of the SSH session so that the process can continue running even if your
connection to the jumpbox fails. The command above uses
nohup, but you could also run
the command in a
Use the optional
--debug flag to enable debug logs.
See the Exit Codes and Logging topic for more information.
Perform the following steps to restore a BOSH Director:
- Ensure the BOSH Director backup artifact is in the folder you will run BBR from.
- Run the BBR restore command from your jumpbox to restore the BOSH Director:
$ nohup bbr director \ --private-key-path PATH_TO_PRIVATE_KEY \ --username USER_NAME \ --host HOST \ restore \ --artifact-path PATH_TO_DIRECTOR_BACKUPReplace the placeholder values as follows:
PATH_TO_PRIVATE_KEY: This is the path to the SSH private key used to connect to the BOSH Director.
USER_NAME: This is the SSH username of the BOSH Director.
HOST: This is the address of the BOSH Director with an optional port that defaults to 22. If the BOSH Director is public, this is a URL, such as
https://my-bosh.xxx.cf-app.com. Otherwise, this is the BOSH Director IP address.
PATH_TO_DIRECTOR_BACKUP: This is the path to the BOSH Director backup you want to restore.
If the command fails, try these steps
Perform the following steps to restore a BOSH deployment:
- Ensure the BOSH deployment backup artifact is in the folder you will run BBR from.
- Run the BBR restore:
$ BOSH_CLIENT_SECRET=BOSH_CLIENT_SECRET \ nohup bbr deployment \ --target BOSH_TARGET \ --username BOSH_CLIENT \ --deployment DEPLOYMENT_NAME \ --ca-cert PATH_TO_BOSH_SERVER_CERT \ restore \ --artifact-path PATH_TO_DEPLOYMENT_BACKUPReplace the placeholder values as follows:
BOSH_CLIENT_SECRET: If you have a BOSH Director with User Account and Authentication (UAA) as the authentication provider, use a UAA client as the username and a UAA client secret as the password. If you have a BOSH Director with basic auth configured, use your username and password.
BOSH_TARGET: This is the FQDN or IP address of your BOSH Director.
DEPLOYMENT-NAME: This is the name of the deployment you want to restore.
PATH_TO_BOSH_CA_CERT: This is the path to the BOSH Director’s Certificate Authority (CA) certificate if the certificate is not verifiable by the local machine’s certificate chain.
PATH_TO_DEPLOYMENT_BACKUP: This is the path to the BOSH deployment backup you want to restore.
If the command fails, try these steps
- Ensure all the parameters in the command are set.
- Ensure the BOSH Director credentials are valid.
- Ensure the specified BOSH deployment exists.
- Ensure that the jumpbox can reach the BOSH Director.
- Ensure the source BOSH deployment is compatible with the target BOSH deployment.
- If you see the error message
Directory /var/vcap/store/bbr-backup already exists on instance, run the relevant cleanup command
- Consult the Exit Codes and Logging topic.
If you need to cancel a restore, perform the following steps:
- Terminate the BBR process by pressing Ctrl-C and typing
- Stopping a restore can leave the system in an unusable state and prevent future restores. Perform the procedures in the Clean Up After Failed Restore section to enable future restores.
If your restore process fails, then the process may leave the BBR restore folder on the instance. As a result, any subsequent restore attempts may also fail. In addition, BBR may not have run the post-restore scripts, which can leave the instance in a locked state.
In order to resolve these issues, run the BBR cleanup script.
To clean up after a failed BOSH Director restore, run the following command:
$ bbr director \ --private-key-path PATH_TO_PRIVATE_KEY \ --username USER_NAME \ --host HOST restore-cleanup
To clean up after a failed BOSH deployment restore, run the following command:
$ BOSH_CLIENT_SECRET=BOSH_CLIENT_SECRET \ bbr deployment \ --target BOSH_TARGET \ --username BOSH_CLIENT \ --deployment DEPLOYMENT_NAME \ --ca-cert PATH_TO_BOSH_CA_CERT \ restore-cleanup
If the cleanup script fails, consult the following table to match the exit codes to an error message.
|8||The post-restore unlock failed. Your BOSH deployment or BOSH Director may be in a bad state and require attention.|
|16||The cleanup failed. This is a non-fatal error indicating that the utility has been unable to clean up open BOSH SSH connections to the deployment VMs. Manual cleanup may be required to clear any hanging BOSH users and connections.|