Backing Up with BOSH Backup and Restore

This topic describes how to use BOSH Backup and Restore (BBR) to back up a BOSH deployment or BOSH Director.

To restore a BOSH deployment or BOSH Director with BBR, see the Restoring with BOSH Backup and Restore topic.

Perform the steps below to back up a BOSH Director, a BOSH deployment, or both.

Prerequisites

To enable your BOSH deployment to be backed up and restored, the deployment must adhere to the BBR contract.

For your backup artifact to be restorable to another environment, you must meet the following conditions:

  • The BOSH topology of a deployment is the same in the restore environment as it was in the backup environment.
  • For any deployment that implements the backup and restore scripts, the instance groups and jobs have the same names.
  • For instance groups or jobs that have backup and restore scripts, they must have the same number of instances.

A change in VM size or underlying hardware should not affect BBR’s ability to restore data, as long as there is adequate storage space to restore the data.

BBR puts the backed up data into the corresponding instance groups and jobs in the restored environment, but cannot validate the restore. For example, if a MySQL encryption key is different in the restore environment, the BBR restore could succeed while the restored MySQL release could be unusable. As a result, operators should validate their backups by using the backup artifacts in a restore.

Back Up a BOSH Director

Step 1: Check Your BOSH Director

Perform the following steps to check that your BOSH Director is reachable and can be backed up:

  1. SSH into your jumpbox:
    $ ssh JUMPBOX_USER/JUMPBOX_ADDRESS -i YOUR_CERTIFICATE.pem
    
  2. Run the BBR pre-backup check:

    $ bbr director \
      --private-key-path PATH_TO_PRIVATE_KEY \
      --username USER_NAME \
      --host HOST \
      pre-backup-check
    

    Use the optional --debug flag to enable debug logs. See the Exit Codes and Logging topic for more information.

    Replace the placeholder values as follows:

    • PATH_TO_PRIVATE_KEY: This is the path to the SSH private key used to connect to the BOSH Director.
    • USER_NAME: This is the SSH username of the BOSH Director.
    • HOST: This is the address of the BOSH Director, with an optional port that defaults to 22. If the BOSH Director is public, this is a URL, such as https://my-bosh.xxx.cf-app.com. Otherwise, this is the BOSH Director IP address.
  3. If the pre-backup check succeeds, continue to the next section. If it fails, the command will print the reason why it cannot back up the deployment with the error code 1. The BOSH Director you selected may not have the correct backup scripts, or the connection to the BOSH Director may have failed.

Step 2: Back Up Your BOSH Director

Run the BBR backup command from your jumpbox to back up your BOSH Director:

$ bbr director \
  --private-key-path PATH_TO_PRIVATE_KEY \
  --username USER_NAME \
  --host HOST \
  backup

Use the optional --debug flag to enable debug logs. See the Exit Codes and Logging topic for more information.

Replace the placeholder values with the same values as above.

If the commands completes successfully, do the following:

  1. Move the backup artifact off the jumpbox to your preferred storage space. BBR stores each backup in the current working directory, in a directory named HOST-TIMESTAMP. The backup created by BBR consists of a folder with the backup artifacts and metadata files.
  2. Compress and encrypt the backup artifacts when storing them.
  3. Make redundant copies of your backup and store them in multiple locations in order to minimize the risk of losing your backups in the event of a disaster.
  4. Attempt a test restore on every backup in order to validate it by performing the procedures in the Restoring with BOSH Backup and Restore topic.

If the command fails, do the following:

  1. If the backup artifact is corrupted, rerun the backup and discard the failing artifacts.
  2. Ensure all the parameters in the command are set.
  3. Ensure the BOSH Director credentials are valid.
  4. Consult the Exit Codes and Logging topic.

Back Up a BOSH Deployment

Step 1: Check Your BOSH Deployment

Perform the following steps to check that your BOSH Director is reachable and has a deployment that can be backed up:

  1. SSH into your jumpbox:
    $ ssh JUMPBOX_USER/JUMPBOX_ADDRESS -i YOUR_CERTIFICATE.pem
    
  2. Run the BBR pre-backup check:

    $ BOSH_CLIENT_SECRET=BOSH_CLIENT_SECRET \
      bbr deployment \
      --target BOSH_TARGET \
      --username BOSH_CLIENT \
      --deployment DEPLOYMENT_NAME \
      --ca-cert PATH_TO_BOSH_SERVER_CERT \
      pre-backup-check
    

    Use the optional --debug flag to enable debug logs. See the Exit Codes and Logging topic for more information.

    Replace the placeholder values as follows:

    • BOSH_CLIENT, BOSH_CLIENT_SECRET: If you have a BOSH Director with User Account and Authentication (UAA) as the authentication provider, use a UAA client as the user name and a UAA client secret as the password. If you have a BOSH Director with basic auth configured, use your user name and password.
    • BOSH_TARGET: This is the FQDN or IP address of your BOSH Director.
    • DEPLOYMENT-NAME: This is the name of the deployment you want to back up. For a list of deployments, run bosh deployments.
    • PATH_TO_BOSH_SERVER_CERT: This is the path to the BOSH Director’s Certificate Authority (CA) certificate, if the certificate is not verifiable by the local machine’s certificate chain.
  3. If the pre-backup check succeeds, continue to the next section. If it fails, the command will print the reason why it cannot back up the deployment with the error code 1. The deployment you selected may not have the correct backup scripts, or the connection to the BOSH Director may have failed.

Step 2: Back Up Your BOSH Deployment

Run the BBR backup command from your jumpbox to back up your BOSH deployment:

$ BOSH_CLIENT_SECRET=BOSH_CLIENT_SECRET \
  nohup bbr deployment \
  --target BOSH_DIRECTOR_IP \
  --username BOSH_CLIENT \
  --deployment DEPLOYMENT_NAME \
  --ca-cert PATH_TO_BOSH_SERVER_CERT \
  backup

Use the optional --debug flag to enable debug logs. See the Exit Codes and Logging topic for more information.

Use the optional --with-manifest to include the deployment manifest with the backup artifact.

Replace the placeholder values with the same values as above.

Note: The BBR backup command can take a long time to complete. You can run it independently of the SSH session, so that the process can continue running even if your connection to the jumpbox fails. The command above uses nohup but you could also run the command in a screen or tmux session.

If the commands completes successfully, do the following:

  1. Move the backup artifact off the jumpbox to your preferred storage space. BBR stores each backup in the current working directory, in a directory named DEPLOYMENT-TIMESTAMP. The backup created by BBR consists of a folder with the backup artifacts and metadata files.
  2. Compress and encrypt the backup artifacts when storing them.
  3. Make redundant copies of your backup and store them in multiple locations in order to minimize the risk of losing your backups in the event of a disaster.
  4. Attempt a test restore on every backup in order to validate it by performing the procedures in the Restoring with BOSH Backup and Restore topic.

If the command fails, do the following:

  1. If the backup artifact is corrupted, rerun the backup and discard the failing artifacts.
  2. Ensure all the parameters in the command are set.
  3. Ensure the BOSH Director credentials are valid.
  4. Ensure the specified deployment exists.
  5. Consult the Exit Codes and Logging topic.

Cancel a Backup

If you need to cancel a backup, perform the following steps:

  1. Terminate the BBR process by pressing Ctrl-C and then typing yes to confirm.
  2. Stopping a backup can leave the system in an unusable state or prevent additional backups. Perform the procedures in the Clean Up After Failed Backup section to enable future backups.

Clean Up After Failed Backup

If your backup process fails, it may leave the BBR backup folder on the instance, causing any subsequent attempts to backup to fail. In addition, BBR may not have run the post-backup scripts, leaving the instance in a locked state.

In order to resolve these issues, run the BBR cleanup script.

To clean up after a failed BOSH Director backup, run the following command:

$ bbr director \
    --private-key-path PATH_TO_PRIVATE_KEY \
    --username USER_NAME \
    --host HOST
    backup-cleanup

To clean up after a failed BOSH deployment backup, run the following command:

$ BOSH\_CLIENT\_SECRET=BOSH\_CLIENT\_SECRET \
    bbr deployment \
    --target BOSH\_TARGET \
    --username BOSH\_CLIENT \
    --deployment DEPLOYMENT\_NAME \
    --ca-cert PATH\_TO\_BOSH\_SERVER\_CERT \
    backup-cleanup

If the cleanup script fails, consult the following table to match the exit codes to an error message.

Value Error
0 Success
1 General failure
8 The post-backup unlock failed. Your BOSH deployment or BOSH Director may be in a bad state and require attention.
16 The cleanup failed. This is a non-fatal error indicating that the utility has been unable to clean up open BOSH SSH connections to the deployment VMs. Manual cleanup may be required to clear any hanging BOSH users and connections.

For more information about how to interpret the exit code, see the Exit Codes section of the Exit Codes and Logging topic.

Create a pull request or raise an issue on the source for this page in GitHub