Backing up with BOSH Backup and Restore

This topic describes how to use BOSH Backup and Restore (BBR) to back up a BOSH deployment or BOSH Director. Perform the steps below to back up a BOSH Director, a BOSH deployment, or both.

To perform a restore, see Restoring with BOSH Backup and Restore.

Prerequisites

If you want to use the result of the backup to restore to a destination environment, verify that the current environment and the destination environment are compatible. For information, see Compatibility of Restore.

Connect to Your Jumpbox

You can establish a connection to your jumpbox in one of the following ways.

For general information about the jumpbox, see Installing BOSH Backup and Restore.

Connect with SSH

SSH into your jumpbox. If you connect to your jumpbox with SSH, you must run the BBR commands in the following sections from within your jumpbox.

Connect with BOSH_ALL_PROXY

Set and use BOSH_ALL_PROXY. Using BOSH_ALL_PROXY opens an SSH tunnel with SOCKS5 to the jumpbox. This tunnel allows you to forward requests to the BOSH Director through the jumpbox from your local machine.

Use one of the following methods to create the tunnel:

  • Tunnel created by BOSH CLI: To provide the BOSH CLI with the SSH credentials it needs to create the tunnel, run the following command:

     export BOSH_ALL_PROXY=ssh+socks5://jumpbox@jumpbox-ip:12345?private_key=jumpbox.key
    
  • Tunnel established separately:

  1. To establish the tunnel and make it available on a local port, run the following command:

    ssh -4 -D 12345 -fNC jumpbox@jumpbox-ip -i jumpbox.key
    
  2. To provide the BOSH CLI with access to the tunnel through use of the BOSH_ALL_PROXY environment variable, run the following command:

    export BOSH_ALL_PROXY=socks5://localhost:12345
    

Note: Using BOSH_ALL_PROXY can result in longer backup and restore times due to network performance degradation. Because all operations must pass through the proxy, moving backup artifacts can be significantly slower.

Back up a BOSH Director

  1. Run the BBR pre-backup check to confirm that your BOSH Director is reachable and can be backed up with BBR:

    bbr director \
    --private-key-path PATH-TO-PRIVATE-KEY \
    --username USER-NAME \
    --host HOST \
    pre-backup-check
    

    Where:

    • PATH-TO-PRIVATE-KEY is the path to the SSH private key used to connect to the BOSH Director.
    • USER-NAME is the SSH username of the BOSH Director.
    • HOST is the address of the BOSH Director with an optional port. This port defaults to 22. If the BOSH Director is public, this is a URL, such as https://my-bosh.xxx.cf-app.com. Otherwise, this is the BOSH Director IP address.

    For example:

    $ bbr director \
    --private-key-path bosh.pem \
    --username vcap \
    --host bosh.example.com \
    pre-backup-check
    
  2. If the pre-check command fails, do the following:

    • Run the command again adding the --debug flag to enable debug logs. For more information, see BBR Logging.
    • Make the fix suggested in the output and run the pre-backup check again.
      For example, the command fails if the BOSH Director selected did not have the correct backup scripts or if the connection to the BOSH Director failed.
  3. Run the BBR backup command to back up your BOSH Director:

    bbr director \
    --private-key-path PATH-TO-PRIVATE-KEY \
    --username USER-NAME \
    --host HOST \
    backup
    

    Where:

    • PATH-TO-PRIVATE-KEY is the path to the SSH private key used to connect to the BOSH Director.
    • USER-NAME is the SSH username of the BOSH Director.
    • HOST is the address of the BOSH Director with an optional port. This port defaults to 22. If the BOSH Director is public, this is a URL, such as https://my-bosh.xxx.cf-app.com. Otherwise, this is the BOSH Director IP address.

    For example:

    $ bbr director \
    --private-key-path bosh.pem \
    --username vcap \
    --host bosh.example.com \
    backup
    
  4. If the backup command completes successfully, follow the steps in Manage Your Backup Artifact below.

  5. If the backup command fails, do the following:

Back up a BOSH Deployment

  1. Run the BBR pre-backup check to confirm that your BOSH Director is reachable and has a deployment that you can back up with BBR:

    BOSH_CLIENT_SECRET=BOSH-CLIENT-SECRET \
    bbr deployment \
    --target BOSH-TARGET \
    --username BOSH-CLIENT \
    --deployment DEPLOYMENT-NAME \
    --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE \
    pre-backup-check
    

    Replace the placeholder text as follows:

    • BOSH-CLIENT-SECRET: If you have a BOSH Director with User Account and Authentication (UAA) as the authentication provider, this is a UAA client secret as the password. If you have a BOSH Director with basic auth configured, this is your password.
    • BOSH-TARGET: The FQDN or IP address of your BOSH Director.
    • BOSH-CLIENT: If you have a BOSH Director with User Account and Authentication (UAA) as the authentication provider, this is a UAA client as the username. If you have a BOSH Director with basic auth configured, this is your username.
    • DEPLOYMENT-NAME: The name of the deployment you want to back up. For a list of deployments, run bosh deployments.
    • PATH-TO-BOSH-CA-CERTIFICATE: The path to the BOSH Director’s Certificate Authority (CA) certificate, if the certificate is not verifiable by the local machine’s certificate chain.
      • If you used bbl, spin up your director to retrieve the CA certificate with bbl director-ca-cert.
      • If you manually deployed, then the certificate may be stored in a secrets.yml or similar file.

    For example:

    $ BOSH_CLIENT_SECRET=p455w0rd \
    bbr deployment \
    --target bosh.example.com \
    --username admin \
    --deployment cf-acceptance-0 \
    --ca-cert bosh.ca.cert \
    pre-backup-check
    
  2. If the pre-backup check command fails, do the following:

    • Run the command again adding the --debug flag to enable debug logs.
      For more information, see BBR Logging.
    • Make the fix suggested in the output and run the pre-backup check again.
      For example, the deployment you selected might not have the correct backup scripts, or the connection to the BOSH Director failed.
  3. If the pre-backup check succeeds, run the BBR backup command to back up your BOSH deployment:

    BOSH_CLIENT_SECRET=BOSH-CLIENT-SECRET \
    nohup bbr deployment \
    --target BOSH-TARGET \
    --username BOSH-CLIENT \
    --deployment DEPLOYMENT-NAME \
    --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE \
    backup
    

    Replace the placeholder text as follows. These are the same values as used in the with the previous command.

    • BOSH-CLIENT-SECRET: If you have a BOSH Director with User Account and Authentication (UAA) as the authentication provider, this is a UAA client secret as the password. If you have a BOSH Director with basic auth configured, this is your password.
    • BOSH-TARGET: The FQDN or IP address of your BOSH Director.
    • BOSH-CLIENT: If you have a BOSH Director with User Account and Authentication (UAA) as the authentication provider, this is a UAA client as the username. If you have a BOSH Director with basic auth configured, this is your username.
    • DEPLOYMENT-NAME: The name of the deployment you want to back up. For a list of deployments, run bosh deployments.
    • PATH-TO-BOSH-CA-CERTIFICATE: The path to the BOSH Director’s Certificate Authority (CA) certificate, if the certificate is not verifiable by the local machine’s certificate chain.
      • If you used bbl, spin up your Director to retrieve the CA certificate with bbl director-ca-cert.
      • If you manually deployed, then the certificate may be stored in a secrets.yml or similar file.

        Note: If you want to include the manifest in the backup artifact, add the --with-manifest flag. However, be aware that the backup artifact then includes credentials that you need to keep secret.

    For example:

    $ BOSH_CLIENT_SECRET=p455w0rd \
    nohup bbr deployment \
    --target bosh.example.com \
    --username admin \
    --deployment cf-acceptance-0 \
    --ca-cert bosh.ca.cert \
    backup

    Note: The BBR backup command can take a long time to complete. You can run it independently of the SSH session so that the process can continue running even if your connection to the jumpbox fails. The command above uses nohup, but you can run the command in a screen or tmux session instead.

  4. If the command completes successfully, follow the steps in What To Do with Your Backup Artifact below.

  5. If the backup command fails, do the following:

Recover from a Failing Command

If the backup fails for the BOSH Director or deployment, follow these steps:

  1. Ensure that you set all the parameters in the command.
  2. Ensure the BOSH Director credentials are valid.
  3. If you are backing up a deployment, ensure the deployment you specify in the BBR command exists.
  4. Ensure that the jumpbox can reach the BOSH Director.
  5. See BBR Logging.
  6. If you see the error message Directory /var/vcap/store/bbr-backup already exists on instance, run the appropriate cleanup command. See Clean up after a Failed Backup below.
  7. If the backup artifact is corrupted, discard the failing artifacts and rerun the backup.

Cancel a Backup

Backups can take a long time. If you need to cancel a backup, for example if you realize that the backup is going to fail or that your developers need to push an app in a hurry, follow these steps:

  1. Terminate the BBR process by pressing Ctrl-C and typing yes to confirm.
  2. Because stopping a backup can leave the system in an unusable state and prevent additional backups, follow the procedures in Clean up after a Failed Backup below.

Clean up after a Failed Backup

If your backup process fails, it might leave the BBR backup folder on the instance, causing any subsequent attempts to backup to fail. In addition, BBR might not have run the post-backup scripts, leaving the instance in a locked state.

Follow the steps below to use the BBR cleanup script after a failed backup attempt.

  1. If the BOSH Director backup failed, run the following command:

    bbr director \
    --private-key-path PATH-TO-PRIVATE-KEY \
    --username USER-NAME \
    --host HOST \
    backup-cleanup
    

    For example:

    $ bbr director \
    --private-key-path bosh.pem \
    --username vcap \
    --host bosh.example.com \
    backup-cleanup
    
  2. If the BOSH deployment backup failed, run the following command:

    BOSH_CLIENT_SECRET=BOSH-CLIENT-SECRET \
    bbr deployment \
    --target BOSH-TARGET \
    --username BOSH-CLIENT \
     --deployment DEPLOYMENT-NAME \
    --ca-cert PATH-TO-BOSH-CA-CERTIFICATE \
    backup-cleanup
    

    For example:

    $ BOSH_CLIENT_SECRET=p455w0rd \
    bbr deployment \
    --target bosh.example.com \
    --username admin \
    --deployment cf-acceptance-0 \
    --ca-cert bosh.ca.crt \
    backup-cleanup
    

Manage Your Backup Artifact

Keep your backup artifact safe by following these steps:

  1. Move the backup artifact off the jumpbox to your storage space. BBR stores each backup in a subdirectory named DEPLOYMENT-TIMESTAMP within the current working directory. The backup created by BBR consists of a folder with the backup artifacts and metadata files.

  2. Compress and encrypt the backup artifacts when storing them.

  3. Make redundant copies of your backup and store them in multiple locations. This minimizes the risk of losing your backups in the event of a disaster.

  4. Each time you redeploy Cloud Foundry, test your backup artifact by following the procedures in Restoring with BOSH Backup and Restore.

Note: Backup artifacts may contain data covered by the European Union’s General Data Protection Regulation (GDPR). Please be mindful that any backup artifacts may contain personal data. For example, a backup of a BOSH deployment could contain user email addresses. As such, backup artifacts should be handled in accordance with organizational policies and applicable laws as they pertain to security, confidentiality, and privacy.

Create a pull request or raise an issue on the source for this page in GitHub