Encrypting Cloud Controller database and rotating keys

Page last updated:

You can encrypt sensitive data in the Cloud Controller (CC) database with Cloud Foundry by using operator provided keys.

Enabling encryption

In the following steps, the examples show a single encryption key being configured and set as the current key, but multiple keys can be stored under keys. Multiple keys are allowed so that CC can decrypt your previously encrypted sensitive data before rotating to a new key. Only the key with its label set as the current_key_label is used for encryption.

1. Choose a value for the encryption key (e.g. "example-random-key-string").

2. Choose a label for the encryption key (e.g. encryption_key_2023_02_20). The label must abide by the following restrictions:

   • 127 character limit
   • Cannot contain a : (colon)

3. Format the encryption key values into YAML as follows:

   database_encryption:
     current_key_label: "encryption_key_2023_02_20"
     keys:
       encryption_key_2023_02_20: "example-random-key-string"
   

   If your CC instance groups have the db_encryption_key key, ensure that you add db_encryption_key to the keys block, and that you set its corresponding key label to value of current_key_label. For example:

   db_encryption_key: "example-random-key-string"
   database_encryption:
     current_key_label: "encryption_key_2023_02_20"
     keys:
       encryption_key_2023_02_20: "example-random-key-string"
   

4. Make your changes by either creating and applying an ops file or directly modifying the manifest.

5. Deploy the changes with the command that corresponds to your use case:

   • With an ops file

     $ bosh deploy -o YOUR_OPS_FILE_NAME PATH_TO_YOUR_MANIFEST
     

   • Direct modification of the manifest

     $ bosh deploy PATH_TO_YOUR_MANIFEST
     

6. Run the bosh errand to encrypt the fields:

   $ bosh run-errand rotate-cc-database-key

7. Once the errand has run successfully, you can safely remove the db_encryption_key from the manifest.

   If you remove db_encryption_key from the manifest before you run the errand, CC no longer is able to decrypt your previously encrypted sensitive data.

Configuring through an ops file

The recommended way of enabling encryption is to use an ops file. YAML anchors (for example, encryption_info in the following example) can also be used to reduce repetition and ensure consistency of encryption key properties across the instance groups.

For example, the following can be used as an ops file:

- type: replace
  path: /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/cc/database_encryption?
  value: &encryption_info
    keys:
      encryption_key_2023_02_20: "example-random-key-string"
    current_key_label: "encryption_key_2023_02_20"

- type: replace
  path: /instance_groups/name=cc-worker/jobs/name=cloud_controller_worker/properties/cc/database_encryption?
  value: *encryption_info

- type: replace
  path: /instance_groups/name=scheduler/jobs/name=cloud_controller_clock/properties/cc/database_encryption?
  value: *encryption_info

Configuring through the manifest

For each CC instance group (api, cc-worker, scheduler) in the manifest, insert the database_encryption YAML block so that the final YAML looks like the following:

instance_groups:
  api:
    jobs:
      cloud_controller_ng:
        properties:
          database_encryption: &encryption_info
            keys:
              encryption_key_2023_02_20: "example-random-key-string"
            current_key_label: "encryption_key_2023_02_20"
  ...
  cc-worker:
    jobs:
      cloud_controller_ng:
        properties:
          database_encryption: *encryption_info
  ...
  scheduler:
    jobs:
      cloud_controller_ng:
        properties:
          database_encryption: *encryption_info

The keys and current_key_label field must match across all instance groups - our example uses YAML anchors to achieve this.

Validating encryption properties

The CC runs validations before and during deployment to ensure that it can decrypt all previously encrypted sensitive data.

The following are validated:

• The db_encryption_key is present if used in the database.
• The current_key_label exists in the keys hash.
• The key associated with an encryption key label in the keys hash does not change.
• All in-use encryption key label/value pairs exist in the keys hash.

In certain situations, you can decide to skip these validations for a deploy to complete successfully. They can do so by setting the database_encryption.skip_validation property to true temporarily.

Skipping database encryption validation is not recommended.

Rotating encryption keys

It is important that the mapping between the encryption key labels and key values in the database_encryption.keys property be maintained. If this mapping is not maintained, it leads to loss of access to encrypted values in the database. The correct way to add a new key is described in the following steps.

In order to rotate an encryption key:

1. Add a new encryption key to the list of keys (e.g. encryption_key_2023_02_21.

2. Update the current_key_label to be the label of the new encryption key (for example, updating the current_key_label to be `“encryption_key_2023_02_21”).

   The following is an example of an ops file used for rotation:

   - type: replace
     path: /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/cc/database_encryption?
     value: &encryption_info
       keys:
         encryption_key_2023_02_20: "example-random-key-string"
         encryption_key_2023_02_21: "other-random-key-string"
       current_key_label: "encryption_key_2023_02_21"
   
   - type: replace
     path: /instance_groups/name=cc-worker/jobs/name=cloud_controller_worker/properties/cc/database_encryption?
     value: *encryption_info
   
   - type: replace
     path: /instance_groups/name=scheduler/jobs/name=cloud_controller_clock/properties/cc/database_encryption?
     value: *encryption_info
   

3. Deploy the changes by running the following command.

   $ bosh deploy -o YOUR_OPS_FILE_NAME PATH_TO_YOUR_MANIFEST

   Whether the errand is run or not, any updates to sensitive data is encrypted with the new key.

4. Run the bosh errand command to rotate the encryption key.

   $ bosh run-errand rotate-cc-database-key

5. Once the errand has run successfully, you can safely remove the old key from the manifest.

   If you remove the old key from the manifest before running the errand, CC is no longer able to decrypt your previously encrypted sensitive data.