Customizing the Cloud Foundry Deployment Manifest Stub for vSphere, vCloud Air, or vCloud Director

Page last updated:

This topic describes how to customize the Cloud Foundry deployment manifest stub for vSphere, vCloud Air, or vCloud Director.

To create a Cloud Foundry manifest, you must perform the following steps:

  1. Use the BOSH CLI to retrieve your BOSH Director UUID, which you use to customize your manifest stub.
  2. Create a manifest stub in YAML format. See the example manifest stub for vSphere, vCloud Air, or vCloud Director below, and follow the editing instructions to customize it for your deployment.
  3. Use a script to combine the manifest stub with other configuration files in the cf-release repository to generate your deployment manifest.

Note: vSphere defaults to using an internal WebDAV blobstore for the Cloud Controller. For alternative blobstore configurations, see Cloud Controller Blobstore Configuration.

Step 1: Retrieve Your BOSH Director UUID

To perform these procedures, you must have installed the BOSH CLI.

  1. Use the bosh target command with the address of your BOSH Director to connect to the BOSH Director. Log in with the default user name and password, admin and admin, or use the username and password that you set when you installed BOSH.
    $ bosh target https://bosh.my-domain.example.com
    Target set to `bosh'
    Your username: admin
    Enter password: *****
    Logged in as 'admin'
    
  2. Use the bosh status --uuid command to view information about your BOSH deployment. Record the UUID of the BOSH Director. You use the UUID when customizing the Cloud Foundry deployment manifest stub.
    $ bosh status --uuid
    abcdef12-3456-7890-abcd-ef1234567890
    

Step 2: Create Your Manifest Stub

Review the example manifest stub, and then follow the editing instructions to customize it for your deployment.

Cloud Foundry Deployment Manifest Stub

---
meta:
  environment: ENVIRONMENT

director_uuid: DIRECTOR_UUID

networks:
  - name: cf1
    subnets:
    - range: 10.85.9.0/24
      gateway: 10.85.9.1
      reserved:
      - 10.85.9.2 - 10.85.9.100
      - 10.85.9.200 - 10.85.9.210
      dns:
      - 10.87.8.10
      - 10.87.8.11
      static:
      - 10.85.9.230 - 10.85.9.254
      cloud_properties:
        name: VSPHERE_NETWORK_NAME_1

  - name: cf2
    subnets:
    - range: 10.85.10.0/24
      gateway: 10.85.10.1
      reserved:
      - 10.85.10.2 - 10.85.10.100
      - 10.85.10.200 - 10.85.10.210
      dns:
      - 10.87.8.10
      - 10.87.8.11
      static:
      - 10.85.10.230 - 10.85.10.254
      cloud_properties:
        name: VSPHERE_NETWORK_NAME_2

properties:
  system_domain: SYSTEM_DOMAIN
  system_domain_organization: SYSTEM_DOMAIN_ORGANIZATION
  app_domains:
    - APP_DOMAIN

  ssl:
    skip_cert_verify: true

  cc:
    staging_upload_user:  STAGING_UPLOAD_USER
    staging_upload_password: STAGING_UPLOAD_PASSWORD
    bulk_api_password: BULK_API_PASSWORD
    db_encryption_key: DB_ENCRYPTION_KEY
    mutual_tls:
      ca_cert: CC_MUTUAL_TLS_CA_CERT
      public_cert: CC_MUTUAL_TLS_PUBLIC_CERT
      private_key: CC_MUTUAL_TLS_PRIVATE_KEY

  blobstore:
    admin_users:
      - username: blobstore-username
        password: blobstore-password
    secure_link:
      secret: blobstore-secret
    tls:
      cert: BLOBSTORE_TLS_CERT
      private_key: BLOBSTORE_PRIVATE_KEY
      ca_cert: BLOBSTORE_CA_CERT
  consul:
    encrypt_keys:
      - CONSUL_ENCRYPT_KEY
    ca_cert: CONSUL_CA_CERT
    server_cert: CONSUL_SERVER_CERT
    server_key: CONSUL_SERVER_KEY
    agent_cert: CONSUL_AGENT_CERT
    agent_key: CONSUL_AGENT_KEY
  etcd:
    require_ssl: true
    ca_cert: ETCD_CA_CERT
    client_cert: ETCD_CLIENT_CERT
    client_key: ETCD_CLIENT_KEY
    peer_ca_cert: ETCD_PEER_CA_CERT
    peer_cert: ETCD_PEER_CERT
    peer_key: ETC_PEER_KEY
    server_cert: ETCD_SERVER_CERT
    server_key: ETCD_SERVER_KEY
  loggregator:
    tls:
      ca_cert: LOGGREGATOR_CA_CERT
      doppler:
        cert: LOGGREGATOR_DOPPLER_CERT
        key: LOGGREGATOR_DOPPLER_KEY
      trafficcontroller:
        cert: LOGGREGATOR_TRAFFICCONTROLLER_CERT
        key: LOGGREGATOR_TRAFFICCONTROLLER_KEY
      metron:
        cert: LOGGREGATOR_METRON_CERT
        key: LOGGREGATOR_METRON_KEY
      syslogdrainbinder:
        cert: LOGGREGATOR_SYSLOGDRAINBINDER_CERT
        key: LOGGREGATOR_SYSLOGDRAINBINDER_KEY
      statsd_injector:
        cert: LOGGREGATOR_STATSDINJECTOR_CERT
        key: LOGGREGATOR_STATSDINJECTOR_KEY
  loggregator_endpoint:
    shared_secret: LOGGREGATOR_ENDPOINT_SHARED_SECRET
  login:
    protocol: http
    saml:
      serviceProviderKey: SERVICE_PROVIDER_PRIVATE_KEY
  nats:
    user: NATS_USER
    password: NATS_PASSWORD
  router:
    status:
      user: ROUTER_USER
      password: ROUTER_PASSWORD
  uaa:
    admin:
      client_secret: ADMIN_SECRET
    ca_cert: UAA_CA_CERT
    cc:
      client_secret: CC_CLIENT_SECRET
    clients:
      cc_routing:
        secret: CC_ROUTING_SECRET
      cloud_controller_username_lookup:
        secret: CLOUD_CONTROLLER_USERNAME_LOOKUP_SECRET
      doppler:
        secret: DOPPLER_SECRET
      gorouter:
        secret: GOROUTER_SECRET
      tcp_emitter:
        secret: TCP-EMITTER-SECRET
      tcp_router:
        secret: TCP-ROUTER-SECRET
      login:
        secret: LOGIN_CLIENT_SECRET
      notifications:
        secret: NOTIFICATION_SECRET
      cc-service-dashboards:
        secret: CC_SERVICE_DASHBOARDS_SECRET
    jwt:
      verification_key: JWT_VERIFICATION_KEY
      signing_key: JWT_SIGNING_KEY
    scim:
      users:
      - name: admin
        password: ADMIN_PASSWORD
        groups:
        - scim.write
        - scim.read
        - openid
        - cloud_controller.admin
        - doppler.firehose
    sslCertificate: UAA_SERVER_CERT
    sslPrivateKey: UAA_SERVER_KEY

  ccdb:
    roles:
    - name: ccadmin
      password: CCDB_PASSWORD
  uaadb:
    roles:
    - name: uaaadmin
      password: UAADB_PASSWORD
  databases:
    roles:
    - name: ccadmin
      password: CCDB_PASSWORD
    - name: uaaadmin
      password: UAADB_PASSWORD

jobs:
- name: ha_proxy_z1
  properties:
    ha_proxy:
      ssl_pem: |
        -----BEGIN RSA PRIVATE KEY-----
        RSA_PRIVATE_KEY
        -----END RSA PRIVATE KEY-----
        -----BEGIN CERTIFICATE-----
        CERTIFICATE
        -----END CERTIFICATE-----

Editing Instructions

Deployment Manifest Stub Contents Editing Instructions

meta:
  environment: ENVIRONMENT
    
Replace ENVIRONMENT with an arbitrary name describing your environment, for example vsphere-prod.

director_uuid: DIRECTOR_UUID
    
Replace DIRECTOR_UUID with the BOSH Director UUID. Run the BOSH CLI command bosh status --uuid to view the BOSH Director UUID.

networks:
  - name: cf1
    subnets:
    - range: 10.85.9.0/24
      gateway: 10.85.9.1
      reserved:
      - 10.85.9.2 - 10.85.9.100
      - 10.85.9.200 - 10.85.9.210
      dns:
      - 10.87.8.10
      - 10.87.8.11
      static:
      - 10.85.9.230 - 10.85.9.254
      cloud_properties:
        name: VSPHERE_NETWORK_NAME_1

  - name: cf2
    subnets:
    - range: 10.85.10.0/24
      gateway: 10.85.10.1
      reserved:
      - 10.85.10.2 - 10.85.10.100
      - 10.85.10.200 - 10.85.10.210
      dns:
      - 10.87.8.10
      - 10.87.8.11
      static:
      - 10.85.10.230 - 10.85.10.254
      cloud_properties:
        name: VSPHERE_NETWORK_NAME_2
    
This examples assumes a two-network deployment. Replace VSPHERE_NETWORK_NAME_1 and VSPHERE_NETWORK_NAME_2 with names of networks in your vSphere datacenter. Update the values for range, reserved, static, dns, and gateway accordingly.

properties:
  system_domain: SYSTEM_DOMAIN
  system_domain_organization: SYSTEM_DOMAIN_ORGANIZATION
  app_domains:
   - APP_DOMAIN
    
Replace SYSTEM_DOMAIN and APP_DOMAIN with the full domain you want associated with applications pushed to your Cloud Foundry installation, for example cloud-09.cf-app.com. You must have already acquired these domains and configured their DNS records so that these domains resolve to your load balancer.

Choose a name for the SYSTEM_DOMAIN_ORGANIZATION. This organization will be created and configured to own the SYSTEM_DOMAIN.

  cc:
    staging_upload_user: STAGING_UPLOAD_USER
    staging_upload_password: STAGING_UPLOAD_PASSWORD
    bulk_api_password: BULK_API_PASSWORD
    db_encryption_key: CCDB_ENCRYPTION_KEY
    mutual_tls:
      ca_cert: CC_MUTUAL_TLS_CA_CERT
      public_cert: CC_MUTUAL_TLS_PUBLIC_CERT
      private_key: CC_MUTUAL_TLS_PRIVATE_KEY
    
The Cloud Controller API endpoint requires basic authentication. Replace STAGING_UPLOAD_USER and STAGING_UPLOAD_PASSWORD with a username and password of your choosing.

Replace BULK_API_PASSWORD with a password of your choosing. The password cannot contain characters that are not allowed for basic authentication unless you encode the characters. Health Manager uses this password to access the Cloud Controller bulk API.

Replace CCDB_ENCRYPTION_KEY with a secure key that you generate to encrypt sensitive values in the Cloud Controller database. You can use any random string. For example, run the following command from a command line to generate a 32-character random string: LC_ALL=C tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 32 ; echo

To generate the certificates and keys for the mutual_tls section, you need the generate-cf-diego-certs script from the cf-release repository. Run the generate-cf-diego-certs script to generate the certificates and keys for Cloud Controller.
For example, run the following in a terminal window:
$ ./scripts/generate-cf-diego-certs
This script outputs a directory named cf-diego-certs that contains a set of files with the certificates and keys you need.
In the stub, replace… with the contents of this file…
CC_MUTUAL_TLS_CA_CERT cf-diego-ca.crt
CC_MUTUAL_TLS_PUBLIC_CERT cloud_controller.crt
CC_MUTUAL_TLS_PRIVATE_KEY cloud_controller.key

  consul:
    encrypt_keys:
      - CONSUL_ENCRYPT_KEY
    ca_cert: CONSUL_CA_CERT
    server_cert: CONSUL_SERVER_CERT
    server_key: CONSUL_SERVER_KEY
    agent_cert: CONSUL_AGENT_CERT
    agent_key: CONSUL_AGENT_KEY
    
See the Security Configuration for Consul topic.

  etcd:
    require_ssl: true
    ca_cert: ETCD_CA_CERT
    client_cert: ETCD_CLIENT_CERT
    client_key: ETCD_CLIENT_KEY
    peer_ca_cert: ETCD_PEER_CA_CERT
    peer_cert: ETCD_PEER_CERT
    peer_key: ETCD_PEER_KEY
    server_cert: ETCD_SERVER_CERT
    server_key: ETCD_SERVER_KEY
    
Generate SSL certs for etcd and replace these values. You can use the scripts/generate-etcd-certs script in the cf-release repository to generate self signed certs.

    loggregator:
      tls:
        ca_cert: LOGGREGATOR_CA_CERT
        doppler:
          cert: LOGGREGATOR_DOPPLER_CERT
          key: LOGGREGATOR_DOPPLER_KEY
        trafficcontroller:
          cert: LOGGREGATOR_TRAFFICCONTROLLER_CERT
          key: LOGGREGATOR_TRAFFICCONTROLLER_KEY
        metron:
          cert: LOGGREGATOR_METRON_CERT
          key: LOGGREGATOR_METRON_KEY
        syslogdrainbinder:
          cert: LOGGREGATOR_SYSLOGDRAINBINDER_CERT
          key: LOGGREGATOR_SYSLOGDRAINBINDER_KEY
      
To generate the certificates and keys for the Loggregator components, you need:
  • The original CA certificate and key used to sign the keypairs for TLS between the Cloud Controller and the Diego BBS
  • The generate-loggregator-certs script from the cf-release repository
Run generate-loggregator-certs CA_CERT CA_KEY to generate the certificates and keys for Loggregator. Replace CA_CERT with the path and filename for the original CA certificate, and CA_KEY with the path and filename for the corresponding key.
For example, run the following in a terminal window:
$ ./scripts/generate-loggregator-certs cf-ca.cert cf-ca.key
This script outputs a directory named loggregator-certs that contains a set of files with the certificates and keys you need for Loggregator.
In the stub, replace… with the contents of this file…
LOGGREGATOR_CA_CERT loggregator-ca.crt
LOGGREGATOR_DOPPLER_CERT doppler.crt
LOGGREGATOR_DOPPLER_KEY doppler.key
LOGGREGATOR_TRAFFICCONTROLLER_CERT trafficcontroller.crt
LOGGREGATOR_TRAFFICCONTROLLER_KEY trafficontroller.key
LOGGREGATOR_METRON_CERT metron.crt
LOGGREGATOR_METRON_KEY metron.key
LOGGREGATOR_SYSLOGDRAINBINDER_CERT syslogdrainbinder.crt
LOGGREGATOR_SYSLOGDRAINBINDER_KEY syslogdrainbinder.key

  loggregator_endpoint:
    shared_secret: LOGGREGATOR_ENDPOINT_SHARED_SECRET
    
Replace LOGGREGATOR_ENDPOINT_SHARED_SECRET with a secure string that you generate.

    login:
      protocol: http
      saml:
        serviceProviderKey: SERVICE_PROVIDER_PRIVATE_KEY
    
Generate a PEM-encoded RSA key pair. You can generate a key pair by running the command openssl req -x509 -nodes -newkey rsa:2048 -days 365 -keyout key.pem -out cert.pem This command creates cert.pem, which contains your public key, and key.pem, which contains your private key. Replace SERVICE_PROVIDER_PRIVATE_KEY with the full private key, include the BEGIN and END delimiter lines, under serviceProviderKey.
For RSA keys, you only need to configure the private key.

  nats:
    user: NATS_USER
    password: NATS_PASSWORD
    
Replace NATS_USER and NATS_PASSWORD with a username and secure password of your choosing. Cloud Foundry components use these credentials to communicate with each other over the NATS message bus.

  router:
    status:
      user: ROUTER_USER
      password: ROUTER_PASSWORD
    
Replace ROUTER_USER and ROUTER_PASSWORD with a username and secure password of your choosing.

  uaa:
    admin:
      client_secret: ADMIN_SECRET
    cc:
      client_secret: CC_CLIENT_SECRET
    clients:
      cc-service-dashboards:
        secret: CC_SERVICE_DASHBOARDS_SECRET
      cc_routing:
        secret: CC_ROUTING_SECRET
      cloud_controller_username_lookup:
        secret: CLOUD_CONTROLLER_USERNAME_LOOKUP_SECRET
      doppler:
        secret: DOPPLER_SECRET
      gorouter:
        secret: GOROUTER_SECRET
      tcp_emitter:
        secret: TCP-EMITTER-SECRET
      tcp_router:
        secret: TCP-ROUTER-SECRET
      login:
        secret: LOGIN_CLIENT_SECRET
      notifications:
        secret: NOTIFICATIONS_CLIENT_SECRET
    
Replace the values for all secret keys with secure secrets that you generate.

You can configure container-to-container networking in this section. If you want to deploy container-to-container networking, see the Enable on an IaaS section of the Administering Container-to-Container Networking topic, beginning with step 4.

    jwt:
      verification_key: JWT_VERIFICATION_KEY
      signing_key: JWT_SIGNING_KEY
    
Generate a PEM-encoded RSA key pair, and replace JWT_SIGNING_KEY with the private key, and JWT_VERIFICATION_KEY with the corresponding public key. Generate a key pair by running the command openssl rsa -in jwt-key.pem -pubout > key.pub. This command creates jwt-key.pem.pub, which contains your public key, and jwt-key.pem, which contains your private key.
Copy in the full keys, including the BEGIN and END delimiter lines.

    scim:
      users:
        - admin|ADMIN_PASSWORD|scim.write,scim.read,o...
    
Generate a secure password and replace ADMIN_PASSWORD with that value to set the password for the Admin user of your Cloud Foundry installation.

    sslCertificate: UAA_SERVER_CERT
    sslPrivateKey: UAA_SERVER_KEY
    
Replace UAA_SERVER_CERT and UAA_SERVER_KEY with UAA certificates. You can use the scripts/generate-uaa-certs script in the cf-release repository to generate self-signed certificates.

  ccdb:
    roles:
    - name: ccadmin
      password: CCDB_PASSWORD
  uaadb:
    roles:
    - name: uaaadmin
      password: UAADB_PASSWORD
  databases:
    roles:
    - name: ccadmin
      password: CCDB_PASSWORD
    - name: uaaadmin
      password: UAADB_PASSWORD
    
Replace CCDB_PASSWORD and UAADB_PASSWORD with secure passwords of your choosing.

  hm9000:
    ca_cert: HM9000_CA_CERT
    server_cert: HM9000_SERVER_CERT
    server_key: HM9000_SERVER_KEY
    agent_cert: HM9000_AGENT_CERT
    agent_key: HM9000_AGENT_KEY
    
Generate SSL certificates for HM9000 and replace these values. You can use the scripts/generate-hm9000-certs script in the cf-release repository to generate self signed certificates.

jobs:
- name: ha_proxy_z1
  properties:
    ha_proxy:
      ssl_pem: |
        -----BEGIN RSA PRIVATE KEY-----
        RSA_PRIVATE_KEY
        -----END RSA PRIVATE KEY-----
        -----BEGIN CERTIFICATE-----
        CERTIFICATE
        -----END CERTIFICATE-----
    
Replace RSA_PRIVATE_KEY and CERTIFICATE with the private key and certificate for SSL termination of the DNS domains you pointed at the HAProxy static IP address. Typically, this certificate is for wildcard subdomains of your SYSTEM_DOMAIN and APPS_DOMAIN.

Note: You can configure blacklists of IP address ranges to prevent future apps deployed to your Cloud Foundry installation from attempting to drain syslogs to internal Cloud Foundry components. See the Log Drain Blacklist Configuration topic for more information.

Note: HAProxy is deployed by default as your external-facing load balancer. In vSphere, you can use F5 instead of HAProxy. However, setting up F5 requires advanced configuration. F5 setup instructions are not covered in this documentation.

Step 3: Generate Your Manifest

To generate a deployment manifest, perform the following steps:

  1. Clone the cf-release GitHub repository. Use git clone to copy the latest Cloud Foundry configuration files onto your computer.

    $ git clone https://github.com/cloudfoundry/cf-release.git
    
  2. From the cf-release directory, run the update script to fetch all the submodules.

    $ cd cf-release
    $ ./scripts/update
    

    Note: Ensure that you have the most up-to-date version of the Cloud Foundry code and all required submodules.

  3. Install spiff, a command line tool for generating deployment manifests.

  4. Run generate_deployment_manifest IAAS PATH-TO-MANIFEST-STUB > cf-deployment.yml from the cf-release directory to create a deployment manifest named cf-deployment.yml.

    • Replace IAAS with aws, openstack, or vsphere. Use vsphere for vSphere, vCloud Air, and vCloud Director.
    • Replace PATH-TO-MANIFEST-STUB with the location of your cf-stub.yml file.

      The following example specifies AWS as an IaaS, and references a cf-stub.yml file located in the in the cf-release directory:
      $ ./scripts/generate_deployment_manifest aws ./cf-stub.yml > cf-deployment.yml
      

      Note: The generate_deployment_manifest script can accept multiple stub files. For example, the following command passes two stub files to the script:
      ./scripts/generate_deployment_manifest vsphere cf-stub.yml cf-consul.yml > cf-deployment.yml

  5. Use the bosh deployment command to set your deployment to the generated manifest:

    $ bosh deployment cf-deployment.yml
    

You are now ready to deploy Cloud Foundry. See the Deploying Cloud Foundry topic for instructions.

Create a pull request or raise an issue on the source for this page in GitHub