Customizing the Cloud Foundry Deployment Manifest for OpenStack

Page last updated:

This topic describes how to customize the Cloud Foundry deployment manifest for OpenStack.

To create a Cloud Foundry manifest, you must perform the following steps:

  1. Use the BOSH CLI to retrieve your BOSH Director UUID, which you use to customize your manifest stub.
  2. Create a manifest stub in YAML format. See the example manifest stub for OpenStack below, and follow the editing instructions to customize it for your deployment.
  3. Use a script to combine the manifest stub with other configuration files in the cf-release repository to generate your deployment manifest.

Note: OpenStack defaults to using an internal WebDAV blobstore for the Cloud Controller. For alternative blobstore configurations, see the Cloud Controller Blobstore Configuration and Using OpenStack Swift as a Cloud Foundry Blobstore topics.

Step 1: Retrieve Your BOSH Director UUID

To perform these procedures, you must have installed the BOSH CLI.

  1. Use the bosh target command with the address of your BOSH Director to connect to the BOSH Director. Log in with the default user name and password, admin and admin, or use the username and password that you set when you installed BOSH.
    $ bosh target https://bosh.my-domain.example.com
    Target set to `bosh'
    Your username: admin
    Enter password: *****
    Logged in as 'admin'
    
  2. Use the bosh status --uuid command to view information about your BOSH deployment. Record the UUID of the BOSH Director. You use the UUID when customizing the Cloud Foundry deployment manifest stub.
    $ bosh status --uuid
    abcdef12-3456-7890-abcd-ef1234567890
    

Step 2: Create Your Manifest Stub

Review the example manifest stub for OpenStack, and then follow the editing instructions to customize it for your deployment.

Cloud Foundry Deployment Manifest Stub for OpenStack

---
director_uuid: DIRECTOR_UUID

meta:
  environment: ENVIRONMENT

  floating_static_ips:
  - 173.1.1.1

networks:
  - name: floating
    type: vip
    cloud_properties:
      net_id: NET_ID
      security_groups: []
  - name: cf1
    type: manual
    subnets:
    - range: 10.10.0.0/24
      gateway: 10.10.0.1
      reserved:
      - 10.10.0.2 - 10.10.0.100
      - 10.10.0.200 - 10.10.0.254
      dns:
      - 8.8.8.8
      static:
      - 10.10.0.125 - 10.10.0.175
      cloud_properties:
        net_id: NET_ID
        security_groups: ["cf"]
  - name: cf2
    type: manual
    subnets: (( networks.cf1.subnets )) # cf2 unused by default with the OpenStack template
                                        # but the general upstream templates require this
                                        # to be a semi-valid value, so just copy cf1

properties:
  system_domain: SYSTEM_DOMAIN
  system_domain_organization: SYSTEM_DOMAIN_ORGANIZATION
  app_domains:
   - APP_DOMAIN

  ssl:
    skip_cert_verify: true

  cc:
    staging_upload_user: STAGING_UPLOAD_USER
    staging_upload_password: STAGING_UPLOAD_PASSWORD
    bulk_api_password: BULK_API_PASSWORD
    db_encryption_key: DB_ENCRYPTION_KEY
    uaa_skip_ssl_validation: true
    tls_port: CC_MUTUAL_TLS_PORT
    mutual_tls:
      ca_cert: CC_MUTUAL_TLS_CA_CERT
      public_cert: CC_MUTUAL_TLS_PUBLIC_CERT
      private_key: CC_MUTUAL_TLS_PRIVATE_KEY

  blobstore:
    admin_users:
      - username: blobstore-username
        password: blobstore-password
    secure_link:
      secret: blobstore-secret
    tls:
      cert: BLOBSTORE_TLS_CERT
      private_key: BLOBSTORE_PRIVATE_KEY
      ca_cert: BLOBSTORE_CA_CERT
  consul:
    encrypt_keys:
      - CONSUL_ENCRYPT_KEY
    ca_cert: CONSUL_CA_CERT
    server_cert: CONSUL_SERVER_CERT
    server_key: CONSUL_SERVER_KEY
    agent_cert: CONSUL_AGENT_CERT
    agent_key: CONSUL_AGENT_KEY
  etcd:
    require_ssl: true
    ca_cert: ETCD_CA_CERT
    client_cert: ETCD_CLIENT_CERT
    client_key: ETCD_CLIENT_KEY
    peer_ca_cert: ETCD_PEER_CA_CERT
    peer_cert: ETCD_PEER_CERT
    peer_key: ETC_PEER_KEY
    server_cert: ETCD_SERVER_CERT
    server_key: ETCD_SERVER_KEY
  loggregator:
    tls:
      ca_cert: LOGGREGATOR_CA_CERT
      doppler:
        cert: LOGGREGATOR_DOPPLER_CERT
        key: LOGGREGATOR_DOPPLER_KEY
      trafficcontroller:
        cert: LOGGREGATOR_TRAFFICCONTROLLER_CERT
        key: LOGGREGATOR_TRAFFICCONTROLLER_KEY
      cc_trafficcontroller:
        cert: LOGGREGATOR_TRAFFICCONTROLLER_CERT
        key: LOGGREGATOR_TRAFFICCONTROLLER_KEY
      metron:
        cert: LOGGREGATOR_METRON_CERT
        key: LOGGREGATOR_METRON_KEY
      syslogdrainbinder:
        cert: LOGGREGATOR_SYSLOGDRAINBINDER_CERT
        key: LOGGREGATOR_SYSLOGDRAINBINDER_KEY
      statsd_injector:
        cert: LOGGREGATOR_STATSDINJECTOR_CERT
        key: LOGGREGATOR_STATSDINJECTOR_KEY
  loggregator_endpoint:
    shared_secret: LOGGREGATOR_ENDPOINT_SHARED_SECRET
  login:
    protocol: http
    saml:
      serviceProviderCertificate: SERVICE_PROVIDER_CERTIFICATE
      serviceProviderKey: SERVICE_PROVIDER_PRIVATE_KEY
  nats:
    user: NATS_USER
    password: NATS_PASSWORD
  router:
    status:
      user: ROUTER_USER
      password: ROUTER_PASSWORD
  uaa:
    admin:
      client_secret: ADMIN_SECRET
    ca_cert: UAA_CA_CERT
    cc:
      client_secret: CC_CLIENT_SECRET
    clients:
      cc_service_key_client:
        secret: CC_SERVICE_KEY_CLIENT_SECRET
      cc_routing:
        secret: CC_ROUTING_SECRET
      cloud_controller_username_lookup:
        secret: CLOUD_CONTROLLER_USERNAME_LOOKUP_SECRET
      doppler:
        secret: DOPPLER_SECRET
      gorouter:
        secret: GOROUTER_SECRET
      tcp_emitter:
        secret: TCP-EMITTER-SECRET
      tcp_router:
        secret: TCP-ROUTER-SECRET
      login:
        secret: LOGIN_CLIENT_SECRET
      notifications:
        secret: NOTIFICATIONS_CLIENT_SECRET
      cc-service-dashboards:
        secret: CC_SERVICE_DASHBOARDS_SECRET
    jwt:
      verification_key: JWT_VERIFICATION_KEY
      signing_key: JWT_SIGNING_KEY
    scim:
      users:
      - name: admin
        password: ADMIN_PASSWORD
        groups:
        - scim.write
        - scim.read
        - openid
        - cloud_controller.admin
        - doppler.firehose
    sslCertificate: UAA_SERVER_CERT
    sslPrivateKey: UAA_SERVER_KEY

  ccdb:
    roles:
    - name: ccadmin
      password: CCDB_PASSWORD
  uaadb:
    roles:
    - name: uaaadmin
      password: UAADB_PASSWORD
  databases:
    roles:
    - name: ccadmin
      password: CCDB_PASSWORD
    - name: uaaadmin
      password: UAADB_PASSWORD
    - name: diego
      password: DIEGODB_PASSWORD

jobs:
  - name: ha_proxy_z1
    networks:
      - name: cf1
        default:
        - dns
        - gateway
    properties:
      ha_proxy:
        ssl_pem: |
          -----BEGIN RSA PRIVATE KEY-----
          RSA_PRIVATE_KEY
          -----END RSA PRIVATE KEY-----
          -----BEGIN CERTIFICATE-----
          SSL_CERTIFICATE_SIGNED_BY_PRIVATE_KEY
          -----END CERTIFICATE-----
  - name: api_z1
    templates:
      - name: consul_agent
        release: cf
        consumes: {consul_client: nil, consul_server: nil, consul_common: nil}
      - name: java-buildpack
        release: cf
      - name: go-buildpack
        release: cf
      - name: binary-buildpack
        release: cf
      - name: nodejs-buildpack
        release: cf
      - name: ruby-buildpack
        release: cf
      - name: php-buildpack
        release: cf
      - name: python-buildpack
        release: cf
      - name: staticfile-buildpack
        release: cf
      - name: dotnet-core-buildpack
        release: cf
      - name: cloud_controller_ng
        release: cf
        consumes: {nats: nil}
      - name: cloud_controller_clock
        release: cf
        consumes: {nats: nil}
      - name: cloud_controller_worker
        release: cf
        consumes: {nats: nil}
      - name: metron_agent
        release: cf
      - name: statsd_injector
        release: cf
      - name: route_registrar
        release: cf
        consumes: {nats: nil}

  - name: api_worker_z1
    instances: 0
  - name: clock_z1
    instances: 1

Editing Instructions

Deployment Manifest Stub Contents Editing Instructions

director_uuid: DIRECTOR_UUID
    
Replace DIRECTOR_UUID with the BOSH Director UUID. Run the BOSH CLI bosh status --uuid command to view the BOSH Director UUID.

meta:
  environment: ENVIRONMENT
    
Replace ENVIRONMENT with an arbitrary name describing your environment, for example openstack-prod.

  floating_static_ips:
  - 198.51.100.1
    
Replace 198.51.100.1 with an existing static IP address for your OpenStack floating network. This is assigned to the ha_proxy job to receive incoming traffic.

networks:
  - name: floating
    type: vip
    cloud_properties:
      net_id: NET_ID
      security_groups: []
  - name: cf1
    type: manual
    subnets:
    - range: 10.10.0.0/24
      gateway: 10.10.0.1
      reserved:
      - 10.10.0.2 - 10.10.0.100
      - 10.10.0.200 - 10.10.0.254
      dns:
      - 8.8.8.8
      static:
      - 10.10.0.125 - 10.10.0.175
      cloud_properties:
        net_id: NET_ID
        security_groups: ["cf"]
    
Update the values for range, reserved, static, and gateway to reflect the available networks and IP addresses in your OpenStack network. Replace NET_ID with the network ID of your OpenStack network.

This also assumes that you have a security group cf suitable for your Cloud Foundry VMs. Change this to the name of your security group if necessary.

properties:
  system_domain: SYSTEM_DOMAIN
  system_domain_organization: SYSTEM_DOMAIN_ORGANIZATION
  app_domains:
   - APP_DOMAIN
    
Replace SYSTEM_DOMAIN and APP_DOMAIN with the full domain you want associated with applications pushed to your Cloud Foundry installation, for example cloud-09.cf-app.com. You must have already acquired these domains and configured their DNS records so that these domains resolve to your load balancer.

Choose a name for the SYSTEM_DOMAIN_ORGANIZATION. This organization will be created and configured to own the SYSTEM_DOMAIN.

  cc:
    staging_upload_user: STAGING_UPLOAD_USER
    staging_upload_password: STAGING_UPLOAD_PASSWORD
    bulk_api_password: BULK_API_PASSWORD
    db_encryption_key: CCDB_ENCRYPTION_KEY
    mutual_tls:
      ca_cert: CC_MUTUAL_TLS_CA_CERT
      public_cert: CC_MUTUAL_TLS_PUBLIC_CERT
      private_key: CC_MUTUAL_TLS_PRIVATE_KEY
    
The Cloud Controller API endpoint requires basic authentication. Replace STAGING_UPLOAD_USER and STAGING_UPLOAD_PASSWORD with a username and password of your choosing.

Replace BULK_API_PASSWORD with a password of your choosing. The password cannot contain characters that are not allowed for basic authentication unless you encode the characters. Health Manager uses this password to access the Cloud Controller bulk API.

Replace CCDB_ENCRYPTION_KEY with a secure key that you generate to encrypt sensitive values in the Cloud Controller database. You can use any random string. For example, run the following command from a command line to generate a 32-character random string: LC_ALL=C tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 32 ; echo

To generate the certificates and keys for the mutual_tls section, you need the generate-cf-diego-certs script from the cf-release repository. Run the generate-cf-diego-certs script to generate the certificates and keys for Cloud Controller.
For example, run the following in a terminal window:
$ ./scripts/generate-cf-diego-certs
This script outputs a directory named cf-diego-certs that contains a set of files with the certificates and keys you need.
In the stub, replace… with the contents of this file…
CC_MUTUAL_TLS_CA_CERT cf-diego-ca.crt
CC_MUTUAL_TLS_PUBLIC_CERT cloud_controller.crt
CC_MUTUAL_TLS_PRIVATE_KEY cloud_controller.key

  blobstore:
    admin_users:
      - username: blobstore-username
        password: blobstore-password
    secure_link:
        secret: blobstore-secret
    tls:
        port: 443
        cert: BLOBSTORE_TLS_CERT
        private_key: BLOBSTORE_PRIVATE_KEY
        ca_cert: BLOBSTORE_CA_CERT
    
Replace blobstore-username and blobstore-password with a username and password of your choosing.

Replace blobstore-secret with a secure secret of your choosing.

Replace BLOBSTORE_TLS_CERT, BLOBSTORE_PRIVATE_KEY, and BLOBSTORE_CA_CERT with the blobstore TLS certificate, private key, and CA certificate.

  consul:
    encrypt_keys:
      - CONSUL_ENCRYPT_KEY
    ca_cert: CONSUL_CA_CERT
    server_cert: CONSUL_SERVER_CERT
    server_key: CONSUL_SERVER_KEY
    agent_cert: CONSUL_AGENT_CERT
    agent_key: CONSUL_AGENT_KEY
    
See the Security Configuration for Consul topic.

  etcd:
    require_ssl: true
    ca_cert: ETCD_CA_CERT
    client_cert: ETCD_CLIENT_CERT
    client_key: ETCD_CLIENT_KEY
    peer_ca_cert: ETCD_PEER_CA_CERT
    peer_cert: ETCD_PEER_CERT
    peer_key: ETCD_PEER_KEY
    server_cert: ETCD_SERVER_CERT
    server_key: ETCD_SERVER_KEY
    
Generate SSL certs for etcd and replace these values. You can use the scripts/generate-etcd-certs script in the cf-release repository to generate self signed certs.

   loggregator:
    tls:
      ca_cert: LOGGREGATOR_CA_CERT
      doppler:
        cert: LOGGREGATOR_DOPPLER_CERT
        key: LOGGREGATOR_DOPPLER_KEY
      trafficcontroller:
        cert: LOGGREGATOR_TRAFFICCONTROLLER_CERT
        key: LOGGREGATOR_TRAFFICCONTROLLER_KEY
      metron:
        cert: LOGGREGATOR_METRON_CERT
        key: LOGGREGATOR_METRON_KEY
      syslogdrainbinder:
        cert: LOGGREGATOR_SYSLOGDRAINBINDER_CERT
        key: LOGGREGATOR_SYSLOGDRAINBINDER_KEY
    
To generate the certificates and keys for the Loggregator components, you need the following:
  • The original CA certificate and key used to sign the keypairs for TLS between the Cloud Controller and the Diego BBS
  • The generate-loggregator-certs script from the cf-release repository
Run generate-loggregator-certs CA_CERT CA_KEY to generate the certificates and keys for Loggregator. Replace CA_CERT with the path and filename for the original CA certificate, and CA_KEY with the path and filename for the corresponding key.
For example, run the following in a terminal window:
$ ./scripts/generate-loggregator-certs cf-ca.cert cf-ca.key
This script outputs a directory named loggregator-certs that contains a set of files with the certificates and keys you need for Loggregator.
In the stub, replace… with the contents of this file…
LOGGREGATOR_CA_CERT loggregator-ca.crt
LOGGREGATOR_DOPPLER_CERT doppler.crt
LOGGREGATOR_DOPPLER_KEY doppler.key
LOGGREGATOR_TRAFFICCONTROLLER_CERT trafficcontroller.crt
LOGGREGATOR_TRAFFICCONTROLLER_KEY trafficontroller.key
LOGGREGATOR_METRON_CERT metron.crt
LOGGREGATOR_METRON_KEY metron.key
LOGGREGATOR_SYSLOGDRAINBINDER_CERT syslogdrainbinder.crt
LOGGREGATOR_SYSLOGDRAINBINDER_KEY syslogdrainbinder.key

  loggregator_endpoint:
    shared_secret: LOGGREGATOR_ENDPOINT_SHARED_SECRET
    
Replace LOGGREGATOR_ENDPOINT_SHARED_SECRET with a secure string that you generate.

    login:
      protocol: http
      saml:
        serviceProviderCertificate: SERVICE_PROVIDER_CERTIFICATE
        serviceProviderKey: SERVICE_PROVIDER_PRIVATE_KEY
    
Generate a PEM-encoded RSA key pair. You can generate a key pair by running the command openssl req -x509 -nodes -newkey rsa:2048 -days 365 -keyout key.pem -out cert.pem This command creates cert.pem, which contains your public key, and key.pem, which contains your private key. Replace SERVICE_PROVIDER_PRIVATE_KEY with the full private key, including the BEGIN and END delimiter lines, under serviceProviderKey.
For RSA keys, you only need to configure the private key.

  nats:
    user: NATS_USER
    password: NATS_PASSWORD
    
Replace NATS_USER and NATS_PASSWORD with a username and secure password of your choosing. Cloud Foundry components use these credentials to communicate with each other over the NATS message bus.

  router:
    status:
      user: ROUTER_USER
      password: ROUTER_PASSWORD
    
Replace ROUTER_USER and ROUTER_PASSWORD with a username and secure password of your choosing.

  uaa:
    admin:
      client_secret: ADMIN_SECRET
    cc:
      client_secret: CC_CLIENT_SECRET
    clients:
      cc-service-dashboards:
        secret: CC_SERVICE_DASHBOARDS_SECRET
      cc_routing:
        secret: CC_ROUTING_SECRET
      cloud_controller_username_lookup:
        secret: CLOUD_CONTROLLER_USERNAME_LOOKUP_SECRET
      doppler:
        secret: DOPPLER_SECRET
      gorouter:
        secret: GOROUTER_SECRET
      tcp_emitter:
        secret: TCP-EMITTER-SECRET
      tcp_router:
        secret: TCP-ROUTER-SECRET
      login:
        secret: LOGIN_CLIENT_SECRET
      notifications:
        secret: NOTIFICATIONS_CLIENT_SECRET
    
Replace the values for all secret keys with secure secrets that you generate.

You can configure container-to-container networking in this section. If you want to deploy container-to-container networking, see the Enable on an IaaS section of the Administering Container-to-Container Networking topic, beginning with step 4.

    jwt:
      verification_key: JWT_VERIFICATION_KEY
      signing_key: JWT_SIGNING_KEY
    
Generate a PEM-encoded RSA key pair, and replace JWT_SIGNING_KEY with the private key, and JWT_VERIFICATION_KEY with the corresponding public key. Generate a key pair by running the command openssl genrsa -out jwt-key.pem 2048 && openssl rsa -pubout -in jwt-key.pem -out jwt-key.pem.pub. This command creates jwt-key.pem.pub, which contains your public key, and jwt-key.pem, which contains your private key.
Copy in the full keys, including the BEGIN and END delimiter lines.

    scim:
      users:
      - name: admin
        password: ADMIN_PASSWORD
    
Generate a secure password and replace ADMIN_PASSWORD with that value to set the password for the Admin user of your Cloud Foundry installation.

    sslCertificate: UAA_SERVER_CERT
    sslPrivateKey: UAA_SERVER_KEY
    
Replace UAA_SERVER_CERT and UAA_SERVER_KEY with UAA certificates. You can use the scripts/generate-uaa-certs script in the cf-release repository to generate self-signed certificates.

  ccdb:
    roles:
    - name: ccadmin
      password: CCDB_PASSWORD
  uaadb:
    roles:
      - name: uaaadmin
        password: UAADB_PASSWORD
  databases:
    roles:
    - name: ccadmin
      password: CCDB_PASSWORD
    - name: uaaadmin
      password: UAADB_PASSWORD
    - name: diego
      password: DIEGODB_PASSWORD
    
Replace CCDB_PASSWORD, UAADB_PASSWORD, and DIEGODB_PASSWORD with secure passwords of your choosing.

  hm9000:
    ca_cert: HM9000_CA_CERT
    server_cert: HM9000_SERVER_CERT
    server_key: HM9000_SERVER_KEY
    agent_cert: HM9000_AGENT_CERT
    agent_key: HM9000_AGENT_KEY
    
Generate SSL certificates for HM9000 and replace these values. You can use the scripts/generate-hm9000-certs script in the cf-release repository to generate self signed certificates.

  login:
    saml:
      serviceProviderKey: SAML_KEY
      serviceProviderKeyPassword: ''
      serviceProviderCertificate: SAML_CERT
    
Generate SSL certificates for SAML and replace these values. You can use the scripts/generate-certs script in the cf-release repository to generate self-signed certificates.

jobs:
  - name: ha_proxy_z1
    networks:
      - name: cf1
        default:
        - dns
        - gateway
    properties:
      ha_proxy:
        ssl_pem: |
          -----BEGIN RSA PRIVATE KEY-----
          RSA_PRIVATE_KEY
          -----END RSA PRIVATE KEY-----
          -----BEGIN CERTIFICATE-----
          SSL_CERTIFICATE_SIGNED_BY_PRIVATE_KEY
          -----END CERTIFICATE-----
    
Replace RSA_PRIVATE_KEY and SSL_CERTIFICATE_SIGNED_BY_PRIVATE_KEY with the PEM-encoded private key and certificate associated with the system domain and apps domains that you configured to terminate at the floating IP address associated with the ha_proxy job.

Note: You can configure blacklists of IP address ranges to prevent future apps deployed to your Cloud Foundry installation from attempting to drain syslogs to internal Cloud Foundry components. See the Log Drain Blacklist Configuration topic for more information.

Step 3: Generate Your Manifest

To generate a deployment manifest, perform the following steps:

  1. Clone the cf-release GitHub repository. Use git clone to copy the latest Cloud Foundry configuration files onto your computer.

    $ git clone https://github.com/cloudfoundry/cf-release.git
    
  2. From the cf-release directory, run the update script to fetch all the submodules.

    $ cd cf-release
    $ ./scripts/update
    

    Note: Ensure that you have the most up-to-date version of the Cloud Foundry code and all required submodules.

  3. Install spiff, a command line tool for generating deployment manifests.

  4. Run generate_deployment_manifest IAAS PATH-TO-MANIFEST-STUB > cf-deployment.yml from the cf-release directory to create a deployment manifest named cf-deployment.yml.

    • Replace IAAS with aws, openstack, or vsphere. Use vsphere for vSphere, vCloud Air, and vCloud Director.
    • Replace PATH-TO-MANIFEST-STUB with the location of your cf-stub.yml file.

      The following example specifies AWS as an IaaS, and references a cf-stub.yml file located in the in the cf-release directory:
      $ ./scripts/generate_deployment_manifest aws ./cf-stub.yml > cf-deployment.yml
      

      Note: The generate_deployment_manifest script can accept multiple stub files. For example, the following command passes two stub files to the script:
      ./scripts/generate_deployment_manifest vsphere cf-stub.yml cf-consul.yml > cf-deployment.yml

  5. Use the bosh deployment command to set your deployment to the generated manifest:

    $ bosh deployment cf-deployment.yml
    

You are now ready to deploy Cloud Foundry. See the Deploying Cloud Foundry topic for instructions.

Create a pull request or raise an issue on the source for this page in GitHub