Administering Container-to-Container Networking

This topic describes how to configure the Container-to-Container Networking feature. For an overview of how Container-to-Container Networking works, see the Understanding Container-to-Container Networking topic.

Enable Container-to-Container Networking

This section presents two procedures for enabling Container-to-Container Networking. See the table below and choose the procedure that corresponds with your use case.

If your deployment runs on … Then follow …
An IaaS, such as AWS or vSphere Enable on an IaaS
BOSH-Lite, a Vagrant box for testing or development Enable on BOSH-Lite

Enable on an IaaS

If your Cloud Foundry (CF) deployment runs on an IaaS, follow these steps to enable Container-to-Container Networking.

  1. Ensure that you have a database ready. To complete this procedure, you need a v5.7 or later MySQL database, or a PostgreSQL database. You can use a database within your Cloud Controller database instance or create a different instance, such as Amazon RDS.
  2. Target your BOSH Director using the BOSH CLI:
    $ bosh target BOSH-DIRECTOR-IP 
  3. If you have not already, upload a stemcell that uses Linux kernel 4.4, such as stemcell 3263.2 or later:
    $ bosh upload stemcell URL-OF-STEMCELL 
  4. Open the CF properties stub that you created when deploying Cloud Foundry and do the following:
    1. Under properties > uaa > scim > users > name: admin > groups add a new group named network.admin.
      scim:
        users:
        - name: admin
          password: 
          groups:
            - scim.write
            - scim.read
            ...
            - routing.router_groups.write
            - network.admin 
    2. Under properties > uaa > clients > cf, to the line beginning scope: add network.admin.
    3. clients:
        cf:
          scope: cloud_controller.read, [...] routing.router_groups.read,network.admin
    4. Under clients, add a network-policy client as follows and replace REPLACE-WITH-UAA-CLIENT-SECRET with a secure password of your choosing.
      clients:
        cf:
          scope: cloud_controller.read, [...] routing.router_groups.read,network.admin
        network-policy:
          authorities: uaa.resource,cloud_controller.admin_read_only
          authorized-grant-types: client_credentials,refresh_token
          secret: REPLACE-WITH-UAA-CLIENT-SECRET
  5. Create a Container-to-Container Networking stub file stubs/cf-networking/stub.yml and copy in the template below:
    ---
    cf_networking_overrides:
      releases:
      - name: cf-networking
        version: latest
      driver_templates:
      - name: garden-cni
        release: cf-networking
      - name: cni-flannel
        release: cf-networking
      - name: netmon
        release: cf-networking
      - name: vxlan-policy-agent
        release: cf-networking
      properties:
        cf_networking:
          vxlan_policy_agent:
            policy_server_url: https://policy-server.service.cf.internal:4003
            ca_cert: |
              -----BEGIN CERTIFICATE-----
              REPLACE-WITH-CA-CERTIFICATE
              -----END CERTIFICATE-----
            client_cert: |
              -----BEGIN CERTIFICATE-----
              REPLACE-WITH-CLIENT-CERTIFICATE
              -----END CERTIFICATE-----
            client_key: |
              -----BEGIN RSA PRIVATE KEY-----
              REPLACE-WITH-CLIENT-KEY
              -----END RSA PRIVATE KEY-----
          garden_external_networker:
            cni_plugin_dir: /var/vcap/packages/flannel/bin
            cni_config_dir: /var/vcap/jobs/cni-flannel/config/cni
          plugin:
            etcd_endpoints:
              - (( config_from_cf.etcd.advertise_urls_dns_suffix ))
            etcd_client_cert: (( config_from_cf.etcd.client_cert ))
            etcd_client_key: (( config_from_cf.etcd.client_key ))
            etcd_ca_cert: (( config_from_cf.etcd.ca_cert ))
          policy_server:
            database:
              type: REPLACE-WITH-DB-TYPE
              username: REPLACE-WITH-USERNAME
              password: REPLACE-WITH-PASSWORD
              host: REPLACE-WITH-DB-HOSTNAME
              port: REPLACE-WITH-DB-PORT
              name: REPLACE-WITH-DB-NAME
            skip_ssl_validation: true
            uaa_client_secret: REPLACE-WITH-UAA-CLIENT-SECRET
            uaa_url: (( "https://uaa." config_from_cf.system_domain ))
            ca_cert: |
              -----BEGIN CERTIFICATE-----
              REPLACE-WITH-CA-CERTIFICATE
              -----END CERTIFICATE-----
            server_cert: |
              -----BEGIN CERTIFICATE-----
              REPLACE-WITH-SERVER-CERT
              -----END CERTIFICATE-----
            server_key: |
              -----BEGIN RSA PRIVATE KEY-----
              REPLACE-WITH-SERVER-KEY
              -----END RSA PRIVATE KEY-----
      garden_properties:
        network_plugin: /var/vcap/packages/runc-cni/bin/garden-external-networker
        network_plugin_extra_args:
        - --configFile=/var/vcap/jobs/garden-cni/config/adapter.json
      jobs:
      - name: policy-server
        instances: 1
        persistent_disk: 256
        templates:
        - name: policy-server
          release: cf-networking
        - name: route_registrar
          release: cf
        - name: consul_agent
          release: cf
        - name: metron_agent
          release: cf
        resource_pool: database_z1
        networks:
          - name: diego1
        properties:
          nats:
            machines: (( config_from_cf.nats.machines ))
            user: (( config_from_cf.nats.user ))
            password: (( config_from_cf.nats.password ))
            port: (( config_from_cf.nats.port ))
          metron_agent:
            zone: z1
          route_registrar:
            routes:
            - name: policy-server
              port: 4002
              registration_interval: 20s
              uris:
              - (( "api." config_from_cf.system_domain "/networking" ))
          consul:
            agent:
              services:
                policy-server:
                  name: policy-server
                  check:
                    interval: 5s
                    script: /bin/true
    
    config_from_cf: (( merge ))
    
  6. Edit the stub file using the table below as a guide:

    Note: For a test environment, you can use a script such as the one in the Container-to-Container Networking Release repository to generate the certificates and keys required for the stub. For a production deployment, use certificates signed by a Certificate Authority (CA).

    Container-to-Container Networking Stub Contents Editing Instructions
    
    properties:
      cf_networking:
        vxlan_policy_agent:
          policy_server_url: [...]
          ca_cert: |
            -----BEGIN CERTIFICATE-----
            REPLACE-WITH-CA-CERTIFICATE
            -----END CERTIFICATE-----
          client_cert: |
            -----BEGIN CERTIFICATE-----
            REPLACE-WITH-CLIENT-CERTIFICATE
            -----END CERTIFICATE-----
          client_key: |
            -----BEGIN RSA PRIVATE KEY-----
            REPLACE-WITH-CLIENT-KEY
            -----END RSA PRIVATE KEY-----
    Copy in certificates and keys for the policy agent. The policy agent communicates with the policy server through TLS. See the Architecture section for more information.
    
    policy_server:
      debug_server_port: REPLACE-WITH-LISTEN-PORT
    
    vxlan_policy_agent:
      debug_server_port: REPLACE-WITH-LISTEN-PORT
    
    By default, the Policy Server and VXLAN Policy Agent listen on port 22222.

    (Optional) Change these port numbers by adding the debug_server_port key pair to the stub file.
    Replace REPLACE-WITH-LISTEN-PORT with a port number.

    For more information, see Manage Debug Logging below.
    
    vxlan_policy_agent:
      iptables_c2c_logging: true
    
    The default value for iptables_c2c_logging is false.

    (Optional) Change the value to true to enable logging for Container-to-Container policy iptables rules.
    
    garden_external_networker:
      iptables_asg_logging: true
        
    The default value for iptables_asg_logging is false.

    (Optional) Change the value to true to enable logging for Application Security Group (ASG) iptables rules.
    
    policy_server:
      uaa_client_secret: REPLACE-WITH-UAA-CLIENT-SECRET
        
    Copy in the REPLACE-WITH-UAA-CLIENT-SECRET value you used in the step above.
    
    database:
      type: REPLACE-WITH-DB-TYPE
      username: REPLACE-WITH-USERNAME
      password: REPLACE-WITH-PASSWORD
      host: REPLACE-WITH-DB-HOSTNAME
      port: REPLACE-WITH-DB-PORT
      name: REPLACE-WITH-DB-NAME
       
    Supply the details for the database from step 1.
    The database type must be postgres or mysql.
    Choose a username and password.
    For host, enter the IP address of the database instance.
    Supply a port. For MySQL, a typical port is 3360.
    Supply the name of the database.
    
      ca_cert: |
        -----BEGIN CERTIFICATE-----
        REPLACE-WITH-CA-CERTIFICATE
        -----END CERTIFICATE-----
      server_cert: |
        -----BEGIN CERTIFICATE-----
        REPLACE-WITH-SERVER-CERT
        -----END CERTIFICATE-----
      server_key: |
        -----BEGIN RSA PRIVATE KEY-----
        REPLACE-WITH-SERVER-KEY
        -----END RSA PRIVATE KEY-----
    garden_external_networker:
      cni_plugin_dir: /var/vcap/packages/flannel/bin
      cni_config_dir: /var/vcap/jobs/cni-flannel/config/cni
        
    Copy in the certificates and keys for the policy server. The policy server communicates with the policy agent through TLS. See the Architecture section for more information.
    
    properties:
      cf_networking:
        network: REPLACE-WITH-OVERLAY-NETWORK-CIDR
        
    (Optional) Enter an IP range for the overlay network. The CIDR must specify an RFC 1918 range. If you do not set a custom range, the deployment uses 10.255.0.0/16.

    See App Instance Communication for more information.
    
    properties:
      cf_networking:
        mtu: REPLACE-WITH-MTU
        
    (Optional) You can manually configure the Maximum Transmission Unit (MTU) value to support additional encapsulation overhead.
  7. Create a file that contains the following bash script. Name the file generate_diego.sh.
    set -e -x -u
    
    environment_path=STUBS-DIRECTORY
    output_path=MANIFEST-DIRECTORY
    diego_release_path=LOCAL-DIEGO-REPO
    
    pushd cf-release
      ./scripts/generate_deployment_manifest aws \
        ${environment_path}/stubs/director-uuid.yml \
        ${diego_release_path}/examples/aws/stubs/cf/diego.yml \
        ${environment_path}/stubs/cf/properties.yml \
        ${environment_path}/stubs/cf/instance-count-overrides.yml \
        ${environment_path}/stubs/cf/stub.yml \
        > ${output_path}/cf.yml
    popd
    
    pushd diego-release
      ./scripts/generate-deployment-manifest \
        -g \
        -c ${output_path}/cf.yml \
        -i ${environment_path}/stubs/diego/iaas-settings.yml \
        -p ${environment_path}/stubs/diego/property-overrides.yml \
        -n ${environment_path}/stubs/diego/instance-count-overrides.yml \
        -N ${environment_path}/stubs/cf-networking/stub.yml \
        -v ${environment_path}/stubs/diego/release-versions.yml \
        > ${output_path}/diego.yml
    popd
    
    Replace the variables as follows:
    • STUBS-DIRECTORY: The directory containing your stubs for CF, Diego, and Container-to-Container Networking.
    • MANIFEST-DIRECTORY: The directory where you want the manifest created.
    • LOCAL-DIEGO-REPO: The directory of the local copy of the diego-release repository.
  8. Enter the following commands to make the script executable and run the script.
    $ chmod u+x generate_diego.sh
    $ ./generate_diego.sh
    
  9. Enter the following command to target your BOSH director:
    $ bosh target BOSH-DIRECTOR-IP
    
    For example,
    $ bosh target 192.0.2.1
  10. Enter the following command to set your CF deployment to the manifest you generated.
    $ bosh deployment ${output_path}/cf.yml
    
  11. Enter the following command to deploy CF.
    $ bosh deploy
  12. Enter the following command to set your Diego deployment to the manifest you generated.
    $ bosh deployment ${output_path}/diego.yml
    
  13. Enter the following command to deploy Diego.
    $ bosh deploy
  14. (Optional) Try the Cats and Dogs example in the Container-to-Container Networking Release repository. In this tutorial, you deploy two apps and create a Container-to-Container Networking policy that allows them to communicate directly with each other.

Enable on BOSH-Lite

If your CF deployment runs on BOSH-Lite, follow these steps to enable Container-to-Container Networking.

  1. Ensure your BOSH-Lite version is 9000.131.0 or later. If you need to upgrade, follow the instructions for Upgrading the BOSH-Lite VM.
  2. Navigate to your bosh-lite directory, for example,
    $ cd ~/workspace/bosh-lite 
  3. To enable bridge-netfilter on the VM running BOSH-Lite, run the following command:
    $ vagrant ssh -c 'sudo modprobe br_netfilter'

    Container-to-Container Networking on BOSH-Lite requires this Linux kernel feature to enforce network policy.
  4. Upload the latest BOSH-Lite stemcell:
    $ bosh upload stemcell https://bosh.io/d/stemcells/bosh-warden-boshlite-ubuntu-trusty-go_agent
  5. To clone the required CF release repos to your workspace, enter the following commands:
    $ git clone https://github.com/cloudfoundry/diego-release
    $ git clone https://github.com/cloudfoundry/cf-release
    $ git clone https://github.com/cloudfoundry-incubator/cf-networking-release
  6. To enable Container-to-Container Networking on BOSH-Lite, navigate to the cf-networking-release directory and run the deploy script:
    $ cd ~/workspace/cf-networking-release
    $ ./scripts/deploy-to-bosh-lite
  7. (Optional) Try the Cats and Dogs example in the Container-to-Container Networking Release repository. In this tutorial, you deploy two apps and create a Container-to-Container Networking policy that allows them to communicate directly with each other.

Manage Logging for Container-to-Container Networking

This section describes how to configure logging for Container-to-Container Networking events by making requests to the running virtual machines (VMs). You can also enable logging for iptables policy rules by editing the manifest in Enable on an IaaS.

Change Log Level for Debugging

By default, the Policy Server logs events at the INFO level. You can capture more information about events by increasing the log level to DEBUG.

To change the log level, follow the steps below:

  1. SSH to either the Policy Server or the VXLAN Policy Agent.

    • Policy Server: SSH directly to the Policy Server VM.
    • VXLAN Policy Agent: SSH to the Diego cell that runs the VXLAN Policy Agent.
  2. To change the log level, run the following command:

    curl -X POST -d 'LOG-LEVEL' localhost:PORT-NUMBER/log-level
    

    The LOG-LEVEL is DEBUG or INFO. The PORT-NUMBER is 22222 unless you specified a different number when you edited the stub file in Enable on an IaaS above.

    The following command increases the log level to DEBUG:

    $ curl -X POST -d 'DEBUG' localhost:22222/log-level
    

    The following command decreases the log level to INFO:

    $ curl -X POST -d 'INFO' localhost:22222/log-level
    

  3. Find the logs in the following locations:

    • Policy Server: /var/vcap/sys/log/policy-server/policy-server.stdout.log
    • VXLAN Policy Agent: /var/vcap/sys/log/vxlan-policy-agent/vxlan-policy-agent.stdout.log

Enable Logging for Container-to-Container Networking Policies

By default, CF does not log iptables policy rules for Container-to-Container network traffic. You can enable logging for iptables policy rules in the manifest in Enable on an IaaS above, or follow the steps below:

  1. SSH to the Diego cell that runs the VXLAN Policy Agent.

  2. To change the log level, run the following command:

    curl -X PUT -d '{"enabled": BOOLEAN}' localhost:PORT-NUMBER/iptables-c2c-logging
    

    The BOOLEAN is true or false. The PORT-NUMBER is 22222 unless you specified a different number when you edited the stub file in Enable on an IaaS above.

    The following command enables logging for iptables policy rules:

    $ curl -X PUT -d '{"enabled": true}' localhost:22222/iptables-c2c-logging
    

    The following command disables logging for iptables policy rules:

    $ curl -X PUT -d '{"enabled": false}' localhost:22222/iptables-c2c-logging
    

  3. Find the logs in /var/log/kern.log.

Use Metrics to Consume Logs

You can stream Container-to-Container Networking component metrics with the Loggregator Firehose.

Container-to-Container Networking logs include the following prefixes:

  • netmon
  • vxlan_policy_agent
  • policy_server

Create Policies for Container-to-Container Networking

This section describes how to create and modify Container-to-Container Networking policies using a plugin for the Cloud Foundry Command Line Interface (cf CLI).

To use the plugin, you must have either the network.write or network.admin UAA scope.

UAA Scope Suitable for… Allows users to create policies…
network.admin operators for any apps in the CF deployment
network.write space developers for apps in spaces that they can access

If you are a CF admin, you already have the network.admin scope. An admin can also grant the network.admin scope to a space developer.

For more information, see Creating and Managing Users with the UAA CLI (UAAC) and Orgs, Spaces, Roles, and Permissions.

Install the Plugin

Follow these steps to download and install the Network Policy plugin for the cf CLI:

  1. Download the network-policy-plugin for your operating system from the Container-to-Container Networking Release repository.

  2. To change the permissions of the plugin file and complete the installation, enter the following commands:

    $ chmod +x ~/Downloads/network-policy-plugin
    $ cf install-plugin ~/Downloads/network-policy-plugin
    

Create a Policy

To create a policy that allows direct network traffic from one app to another, enter the following command:

$ cf allow-access SOURCE-APP DESTINATION-APP --protocol PROTOCOL --port PORT

Replace the placeholders in the above command as follows:

  • SOURCE-APP is the name of the app that will send traffic.
  • DESTINATION-APP is the name of the app that will receive traffic.
  • PROTOCOL is one of the following: tcp or udp.
  • PORT is the port at which to connect to the destination app. The allowed range is from 1 to 65535.

The following example command allows access from the frontend app to the backend app over TCP at port 8080:

$ cf allow-access frontend backend --protocol tcp --port 8080
Allowing traffic from frontend to backend as admin...
OK 

List Policies

You can list all the policies in your deployment or just the policies for which a single app is either the source or the destination:

  • To list the all the policies in your deployment, enter the following command:

    $ cf list-access
    
  • To list the policies for an app, enter the following command:

    $ cf list-access --app MY-APP
    

    The following example command lists policies for the app frontend:

    $ cf list-access --app frontend
    Listing policies as admin...
    OK
    
    Source    Destination    Protocol    Port
    frontend  backend        tcp         8080
    

Delete a Policy

To delete a policy that allows direct network traffic from one app to another, enter the following command:

$ cf remove-access SOURCE-APP DESTINATION-APP --protocol PROTOCOL --port PORT

The following command deletes the policy that allowed the frontend app to communicate with the backend app over TCP on port 8080:

$ cf remove-access frontend backend --protocol tcp --port 8080
Denying traffic from frontend to backend as admin...
OK 

Create a pull request or raise an issue on the source for this page in GitHub