Configuring Container-to-Container Networking

This topic describes how to configure the Container-to-Container Networking feature, which allows direct network traffic between apps. For an overview of how Container-to-Container Networking works, see the Understanding Container-to-Container Networking topic.

Enable Container-to-Container Networking

Container networking is installed by default when you install Cloud Foundry using cf-deployment. For instructions for using cf-deployment, see Deploying Cloud Foundry with cf-deployment.

Container networking has properties that you can configure to change their default behavior. The table below has a list of properties and instructions for editing them. For more information, see Configuration Information for Operators.

Container-to-Container Networking Opsfiles Description

    - type: replace
    path: /instance_groups/name=diego-cell/jobs/name=vxlan_policy_agent/properties/iptables_logging?
    value: true
    
The default value for iptables_logging is false.

(Optional) Change the value to true to enable logging for Container-to-Container policy iptables rules.

    - type: replace
    path: /instance_groups/name=diego-cell/jobs/name=cni/properties/iptables_logging?
    value: true
    
The default value for iptables_logging is false.

(Optional) Change the value to true to enable logging for Application Security Group (ASG) iptables rules.

    - type: replace
    path: /instance_groups/name=diego-cell/jobs/name=silk-controller/properties/network?
    value: REPLACE-WITH-OVERLAY-NETWORK-CIDR
    
(Optional) Enter an IP range for the overlay network. The CIDR must specify an RFC 1918 range. If you do not set a custom range, the deployment uses 10.255.0.0/16.

See App Instance Communication for more information.

    - type: replace
    path: /instance_groups/name=diego-cell/jobs/name=cni/properties/mtu?
    value: REPLACE-WITH-MTU
    
(Optional) You can manually configure the Maximum Transmission Unit (MTU) value to support additional encapsulation overhead.

To see how container networking works with and without service discovery, see Cats and Dogs with Service Discovery in GitHub. In this tutorial, you deploy two apps and create a Container-to-Container Networking policy that allows them to communicate directly with each other.

Configure the Overlay Network

Container-to-Container Networking uses an overlay network to manage communication between app instances. By default, each Diego cell in the overlay network is allocated a /24 range that supports 254 containers per cell, one container for each of the usable IP addresses, .1 through .254. For more information about the overlay network, see Overlay Network in Understanding Container-to-Container Networking.

Configure the Number of Diego Cells

If you want to modify the number of Diego cells supported by the overlay network in your Cloud Foundry deployment, edit the cf_networking.network property in your cf-networking-release manifest, and then re-deploy Cloud Foundry. See the examples below:

Overlay subnet mask Number of cells Containers per cell
/20 63 254
/16 255 254
/12 4,095 254

Warning: The overlay network IP address range must not conflict with any other IP addresses in the network. If a conflict exists, Diego cells cannot reach any endpoint that has a conflicting IP address.

Configure the Number of Containers per Cell

If you want to modify the number of containers per Diego cell in your Cloud Foundry deployment, edit the cf_networking.subnet_prefix_length property in your cf-networking-release manifest, and then re-deploy Cloud Foundry. See the examples below:

Overlay subnet mask Number of cells Cell prefix length Containers per cell
/16 255 /24 254
/16 255 /26 62
/16 255 /28 14

Manage Logging for Container-to-Container Networking

This section describes how to configure logging for Container-to-Container Networking events by making requests to the running virtual machines (VMs). You can also enable logging for iptables policy rules by editing the manifest in Enable on an IaaS.

Change Log Level for Debugging

By default, the Policy Server logs events at the INFO level. You can capture more information about events by increasing the log level to DEBUG.

To change the log level, follow the steps below:

  1. SSH to either the Policy Server or the VXLAN Policy Agent.

    • Policy Server: SSH directly to the Policy Server VM.
    • VXLAN Policy Agent: SSH to the Diego cell that runs the VXLAN Policy Agent.
  2. To change the log level, run the following command:

    curl -X POST -d 'LOG-LEVEL' localhost:PORT-NUMBER/log-level
    

    The LOG-LEVEL is DEBUG or INFO. The PORT-NUMBER is 22222 unless you specified a different number when you edited the stub file in Enable on an IaaS above.

    The following command increases the log level to DEBUG:

    $ curl -X POST -d 'DEBUG' localhost:22222/log-level
    

    The following command decreases the log level to INFO:

    $ curl -X POST -d 'INFO' localhost:22222/log-level
    

  3. Find the logs in the following locations:

    • Policy Server: /var/vcap/sys/log/policy-server/policy-server.stdout.log
    • VXLAN Policy Agent: /var/vcap/sys/log/vxlan-policy-agent/vxlan-policy-agent.stdout.log

Enable Logging for Container-to-Container Networking Policies

By default, CF does not log iptables policy rules for Container-to-Container network traffic. You can enable logging for iptables policy rules in the manifest in Enable on an IaaS above, or follow the steps below:

  1. SSH to the Diego cell that runs the VXLAN Policy Agent.

  2. To change the log level, run the following command:

    curl -X PUT -d '{"enabled": BOOLEAN}' localhost:PORT-NUMBER/iptables-c2c-logging
    

    The BOOLEAN is true or false. The PORT-NUMBER is 22222 unless you specified a different number when you edited the stub file in Enable on an IaaS above.

    The following command enables logging for iptables policy rules:

    $ curl -X PUT -d '{"enabled": true}' localhost:22222/iptables-c2c-logging
    

    The following command disables logging for iptables policy rules:

    $ curl -X PUT -d '{"enabled": false}' localhost:22222/iptables-c2c-logging
    

  3. Find the logs in /var/log/kern.log.

Use Metrics to Consume Logs

You can stream Container-to-Container Networking component metrics with the Loggregator Firehose.

Container-to-Container Networking logs include the following prefixes:

  • netmon
  • vxlan_policy_agent
  • policy_server

Create and Manage Networking Policies

This section describes how to create and modify Container-to-Container Networking policies using the Cloud Foundry Command Line Interface (cf CLI). The cf CLI only supports configuring policies for apps within the same space. To configure policies for apps in different orgs and spaces, use the Policy Server External API.

Note: With the NSX-T integration, container networking policies and ASGs continue to work as normal. Advanced ASG logging is not supported with NSX-T.

Prerequisites

  • Ensure that you are using cf CLI v6.30 or higher:
    $ cf version
    
    For more information about updating the cf CLI, see the Installing the cf CLI topic.

Grant Permissions

CF admins use the following UAA scopes to grant specific users or groups permissions to configure network policies:

UAA Scope Suitable for… Allows users to create policies…
network.admin operators for any apps in the CF deployment
network.write space developers for apps in spaces that they can access

If you are a CF admin, you already have the network.admin scope. An admin can also grant the network.admin scope to a space developer.

For more information, see Creating and Managing Users with the UAA CLI (UAAC) and Orgs, Spaces, Roles, and Permissions.

To grant all Space Developers permissions to configure network policies, edit your BOSH manifest to include the cf_networking.enable_space_developer_self_service property and set that property to true.

Add a Network Policy

To add a policy that allows direct network traffic from one app to another, run the following command:

cf add-network-policy SOURCE_APP --destination-app DESTINATION_APP --protocol (tcp | udp) --port RANGE

Replace the placeholders in the above command as follows:

  • SOURCE_APP is the name of the app that sends traffic.
  • DESTINATION_APP is the name of the app that will receive traffic.
  • PROTOCOL is one of the following: tcp or udp.
  • RANGE are the ports at which to connect to the destination app. The allowed range is from 1 to 65535. You can specify a single port, such as 8080, or a range of ports, such as 8080-8090.

The following example command allows access from the frontend app to the backend app over TCP at port 8080:

$ cf add-network-policy frontend --destination-app backend --protocol tcp --port 8080
Adding network policy to app frontend in org my-org / space dev as admin...
OK

List Policies

You can list all the policies in your space, or just the policies for which a single app is the source:

  • To list the all the policies in your space, run cf network-policies.

    $ cf network-policies
    
  • To list the policies for an app, run cf network-policies --source MY-APP. Replace MY-APP with the name of your app.

    $ cf network-policies --source example-app
    

    The following example command lists policies for the app frontend:

    $ cf network-policies --source frontend
    Listing network policies in org my-org / space dev as admin...
    
    source      destination   protocol   ports
    frontend    backend       tcp        8080
    

Remove a Network Policy

To remove a policy that allows direct network traffic from an app, run the following command:

cf remove-network-policy SOURCE_APP --destination-app DESTINATION_APP --protocol PROTOCOL --port RANGE

Replace the placeholders in the above command to match an existing policy, as follows:

  • SOURCE_APP is the name of the app that sends traffic.
  • DESTINATION_APP is the name of the app that receives traffic.
  • PROTOCOL is either tcp or udp.
  • PORTS are the ports connecting the apps. The allowed range is from 1 to 65535. You can specify a single port, such as 8080, or a range of ports, such as 8080-8090.

The following command deletes the policy that allowed the frontend app to communicate with the backend app over TCP on port 8080:

$ cf remove-network-policy frontend --destination-app backend --protocol tcp --port 8080
Removing network policy to app frontend in org my-org / space dev as admin...
OK

App Service Discovery (Experimental)

When app service discovery is enabled, apps pushed to Cloud Foundry can establish container-to-container communications through a known route served by internal BOSH DNS. This allows front-end apps to easily connect with back-end apps.

To establish container-to-container communications between a front-end and back-end app, a developer:

  1. Launches a back-end app that publishes a local endpoint.
  2. Maps a named route to the endpoint.
  3. Creates a network policy that allows direct traffic from the front-end to the back-end app.
  4. Launches the front-end app.

See Cats and Dogs with Service Discovery in GitHub for an example, written in Go, that demonstrates communication between front-end and back-end apps.

Enable App Service Discovery

To enable app service discovery, include three ops files in your CF deployment, as described in the CF App Service Discovery Release on GitHub.

Create a pull request or raise an issue on the source for this page in GitHub