HTTP Routing

Page last updated:

This topic describes how the Gorouter, the main component in Cloud Foundry’s routing tier, routes HTTP traffic within Cloud Foundry (CF).

HTTP Headers

HTTP traffic passed from the Gorouter to an app includes the following HTTP headers:

  • X-Forwarded-Proto gives the scheme of the HTTP request from the client. The scheme is HTTP if the client made an insecure request or HTTPS if the client made a secure request. Developers can configure their apps to reject insecure requests by inspecting the HTTP headers of incoming traffic and rejecting traffic that includes X-Forwarded-Proto with the scheme of HTTP.

  • X-Forwarded-For gives the IP address of the client originating the request.

If your load balancer terminates TLS on the client side of the Gorouter, it must append these headers to requests forwarded to the Gorouter. For more information, see the Securing Traffic into Cloud Foundry topic.

HTTP Headers for Zipkin Tracing

Zipkin is a tracing system that enables app developers to troubleshoot failures or latency issues. Zipkin provides the ability to trace requests and responses across distributed systems. See Zipkin.io for more information.

When the Zipkin feature is enabled in Cloud Foundry, the Gorouter examines the HTTP request headers and performs the following:

  • If the X-B3-TraceId and X-B3-SpanId HTTP headers are not present in the request, the Gorouter generates values for these and inserts the headers into the request forwarded to an application. These values are also found in the Gorouter access log message for the request: x_b3_traceid and x_b3_spanid.
  • If both X-B3-TraceId and X-B3-SpanId HTTP headers are present in the request, the Gorouter forwards the same value for X-B3-TraceId, generates a new value for X-B3-SpanId, and adds the X-B3-ParentSpan header, and sets to the value of the span id in the request. In addition to these trace and span ids, the Gorouter access log message for the request includes x_b3_parentspanid.

Developers can then add Zipkin trace IDs to their application logging in order to trace app requests and responses in Cloud Foundry.

After adding Zipkin HTTP headers to app logs, developers can use cf logs myapp to correlate the trace and span ids logged by the Gorouter with the trace ids logged by their app. To correlate trace IDs for a request through multiple apps, each app must forward appropriate values for the headers with requests to other applications.

HTTP Headers for App Instance Routing

Developers who want to obtain debug data for a specific instance of an app can use the HTTP header X-CF-APP-INSTANCE to make a request to an app instance.

Perform the following steps to make an HTTP request to a specific app instance:

  1. Obtain the GUID of your app:
    $ cf app YOUR-APP --guid
  2. List your app instances and retrieve the index number of the instance you want to debug:
    $ cf app YOUR-APP
  3. Make a request to the app route using the HTTP header X-CF-APP-INSTANCE set to the concatenated values of the app GUID and the instance index:
    $ curl app.example.com -H "X-CF-APP-INSTANCE":"YOUR-APP-GUID:YOUR-INSTANCE-INDEX"

Forward Client Certificate to Applications

Applications that require mutual TLS (mTLS) need metadata from client certificates to authorize requests. Cloud Foundry supports this use case without bypassing layer-7 load balancers and the Gorouter.

The HTTP header X-Forwarded-Client-Cert (XFCC) may be used to pass the originating client certificate along the data path to the application. Each component in the data path must trust that the back-end component has not allowed the header to be tampered with.

If you configure the load balancer to terminate TLS and set the XFCC header from the received client certificate, then you must also configure the load balancer to strip this header if it is present in client requests. This configuration is required to prevent spoofing of the client certificate.

The following sections describe supported deployment configurations.

Terminating TLS for the First Time at the Load Balancer

By default, CF forwards arbitrary headers that are not otherwise mentioned in the docs, and a load balancer can be configured to put the certificate of the originating client, received during the mutual TLS handshake, into an HTTP header that the load balancer forwards upstream. We recommend the header XFCC for this use case, as this header is used in other configuration modes described below. The value of the header should be the base64-encoded bytes of the certificate; equivalent to a PEM file with newlines, headers, and footers removed.

This mode is enabled when router.forwarded_client_cert: always_forward.

Alternatively, the operator can configure the Gorouter to forward the XFCC header set by the load balancer only when the connection with the load balancer is mutual TLS. The client certificate received by the Gorouter in the mutual TLS handshake will not be forwarded in the header.

This mode is enabled when router.forwarded_client_cert: forward.

Terminating TLS for the First Time at the Gorouter

If the Gorouter is the first component to terminate TLS, such that it receives the certificate of the originating client in the mutual TLS handshake, the operator should configure the Gorouter to strip any instances of the XFCC header from client requests, set the value of the header to the base64-encoded bytes of the client certificate received in the mutual handshake, and forward the header upstream to the application.

This mode is enabled when router.forwarded_client_cert: sanitize_set.

If TLS is terminated for the first time at Gorouter, the Gorouter must be configured to trust the root certificate authority used to sign the Diego intermediate certificate authority, which in turn is used to sign certificates generated for each Diego container. This trust enables mutual authentication between applications that are running on Cloud Foundry. Operators should configure the router.ca_certs property for the Gorouter job with the root certificate authority in their BOSH deployment manifest.

    jobs:
      properties:
        router:
          ca_certs:
            -----BEGIN CERTIFICATE-----
            (contents of certificate)
            -----END CERTIFICATE-----

The router.ca_certs property is a string of concatenated certificate authorities in PEM format.

Client-Side TLS

Depending on your needs, you can configure your deployment to terminate TLS at the Gorouter, at the Gorouter and the load balancer, or at the load balancer only. For more information, see the Securing Traffic into Cloud Foundry topic.

TLS to Apps and Other Back-End Services

The Gorouter supports TLS and mutual authentication to back-end destinations, including app instances, platform services, and any other routable endpoints.

This is enabled by configuring the following manifest properties:

  • router.backends.enable_tls: true
  • router.ca_certs: must include the CA certificate used to sign any back end to which the Gorouter initiates a TLS handshake.

The Gorouter does not automatically initiate TLS handshakes with back-end destinations. To register that the Gorouter must initiate a TLS handshake before forwarding HTTP requests to a component, the component must include the following optional fields in its route.register message to NATS:

  • "tls_port": The port to which the Gorouter should open a connection and initiate a TLS handshake
  • "server_cert_domain_san": An identifier for the back end which the Gorouter expects to find in the Domain Subject Alternative Name of the certificate presented by the back end. This is used to prevent misrouting for back ends whose IPs and/or ports are expected to change; see Consistency.

To enable TLS from the Gorouter to application instances, some additional configuration is required in addition to the properties above; see Configuring Consistency with TLS.

Authors of routable components can configure Route Registrar to send the necessary NATS message automatically.

To enable mutual authentication between the Gorouter and back ends, operators configure the Gorouter with a certificate and private key using the following manifest properties:

  • router.backends.cert_chain
  • router.backends.private_key

Gorouter presents the certificate if requested by the back end during the TLS handshake.

Preventing Misrouting

As CF manages and balances apps, the internal IP address and ports for app instances change. To keep the Gorouter’s routing tables current, a Route Emitter on each Diego cell sends a periodic update to all Gorouters via NATS, reminding them of the location of all app instances; the default frequency of these updates is 20 seconds. The Gorouter tracks a time-to-live (TTL) for each route to back-end mapping; this TTL defaults to 120 seconds and is reset when the Gorouter receives an updated registration message.

Network partitions or NATS failures can cause the Gorouter’s routing table to fall out of sync, as CF continues to re-create containers across hosts to keep apps running. This can lead to routing of requests to incorrect destinations.

You can configure the Gorouter to deal with this problem in two ways:

Consistency mode Security of traffic between Gorouter and Containers Gorouter Route Pruning
With TLS Enabled Encrypted via TLS Routes are pruned on failure of TLS handshake only. See Gorouter TLS pruning behavior.
Without TLS Enabled Plain text Routes are pruned on TTL expiry

Both modes are described below.

With TLS Enabled

This mode is newer and has the following benefits:

  • Improved availability for applications by keeping routes in the Gorouters routing table when TTL expires
  • Increased guarantees against misrouting, by validating the identity of back ends before forwarding requests
  • Increased security by encrypting data in flight from the Gorouter to back ends

In this mode, the Diego Route Emitters send a modified route registration message to NATS that includes a unique identifier for the app instance, as well instructions to use TLS when communicating with the instance. See TLS to Apps and Other Back-End Services for details.

Before forwarding traffic to an app instance, the Gorouter initiates a TLS handshake with an Envoy proxy that runs in each app container and listens on port 8443. In the TLS handshake, the Envoy proxy presents a certificate generated by Diego for each container which uniquely identifies the container using the same app instance identifier sent by the Router Emitter, configured in the certificate as a domain Subject Alternative Name (SAN).

If the Gorouter confirms that the app instance identifier in the certificate matches the one received in the route registration message, the Gorouter forwards the HTTP request over the TLS session. If the instance identifiers do not match, the Gorouter transparently retries another instance of the app. If the TTL of the failed back end has expired, Gorouter prunes the route to it from its routing table.

The following table explains how the Gorouter keeps routes in its routing table after TTL has expired, as long as TLS handshakes continue to succeed.

  TLS handshake succeeds AND app instance identity valid TLS handshake fails OR app instance identity invalid
TTL has expired Request forwarded Route is pruned; transparent retry of another instance
TTL has not expired Request forwarded Route is not pruned but ineligible for requests for 30 seconds; transparent retry of another instance

Configure Validation of App Instance Identity with TLS

To enable validation of app instance identity, follow these steps:

  1. Configure the Gorouter to initiate a TLS handshake to back ends. See instructions at TLS to Apps and Other Back-End Services. Note: router.ca_certs should include the CA certificate used to sign the certificate configured in diego.executor.instance_identity_ca_cert.
  2. Configure Diego to run a proxy in each container with the following manifest property:
    • containers.proxy.enabled: true: This proxy will be used to terminate TLS for requests from Gorouter and present the App Instance certificate in the handshake.

      You can configure the above properties by using this ops file with your cf-deployment manifest.
  3. Configure Diego to generate app instance identity credentials with the following manifest properties:
    • diego.executor.instance_identity_ca_cert: the signing certificate for the Diego cell rep
    • diego.executor.instance_identity_key: the private key for the Diego cell rep

      You can configure the above properties by using this ops file with your cf-deployment manifest. You may have already configured these properties to support apps retrieving service binding credentials from runtime CredHub. See the Instance Identity notes in diego-release for details.

Without TLS Enabled

In this mode, the Diego Route Emitters on each cell send route registration messages that include instructions for the Gorouter to send unencrypted requests to the app instance. If the Gorouter does not receive an update for the route within the time-to-live (TTL) interval, the route is pruned from the Gorouter’s routing table. See TLS to Apps and Other Back-End Services for details.

This pruning method was developed before the alternative was available.

Consistency Behavior can vary by Instance Group

The Gorouter can validate app instance identity using TLS only when Diego Cells are configured appropriately. As this configuration is scoped to an instance group of Cells, some Cells (such as in an Isolation Segment) can be configured to receive TLS requests from the Gorouter and others plain text without validation of instance identity. This behavior can be

Validation of app instance identity using TLS is supported for Linux Cells only. Support for Windows Cells may be added later. The Gorouter forwards requests to Windows apps over plain text and prunes based on route TTL.

Transparent Retries

If the Gorouter cannot establish a TCP connection with a selected application instance, the Gorouter considers the instance ineligible for requests for 30 seconds, and the Gorouter transparently attempts to connect to another application instance. Once the Gorouter has established a TCP connection with an application instance, the Gorouter forwards the HTTP request.

See the Round-Robin Load Balancing section below for more information about how the Gorouter forwards requests to application instances.

Round-Robin Load Balancing

The Gorouter uses the round-robin algorithm for load balancing incoming requests to application instances. The Gorouter maintains a dynamically updated list of application instances for each route, and forwards each request for a given route to the next application instance in the list.

WebSockets

WebSockets is a protocol providing bi-directional communication over a single, long-lived TCP connection, commonly implemented by web clients and servers. WebSockets are initiated through HTTP as an upgrade request. The Gorouter supports this upgrade handshake, and holds the TCP connection open with the selected application instance. To support WebSockets, the operator must configure the load balancer correctly. Depending on the configuration, clients may have to use a different port for WebSocket connections, such as port 4443, or a different domain name. For more information, see the Supporting WebSockets topic.

Session Affinity

The Gorouter supports session affinity, or sticky sessions, for incoming HTTP requests to compatible apps.

With sticky sessions, when multiple instances of an app are running on CF, requests from a particular client always reach the same app instance. This allows apps to store session data specific to a user session.

  • To support sticky sessions, configure your app to return a JSESSIONID cookie in responses. The app generates a JSESSIONID as a long hash in the following format:

    1A530637289A03B07199A44E8D531427
    

  • If an app returns a JSESSIONID cookie to a client request, the CF routing tier generates a unique VCAP_ID for the app instance based on its GUID in the following format:

    323f211e-fea3-4161-9bd1-615392327913
    

  • On subsequent requests, the client must provide both the JSESSIONID and VCAP_ID cookies.

The CF routing tier uses the VCAP_ID cookie to forward client requests to the same app instance every time. The JSESSIONID cookie is forwarded to the app instance to enable session continuity. If the app instance identified by the VCAP_ID crashes, the Gorouter attempts to route the request to a different instance of the app. If the Gorouter finds a healthy instance of the app, it initiates a new sticky session.

Note: CF does not persist or replicate HTTP session data across app instances. If an app instance crashes or is stopped, session data for that instance is lost. If you require session data to persist across crashed or stopped instances, or to be shared by all instances of an app, store session data in a CF marketplace service that offers data persistence.

Keepalive Connections

From Frontend Clients

The Gorouter supports keepalive connections from clients; it does not close the TCP connection with clients immediately after returning an HTTP response. It is up to clients to close these connections.

To Back-End Servers

By default, the Gorouter closes the TCP connection with an app instance or system component after receiving an HTTP response.

When Keepalive Connections is enabled, the Gorouter maintains established TCP connections to back ends, with a configurable maximum that limits the number of total idle connections from each Gorouter instance. The Gorouter reuses idle connections for routing of subsequent requests if one exists for the selected back end, lowering latency and increasing throughput. If no idle connection exists, the Gorouter establishes a new connection and maintains the connection unless it has reached its idle connections limit. The number of idle connections from each Gorouter instance to an individual back end is limited to 100. Operators should see Router Idle Keepalive Connections for details.

Create a pull request or raise an issue on the source for this page in GitHub