Troubleshooting TCP routing in Cloud Foundry

Page last updated:

Are your TCP routing issues related to DNS and load balancer misconfiguration, the TCP routing tier, or the routing subsystem? Use the following steps to find out.

Rule out the app

If you are have TCP routing issues with an app, the following procedure determines what to troubleshoot:

Note If you have mutual TLS app identity verification enabled, you must also enable support for TLS backends with TCP Router via the tcp_router.backend_tls.enabled property, and on Route Emitter via the tcp.enable_tls property..

Prerequisites

This procedure requires that you have the following:

• An app with TCP routing issues.
• A TCP domain. See Routes and Domains for more information about creating a TCP domain.
• A simple HTTP web app that you can use to curl.

Procedure

1. Push a simple HTTP app using your TCP domain (cf CLI v6 only) by entering the following command:

   $ cf push MY-APP -d tcp.MY-DOMAIN --random-route
   

For cf CLI v7 and later, the -d flag is not supported. Domain is set in the manifest, in the routes property.

  $ cf push MY-APP --random-route
  

1. Curl your app on the port generated for the route. For example, if the port is 1024:

   $ curl tcp.MY-DOMAIN:1024
   

2. If the curl request fails to reach the app, proceed to the next section: Rule Out DNS and the Load Balancer.

3. If the curl request to your simple app succeeds, curl the app you are having issues with.

4. If you cannot successfully curl your problem app, TCP routing is working correctly. There is an issue with the app you cannot successfully curl.

Rule out DNS and the load balancer

1. Curl the TCP router healthcheck endpoint:

   $ curl tcp.MY-DOMAIN:80/health -v
   

2. If you receive a 200 OK response, proceed to the next section: Rule Out the Routing Subsystem.

3. If you do not receive a 200 OK, your load balancer might not be configured to pass through the healthcheck port. Continue following this procedure to test your load balancer configuration.

4. Confirm that your TCP domain name resolves to your load balancer:

   $ dig tcp.MY-DOMAIN
   ...
   tcp.MY-DOMAIN. 300 IN  A 123.456.789.123
   

5. As an admin user, list the reservable ports configured for the default-tcp router group:

   $ cf curl /routing/v1/router_groups
   [
     {
       "guid": "d9b1db52-ea78-4bb9-7473-ec8e5d411b14",
       "name": "default-tcp",
       "type": "tcp",
       "reservable_ports": "1024-1123"
     }
   ]
   

6. Choose a port from the reservable_ports range and curl the TCP domain on this port. For example, if you chose port 1024:

   $ curl tcp.MY-DOMAIN:1024 -v
   

7. If you receive an immediate rejection, then the TCP router likely rejected the request because there is no route for this port.

8. If your connection times out, then you need to configure your load balancer to route all ports in reservable_ports to the TCP routers.

Rule out the routing subsystem

Send a direct request to the TCP router to confirm that it routes to your app.

1. SSH into your TCP router.

2. Curl the port of your route using the IP address of the router itself. For example, if the port reserved for your route is 1024, and the IP address of the TCP router is 10.0.16.18:

   $ curl 10.0.16.18:1024
   

3. If the curl is successful, then the load balancer either:

   • cannot reach the TCP routers, or
   • is not configured to route requests to the TCP routers from the reservable_ports

4. If you cannot reach the app by curling the TCP router directly, perform the following steps to confirm that your TCP route is in the routing table.

   1. Obtain the secret for your tcp_emitter OAuth client from your manifest. This OAuth client has permissions to read the routing table.

   2. Install the UAA CLI uaac:

      $ gem install cf-uaac
      

   3. Obtain a token for this OAuth client from UAA by providing the client secret:

      $ uaac token client get tcp_emitter
      Client secret:
      

   4. Obtain an access_token:

      $ uaac context
      

   5. Use the routing API to list TCP routes:

      $ curl api.MY-DOMAIN/routing/v1/tcp_routes -H "Authorization: bearer TOKEN"
      [{"router_group_guid":"f3518f7d-d8a0-4279-4e89-c058040d0000",
      "backend_port":60000,"backend_ip":"10.244.00.0","port":60000,"modification_tag":{"guid":"d4cc3bbe-c838-4857-7360-19f034440000",
      "index":1},"ttl":120}]
      

   6. In this output, each route mapping has the following:

      • port: your route port
      • backend_ip: an app instance mapped to the route
      • backend_port: the port number for the app instance mapped to the route

   7. If your route port is not in the response, then the tcp_emitter might be unable to register TCP routes with the routing API. Look at the logs for tcp_emitter to see if there are any errors or failures.

   8. If the route is in the response, but you were not able to curl the port on the TCP route directly, then the TCP router might be unable to reach the routing API. Look at the logs for tcp_router to see if there are any errors or failures.