Streaming App Logs with Fluentd

Page last updated:

Fluentd is an open source log collector that allows you to implement unified logging layers. With Fluentd, you can stream app logs to different backends or services like Elasticsearch, HDFS and Amazon S3. This topic explains how to integrate Fluentd with Cloud Foundry apps.

Step 1: Create a Cloud Foundry Syslog Drain for Fluentd

  1. In Cloud Foundry, create a syslog drain user-provided service instance as described in Using Third-Party Log Management Services.

  2. Choose one or more apps whose logs you want to drain to Fluentd through the service.

  3. Bind each app to the service instance, and restart the app.

  4. Note the GUID for each app, the IP address of the Loggregator host, and the port number for the service.

  5. Locate the port number in the syslog URL. For example:

    syslog://logs.example.com:5140

Step 2: Set up Fluentd for Cloud Foundry

This section assumes you have an active Fluentd instance running. If you do not have an active Fluentd instance, refer to the Fluentd Documentation/Install steps for more details. If you use cf to deploy your fluentd instances, you will have to use tcp routing.

Fluentd comes with native support for syslog protocol. To set up Fluentd for Cloud Foundry, configure the syslog input of Fluentd as follows.

  1. In your main Fluentd configuration file, add the following source entry:

    <source>
      @type syslog
      port 8080
      bind 0.0.0.0
      tag cf.app
      message_length_limit 99990
      frame_type octet_count
      <transport tcp>
      </transport>
      <parse>
        message_format rfc5424
      </parse>
    </source>
    
  2. Restart the Fluentd service.

Note: The Fluentd syslog input plugin supports tls and tcp options. Make sure to use the same transport that Cloud Foundry is using.

Fluentd will start listening for Syslog message on port 8080 and tagging the messages with cf.app, which can be used later for data routing. For more details about the full setup for the service, refer to the Config File article.

If your goal is to use an Elasticsearch or Amazon S3 backend, read the following guide: http://www.fluentd.org/guides/recipes/elasticsearch-and-s3

Create a pull request or raise an issue on the source for this page in GitHub