Configuring System Logging
Page last updated:
This topic explains how to configure the Cloud Foundry Loggregator system.
Scaling Loggregator
Cloud Foundry system components and apps constantly generate log and metrics data. The Metron agent running on each component or application VM collects and sends this data out to Doppler components, which temporarily buffer the data before periodically forwarding it to the Traffic Controller. The Traffic Controller then serves the aggregated data stream through the Firehose WebSocket endpoint.
When the log and metrics data input to a Doppler exceeds its buffer size for a given interval, data can be lost. You can take several actions to minimize this loss.
Increase Doppler buffer size
You can increase the Doppler internal buffer size by increasing the doppler.message_drain_buffer_size property from its default value of 10000 in your Cloud Foundry BOSH deployment manifest.
Add additional Doppler server instances
You can increase the number of Doppler servers by increasing the instances property for the doppler_z1 and doppler_z2 jobs in your Cloud Foundry BOSH deployment manifest.
Enabling System Log Forwarding
Cloud Foundry can forward syslog data to an external aggregator service instead of routing it to the Loggregator Firehose.
To do this, add the following properties to the properties hash in your CF deployment manifest:
properties:
syslog_daemon_config:
address: YOUR-SYSLOG-AGGREGATOR-IP
port: YOUR-SYSLOG-AGGREGATOR-TCP-PORT
transport: YOUR-TRANSPORT-PROTOCOL
Replace YOUR-TRANSPORT-PROTOCOL with one of the following transport protocols:
tcpudprelp
Customizing Loggregator Components
You can customize each Loggregator component by changing its properties in the CF deployment manifest. The following details some of the most commonly used changes.
Doppler
| Property | Description | Default |
|---|---|---|
| doppler.zone | Zone of the doppler server | no default |
| doppler.debug | Boolean value to enable verbose logging for Diego and Doppler server (the Doppler system) | false |
| doppler.maxRetainedLogMessages | Number of log messages to retain per application | 100 |
| doppler.dropsonde_incoming_port | Port for incoming messages in the dropsonde format | 3457 |
| doppler.incoming_tcp_port | Port for incoming TCP messages | 3458 |
| doppler.tls.enable | Enable TLS listener on doppler so it can receive dropsonde envelopes over TLS transport. If enabled, you must specify Certificate and Key files. | false |
| doppler.tls.port | Port for incoming messages in the dropsonde format over TLS listener | 3459 |
| doppler.tls.server_cert | TLS server certificate | no default |
| doppler.tls.server_key | TLS server key | no default |
| loggregator.tls.ca_cert | CA root required for key/certificate verification | no default |
| loggregator.etcd.require_ssl | Enable SSL for all communication with ETCD | false |
| loggregator.etcd.ca_cert | PEM-encoded CA certificate | no default |
| loggregator.etcd.client_cert | PEM-encoded client certificate | no default |
| loggregator.etcd.client_key | PEM-encoded client key | no default |
| doppler.outgoing_port | Port for outgoing log messages | 8081 |
| doppler.websocket_write_timeout_seconds | Interval before aborting unsuccessful WebSocket write | 60 |
| doppler.blacklisted_syslog_ranges | Blacklist for IP addresses that should not be used as syslog drains. For example, internal IP addresses | no default |
| doppler.container_metric_ttl_seconds | Time to live (TTL), in seconds, for container usage metrics | 120 |
| doppler.unmarshaller_count | Number of parallel unmarshallers to run within Doppler | 5 |
| doppler.sink_inactivity_timeout_seconds | Interval before removing a sink due to inactivity | 3600 |
| doppler.sink_dial_timeout_seconds | Dial timeout for sinks | 1 |
| doppler.sink_io_timeout_seconds | I/O Timeout on sinks | 0 |
| doppler_endpoint.shared_secret | Shared secret used to verify cryptographically signed dropsonde messages | no default |
| doppler.message_drain_buffer_size | Size of the internal buffer used by doppler to store messages for output to firehose or Cloud Foundry logs. If buffer fills, Doppler drops messages. | 10000 |
| doppler.syslog_skip_cert_verify | Boolean value to disable certificate verification for syslog sink when connecting over TLS | true |
| doppler.locked_memory_limit | Shell’s locked memory limit size. Accepts numeric values interpreted as KsB, or the following non-numeric values: kernel, soft, hard, unlimited. kernel sets limit to the kernel’s default. |
unlimited |
| loggregator.etcd.machines | IP addresses pointing to the ETCD cluster | no default |
| metron_endpoint.host | Host used to emit messages to the Metron agent | 127.0.0.1 |
| metron_endpoint.dropsonde_port | Port used to emit dropsonde messages to the Metron agent | 3457 |
Traffic Controller
| Property Name | Description | Default |
|---|---|---|
| traffic_controller.debug | Boolean value to enable verbose logging for Diego and Loggregator server (the Loggregator system) | false |
| traffic_controller.disable_access_control | Boolean value to set Traffic Controller to bypasses authentication with the UAA and Cloud Controller | false |
| traffic_controller.locked_memory_limit | Shell’s locked memory limit size. Accepts numeric values interpreted as KsB, or the following non-numeric values: kernel, soft, hard, unlimited. kernel sets limit to the kernel’s default. |
unlimited |
| loggregator.outgoing_dropsonde_port | Port for outgoing dropsonde messages | 8081 |
| traffic_controller.security_event_logging.enabled | Boolean value to enable logging of all requests made to the Traffic Controller in CEF format | false |
| doppler.uaa_client_id | Doppler’s client id to connect to UAA | doppler |
| uaa.clients.doppler.secret | Doppler’s client secret to connect to UAA | no default |
| uaa.url | URL of UAA | no default |
| login.protocol | Protocol to use to connect to UAA if uaa.url is not set | https |
| loggregator.etcd.require_ssl | Enable SSL for all communication with ETCD | false |
| loggregator.etcd.machines | IP addresses pointing to the ETCD cluster | no default |
| loggregator.etcd.maxconcurrentrequests | Maximum number of concurrent requests to ETCD | 10 |
| loggregator.etcd.ca_cert | PEM-encoded CA certificate | no default |
| traffic_controller.etcd.client_cert | PEM-encoded client certificate | no default |
| traffic_controller.etcd.client_key | PEM-encoded client key | no default |
| traffic_controller.pprof_port | pprof port for runtime profiling data | 6060 |
| system_domain | Domain reserved for Cloud Foundry operator, and the base URL where login, UAA, and other non-user apps listen | no default |
| ssl.skip_cert_verify | Boolean value to ignore bad SSL certificates when connecting over https | false |
| cc.srv_api_uri | API URI of Cloud Controller | no default |
Metron Agent
| Property Name | Description | Default |
|---|---|---|
| syslog_daemon_config.enable | Boolean value to enable rsyslog configuration for forwarding syslog messages into Metron | true |
| syslog_daemon_config.address | IP address for syslog aggregator | no default |
| syslog_daemon_config.port | TCP port of syslog aggregator | no default |
| syslog_daemon_config.transport | Transport to use when forwarding logs. Accepts the following values: tcp, udp, or relp |
tcp |
| syslog_daemon_config.fallback_addresses | IP addresses of fallback servers to use if primary syslog server is unavailable. Only tcp and relp supported. Each list entry must consist of address, transport, and port keys. |
[] |
| syslog_daemon_config.custom_rule | Custom rule for syslog forward daemon | no default |
| syslog_daemon_config.max_message_size | Maximum message size | 4k |
| metron_endpoint.shared_secret | Shared secret used to verify cryptographically signed dropsonde messages | no default |
| metron_agent.listening_port | Port on which the Metron agent listens to receive dropsonde log messages | 3457 |
| metron_agent.listening_address | IP address on which the Metron agent listens to receive dropsonde log messages provided for BOSH links, should not be overwritten | 127.0.0.1 |
| metron_agent.debug | Boolean value to enable verbose mode | false |
| metron_agent.protocols | A priority list of protocols over which Metron connects to Doppler. Metron will refuse to connect to Doppler over any protocol not on this list. | ["udp"] |
| metron_agent.tls.client_cert | TLS client certificate | no default |
| metron_agent.tls.client_key | TLS client key | no default |
| metron_agent.tls.ca_cert | CA root required for key/certificate verification | no default |
| metron_agent.zone | Availability zone where this agent runs | no default |
| metron_agent.deployment | Name of deployment. Added as tag on all outgoing metrics. | no default |
| metron_agent.tcp.batching_buffer_bytes | Number of bytes which can be buffered prior to TCP write. This applies to TLS over TCP. | 10240 |
| metron_agent.tcp.batching_buffer_flush_interval_milliseconds | Maximum time a message can stay in the batching buffer before being flushed | 100 |
| metron_agent.logrotate.freq_min | Frequency, in minutes, with which logrotate rotates VM logs | 5 |
| metron_agent.logrotate.rotate | Number of files that logrotate retains on the VM | 7 |
| metron_agent.logrotate.size | Size at which logrotate rotates log file | 50M |
| loggregator.etcd.require_ssl | Boolean value to enable SSL for all communication with ETCD | false |
| loggregator.etcd.machines | IP addresses pointing to the ETCD cluster | no default |
| loggregator.etcd.maxconcurrentrequests | Maximum number of concurrent requests to ETCD | 10 |
| loggregator.etcd.ca_cert | PEM-encoded CA certificate | no default |
| metron_agent.etcd.client_cert | PEM-encoded client certificate | no default |
| metron_agent.etcd.client_key | PEM-encoded client key | no default |
| metron_agent.pprof_port | pprof port for runtime profiling data | 6061 |
Syslog Drain Binder
See Using Log Management Services.
| Property Name | Description | Default | metron_endpoint.host | Host used to emit messages to the Metron agent | 127.0.0.1 |
|---|---|---|
| metron_endpoint.dropsonde_port | Port used to emit dropsonde messages to the Metron agent | 3457 |
| loggregator.etcd.require_ssl | Boolean value to enable SSL for all communication with ETCD | false |
| loggregator.etcd.machines | IP addresses pointing to the ETCD cluster | no default |
| loggregator.etcd.maxconcurrentrequests | Maximum number of concurrent requests to ETCD | 10 |
| loggregator.etcd.ca_cert | PEM-encoded CA certificate | no default |
| syslog_drain_binder.etcd.client_cert | PEM-encoded client certificate | no default |
| syslog_drain_binder.etcd.client_key | PEM-encoded client key | no default |
| system_domain | Domain reserved for Cloud Foundry operator, and the base URL where login, UAA, and other non-user apps listen | no default |
| syslog_drain_binder.drain_url_ttl_seconds | Time to live (TTL), in seconds, for drain URLs | 60 |
| syslog_drain_binder.update_interval_seconds | Interval, in seconds, on which to poll Cloud Controller | 15 |
| syslog_drain_binder.polling_batch_size | Batch size for the poll from Cloud Controller | 1000 |
| syslog_drain_binder.debug | Boolean value to enable verbose logging for syslog_drain_binder | false |
| syslog_drain_binder.locked_memory_limit | Shell’s locked memory limit size. Accepts numeric values interpreted as KsB, or the following non-numeric values: kernel, soft, hard, unlimited. kernel sets limit to the kernel’s default. |
unlimited |
| cc.bulk_api_password | Password for the bulk API | no default |
| cc.srv_api_uri | API URI of Cloud Controller | no default |
| ssl.skip_cert_verify | Boolean value to ignore bad SSL certificates when connecting over https | false |