Configuring System Logging

Page last updated:

This topic explains how to configure the Cloud Foundry Loggregator system.

Scaling Loggregator

Cloud Foundry system components and apps constantly generate log and metrics data. The Metron agent running on each component or application VM collects and sends this data out to Doppler components, which temporarily buffer the data before periodically forwarding it to the Traffic Controller. The Traffic Controller then serves the aggregated data stream through the Firehose WebSocket endpoint.

When the log and metrics data input to a Doppler exceeds its buffer size for a given interval, data can be lost. You can take several actions to minimize this loss.

Add additional Doppler server instances

You can increase the number of Doppler servers by increasing the instances property for the doppler_z1 and doppler_z2 jobs in your Cloud Foundry BOSH deployment manifest.

Enabling System Log Forwarding

Cloud Foundry can forward syslog data to an external aggregator using the syslog-release.

Customizing Loggregator Components

You can customize each Loggregator component by changing its properties in the CF deployment manifest. The following details some of the most commonly used changes.

Doppler

Property Description Default
doppler.zone Zone of the doppler server no default
doppler.debug Boolean value to enable verbose logging for Diego and Doppler server (the Doppler system) false
doppler.maxRetainedLogMessages Number of log messages to retain per application 100
doppler.dropsonde_incoming_port Port for incoming messages in the dropsonde format 3457
doppler.incoming_tcp_port Port for incoming TCP messages 3458
doppler.tls.enable Enable TLS listener on doppler so it can receive dropsonde envelopes over TLS transport. If enabled, you must specify Certificate and Key files. false
doppler.tls.port Port for incoming messages in the dropsonde format over TLS listener 3459
doppler.tls.server_cert TLS server certificate no default
doppler.tls.server_key TLS server key no default
loggregator.tls.ca_cert CA root required for key/certificate verification no default
loggregator.etcd.require_ssl Enable SSL for all communication with ETCD false
loggregator.etcd.ca_cert PEM-encoded CA certificate no default
loggregator.etcd.client_cert PEM-encoded client certificate no default
loggregator.etcd.client_key PEM-encoded client key no default
doppler.outgoing_port Port for outgoing log messages 8081
doppler.websocket_write_timeout_seconds Interval before aborting unsuccessful WebSocket write 60
doppler.blacklisted_syslog_ranges Blacklist for IP addresses that should not be used as syslog drains. For example, internal IP addresses no default
doppler.container_metric_ttl_seconds Time to live (TTL), in seconds, for container usage metrics 120
doppler.unmarshaller_count Number of parallel unmarshallers to run within Doppler 5
doppler.sink_inactivity_timeout_seconds Interval before removing a sink due to inactivity 3600
doppler.sink_dial_timeout_seconds Dial timeout for sinks 1
doppler.sink_io_timeout_seconds I/O Timeout on sinks 0
doppler_endpoint.shared_secret Shared secret used to verify cryptographically signed dropsonde messages no default
doppler.message_drain_buffer_size Size of the internal buffer used by doppler to store messages for output to firehose or Cloud Foundry logs. If buffer fills, Doppler drops messages. 10000
doppler.syslog_skip_cert_verify Boolean value to disable certificate verification for syslog sink when connecting over TLS true
doppler.locked_memory_limit Shell’s locked memory limit size. Accepts numeric values interpreted as KsB, or the following non-numeric values: kernel, soft, hard, unlimited. kernel sets limit to the kernel’s default. unlimited
loggregator.etcd.machines IP addresses pointing to the ETCD cluster no default
metron_endpoint.host Host used to emit messages to the Metron agent 127.0.0.1
metron_endpoint.dropsonde_port Port used to emit dropsonde messages to the Metron agent 3457

Traffic Controller

Property Name Description Default
traffic_controller.debug Boolean value to enable verbose logging for Diego and Loggregator server (the Loggregator system) false
traffic_controller.disable_access_control Boolean value to set Traffic Controller to bypasses authentication with the UAA and Cloud Controller false
traffic_controller.locked_memory_limit Shell’s locked memory limit size. Accepts numeric values interpreted as KsB, or the following non-numeric values: kernel, soft, hard, unlimited. kernel sets limit to the kernel’s default. unlimited
loggregator.outgoing_dropsonde_port Port for outgoing dropsonde messages 8081
traffic_controller.security_event_logging.enabled Boolean value to enable logging of all requests made to the Traffic Controller in CEF format false
doppler.uaa_client_id Doppler’s client id to connect to UAA doppler
uaa.clients.doppler.secret Doppler’s client secret to connect to UAA no default
uaa.url URL of UAA no default
login.protocol Protocol to use to connect to UAA if uaa.url is not set https
loggregator.etcd.require_ssl Enable SSL for all communication with ETCD false
loggregator.etcd.machines IP addresses pointing to the ETCD cluster no default
loggregator.etcd.maxconcurrentrequests Maximum number of concurrent requests to ETCD 10
loggregator.etcd.ca_cert PEM-encoded CA certificate no default
traffic_controller.etcd.client_cert PEM-encoded client certificate no default
traffic_controller.etcd.client_key PEM-encoded client key no default
traffic_controller.pprof_port pprof port for runtime profiling data 6060
system_domain Domain reserved for Cloud Foundry operator, and the base URL where login, UAA, and other non-user apps listen no default
ssl.skip_cert_verify Boolean value to ignore bad SSL certificates when connecting over https false
cc.srv_api_uri API URI of Cloud Controller no default

Metron Agent

Property Name Description Default
syslog_daemon_config.enable Boolean value to enable rsyslog configuration for forwarding syslog messages into Metron true
syslog_daemon_config.address IP address for syslog aggregator no default
syslog_daemon_config.port TCP port of syslog aggregator no default
syslog_daemon_config.transport Transport to use when forwarding logs. Accepts the following values: tcp, udp, or relp tcp
syslog_daemon_config.fallback_addresses IP addresses of fallback servers to use if primary syslog server is unavailable. Only tcp and relp supported. Each list entry must consist of address, transport, and port keys. []
syslog_daemon_config.custom_rule Custom rule for syslog forward daemon no default
syslog_daemon_config.max_message_size Maximum message size 4k
metron_endpoint.shared_secret Shared secret used to verify cryptographically signed dropsonde messages no default
metron_agent.listening_port Port on which the Metron agent listens to receive dropsonde log messages 3457
metron_agent.listening_address IP address on which the Metron agent listens to receive dropsonde log messages provided for BOSH links, should not be overwritten 127.0.0.1
metron_agent.debug Boolean value to enable verbose mode false
metron_agent.protocols A priority list of protocols over which Metron connects to Doppler. Metron will refuse to connect to Doppler over any protocol not on this list. ["udp"]
metron_agent.tls.client_cert TLS client certificate no default
metron_agent.tls.client_key TLS client key no default
metron_agent.tls.ca_cert CA root required for key/certificate verification no default
metron_agent.zone Availability zone where this agent runs no default
metron_agent.deployment Name of deployment. Added as tag on all outgoing metrics. no default
metron_agent.tcp.batching_buffer_bytes Number of bytes which can be buffered prior to TCP write. This applies to TLS over TCP. 10240
metron_agent.tcp.batching_buffer_flush_interval_milliseconds Maximum time a message can stay in the batching buffer before being flushed 100
metron_agent.logrotate.freq_min Frequency, in minutes, with which logrotate rotates VM logs 5
metron_agent.logrotate.rotate Number of files that logrotate retains on the VM 7
metron_agent.logrotate.size Size at which logrotate rotates log file 50M
loggregator.etcd.require_ssl Boolean value to enable SSL for all communication with ETCD false
loggregator.etcd.machines IP addresses pointing to the ETCD cluster no default
loggregator.etcd.maxconcurrentrequests Maximum number of concurrent requests to ETCD 10
loggregator.etcd.ca_cert PEM-encoded CA certificate no default
metron_agent.etcd.client_cert PEM-encoded client certificate no default
metron_agent.etcd.client_key PEM-encoded client key no default
metron_agent.pprof_port pprof port for runtime profiling data 6061

Syslog Drain Binder

See Using Log Management Services.

Property Name Description Default
metron_endpoint.host Host used to emit messages to the Metron agent 127.0.0.1
metron_endpoint.dropsonde_port Port used to emit dropsonde messages to the Metron agent 3457
loggregator.etcd.require_ssl Boolean value to enable SSL for all communication with ETCD false
loggregator.etcd.machines IP addresses pointing to the ETCD cluster no default
loggregator.etcd.maxconcurrentrequests Maximum number of concurrent requests to ETCD 10
loggregator.etcd.ca_cert PEM-encoded CA certificate no default
syslog_drain_binder.etcd.client_cert PEM-encoded client certificate no default
syslog_drain_binder.etcd.client_key PEM-encoded client key no default
system_domain Domain reserved for Cloud Foundry operator, and the base URL where login, UAA, and other non-user apps listen no default
syslog_drain_binder.drain_url_ttl_seconds Time to live (TTL), in seconds, for drain URLs 60
syslog_drain_binder.update_interval_seconds Interval, in seconds, on which to poll Cloud Controller 15
syslog_drain_binder.polling_batch_size Batch size for the poll from Cloud Controller 1000
syslog_drain_binder.debug Boolean value to enable verbose logging for syslog_drain_binder false
syslog_drain_binder.locked_memory_limit Shell’s locked memory limit size. Accepts numeric values interpreted as KsB, or the following non-numeric values: kernel, soft, hard, unlimited. kernel sets limit to the kernel’s default. unlimited
cc.bulk_api_password Password for the bulk API no default
cc.srv_api_uri API URI of Cloud Controller no default
ssl.skip_cert_verify Boolean value to ignore bad SSL certificates when connecting over https false
Create a pull request or raise an issue on the source for this page in GitHub