Loggregator Architecture

Page last updated:

This topic describes the Loggregator architecture and components. It also describes the Cloud Foundry components that send BOSH-reported VM metrics to Loggregator.

Overview

Loggregator gathers and streams logs and metrics from user apps in an Cloud Foundry deployment. It also gathers and streams metrics from Cloud Foundry components and health metrics from BOSH-deployed VMs. Loggregator allows you to view these logs and metrics through the Loggregator CLI plugins or through a third-party service. For more information, see the Loggregator repository on GitHub.

Loggregator architecture includes components for collecting, storing, and forwarding logs and metrics.

Loggregator Architecture and Components

This section includes the following Loggregator architecture diagrams, describing parallel paths for log and metric delivery:

Firehose Architecture

The following diagram shows the architecture of log/metric delivery via the Loggregator firehose.

A Forwarder Agent appears in squares that depict 'Diego Cell VMs' and 'Component VMs'. Within the Diego Cell VMs, applications, Diego Rep and Prom Scraper send logs and metrics to the Forwarder Agent. Within the Component VMs, Diego, Prom Scraper and Statsd Injector send metrics to the Forwarder Agent. The Forwarder Agent sends to the Loggregator Agents, which in turn send to the Dopplers. Dopplers send to Traffic Controllers and Reverse Log Proxies, which each send logs and metrics to different consumers. Traffic Controllers send to V1 Firehose Consumers, and Reverse Log Proxies send to V2 Firehose Consumers. V2 Firehose External Consumers are serviced through the Reverse Log Proxy Gateway. Logs and metrics are also sent to Log Cache via the Log Cache Nozzle. Cloud Controller consumes app metrics from Log Cache, and developer tools consume logs and metrics via the Log Cache CF Auth Proxy. For more descriptions of all of the components in Loggregator, see the list of components below.

View a larger version of this image.

The following components are included in the Loggregator Firehose, as shown in the diagram above:

  • Prom Scraper: Prom Scrapers run on both Cloud Foundry component VMs and Diego Cell VMs. They aggregate metrics from Cloud Foundry components located on those VMs via Prometheus exposition. Prom Scrapers then forward those metrics to Forwarder Agents.

  • Statsd Injector: Statsd Injectors run on Cloud Foundry component VMs. They receive metrics from Cloud Foundry components over the Statsd protocol. Statsd Injectors then forward those metrics to Forwarder Agents.

  • Forwarder Agent: Forwarder Agents run on both Cloud Foundry component VMs and Diego Cell VMs. They receive logs and metrics from the apps and Cloud Foundry components located on those VMs. Forwarder Agents then forward the logs and metrics to Loggregator Agents and other agents.

  • Loggregator Agent: Loggregator Agents run on both Cloud Foundry component VMs and Diego Cell VMs. They receive logs and metrics from the Forwarder Agents, then forward the logs and metrics via multiple downstream Dopplers.

  • Doppler: Dopplers receive logs and metrics from Loggregator Agents, store them in temporary buffers, and forward them to Traffic Controllers and Reverse Log Proxies.

  • Traffic Controller: Traffic Controllers poll Doppler servers for logs and metrics, then translate these messages from the Doppler servers as necessary for external and legacy APIs. It services the Firehose Endpoint (also known as the V1 Firehose). The Firehose cf CLI plugin allows you to view the output of the Firehose. For more information about the Firehose plugin, see Installing the Loggregator Firehose Plugin for cf CLI.

  • Reverse Log Proxy (V2 Firehose): Reverse Log Proxies (RLPs) collect logs and metrics from Dopplers and forward them to Log Cache and other consumers via gRPC. Operators can scale up the number of RLPs based on overall log volume.

  • Reverse Log Proxy Gateway (V2 Firehose): Reverse Log Proxies Gateways (RLP Gateways) collect logs and metrics from Reverse Log Proxies and forward them to consumers via HTTP. Collecting logs and metrics via the RLP Gateway will have lower throughput compared to consuming from the Traffic Controller or Reverse Log Proxy.

  • Log Cache:

    Log Cache allows you to view logs and metrics over a specified period of time. The Log Cache includes API endpoints and a CLI plugin to query and filter logs and metrics. To download the Log Cache CLI plugin, see Cloud Foundry Plugins. The Log Cache API endpoints are available by default. For more information about using the Log Cache API, see Log Cache on GitHub.

  • rsyslog: While not part of the firehose itself, rsyslog is responsible for delivering logs from platform components to outside consumers.

Shared-Nothing Architecture

The following diagram shows the architecture of log/metric delivery via the Shared-Nothing Architecture.

A Forwarder Agent appears in squares that depict 'Diego Cell VMs' and 'Component VMs'. Within the Diego Cell VMs, applications, Diego Rep and Prom Scraper send logs and metrics to the Forwarder Agent. Within the Component VMs, Diego, Prom Scraper and Statsd Injector send metrics to the Forwarder Agent. The Forwarder Agent sends to both Syslog Agents and Metrics Agents on the same VM. Syslog Agents send logs to Syslog Drain Consumers and logs/metrics to the Log Cache Syslog Server via Syslog RFC 5424. Metrics Agents expose metrics via Prometheus exposition which can be scraped by Metric Scrapers. Metrics Discovery Registrars on the same VM register the Metrics Agents' endpoints with NATS for discovery by Metric Scrapers. Cloud Controller consumes app metrics from Log Cache, and developer tools consume logs and metrics via the Log Cache CF Auth Proxy. For more descriptions of all of the components in Loggregator, see the list of components below.

View a larger version of this image.

The following components are included in the Shared-Nothing Architecture, as shown in the diagram above:

  • Prom Scraper: Prom Scrapers run on both Cloud Foundry component VMs and Diego Cell VMs. They aggregate metrics from Cloud Foundry components located on those VMs via Prometheus exposition. Prom Scrapers then forward those metrics to Forwarder Agents.

  • Statsd Injector: Statsd Injectors run on Cloud Foundry component VMs. They receive metrics from Cloud Foundry components over the Statsd protocol. Statsd Injectors then forward those metrics to Forwarder Agents.

  • Forwarder Agent: Forwarder Agents run on both Cloud Foundry component VMs and Diego Cell VMs. They receive logs and metrics from the apps and Cloud Foundry components located on those VMs. Forwarder Agents then forward the logs and metrics to Loggregator Agents and other agents.

  • Syslog Agent:

    Syslog Agents run on Cloud Foundry component VMs and host VMs to collect and forward logs to configured syslog drains. This includes syslog drains for individual apps as well as aggregate drains for all apps in your foundation. You can specify the destination for logs as part of the individual syslog drain or in the CF tile.

  • Syslog Binding Cache (not pictured): Syslog Agents can overwhelm CAPI when querying for existing bindings. This component acts a a caching proxy between the Syslog Agents and CAPI.

  • Metrics Agents: Metrics Agents receive metrics from Forwarder Agents, and make them available to Metric Scrapers via Prometheus Exposition.

  • Metrics Discovery Registrars: Metrics Discovery Registrars register each scrapeable endpoint with NATS, for discovery by Metrics Scrapers.

  • Log Cache:

    Log Cache allows you to view logs and metrics over a specified period of time. The Log Cache includes API endpoints and a CLI plugin to query and filter logs and metrics. To download the Log Cache CLI plugin, see Cloud Foundry Plugins. The Log Cache API endpoints are available by default. For more information about using the Log Cache API, see Log Cache on GitHub.

  • Log Cache Syslog Server: The Log Cache Syslog Server takes the place of the Log Cache Nozzle (only one will be active at a time). It receives logs and metrics from Syslog Agents and sends them to Log Cache.

System Metrics Agents Architecture

The following diagram shows the architecture of a Loggregator deployment that uses System Metrics Agents to collect VM and system-level metrics.

System Metrics Agent appear in squares that depict 'Host VMs'. Represented by arrows, the System Metrics Agents send VM system-level metrics to a System Metrics Scraper, which forwards these metrics to Loggregator Agents over mTLS. For more descriptions of all of the components in Loggregator, see the 'Loggregator Components' section below.

View a larger version of this image.

The following describes the components of a Loggregator deployment that uses System Metrics Agents, as shown in the diagram above:

  • System Metrics Agent: A standalone agent to provide VM system metrics using a Prometheus-scrapable endpoint.

  • System Metrics Scraper: The System Metrics Scraper forwards metrics from System Metrics Agents to Loggregator Agents over mTLS.

This section describes the Cloud Foundry components that forward BOSH-reported VM metrics to Loggregator. BOSH-reported VM metrics measure the health of BOSH-deployed VMs on which apps and Cloud Foundry components are deployed. Loggregator streams BOSH-reported VM metrics through the Firehose.

The following are the Cloud Foundry components that send BOSH-reported VM metrics to Loggregator:

  • BOSH Agent: BOSH Agents are located on component VMs and Diego Cell VMs. They collect metrics, such as Diego Cell capacity remaining, from the VM and forward them to the BOSH Health Monitor.

  • BOSH Health Monitor: The BOSH Health Monitor receives metrics from the BOSH Agents. It then forwards the metrics to a third-party service or to the BOSH System Metrics Forwarder.

  • BOSH System Metrics Plugin: This plugin reads health events, such as VM heartbeats and alerts from the BOSH Health Monitor, and streams them to the BOSH System Metrics Server.

  • BOSH System Metrics Server: The BOSH System Metrics Server streams metrics and heartbeat events to the BOSH System Metrics Forwarder over gRPC.

  • BOSH System Metrics Forwarder: The BOSH System Metrics Forwarder is located on the Loggregator Traffic Controller. It forwards heartbeat events from the BOSH System Metrics Server as envelopes to Loggregator through a Loggregator Agent.

Create a pull request or raise an issue on the source for this page in GitHub