Configuring the OpenTelemetry Collector

Page last updated:

You can deploy an OpenTelemetry Collector for Cloud Foundry to egress traces, metrics, and logs.

When the OpenTelemetry (OTel) Collector is deployed, the Loggregator Forwarder Agent forwards platform and application metrics to the OTel Collector, which then processes traces, metrics, and logs using one or more pipelines that you configure.

Collector configuration structure

The OTel Collector has a standard YAML-based file format for configuration.

The structure of Cloud Foundry configuration file consists of three classes of pipeline components that access telemetry data:

After each pipeline component is configured, add it in the service.pipelines section of the Collector configuration file.

Exporter configuration

For example, to configure the OpenTelemetry Collector to egress metrics using OTLP over gRPC, you can provide the following minimal configuration. This will use the defaults for gRPC and TLS configuration.

otlp_grpc:
  endpoint: 203.0.113.10:4317

As the OTel Collector is deployed on all VMs in Cloud Foundry, you must ensure that all VMs in your deployment are allowed to connect to the OTLP server endpoint.

Note The OTLP exporter config key names changed in this release. Use otlp_grpc for gRPC and otlp_http for HTTP. The previous names otlp and otlphttp are deprecated but continue to work.

Included exporters

Exporter Stability
otlpexporter stable
otlphttpexporter stable
nopexporter beta
fileexporter alpha
prometheusexporter stable
prometheusremotewriteexporter beta
splunkhecexporter beta
datadogexporter beta
kafkaexporter beta
syslogexporter alpha

mTLS

To configure the exporter to use an mTLS connection to the remote OTLP endpoint, you can specify additional TLS configuration.

otlp_grpc:
  endpoint: 203.0.113.10:4317
  tls:
    cert_pem: |
      PEM_ENCODED_CERTIFICATE
    key_pem: |
      PEM_ENCODED_PRIVATE_KEY
    ca_pem: |
      PEM_ENCODED_CERTIFICATE

Multiple exporters

The OTel Collector supports defining multiple exporters, each with its own configuration. For example, you can define two OTLP gRPC exporters that egress metrics to different destinations.

otlp_grpc:
  endpoint: 203.0.113.10:4317
otlp_grpc/another:
  endpoint: 203.0.113.11:4317

Receiver configuration

The release always supplies the managed otlp/cf-internal-local receiver for telemetry from the Loggregator Forwarder Agent. Its definition comes from the BOSH job’s ingress.grpc.* properties and overrides any operator definition with the same name.

To use custom receivers, define them in config.receivers and reference them in the appropriate config.service.pipelines receiver lists.

The BOSH job property ingress.pipelines controls how the managed receiver is wired:

ingress.pipelines Receiver definitions Pipeline receiver lists
Unset (default) Replaced with only otlp/cf-internal-local All replaced with [otlp/cf-internal-local]
[] Custom definitions preserved; managed receiver supplied Unchanged; wire all receivers manually
Nonempty list Custom definitions preserved; managed receiver supplied Only selected pipelines’ lists replaced with [otlp/cf-internal-local]; other lists unchanged

A nonempty list must contain exact configured pipeline names, for example [metrics/inbound]. Selected receiver lists are replaced, not extended. To combine managed and custom receivers in the same pipeline, use [] and explicitly list both receivers, as in the custom receiver example.

Only the default, unset mode adds nop pipelines for missing traces, metrics, or logs. Explicit modes ([] or a nonempty list) add no nop fallbacks. Operators must configure pipelines to handle unwanted signals, for example with a nop exporter, or disable emission of those signals at their sources.

Migration warning Default behavior remains compatible. Before switching to an explicit mode, remove obsolete placeholder receiver definitions such as an empty otlp/placeholder. Explicit modes preserve these definitions instead of discarding them, and invalid definitions can prevent the Collector from starting.

See the deployment instructions for release requirements and Configuring custom receiver wiring in CF Deployment for the supplementary BOSH operations file.

Included receivers

Receiver Stability
otlpreceiver stable
filelogreceiver beta

gRPC compression

You can configure the type of compression used for OTLP gRPC exporters or receivers. The OTLP gRPC exporter uses gzip compression by default. To reduce CPU usage of the OTel Collector (increasing the size of the payload) you can optionally configure the OTLP gRPC exporter to use snappy compression. Validate that the server you are sending metrics to support snappy compression before making this change.

otlp_grpc:
  endpoint: 203.0.113.10:4317
  compression: snappy

Processor configuration

Processors are used at various stages of a pipeline to pre-processes data before it is exported. By default, no processors are enabled. The order of the processors in a pipeline determines the order of the processing operations that the Collector applies to the signal.

Refer to the individual processor documentation for more information.

Currently, the Cloud Foundry distribution of the OpenTelemetry Collector supports the following processors out of the box: batch, memorylimiter. You can configure processors using the processors section of the Collector configuration file.

processors:
  batch:

Included processors

Processor Stability
batchprocessor stable
memorylimiterprocessor stable
transformprocessor beta
filterprocessor alpha

Authentication

The Cloud Foundry distribution of the OpenTelemetry Collector does not currently include any authenticator extensions. Please let us know in the Cloud Foundry Slack #logging-and-metrics channel if there are authenticator extensions you would find useful.

Reviewing the available components

The Cloud Foundry distribution of the OpenTelemetry Collector ships with a number of components. To see the components that are available for use, run the following command on a VM that has otel-collector deployed.

/var/vcap/packages/otel-collector/otel-collector components

Service section

The service section is used to configure what components are enabled in the Collector based on the configuration found in the receivers, processors, exporters, and extensions sections. If a component is configured, but not defined referenced the service section, then it’s not enabled.

The service section consists of two subsections:

Extensions

You can enable desired extensions using extensions subsections:

service:
  extensions: [pprof]

Included Extensions

Included extensions

Extension Stability
opampextension alpha
pprofextension beta

Pipelines

A pipeline is a set of components used to process a signal. Before including a receiver, processor, or exporter in a pipeline, make sure to define its configuration in the appropriate section.

You can use the same receiver, processor, or exporter in more than one pipeline. When a processor is referenced in multiple pipelines, each pipeline gets a separate instance of the processor.

The following is a Collector configuration fragment, under the BOSH job’s config property. The release supplies the managed receiver; define the referenced processors, exporters, and extensions separately. Note that the order of processors dictates the order in which data is processed:

service:
  extensions: [pprof]
  pipelines:
    traces:
      receivers: [otlp/cf-internal-local]
      processors: [batch]
      exporters: [otlp_grpc]
    metrics:
      receivers: [otlp/cf-internal-local]
      processors: [batch]
      exporters: [otlp_grpc]

Example config

Below is a complete input for the BOSH job’s config property, not a complete BOSH properties map. Save this Collector configuration as the file supplied to otel_collector_config in the deployment command. For receiver wiring options, see Receiver configuration.

processors:
  batch:

exporters:
  otlp_grpc:
    endpoint: otelcol:4317

extensions:
  pprof:

service:
  extensions: [pprof]
  pipelines:
    traces:
      receivers: [otlp/cf-internal-local]
      processors: [batch]
      exporters: [otlp_grpc]
    metrics:
      receivers: [otlp/cf-internal-local]
      processors: [batch]
      exporters: [otlp_grpc]

Custom filelog receiver example

This BOSH job properties example collects logs from files on the VM and combines them with logs from the managed receiver. Traces and metrics use only the managed receiver. For receiver wiring options, see Receiver configuration. Only the contents of config are Collector configuration.

ingress:
  pipelines: []
config:
  receivers:
    filelog/custom:
      include:
        - /var/vcap/sys/log/YOUR_JOB/*.log
      start_at: end
  processors:
    batch:
  exporters:
    otlp_grpc:
      endpoint: otelcol:4317
  service:
    pipelines:
      traces:
        receivers: [otlp/cf-internal-local]
        processors: [batch]
        exporters: [otlp_grpc]
      metrics:
        receivers: [otlp/cf-internal-local]
        processors: [batch]
        exporters: [otlp_grpc]
      logs:
        receivers: [otlp/cf-internal-local, filelog/custom]
        processors: [batch]
        exporters: [otlp_grpc]

Set ingress.pipelines using a supplementary operations file as shown in Configuring custom receiver wiring in CF Deployment.

Deploying the OpenTelemetry Collector

CF Deployment provides experimental operations files that you can use to deploy the OTel Collector:

Custom receiver wiring via ingress.pipelines is available in otel-collector-release v0.11.13 or later.

For example, to deploy a fresh CF Deployment with OTel Collector enabled, run:

bosh deploy cf-deployment.yml \
  --ops-file=operations/experimental/add-otel-collector.yml \
  --var-file=otel_collector_config=YOUR_COLLECTOR_CONFIG_FILE_PATH.yml \
  --var=system_domain=YOUR_SYSTEM_DOMAIN

The supplied file must contain the full Collector configuration, such as the example config. The --var-file option passes the file contents as a string, which the BOSH job template parses as YAML. The operations file places it under the job’s config property and supplies the managed receiver’s TLS certificates.

Configuring custom receiver wiring in CF Deployment

When using a release that supports ingress.pipelines, save the following supplementary BOSH operations file as custom-otel-ingress.yml to enable manual receiver wiring on Linux:

- type: replace
  path: /addons/name=otel-collector/jobs/name=otel-collector/properties/ingress/pipelines?
  value: []

Apply it with --ops-file=custom-otel-ingress.yml after the addon operations files. To select pipelines instead of manual wiring, replace [] with their exact names, for example [metrics]; each selected receiver list will be replaced, not extended.

Create a pull request or raise an issue on the source for this page in GitHub