Configuring the OpenTelemetry Collector
Page last updated:
You can deploy an OpenTelemetry Collector for Cloud Foundry to egress traces, metrics, and logs.
When the OpenTelemetry (OTel) Collector is deployed, the Loggregator Forwarder Agent forwards platform and application metrics to the OTel Collector, which then processes traces, metrics, and logs using one or more pipelines that you configure.
Collector configuration structure
The OTel Collector has a standard YAML-based file format for configuration.
The structure of Cloud Foundry configuration file consists of three classes of pipeline components that access telemetry data:
After each pipeline component is configured, add it in the service.pipelines section of the Collector configuration file.
Exporter configuration
For example, to configure the OpenTelemetry Collector to egress metrics using OTLP over gRPC, you can provide the following minimal configuration. This will use the defaults for gRPC and TLS configuration.
otlp_grpc:
endpoint: 203.0.113.10:4317
As the OTel Collector is deployed on all VMs in Cloud Foundry, you must ensure that all VMs in your deployment are allowed to connect to the OTLP server endpoint.
Note
The OTLP exporter config key names changed in this release. Use otlp_grpc
for gRPC and otlp_http for HTTP. The previous names otlp and
otlphttp are deprecated but continue to work.
Included exporters
| Exporter | Stability |
|---|---|
otlpexporter |
stable |
otlphttpexporter |
stable |
nopexporter |
beta |
fileexporter |
alpha |
prometheusexporter |
stable |
prometheusremotewriteexporter |
beta |
splunkhecexporter |
beta |
datadogexporter |
beta |
kafkaexporter |
beta |
syslogexporter |
alpha |
mTLS
To configure the exporter to use an mTLS connection to the remote OTLP endpoint, you can specify additional TLS configuration.
otlp_grpc:
endpoint: 203.0.113.10:4317
tls:
cert_pem: |
PEM_ENCODED_CERTIFICATE
key_pem: |
PEM_ENCODED_PRIVATE_KEY
ca_pem: |
PEM_ENCODED_CERTIFICATE
Multiple exporters
The OTel Collector supports defining multiple exporters, each with its own configuration. For example, you can define two OTLP gRPC exporters that egress metrics to different destinations.
otlp_grpc:
endpoint: 203.0.113.10:4317
otlp_grpc/another:
endpoint: 203.0.113.11:4317
Receiver configuration
The release always supplies the managed otlp/cf-internal-local receiver for telemetry from the Loggregator Forwarder Agent. Its definition comes from the BOSH job’s ingress.grpc.* properties and overrides any operator definition with the same name.
To use custom receivers, define them in config.receivers and reference them in the appropriate config.service.pipelines receiver lists.
The BOSH job property ingress.pipelines controls how the managed receiver is wired:
ingress.pipelines |
Receiver definitions | Pipeline receiver lists |
|---|---|---|
| Unset (default) | Replaced with only otlp/cf-internal-local |
All replaced with [otlp/cf-internal-local] |
[] |
Custom definitions preserved; managed receiver supplied | Unchanged; wire all receivers manually |
| Nonempty list | Custom definitions preserved; managed receiver supplied | Only selected pipelines’ lists replaced with [otlp/cf-internal-local]; other lists unchanged |
A nonempty list must contain exact configured pipeline names, for example [metrics/inbound]. Selected receiver lists are replaced, not extended. To combine managed and custom receivers in the same pipeline, use [] and explicitly list both receivers, as in the custom receiver example.
Only the default, unset mode adds nop pipelines for missing traces, metrics, or logs. Explicit modes ([] or a nonempty list) add no nop fallbacks. Operators must configure pipelines to handle unwanted signals, for example with a nop exporter, or disable emission of those signals at their sources.
Migration warning
Default behavior remains compatible. Before switching to an explicit mode, remove obsolete placeholder receiver definitions such as an empty otlp/placeholder. Explicit modes preserve these definitions instead of discarding them, and invalid definitions can prevent the Collector from starting.
See the deployment instructions for release requirements and Configuring custom receiver wiring in CF Deployment for the supplementary BOSH operations file.
Included receivers
| Receiver | Stability |
|---|---|
otlpreceiver |
stable |
filelogreceiver |
beta |
gRPC compression
You can configure the type of compression used for OTLP gRPC exporters or receivers. The OTLP gRPC exporter uses gzip compression
by default. To reduce CPU usage of the OTel Collector (increasing the size of the payload) you can optionally configure
the OTLP gRPC exporter to use snappy compression. Validate that the server you are sending metrics to
support snappy compression before making this change.
otlp_grpc:
endpoint: 203.0.113.10:4317
compression: snappy
Processor configuration
Processors are used at various stages of a pipeline to pre-processes data before it is exported. By default, no processors are enabled. The order of the processors in a pipeline determines the order of the processing operations that the Collector applies to the signal.
Refer to the individual processor documentation for more information.
Currently, the Cloud Foundry distribution of the OpenTelemetry Collector supports the following processors out of the box: batch, memorylimiter. You can configure processors using the processors section of the Collector configuration file.
processors:
batch:
Included processors
| Processor | Stability |
|---|---|
batchprocessor |
stable |
memorylimiterprocessor |
stable |
transformprocessor |
beta |
filterprocessor |
alpha |
Authentication
The Cloud Foundry distribution of the OpenTelemetry Collector does not currently include any authenticator extensions. Please let us know in the Cloud Foundry Slack #logging-and-metrics channel if there are authenticator extensions you would find useful.
Reviewing the available components
The Cloud Foundry distribution of the OpenTelemetry Collector ships with a number of components. To see the components that are available for use, run the following command on a VM that has otel-collector deployed.
/var/vcap/packages/otel-collector/otel-collector components
Service section
The service section is used to configure what components are enabled in the Collector based on the configuration found in the receivers, processors, exporters, and extensions sections. If a component is configured, but not defined referenced the service section, then it’s not enabled.
The service section consists of two subsections:
Extensions
You can enable desired extensions using extensions subsections:
service:
extensions: [pprof]
Included Extensions
Included extensions
| Extension | Stability |
|---|---|
opampextension |
alpha |
pprofextension |
beta |
Pipelines
A pipeline is a set of components used to process a signal. Before including a receiver, processor, or exporter in a pipeline, make sure to define its configuration in the appropriate section.
You can use the same receiver, processor, or exporter in more than one pipeline. When a processor is referenced in multiple pipelines, each pipeline gets a separate instance of the processor.
The following is a Collector configuration fragment, under the BOSH job’s config property. The release supplies the managed receiver; define the referenced processors, exporters, and extensions separately. Note that the order of processors dictates the order in which data is processed:
service:
extensions: [pprof]
pipelines:
traces:
receivers: [otlp/cf-internal-local]
processors: [batch]
exporters: [otlp_grpc]
metrics:
receivers: [otlp/cf-internal-local]
processors: [batch]
exporters: [otlp_grpc]
Example config
Below is a complete input for the BOSH job’s config property, not a complete BOSH properties map. Save this Collector configuration as the file supplied to otel_collector_config in the deployment command. For receiver wiring options, see Receiver configuration.
processors:
batch:
exporters:
otlp_grpc:
endpoint: otelcol:4317
extensions:
pprof:
service:
extensions: [pprof]
pipelines:
traces:
receivers: [otlp/cf-internal-local]
processors: [batch]
exporters: [otlp_grpc]
metrics:
receivers: [otlp/cf-internal-local]
processors: [batch]
exporters: [otlp_grpc]
Custom filelog receiver example
This BOSH job properties example collects logs from files on the VM and combines them with logs from the managed receiver. Traces and metrics use only the managed receiver. For receiver wiring options, see Receiver configuration. Only the contents of config are Collector configuration.
ingress:
pipelines: []
config:
receivers:
filelog/custom:
include:
- /var/vcap/sys/log/YOUR_JOB/*.log
start_at: end
processors:
batch:
exporters:
otlp_grpc:
endpoint: otelcol:4317
service:
pipelines:
traces:
receivers: [otlp/cf-internal-local]
processors: [batch]
exporters: [otlp_grpc]
metrics:
receivers: [otlp/cf-internal-local]
processors: [batch]
exporters: [otlp_grpc]
logs:
receivers: [otlp/cf-internal-local, filelog/custom]
processors: [batch]
exporters: [otlp_grpc]
Set ingress.pipelines using a supplementary operations file as shown in Configuring custom receiver wiring in CF Deployment.
Deploying the OpenTelemetry Collector
CF Deployment provides experimental operations files that you can use to deploy the OTel Collector:
Custom receiver wiring via ingress.pipelines is available in otel-collector-release v0.11.13 or later.
For example, to deploy a fresh CF Deployment with OTel Collector enabled, run:
bosh deploy cf-deployment.yml \
--ops-file=operations/experimental/add-otel-collector.yml \
--var-file=otel_collector_config=YOUR_COLLECTOR_CONFIG_FILE_PATH.yml \
--var=system_domain=YOUR_SYSTEM_DOMAIN
The supplied file must contain the full Collector configuration, such as the example config. The --var-file option passes the file contents as a string, which the BOSH job template parses as YAML. The operations file places it under the job’s config property and supplies the managed receiver’s TLS certificates.
Configuring custom receiver wiring in CF Deployment
When using a release that supports ingress.pipelines, save the following supplementary BOSH operations file as custom-otel-ingress.yml to enable manual receiver wiring on Linux:
- type: replace
path: /addons/name=otel-collector/jobs/name=otel-collector/properties/ingress/pipelines?
value: []
Apply it with --ops-file=custom-otel-ingress.yml after the addon operations files. To select pipelines instead of manual wiring, replace [] with their exact names, for example [metrics]; each selected receiver list will be replaced, not extended.