Your OpenTelemetry Collector configuration affects how much telemetry reaches your backend and how the Collector handles load. Filtering and sampling reduce the volume of data exported, while batching reduces the number of export requests. Memory limits and queues help manage traffic spikes. Gartner figures cited by The New Stack show that log contracts covering tens to hundreds of gigabytes per day a decade ago now cover tens to hundreds of terabytes, and that as of 2023, 37% of enterprises ingested more than 1 TB of log data daily.
This guide explains how OpenTelemetry Collector pipelines process telemetry, how to configure their components, and how to choose deployment patterns and distributions. It also covers how to validate and troubleshoot Collector configurations and how groundcover complements OpenTelemetry with eBPF-based telemetry and AI-assisted investigations.
What Is an OpenTelemetry Collector Pipeline?
An OpenTelemetry Collector pipeline is a named dataflow, declared under service.pipelines in the Collector's YAML config, that carries one signal type: traces, metrics, logs, or profiles. To configure the Collector, define receivers, processors, and exporters in their respective sections, then connect them under service.pipelines in execution order. Components not referenced by a pipeline remain inactive. Connectors link pipelines, while extensions are enabled separately under service.extensions. A minimal traces pipeline looks like this:
Pipeline names follow the type[/name] format, so traces, traces/sampled, and metrics/internal can coexist in one config. That naming scheme lets you run multiple pipelines for the same signal type, each with its own processor chain and export destinations.
How Collector Pipelines Work
Data flows through a pipeline in one direction. Receivers ingest before processors mutate or drop data. Exporters then ship the result.
Receivers, Processors, and Exporters
Receivers get data into the Collector, either by listening for pushed telemetry (OTLP, Zipkin) or by scraping it (Prometheus, hostmetrics). A receiver converts incoming data to the Collector's internal representation and hands it to the first consumer in the pipeline. From there, each processor runs in sequence and passes its output to the next stage.
Exporters sit at the end and translate the internal representation into a backend's wire format. When a pipeline lists multiple exporters, a fan-out consumer forwards telemetry to each one, copying data when necessary to prevent modifications from affecting other exporters. Handoffs at every boundary are plain function calls, a detail that matters later when we look at fan-out behavior.
The Service Block and Pipeline Definition
The service.pipelines block connects the receivers, processors, and exporters defined in the Collector's YAML configuration. Each pipeline handles one signal type and specifies where telemetry enters, which processors handle it, and where the resulting data is exported. The processor list also determines execution order. The following example shows how a single OTLP receiver definition supports separate pipelines for traces, metrics, and logs:
Each pipeline passes telemetry through memory_limiter and batch before exporting it through otlp_http. The three pipelines share one OTLP receiver instance, but each has its own processor instances. The service.pipelines block determines how these components are connected and which signals they handle.
OTLP as the Primary Protocol
OTLP is the Collector's native protocol for both receiving and exporting, and it comes in two transports. OTLP/gRPC sends unary Protobuf RPCs exclusively over HTTP/2 and defaults to port 4317; OTLP/HTTP sends POST requests carrying binary or JSON-encoded Protobuf to port 4318, with signal paths like /v1/traces and /v1/metrics.
In Kubernetes, gRPC multiplexes requests over one long-lived TCP connection, and standard Layer 4 Services balance connections rather than individual requests, so an agent can hold one connection to a single gateway pod and send all of its telemetry there.
Two approaches address that connection pinning:
- Keep gRPC and rebalance connections: Put an L7-aware load balancer such as Envoy in front of the gateway pods, or recycle server-side connections with GoAway frames.
- Use OTLP/HTTP: OTLP/HTTP runs over HTTP/1.1 or short-lived HTTP/2 connections, so it avoids the same connection pinning.
Raw efficiency differences are modest. For example, the Thyme benchmark on AWS EKS at roughly 100,000 log records per second found gRPC adds about 25% CPU overhead versus HTTP when both use zstd compression, and the two perform similarly uncompressed.
Processor Ordering
The processors list inside each service.pipelines entry sets execution order, while the top-level processors section only defines the processors, and getting the order wrong wastes work or drops data at the worst moment. Use the following recommended processor order, including only the processors you configure:
memory_limiter: Checks the Collector's memory usage and refuses incoming data when memory pressure becomes too high. Placing it first allows the Collector to signal upstream senders to slow down or retry before other processors consume more memory.- Sampling and filtering processors: Remove telemetry you don't need before it reaches later stages. This reduces the amount of data that other processors must handle, saving processing time and memory.
- Processors that depend on request context: Components such as
k8sattributescan use information from incoming connections to identify the Kubernetes workloads producing telemetry. Run them beforebatch, which can discard the original request context. - Transform and enrichment processors: Components such as
attributesandtransformadd, modify, or remove telemetry attributes. At this stage, they process the data that remains after filtering and prepare it for export. batch: Groups processed telemetry into larger batches before export, reducing the number of requests sent to the backend. Place it last so filtering, enrichment, and transformations happen before data is batched.
Fan-Out Across Multiple Pipelines
A single receiver can feed multiple pipelines without creating a separate receiver instance for each one. For example, the traces and traces/sampled pipelines can both use the otlp receiver while applying different processors to the incoming spans. The Collector creates one receiver instance and uses a fan-out consumer to send telemetry to each pipeline's processor chain. Each pipeline has its own processor instances and state. The Collector also uses the processors' MutatesData capability to determine when telemetry needs to be copied, preventing changes in one pipeline from affecting another.
However, sharing a receiver does not make the pipelines independent. The Collector forwards data through synchronous function calls, meaning a processor must return before the receiver can complete that forwarding operation. If a processor in traces/sampled blocks, it can also delay the traces pipeline and prevent the shared receiver from forwarding new spans. This means a problem in one pipeline can affect others that use the same receiver.
The sending_queue helps when an exporter is slow. Exporters using exporterhelper enable it by default, with 10 consumers and a capacity of 1,000 requests. The queue allows the exporter to accept telemetry without waiting for every request to reach the backend. However, it cannot prevent delays caused by a processor blocking earlier in the pipeline. Once the queue fills, it also starts rejecting new data by default. Keep processor chains short, monitor exporter queues, and use separate Collector instances when pipelines require stronger failure isolation.
Key Components and Capabilities
Four component families do most of the practical work in production pipelines: the stability pair (memory_limiter and batch), OTTL-driven filters and transforms, connectors, and extensions.
Batch Processor and Memory Limiter
The memory_limiter processor helps prevent the Collector from exhausting its available memory. You must configure either limit_mib or limit_percentage, since neither has a usable default.
The processor uses two thresholds: a soft limit at which it starts refusing data and a hard limit at which it forces garbage collection. With limit_mib: 4000 and spike_limit_mib: 800, the soft limit is 3200 MiB, while the hard limit is 4000 MiB.
Recommended Memory Limiter Settings
- Memory limits: Use
limit_percentage(around 80%) in containers. Uselimit_mibon bare metal or VMs with known throughput. - Check interval: Set
check_intervalto1sto check memory usage regularly. - Runtime: Set the
GOMEMLIMITenvironment variable to 80% of the hard limit and account for additional process memory overhead beyondlimit_mib.
The memory_limiter processor has alpha stability for profiles, so check its compatibility before using it in a profiles pipeline.
The batch processor groups telemetry into fewer export requests to reduce network and backend overhead. It sends a batch when it reaches send_batch_size (default 8192) or when timeout expires (default 200ms). The send_batch_max_size setting defaults to 0, meaning there is no configured upper limit on the number of items in a batch.
These processors help the Collector manage memory pressure and export overhead. The memory_limiter controls how much data the Collector can accept under memory pressure, while batch controls how telemetry is grouped before export.
Filtering and Transforming With OTTL
The OpenTelemetry Transformation Language (OTTL) lets you evaluate conditions and modify telemetry using OpenTelemetry data fields. An OTTL statement can reference paths such as resource.attributes["key"], apply conditions using where and comparison operators, and call functions such as set, delete_key, and IsMatch.
The filter processor uses OTTL conditions to discard telemetry, while the transform processor modifies data that continues through the pipeline.
The filter processor drops telemetry when any configured condition matches. For multiple conditions, it uses OR logic.
Here, the trace condition drops matching spans, while the log conditions discard records containing "password" or having a severity below WARN. The IsMatch condition drops the entire matching log record rather than masking the sensitive text.
To modify telemetry without dropping it, use the transform processor:
This example copies the Kubernetes namespace into a new resource attribute, removes the original attribute, and limits string values in span attributes to 4,096 bytes.
Both processors default error_mode to ignore, which logs statement errors and continues processing. Setting it to propagate returns the error to the pipeline, potentially causing the affected data to be dropped.
OTTL support for profiles is still at development stability, so verify which operations your Collector version supports before using it with profile data.
Connectors
A connector links two pipelines inside one Collector process. It acts as an exporter at the end of one pipeline, and as a receiver at the beginning of another, so telemetry passes from the first pipeline into the second. Each connector must appear in both roles, in separate pipelines:
The span_metrics connector is the canonical connector use case: it ends the traces pipeline and consumes every span that reaches it, then aggregates Request, Error, and Duration (RED) metrics per unique set of dimensions. It emits them into the metrics pipeline as traces.span.metrics.calls (a count of spans per dimension set) and traces.span.metrics.duration (a histogram of span durations), flushing every 60s by default.
Placement interacts with sampling: span-to-metrics conversion must run before any tail sampler discards spans; otherwise your RED metrics undercount. The routing connector rounds out the family by steering telemetry to different pipelines based on OTTL conditions against resource attributes.
Extensions
Extensions provide capabilities outside the data path, including health checks, diagnostics, persistent storage, and authentication. The health_check extension exposes an HTTP endpoint, which defaults to localhost:13133. For Kubernetes HTTP probes targeting the Pod IP, configure an address the probes can reach while restricting access to the endpoint. Leave check_collector_pipeline disabled because it does not work reliably.
Beyond health checks, three extension groups cover most production needs:
- Diagnostic endpoints: pprof (default
localhost:1777) provides a Go profiling endpoint for diagnosing CPU and memory behavior, while zpages (defaultlocalhost:55679) provides live in-process diagnostics, including a/debug/pipelinezview of running pipelines. - Persistent storage: file_storage provides bbolt-backed persistence for exporter sending queues, so queued data survives a restart.
- Authentication: Authenticator extensions support mechanisms such as basic authentication, bearer tokens, and OAuth2 client credentials. They can attach credentials to outgoing exporter requests, while basic auth and bearer token authenticators can also authenticate incoming receiver connections.
To avoid storing credentials directly in YAML, place them in environment variables and reference them through the Collector's environment variable config provider (envprovider). The exporter or receiver then references the configured authenticator extension.
The Core distribution includes envprovider for reading environment variables, but the authenticator extensions themselves are only available in otelcol-contrib and otelcol-k8s.
Deployment Topologies and Distributions
Pipeline configs differ depending on whether the Collector runs beside the workload or as shared infrastructure, and on which binary you deploy.
Agent vs. Gateway
An agent Collector runs on every host, and a gateway runs as a centralized service. In the combined agent-to-gateway pattern, the two tiers split responsibilities:
- Agent responsibilities: Agents in a DaemonSet collect from services on their node and gather the node's own telemetry. Use the agent tier for metadata enrichment with
k8sattributes, log file collection, and host metrics. - Gateway responsibilities and justification: Gateway collectors receive data from agents and perform centralized processing such as filtering and sampling before exporting data to backends. Add a gateway when processing needs a global view, as tail-based sampling does, when one egress point should hold backend credentials, or when the deployment requires network isolation.
Choose the agent-only pattern when you need to collect and process telemetry on individual nodes without centralized processing. Use gateway-only when applications or other Collectors send OTLP to a shared endpoint and node-local collection isn't required. Use both when agents need to collect telemetry locally while gateways handle processing that requires data from multiple nodes, such as tail-based sampling. This requires deploying and managing both sets of Collectors.
When running multiple gateway replicas, follow the single-writer principle: each metric data stream must have one writer and a globally unique identity. For example, running the same Prometheus receiver configuration on every replica can cause duplicate scrapes. Assign scrape targets to specific Collector instances to avoid collecting the same metrics more than once.
Kubernetes Patterns
The OpenTelemetry Operator supports four deployment modes: DaemonSet, Sidecar, StatefulSet, and Deployment. A DaemonSet runs a Collector on each node, making it suitable for receivers such as kubeletstats and filelog. Sidecars run alongside individual applications and can help distribute gRPC connections across gateway replicas without an L7 load balancer. They also isolate collection failures to individual pods, although the k8sattributes processor isn't supported in sidecar mode.
Scaling Collectors introduces a problem for tail-based sampling. All spans belonging to a trace must reach the same Collector instance, or different replicas may make inconsistent sampling decisions. To address this, use a load-balancing exporter in the first tier to route spans by trace ID to a second tier running the tail sampler. Capital One reduced tracing data volume by 70% using a layered Collector architecture, while Skyscanner processes 1.8 million spans per second with fewer than 100 Collector replicas and retains 4.5% of traces.
The tail sampler holds spans in memory while waiting to make a decision. Its decision_wait defaults to 30 seconds, while num_traces defaults to 50,000. When the buffer fills, older traces can be removed before sampling decisions are made. If otelcol_processor_tail_sampling_sampling_trace_dropped_too_early increases, consider raising num_traces, accounting for the additional memory required.
Tail sampling requires the Contrib tailsamplingprocessor. Place it after processors that depend on request context, such as k8sattributes, and generate RED metrics before sampling discards spans. If you don't need tail-based decisions, the probabilistic_sampler processor provides an alternative.
Core vs. Contrib Distributions
The Collector distribution you install determines which receivers, processors, exporters, and extensions are available. Core (otelcol) includes common components, while Contrib (otelcol-contrib) adds components such as the Kubernetes attributes processor, tail-sampling processor, and load-balancing exporter.
Before configuring an exporter, check whether your distribution includes it. Core provides otlpexporter and otlphttpexporter (configured as otlp_http), which can send telemetry to backends that accept OTLP. It also includes exporters for Prometheus, Zipkin, Kafka, and files. Contrib supports additional destinations, including AWS, Azure, Google Cloud, ClickHouse, Elasticsearch, Datadog, and Splunk. For backends such as Jaeger, Tempo, or OpenSearch, use an OTLP exporter if the backend accepts OTLP. Otherwise, you'll need a distribution containing a compatible exporter.
Using Contrib when you only need a few components increases binary size and can expand the attack surface. The OpenTelemetry Collector Builder (ocb) lets you build a distribution containing only the components you need. You can also use existing distributions such as otelcol-k8s or otelcol-otlp when they include the required components.
Unified OpenTelemetry and eBPF Observability With groundcover
groundcover combines telemetry from your existing OpenTelemetry Collector pipelines with data captured by its eBPF sensor. You can keep your current instrumentation and export destinations while using both sources to investigate application and infrastructure behavior.
Send OpenTelemetry Data to groundcover
To send Collector telemetry to groundcover, add an otlp_http exporter configured with your BYOC endpoint and ingestion key. List it alongside your existing exporters under service.pipelines, so traces, metrics, and logs can reach groundcover without replacing your current backend. You can follow the Collector integration guide to configure the exporter and authentication. Any sampling must happen in the Collector because groundcover's BYOC ingestion endpoint does not sample incoming telemetry.
Under groundcover's BYOC architecture, the observability data plane runs in your own cloud account. ClickHouse stores logs, traces, and events, while VictoriaMetrics stores metrics with PromQL compatibility. This keeps telemetry in your environment while allowing you to query it through groundcover.
Combine OpenTelemetry With eBPF Telemetry
The groundcover eBPF sensor runs as a DaemonSet on Kubernetes nodes and collects telemetry without requiring changes to application code. It captures service activity, dependencies, traces, logs, metrics, and Kubernetes events, including information from workloads that haven't been instrumented with OpenTelemetry.
OpenTelemetry SDK spans provide application-level context from instrumented services, while eBPF adds visibility into other workloads and their interactions. groundcover correlates these signals with Kubernetes workload information, allowing you to investigate services and dependencies without switching between separate monitoring tools.
Investigate Telemetry With Agent Mode
groundcover Agent Mode lets you investigate logs, traces, metrics, and Kubernetes events using natural language. You can ask why a service is returning errors, identify changes in a namespace, or examine dependencies between workloads. Agent Mode runs queries against the collected telemetry and makes those queries available for inspection.
It can also create dashboards, monitors, and OTTL-based log parsing or drop rules. This lets you investigate data from your OpenTelemetry pipelines alongside eBPF-collected telemetry and use the findings to improve monitoring and filtering.
Validating and Troubleshooting Pipelines
Run otelcol validate --config=customconfig.yaml before every deploy. The subcommand does two things: it validates the configuration and YAML, then builds every component without running it, surfacing errors from either step. That catches unparseable YAML, references to undefined components or extensions, pipeline cycles, and missing component factories.
Once validation succeeds, start the Collector with otelcol --config=customconfig.yaml. If you're using the Contrib distribution, use otelcol-contrib instead. Validation checks the configuration, but the Collector must be running before you can verify that telemetry reaches its destination.
Know what validate cannot tell you. It does not test whether receivers can bind their ports and accept data, whether the Collector can authenticate to a backend, or whether exporters can reach their endpoints, so unreachable endpoints, bad credentials, and ports that fail to bind all pass validation and fail at runtime.
Four failure patterns recur in practice:
- Distribution mismatches: Factory checks are distribution-specific, so a config that validates against
otelcol-contribfails on the Core binary with a missing-factory error. OTelBin's per-distribution validation catches this before deploy. - Receiver not enabled in a pipeline: A receiver declared at the top level but absent from
service.pipelinesdoes nothing, because configuration outside the service block is inert. A receiver listed only in another signal's pipeline does not ingest your signal either. - Incorrect exporter endpoints: The config validates, but exports fail at runtime. The exporter retries with its defaults (
retry_on_failureenabled, a 5s initial interval, a 30s maximum interval, and giving up after 300s) while thesending_queue(default size 1000) fills, and once the queue is full, the exporter rejects new data rather than blocking. Check that the port and transport match the backend (4317 for OTLP/gRPC, 4318 for OTLP/HTTP), and look for export errors in the Collector's own logs. - Authentication failures: Bad or missing credentials also pass validation, and the backend rejects your requests at runtime. Configure credentials through an authenticator extension (basic auth, bearer token, or OAuth2 client credentials) and reference it from the exporter. These extensions ship in the
otelcol-contribandotelcol-k8sdistributions, not Core, and you must define every extension and list it underservice.extensions.
When telemetry isn't arriving, work through these checks in order, from the Collector's inputs to its outputs:
- Check receiver configuration and pipeline references. Confirm that the receiver's endpoint and protocol match what your senders use and that the receiver is listed under
service.pipelinesfor the correct signal. - Add the debug exporter to confirm that data reaches the pipeline.
- If data appears in the debug exporter, check exporter endpoints, credentials, and connectivity. Look for export errors in the Collector's logs.
To run the second check, wire the debug exporter into a pipeline next to your real exporter:
The debug exporter behaves as follows:
verbosity: basic(the default) prints a one-line summary per batch,normalprints roughly one line per record, anddetailedprints the full payload.sampling_thereafteralready defaults to 1, which disables output sampling, so you don't need to set it while debugging.- By default, the output goes through the Collector's own logger.
- The output format is unstable and can change at any time without a breaking change, so never parse it in automation.
FAQs
Yes. The Collector runs processors in the order listed under service.pipelines. Place memory_limiter first to handle memory pressure, filtering and sampling processors before enrichment, and batch last to avoid batching data that other processors would discard.
The Collector creates one receiver instance and forwards its data to each pipeline's separate processor chain. Because this forwarding uses synchronous calls, a blocking processor in one pipeline can delay the others sharing that receiver.
Receivers ingest telemetry, while exporters send it to destinations. A connector links pipelines within the same Collector process, acting as an exporter in one and a receiver in another. For example, span_metrics converts trace spans into RED metrics.
Use an agent for node-local collection and a gateway for centralized processing, such as tail-based sampling. Run both when you need local collection and processing across multiple nodes. A gateway can also run alone when applications send telemetry directly to it.
memory_limiter controls memory pressure by refusing data when usage exceeds its soft limit and triggering garbage collection at the hard limit. batch groups telemetry into fewer export requests, reducing processing and network overhead. Together, they help the Collector handle load, although they cannot prevent every overload failure.
