Building an OpenTelemetry backend architecture

OpenTelemetry standardizes telemetry production and transport: SDKs generate traces, metrics, and logs, and OTLP carries them. Storage and query belong to the backend, and OpenTelemetry deliberately leaves that choice to you. The Collector's exporter layer is where you make that choice, and it is the one point in the architecture that determines whether you can swap backends next year or whether you are locked in.

What an OpenTelemetry backend architecture is

An OpenTelemetry backend architecture is the set of storage and query systems that sits behind the Collector, plus the pipeline design that routes telemetry to them. OpenTelemetry itself covers instrumentation (SDKs and auto-instrumentation) and transport (OTLP over gRPC on port 4317 or HTTP on port 4318). It ships no database and no query engine.

That gap is deliberate. The OTLP specification, stable at version 1.11.0 for traces, metrics, and logs, defines the wire format, and any backend that accepts OTLP can receive your telemetry. Vendor neutrality falls out of this separation: as long as your services emit OTLP and your Collector exports OTLP, the backend behind it is a configuration detail rather than an architectural commitment.

Core Collector components: receivers, processors, exporters, pipelines

A Collector pipeline has three stages. Receivers ingest telemetry. Processors transform it in order. Exporters deliver it. A minimal trace pipeline uses the OTLP receiver to accept spans, the memory_limiter and batch processors to protect the process and reduce network chatter, and an OTLP exporter pointed at a trace backend such as Jaeger. The Jaeger project built v2 on the Collector framework, and Jaeger v2 accepts OTLP natively.

Two distributions matter for backend work, both at v0.157.0 as of July 2026:

• Core (otelcol): The minimal build ships OTLP, Prometheus, Jaeger, and Zipkin receivers; the batch, memory_limiter, and probabilistic sampling processors; and OTLP gRPC/HTTP, Prometheus, and Prometheus Remote Write exporters.
• Contrib (otelcol-contrib): The community build adds the components a serious backend design needs: the load_balancing exporter, the tail_sampling processor, the k8sattributes processor, and the ClickHouse exporter, among hundreds of others.

The project also publishes otelcol-k8s, a curated Kubernetes subset that keeps the load balancing exporter and tail sampling processor while excluding vendor-specific components.

Pipelines fan out. When two pipelines declare the same receiver, each pipeline gets its own copy of the data, so one OTLP receiver can feed a trace pipeline bound for Jaeger and a second pipeline bound for a long-term archive without either interfering with the other.

The three signal types and their pipelines

Traces, metrics, and logs each run through a separate pipeline declared under service.pipelines, and every component is typed per signal. Maturity differs by signal even within one component: the load_balancing exporter is beta for traces and logs, but the load balancing README still lists metrics as in development. Separate pipelines also mean separate processor chains, so you can tail-sample traces while shipping logs unsampled through a different set of processors.

Why OTLP is the decoupling point

Standardize on OTLP at every Collector boundary and the backend becomes swappable. That holds only if nothing proprietary leaks into the pipeline, which is why vendor-specific exporters belong at the edge of your topology or nowhere. The spec also protects you from version skew between tiers: "Old clients must be able to talk to new servers and vice versa," so agent, gateway, and backend upgrades don't have to move in lockstep.

Agent vs. gateway: choosing the right deployment topology

An agent Collector runs next to the workload, as a DaemonSet pod on each node or a sidecar in each pod. A gateway Collector runs as a centralized, horizontally scaled service that all agents forward to. The official deployment docs describe the resulting network layout: "Applications communicate with local agents using the internal host network, agents communicate with gateways over the internal cluster network, and gateways securely communicate with external backends using TLS."

Agents earn their place through locality. They collect node-local sources such as kubelet stats and container log files. They also enrich telemetry with host metadata and reduce what crosses the network. Gateways concentrate the concerns you don't want copied onto every node: backend credentials and egress TLS, plus tail sampling, rate limiting, and fan-out to multiple backends.

The gateway tier carries one hard rule from the official gateway guidance: follow the single-writer principle, because "multiple collectors must not modify or report on the same data series concurrently." Small deployments can skip the gateway and let agents export directly. Add the gateway tier when you need tail sampling, multiple backends, or credential isolation.

Kubernetes-native deployment patterns

Kubernetes gives you three ways to run Collectors, and they map to different problems:

• DaemonSet: One Collector per node handles node-scoped collection: container log files, kubelet stats, host metrics. This is the standard agent tier, and its resource cost scales with node count rather than pod count.
• Sidecar: One Collector per pod gives per-workload isolation and configuration. The OpenTelemetry Operator injects it via the sidecar.opentelemetry.io/inject annotation, which must sit on spec.template.metadata.annotations, and it sets OTEL_RESOURCE_ATTRIBUTES with Kubernetes resource attributes automatically.
• Operator-managed: The OpenTelemetryCollector CRD's .spec.mode accepts Deployment (the default), DaemonSet, StatefulSet, and Sidecar, so one declarative resource covers both agent and gateway tiers.

The Operator (v0.156.0, released July 2026) does more than deploy Collectors:

• Instrumentation CRD: Injects auto-instrumentation per language via annotations, with Java, Node.js, Python, .NET, and Apache HTTPD enabled by default and Go and NGINX behind feature gates.
• Target Allocator: Decouples Prometheus service discovery from metric collection, distributing scrape targets across StatefulSet or DaemonSet Collectors.
• Collector upgrades: Upgrades managed Collectors automatically, which cuts one recurring maintenance task but means you should pin versions deliberately if you need to hold one back.

Scaling the Collector: multi-tier architecture and backpressure

At hundreds of services, the standard topology is two tiers: a DaemonSet agent tier that collects and enriches locally, forwarding to a horizontally scaled gateway tier that samples, routes, and exports. Whether a traffic spike loses data comes down to two controls inside each Collector.

The first is the memory_limiter processor, which must be the first processor in every pipeline. It enforces a soft limit (limit_mib minus spike_limit_mib, roughly 20% of the hard limit) at which it refuses incoming data and pushes backpressure to receivers, and a hard limit at which it forces garbage collection. The README carries a warning worth reading twice: data is permanently lost if the component preceding the memory limiter does not retry after refusal, which is exactly why agents with retry-enabled OTLP exporters should sit in front of gateways. Pair it with GOMEMLIMIT set to 80% of the container's memory limit; the Kubernetes Helm chart defaults to limit_percentage: 80 with spike_limit_percentage: 25.

The second control is the exporter queue, which absorbs backend slowness. The official scaling guide gives concrete signals: scale out when otelcol_exporter_queue_size reaches 60–70% of otelcol_exporter_queue_capacity, watch otelcol_processor_refused_spans for memory pressure, and keep at least three replicas per tier. Do not scale when otelcol_exporter_send_failed_spans is rising; that signals a failing backend, and adding Collectors only adds pressure on it.

Neither control fixes CPU saturation. In the project's own CI load tests, the 10k spans/sec no-backend case failed at roughly 122% CPU, and the memory-limited variant still consumed 113.8%. Size CPU per expected throughput; Splunk's sizing guidance for its distribution suggests planning around 15,000 spans/sec per core.

Sampling strategies and trace-aware load balancing

Head-based sampling decides at the root span, before the trace's outcome exists. The probabilistic sampler ships in core, costs almost nothing, and treats a failed checkout identically to a healthy health check. Tail-based sampling waits for the whole trace: the tail_sampling processor holds up to 50,000 traces in memory (the num_traces default), waits decision_wait (default 30s), then applies policies such as latency, status_code, probabilistic, rate_limiting, and ottl_condition, so you can keep every error and slow request while dropping most healthy traffic.

Tail sampling imposes a hard constraint: "All spans for a given trace MUST be received by the same Collector instance." A round-robin load balancer in front of a scaled gateway tier violates this on every request, scattering one trace's spans across instances so each Collector evaluates a fragment and decides wrong. The fix is the load_balancing exporter running in a routing tier ahead of the sampling tier, with routing_key: traceID (the default for traces) hashing every span of a trace to the same downstream instance. It resolves backends via static, dns, k8s, or aws_cloud_map resolvers; the k8s resolver needs a service account with get/list/watch on EndpointSlice objects, and missing RBAC leaves the resolver cache empty.

Know the failure modes before you rely on this pattern:

• Route churn: When the backend list changes, the load-balancing exporter reroutes approximately R/N routes (R total routes, N backends), which breaks trace completeness during scale events.
• No default failover rerouting: The exporter does not re-route data to a healthy endpoint on delivery failure by default.
• Duplicate retry risk: Open issues include a duplicate retry issue.
• Single-gateway recommendation: The official docs recommend preferring "a single well-resourced tail-sampling gateway unless you have a robust sticky-routing strategy."

Self-hosted open-source backend options

Each open-source backend commits to a different storage architecture, and that choice sets your operational profile more than any feature list:

Jaeger v2 is a single binary on the Collector framework, configurable as collector, query, ingester, or all-in-one, with Kafka available as a persistent queue between collectors and storage. The docs recommend OpenSearch over Cassandra for large-scale production, and the ClickHouse backend sits behind a feature gate as experimental. Jaeger v1 reached end-of-life on December 31, 2025, so any v1 deployment should treat migration as urgent.

Tempo takes the opposite bet: no index cluster at all. It writes Apache Parquet blocks to object storage, prunes trace-ID lookups with bloom filters, and in microservices mode runs at replication factor 1 because the Kafka WAL provides durability. Grafana Labs reported tested clusters at 600 MB/s and 1.2 million spans per second. The trade is query latency against storage cost: scanning Parquet in S3 is slower than an indexed lookup, and lower-cost to retain.

Prometheus remains the metrics default. Its own docs are direct about the ceiling: "Prometheus local storage is not clustered or replicated: it is not arbitrarily scalable or durable in the face of drive or node outages and should be managed like any other single node database." Default retention is 15 days, and a single instance tops out at tens of millions of active series.

SigNoz consolidates everything into three runtime components (the SigNoz binary, a SigNoz OTel Collector, and ClickHouse), which makes it the fastest path to an all-in-one self-hosted stack. The concentration cuts the system count and moves the burden into ClickHouse itself: one independent review notes that managing ClickHouse at production scale can act as "a 'second job' for small teams."

Routing telemetry to multiple backends with OTLP exporters

Fan-out is a configuration change, not an architecture change. List multiple exporters in one pipeline and each receives a full copy of the data:

service:  pipelines:    traces:      receivers: [otlp]      processors: [memory_limiter, batch]      exporters: [otlp/jaeger, otlp/tempo]

Because both exporters speak OTLP, the backends differ only by endpoint. This is also the migration pattern that makes vendor neutrality practical: dual-write to the incumbent and the candidate backend for a few weeks, compare query results, then delete one exporter. No re-instrumentation, no agent swap.

Metrics have an equivalent escape hatch: Prometheus supports remote write and read integrations with external storage systems.

Cross-signal correlation across traces, metrics, and logs

Correlation depends on shared identity, not on the backend. A trace ID in a log line is only useful if the log and the span carry the same service.name and resource attributes, which is what OpenTelemetry's semantic conventions standardize. Get this right in the pipeline: the Operator's sidecar injection stamps Kubernetes resource attributes automatically, and the k8sattributes processor in contrib does the same enrichment at the Collector for workloads that lack it.

The failure modes are quiet. One self-hosted SigNoz report found that if an application emits logs after a span closes, those logs arrive with empty trace_id and span_id, silently breaking log-to-trace navigation in the UI. On the metrics side, Prometheus exemplars link trace IDs only, a thin bridge compared to attribute-level correlation.

Splitting signals across backends adds a coordination tax on top. Grafana's LGTM stack means operating separate systems for logs, metrics, and traces. A single investigation crosses three query dialects. Consistent attributes make that crossing possible; nothing makes it free.

Security between Collector tiers and backends

The Collector's TLS configuration defaults are sane: insecure and insecure_skip_verify are false, and min_version is TLS 1.2. For the agent-to-gateway hop inside the cluster, enable mutual TLS by setting client_ca_file on the gateway's receiver, which switches client verification to RequireAndVerifyClientCert; reload_interval handles certificate rotation without restarts. Since Collector v0.110.0, servers bind to localhost by default, and the security guidance is explicit on the rest: avoid 0.0.0.0 bindings, avoid running the Collector as root, and "Your OTel Collector configuration should include encryption and authentication."

Backend authentication runs through extensions rather than exporter config. The gateway's exporters attach bearertokenauthextension for static token auth (default header Authorization, scheme Bearer), oauth2clientauthextension for client-credentials flows with automatic token refresh, or sigv4authextension for AWS-signed requests. On the receiving side, oidcauthextension validates JWTs on incoming traffic, so a gateway can require identity from every agent. Add the redaction processor before egress to strip attributes that don't match an allowlist.

The operational cost of self-hosting

Every backend in this article works, and every one of them bills you in engineering time.

• Engineering overhead: The full open-source stack means Prometheus, Loki, Tempo, and Alloy or an OTel Collector as separate systems, each with its own versioning, configuration, and upgrade path.
• Storage dependencies: Tempo's production mode requires a Kafka-compatible WAL, and distributed ClickHouse under SigNoz requires ZooKeeper or ClickHouse Keeper.
• HA caveats: The Prometheus pattern of two parallel servers duplicates data, "causing graphs to look jagged."
• Version churn: v0.157.0 replaced the histogram bucket boundaries on batch-size metrics, which breaks any dashboard or alert with hard-coded le values, and contrib fixed metric units across a dozen receivers with the same dashboard-breaking effect.

There is a middle path between running that fleet and shipping telemetry to a SaaS vendor's cloud. groundcover's BYOC deployment (Bring Your Own Cloud: your data plane runs in your own cloud account) keeps ClickHouse and VictoriaMetrics inside your VPC while groundcover manages the control plane and orchestration. It ingests OTLP as a first-class source alongside its eBPF signals. Your existing Collector pipelines keep working; the OTLP exporter points at an endpoint in your own VPC instead of a self-managed Jaeger or Tempo cluster. You can validate the full BYOC architecture in the free tier at $0/host/month on a single cluster before touching the rest of your topology.