Securing customer-owned telemetry planes: architecture, threats, and controls

Moving your telemetry plane into your own VPC removes a vendor's cloud from your threat model and adds your own. Every telemetry component that used to be someone else's problem becomes infrastructure you patch and credentials you rotate, with evidence you defend, and the stakes are concrete: Aqua Security found exposed Prometheus servers, over 336,000 servers and exporters exposed to the public internet, queryable for API keys and credentials. Securing customer owned telemetry plane infrastructure starts with knowing exactly what you own.

What is a customer-owned telemetry plane

A customer-owned telemetry plane is the infrastructure you control for the full lifecycle of your observability signals: logs, metrics, traces, and Kubernetes events. It spans collectors on your nodes, brokers or pipeline processors in the middle, and storage backends like ClickHouse and VictoriaMetrics at the end.

The data plane holds and moves the telemetry itself; the control plane handles orchestration, configuration, and the UI. Ownership models split along that line:

• SaaS: The vendor owns both planes. Your telemetry egresses to vendor-managed infrastructure, and the vendor has full access to raw data.
• BYOC (Bring Your Own Cloud): Your data plane runs in your own cloud account. The vendor manages the control plane and the software lifecycle, but raw telemetry stays inside your VPC. ClickHouse documentation draws the line precisely: "BYOC separates the ClickHouse control plane, which runs in the ClickHouse VPC, from the data plane, which runs entirely in your cloud account."
• Self-hosted: You own both planes. Elastic Cloud Enterprise, for example, "supports sovereign cloud by running entirely on your infrastructure," with no external connectivity required, including fully air-gapped deployments.

Your data plane location sets data residency. WarpStream states that raw data never leaves your VPC or object storage buckets; Redpanda's BYOC architecture documentation makes the same guarantee for its data plane. That residency is why BYOC and self-hosted models are attractive in regulated sectors, and it is also why the security burden shifts to you: the vendor no longer stands between your telemetry and an attacker.

How securing a customer-owned telemetry plane works

Securing the plane means applying controls at each pipeline stage, then validating the failure modes attackers use to steal telemetry or manipulate the pipeline. Security controls map to the stages and trust boundaries attackers target.

Signals and pipeline stages

Four signal types flow through the plane: logs, metrics, traces, and events. They pass through three stages, and each stage takes different controls:

• Collector: DaemonSets, OpenTelemetry Collectors, and log shippers run on your nodes. This is where redaction, authentication of inbound data, and TLS origination belong, because anything scrubbed here never exists downstream.
• Broker or pipeline: Kafka, NATS, or a processing tier buffers and routes signals. Controls here are mutual authentication, replication and durability settings, and integrity verification at ingestion.
• Storage: ClickHouse holds logs, traces, and events; VictoriaMetrics holds metrics. Controls here are encryption at rest, RBAC on queries, retention policy, and immutability for audit-grade data.

Redaction at storage time means secrets already transited your network in cleartext, and authentication only at storage means anyone who can reach the broker can inject forged telemetry.

Threat vectors and attack surface

Security researchers documented five recurring incident vectors from 2024 to 2026, with credentials written into telemetry storage as the most frequently exploited class:

• Secrets in logs and metrics: Fluentd's Monitor Agent REST API leaked database passwords, API keys, and cloud credentials embedded in plugin configuration (CVE-2026-44025, CVSS 7.5; Fluentd fixed it in 1.19.3).
• Unencrypted or improperly verified transport: Logstash's TCP output plugin failed to validate TLS certificates even with ssl_verification_mode => full configured (CVE-2025-37730), opening telemetry in transit to man-in-the-middle interception.
• Exposed collector and dashboard endpoints: Beyond the 336,000 exposed Prometheus instances, Grafana public dashboards leaked all direct data-source passwords, including for datasources no dashboard used (NVD CVE record). Up to v0.103.0, OTLP receivers in the OpenTelemetry Collector bound to 0.0.0.0 by default, accepting unauthenticated data from any source.
• Supply chain compromise: The SolarWinds SUNBURST breach inserted malicious code into a management platform's software update mechanism and reached U.S. government agencies and Fortune 500 firms. The May 2026 "Mini Shai-Hulud" npm campaign went further: it exfiltrated stolen CI credentials to t.m-kosche.com:443/api/public/otel/v1/traces, an OTLP-shaped path, because attackers know telemetry endpoints are whitelisted in egress controls.
• Non-human identity misconfiguration and pipeline suppression: Overprivileged ingest tokens and misconfigured alerting paths turn availability into an integrity risk. An attacker who suppresses your pipeline gets the same outcome: missing detections.

Encryption in transit and at rest

NIST TLS guidance sets TLS 1.2 as the floor and TLS 1.3 where supported, and treat TLS 1.0 and 1.1 as prohibited: RFC 8996 states implementations "MUST NOT negotiate" them. This maps directly to NIST CSF 2.0 subcategory PR.DS-02: "The confidentiality, integrity, and availability of data-in-transit are protected."

Defaults across the pipeline do not enforce this for you:

• OpenTelemetry Collector: configtls defaults min_version to "1.2", but you must configure TLS explicitly; without it, telemetry travels unencrypted between Collector and backend.
• Fluent Bit: ships with tls set to off, and even when enabled, tls.verify_hostname is off by default. Client certificate verification (tls.verify_client_cert), the knob that turns on mTLS for input plugins, is also off.
• OTel option naming: in the otlphttp exporter, insecure: true does not skip CA validation, while in Kafka components insecure disables TLS entirely.

Use mTLS on segments where both endpoints are yours: collector to broker, broker to storage. Server-side TLS proves the destination's identity; mutual TLS also proves the sender's, which is what stops a rogue device on your network from injecting forged telemetry into an otherwise encrypted stream. Kafka deserves a specific audit here, because its PLAINTEXT protocol "provides no security" and security.inter.broker.protocol defaults to PLAINTEXT if not specified.

At rest, self-hosted storage encryption is your responsibility: Elastic, for example, requires customer-managed disk-level encryption such as dm-crypt for self-managed clusters.

Access control and role separation

Least privilege in a telemetry plane means separating three populations: pipeline operators, security consumers, and the non-human identities (NHIs) that make up most of its access graph. NIST SP 800-53 Rev. 5 AC-6(2) requires that accounts with access to security functions use non-privileged accounts for everything else; applied here, the engineer who tunes collector configs should not hold the role that can delete audit indices, and the security analyst reading detections should not hold pipeline admin.

NHIs concentrate the exposure because they outnumber users and often lack lifecycle controls. Cloud Security Alliance NHI researchputs NHIs at 45:1 over human users on average, and 144:1 in cloud-native environments, and only 20% of organizations have formal offboarding and revocation processes for API keys.

GitGuardian measured the persistence problem: 60.4% of flagged NHI issues are long-lived secrets, and 64% of secrets leaked in 2022 remained valid in January 2026. Static bearer tokens used by OpenTelemetry collectors are, in GitGuardian's words, "notoriously difficult to rotate across hundreds of microservices without downtime."

The fix NIST prescribes is short-lived, cryptographically verifiable workload identity. SP 800-207A recommendation ID-SEG-REC-2 states that each service "should present a short-lived cryptographically verifiable identity credential to other services that is authenticated per connection and reauthenticated regularly," and it names SPIFFE/SPIRE as the mechanism, with operators typically rotating X.509-SVIDs at ~80% of a roughly 1-hour lifetime. Give each pipeline component its own identity: sharing one monitoring service account across collector instances is exactly the pattern OWASP catalogs as NHI9:2025, NHI Reuse.

Sensitive data redaction

Redact at the collector or broker stage, before secrets reach storage, and understand precisely what your redaction tooling can and cannot catch. The OpenTelemetry Collector's redaction processor works fail-closed: it deletes attributes not on the allowed_keys allowlist, then masks values matching blocked_values regex patterns, with hashing options including hmac-sha256 and hmac-sha512 (added February 2026 for GDPR-compliant pseudonymization).

Regex-based redaction fails on semantic PII, unstructured bodies, and silent rule failures:

• Regex catches syntax, not semantics: A 2026 benchmark of 1,300 prompts and 4,014 annotated sensitive spans found regex-only detection achieves a zero leak rate on emails, IPs, AWS access keys, PEM markers, and bearer tokens, but misses semantic PIIincluding all person names, organization names, addresses, SSNs, employee IDs, hostnames, and passwords. None of the three major collectors (OTel Collector, Fluent Bit, Vector) includes built-in NER or ML-based detection.
• Unstructured log bodies escape the allowlist: log.body is not natively redacted by the OTel redaction processor; free-text redaction requires parsing bodies into structured fields first.
• Redaction rules fail silently: As of OTel Collector Contrib v0.157.0, the transform processor's default error_mode is permanently ignore, so a misconfigured redaction rule drops nothing and warns no one. A v0.131.0 bug where ^ and $ anchors prevented mid-string matches caused real PII leaks before maintainers fixed it.

Fluent Bit's Modify filter has no value-masking action (Remove_regex deletes the key entirely), and Vector's VRL redact function ships exactly one named preset, us_social_security_number.

Pipeline integrity and anti-tampering

An attacker who can modify telemetry, or the pipeline's own configuration, controls what your security team believes happened. Three controls address this, each mapped to a documented requirement:

• Cryptographic integrity at ingestion and storage: NIST SP 800-53 AU-9(3) requires cryptographic mechanisms to prevent unauthorized modification of audit information, and NIST SP 800-92 specifies the pattern: compute a message digest per log file (SHA, not MD5, for federal use) and protect the digests via FIPS-approved encryption or read-only media. AWS CloudTrail's implementation is a working reference: hourly digest files hashed with SHA-256 and signed with SHA-256/RSA using a per-Region private key.
• Immutable audit logs of configuration changes: Ingest-only tokens with write access to configuration create config-plane tampering risk. Every change to collector configs, sampling rules, and redaction policies belongs in an append-only audit trail, and PCI DSS 10.3.5 requires file-integrity monitoring or change detection on the logs themselves.
• Verified software supply chain: SUNBURST worked because operators trusted the update mechanism implicitly. Pin collector versions, verify signatures on images and packages, and never resolve pipeline dependencies at runtime; the March 2026 axios compromise hit Datadog's own CI action precisely because it ran npx @datadog/datadog-ci@latest with no lockfile or integrity verification.

Key features and capabilities

Beyond the per-stage controls, three architectural decisions determine whether the plane holds up under an active adversary rather than an ordinary outage.

Separating security telemetry from observability pipelines

Observability pipelines are tuned for operational truth, and the tuning mechanisms, especially sampling and relevance filtering, degrade adversarial truth. Call this the Telemetry Intent Gap: the same trace your latency analysis treats as statistically redundant may be the single record of an attacker's lateral movement. OpenTelemetry frames sampling as a way to reduce observability costs without losing visibility, but the security failure mode is sampling blind spots: rare errors and slow requests dropped by head sampling.

The OTel tail sampling processor shows how signal loss happens mechanically. Its circular buffer defaults to 50,000 traces, and the sampler removes the oldest trace when a new trace arrives, which can drop a trace before it is sampled; the processor cannot retroactively upgrade a "do not sample" decision when an error span arrives late; and horizontal scaling splits a trace's spans across collectors that reach different decisions. The official OpenTelemetry docs carve out the case directly: users should avoid sampling if regulation prohibits dropping data and they cannot route unsampled data to low-cost storage.

The architectural answer is a dedicated security pipeline: full-fidelity, unsampled, with its own retention and access policy, running parallel to the cost-optimized observability path. At the collection layer, groundcover's Flora eBPF sensor, for instance, ingests 100% of signals with zero sampling from a single DaemonSet per node.

Audit logging and anomaly detection

The pipeline needs to watch itself. That means auditing every access to pipeline configuration, including who changed it and when. It also means baselining normal collector and broker behavior and alerting on deviations: a collector executing unexpected binaries, an unrecognized device shipping telemetry to your broker, an export destination that was never configured. NIST SP 800-171r3 explicitly lists "failures in audit log capturing mechanisms" and reaching audit log storage capacity as failures you must detect, and SP 800-92r1 requires monitoring logging configurations and operational statuses to catch them.

ML-driven anomaly detection helps here, with one caution that follows from the Telemetry Intent Gap: AI relevance scoring built for noise reduction can suppress security signals. Volume-reduction filters cut data for cost, and a low-frequency attack signal looks exactly like the noise those mechanisms exist to drop. Keep volume-reduction AI out of the security pipeline, or gate it behind rules that exempt authentication, authorization, and configuration events.

Network segmentation, HA, and failover

Isolate the telemetry plane on a dedicated network segment with explicit ingress and egress rules; the Mini Shai-Hulud campaign's OTLP-shaped exfiltration endpoint is proof that attackers count on telemetry egress being unscrutinized. For availability, run brokers Active/Active with quorum-based replication, and audit the durability settings, because several defaults trade acknowledged writes for performance:

• Kafka: unclean.leader.election.enable allows a replica outside the in-sync set to become leader "as a last resort, which may result in data loss." For security telemetry, that means audit records the broker already acknowledged can vanish; the safe setting prefers unavailability over loss, paired with acks=all and min.insync.replicas.
• NATS JetStream: file-based streams do not immediately fsync, using a default 2-minute sync_interval. Jepsen's analysis of NATS 2.12.1 found acknowledged writes could be lost and the cluster entered "a persistent split-brain which continued after all nodes were restarted"; sync_interval: always closes the window at a performance cost.
• OTel Collector: the default sending queue holds 1000 batches, and when it fills during an outage, new data is dropped. OpenTelemetry documents the consequence in the memory limiter README: data "will be permanently lost" if the preceding component does not correctly retry after refusal, and Prometheus remote write permanently loses unsent data to WAL compaction after a remote endpoint is unreachable for more than 2 hours.

Decide explicitly what happens when the telemetry link is lost. Persistent WAL storage on collectors reduces loss but does not eliminate it, and Fluent Bit discards its oldest buffered chunk when storage.total_limit_size is reached. NIST SP 800-92r1 requires you to determine how to handle log storage and transfer errors and to use out-of-band methods where appropriate; an attacker-induced partition that silently drops audit records is a detection failure.

How it fits in: framework mapping and BYOC architecture

Three frameworks your auditors and architects already use cover these controls directly.

• NIST CSF 2.0: Published February 26, 2024, with six functions and 106 subcategories, it covers the plane through PR.DS-02 for transport encryption, PR.PS-04 ("Log records are generated and made available for continuous monitoring"), DE.CM-06 for monitoring external service providers (your telemetry vendor), and DE.AE-03 for correlating information from multiple sources. In the 2.0 delta, supply chain risk moved from ID.SC into the new GOVERN function as GV.SC.
• NIST SP 800-207: Treats the telemetry plane as both a protected surface and a Zero Trust enabler. Tenet 1 makes collectors, brokers, and exporters "resources"; tenet 2 requires all communication secured regardless of network location (mTLS on internal hops and at the edge); tenet 3 demands per-session least privilege, which is the SPIFFE short-lived credential model from the access-control section; and tenet 7 states the enterprise "collects as much information as possible about the current state of assets" to improve posture. Your pipeline is the instrument that satisfies tenet 7, which is why its own integrity carries Zero Trust weight.
• OWASP NHI Top 10 (2025): Reads like a telemetry pipeline audit checklist. NHI2 (Secret Leakage) covers tokens transiting logs, NHI4 (Insecure Authentication) covers OTel Collector admin endpoints, which the project notes "are unauthenticated by default" and should bind only to localhost, NHI5 and NHI7 cover overprivileged and long-lived collector tokens, and NHI9 covers shared identities across pipeline components.

Your deployment model assigns the security burden:

Every BYOC implementation reviewed requires the vendor to hold cross-account permissions to manage the data plane; ClickHouse's bootstrap role, for example, is scoped to resources prefixed clickhouse-cloud and documented as least-privilege. Data sovereignty and IAM exposure are separate questions, so audit the vendor's role scope even when the raw data never leaves your VPC.

groundcover's BYOC architecture applies the split-plane model to observability at every tier, including free: the entire data plane, ClickHouse for logs, traces, and events and VictoriaMetrics for metrics, runs inside your VPC, while groundcover manages only the control plane (UI and orchestration). Raw telemetry stays inside your VPC. To validate the data-residency boundary on your own cluster, you can use the free tier, which includes the same in-VPC data plane as the paid tiers.

What's next for securing customer-owned telemetry planes

Four developments will reshape the discipline over the next few years, and each already has a concrete standard or regulation behind it.

• SBOMs for pipeline components: SUNBURST is the driver, and the standards are now mature. Ecma published CycloneDX 1.7 as ECMA-424 in December 2025 with a dedicated Vulnerabilities object and VEX support, and SPDX 3.0.1 carries cve and packageUrl identifiers. CISA's draft SBOM minimum elements pushes component transparency, and tools like OWASP Dependency-Track already correlate pipeline SBOMs against NVD, OSV, and GitHub Advisories continuously. Given the 2025–2026 CVE record in Fluentd, Logstash, and Prometheus (two of them CVSS 9.8), continuous CVE tracking against a live SBOM is becoming baseline.
• Long-horizon forensic retention on immutable storage: PCI DSS v4.0 requires 12 months of audit history with 3 months immediately available; HIPAA requires 6 years of documentation retention; OMB M-21-31 requires federal criticality-level-0 logs kept 12 months active plus 18 months cold. WORM storage is the enforcement mechanism: S3 Object Lock in Compliance mode prevents deletion by any user including root, and Azure Monitor Logs supports total retention up to 12 years. Expect security telemetry to split from operational telemetry on retention policy as well as sampling.
• ML-based detection closing the redaction gap: The regex ceiling documented earlier (zero coverage of names, addresses, and passwords) is addressable. Adding NER via tools like Presidio cut person-name leak rate to 0.123 in the 2026 benchmark, at under 50ms overhead. Semantic PII detection at the collector stage is the obvious next step for pipeline vendors.
• Disclosure obligations that reach telemetry data: The SEC's definition of a cybersecurity incident covers "any information residing therein" on a registrant's systems, which includes logs and telemetry, with disclosure due 4 business days after a materiality determination. NIS2 requires an early warning within 24 hours and notification within 72; NYDFS Part 500 requires 72 hours, including for incidents at third-party service providers; the U.S. banking rule requires 36. For regulated operators, a breach of the telemetry plane is a reportable event on a clock, which makes the integrity and audit controls above compliance infrastructure.

‍