Reducing Kubernetes monitoring costs with eBPF

Dynatrace's State of Log Management 2026 found telemetry and log volume grew 93% on average in the past 12 months, with 20% of organizations seeing growth above 150%. On volume-based pricing, teams facing that curve drop signals or watch the invoice climb, which is why Kubernetes monitoring cost climbs faster than the cluster does. (The widely repeated claim that telemetry doubles every 15 months has no retrieved Gartner or IDC report behind it; the closest sourced figure is a 29% CAGR attributed to IDC, roughly an 18-month doubling.) eBPF-based collection changes the economics of that choice rather than managing it.

What drives Kubernetes monitoring costs

Kubernetes monitoring cost grows along billing dimensions that VM monitoring never had. A VM is one host, one agent, one line item. A Kubernetes node carries dozens of pods, each with labels and short lifecycles that most SaaS vendors meter separately:

• Per-container and per-pod meters: Datadog includes 5 containers per host on Infrastructure Pro and charges $0.002 per container per hour beyond that, roughly $1.44/month per always-on container. Dynatrace bills Kubernetes Platform Monitoring at $0.002 per pod-hour, about $1.40 per pod per month, and meters pod usage in 15-minute intervals.
• Dynamic pod lifecycles: Every pod restart and deployment rollout mints new label values. On per-metric pricing, each new label combination is a new billable timeseries.
• The allocated-versus-actual gap: You pay to monitor provisioned capacity, and in Kubernetes most of that capacity sits idle. CAST AI, Datadog, and Google quantify that gap below.

Grafana Cloud moved self-serve Kubernetes Monitoring billing to active host hours and container hours on September 17, 2024, with contract customers moving on February 13, 2026, which shows how directly pod density now maps to invoices across the vendor field.

The main cost drivers

Most spend comes from the drivers below.

Metric cardinality

Datadog defines a custom metric as a unique combination of metric name and tag values, includes 100 per host on Infrastructure Pro, and charges $0.10 per 100 ingested custom metrics above the allotment. Distribution metrics carry a 5x multiplier: one metric name with 4 tag combinations generates 20 billable custom metrics. Kubernetes makes this explosive because the Datadog Agent auto-assigns tags from pod labels, and the high cardinality setting includes container IDs, multiplying timeseries for every metric carrying those tags.

Production teams have documented the failure modes. One service emitted a counter tagged with user_id and created 400,000 series.

Another team found its Datadog bill running $6,780/month against $2,100/month of AWS infrastructure; after cutting custom metrics from 47,000 to 800, the bill fell to $1,490/month. One team left the GKE Autopilot GCP metric defaults in place and went from a few hundred custom metrics to 20,000+, a 20x bill spike until they filtered.

Self-hosted Prometheus converts the same problem into memory instead of dollars. Estimates run from ~732 bytes per series (Brian Brazil) to Cloudflare's ~4 KiB/series, and there is no built-in hard cap on series count.

One documented case saw a Prometheus instance cross 60 GB of memory in an OOM-killed case on a 50-node cluster with 2 million active series. Grafana Cloud, meanwhile, bills active series at the 95th percentile of hourly measurements, so a sustained cardinality spike lands directly on the invoice.

Over-provisioning and idle resources

CAST AI's 2026 optimization report, drawn from 23,000+ clusters, found average CPU utilization of 8% and memory utilization of 20%, with a 69% gap between provisioned and requested CPU. CAST AI sells optimization, so read its sample with that in mind, but Datadog and Google report similar under-utilization patterns: Datadog's 2025 container report found 72% of workloads used less than half their requested CPU, and Google's Borg trace showed production-tier CPU usage around 30% of allocation.

Every one of those idle nodes carries a per-host monitoring charge and emits a full complement of metrics. Right-sizing cuts the monitoring bill as well as the compute bill; the autoscaling and right-sizing section below covers the mechanics.

Log ingestion and retention

Logs are another major lever, and vendors meter them at every stage. Datadog charges $0.10/GB to ingest plus $1.06 to $2.50 per million indexed events depending on retention. Dynatrace's pay-per-query model stacks $0.20/GiB ingest, $0.0007/GiB-day retention, and $0.0035/GiB-scanned at query time, so you pay a third time when you read the data.

The behavioral result is rationing. Dynatrace's own survey found organizations discard or never collect 86% of their logs because of cost, while spending nearly $2.5 million annually on logging. When an incident hits, the signal you need is often in the logs nobody kept.

How eBPF changes the cost equation

eBPF moves telemetry collection from the application layer to the Linux kernel, which collapses the agent stack.

groundcover's eBPF sensor deploys as a single DaemonSet, one pod per node, and captures metrics, traces, logs, and Kubernetes events directly from the kernel. No SDKs or language-specific agents. No restarts.

The application and infrastructure telemetry stack, plus separate upgrade cycles, reduce to one deployment. Every new pod is visible the moment it schedules.

The overhead difference is measurable, with a caveat worth stating plainly: no neutral head-to-head benchmark of the Datadog Agent against eBPF collectors exists in the public record. The only published comparison is groundcover's own 2023 benchmark, which measured 249% CPU overhead above baseline for Datadog at 3,000 req/s versus 32% for eBPF-based Pixie and 59% for OpenTelemetry. groundcover states that eBPF runs on 73% less CPU than the Datadog Agent. Treat both as vendor-run figures and test them on your own cluster; the test costs an afternoon.

Datadog applies eBPF in scoped products such as Cloud Network Monitoring and workload protection, rather than for universal telemetry. That does not replace the billing model that instrumentation feeds.

That billing model is the larger change. The groundcover’s eBPF sensor collection cost matters, but flat per-node pricing matters more: groundcover charges a flat per-node rate with no per-GB, per-metric, or per-seat meters, so a 10x log spike during an incident, a cardinality explosion from a bad label, or a decision to trace at 100% changes nothing on the bill.

groundcover publishes named customer outcomes around the same mechanism. BigBasket cut observability costs 50% while expanding coverage, and Afida expanded from 10 to approximately 100 users without incremental per-user cost because groundcover does not charge per seat. Ingestion pricing forces you to choose which signals you can afford; flat per-node pricing removes the choice, and the 86% of discarded logs comes back.

Allocating costs by namespace, workload, and team

Cluster-level bills hide who spends what, and shared clusters make the problem structural. The FinOps Foundation's container cost allocation guidance recommends labeling namespaces and workloads with cost-center and team, plus department or business-unit, so spend maps to the organization that generated it.

The OpenCost specification defines the math: total cluster cost decomposes into workload costs, idle costs, and overhead, with workload cost calculated at the container level as max(request, usage). That max() is the accountability mechanism. A team that requests 4 CPUs and uses 0.5 gets charged for 4, which turns the over-provisioning gap into a number on someone's showback report instead of an invisible platform subsidy.

It works at enterprise scale. Nationwide told the FinOps Foundation: "For shared platforms, like containers, we primarily leverage a label strategy for cost allocation. We allocate cluster costs in proportion of each cost center label's usage," with chargeback covering chargeback coverage for "the vast majority of our expenses". For SaaS companies, the same per-namespace breakdown feeds COGS reporting: per-customer or per-product infrastructure cost becomes a queryable number.

Reducing metric cardinality and ingestion costs

Most scraped metrics are never read. An analysis of a default Kubernetes deployment found only 270 metrics in any dashboard or rule, roughly 80% waste. Two tools find yours: promtool tsdb analyze reports series counts and churn per label pair, and the mimirtool analyze workflow cross-references your Grafana dashboards and ruler rules against ingested series, outputting in_use_active_series versus additional_active_series, your drop candidates.

Dropping happens in metric_relabel_configs, which runs after the scrape but before storage (unlike relabel_configs, which selects targets before the scrape):

metric_relabel_configs:  - source_labels: [__name__]    regex: 'expensive_metric_name'    action: drop

The returns are documented. One team cut a single metric from 13,032 series to 432 by removing a type label with 162 distinct values, and its combined optimization work cut CPU and memory by 46% and 58%.

Widening scrape_interval to 60s can cut Grafana Cloud metrics costs by 75%. Robust Perception's caveat applies: relabel-dropping is a stopgap, since Prometheus still pulls every sample from the target before relabeling drops it; the durable fix is at the instrumentation source.

Platform teams can attack the same problem architecturally by changing storage backends. eBPF deploys as a DaemonSet, one pod per node, and groundcover keeps logs and traces in ClickHouse and metrics in VictoriaMetrics. That architecture aggregates and stores telemetry in the backend before teams make retention choices, rather than depending only on after-the-fact deletion.

SaaS vs self-hosted monitoring stacks

SaaS consumption pricing stacks meters. Datadog charges $15 to $27 per host for infrastructure, $31 to $48 per APM host, $0.10/GB for ingested spans, $1.27 to $2.50 per million indexed spans, and $0.15 to $4.50 per thousand RUM sessions, each billed separately.

Dynatrace's DPS runs about $58/month per 8 GiB full-stack host.

Self-hosting trades those invoices for engineering time, and the time is not small:

• Scaling ceilings: Exoscale puts a single Prometheus server's ceiling at roughly 1 to 2 million active series before OOM risk, and HA means running duplicate servers plus external deduplication. Zepto's HA pair failed simultaneously after both Prometheus replicas OOM-killed, taking out both replicas at once.
• Component sprawl: Thanos decomposes into six components. Medallia's production deployment ran 402 Prometheus instances, 286 sidecars, 88 stores, and dozens of queriers, rulers, and compactors, consuming 450 to 650 CPU cores and 7 to 13 TiB of working set memory.
• VictoriaMetrics as the lighter path: VictoriaMetrics runs the same PromQL workloads on a smaller footprint than Prometheus plus Thanos, with Prezi reporting roughly 30% lower total cost and 70% less storage and Grammarly reporting an AWS bill roughly 10x lower. It brings its own operations, though: replication-factor and HA configuration, plus cross-AZ traffic charges, which Prezi hit when it spread cluster components across zones.
• Upgrade risk: Grafana 11 removed legacy alerting entirely; upgrading without migrating "will result in losing all of your alerts".

BYOC (Bring Your Own Cloud) is the third path: the data plane runs in your own cloud account while the vendor operates it. In groundcover's version, ClickHouse and VictoriaMetrics run inside your VPC and groundcover manages installation, upgrades, and operations through its control plane, so telemetry never leaves your infrastructure and the Thanos-scale operational burden never arrives. groundcover's BYOC architecture is the default at every tier, including free.

Kubernetes cost monitoring tools

Visibility into monitoring spend and cluster spend requires tooling of its own, and the main options split by what they optimize:

• Kubecost (IBM): Cost allocation by namespace, deployment, service, and team labels, plus cloud bill reconciliation and request right-sizing recommendations. IBM acquired it in September 2024; the free Foundations tier covers 250 cores with 15-day retention, and multi-cluster unified views require Enterprise.
• CloudZero: Unit economics is the differentiator: cost per customer, per product, per AI feature, with 100% spend allocation through its CostFormation model. PeerSpot reviewers cite CloudZero pricing as a "1% of cloud usage" model and flag heavy YAML configuration.
• CAST AI: Cost monitoring is free for unlimited clusters; the paid product is an autonomous optimizer with a 15-second autoscaler sync, bin packing, and Spot instance management. PeerSpot reviewers report 20 to 40% infrastructure savings but weaker cost allocation than competitors.
• Datadog: Datadog covers a broad observability surface, but the multi-dimensional billing dissected above applies to the cost tooling's own parent invoice.
• Prometheus: Prometheus costs nothing to license and brings the PromQL ecosystem, but it handles metrics only and carries the cardinality and HA limits covered earlier.

Kubecost runs self-hosted or as managed SaaS; CloudZero and CAST AI are SaaS only. Pick based on whether you need chargeback reporting (Kubecost, CloudZero) or automated remediation (CAST AI), and remember none of them reduces the monitoring bill itself; they expose where it goes.

Autoscaling and right-sizing that cut costs

Since vendors meter monitoring per host, pod, or container, every node you consolidate away cuts the observability bill along with the compute bill. Karpenter manages instances directly without node groups, batches pending pods, and bin-packs them against CPU, memory, GPU, and DaemonSet overhead before provisioning; its WhenEmptyOrUnderutilized consolidation policy actively removes or replaces underutilized nodes.

AWS-published case studies, vendor-sourced but named and quantified:

• HP: a 40% improvement in node utilization worth $125K+ per year.
• Grover: 80% on Spot in production.
• Yelp: a 25% spending efficiency improvement.

At the workload level, HPA scales replicas while VPA resizes requests, and the two must not target the same resource metric: VPA inflates requests, HPA sees utilization drop and scales in, load concentrates, and the loop repeats.

Run VPA in Off or Initial mode in production to collect recommendations without eviction risk. In-place pod resize reached GA in Kubernetes v1.35, which softens the restart penalty that made VPA scary, though VPA's own InPlace and InPlaceOrRecreatemodes are still alpha as of VPA 1.7.0. Feed the recommendations from the utilization data your monitoring already collects; the 8% CPU utilization figure from earlier is the budget you claw back here.

Connecting cost data to FinOps

FinOps teams turn cluster-level cost visibility into governance by feeding it into a reporting loop. The FinOps Foundation defines the two reporting modes precisely: showback reports actual consumption for awareness, while chargeback charges that consumption to P&L budgets in the financial system. The namespace and label allocation described earlier is the input; teams use showback to build the habit, then chargeback to create the incentive.

Enterprise readers will also meet TBM (Technology Business Management), the TBM Council's taxonomy that classifies technology spend through four layers: cost pools, resource towers, solutions, and consumers. TBM Taxonomy 5.0 adds AI Compute and AI Models as distinct domains.

The two disciplines divide the work: per the FinOps Foundation, TBM looks across the whole IT portfolio top-down, while FinOps handles the variable, dynamic cloud spend that TBM is not designed to address. In practice that means mapping Kubernetes namespaces and workloads to TBM sub-towers via the same labels teams use for chargeback, so observability spend appears in COGS and portfolio reporting instead of a flat "cloud" line.

Quick wins checklist

Ordered by effort-to-impact, these steps come straight from the mechanisms above:

1. Run mimirtool analyze against your dashboards and rules to find unused series; expect a large fraction of scraped metrics to be dead weight.
2. Drop the worst offenders with metric_relabel_configs and set sample_limit per scrape job as a circuit breaker against cardinality explosions.
3. Audit tags feeding your SaaS vendor: hunt for user_id, request IDs, and container IDs in metric labels, and check distribution metrics for the 5x multiplier.
4. Widen scrape_interval to 60s for metrics that don't need 15s resolution.
5. Label namespaces and deployments with cost-center and team, then stand up showback with Kubecost's free tier or OpenCost before attempting chargeback.
6. Compare requested versus actual CPU and memory per workload, and start VPA in Off mode to collect right-sizing recommendations.
7. Enable Karpenter consolidation on non-critical node pools; fewer nodes means fewer per-host monitoring charges immediately.
8. Price your current stack per node: total monthly observability spend divided by node count, so alternatives compare on one number.

If you want to test whether kernel-level collection replaces your agent stack, groundcover's free tier includes the full BYOC architecture, so your telemetry stays in your VPC from day one. Deploy eBPF on a single cluster, no credit card required, and you have full-cluster visibility within hours to compare against your current agents' footprint and coverage.

‍