Kubernetes log bills grow along four levers: the raw volume your workloads emit, the CPU and memory your log shippers burn on every node, the per-GB or per-event pricing of your backend, and the network egress fees for shipping data out of your cluster. Most of that spend is avoidable. Riot Games cut log ingest from 2.3 PB/month to 350 TB/month, an 85% reduction, without losing the signals its SREs needed (SREcon EMEA 2025). This guide covers Kubernetes log cost reduction lever by lever, with measured numbers where they exist and honest flags where they don't.
Why Kubernetes logging costs spiral
Log spend compounds because the four cost drivers multiply rather than add. A verbose service inflates volume; that volume raises shipper CPU on every node; the backend charges per GB on the inflated total; and every gigabyte crossing an AZ boundary or exiting to a SaaS backend pays egress on top.
- Log volume: Edge Delta's Charting Observability 2023 survey of 200 DevOps and SRE professionals found log data grew 5x over three years, roughly one doubling every 18 months, with 22% of respondents reporting 10x or more. A "doubles every 15 months" figure circulates in vendor marketing, but no primary source for it exists; Cribl's Telemetry Trends Report documents core inputs like syslog and TCP JSON growing more than 60% year over year, which is the defensible number.
- Shipper overhead: Every node runs a DaemonSet shipper pod whether it handles 100 logs/sec or 10,000. That fixed per-node tax scales with cluster size, not usefulness.
- Backend pricing: Per-GB and per-event models turn every unfiltered DEBUG line into a line item.
- Egress fees: Cross-AZ transfer and internet egress to external SaaS backends sit outside the observability budget line, so nobody attributes them to logging until the cloud bill arrives.
How Kubernetes logging works
The overhead starts with a convention. The Twelve-Factor App mandates that an app "never concerns itself with routing or storage of its output stream" and instead "writes its event stream, unbuffered, to stdout." Kubernetes implements this through a chain: the container runtime captures stdout/stderr, containerd decorates each line with an RFC3339Nano timestamp and CRI tags, and the kubelet directs the runtime to write logs into /var/log/pods (Kubernetes docs).
A DaemonSet-based shipper, usually Fluent Bit or Vector, then tails those files on every node and forwards them to a backend. File tailing is where the CPU goes. Fluent Bit's tail plugin watches files via inotify and tracks offsets in a SQLite database, and large pod counts can force you to raise the kernel's fs.inotify.max_user_watches limit (Fluent Bit tail docs). Fluentd's own documentation warns that "CPU and I/O overhead can be high when monitoring many files" because of an internal 1-second watch timer (Fluentd in_tail docs).
The official Kubernetes docs add a storage warning most teams miss: writing logs to a file and then streaming them to stdout "can double how much storage you need on the node." Each optimization below intervenes at a specific point in the chain: log levels reduce what enters it, filters cut what the shipper forwards, and backend choice sets the price of everything that survives.
Measure first: log volume and spend by namespace
Attribute volume to namespaces and workloads before touching a single drop rule, because savings projections without a baseline are guesses. Build attribution from the labels your pipeline already emits: query your backend for bytes ingested grouped by namespace and workload labels, or read the shipper's own throughput metrics per input path.
The pattern in every documented success story is measurement preceding optimization. AppsFlyer measured that "only a single-digit percentage of logs" contributed to debugging or incident response before building its suppression strategy, which then cut Kafka infrastructure log volume by 50% on average and 66% overall (AppsFlyer Engineering). Expect a familiar shape in your own data: a handful of chatty services, ingress controllers, and health-check endpoints producing a disproportionate share of total bytes.
Tune log levels in production
Disabling DEBUG and TRACE in production is the highest-impact, lowest-effort change available. DEBUG can generate 10 to 100 times more log volume than INFO (Frugal, Sawmills), so on a per-GB backend the log level is a direct bill multiplier. Researchers have not published a controlled production benchmark that isolates the DEBUG-to-INFO switch, but the case studies are consistent:
- GCP Cloud Logging: One team traced its overage to DEBUG logs left enabled in production, enforced DEBUG-in-dev-only, and eliminated 2 million logs/day, saving $1,000/month (writeup).
- ECS/CloudWatch: A 15-service fleet switching from INFO to WARN cut CloudWatch costs from $135/month to $30/month, a 78% drop (Fortem).
- Spotify: Set default logging to Google Cloud Logging at WARN and ERROR only as part of its high-throughput logging rework (SREcon22 EMEA).
- GKE system logs: Excluding info-level Kubernetes system logs saved 60% on a 5-node cluster and 67% on a 50-node cluster, roughly $550/month at $0.50/GiB (OneUptime).
Enforce the policy structurally: DEBUG for local and dev, INFO for production non-error traces, WARN and ERROR for problems. A ConfigMap-driven log level per environment prevents the "left it on after the incident" failure mode.
Filter and drop noisy logs at the source
Health-check, liveness/readiness probe, and ingress access logs are the biggest deletable category. Grepr estimates health-check logs alone account for 15 to 40 percent of total volume, with a worked example: 200 pods probed at 10-second intervals generate roughly 28.8 million lines (8.6 GB) per day of logs nobody will ever read. You will see a "30-50% volume cut" figure quoted for this category of filtering; treat it as an aggregate heuristic, since no primary source measures it in isolation. The best isolated estimate for health-check filtering alone is 15-25%, from CloudToolStack, an industry blog rather than independent research.
Drop these logs at the shipper, before they cross the network or hit the meter. adjoe, running 8 billion logs/day on EKS, deployed Fluent Bit with grep filters and Lua scripts to drop unneeded logs at the node level, contributing to a 60% reduction in log costs while extending retention to 60 days (adjoe engineering blog). Vector supports equivalent drop transforms in its pipeline, and if you run an OpenTelemetry Collector, the filter processor drops log records matching OTTL conditions:
processors: filter: error_mode: propagate log_conditions: - IsMatch(log.body, ".*password.*") - log.severity_number < SEVERITY_NUMBER_WARNMatch probe traffic on the user-agent (kube-probe/*), the request path (/healthz, /readyz), or the ingress controller's access-log format, and drop on any condition.
Sample high-throughput services
Sampling complements filtering when dropping an entire category is too blunt: you keep a statistically representative fraction of a high-volume stream instead of all or nothing. OpenTelemetry draws this line explicitly, noting that filtering and aggregation "do not adhere to the concept of representativeness" the way probabilistic sampling does (OTel sampling concepts).
Two mechanisms exist in the OTel Collector. Head sampling decides early using the trace ID and a fixed percentage; it is cheap but cannot guarantee that traces containing errors get sampled. Tail sampling decides after seeing the whole trace, so it can keep every error and slow request, but it requires stateful collectors that OTel warns may need "dozens or even hundreds of compute nodes" at scale. For logs specifically, the probabilistic sampler processor (alpha stability for logs) supports a hash-seed mode that samples on attributes other than trace ID, such as service.instance.id for pod-level sampling, plus a sampling_priority attribute to override the rate per record so errors always pass.
The production evidence supports aggressive rates on the right streams. Riot Games applied 1% sampling at the collector level as part of its 85% reduction. Spotify automated the response: GCP alerts on high-volume projects trigger a switch to a 10% sampling rate, replacing a regime where the overloaded pipeline dropped an estimated 80% of logs without surfacing it.
Sampling reduces volume only after you define the streams that must remain complete. Start by separating compliance-scoped logs from routine application logs, then apply sampling only to streams where representativeness is acceptable.
Compliance-critical logs to never drop
Whitelist audit-trail and compliance sources before any broad drop or sample rule ships, because the retention floors are hard requirements, not guidelines:
- PCI-DSS v4.0 (Req. 10.5.1): Retain audit trail history at least one year, with three months immediately available for analysis (PCI SSC guidance).
- FedRAMP Moderate and High (AU-11): Audit records stay online at least 90 days (FedRAMP AU-11).
- HIPAA (45 CFR 164.316(b)(2)(i)): Six-year retention for Security Rule documentation; audit log retention for ePHI systems is risk-based with no HHS-prescribed minimum (HHS OCR).
- SOC 2: No fixed period, but logs must cover the full Type II observation window, typically 6-12 months (Safeguard.sh).
GDPR runs the other direction: Article 5(1)(e)'s storage limitation principle makes short retention favorable for logs containing personal data. Tag compliance-scoped namespaces and route them around your drop and sample pipeline entirely.
Choose an efficient log shipper
In the VictoriaMetrics benchmark, Fluent Bit used less CPU and memory than Vector at 10K logs/sec, while Fluentd dropped logs before reaching that rate. The VictoriaMetrics benchmark (March 2026, methodology on GitHub) tested Fluent Bit v4.2.3, Vector v0.53.0, and Fluentd v1.19 under identical 1 CPU / 1 GiB limits:
VictoriaMetrics excluded Fluentd from the 10K comparison because it was "already losing logs at this throughput level." Its own docs describe single-process tuning as sufficient only up to 5,000 messages/sec. If you run Fluentd today, migrating to Fluent Bit or Vector is a direct per-node resource cut across the fleet; Fluent Bit v5.0 additionally claims up to 4x faster OTLP payload handling than v4.2, though that claim has no independent Kubernetes benchmark yet (release announcement).
One further reduction applies regardless of shipper: skipping file tailing entirely. groundcover's eBPF sensor collects logs at the Linux kernel level via eBPF as a single DaemonSet, so there is no inotify watch set, no SQLite offset database, and no per-file tail loop. Treat the sensor's resource footprint as workload-dependent; the mechanism, one kernel-level DaemonSet replacing file tailers plus per-language APM agents, explains where the savings come from.
Backend selection and its cost multiplier
The backend's pricing model multiplies every upstream saving or waste, so it deserves as much scrutiny as any filter rule. Common backend architectures price the same gigabyte very differently:
Indexing strategy explains much of the spread. Loki indexes only labels, not log content, which keeps storage cheap but limits query patterns to LogQL over label sets. Elastic Cloud Serverless uses full-text indexing. Qonto's measured comparison for ~30 TB compressed/month put Loki at $6,600-$7,600 versus $12,600-$13,600 for equivalent Elasticsearch.
Columnar stores change the storage math again. ClickHouse's LogHouse post reports a 17x compression ratio for its internal logging platform, which stores over 19 PiB uncompressed for $125K/month in infrastructure, against a modeled (not invoiced) Datadog list-price projection of ~$26M/month (ClickHouse blog). Structured JSON logging compounds the advantage: it reduces tokenization work in Elasticsearch indexes and improves ClickHouse compression ratios, though no source quantifies the percentage.
groundcover stores telemetry data in ClickHouse deployed inside your own VPC, priced flat per node rather than per GB. On a volume-decoupled backend, a noisy deploy raises disk usage, not your invoice, which changes filtering from a billing emergency into an efficiency task.
Retention policies and tiered storage
Retention is a hot-window question plus a cold-storage question. Keep a short hot tier sized to your actual query patterns, then route older logs to object storage. AWS S3 tiers make the gradient concrete:
(AWS Glacier storage classes, restore options). Watch the minimum storage durations: 90 days for the Glacier tiers, 180 for Deep Archive, so don't route logs there that a lifecycle policy will delete sooner.
The backends offer their own tiering. Datadog's Flex Logs decouples ingestion from indexing, with rehydration from archives at $0.10 per compressed GB scanned. Balance the tiers against query reality: adjoe first cut retention from 30 to 7 days, then used its filtering savings to extend it to 60 days, ending with longer retention and lower cost. The compliance floors from the sampling section apply here too; PCI's three-month immediately-available requirement and FedRAMP's 90-day online minimum set your hot-tier lower bound for in-scope systems.
Node-level rotation is the layer under all of this. The kubelet defaults to containerLogMaxSize: 10Mi and containerLogMaxFiles: 5, so 50Mi maximum per container, and the kubelet exposes only the latest file to kubectl logs(kubelet config API). EKS and GKE inherit these defaults; GKE allows tuning up to 500Mi and 10 files but caps total per-container log size at 1% of node storage (GKE node system config). Legacy Docker json-file clusters default to max-size: -1, meaning no rotation at all until you set it.
Reduce egress costs
Egress is the cost lever most teams never attribute to logging. Shipping logs from AWS to an external SaaS backend over the public internet costs $0.09/GB for the first 10 TB/month (AWS EC2 pricing). Compare that to Datadog's $0.10/GB ingestion price: the network path nearly doubles the effective per-GB cost. Ship from private subnets through a NAT Gateway and you add another $0.045/GB in processing fees (AWS APN guide). GCP Premium Tier internet egress runs $0.12/GiB to North America and Europe; Azure charges $0.087/GB from those regions.
Fix the network path in stages:
- Keep traffic in-region and same-AZ where possible. Cross-AZ transfer costs $0.01/GB in each direction on AWS and $0.01/GiB cross-zone on GCP; same-AZ private-IP traffic is free on all three clouds.
- Use private connectivity for SaaS backends. Shipping logs to Datadog US East over AWS PrivateLink costs $0.01/GB versus $0.09/GB over the public internet. Datadog also supports Azure Private Link and GCP Private Service Connect on specific sites.
- Don't egress at all. A BYOC architecture stores telemetry inside your own VPC, so log data never crosses an internet or cross-region boundary. In groundcover's model, the data plane runs in your cloud account while only the managed control plane and UI live outside it. Your log bytes never generate an egress line item.
A measured example of what the wire costs: one Cribl customer cut log egress from 600 MBps to 200 MBps, roughly 131 TB/month of transfer removed, alongside a $70K-to-$30K monthly storage reduction.
Prioritized log cost reduction checklist
Every large documented reduction (Riot Games, adjoe, Clever, AppsFlyer) combined at least two techniques; none came from a single change. Work this list in order:
- Baseline volume by namespace and workload. You cannot rank the rest of this list for your cluster without it.
- Enforce production log levels. DEBUG off in production, INFO as the ceiling for non-error traces. Cheapest change with the largest documented single-lever impact.
- Whitelist compliance sources. Tag PCI, FedRAMP, and audit-trail namespaces before writing any drop rule.
- Drop probe and health-check logs at the shipper. Fluent Bit grep filters, Vector transforms, or OTel filter-processor OTTL conditions.
- Sample the high-throughput remainder. Probabilistic sampling at 1-10% on chatty services, with priority overrides so errors always pass.
- Replace Fluentd if you run it. The benchmark gap to Fluent Bit is 6x on throughput under identical limits.
- Tier retention. Short hot window sized to query patterns and compliance floors, Glacier-class object storage behind it, kubelet rotation confirmed at the node.
- Fix the network path. In-region shipping, PrivateLink for SaaS, no NAT Gateway in the log path.
- Re-examine the backend's pricing model. This is the multiplier on everything above.
Steps 1 through 8 assume a per-GB backend, where every technique exists to shrink a meter. groundcover takes a different architecture path: flat per-node pricing at roughly $35/node/month, so log volume, retention within your tier, and query load do not change the bill. That means you can keep the data engineers need instead of sampling it away just to stay under an ingest meter.
The BYOC data plane keeps ClickHouse and VictoriaMetrics in your VPC. Only the managed control plane and UI live outside it. groundcover's eBPF sensor replaces the file-tailing DaemonSet and per-language agents with one sensor per node, while edge stream processing applies OTTL drop rules and enrichment in-cluster before storage.
groundcover documents an 87% cost reduction versus Datadog for a 700-node Kubernetes environment. Treat that as an outcome of the architecture. The starting point is complete telemetry without shipping your data out or pricing every GB. The Free tier runs on the same BYOC architecture with 12-hour retention, needs no credit card, and deploys in 20-30 minutes via the groundcover Console if you want to see the BYOC data plane and groundcover's eBPF sensor DaemonSet on your own cluster.
.svg)
