How to reduce log ingestion costs

Log bills climb faster than the infrastructure that produces them. Log data grew 250% year over year on average in a 2024 Chronosphere survey covered by The New Stack, and every one of those gigabytes can hit three or four separate meters on the way through a platform: ingestion, retention, query, and egress. This guide works through the problem in order: what drives log ingestion cost, how the major pricing models and retention tiers work, how Azure’s Analytics, Basic, and Auxiliary log plans compare, how to find your biggest volume sources, which reduction strategies to run first, when commitment pricing breaks even, what 100 GB/day costs across four platforms, and how groundcover’s per-node model changes the equation.

What drives log ingestion cost

Volume is the primary lever because every other meter multiplies it. A gigabyte you never ingest costs nothing to retain, scan, or move between clouds, which is why pre-ingestion reduction outperforms any downstream optimization.

Four drivers set the bill:

• Volume: The base unit for every charge. Per-GB and per-event ingestion fees apply to what arrives at the platform, and the sections below show that several vendors meter the raw payload regardless of what the platform drops afterward.
• Retention: A recurring monthly charge on everything you keep. Retained volume compounds: at 100 GB/day with 30-day retention, you are always paying storage on roughly 3 TB.
• Queries: Several platforms bill per GB scanned, so query cost scales with retained volume even when the answer is small.
• Egress: Logs that cross cloud boundaries pay bandwidth rates. InfoQ analysis documents rates as high as $0.16/GB in multi-cloud observability deployments, where a single server producing 1 GB/hour costs about $115/month in egress alone.

Having named the drivers, the next question is how vendors turn them into pricing models.

How log ingestion pricing models work

The major platforms price the same four drivers in structurally different ways, and the structure determines which reduction lever works:

• Pay-as-you-go per GB: Google Cloud Logging charges $0.50/GiB for standard logs, with the first 50 GiB per project per month free, 30 days of storage included, and no query charges.
• Decoupled ingest plus event-based indexing: Datadog charges $0.10 per ingested GB on all logs, then $1.06 to $2.50 per million indexed events per month depending on retention (annual billing). On-demand indexing rates run 50% above annual rates.
• Multi-meter usage: The Dynatrace rate card lists $0.20/GiB for Ingest & Process, $0.0007/GiB-day for Retain, and $0.0035/GiB-scanned for Query. Dynatrace measures raw bytes before enrichment.
• Included-query retention: Dynatrace’s Retain with Included Queries option bundles query usage at $0.02/GiB-day for 10 to 35 days of retention.
• Commitment tiers: Azure Monitor discounts Analytics Logs at committed daily volumes from 100 GB/day up to 50,000 GB/day, covered in the commitment section below.
• Flat per-node: groundcover charges per monitored node with no per-GB component, covered in the final section.

Two of these drivers, retention and queries, hide the most surprises, so they get their own breakdowns.

Log data retention costs

Retention is priced in distinct tiers, and the rate gap between interactive and archive storage is large enough to reorder your architecture around. The rates below come from official vendor documentation:

Sources: Azure Monitor cost documentation, Google Cloud Observability pricing, Datadog pricing, Dynatrace rate card.

The Datadog row deserves a second look: Flex Logs Storage at $0.05 per million events versus Standard 15-day annual indexing at $1.70 per million is a 34× storage rate difference, before Flex compute charges for scalable query tiers.

Hidden costs: queries, rehydration, data transfer

Fees on data you already paid to ingest are the most commonly overlooked line items. These fees recur across platforms.

Search and scan fees apply even on low-cost tiers. Azure search jobs cost $0.005/GB scanned across the entire queried time range: the query example shows Azure billing 300 GB for a query spanning 3 days of a table ingesting 100 GB/day, regardless of how few rows match.

Rehydration and restore carry minimums and scan-everything mechanics. Azure Data Restore costs $0.10/GB per day with a 2 TB and 12-hour minimum. Datadog rehydration charges $0.10 per compressed GB scanned against the total archive volume in the time range, not the matching logs, because Datadog applies the query filter after download; pulling a small subset out of a month-long archive can scan hundreds of GB plus indexing fees.

Outbound transfer stacks on top. Datadog charges $0.25/GB per destination for forwarding to SIEM or BI tools, and cloud egress runs $0.09/GB on AWS according to AWS egress pricing, $0.087/GB on Azure according to Azure bandwidth pricing, and $0.12/GiB on GCP Premium Tier according to GCP network pricing at first-tier volumes. With the meters mapped, the next question is which tier each log stream belongs in.

Log tier plans compared: Analytics, Basic, Auxiliary

For Azure workloads with large verbose tables, routing each table by query frequency can produce a large reduction because the per-GB spread between tiers is roughly 46×. One caveat on the numbers: the official Azure Monitor pricing page rendered placeholder values at the time of research, so the rates below come from the Azure Retail Prices API (East US), and third-party analysis corroborates them; check your region in the Azure portal before committing budget.

Tier structure per Microsoft cost documentation.

Two constraints shape the routing decision. Commitment-tier discounts apply only to Analytics Logs, so Basic and Auxiliary volumes never earn volume discounts. And effective October 1, 2025, Auxiliary tables carry an Auxiliary processing charge of $0.10/GB on all incoming data, which triples the effective Auxiliary rate and needs to be in the model.

Datadog’s tiers impose an analogous trade. Datadog supports monitors only for Standard Tier logs, and Datadog hides Flex logs in Log Explorer by default behind an “Include Flex Logs” toggle. Route a log stream to a low-cost tier only after confirming nothing alerts on it. Knowing where logs should go is half the problem; the other half is knowing which streams are inflating the bill in the first place.

How to find your biggest log volume offenders

Volume reduction starts with attribution, and most platforms give you the data to rank sources by ingested GB. Work through these steps before touching any filter:

1. Rank tables and sources by ingested volume in your workspace usage data or the platform’s cost dashboard, then drill to column level. Microsoft’s Sentinel team documented the finding that the ConditionalAccessPolicies column alone typically represents 50–80% of the AADNonInteractiveSigninLogs table.
2. Audit collector and agent defaults.
3. Run cardinality analysis on metrics and log labels.
4. Check network flow logs specifically. Flow logs can become major volume sources at scale, and the next section quantifies that risk.

Cloud cost management tools close the loop by tying each source to a dollar figure, which turns the filter conversation from “this looks noisy” into “this table costs $4,000/month and nobody queries it.”

Strategies to reduce log ingestion cost

Run the levers in this order, because each one changes the economics of the levers after it:

1. Filter before ingestion. The only lever that reduces every downstream meter at once.
2. Route surviving logs to the right tier. Match query frequency to storage cost.
3. Convert high-volume streams to metrics. Keep the signal, drop the raw events.
4. Set caps and budget alerts. A backstop against the failure modes the first three can’t predict.

Filtering, transformations, and sampling before ingestion

Where you filter determines whether you actually save money. Datadog meters ingestion on the original raw payload, so pipeline processors that drop fields after arrival reduce nothing; filtering has to happen upstream, via Observability Pipelines ($0.095–$0.12/GB) or your own collectors.

Azure filters through DCR transformations, KQL applied before data reaches the workspace: where drops rows, project-away drops columns. Two documented gotchas apply. Filtering more than 50% of incoming data on Analytics or Basic tables triggers a processing charge on the excess, calculated as GB dropped minus half the incoming GB. Microsoft exempts workspaces with Sentinel enabled. And for Windows events, XPath filtering guidance identifies XPath filtering as more efficient than transformKql.

VPC Flow Logs deserve their own line item. DoIT’s modeling puts 10 NAT gateways at 1,000 requests/second each at roughly 5.2 TB/month, about $2,600/month at CloudWatch’s $0.50/GB standard rate. Volume controls and aggregation cut that volume:

• Volume controls: GCP’s CEL filter expressions discard flow records before they reach Cloud Logging’s billing meter. Cloud Logging exclusion filters, by contrast, apply after API receipt: they can reduce chargeable logs and storage consumption, but do not reduce API calls or entries.write quota usage. GCP also exposes a secondary sampling rate from 0.0 to 1.0, with the Compute Engine API defaulting to 0.5. AWS lets you capture only accepted or only rejected traffic at flow log creation.
• Aggregation: Splunk Lantern documents in-stream 1-minute aggregation cutting a ~1 TB/day flow log feed to 67 GB/day, a 93% reduction, by collapsing 47 raw records into one aggregated event.

Tiering and retention tuning

Move infrequently queried logs to cold storage and the recurring retention charge drops by multiples. Routing GCP logs through a sink to Cloud Storage puts them in Coldline at ~$0.004/GiB/month or Archive at ~$0.0012/GiB/month according to Cloud Storage pricing, against $0.01/GiB/month for extended Cloud Logging retention: a 2.5× to 8× storage reduction before retrieval fees. The query trade-off is concrete. Mercari’s engineering team found BigQuery external-table queries over GCS ran up to 100× slower than standard tables, with single queries exceeding $10.

Compliance does not block this. PCI-DSS v4.0.1 requirement 10.5.1 states: “Retain audit log history for at least 12 months, with at least the most recent three months immediately available for analysis” (PCI SSC document library). Three months hot, nine months in archive satisfies it. FedRAMP’s AU-11 LOW baseline is explicitly two-tiered: at least ninety days online, the remainder preserved offline per NARA requirements.

Logs-based metrics

Converting high-volume log streams into aggregated metrics preserves the trend while eliminating raw event storage, but cardinality decides whether it saves or costs money. Datadog generates log-based metrics at 10-second intervals and bills them as custom metrics at $0.05 per metric per month.

The math turns fast. Sawmills’ analysis works the example: one metric across 10 endpoints, 3 status codes, and 3 customer tiers is 90 custom metrics, about $4.50/month above allotment. Add a user_id tag with 50,000 users and the same metric becomes 50,000 custom metrics, roughly $2,500/month. Datadog’s own docs warn against grouping by unbounded attributes like user IDs, request IDs, or session IDs. Aggregate bounded dimensions only; keep high-cardinality questions on raw logs in a low-cost tier.

Daily caps and budget alerts

Caps and alerts protect against runaway ingestion that the earlier controls cannot predict.

Set the backstop in two steps:

1. Enable a daily cap or hard budget cap where the platform supports one, set above your 95th-percentile daily volume so normal variance never trips it.
2. Configure budget alerts first at 50% and later at 80%; issue a final alert at 100% of the monthly target, routed to the team that owns the noisiest sources. Also alert on ingestion-rate anomalies per source, not just on spend, so you catch the runaway log loop hours before the invoice does.

One warning on enforcement: a cap that halts ingestion also blinds every monitor that evaluates those logs, because an alert cannot fire on events the platform never ingested. Pair any hard cap with the ingestion-anomaly alert from step 2 so a human intervenes before the cap does.

Commitment tiers, dedicated clusters, and volume discounts

Commitment pricing pays off only when your trailing volume reliably exceeds the break-even point, and that point sits below the tier’s nominal size. Azure Monitor’s commitment tiers run from 100 GB/day to 50,000 GB/day, save up to 30% versus pay-as-you-go, lock for 31 days per change, and bill overage at the committed tier rate rather than PAYG. Third-party break-even estimates for the 100 GB/day tier range from ~70 GB/day to ~85 GB/day, with the spread reflecting regional pricing and assumed PAYG rates; use the conservative figure.

Dedicated clusters require a minimum 100 GB/day commitment via the REST API (500 GB/day via CLI or ARM templates) and offer no pay-as-you-go option, so they only make sense once commitment-tier volume is already a given.

Datadog structures commitment differently. There is no dedicated cluster or reserved-capacity construct for Log Management; instead, on-demand indexing runs 50% above annual rates, so an annual commitment wins at any positive volume. Datadog charges usage above commitment on demand at month-end, volume discounts open at 3 billion or more events per month, and practitioner reporting puts negotiated discounts at 20–40% for annual contracts above $5,000/month.

The decision framework follows from the mechanics: commit to your trailing observed baseline, never to projected growth, since Azure lets you resize every 31 days and Datadog bills the gap above commitment at the on-demand premium anyway. Forecasting risk is the real cost of committing, which the fixed-scenario comparison below makes concrete.

Log ingestion cost comparison across platforms

Fix the scenario at 100 GB/day, roughly 3,000 GB/month, with 30-day retention, and the same workload spans more than an order of magnitude across platforms. We derived the Azure figures from the Retail Prices API and the cited third-party analysis as flagged above; for this comparison, we treat GiB and GB as approximately equal:

Rate sources: Azure pricing, Datadog, Google Cloud, Dynatrace.

The Datadog cell is deliberately incomplete: Datadog publishes no GB-to-event conversion, so total cost at 100 GB/day depends entirely on average event size and how much of the stream you index versus route to Flex or archives. Every row shares one property, though: cost scales with data. That shared property is what the final model on the list removes.

How groundcover changes the cost equation

Comprehensive telemetry access starts with the collection architecture. The groundcover eBPF sensor deploys as a Kubernetes DaemonSet, one pod per node, and captures logs, metrics, traces, and Kubernetes events directly from the Linux kernel with zero application instrumentation. That collection path lets teams retain the signals they need without configuring each service separately.

groundcover’s flat per-node pricing then removes the vendor-side volume meter: $30/node/month on Pro and $35/node/month on Enterprise according to groundcover pricing, with no per-GB ingestion charges, custom-metric surcharges, or per-user fees. Log volume, retention within your tier, and query load do not change groundcover’s per-node unit price. Customer-owned compute, storage, and cloud-network usage remain separate and can change as data volume grows.

Without a groundcover per-GB charge, teams can retain debug logs based on engineering value rather than vendor ingestion fees. Volume controls and tier routing can remain engineering decisions because groundcover does not add vendor-side per-GB fees.

Deployment is BYOC, Bring Your Own Cloud: the data plane, including compute, storage, ingestion, and telemetry processing, runs inside your own cloud account, while groundcover manages the control plane for the hosted UI, APIs, authentication, and orchestration. ClickHouse stores logs, traces, and Kubernetes events, while VictoriaMetrics stores metrics inside the customer environment. This split-plane architecture means groundcover has no vendor-side egress, rehydration, or per-GB-scanned query meter. Retention windows become your own object-storage decision rather than a line item competing with ingestion.

On the numbers, groundcover’s published Datadog pricing comparison models a team running ~700 Kubernetes nodes with 5 TB/day of logs and 500K custom metrics: $2.54M/year on Datadog versus $297K/year on groundcover, an 87% reduction with full APM enabled and no trace sampling. Those figures are groundcover’s own modeled comparison with stated assumptions, not an audited third-party benchmark. Among named customers, BigBasket reported a 50% observability cost reduction while expanding coverage across production plus development and testing environments; per-node pricing meant adding environments changed the node count, not a volume multiplier.

For engineering leaders managing procurement, groundcover is available through the AWS and Google Cloud Marketplaces, plus Azure Marketplace, so eligible observability spend can draw against an existing cloud commitment without opening a separate procurement path.

FAQ

These answers summarize the article’s main pricing, tiering, filtering, commitment, and retention decisions.

Which cost driver should we tackle first?

Volume. It is the multiplicand for retention, query, and egress charges, so a gigabyte filtered before ingestion saves on every meter at once, while a retention or query optimization saves on one.

How do we find our top log volume sources?

Rank tables by ingested GB in your workspace usage data or cost dashboard, drill to column level, and audit collector defaults. Microsoft’s Sentinel guidance identifies single columns consuming most of a table’s volume, and flow logs can become major volume sources at scale.

When should logs go to Analytics vs Basic vs Auxiliary?

Analytics for anything feeding alerts or frequent queries, Basic for occasionally queried debug logs, Auxiliary for verbose streams headed to long-term retention. Factor in the $0.005/GB scan fee on Basic and Auxiliary and the Auxiliary processing charge that took effect October 1, 2025.

At what volume does commitment pricing beat pay-as-you-go?

For Azure’s 100 GB/day tier, third-party estimates put break-even between roughly 70 and 85 GB/day of sustained volume; commit at the conservative end. On Datadog, annual commitment beats on-demand at any volume because on-demand indexing carries a 50% premium.

Does filtering before ingestion always reduce the bill?

Not automatically. Datadog meters the raw payload, so filters must run upstream of its intake endpoint, and Azure charges a processing fee when a DCR transformation drops more than half of incoming data on Analytics or Basic tables.

Can we shorten retention and still meet compliance?

Yes. PCI-DSS v4.0.1 requires 12 months of audit log history but only the most recent three months immediately available, and FedRAMP’s AU-11 LOW baseline requires 90 days online with the remainder offline, so the majority of a compliance window can live in archive-class storage.

Do low-cost log tiers charge for queries?

They do. Azure Basic and Auxiliary bill $0.005/GB scanned, Dynatrace’s pay-per-query model bills $0.0035/GiB-scanned even on retained data, and Datadog rehydration scans the entire archive volume in the queried time range at $0.10 per compressed GB.

What does 100 GB/day cost on each major platform?

Roughly $6,900/month on Azure Analytics pay-as-you-go, $1,475 on Google Cloud Logging, and $2,400 on Dynatrace with bundled queries, while Datadog runs $300 in ingestion plus event-count-dependent indexing. On groundcover the same volume changes nothing in the per-node unit price, because billing follows node count rather than data.