Reducing observability agent CPU overhead

Your observability stack should watch production without competing with it. On busy Kubernetes nodes, log shippers and parsing agents routinely burn a full CPU core or more, and OpenAI’s applied observability team recovered 30,000 CPU cores at peak by changing one line of Fluent Bit configuration. Comparable headroom can be hiding in a cluster; finding it requires understanding exactly where parse-time CPU goes.

The pressure behind that work is worth naming, because it applies to anyone running a DaemonSet. OpenAI needs more CPU than it can buy, so scaling out is not always an option and optimizing is not optional. Every core a per-node agent reserves is a core the node no longer contributes to the cluster, which is why their team treats each new DaemonSet as something that has to justify itself. Return one core per node and you may fit one more service per host.

What is log parsing CPU overhead

Log parsing CPU overhead is the compute cost an agent or pipeline spends turning raw log bytes into structured events: matching regex patterns, splitting fields, reassembling multiline records, serializing output, and shuttling data through syscalls. On a node with a fixed CPU budget, every millicore the parser consumes is a millicore your workloads don’t get, and under Kubernetes CPU limits the parser can also throttle itself into falling behind.

Four mechanisms account for most of the waste:

• Regex backtracking: ambiguous patterns like grok’s GREEDYDATA force the engine to explore exponentially many match paths on failure.
• Busy-wait polling: tight loops and zero-timeout event calls that spin the CPU even when no logs arrive.
• Per-line memory allocation: heap allocations for every line, key, and value, paid again in garbage collection.
• Syscall and context-switch overhead: stat storms, fsync on every checkpoint, futex contention, and epoll_wait churn.

Each of these has measured, reproducible costs. Those four mechanisms produce distinct signatures in CPU profiles and require different fixes.

How log parsing burns CPU

The four root causes above are not equal offenders on every node, so it helps to know what each one looks like in a profile and how expensive it gets in the field.

Regex backtracking in grok and multiline parsers

Grok patterns are regexes underneath, and a backtracking regex engine explores match alternatives depth-first. When a pattern is ambiguous, a failing input forces the engine to revisit every choice point: Microsoft’s documentation puts a 30-character worst-case input at roughly 1,073,741,824 comparisons. Elastic’s own testing found failed grok matches run up to 6× slower than successful ones.

The classic production anti-pattern is two consecutive %{GREEDYDATA} captures. The engine has no way to tell where one greedy capture should end and the next should begin, so it hands as much input as it can to the first and backtracks from there — the explanation given in grok issue #37. In that report, CPU on an EC2 m3.xlarge hit 100% within minutes; removing the problematic suffix dropped it to roughly 20%, and adding guards made parse failure 2×–5× faster.

The fixes follow directly from the mechanism:

• Anchor your patterns: adding ^ and $ made initial match-failure detection ~10× faster in Elastic’s tests, because the engine stops retrying the match at every substring offset.
• Replace ambiguous captures with specific ones: swapping IPORHOST for NOTSPACE in one Guance pipeline cut per-event cost from 67,006 ns to 7,860 ns in a pipeline benchmark, about 8.5×.
• Prefer delimiter-based parsers: Logstash’s dissect filter scans left-to-right for delimiters once, with no regex engine. A 10-million-message benchmark on Logstash 7.9 measured 40% versus 60% CPU for dissect and grok, respectively.
• Keep timeouts on: Logstash grok’s timeout_millis defaults to 30,000 ms. Disabling it means a single pathological input can tie up a worker indefinitely.

Regex engines built on Thompson NFA simulation, the approach Go and Rust use, run in O(mn) time and cannot backtrack catastrophically. Where your agent lets you choose the engine, Thompson NFA simulation can eliminate catastrophic backtracking entirely.

Busy-wait polling vs event-driven file tailing

Pattern cost is only half the story; how the agent discovers new log data drives CPU too. A polling tailer wakes on a timer and calls stat() on every watched file whether or not anything changed. An idle Fluentd instance watching 2,000 files burned 12% CPU on its 1-second timer in a Fluentd polling test, with strace -c recording 15,337 fstat calls per polling cycle. Event-driven tailing via Linux inotify instead blocks on a file descriptor until the kernel reports a change.

The conceptual contrast is simple:

# Polling: wake, stat every file, sleep, repeat
loop { for f in files { stat(f) }; sleep(interval) }

# Event-driven: block until the kernel says something changed
loop { events = read(inotify_fd)  # blocks; wakes only on writes }

But event-driven is not automatically cheaper at high log rates, and OpenAI’s case is the clearest public demonstration of why.

Running Fluent Bit as a DaemonSet across their fleet, OpenAI emits just under 10 PB of logs a day. Their engineer went into the investigation expecting the hot path to be string processing, since enrichment and filtering run in Lua. perf report said otherwise: fstat64 accounted for about 35% of the thread’s time. The largest single use of Fluent Bit’s CPU was working out how big log files were before reading them.

Two places in the tail plugin call stat. The first is the periodic refresh_interval, which OpenAI had set to 5 seconds — frequent, but with a few dozen pods per node the arithmetic doesn’t reach 35%. That left inotify, which is on by default.

The reason inotify costs so much is that its events say a write happened, not how much was written. Fluent Bit needs the size to update the state database tracking its read offset in each file, so every notification triggers a stat and then an immediate read. A process logging line by line puts Fluent Bit in a race with the writer: one notification, one stat, one read, per line. At 1,000 lines/second on one file, that is 1,000 wakeups and 1,000 stats a second.

The fix was a one-line change turning inotify off. On a single test cluster, CPU halved for the same work; rolled out to production, the result held, returning about 30,000 cores at peak. The 50% figure applies to the most heavily loaded Fluent Bit pods, and OpenAI was careful to note that hosts differ in load and in the logging behavior of what they run. A Kubernetes user in Fluent Bit issue #11096 saw CPU fall from 117m to 13m millicores with the same inotify_watcher: false change.

OpenAI has said it is working on an upstream patch to expose a minimum interval, so other users could get the same benefit without giving up inotify. Worth watching before you standardize on disabling it.

The lesson is to match the mechanism to your write rate: event-driven wins for many mostly-idle files, periodic polling wins for a few files written thousands of times per second. And never set poll intervals to zero: Filebeat’s docs warn that scan_frequency: 0 makes it scan the disk in a tight loop, and rsyslog’s docs say PollingInterval: 0 “will make rsyslogd become a CPU hog.”

Multiline parsing cost

Stack traces and slow-query logs force the parser to hold state across lines, and that statefulness has both a steady cost and a history of pathological bugs. Fluent Bit’s CRI parser benchmark measured 49.2 MB/s with no parser, 18.6 MB/s with the standard regex CRI parser, and 27.6 MB/s with a static state-machine parser. The static parser increased throughput about 49% over the regex parser and recovered about 29% of the gap to the no-parser baseline.

The pathological side is versioned. Fluent Bit’s multiline filter failed to reset its group metadata buffer on flush in 2.1.x, 2.2.x, 3.0.4, 3.0.6, and 3.1.7, driving CPU to 100% and memory growth to OOM at 1,000+ lines/second (issue #7782); Fluent Bit maintainers fixed the bug in v3.1.9 and v3.2.0. Separately, running multiline or rewrite_tag as an input processor on a threaded tail input in 4.0.3 produced a tight epoll_wait(0) spin at 100% CPU; the workaround is disabling threaded: true or running multiline as a regular filter.

Regex engine choice compounds here too. An OTel Collector test with 2 KiB MySQL slow-query lines capped at ~25,000 RPS with Go’s standard regexp multiline and sustained ~45,000 RPS with go-re2. Prefer built-in parsers (docker, cri) over custom multiline regexes, and set a bounded flush timeout so partial groups don’t accumulate.

Memory allocation and zero-copy parsing

Beyond pattern matching and I/O, parsers pay for how they hold each line in memory. Allocating fresh strings for log records and their fields adds allocator time plus garbage collection time; borrowing views into an existing buffer (string_view in C++, &str/Cow in Rust, []byte in Go) skips both.

You will see a 30–60% CPU reduction quoted for zero-copy parsing. The evidence partially supports that range, but the honest span is wider, roughly 12% to 8×. Results depend on record size and escape frequency, especially how completely the implementation eliminates allocations:

• Datadog’s log tokenizer switched to reusable borrowed buffers and got a geomean 2.12× speedup (3.22× on short lines) with allocations going from 2 to 0 per operation.
• Grafana Loki’s logfmt parser got 46.65% faster from string-to-byte-slice conversion; a log-reader change cut 26% of time and 91% of bytes allocated in a log-reader benchmark.
• Fluent Bit’s arena allocator for large protobuf log requests cut 16.66% of task-clock in C.
• A memory-mapped std::string_view approach in C++ ran 2.3–2.8× faster; a Rust borrow-vs-copy benchmark measured 6× on 10,000 lines, 30,000 allocations down to 1.

For context on the ceiling, a 2024 ASPLOS warehouse-scale study found malloc/free consuming 4.3% of fleet CPU cycles overall, while one parser case study found 25% of runtime in new. In a parser, allocation is a much bigger slice than fleet averages suggest.

Syscall and context-switch overhead

The last driver hides below the application code entirely: kernel crossings. Each one has a documented cost profile in log pipelines, especially Go-based ones.

• stat storms: covered above; the single largest CPU consumer in OpenAI’s Fluent Bit profile, at roughly 35% of thread time.
• fsync on checkpoints: Grafana Loki’s compaction spent 75.75% of syscall time in futex and 19,479 µs per fsync call; swapping fsync for fdatasync cut the job from 4 minutes to under 40 seconds in a Loki compaction test. Busy Filebeat instances run 10–20 fsync operations/second, with some configurations hitting 1,400 registry writes/second.
• futex contention: Google reported ~5% of fleet CPU cycles in futex calls through fleet futex data. One Go service with 200 goroutines contending on a single channel showed 55–70% of CPU in runtime.futex in a channel-contention profile.
• epoll_wait churn: a 192-core Go program with >2,500 sockets spent ~65% of total CPU in netpoll → epoll_wait in an epoll profile. One Go streaming service found zero-timeout epoll_wait calls returning nothing; changing the timeout from 0 to 1 ms cut staging CPU from 6% to 1% and 30–70% in production.

Kernel crossings appear in system profiles, so diagnosis has to start there.

How to profile log parser CPU overhead

Engineers found every fix above by profiling the running pipeline. Two things are worth taking from how OpenAI ran theirs. First, start with a hypothesis, then let the profile overturn it — theirs pointed at Lua string processing and the answer turned out to be a syscall, and a team that had trusted the hypothesis would have spent its effort in the wrong place. Second, the barrier is lower than it looks. You don’t need to write C to run perf; you need root, fifteen minutes, and enough patience to read the output (and a model can help you get the flags right).

Before touching config, follow this workflow:

1. Start with the agent’s own metrics. Logstash exposes a Hot Threads API and per-plugin events.duration_in_millis via the Node Stats API; worker_utilization near 100% means workers are saturated, and worker_millis_per_event names the expensive plugin. One Elastic example traced 84,193 ms of cumulative grok duration across 49,483 events to backtracking.
2. Disable filters one at a time. Binary-search your pipeline: comment out a filter stage, replay traffic, measure. This is crude but isolates the costly stage, whether parsing, buffering, output, or transport.
3. Profile Go agents with pprof. For OTel Collector, Vector-adjacent Go tooling, or custom shippers: go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30, then top5 -cum and list <funcname>. This is how an OTel Collector maintainer found filename metadata parsing eating ~30% of log-parsing time, and how Coralogix found GC consuming ~29% of a 10B-logs/day service’s CPU, recovering ~10% with a GOGC change.
4. Generate a flame graph for C or mixed workloads. perf record -F 99 -p PID -g -- sleep 30, then Brendan Gregg’s stackcollapse and flamegraph scripts. A wide stat or futex tower identifies syscall overhead.
5. Check syscalls last, and carefully. strace pauses the target at every syscall entry and exit and has slowed workloads up to 442×; use strace -c briefly for summaries only. perf trace caused only a 1.36× slowdown in a separate dd benchmark, where strace itself caused a 173× slowdown, and bpftrace aggregates in-kernel for production-safe counting.

Once a symbol is hot, account for it before you change anything. OpenAI’s team asked which code paths could produce fstat calls, found two, and eliminated the periodic one on volume alone before touching inotify. Then they validated the one-line fix on a single test cluster before the fleet. Use the profile to identify whether CPU is going to pattern matching, allocation, polling, or kernel calls. Configuration is where you act on the result.

Tuning Fluent Bit and Logstash to reduce CPU

Once the profile points at a stage, these are the highest-yield knobs in the two most common pipelines.

Fluent Bit tuning

For Fluent Bit:

• inotify_watcher: defaults to true. At high per-file write rates, set it false for stat-based watching; measured wins range from 50% (OpenAI, on their busiest pods) to ~9× (issue #11096). Check the changelog first — OpenAI is proposing an upstream minimum-interval option that would get you most of this without turning inotify off.
• refresh_interval: defaults to 60s; the docs note that increasing scanning frequency raises CPU load. Don’t lower it without a reason — though note that OpenAI ran it at 5s and it still wasn’t their problem. Periodic polling scales with file count; inotify scales with write rate, and write rate is usually the larger number.
• Lua filters: a 1M-sample benchmark ran 2.70s without a Lua filter and 4.98s with one, and Lua fully occupies a single core under load because all multi-threaded inputs converge into one single-threaded filter (issue #8088). Move logic into built-in filters where possible. Real cost, but not automatically your largest one: OpenAI does its enrichment in Lua and syscalls still dominated the profile.
• Version pinning: several of the worst CPU incidents come from version-specific bugs. Fluent Bit maintainers fixed the multiline buffer bug in v3.1.9/v3.2.0 and the rewrite_tag emitter busy loop affecting 3.0.0–3.2.4 (issue #9939). They also fixed the in_forward 100% CPU hang in v4.2.7, while setting net.io_timeout mitigated a TLS tight retry loop in 4.2.2 (issue #11551). Track changelogs like you track CVEs.
• Retries: set a finite retry_limit (maintainers suggest 5–10); one unreachable destination generated 1.2M DNS requests in under 15 minutes during a DNS retry storm before the v3.2.0 fix.

Logstash tuning

For Logstash:

• pipeline.workers: defaults to host core count. Elastic warns that for CPU-bound plugins, extra concurrency lowers throughput as workers contend and context-switch, and advises changing one thing at a time. On multi-pipeline hosts the guidance gets aggressive: for 26 pipelines on 16 cores, Elastic suggested possibly as low as 1 worker per pipeline in its multi-pipeline guidance.
• Queue type: on an 8-core host, memory queues kept scaling to 15,550 eps at 32 workers in a queue benchmark while persistent queues plateaued around 9,500. If you need PQ durability, never set queue.checkpoint.writes: 1; Elastic’s docs say forcing an fsync per event “can severely impact performance.”
• JVM heap: set 4–8 GB with Xms = Xmx. An undersized heap turns CPU charts spiky because the JVM stops the world for full GCs.
• Filter choice: break_on_match: true (the default) stops grok after the first hit; ordering most-matched patterns first bought a 9.2% throughput gain in one Cisco ASA pipeline, and switching to dissect more than doubled per-doc speed in an Apache Tomcat pipeline.

Tuning gets you a well-behaved agent on an idealized node. Kubernetes then adds its own multipliers.

Kubernetes and container factors that amplify overhead

The same agent config costs more inside a throttled container than on bare metal, for reasons that live in the kernel scheduler and the container runtime.

• CFS throttling: a 100m CPU limit means 10ms of quota per 100ms period; when the quota runs out, every thread halts for the rest of the period. The kubernetes-mixin CPUThrottlingHigh alert fires at 25% throttled periods, and above 50% latency impact is near-certain. A throttled log agent falls behind, buffers grow, and the catch-up burst throttles it again. The failure isn’t contained to the agent: OpenAI’s reason for keeping Fluent Bit lean is that an agent exceeding its quota can drop logs or degrade a neighboring application on the same node.
• GOMAXPROCS mismatch: Go 1.5–1.24 defaulted GOMAXPROCS to total host cores, ignoring container limits. Uber’s automaxprocs data shows that under a 2-core quota, a service delivered 22,191 RPS at GOMAXPROCS=24 versus 44,715 RPS at GOMAXPROCS=2 in an automaxprocs benchmark, with throttling eliminated by matching the quota. Go 1.25 finally made GOMAXPROCS container-aware; older Go-based agents (Promtail, Filebeat, OTel Collector builds) need automaxprocs or an explicit env var.
• Container log routing: containerd’s CRI format wraps every line with a timestamp, stream, and partial/full tag, using a 4096-byte read buffer, so long lines arrive as P, P, ..., F chunks your agent must reassemble. Kubelet rotates at containerLogMaxSize: 10Mi with 5 files, so tailers constantly re-discover files. One measurement found file tailing used 30–50% more CPU than forward-socket delivery in a Kubernetes tailing test at the same log rate, driven by lines per second rather than file count.
• EKS and Bottlerocket specifics: AL2023 (cgroup v2) is the current EKS default, AL2 AMIs stopped publishing November 26, 2025, and Bottlerocket defaulted new variants to cgroup v2 from release 1.13.0. Watch DaemonSet-specific bugs too: OTel Collector’s kubeletstats receiver spent ~77% of pprof CPU on repeated TLS handshakes per scrape in an EKS kubeletstats profile on AL2023 EKS nodes.

All of this assumes one agent per node. Most clusters run several, and that multiplies every cost in this article.

How log parsing overhead fits the broader observability cost problem

Agent sprawl is the pattern where each node runs an APM agent, a log shipper, a metrics exporter, and an OTel collector, each tailing files or intercepting traffic, each with its own parse path and failure modes. Every driver above (backtracking, polling, allocation, syscalls) is paid once per agent.

The cost is easy to underweight because it is spread thin. OpenAI’s framing makes it concrete: whatever a DaemonSet reserves, every node in the fleet stops contributing to the cluster. That turns a per-pod number into a fleet-wide subtraction, and it sets a high bar for adding another one — their stated goal is fewer DaemonSets, each using less. A cluster running four per-node agents has made that subtraction four times, mostly without deciding to.

Each agent’s implementation language sets its performance ceiling. The 2026 Kubernetes benchmark, independent of all tested vendors and reproducible from a public repo, capped each collector at 1 CPU and 1 GiB and measured maximum throughput on ~216-byte JSON logs:

C and Rust lead. Go sits in the middle, while Ruby trails by 6×. VictoriaMetrics did not test Logstash; the most recent figures covering it are Vector’s own test harness, where Logstash managed 4.6 MiB/s on regex parsing vs. Fluent Bit’s 20.5 MiB/s, with the caveat that Vector publishes that harness.

But even the fastest user-space parser still tails files, still parses per line, and still runs once per signal type. Kernel-level collection changes what work exists at all.

How groundcover’s Flora eBPF sensor avoids parse-time CPU

groundcover’s Flora eBPF sensor deploys as a single DaemonSet, one pod per node, and captures metrics, traces, logs, and Kubernetes events directly from the Linux kernel. Flora replaces per-service log shippers and language-specific SDKs with one node-level sensor. That architecture prevents a second or third agent from multiplying the tail-and-parse cost across the node. The reduction is architectural: a kernel-level sensor performs the collection work once per node rather than once per agent.

groundcover’s own benchmark used a Golang HTTP server at 3,000 requests/second on an isolated Kubernetes cluster. The groundcover benchmark measured Flora at +9% application CPU overhead versus +249% for the Datadog agent, and total combined CPU 73% below Datadog’s.

That 73% figure remains current as of August 2026, and groundcover last updated the post on July 24, 2026. The metric measures combined agent-plus-application CPU; it does not report standalone sensor cost. groundcover published the benchmark, and its own guide states that no neutral head-to-head benchmark of the Datadog Agent against eBPF collectors exists in the public record under its neutral benchmark caveat and advises treating both sides’ numbers as vendor-run figures and testing on your own cluster.

You can validate the result on one cluster with the free tier. It includes the full BYOC architecture, keeps the data plane in your VPC, and provides full-cluster visibility within hours without a credit card.

What’s next for reducing parsing overhead

Architecture changes where parsing happens; the frontier of research changes how fast parsing itself can go.

• SIMD/vectorized parsing: simdjson parses NDJSON at 3.5 GB/s on a single core, and Fluent Bit v4.1.0 shipped SIMD JSON-encoding improvements (via yyjson) making applicable plugins 2.5× faster. ByteDance’s Sonic brings the same approach to Go, with an independent 2025 benchmark measuring +548% over the standard library in an independent JSON benchmark.
• Binary and structured formats: Fluent Bit already converts every event to MessagePack internally for performance. A Go benchmark puts protobuf unmarshal at ~5× JSON speed, and ClickHouse ingested Native+LZ4 51% faster than JSONEachRow in a ClickHouse format benchmark. The caveat from JVM benchmarks: library choice can dominate format choice, since dsl-json beat protobuf on combined throughput there.
• Node-level stream processing: processing at the edge before shipping, the model Flora uses and the direction research like LogCrisp research (AVX SIMD over compressed logs, 3.8× higher ingestion speed) points toward, cuts both parse CPU and egress volume.

One more thing the OpenAI result suggests: the wins are still out there. A team running one of the largest logging pipelines in the world found half its agent’s CPU going somewhere nobody expected, in mature open-source software that thousands of companies run, and fixed it in a line. Most of that software has never been profiled under a load like yours.