Reducing log filtering CPU overhead

Log filtering CPU overhead is measurable, and you can often fix it without dropping signals you care about. On Fluent Bit, a chain of filters can cut throughput by roughly 89% versus an unfiltered pipeline. A production profile attributed 10% of total OpenTelemetry Collector CPU to regex matching inside a transform processor. This guide works through the problem in order:

1. The mechanism: why per-record filter evaluation compounds at high throughput
2. Diagnosis: profiling and attributing cost to a specific filter
3. The regex tax and its fixes
4. Filter placement: drop early to pay less
5. WASM and Lua filter overhead in Fluent Bit
6. Batching and backpressure tuning
7. Compression and rotation CPU cost
8. Architectural escape valves
9. An optimization checklist and node-level validation option

Why log filtering rules drive CPU overhead

Every filter you add executes against every record, and in Fluent Bit that execution happens single-threaded in the main engine thread. The official documentation states it plainly: “Filters always run in the main thread, and using multiple filters can introduce performance overhead, particularly under heavy workloads.” At 10,000 or 100,000 records per second, a few microseconds of per-record filter cost becomes the pipeline’s dominant CPU consumer.

Fluent Bit’s PR #9399 quantified the filter-on versus filter-off delta on the same hardware across Kubernetes log pipeline configurations:

The PR author describes the failure mode directly: “The input pauses constantly because the engine thread is backed up since all filters are executed single-thread in the engine thread.” A separate report, issue #7633, found the Docker parser filter alone dropped throughput from 100 MB/s to 15 MB/s while the pipeline thread pegged at 100% CPU.

For baseline context, the VictoriaMetrics benchmark measured Fluent Bit at 0.260 cores and the OTel Collector at 0.491 cores at roughly 10,000 logs per second on identical hardware. That benchmark explicitly cannot separate parsing and filtering costs from transformation costs, which is exactly why the next section exists: you have to attribute the cost yourself before you can cut it.

How to diagnose which filter is burning CPU

Attribute CPU to a named pipeline stage before changing a rule. First rule out scheduler throttling, then use profiles to map the remaining hot frames to a filter or processor.

Step 1: profile the agent with pprof and flame graphs

The OTel Collector is a Go binary and ships a pprof extension (beta) that exposes Go’s net/http/pprof endpoint on port 1777:

extensions:
  pprof:
    endpoint: localhost:1777

service:
  extensions: [pprof]

Capture a 30-second CPU profile with go tool pprof http://localhost:1777/debug/pprof/profile?seconds=30. On Kubernetes, follow the profiling workflow: port-forward the pprof service and curl the profile, then open it with go tool pprof -http=: for the interactive flame graph UI. As of Go 1.26, the flame graph is the default view.

Read the flame graph using these rules:

• Frame width: Width indicates how often a frame appeared in samples.
• Horizontal position: The flame graph x-axis sorts frames alphabetically.
• Top edge: The top edge shows what was on-CPU, and top-edge plateaus identify hot functions.
• Period comparison: Compare low-CPU and high-CPU periods side by side, a workflow Grafana Labs documented when a single mutex function jumped from 33% to 71% of CPU between periods.

Fluent Bit is written in C and does not expose a pprof-style endpoint. Its built-in HTTP monitoring server (http_server on, port 2020) exposes per-plugin throughput and error metrics at /api/v1/metrics; those metrics do not attribute CPU consumption. For actual CPU flame graphs you need an external eBPF profiler, covered under escape valves below. Expected outcome of this step: a named function owning a disproportionate share of samples, or plugin metrics that narrow the investigation.

Step 2: check GOMAXPROCS against container CPU limits

Before blaming a filter, rule out CFS throttling, because it produces the same symptom: high latency and apparent CPU saturation that does not correspond to actual work. On Go versions before 1.25, GOMAXPROCS defaults to the host’s logical CPU count and ignores cgroup quotas, so a Collector limited to 2 CPUs on a 64-core node runs 64 scheduler threads and gets paused by the kernel for the remainder of each 100 ms CFS period.

The documented incidents share one signature. Uber’s internal load balancer on a 2-core quota went from 22,191 to 44,715 RPS and from 131,923 throttled periods to zero by setting GOMAXPROCS from the default 24 down to 2. A November 2025 benchmark by Michal Drozd on a 64-CPU host with a 2-CPU limit measured P99 latency dropping from 890 ms to 35 ms and throttle rate from 72% to 3% after the same fix. Grafana Tempo’s maintainers documented the identical failure mode on large Kubernetes nodes.

Two fixes exist. Go 1.25 makes the runtime container-aware by default, deriving GOMAXPROCS from the cgroup CPU bandwidth limit and updating it periodically; for older builds, import uber-go/automaxprocs or set the GOMAXPROCS environment variable to match the limit. Confirm the diagnosis with the container_cpu_cfs_throttled_seconds_total metric: if it is climbing, fix the quota alignment first, then re-profile. With throttling ruled out, the remaining CPU belongs to the pipeline itself.

Step 3: attribute cost to a specific filter or processor

Map the hot frames from step 1 to a named pipeline stage. Regex cost shows up as Go regexp frames under the transform or filter processor in the Collector and as Oniguruma/Joni frames in Logstash. For scripted filters, WASM cost appears as wasm_runtime_call_wasm and buffer-copy functions in Fluent Bit, while Lua cost appears under the Lua filter’s callback path.

A OneUptime production report from February 2026 shows what a completed attribution looks like: the transform processor consumed 35% of total Collector CPU. Within that total, OTTL execution accounted for 25% of Collector CPU and regex matching accounted for 10%. That level of specificity, a percentage attached to a named processor, is the exit criterion for diagnosis. Once you have it, regex is a common culprit in production profiles, which is where the fixes start.

The regex tax and how to fix it

Backtracking regex engines pay their worst-case cost on lines that do not match. Elastic’s engineering analysis of Grok found that checking a non-matching line can be up to 6 times slower than a successful match, because without anchors the engine retries the pattern against substrings of the input. Greedy quantifiers compound this: Microsoft’s regex documentation shows a single greedy .* forcing character-by-character retreat, while nested quantifiers produce comparison counts that increase exponentially, with a 40-character input requiring approximately 1 trillion comparisons.

Elastic’s guidance applies to every backtracking engine in this space: “In regular expression design, the best thing you can do to aid the regex engine is to reduce the amount of guessing it needs to do. This is why greedy patterns are generally avoided.” Two concrete fixes follow, one for OTel-style pipelines and one for Logstash.

Exact match vs. regex

Replacing regex with exact or prefix matching is the highest-ROI change available, and the OTel Collector’s own development history proves it. The maintainers added HasPrefix and HasSuffix functions explicitly “to avoid regex cost (when using IsMatch) to check for prefix and suffix for strings.” Equality comparisons like log.severity_number < SEVERITY_NUMBER_WARN or attributes["service"] == "checkout" skip the regex engine entirely.

When regex is unavoidable, two rules from Collector profiling apply:

• Use static patterns with IsMatch. The Collector caches compiled patterns only when the configuration uses string literals; a dynamic-pattern issue documents that dynamic patterns force regex compilation on every evaluation.
• Anchor everything. The OneUptime team’s fix for the regex share of CPU they profiled was to anchor patterns with ^ and $ and remove greedy .* quantifiers, because anchored patterns let the engine fail fast on non-matching strings.

The before/after framing is worth internalizing: when anchoring preserves the intended match boundaries, it can reduce CPU without changing output. The CPU difference comes from how much guessing the engine performs per record. Logstash users face the same tax under a different name.

Grok pattern optimization in Logstash

GREEDYDATA is the Logstash spelling of the greedy-quantifier problem, and anchoring is again the fix. Elastic’s benchmarks found that adding ^ and $ anchors made initial match-failure detection roughly 10 times faster, and that tiered matching combined with anchors is always the recommended structure. For delimited formats, skip regex entirely: Elastic’s Dissect filter measured 60.99 µs per event against 199.89 µs for Grok + CSV on the same Palo Alto syslog workload.

Cap runaway patterns with the Grok plugin’s timeout settings. timeout_millis defaults to 30000 ms per pattern with 250 ms quantization, and timed-out events receive the _groktimeout tag. Set timeout_scope => event to enforce one timeout across all patterns on an event; the docs state the plugin “can achieve similar safeguard against runaway matchers with significantly less overhead” in that mode.

Anchoring and timeouts bound the cost of each filter evaluation. The next lever bounds how many records ever reach a filter at all.

Filter placement: drop early to pay less

A record dropped at the input stage costs nothing downstream. Red Hat’s OpenShift sequencing guide makes the order explicit: filters execute in pipeline order, and “a log record dropped by an earlier filter does not reach subsequent filters.” The guide directs operators to “apply drop filters first to remove unwanted log records, then apply prune filters to remove fields from the remaining records. This reduces processing work.” Dropping a whole record requires one decision. Pruning fields requires the pipeline to walk and mutate every surviving record, so drop rules always come first. OpenShift 4.16 goes further, documenting that stream exclusion guidance reduces collector CPU and memory load.

Fluent Bit gives you four placement options, cheapest first:

• Input path exclusion: The tail input’s exclude_path glob skips files before records exist.
• Pod-level exclusion: The Kubernetes filter’s pod exclusion annotation lets workloads opt out of collection with fluentbit.io/exclude.
• Input-attached processors (YAML only): Processors run in the input plugin’s thread instead of the main engine thread, and the processor performance docs state that “processors perform better than filters, and when chaining them, there are no encoding/decoding performance penalties.”
• Global filters: The grep exclusion rule drops records by field regex, but it runs in the contended main thread.

The OTel Collector mirrors the same hierarchy. The filelog receiver’s include/exclude globs and filter operator act before records enter the pipeline, and the OTel blog’s collection-stage guidance is to “do as much work during the collection as possible.” Within the filter processor, condition placement matters enormously: a collector-contrib benchmark measured a resource-level condition at 116,931 ns/op against 528,010 ns/op for the same logic repeated in a per-record where clause, roughly 4.5× faster. Since v0.146.0, hierarchical filtering lets the filter processor drop a matching resource subtree without evaluating lower levels.

Placement solves the where; the next two sections address filters whose per-record cost is high wherever they sit.

WASM and Lua filter overhead in Fluent Bit

Misconception: “Fluent Bit recreates the WASM instance for every log line, which is why WASM filters are slow.” The current source code shows the opposite: cb_wasm_init and cb_wasm_filter instantiate the WASM module once at filter startup and retrieve that persistent instance for every record.

The per-record invocation path includes the WASM buffer copy during function dispatch, followed by the sandboxed call. JSON and msgpack encoding round-trips add another cost that a maintainer identified as “one of the culprits of consuming CPU” in an encoding overhead issue.

The measured gap between scripted and native filters is large. A benchmark reported in Fluent Bit issue #7112 for version 2.0 processed 1 million records in 2.701 s with no filter and 4.984 s with a Lua filter. The interpreted no-op WASM filter took 711 s, while the benchmark authors AOT-compiled the same WASM filter with flb_wamrc and measured 124 s.

A DNS analytics case study from March 2026 reported a Go/WASM filter running at 60–90% CPU where a native C filter did the same work at 1–3%. The case study attributed the gap to per-event serialization and sandbox context switches. It also identified Go runtime overhead inside the module.

Four mitigations follow directly from the mechanism. Enable AOT compilation; the official WASM docs say Fluent Bit disables it by default. Set event_format msgpack to use the MsgPack pass-through added in PR #8431 and skip the JSON round-trip.

Configure wasm_heap_size and wasm_stack_size, available since v3.1.5, to prevent allocation failures for larger records; the research does not demonstrate a CPU benefit. If the logic is simple enough to express as grep rules or native filters, the case-study results support rewriting hot-path logic as a native filter instead of continuing to tune the scripted implementation. Scripted filters are a per-record cost; batching, next, is how you amortize the fixed costs that remain.

Tuning batch processors and backpressure

The OTel Collector’s batch processor amortizes per-export overhead. Its v0.159.0 defaults set send_batch_size: 8192 and timeout: 200ms. The send_batch_max_size: 0 default leaves the maximum unlimited.

Placement and sizing follow official guidance:

• Batch semantics: send_batch_size triggers a send when the batch reaches 8192 items. send_batch_max_size sets the cap, while setting timeout to zero causes immediate sends that discard the batching benefit entirely.
• Order: Follow the processor order guidance: memory_limiter first, then sampling and filtering, batch last, because “batching should happen after any data drops such as sampling.”
• Edge vs. gateway sizing: The agent-to-gateway guidance recommends that agents use “smaller batch sizes and shorter timeouts to minimize latency and memory usage” while gateways use “larger batch sizes and longer timeouts for better throughput.”
• Saturation guard: The memory limiter with a 1-second check_interval and spike_limit_mib around 20% of limit_mib backpressures the pipeline before it OOMs, and the otelcol_exporter_queue_size reaching 60–70% of capacity is the documented scale-up signal.

Plan for the batch processor’s retirement. The migration RFC schedules a deprecation warning at v0.160.0 and removal from the default distribution at v0.166.0. Exporter-level sending_queue::batch and the development-stability queuebatchprocessor are the replacements. Filtering, batching, and backpressure cover the pipeline itself; the remaining CPU consumers sit at the edges, starting with compression.

Log compression and rotation CPU cost

On one compression core, the USENIX HotEdge20 study found bzip2 CPU-bound while zstd’s end-to-end time changed little as cores increased. The study measured 84 seconds end-to-end for bzip2 against 38 seconds for zstd, concluding that “with only one core, compute capability is the main bottleneck of bzip2.” Adding a second core cut bzip2’s time from 84 to 44 seconds. Four cores cut the time again to 36 seconds, while zstd stayed nearly flat.

Single-thread throughput numbers explain why. lzbench results on a single EPYC thread measured zstd -1 at 422 MB/s compression against 93 MB/s for zlib -1 and 13.1 MB/s for bzip2 -9. lzbench measured bzip2 decompression at 37.5 MB/s, the lowest decompression throughput in the test. If log rotation runs on a one- or two-core edge box, algorithm choice is the whole ballgame; size the rotation frequency so a compression job finishes well within the rotation interval, or a backlog of uncompressed files stacks up behind a pegged thread.

Check what your stack supports before choosing. Neither Fluent Bit’s HTTP output documentation nor Logstash’s output documentation lists bzip2. Fluent Bit’s HTTP and S3 outputs support zstd. Logstash offers zstd only through the Kafka output. For file rotation, logrotate defaults to /bin/gzip -6 and switches to zstd with a one-line compresscmd /usr/bin/zstd. When tuning within the agent is exhausted, the remaining options change the architecture.

Architectural escape valves

Four structural changes remain when rule-level tuning hits its floor.

Audit OTTL expression complexity. Collector engineering issues document the Time function consuming roughly 5% of compute in a real configuration. Production profiles also found boxing/unboxing to any exceeding 5% CPU.

The replaceAllPatterns benchmark found mode_key running ~61% slower than mode_value. Run make benchmark-ottl from pkg/ottl to measure your statements before and after changes.

Pick the exporter transport deliberately. The OTLP spec recommends http/protobuf as the default, and the Thyme benchmark at 100k logs/sec measured OTLP/HTTP with zstd at 1.70 cores against 2.12 for OTLP/gRPC with zstd. The uncompressed variants used near 3.7–3.8 cores for both.

Compression choice matters more than transport in those results. The Collector’s own configgrpc benchmarks show zstd compressing large log payloads at 286 MB/s against gzip’s 105 MB/s, while both algorithms expand small payloads. Note that collector-contrib’s own testbed results conflict on CPU, so benchmark with your payloads.

Offload filtering downstream. Shipping raw logs to a remote syslog target or SIEM moves filter CPU off the edge entirely. The trade is edge CPU for network egress and ingest-side cost, which pays off when edge hardware is the scarce resource.

Run continuous profiling. Grafana Pyroscope aggregates always-on profiles so the low-versus-high CPU comparison from the diagnosis section becomes routine during normal operations and incidents. For the Collector, an unmerged Pyroscope pull request describes configuring Grafana Agent to scrape profiles from the Collector’s pprof extension.

For Fluent Bit, the Alloy pyroscope.ebpf component profiles compiled C binaries system-wide on Linux kernel 4.9 or later when Alloy runs as root in the host PID namespace, though this path is not explicitly documented for Fluent Bit. The OTLP Profiles pipeline into Pyroscope 2.0+ is an alternative, with the caveat that the profiles signal is alpha.

There is also a fifth valve: change where filtering happens in the first place, which the checklist below closes on.

Optimization checklist for Fluent Bit and OTel Collector

Work through these in order; the earlier items are easier to apply and have the strongest evidence behind them.

• Profile before changing anything. Enable the Collector’s pprof extension or Fluent Bit’s HTTP monitoring server and get a named stage with a CPU percentage attached where profiling data supports it.
• Align GOMAXPROCS with the container CPU limit. Upgrade to Go 1.25 builds or import automaxprocs, and watch the CFS throttling metric to confirm the fix landed.
• Convert regex to exact, prefix, or substring matching. Use OTTL equality and HasPrefix/HasSuffix; where regex stays, anchor with ^ and $, keep patterns static, and kill greedy quantifiers including GREEDYDATA.
• Set Grok timeouts. timeout_millis with timeout_scope => event bounds worst-case pattern cost in Logstash.
• Drop early, prune late. Exclude at the input with glob paths, pod annotations, or receiver operators. Place drop conditions at resource level and run drop filters before prune filters.
• Prefer Fluent Bit processors over filters. Input-attached processors escape the single-threaded engine thread and skip encoding penalties.
• Contain scripted filter cost. AOT-compile WASM, switch to msgpack event format, and rewrite hot-path logic natively when the profile justifies it.
• Tune batching and backpressure. memory_limiter first, batch last, smaller batches at the edge, larger at the gateway, and plan the batch processor migration before v0.166.0.
• Replace gzip and bzip2 with zstd wherever the output or rotation tooling supports it, especially on low-core hosts.

Node-level filtering moves drop rules into the sensor before network transfer or storage. groundcover’s Flora eBPF sensor deploys as a single DaemonSet, one pod per node, and captures logs, metrics, traces, and Kubernetes events directly from the Linux kernel with zero application instrumentation. Log pipelines run inside the sensor, so drop rules using OTTL set(drop, true) statements execute at the node before network or storage; IsMatch() supports substring and regex matching.

The recommended rule order matches this article: drop rules, then quick parsing, then complex parsing, then obfuscation. The pipeline reports its own processing latency and flags anything above 10 ms with guidance to simplify the regex.

groundcover’s own April 2023 benchmark, a vendor-stated result against a Go HTTP server at 3,000 requests per second, measured Flora using 73% less total CPU than the Datadog agent. If you want to validate the architecture on your own cluster, the free plan includes BYOC (Bring Your Own Cloud, where your data plane runs in your own cloud account) and requires no credit card. Deploy Flora on one cluster and evaluate full-cluster visibility within hours.

FAQ

These answers summarize the diagnostic and optimization decisions operators most often need to make.

How much CPU does log filtering cost on edge agents?

It depends entirely on the filter chain, and the measured range is wide. Fluent Bit PR #9399 shows an unfiltered pipeline sustaining over 41 MB/s while a chained filter configuration on identical hardware managed under 5 MB/s, and a Docker parser filter alone cut throughput by about 85% in a separate report. Treat any per-record filter as a multiplier on your log rate.

How do I identify which filter rule is the culprit?

Profile the agent instead of reading configs. On the OTel Collector, enable the pprof extension, pull a CPU profile, and find the widest plateaus in the flame graph. On Fluent Bit, use the monitoring API’s per-plugin throughput and error metrics to narrow the investigation, then use an external eBPF profiler for CPU attribution. First rule out CFS throttling by checking GOMAXPROCS against the container limit, since throttling mimics filter overhead.

Why does regex cause CPU spikes?

Backtracking engines pay the most on lines that fail to match, up to 6× the cost of a successful match in Elastic’s Grok measurements, and greedy or nested quantifiers multiply the guessing. High-volume non-matching traffic hitting an unanchored pattern is the classic spike signature.

Should I filter at the input or the output?

Input, wherever possible. A record excluded at the input stage never consumes parser, filter, buffer, or exporter CPU, and both Red Hat’s OpenShift Logging docs and the OTel container-log guidance recommend doing the work as early in the pipeline as collection allows.

Are WASM and Lua filters slower than native filters?

Yes, by a wide margin. In a controlled Fluent Bit test, a Lua filter added roughly 84% to processing time while an interpreted WASM filter was orders of magnitude slower; AOT compilation recovers much of the WASM gap but native filters remain far cheaper. The instance itself is persistent, so the cost comes from buffer handling and sandbox calls, including encoding round-trips.

What profiling tools should I use?

Use Go’s pprof with flame graphs for the OTel Collector. For Fluent Bit, use the built-in monitoring API and an external eBPF profiler; Grafana Pyroscope via Alloy’s eBPF component or the OTLP Profiles pipeline supports continuous profiling across both. Pyroscope’s eBPF path profiles C binaries, which covers Fluent Bit even though that integration is undocumented.

How do batch and backpressure settings affect CPU?

Batching amortizes fixed per-export costs, so undersized batches or a zero timeout waste CPU on small sends. Keep memory_limiter as the first processor to backpressure the pipeline before saturation, and scale out when the exporter queue passes roughly two-thirds of capacity.

Is dropping a whole record cheaper than pruning fields?

Yes. Neither the OTel nor OpenShift docs publish a per-record CPU number for the comparison, but OpenShift’s guidance to apply drop filters before prune filters “reduces processing work” for a structural reason: a dropped record skips every downstream filter, while a pruned record must still be walked and mutated by each remaining stage.