Quantifying Python logging performance overhead: benchmarks, mechanics, and production fixes

Synchronous I/O handlers in CPython’s logging module, including StreamHandler and FileHandler, write and flush each emitted record on the calling thread, under a lock, before your code continues. In a high-throughput Python service, that design turns each log line into a synchronous tax of several microseconds. The tax compounds in tight loops and concurrent execution, including threads and asyncio event loops.

This article first defines Python logging performance overhead and quantifies it with published benchmarks. It then traces the mechanics of a single log call before covering configuration-level fixes such as QueueHandler offloading, isEnabledFor guards, metadata toggles, and buffering decisions. Finally, it compares handlers, evaluates third-party libraries, examines the asyncio and PyPy gaps, and closes with a production checklist and FAQ.

What is Python logging performance overhead

Python logging performance overhead is the per-call CPU time and latency your application pays for every call into the standard library logging module, whether or not a handler writes the record. The overhead splits into two categories that respond to different fixes. CPU cost covers level checks, argument formatting, LogRecord construction, and handler dispatch; I/O cost covers synchronous writes and flushes from I/O handlers that block the calling thread until the stream’s write and flush methods return.

The distinction matters because most teams tune the wrong half. Filtered calls (a logger.debug() in production at INFO level) pay only CPU cost, and eager f-string arguments can multiply that cost twentyfold. Calls emitted through synchronous I/O handlers pay both, and the I/O half runs under a per-handler lock that serializes every thread in the process. The next section puts numbers on both paths.

How much overhead does Python logging add

An emitted stdlib log call costs 5–7 microseconds of dispatch, while a disabled-level call costs about 60 nanoseconds, a roughly 100× gap.

Published benchmark results

The October 2024 benchmark runs on Windows 11 measured Python 3.13.0 logging_simple (an emitted message, no interpolation) at 5.46 µs on an AMD Ryzen 9 7900 and 6.11 µs on an Intel Core i3-1315U. The same runs measured logging_format (%-style argument substitution) at 5.92–6.60 µs and put logging_silent, a logger.debug() call at a disabled level, at 59.6–63.3 ns. These tests form part of the pyperformance suite.

Two methodology notes qualify those figures. First, pyperformance’s logging benchmarks write to an in-memory io.StringIO sink to isolate dispatch cost from disk I/O (bugs.python.org #30815), so real file handlers add flush latency on top. Second, the Python version stamps every figure: logging_silent measured 102–106 ns on Python 3.10 on the same hardware (2023 run), and interpreter-wide work such as the PEP 659 specializing adaptive interpreter drove the improvement to ~60 ns by 3.13. The What’s New documents for 3.11 through 3.13 list no logging-specific performance optimizations.

Where the cost comes from

The per-call price combines synchronous handler I/O with two CPU costs: string formatting and LogRecord construction with metadata collection. For records that reach a real I/O handler, the flush dominates: StreamHandler.emit() calls self.flush() after every single record, under the handler lock, in the CPython source.

For the far more numerous filtered calls, argument construction dominates, and call style decides the bill. A 2026 practitioner harness using time.perf_counter_ns (median of 3 batches × 100,000 iterations; the benchmark authors did not state the CPython version) measured a disabled %-style call at 210 ns and a disabled f-string call at 4,180 ns. Python evaluates the f-string before logging checks the level, which makes the filtered call 20× more expensive for zero output.

How to measure it in your own app

Measure the disabled dispatch path and an enabled call to your real handler, then calculate the delta that a level change or queue offload can recover. The timeit docs recommend taking the min() of timeit.repeat() rather than the mean. The timing tool disables GC by default, so re-enable it with gc.enable() in setup because repeated LogRecord creation triggers collection in real services.

import timeit

setup = """
import gc, logging
gc.enable()
logging.basicConfig(filename="bench.log", level=logging.INFO)
logger = logging.getLogger("bench")
"""

emitted = min(timeit.repeat('logger.info("payload %s", 123)', setup=setup, number=10_000, repeat=5))
disabled = min(timeit.repeat('logger.debug("payload %s", 123)', setup=setup, number=10_000, repeat=5))
print(emitted / 10_000, disabled / 10_000)

For publishable numbers, use pyperf instead of raw timeit. It spawns 20 processes by default to control for ASLR and Python’s randomized hash function. pyperf system tune sets the CPU governor, and pyperf can pin workers to selected CPUs. Having established what the overhead costs and how to measure it, the next question is where each microsecond goes inside a single call.

How Python logging overhead works

A single emitted log call travels from the logger’s level check through LogRecord creation, handler filtering, lock acquisition, formatting, and a write-plus-flush to the sink. Each stage below is a distinct cost you can measure and, in most cases, remove.

Synchronous blocking I/O

A synchronous I/O handler blocks the calling thread until its flush completes because StreamHandler.emit() formats and writes the record, then calls self.flush() before returning, all inside the handler lock. FileHandler.emit() delegates to that same path after opening the file if needed.

FileHandler._open() opens a block-buffered stream with default buffering, typically 128 KB. That buffer buys nothing, because the per-record flush drains it to the OS on every emission. The per-record flush pushes each record through the buffered I/O layer to the OS page cache, which is the primary latency source for emitted records and the reason the QueueHandler fix later in this article exists.

Log level filtering and CPU cost

A DEBUG logger left enabled in production pays the full emit cost on every debug call, so the level check is your cheapest optimization surface. Logging methods short-circuit through isEnabledFor() before calling _log(), and Python 3.7 introduced per-level caching that made ignored calls about 50% faster.

The cached disabled path is the ~60 ns figure from the pyperformance data above. That price holds only if the arguments are free to construct, which is the subject of the next stage.

String formatting and lazy evaluation

%-style formatting is lazy and f-strings are not, which is the whole performance argument for the older syntax. The Logging HOWTO states that “Formatting of message arguments is deferred until it cannot be avoided”: msg % args runs inside Formatter.format() only when a record reaches a handler, so logger.debug("state: %s", obj) never touches obj’s __str__ at a disabled level.

Python builds the full string for logger.debug(f"state: {obj}") before logging checks the level. The 20× gap between disabled %-style and disabled f-string calls measured earlier comes from this mechanism, and at scale the call style matters more than the library choice for filtered events.

LogRecord creation and metadata collection

Every enabled call constructs a LogRecord, and its __init__ collects metadata whether your format string uses it or not. With defaults on, it calls threading.get_ident() and threading.current_thread().name, os.getpid(), the multiprocessing process name, and on Python 3.12+ the asyncio task name.

Caller location is the expensive field. Enabled logger calls run findCaller() via sys._getframe() unless logging._srcfile = None disables the lookup, regardless of whether the formatter renders %(filename)s or %(lineno)d. The field-attribution benchmark measured a format with caller info at 13,980 ns per emitted call against a 9,640 ns all-fields baseline, a 4,340 ns premium per line.

Locking and thread safety overhead

Every handler wraps emit() in a threading.RLock, so N threads sharing one handler serialize on it rather than logging in parallel. The mechanism sits in Handler.handle() (CPython source):

def handle(self, record):
    rv = self.filter(record)
    if rv:
        with self.lock:
            self.emit(record)
    return rv

The practitioner contention benchmark used 20,000 iterations per thread with a threading.Barrier start. It measured 31,200 ns per call at 1 thread, 48,700 at 2, 79,300 at 4, and 142,500 at 8.

The GIL compounds the problem by serializing the Python-level work around the lock. Free-threaded CPython builds under PEP 703 retain logging’s RLock, so they do not remove handler-level serialization. The Python 3.14 What’s New reports 5–10% single-threaded overhead for those builds.

The author-affiliated rapidlog benchmark shows the same shape in throughput terms for Python 3.13 on Windows with stdlib JSON configuration. It measured 9,317 logs/s single-threaded, dropping to 6,487 at 4 threads and 6,441 at 8. Having walked the mechanics, the next section turns each cost center into a fix.

Key optimizations and capabilities

Each fix below targets one of the cost centers above, and all of them are configuration-level changes rather than rewrites. Apply them in the order listed; the queue offload alone removes the largest I/O term from the caller thread.

Non-blocking logging with QueueHandler and QueueListener

QueueHandler removes downstream handler I/O from the caller thread, but its default caller-side work includes more than queue.put_nowait(). Before queuing, QueueHandler.prepare() formats and shallow-copies the record before clearing fields that cannot be pickled. The QueueListener thread then runs its configured downstream handlers, including their formatting and I/O. Override prepare() if your application must preserve record.msg and record.args so the listener can defer message formatting. The Logging Cookbook documents QueueHandler as the standard pattern for performance-critical threads:

import logging
import logging.handlers
import queue

log_queue = queue.Queue()  # unbounded: no records dropped under burst

queue_handler = logging.handlers.QueueHandler(log_queue)

file_handler = logging.FileHandler("app.log")
file_handler.setFormatter(
    logging.Formatter("%(asctime)s %(levelname)s %(name)s %(message)s")
)

listener = logging.handlers.QueueListener(
    log_queue, file_handler, respect_handler_level=True
)
listener.start()

root = logging.getLogger()
root.addHandler(queue_handler)
root.setLevel(logging.INFO)

# at clean shutdown, after application work completes:
listener.stop()

Four operational caveats come straight from CPython:

• Queue capacity: On a bounded queue, put_nowait() raises queue.Full and the record encounters the documented queue-full behavior: logging either drops it silently or prints an error to stderr, with no backpressure (gh-81651). Use an unbounded queue or size the queue for burst capacity.
• Shutdown ordering: You must call listener.stop() yourself because logging.shutdown() runs at atexit but does not stop the listener, since QueueListener is not a Handler.
• Sentinel timing: Stop application logging before calling listener.stop(). The listener exits when it reads the stop sentinel and never processes records that producers enqueue after that sentinel.
• Version integration: Python 3.12 added dictConfig integration with a handler.listener accessor. Python 3.14 lets QueueListener act as a context manager (gh-132106).

Guarding hot paths with isEnabledFor

Wrap any log call with expensive arguments in an explicit level guard so the argument work never runs at disabled levels:

if logger.isEnabledFor(logging.DEBUG):
    logger.debug("cart state: %s", expensive_serialize(cart))

The official docs recommend exactly this pattern, along with logging.disable(logging.DEBUG) as a process-wide floor when you want logging to short-circuit every debug call. The guard costs one cached level check; without it, a serialization call in a tight loop pays its full price on every iteration regardless of output.

Disabling expensive metadata collection

The Logging HOWTO’s optimization table documents module-level toggles for every metadata field LogRecord collects:

import logging

logging._srcfile = None          # skip sys._getframe() caller lookup
logging.logThreads = False       # skip thread ident and name
logging.logProcesses = False     # skip os.getpid()
logging.logMultiprocessing = False
logging.logAsyncioTasks = False  # Python 3.12+

The savings are measurable. The pyftpdlib project measured 100,000 logger.error() calls dropping from 1.838 s to 1.381 s after setting _srcfile = None, roughly 28%, and the field-attribution harness cited earlier found disabling PID collection saved 520 ns per call, rising to 1,180 ns with thread and multiprocessing fields off too. One warning: _srcfile is a single-underscore attribute outside logging.__all__, named in the official HOWTO but carrying no public API stability guarantee, so pin your Python version in tests if you rely on it.

fsync and OS write buffering

FileHandler never calls os.fsync(), and that default is the throughput-friendly choice. Its per-record flush() drains Python’s buffers to the OS page cache via write(2); forcing the page cache to physical storage is a separate os.fsync() syscall that the FileHandler implementation does not make.

The trade is durability for speed. A crash or power failure can erase records that the handler flushed but did not sync, and if you need per-record durability you must subclass the handler and call os.fsync(self.stream.fileno()) after flushing, paying a device flush per line. The delay=True constructor parameter changes none of this; it only controls deferred opening behavior. With the individual fixes covered, the next question is which handler to standardize on.

Handler showdown: FileHandler vs StreamHandler vs QueueHandler

The handlers differ in what the calling thread pays before continuing, which is the number that shows up in your request latency:

Treat the absolute milliseconds in that README as directional; they are maintainer-stated, from richbench on macOS 11, and the benchmark authors did not state the Python version. No authoritative public benchmark isolates FileHandler versus StreamHandler end-to-end with a documented CPython 3.10+ version and hardware, a gap worth knowing before you quote per-handler numbers in a design review. The structural conclusion holds regardless: QueueHandler is the only stdlib option that moves sink I/O off the caller thread, although default prepare() still leaves record formatting and copying there. That offload is why QueueHandler anchors the checklist later. Before that, the standard library deserves a comparison against its challengers.

How Python logging fits in: standard library vs alternatives

The alternatives differ in raw throughput, filtering behavior, and current Python support. The available benchmarks do not use one common methodology, so each result needs its stated scope.

• picologging: picologging is the fastest measured alternative, but its version support ends the discussion for many teams. The Microsoft project’s maintainer-stated benchmarks claim 2.5× faster FileHandler and 12.3× faster emitted debug calls than stdlib. They also claim 19.1× faster Formatter().format(), while the independent NexusLog suite measured 490,626 msgs/sec against stdlib’s 188,206. Against that, the project is beta. The latest stable PyPI release is 0.9.3 from 2023-09-29, and the project publishes no wheels for Python 3.13+ as of August 2026, so treat it as unsupported on current CPython unless you build from source.
• Loguru: Loguru’s README claims “10x faster than built-in logging,” and the maintainer’s own measurements contradict it. In Loguru issue #667, the maintainer measured Loguru at 43.1 µs/call against stdlib’s 34.4 µs to the same StringIO sink, stating “Loguru is not the best candidate for fast logging (yet). It’s currently slower than standard logging.” NexusLog independently measured Loguru about 1.44× slower than stdlib. A 2024 datetime-format regression made 1M records take 44 s versus stdlib’s 10 s before the maintainer fixed it in PR #1202.
• structlog: structlog publishes no numeric stdlib comparison; its performance docs offer configuration guidance only. Its measurable advantage is the filtering model: make_filtering_bound_logger() returns before constructing an event dict for filtered calls. The independent LogXide suite (macOS ARM64, Python 3.12.11/3.14.2) put structlog at 165,936–179,948 ops/sec against picologging’s 352,800–371,144 in the same runs.

Structured JSON output carries its own premium over the numbers above. A cross-library harness used pytest-benchmark with a StringIO sink; the benchmark authors did not state the CPython version. It measured emitted JSON calls at:

Two gaps qualify all of this. That table compares JSON against JSON; no public benchmark isolates the delta between plain %-style logging.Formatter output and JSON under identical conditions. And no single independent benchmark covers stdlib plus all three libraries with a stated Python version and hardware, so run your own harness before switching libraries. The stdlib still performs handler I/O synchronously and relies on costly frame introspection, so the next section examines asyncio and alternative-runtime behavior.

What’s next for low-overhead Python logging

Asyncio-compatible logging remains an open problem the core team has declined to solve in the module itself. No Python contributor has filed a PEP for native async logging, and a December 2024 discuss.python.org thread concluded “There’s not really a good reason to make logging async.” A March 2025 thread explains the safety constraint: the logging module is synchronous by design, and an async context switch during message handling is unsafe when the handler holds arbitrary threading locks.

Synchronous file and network handlers can stall an event loop. The asyncio dev docs warn that network logging can block the loop. DatagramHandler performed a DNS lookup per message that blocked for seconds on slow DNS (gh-91305). SMTPHandler and HTTPHandler run full synchronous round trips inside emit().

The Cookbook’s answer is the same QueueHandler recipe, extended explicitly to async code so “any blocking code runs only in the QueueListener thread.” The third-party aiologger offloads file I/O to a thread pool via aiofiles for teams wanting an async-native API.

Alternative runtimes add a frame-introspection wrinkle. PyPy’s performance page states that sys._getframe() incurs “a performance penalty that can be huge by disabling the JIT over the enclosing JIT scope,” and the logging module calls it in findCaller(). The Logging HOWTO names PyPy specifically when recommending logging._srcfile = None, and CPython keeps its own experimental JIT (PEP 744) disabled by default in 3.13 while reporting modest gains so far. The offload-and-strip-metadata playbook therefore holds across runtimes. Everything above condenses into an auditable configuration.

Production configuration checklist

Audit an existing service’s logging setup against these items, in order of impact:

• Offload I/O: Route all hot-path loggers through QueueHandler with a QueueListener owning the real handlers, and use an unbounded queue unless you have measured burst capacity. Default QueueHandler.prepare() still formats and copies the record on the caller thread, so override it when you need to defer that work too.
• Shut down cleanly: Call listener.stop() at application shutdown, before handlers close; atexit will not do it for you.
• Set the level floor: Run production at INFO or above, and wrap any debug call with expensive arguments in isEnabledFor(logging.DEBUG).
• Format lazily: Use %-style arguments (logger.info("user %s", uid)) everywhere; ban f-strings inside log calls in code review.
• Strip unused metadata: Set logging._srcfile = None and disable logThreads, logProcesses, logMultiprocessing, and logAsyncioTasks for fields your format string never renders.
• Leave fsync off: Accept OS write buffering unless a compliance requirement demands per-record durability, and budget the device-flush cost if it does.
• Benchmark on your version: Interpreter releases move logging numbers 10–40% with no logging code changes, so re-measure with pyperf after every Python upgrade.

Application-level efficiency is only half of the logging problem at platform scale. Per-GB observability pricing makes every emitted log line increase the bill, which pushes teams to discard telemetry they tuned their applications to produce. groundcover instead uses flat per-node pricing, so log volume does not change the per-node unit price.

groundcover’s BYOC architecture, short for Bring Your Own Cloud, runs the data plane in your own cloud account. Its storage architecture keeps logs and traces in ClickHouse alongside Kubernetes events, all inside the customer environment. That combination removes the volume penalty while keeping the telemetry data plane in your account.

If you want to validate the architecture on your own cluster, the free plan includes BYOC and requires no credit card. Deploy the Flora eBPF sensor on one cluster and evaluate full-cluster visibility within hours.