Understanding log severity levels and the standard hierarchy

A Loki 2.9 upgrade shipped two faults at once: a Helm chart regression that left log files unrotated, and a config merge that flipped log.level from info to debug. The level change alone multiplied Loki’s own output tenfold, and with rotation disabled the files filled node disks over the next 48 hours until the kubelet’s disk-pressure threshold evicted 45 pods across 3 nodes (postmortem). Neither fault would have been enough on its own, which is the point: severity levels are the control surface that decides what your pipeline stores, where it routes records, what your team pays for, and what pages an engineer at 2 AM. This guide covers, in order: what a severity level does, the standard TRACE-to-FATAL hierarchy, the threshold filtering model, syslog’s numeric codes, cross-framework mapping, level selection rules, production versus development configuration, runtime level changes, and alert routing discipline.

What is a log severity level

A log severity level is a ranked field attached to every log record that filtering and routing systems compare against thresholds. The label matters less than the comparison it enables: a logger with a threshold of INFO drops every DEBUG and TRACE record before it costs anything downstream, and an alert route matching only ERROR and above keeps WARN records out of the pager. The OpenTelemetry spec makes the comparison explicit, directing implementations to use the numeric SeverityNumber field wherever severity takes part in greater-than or less-than comparisons (OTel logs data model).

Severity therefore controls two areas: the resource footprint, including output volume and storage duration, and the destination each record reaches. Get the level wrong at the source and every downstream decision inherits the error. The next section defines each level so those source decisions rest on a shared vocabulary.

The standard log severity hierarchy

The standard log severity levels hierarchy runs from TRACE at the most verbose end to FATAL at the most severe, matching the six named ranges in the OpenTelemetry logs data model. Each level pairs with a concrete operational trigger:

1. TRACE: Fine-grained execution detail. Enable it when you need to reconstruct a specific code path step by step.
2. DEBUG: Diagnostic events for development. Enable it when investigating behavior in a dev environment or during an active incident.
3. INFO: A normal event happened: service startup, configuration load, request completion, or shutdown.
4. WARN: The system degraded but is still serving: a retry eventually succeeded or a connection pool is near its limit.
5. ERROR: An operation failed and needs attention, but the process survived.
6. FATAL: The process cannot continue. This is usually its last log line.

The three pairs below draw the boundaries that trip teams up most often.

TRACE and DEBUG

TRACE records execution flow: loop iterations, function entry and exit, branch transitions, and intermediate state. Java Util Logging makes the granularity explicit in its equivalents, reserving FINER for method entry, return, and thrown exceptions, and FINEST for tracing detail finer still (JUL Level Javadoc). DEBUG records the state a developer inspects to understand behavior: computed values and branch decisions.

The boundary is contested even among framework maintainers. SLF4J only added TRACE in version 1.4.0 on May 16, 2007 (SLF4J release codes), and its FAQ still discourages the level, conceding that the project “decided to bow to popular demand” (SLF4J FAQ). OTel expects TRACE to be off in default configurations, which is the practical rule: TRACE reconstructs flow at a verbosity too high for routine debugging.

INFO and WARN

INFO carries no problem signal at all. OTel defines it as an informational record marking that an event occurred, which makes INFO the audit trail of normal operation. WARN sits one range above, defined as “not an error but likely more important than informational” (OTel logs data model).

The operational test for WARN: the system is degraded but still serving, and the record should be visible before the degradation becomes a failure. Route WARN to next-business-day review unless a service-specific condition makes it immediately actionable.

ERROR and FATAL

ERROR records a failed operation the process survived: a rejected write, an unreachable dependency with an exhausted retry budget. The spec attaches concrete semantics to the boundary, treating any record at SeverityNumber 17 or above as an indication that something erroneous occurred. FATAL, in OTel’s wording, covers an application or system crash, an unrecoverable condition.

Not every framework carries FATAL. The SLF4J FAQ explains that the Marker interface “renders the FATAL level largely redundant,” since an error needing attention beyond the ordinary can simply carry a marker named FATAL. Having defined the six levels, the next question is how a configured level turns into a filter.

How log level filtering works

Setting a level establishes a cumulative threshold: the logger passes that level and every more severe level. Logback states the rule precisely, enabling a request of level p on a logger with effective level q whenever p is greater than or equal to q (Logback architecture manual). OpenTelemetry SDKs enforce the same model normatively: a Logger MUST drop any record whose SeverityNumber is set and falls below the configured minimum_severity (OTel logs SDK).

The threshold is cumulative, so each step down the table sheds an entire band of volume:

One threshold change moves whole bands at once, which is why the single info to debug flip in the Loki incident above multiplied Loki’s own output by 10x. Numeric direction varies by system: SLF4J, JUL, and OTel count upward toward severity, while Log4j 2 and syslog count downward. That inversion is the next thing to pin down.

Syslog severity levels (RFC 5424)

RFC 5424 defines eight severities numbered 0 through 7, and the scale is inverted relative to application frameworks: a lower number means a more severe condition (RFC 5424). The full set, from Section 6.2.1:

The numeric scale predates the RFC by decades. Eric Allman wrote both sendmail and syslog at UC Berkeley, and the pair “became part of BSD in 1981” (USENIX ;login: interview). The 4.3BSD syslog.h header from 1986 codifies the canonical constants, LOG_EMERG 0 through LOG_DEBUG 7 (4.3BSD source). RFC 3164 first documented the observed behavior as an Informational RFC in August 2001, and RFC 5424 replaced it as a Proposed Standard in March 2009 with the identical scale.

The severity travels in the PRI field as PRIVAL = (Facility × 8) + Severity (RFC 5424), but filtering semantics are implementation-defined, not part of the RFC. rsyslog’s facility.priority selectors log “all messages of the specified priority and higher” (rsyslog docs), and syslog-ng’s level() filter accepts ranges such as level(err..emerg) (syslog-ng docs). Where application logs and syslog meet in one pipeline, the two opposing scales must be reconciled explicitly, which is the job of the mapping table below.

Cross-framework level mapping

Multi-stack environments run Log4j 2, SLF4J, JUL, syslog daemons, and OpenTelemetry collectors side by side, each with its own numeric convention. The table aligns them; the OTel column follows the mappings in the OTel data model appendix, which the spec labels as examples “provided purely for demonstrative purposes”:

Four details in this table cause silent mis-routing in practice:

• Direction inversion: Log4j 2’s intLevel and syslog both count downward toward severity (FATAL is 100, Emergency is 0), while SLF4J, JUL, and OTel count upward (Log4j 2 custom levels). A numeric comparison copied between systems without checking direction inverts the filter.
• JUL levels with no clean counterpart: JUL’s CONFIG (700) exists nowhere else; the Log4j 2 JUL adapter maps it to a custom CONFIG level at numeric 450 (Log4j 2 system properties), and OTel places it at DEBUG3 (7). JUL’s two debug-adjacent levels split as well: OTel puts FINER at DEBUG (5) but FINE one slot higher at DEBUG2 (6), so collapsing both to DEBUG loses a distinction the appendix preserves.
• Syslog levels with no direct counterpart: Syslog’s Notice (5) lands at OTel INFO2 (10), Alert (1) at ERROR3 (19), Critical (2) at ERROR2 (18), and Emergency (0) at FATAL (21) (OTel appendix).
• Bridge disagreements: SLF4JBridgeHandler maps JUL FINER to DEBUG (SLF4J Javadoc), while the Log4j 2 JUL adapter maps FINER to TRACE. And because SLF4J has no FATAL, the log4j-to-slf4j bridge collapses both FATAL and ERROR into SLF4J ERROR (Log4j 2 docs).

OTel reserves four numeric slots per named range (WARN spans 13–16) to absorb exactly this granularity: mappings must assign by range meaning, and a source with a single matching level should take the smallest value in the range (OTel logs data model). With the vocabulary and mappings settled, the harder problem is judgment: which level a given event deserves.

Choosing the right log level

Developers misassign levels often enough to be a measured phenomenon. An ICSE 2012 study of 2,389 verbosity-level modifications found developers “often do not assign the right verbosity level at the first time,” with level adjustments accounting for 26% of all logging improvements (Yuan, Park & Zhou). A decision rule per level cuts that churn:

• TRACE: Only when reconstructing an execution path step by step. Expect it disabled everywhere by default.
• DEBUG: State a developer needs to diagnose behavior. If it only helps someone reading code, it is DEBUG, not INFO.
• INFO: A state change worth an audit trail: startup, shutdown, config reload, job completion.
• WARN: The system absorbed a problem and kept serving. If it repeats, someone should investigate before it escalates.
• ERROR: An operation failed and a human needs to act. If no action follows, it is not an ERROR.
• FATAL: The process is about to exit and cannot recover.

The costliest misassignment is inflating WARN and ERROR. Mike Shi’s guidance in The New Stack is to be deliberate about the ERROR label and reserve it for conditions that genuinely require action, on the grounds that overusing it numbs alerts and teaches teams to ignore what matters (The New Stack).

Emit severity as a field in structured JSON rather than a substring in a message line, because downstream systems route on the field. AWS Lambda’s Advanced Logging Controls filter by level only when logs are in JSON format (AWS Lambda docs), and OTel’s SeverityNumber is the numeric field collectors compare, as covered earlier. Level choice per statement is half the discipline; the threshold you run per environment is the other half.

Production vs development log level configuration

Run production at INFO or WARN and reserve DEBUG and TRACE for development or active incidents. INFO is the threshold vendors ship as the default: Loki’s server logging defaults to info and AWS Lambda’s application log level defaults to INFO (AWS Lambda docs). AWS Prescriptive Guidance pushes high-volume services further, recommending that both info and debug be disabled in production because of the volume they generate (AWS Prescriptive Guidance). The two positions are not in conflict once volume is treated as the variable: INFO is the right default for an ordinary service, and WARN becomes the right floor at the point where a service’s INFO output is itself the cost problem.

Two forces make verbosity expensive in production:

• Resource consumption: Uber’s Spark cluster generates up to 200 TB of logs per day at the default INFO level, 5.38 PB uncompressed over 30 days (Uber engineering). That INFO baseline shows how expensive routine logging can become; enabling DEBUG adds another band of records whose volume depends on the workload. Coleman Parkes surveyed 450 senior enterprise leaders for Dynatrace’s vendor-stated State of Log Management 2026 survey, which puts log management at 45% of observability budgets and nearly $2.5 million in average annual spend per organization.
• PII exposure: Verbose levels capture payloads, tokens, stack traces, and plaintext passwords. Twitter (Krebs on Security) and GitHub (BleepingComputer) both logged plaintext passwords to internal logs in 2018, and Facebook disclosed in 2019 that it stored passwords “in a readable format within our internal data storage systems” (Facebook Newsroom). The UK Home Office engineering guidance states it directly: “Ensure DEBUG or TRACE log level is not enabled in production” (Home Office).

When you do need DEBUG in production, scope it to the incident. Datadog documents the toggle pattern: set a 100% exclusion filter on status:DEBUG, then switch it off from the UI or the API for the duration of an investigation. Toggling verbosity mid-incident presumes you can change levels without redeploying, which is what inheritance and runtime reconfiguration provide.

Level inheritance and dynamic runtime configuration

Logger hierarchies let you set one threshold globally and override it per package. Frameworks name loggers with dot separators, so com.example.payments is a child of com.example, and a logger without an explicit level inherits the nearest configured level found by walking up the hierarchy toward the root, the effective level in Logback’s terms (Logback architecture). Root at INFO with com.example.payments at DEBUG gives one noisy module full verbosity while the rest of the service stays quiet.

Root defaults diverge across frameworks, which bites when a new logger inherits silently. Log4j 2’s LoggerConfig defaults to ERROR (Log4j 2 architecture). Logback’s root defaults to DEBUG, while Python’s root logger defaults to WARNING (Python logging). Python adds a propagation caveat: records go directly to ancestor handlers, and ancestor logger levels and filters do not run again (Python logging).

For incident investigation, four mechanisms change levels without a restart:

• Log4j 2: Set monitorInterval (seconds) on the configuration element; Log4j polls the file and “automatically reconfigures the logger context” on change, with no log events lost during reload (Log4j 2 configuration). Configurator.setLevel() does the same programmatically (Log4j 2 FAQ).
• Logback: scan="true" watches the configuration file, scanning once per minute by default (Logback configuration).
• Spring Boot: The Actuator loggers endpoint accepts POST /actuator/loggers/{logger.name} with body { "configuredLevel": "DEBUG" } (Spring Boot Actuator). The POST only changes the instance that received it, so each replica must be addressed individually (seriesci).
• Kubernetes: A config file mounted from a ConfigMap volume propagates without a pod restart, delayed by up to the kubelet sync period plus cache TTL, about 2 minutes at defaults, provided the application polls or watches the file (Kubernetes docs). ConfigMaps consumed as environment variables, and subPath mounts, do not update and require a restart (Kubernetes ConfigMap docs).

Runtime control is what makes the final discipline practical: production stays quiet by default and gets loud only on demand, so severity remains a trustworthy alert signal.

Preventing alert fatigue with level discipline

Alert routing only works if severity means something. A Huawei Cloud study of over 4 million production alerts across two years found 88.9% of on-call engineers agreed misleading severity had an impact, and 17 of 18 interviewed engineers said alert storms greatly fatigued them (DSN 2022 study). Google’s SRE book describes the end state, where pages arriving too often lead staff to second-guess, skim, or ignore them, and a real page gets lost in the noise (Google SRE book).

Route by the hierarchy you enforced at the source. Page on FATAL and actionable ERROR only. Ticket WARN for next-business-day review, and keep INFO in dashboards and searches; Google SRE recommends tuning noisy alerts toward “a 1:1 alert/incident ratio” (Being On-Call). Hudson River Trading took three years to work its way down from as many as 2,400 high-urgency pages a month in mid-2015 to roughly 1,000 by mid-2018, standardizing its Nagios deployment templates and adding Python scripts that filtered unnecessary alerts before they reached PagerDuty (TechTarget).

Per-GB pricing often pushes engineering teams to set retention by severity: models such as Datadog’s $0.10/GB ingestion plus per-million-event indexing charge for every DEBUG line you keep. groundcover runs the data plane in your own cloud account under BYOC (Bring Your Own Cloud). The Flora eBPF sensor captures logs at the Linux kernel with no per-service agents, while flat per-node pricing means log volume does not change the bill, so severity stays a routing signal rather than a cost lever. If you want to validate the architecture on your own cluster, the free plan includes BYOC and requires no credit card; deploy Flora on one cluster and evaluate full-cluster visibility within hours.