Designing hot and cold log storage architecture for cost and performance

Log volumes are outgrowing storage budgets. Dynatrace’s vendor-stated State of Log Management 2026 survey of 450 senior IT leaders reported a 93% average increase in log and telemetry volume over the prior year. Teams that keep everything on SSD-backed hot storage often pay block-storage prices for logs that receive few queries after the initial incident window, and teams that cut retention lose the data they need for incidents and audits; a hot cold log storage architecture removes that trade-off by matching each log’s age and access pattern to storage priced for it. First, define the tiers and architecture. Second, map lifecycle automation and retrieval costs. Third, compare platform implementations and compliance constraints before looking ahead.

What hot, warm, and cold log storage actually mean

First, the tiers describe access patterns and then the hardware that supports them. Elasticsearch’s index lifecycle management (ILM) model gives the clearest working definitions: the hot tier holds data that applications actively write and engineers query, the warm tier holds data that applications no longer write but engineers still query regularly, and the cold tier holds data that engineers query infrequently and where engineers accept slower responses. Elastic adds a frozen phase for data queried rarely enough that very slow queries are tolerable, plus a delete phase, for five ILM phases in total.

For logs specifically, hot means the live tail: active incident investigation plus dashboard and alert evaluation over the last few days. Warm covers recent history that on-call engineers still search, such as last month’s deploys. Cold and archive retain data for compliance work and occasional retrospective queries.

The tiers pair access frequency with an expected latency budget:

How hot and cold log storage architecture works

Logs enter the pipeline once and descend through the tiers as they age.

The storage tiers: hot, warm, cold, archive

In a conventional multi-tier index architecture, each log line follows the same lifecycle: it lands in the hot tier at ingest, where the search platform indexes it on fast local storage and serves queries at millisecond latency. Lifecycle triggers based on age or thresholds for size and document count then roll the underlying index or segment to warm, later to cold, and finally to archive or deletion. In Elasticsearch, the rollover action fires on conditions such as max_age or max_primary_shard_size; subsequent phase transitions use min_age measured from rollover time, not index creation.

Each tier occupies a distinct point on the latency and cost spectrum for its use case:

Retrieval latency and query speed per tier

Order-of-magnitude expectations matter more than exact numbers, because the gap between tiers is 10× to 1,000×, not 20%. Elastic’s searchable snapshots benchmark on a 4 TB dataset measured hot/warm queries at 92 ms on first run and 29 ms on repeat, cold at 95 ms and 38 ms, and frozen at 6,257 ms on first run dropping to 76 ms once cached. First-run frozen queries pay nearly two orders of magnitude in latency because the query engine must fetch the data from object storage before searching it.

Larger datasets produce a wider latency gap. Against a 1 PB frozen dataset, a simple term query’s first run took 554 seconds. Archive classes sit another order of magnitude out: retrieving from deep archive storage is measured in hours, which makes Deep Archive unsuitable as the primary incident-response tier.

Storage media behind each tier

The latency ladder maps directly to physical media, and cloud providers price storage partly according to media performance and retrieval latency. Enterprise NVMe drives such as the Micron 9400 deliver 4K random reads at a 69 µs median, which is what makes millisecond hot-tier queries possible. Enterprise 7200 RPM HDDs carry 4.16 ms of average rotational latency and sustain only 40–170 random read IOPS at low queue depths, which suits read-mostly warm data at a fraction of SSD cost.

Cold and archive tiers leave block storage entirely. S3 Standard GETs show a p90 time-to-first-byte around 20 ms regardless of object size, acceptable for infrequent queries but not for interactive search across thousands of chunks. At the far end, LTO-9 tape needs roughly 17 seconds to load an initialized cartridge plus about 45 seconds average to position to a record. You pay for latency, so the design goal is to hold each log on the slowest medium its access pattern tolerates.

Object storage as the cold-tier backbone

S3-compatible object storage is the default cold tier:

• Operational properties: AWS S3 states 99.999999999% (11 nines) durability, while Azure storage redundancy and GCS durability documentation state the same durability level. AWS spreads S3 Standard across at least three Availability Zones, and coding parameters are undisclosed for AWS and GCS. A uniform HTTP interface makes objects addressable, so query engines can search cold data in place instead of restoring it first.
• Cost: object storage prices sit an order of magnitude below SSD block storage, and archive classes another order below that.

Elastic, Splunk, and Grafana engineering teams adopted object storage for older data. Elastic’s cold and frozen phases mount searchable snapshots from object storage, Splunk’s SmartStore uses S3-compatible remote storage as its system of record, and Grafana Loki stores both index and chunks in a single object storage backend across the three major clouds.

Key features and capabilities

Having established the tiers and their media, the next step is to map storage classes, retention windows, lifecycle automation, and the restore path that turn the model into a running system.

Cloud storage services mapped to tiers

Each major cloud publishes storage classes that slot cleanly into the tier model. The cloud providers publish the prices below for AWS us-east-1, Azure East US (LRS, first 50 TB band), and GCS us-central1 regional; the table derives the GCS Nearline, Coldline, and Archive monthly equivalents from per-GiB-hour rates:

Minimum storage durations mean an object deleted or re-tiered early still incurs the full minimum charge, so windows shorter than the class minimum belong in a warmer class. Cloud providers also change class names and prices (AWS, for example, removed the 30-day transition minimum for Standard-IA in July 2026, though the 30-day billing minimum remains), so verify current class names and pricing against provider documentation before committing lifecycle rules.

Designing log retention policy per tier

A defensible starting policy is 7 days hot, 30 days warm, and 365 days cold. A seven-day hot window is a starting point, while AWS notes that most OpenSearch Service customers run hot retention between 7 and 14 days. The 30-day warm window absorbs investigations that trail an incident, and the 365-day cold window satisfies annual audit cycles.

Kaltura’s and Adaptavist’s production policies follow this pattern. Kaltura, ingesting 8 TB/day across 6 AWS Regions, runs rollover at 30 GB primary shard size, hot to warm at 2 days, warm to cold at 14 days, and delete at 60 days; the redesign extended critical-log real-time retention from 5 to 30 days while cutting observability costs 60%. Adaptavist, retaining ~120 GB/day of audit data, keeps 31 days on NVMe-backed hot nodes and 315 days in UltraWarm for a 346-day total. Set the hot window from your incident query patterns and the cold window from your longest audit obligation, then let automation enforce the boundaries.

Automating tier transitions with lifecycle policies

Lifecycle policies make tiering hands-off. An S3 lifecycle configuration is XML (JSON via the CLI), supports up to 1,000 rules per bucket, and only moves objects toward colder classes. A rule that archives logs after a year and deletes them after ten looks like this:

<LifecycleConfiguration>
  <Rule>
    <ID>Transition and Expiration Rule</ID>
    <Filter>
      <Prefix>tax/</Prefix>
    </Filter>
    <Status>Enabled</Status>
    <Transition>
      <Days>365</Days>
      <StorageClass>GLACIER</StorageClass>
    </Transition>
    <Expiration>
      <Days>3650</Days>
    </Expiration>
  </Rule>
</LifecycleConfiguration>

One constraint bites log workloads specifically: since September 2024, objects smaller than 128 KB do not transition by default, so small log objects need an explicit object-size filter (ObjectSizeGreaterThan or ObjectSizeLessThan) or, better, compaction into larger objects before archiving. Inside a search platform, the equivalent automation is Elasticsearch ILM, with phase transitions based on age or index thresholds for size and document count, or OpenSearch ISM, whose managed-service variant adds warm_migration, cold_migration, and cold_delete operations for UltraWarm and cold storage.

Cost trade-offs across tiers

The per-GB spread between hottest and coldest object classes runs roughly 9–23× on every cloud: about 23× from S3 Standard to Glacier Deep Archive, about 9× from Azure Hot to Archive, and about 16.7× from GCS Standard to Archive. The practical multiple is larger, because hot log tiers usually sit on replicated SSD block storage at $0.08–0.10/GB-month, pushing the effective hot-to-cold ratio to 50–100× after counting replicas.

Named outcomes show what the ratios translate to in practice:

Restoring and querying archived logs

The archive tier’s cost comes with a retrieval contract you must design around. S3 Glacier Flexible Retrieval offers three tiers: expedited at 1–5 minutes for objects under 250 MB ($0.03/GB), standard at 3–5 hours ($0.01/GB), and bulk at 5–12 hours (free). Glacier Deep Archive restores take 9–12 hours standard or up to 48 hours bulk. Retrieval fees mean a poorly scoped restore of a large archive can cost more than a month of its storage.

Platform-level thaw workflows add steps. Splunk’s classic path copies an archived bucket into thaweddb/, runs splunk rebuild, and restarts the indexer; Splunk excludes thawed data from further aging. Elastic’s searchable snapshots avoid a thaw step entirely, at the first-query latency penalty shown earlier.

During an incident, cold-tier query limits are the constraint that matters. A community Loki operator running v3.x against S3 without a warm cache reported 1-day queries taking about 70 seconds and 10-day queries timing out at the gateway. Size your hot and warm windows so that incident investigation never depends on a cold read, and treat archive restores as planned operations with hours of lead time. Those retrieval contracts are where platform implementations diverge most.

How hot and cold log storage fits in

With lifecycle mechanics and retrieval costs established, the third step is to compare platform implementations and the compliance constraints that shape them. A single-tier setup forces one price on every log regardless of value, keeping 100% of retained data on replicated SSD block storage at $0.08–0.10/GB-month; tiering is how mature platforms escape that.

Platform-specific architectures

Elasticsearch is the most complete end-to-end example. ILM moves indices through hot, warm, cold, frozen, and delete phases across dedicated node roles (data_hot through data_frozen); rollover happens only in the hot phase, cold indices mount searchable snapshots fully cached on local disk, and frozen indices mount them partially, through a fixed-size shared cache. The result is one query interface over data spanning SSD to object storage.

Splunk expresses the same lifecycle as bucket stages: hot, warm, cold, frozen, and thawed, with rolls triggered by size thresholds and maxWarmDBCount (default 300), followed by frozenTimePeriodInSecs (default roughly 6 years). SmartStore collapses this to two states, hot buckets on local disk and warm buckets on remote object storage fronted by an LRU cache, and buckets roll to frozen directly from warm. One operational warning from Splunk’s own community: SmartStore migration is one-way, so cache sizing is an architectural decision, not a tuning knob.

Grafana Loki skips local tiers altogether: index and chunks both live in object storage, the TSDB index shipper handles metadata, and a compactor enforces retention rules instead of ILM phases. Performance then depends on caching; one production deployment with split_queries_by_interval: 15m and a 97.8% Memcached hit rate held p99 query latency around 2.75 seconds while scanning terabytes. Tier boundaries in Loki are cache boundaries, which shifts the retention question onto the rules a compactor enforces.

groundcover expresses tiering as a retention policy rather than index phases or bucket stages. Storage Management, configured per backend under Settings, gives each data type — logs, traces, and Kubernetes events — a default retention window plus an optional move to cold storage after a set time in hot, with the cold destination fixed to the cold volume in the current release. On top of that age axis it adds a content axis: indexes, each a gcQL query filter carrying its own retention period, evaluated top to bottom so the first match wins. Order matters, and the failure mode is quiet — a broad cluster:prod rule placed above a narrow cluster:prod level:error rule swallows the errors you meant to keep longer, so narrow filters go first. Cold storage applies at the default policy level rather than per index, and metrics retention is set separately. Where Elastic tiers by index age and Loki tiers by cache boundary, groundcover tiers by age and by query filter, and the rules for both usually come from compliance.

Compliance, immutability, and audit retention

Compliance frameworks constrain retention length more than tier placement, and reading them precisely saves money.

Immutability is where cold-tier design meets security. S3 Object Lock in Compliance mode makes a locked object version undeletable by any user, including root, until retention expires, with retention up to 100 years; Governance mode allows bypass only with the s3:BypassGovernanceRetention permission. Azure’s locked time-based immutability policies and GCS Bucket Lock, which becomes irreversible once locked, provide the equivalent WORM guarantees; Cohasset Associates assessed each service against SEC Rule 17a-4(f) and related rules. The ransomware implication is direct: an attacker who compromises credentials can drop hot indices, but a compliance-mode-locked cold copy cannot be deleted, which makes the immutable cold tier the recovery floor for audit data.

The groundcover approach

Storage physics do not change with the pricing model. Replicated SSD still runs 50–100× the cost of object storage, and a year of audit logs still belongs on the cheap medium. What changes is who sets the retention window.

Per-GB pricing turns retention into rationing. When every retained gigabyte carries an ingestion and storage meter, teams shorten windows, sample streams, and drop non-production logs to control the bill. Incident and audit needs should set retention, not the invoice.

groundcover removes that volume penalty structurally. The deployment is BYOC (Bring Your Own Cloud, where your data plane runs in your own cloud account): logs and traces, plus Kubernetes events, live in ClickHouse inside your VPC, with metrics in VictoriaMetrics beside them. Flat per-node pricing has no volume-based ingestion or custom-metric charges and no per-user fees, so data volume and retention length do not change the unit price.

Tiering then becomes an engineering decision rather than a budget one. Storage Management sets a default retention window per data type and an optional move to cold storage after a chosen time in hot, and indexes let a query filter carry its own retention — production errors at 90 days ordered above production logs at 30, say — so you shorten the window for the data you do not need instead of for everything. The hot tier, the cold volume, and the bucket all sit in your account, which is what complete telemetry access means in practice.

Three current limits are worth knowing before you plan around it: self-serve Storage Management is in gradual rollout, per-index cold storage is not supported in this release, and metrics retention is set with the groundcover team rather than from the page.

If you want to validate the architecture on your own cluster, the free plan includes BYOC and requires no credit card. Deploy Flora on one cluster and evaluate full-cluster visibility within hours.

What’s next for hot and cold log storage

Finally, future tiering work focuses on removing restore steps and improving cold-media efficiency. Elastic’s searchable snapshots already query frozen data without a restore step, and the trajectory is toward eliminating thaw workflows entirely, leaving latency as the only remaining difference between tiers.

Kafka and Pulsar follow the active-tail replication pattern, while cold durability moves to erasure-coded object stores. Loki relies on its object storage provider’s internal redundancy for cold chunk durability. MinIO states its 12:4 configuration delivers eleven nines of durability at 33% storage overhead versus 200% for three-way replication; HDFS erasure coding’s lack of append support keeps EC out of the active write path.

FAQ

What is the difference between hot, warm, and cold log storage? Hot storage holds actively written and queried logs on SSD for millisecond search; warm holds read-only recent history on lower-cost media; cold holds infrequently queried logs on object storage where first-query latency of seconds to minutes is acceptable. Archive sits below cold and usually requires a restore before querying.

What query latency should I expect from each tier? Milliseconds for hot and warm, seconds to minutes for the first query against cold or frozen data, and hours for archive restores. Elastic’s benchmark makes the gap concrete: 92 ms on hot versus 6,257 ms on an uncached frozen first run of the same 4 TB dataset.

What retention windows should I set per tier? Start from 7 days hot, 30 days warm, and 365 days cold, then adjust: hot from your incident query patterns (most OpenSearch users run 7–14 days), cold from your longest audit obligation. Kaltura’s production policy (2 days hot, 14 warm, 60 delete) shows aggressive windows work when query patterns support them.

Can I set different retention for different logs? Only if the platform tiers by content, not just by age. Elasticsearch ILM and Loki’s compactor both key off index or chunk age, so a longer window for error logs means a longer window for everything in that index. groundcover pairs a default retention window and an optional hot-to-cold move with indexes: each index is a query filter with its own retention, evaluated top to bottom, first match wins. Put narrow filters above broad ones, or the broad rule claims the rows first.

How do I automate tier transitions? Use the native lifecycle mechanism at whichever layer owns the data: S3 lifecycle rules for raw objects, Elasticsearch ILM or OpenSearch ISM for indices, and Loki’s compactor for chunk retention. All trigger transitions on age or size without manual intervention.

Which cloud storage classes map to which tier? S3 Standard and Azure Hot back the hot/warm object layer; GCS Standard does too. Glacier Instant Retrieval and Azure Cool serve warm, as does GCS Nearline, while Glacier Flexible Retrieval and Azure Cold serve cold alongside GCS Coldline. Glacier Deep Archive and Azure Archive serve deep archive, as does GCS Archive. Confirm current names and minimum-duration terms on provider pricing pages before building rules.

How much does moving logs from hot to cold save? Object storage class spreads run roughly 10–23× per cloud, and the effective saving reaches 50–100× when the hot tier is replicated SSD block storage. Documented outcomes range from Trellix’s 35% to the AWS reference design’s ~48× annual cost reduction.

How does restoring archived logs work? Glacier-class restores take 1–5 minutes for expedited retrieval of objects under 250 MB, or up to 48 hours for Deep Archive bulk, and carry per-GB retrieval fees. Splunk thaws archives via thaweddb/ and splunk rebuild. Elastic queries frozen snapshots in place with no restore step. Plan archive access as a scheduled operation, never an incident dependency.

What do compliance frameworks require for log retention? PCI DSS v4.0.1 requires 12 months with three months “immediately available,” which PCI SSC says can be archived or restorable rather than hot. HIPAA’s 6-year rule covers Security Rule documentation rather than every operational log, and SOC 2 sets no numeric window; pair the required window with WORM controls such as S3 Object Lock Compliance mode when immutability is in scope.