ClickHouse versus S3 for log storage

Amazon S3 and ClickHouse solve different halves of the log storage problem. S3 holds bytes at $0.023 per GB-month and charges for requests, but adds no continuously running query compute. ClickHouse is a columnar analytics database that can mount S3 as the backing disk for its MergeTree tables, adding a sparse primary index and aggressive compression. A warm cache can then produce millisecond queries on top of the same object store.

The decision is not binary. The practical comparison in S3 vs ClickHouse log storage is raw S3 queried ad hoc through Athena or DuckDB versus ClickHouse running with S3 as its storage backend. This article works through the trade-offs in order: a summary table and MergeTree-on-S3 mechanics, then storage costs and query benchmarks. It finishes with compression, TTL tiering, operational failure modes, object storage providers, and a decision matrix.

S3 vs ClickHouse for log storage at a glance

First, the table below compresses the architecture decision into the dimensions platform teams actually budget for:

Each row above hides a mechanism. Platform teams should base the outcome on those mechanisms, starting with how ClickHouse lays data onto S3.

How ClickHouse stores logs on S3

ClickHouse-on-S3 separates storage from compute: MergeTree data parts live as objects in the bucket, while ClickHouse stores part metadata on each server’s local disk. That split means storage scales at S3 prices while compute scales independently, but every read that misses the local cache pays S3’s latency profile: ClickHouse’s own measurements put S3 at 500 ms P99.99 latency and ~5K IOPS versus 1 ms and 100K IOPS for local SSD.

MergeTree parts, background merges, and write amplification

Every insert creates a sorted, immutable data part, and background threads continuously merge small parts into larger ones toward a compressed size target of 150 GiB (max_bytes_to_merge_at_max_space_in_pool, default 161,061,273,600 bytes). Each merge decompresses source columns, merge-sorts them, rebuilds the sparse index, and writes a new part; ClickHouse deletes source parts after a configurable delay, 8 minutes by default (old_parts_lifetime, 480 seconds).

On S3 this cycle gets expensive. As one ClickHouse GitHub issue puts it: “When ClickHouse is merging data from S3, it has to pull all of the data across the network from S3, write that data to a tmp file, and then write it all back across the network again.” (Issue #41316) The pathological case: a large part already on S3 receives a small insert into the same partition, and ClickHouse downloads the entire large part, merges, and re-uploads it (Issue #38315).

These tuning habits contain the damage:

• Batch inserts: Healthy batching averages 10,000+ rows per insert. ClickHouse delays inserts at parts_to_delay_insert (default 1,000 active parts per partition) and throws “Too many parts” at parts_to_throw_insert (default 3,000). Both defaults changed in 23.6 — they were 150 and 300 before that, and older blog posts still quote the old numbers. A third limit, max_parts_in_total (default 100,000), applies across every partition of a table, and fine-grained partitioning usually trips that one first.
• Avoid prefer_not_to_merge as a first resort: The volume-level setting skips S3 merges entirely but can trigger the same “Too many parts” failure, and guidance in Issue #41316 conflicts on when it is safe.
• Merge locally, store remotely: Keep the actively merging partition on block storage and move it to S3 only after merges settle. The tiering section below shows the TTL pattern.

Merges explain the write cost. The metadata layer explains the failure modes you will debug.

Metadata and S3 write-once semantics

S3 objects are immutable, so ClickHouse keeps local metadata files on each server that map part names to S3 object keys; renames and hard links happen in that local layer, never in the bucket. The consequence, documented in ClickHouse’s SharedMergeTree engineering post, is that metadata remains coupled to compute: lose the local metadata and the S3 objects become unreadable orphans.

This layering shows up in production errors. Issue #70937 reports part fetches failing with Remote metadata 'count.txt' is not exists. (CORRUPTED_DATA) on 24.9.2.42, an error that only makes sense once you know the local metadata and the remote objects can drift apart. When you see NoSuchKey or missing-metadata errors on an S3-backed table, check the local metadata directory before blaming the bucket.

Storage cost breakdown

ClickHouse-on-S3 shifts the cost model from storage capacity toward compute and S3 request volume. The following sections separate capacity, request, and merge-amplification costs.

Per-GB storage pricing

S3 Standard at $0.023/GB-month is 3.5× less than gp3 EBS at $0.08/GB-month on raw storage, before EBS IOPS and throughput provisioning. A fully provisioned 1 TiB gp3 volume at 10,000 IOPS and 1,000 MiB/s runs $151.92/month.

Full TCO follows the same direction. A documented 3-node comparison prices r5b.xlarge nodes with 1 TiB gp3 each at $1,112.38/month against r7gd.xlarge nodes with ephemeral NVMe cache and an S3 backend at $666.78/month, a 40% reduction. At larger scale, ClickHouse’s internal LogHouse platform reports an all-in cost of $23.83 per TiB of uncompressed logs ingested per month, vendor-stated, on 19 PiB of uncompressed data.

S3 API request charges (GET, PUT, LIST)

AWS charges $0.005 per 1,000 PUT/COPY/POST/LIST requests and $0.0004 per 1,000 GETs. Those rates look negligible until merges multiply them: Luciq attributed ~96% of their S3 bill, with storage bytes at ~4% to PUT requests in one production deployment. Luciq also measured a single 171 MiB table emitting tens of thousands of PUTs per hour; its long tail of small tables totaling a few hundred GB generated ~41% of their entire PUT bill at hundreds of millions of PUTs per day.

Request rate is a second constraint alongside cost. Tinybird documents a cluster of 10 nodes at 500 PUT/s each pushing 5,000 PUT/s against S3’s 3,500 PUT/s per-prefix limit, producing 503 SlowDown errors; the client-side s3_max_put_rps and s3_max_get_rps settings cap per-node request rates. Part count and merge frequency are the levers, which leads directly to the write amplification question.

Write amplification overhead

No published source gives a universal bytes-written-to-bytes-ingested multiplier for MergeTree-on-S3; the closest figure is a simulated WA of 3.31 from a ClickHouse merge-selector PR. In practice, object-count amplification drives the dominant cost, because every file in a Wide part becomes its own PUT.

The worst case documented so far comes from JSON columns rather than ordinary schemas. With the default max_dynamic_paths = 1024, each dynamic path inside a JSON, Object, or Dynamic column materializes its own .bin and mark substreams. The reporter on Issue #100960 measured roughly 25,000 PUT requests for 2 GB of data, with more than 30,000 uploads under 1 MiB each. A proposed fix benchmarked a 15-column schema plus one JSON column with 50 dynamic paths and found 664 files per Wide part against 10 in Compact — a 66× difference on local disk, with inserts, merges, and every tested query also faster in Compact. The PR author closed it in favor of the simpler existing knob.

That knob is min_bytes_for_wide_part (default 10 MiB uncompressed). Raising it forces more parts into Compact format: Luciq cut the Wide share of cold merges from ~34% to ~1.5%, dropped daily PUTs ~59% on the first day, and projected ~92% fleet-wide. Altinity measured a related pathology on a 109-column table, 227 files per part, vendor-stated, and identifies file count as the main cause of slow S3 inserts and deletes. Wide columnar layout is the right default on local disk, where per-column read locality is free; on S3 each file carries a request charge, and the trade flips.

Platform teams must next compare query latency.

Query performance: ClickHouse-on-S3 vs raw S3

The warm/cold cache gap matters more than the storage medium, while indexing strategy can outweigh both. The benchmarks below separate those effects before comparing ClickHouse with Athena, DuckDB, and Parquet.

ClickHouse columnar queries vs Athena, DuckDB, and Parquet

In ClickHouse’s distributed cache benchmark (Amazon reviews dataset, ~151M rows, vendor-stated), a cold full-table scan took 29.7 s on self-managed gp3 EBS and 18.7 s on S3 with a local filesystem cache; warm runs converged to 5.4 s and 3.8 s. Scattered latency-sensitive reads inverted: 0.179 s cold on EBS versus 0.461 s cold on S3, 2.6× slower, with both warming to ~60 ms.

Indexing and file layout outweigh both. Altinity’s MergeTree vs Parquet comparison (ClickHouse 23.4, vendor-stated) ran two datasets. On the 200M-row OnTime dataset, three selective queries took 80 ms, 23 ms, and 80 ms against MergeTree on EBS versus 800 ms, 600 ms, and 2,000 ms against Parquet on S3 — a 10× to 25× gap. On the 600M-row Star Schema Benchmark, with more columns and more complex filters, the full query suite totaled 0.824 s on MergeTree against 117 s on Parquet. Altinity attributes much of the difference to per-file S3 request overhead rather than scan speed alone: repartitioning the same Parquet data from 35 files into 408 made every query at least 5× slower, while the equivalent MergeTree change cost only 1.8×. Their own summary is that Parquet on S3 runs 10 to 100 times slower than MergeTree, and unpredictably so. For log analytics, where queries filter on time ranges and a handful of fields, the sparse index plus a small number of large parts is the argument for ClickHouse.

Athena charges $5.00 per TB scanned with a 10 MB per-query minimum, so its economics depend entirely on partitioning and format. One practitioner cut a year-of-logs query from 36 TB scanned ($180) to 10–50 GB ($0.25) with day-level partitions and Parquet. Latency stays in the seconds: an independent benchmark on ~20 GB of ZSTD Parquet measured Athena’s warm median at 4,211 ms against 505 ms for DuckDB on local SSD and 881 ms for DuckDB on Cloudflare R2, with 20–30 s cold starts from remote Parquet metadata fetches.

Against ClickHouse the gap widens for repeated interactive queries. Cloud CIRCUS migrated CloudFront log analytics from Athena to a single 4 vCPU ClickHouse instance and saw a daily access-count query drop from ~16 seconds to 0.043 seconds, at roughly $300/month versus $100/month for Athena, vendor-stated. The source does not state whether the MergeTree data used local storage or S3. On raw Parquet without MergeTree, ClickHouse loses its index advantage: in ClickBench data-lake results, vendor-maintained, DuckDB beats ClickHouse on many individual S3 Parquet queries, though ClickHouse posts the better combined relative score (×11.73 vs ×20.29 on c6a.4xlarge, lower being better). No controlled three-way benchmark of ClickHouse-on-S3, Athena, and DuckDB exists in the retrieved literature; run your own against your schemas.

Platform teams should use Athena or DuckDB for occasional scans over well-partitioned Parquet. ClickHouse suits frequent selective queries where the sparse index and a warm cache pay off. The next decision is cache sizing, so configure it deliberately.

Local disk cache configuration

ClickHouse’s filesystem cache wraps the S3 disk with a type=cache disk; path and max_size requirements have no defaults. A minimal storage_configuration:

<clickhouse>
  <storage_configuration>
    <disks>
      <s3_disk>
        <type>s3</type>
        <endpoint>https://bucket.s3.amazonaws.com/logs/</endpoint>
      </s3_disk>
      <s3_cache>
        <type>cache</type>
        <disk>s3_disk</disk>
        <path>/var/lib/clickhouse/disks/s3_cache/</path>
        <max_size>200Gi</max_size>
      </s3_cache>
    </disks>
  </storage_configuration>
</clickhouse>

Eviction is LRU at max_size; cache_on_write_operations (default false) enables write-through caching of inserts and merges, and system.filesystem_cache exposes hit metrics. Size the cache around the repeatedly queried working set: Pulse’s guidance is that a 10 TB table where queries hit the last 100 GB needs roughly a 200 GB cache for near-local performance. Their 50 GiB deployment measured 87.42M rows/s on local EBS, 37.59M rows/s on cold S3, and 115.27M rows/s warm. Tinybird runs a 500 GiB cache targeting an 80%+ hit rate, holding cached queries at 50–200 ms against 1–2 s cold.

Compression and storage efficiency

Columnar compression is the quiet multiplier in this comparison: it shrinks both the storage bill and the S3 GET volume per query. Self-managed ClickHouse defaults to LZ4; ClickHouse Cloud defaults to ZSTD (codec docs). Altinity’s Star Schema benchmark, vendor-stated and dating to 2017, measured ZSTD ~35% smaller than LZ4 but LZ4 1.75× faster on hot queries, because decompression becomes the CPU bottleneck once I/O stops being the constraint. Both codecs have improved since, so treat the ratio as directional rather than current. For S3-backed cold data, ZSTD(1) usually wins; Cloudflare’s DNS log tests show level 9 costing 7.3× the compression speed for only ~10% more density, so stay at level 1.

Measured ratios on real log data, all vendor-stated by ClickHouse:

• Agent-shaped observability logs at ZSTD(1): 33× for Fluent Bit schemas, 21× for Vector, 14.1× for OpenTelemetry.
• Nginx access logs with tuned types and sort keys: 178× versus 35× for the unstructured schema, same 66.7M-row dataset.
• Parquet on S3: Parquet+ZSTD(1) is 2.2× smaller than JSONEachRow+ZSTD(1), so raw JSON is the worst archive format on every axis.

The practical read: measured ClickHouse log schemas compress raw data by 14.1× to 178×, while Parquet+ZSTD(1) is 2.2× smaller than JSONEachRow+ZSTD(1). Those ratios feed directly into how much data you can afford to keep hot. Your retention policy determines where each age of data lives.

Hot/cold tiering with TTL

The TTL MOVE pattern keeps recent partitions on local SSD, where merges are fast and free of API charges, and ages settled data to S3. A ClickHouse maintainer states the rationale directly: keep the last partition on hot EBS and move data to cold S3 via TTL after it has merged, because “insertion already goes into a new partition” and you “avoid download/upload of data from/to S3 because of merges” (Discussion #77681). The table definition:

CREATE TABLE logs (...)
ENGINE = MergeTree
PARTITION BY toDate(timestamp)
ORDER BY (service, timestamp)
TTL timestamp + INTERVAL 7 DAY TO VOLUME 'cold'
SETTINGS storage_policy = 'tiered';

Choose the move threshold from your query distribution: most log queries hit recent data, and Altinity’s best-practices guidance, vendor-stated, is to keep actively merging and frequently queried data on block storage precisely because it cuts S3 I/O and API costs. Set perform_ttl_move_on_insert = false in the same policy; otherwise ClickHouse applies the TTL move on insert and data merges in S3 instead of locally.

The savings are real but sensitive to which SSD you price against. For a 10 TB dataset split 2 TB hot and 8 TB cold, oneuptime calculates $392/month versus $1,024 all-SSD, a 62% reduction — figures that imply a $0.10/GB SSD rate, closer to gp2 or io1 than to gp3. Run the same split against gp3 at $0.08/GB and the all-SSD baseline drops to roughly $819, making the saving about 52%. Either way the direction holds; the exact percentage depends on your volume type.

For multi-year retention, use S3 lifecycle policies as a separate archival path beyond active ClickHouse storage, with a tested restore workflow before querying archived data. Glacier Deep Archive stores at $0.00099/GB-month with a 180-day minimum and standard restores within 12 hours. Two traps: AWS bills roughly 40 KB of metadata per archived object (32 KB at the Deep Archive rate plus 8 KB at the S3 Standard rate), so millions of small log files can cost more in overhead than data, and lifecycle transition requests bill per object. AWS documents 10 million daily log objects costing $200/day in transition fees alone, cut to one request per day by aggregating with s3tar. Tiering handles data placement; the remaining overhead is everything else you now operate.

Operational overhead

Running ClickHouse-on-S3 means owning merge behavior and cache sizing. You also own credentials and backups, while replication carries the sharpest edge.

Zero-copy replication mechanics and caveats

Do not enable zero-copy replication in production unless you are prepared to maintain a fork. The feature (allow_remote_fs_zero_copy_replication, default false since 22.8) lets replicas share S3 objects instead of copying data, coordinating reference counts through ZooKeeper/Keeper. The official docs label it “not ready for production”, and maintainer Alexey Milovidov closed a fix PR in October 2025 with “We don’t support zero-copy replication. Closing.” (PR #77907)

The architecture creates four operational limitations (ClickHouse engineering blog):

• Each replica changes metadata independently, so ClickHouse cannot make atomic commits across all replicas.
• Merges and mutations contend on exclusive Keeper locks and block each other.
• Durability requires object storage plus both Keeper and local disk.
• A high server count creates contention on the replication log.

At least six merged PRs fix data-loss bugs in the reference-counting path. The worst documented outcome is PostHog’s: on 19 February 2026, a routine mutation to materialize an index triggered a zero-copy bug that convinced one replica every data part was unreferenced. Over the next eight and a half hours the replicas deleted the entire database from S3, losing all US logs data older than three days. PostHog had no S3 Versioning on the bucket, and their first two remediations were to disable zero-copy everywhere and turn versioning on.

Managed vendors run this feature only with engineering behind it. Tinybird operates zero-copy replication on a ClickHouse fork with its own maintenance layer, and still spent a cleanup project tracking down petabytes of orphaned objects — nearly losing live data to their own deletion tooling in the process. ClickHouse Cloud sidesteps the problem entirely with SharedMergeTree and leaderless replication, but that engine is Cloud-only. Self-hosted teams should replicate with full copies or lean on hot/cold tiering, and enable S3 Versioning regardless, so a runaway delete stays recoverable.

IAM and credential configuration

Attach S3 permissions through IAM instance profiles rather than static access keys in the ClickHouse XML config; hardcoded credentials end up in config management history. Scope the role to the specific bucket and prefix the disk uses. Pair the role with bucket-level S3 Versioning: as the PostHog incident shows, the ClickHouse process holds delete permissions by design, so versioning is the layer that makes a bad delete reversible.

Backup and snapshot strategies

Two documented constraints shape the backup strategy. SYSTEM RESTORE REPLICA does not work with S3 zero-copy replication, one more reason the feature stays off. ClickHouse’s DR guidance also requires data and metadata to exist in multiple regions for disaster recovery, so replicate backups cross-region rather than trusting a single bucket. Incremental part-level backups to a separate S3 bucket, plus versioning on the primary, cover the realistic failure set for log archives.

Ingestion pipeline options

The shipper choice depends on the destination. Vector maintains a native ClickHouse sink rated Stable with at-least-once delivery, JSONEachRow by default, and async insert support, plus a Stable S3 sink writing Parquet among other formats. Fluent Bit has no dedicated ClickHouse output; ClickHouse’s own tutorials route it through the generic HTTP output with FORMAT JSONEachRow and async_insert=1, and the community plugins that once filled the gap are deprecated. Its official S3 plugin is mature. Logstash offers only a community ClickHouse plugin that Elastic does not support.

Throughput benchmarks split by scenario. A VictoriaMetrics Kubernetes benchmark measured Fluent Bit at 31,300 logs/s versus Vector’s 25,000 at equal 1 vCPU/1 GiB budgets, with Fluent Bit using less CPU and memory; a practitioner benchmark to Kafka put Vector ahead at 128k versus 105k logs/s at higher resource cost. For ClickHouse destinations, Vector is the default when native, Stable sink support matters. For pure S3 archiving under the cited 1 vCPU/1 GiB resource limit, Fluent Bit delivered the smaller footprint. With the pipeline settled, the last architectural variable is which object store sits underneath.

Object storage provider comparison

AWS S3 Standard is the baseline, but two alternatives shift the performance and cost position.

S3 Express One Zone stores at $0.11/GB-month, 4.8× S3 Standard, and the April 2025 price cuts made both request types cheaper than Standard: GETs fell 85% to $0.00003 per 1,000 (13× less than Standard’s $0.0004) and PUTs fell 55% to $0.00113 per 1,000, roughly 4.4× less than Standard’s $0.005. For a merge-heavy MergeTree workload where PUT count dominates the bill, that second number matters more than the first. The catch is on the other side of the same announcement: per-GB upload and retrieval charges now apply to every byte transferred rather than only the portion of a request above 512 KB, at $0.0032/GB up and $0.0006/GB down. Merges that repeatedly rewrite multi-gigabyte parts pay that charge on each rewrite, so model the byte-transfer line before assuming Express is cheaper. AWS states single-digit-millisecond first-byte latency, and ClickHouse’s own 24.3 benchmark, vendor-stated, ran a count() over a trillion-row Parquet dataset 7.4× faster on Express One Zone than Standard S3. Express suits a read-heavy hot tier, not an archive, and its single-AZ durability model means you need a separate durable copy elsewhere.

MinIO covers the on-prem case, with a licensing picture that has changed sharply. The company put its open-source repository into maintenance mode in December 2025, relabeled it “no longer maintained” on 12 February 2026, and archived it as read-only that April. The code stays AGPLv3, but there are no official binaries, no reviewed patches, and no upstream watching for CVEs — running it now means owning a Go build pipeline for your storage layer. The commercial AIStor line offers an unrestricted single-node free tier and a multi-node Enterprise Lite tier for deployments under 400 TiB, quoted through sales rather than listed; press coverage of the Enterprise tier has cited entry pricing near $96,000 per year for 400 TiB, so budget accordingly and confirm directly.

ClickHouse’s MinIO compatibility docs state that all S3 functions and tables are compatible with MinIO and note network locality can beat AWS S3 throughput. One configuration quirk is that the MinIO endpoint requires a double slash to mark the bucket root.

MinIO’s published throughput figures need a workload caveat. Vendor benchmarks reaching 249.5 GiB/s GET on 8 nodes use 64–256 MiB objects and MinIO’s own WARP tool, so they do not transfer directly to small-object log workloads.

Which should you choose?

Choose the architecture from the query pattern. Frequent interactive queries justify ClickHouse’s compute and operational surface; rare audits do not.

Choose raw S3 if:

• Queries are rare and ad hoc, and Athena’s $5/TB scan cost over partitioned Parquet stays under the fixed cost of a running cluster.
• The workload is a pure compliance archive where Glacier Deep Archive’s $0.00099/GB-month and 12-hour restores are acceptable.
• Your team wants lifecycle policies and bucket permissions as the entire operational surface.

Choose ClickHouse-on-S3 if:

• Engineers query logs interactively every day and need warm-cache responses in the tens of milliseconds.
• Queries are selective (time range, service, trace ID), where MergeTree beat unindexed Parquet by 10–25× on simple filters and by two orders of magnitude on Altinity’s complex multi-column suite.
• You want tiered retention across SSD and S3, with a separate tested archival workflow for Glacier.

There is a third path for teams that want the ClickHouse query model without owning merge tuning, cache sizing, and replication design. groundcover runs ClickHouse inside your VPC under a BYOC model (Bring Your Own Cloud, where the data plane runs in your own cloud account) with flat per-node pricing decoupled from data volume. The data plane runs in your cloud account and includes compute, storage, and telemetry ingestion and processing. groundcover manages the control plane for the hosted UI and APIs, along with authentication and orchestration. If you want to validate the architecture on your own cluster, the free plan includes BYOC and requires no credit card.