← All results

ClickHouse Kafka table engine

native · database-native · Apache-2.0 · active · cpp

Homepage · Source · Its configuration in this repository

Has this configuration been reviewed upstream?

Not yet. This configuration has not been checked by the people who maintain this system, and until it has, treat its numbers as ours rather than theirs.

What it was asked to guarantee

at-least-once · offset-commit every 5s

The envelope it ran in

32 CPU · 96g memory

• sut — data-plane · 32 CPU · 96g

Its arms

• Distributed forward distributed · realistic · native · default
  num_consumers 32 · block_msgs 16384 · flush_ms 5000 · poll_timeout_ms 500

Where it has been measured

Configurations it refuses

• flush_ms must exceed poll_timeout_ms. The Kafka engine's stream thread evaluates the flush deadline between polls of up to poll_timeout_ms, so flush_ms <= poll_timeout_ms yields flushes paced by poll returns rather than by the configured cadence, and the [guarantees] interval the record declares would not be the one in force.

Declared deviations

• The data plane is a database server acting as a dedicated ingest tier — the container consumes, decodes, transforms and forwards like every arm's client, but carries a server's own overhead (system logs, scheduler, an idle SQL endpoint) inside the envelope; it does no MergeTree storage, which lands on the shared infra ClickHouse outside every arm's budget equally. — The honest resolution of an engine that runs inside a database — published as "ClickHouse as its own ETL tier", one network hop to storage like every arm, not a zero-hop baseline. · affects envelope
• Wire format is native, but via the Distributed engine's interserver transfer (lz4-compressed columnar blocks), not a client insert API any external program could call. The headroom gate divides this arm's rate by the measured native ceiling, which was measured with a CLIENT posting FORMAT Native over HTTP — a transport this arm never uses; probably conservative for interserver blocks, but unmeasured. — Least server-side work of any arm's format; read against spate native, not flink rowbinary_nt, and read the headroom share with the ceiling's transport caveat in mind. · affects wire-format
• Durability cadence is "at most 5 s" — offsets commit per flushed block, and blocks sealing on block_msgs commit sooner. — Stricter than the convention, never looser; disclosed rather than averaged away. · affects durability
• The latency floor is the flush cadence — rows wait up to flush_ms = 5000 in the engine's block buffer, and ingest_ts is stamped by the shared server when the forwarded insert lands. That is the same linger-for-throughput trade as Spate's linger_ms = 500, at 10x the floor — and here the same knob is also the offset-commit cadence, so tightening it buys latency by committing more often rather than being an independent choice. — The honest cost of a server-batched design; the 10x factor is stated so the sustained-mode latency panel is read as a knob consequence, not a mystery. · affects latency
• No insert_deduplication_token is sent (like every arm except Spate). A replay after a crash mid-block is suppressed by the shared table's non_replicated_deduplication_window=1000 when the re-consumed block is byte-identical, and surfaces as the duplicate metric when re-framing produced a different block. — The same content-hash dedup every non-token arm gets; duplicates that do land are reported, never hidden — but "reported" must not be read as "replays cannot be absorbed", because absorbing identical replays is what the window is for. · affects dedup
• At block_msgs = 16384 (~1.2M rows) this arm sends the largest insert blocks of any arm — 4.6x Spate's 262144-row batches, 24x Flink's 50000 — so it holds the smallest share of the background-merge cost that the server-side figure excludes by construction (merges appear in part_log, not query_log). — methodology/measurement.md requires records to say so: the merge cost an insert makes inevitable lands nowhere in the per-row server-side number, and the asymmetry scales inversely with block size. · affects server-side
• Server-side attribution for this arm runs under the INVERTED predicate NOT is_initial_query (declared by [clickhouse].forwarded_inserts): its inserts reach the shared server as Distributed forwards, whose initial query ran on the arm's own node. Every other arm is measured under is_initial_query. — The strict predicate would attribute nothing to this arm; the inversion is declared here so a reader of the record knows this arm's server-side figure came from the complementary predicate, not the one the methodology describes for client arms. · affects server-side
• Drain-mode windows for this arm include the official image's two-phase boot — an init server that runs the DDL and the assert script, then the real server — before the first block can flush; a start-up cost with no analogue in the other arms' single-process clients. — Drain charges start-up for every arm by design; declared because the magnitude here is structurally arm-specific (two server boots plus ~10 assert queries), not because the rule differs. · affects measurement

Tell us we got this wrong

If this system is configured badly here, that is a bug in this benchmark rather than a result about ClickHouse Kafka table engine, and the pull request that fixes it is the most valuable one this repository can receive. How to send one. The whole configuration is at entrants/clickhouse-kafka-engine.

This system also reports CPU per row, Cores, data plane, Peak memory, data plane, Peak charged memory, Throttled, ClickHouse CPU per row, ClickHouse CPU, ClickHouse CPU per written row, ClickHouse CPU wait, Bytes inserted, Rows per insert, Rows written server-side. Those figures are on each arm’s own disclosure in the results table.