← All results

Kafka Connect + clickhouse-kafka-connect

jvm · connector-runtime · Apache-2.0 · active · java

Homepage · Its configuration in this repository

Has this configuration been reviewed upstream?

Not yet. This configuration has not been checked by the people who maintain this system, and until it has, treat its numbers as ours rather than theirs.

What it was asked to guarantee

at-least-once · offset-commit every 5s

The envelope it ran in

32 CPU · 96g memory

• worker — data-plane · 32 CPU · 96g

Its arms

• clickhouse-kafka-connect · RowBinary + MV rowbinary-mv · realistic · rowbinary · default
  tasks 32 · buffer_count 200 · poll_records 200 · buffer_flush_ms 1000 · client_version V2 · heap_mib 63488 · jvm_opts -XX:+UnlockExperimentalVMOptions -XX:ObjectAlignmentInBytes=16 -XX:G1NewSizePercent=60 -XX:ConcGCThreads=12 -XX:+AlwaysPreTouch -XX:+ExitOnOutOfMemoryError

Where it has been measured

Configurations it refuses

• buffer_flush_ms must exceed . bufferFlushTime=0 disables the connector's time-based flush, so any tail smaller than bufferCount is never inserted and a drain runs to its deadline instead of completing — minutes per cell, reported as a timeout for an arm that is working.
• poll_records must exceed . Renders consumer.max.poll.records, which Connect refuses to start on at 0.

Declared deviations

• Kafka Connect has no fan-out operator — one Kafka record cannot become ~100 rows inside the runtime — so the connector lands the nested batch (Null-engine landing table, per-entrant DDL) and a ClickHouse materialized view performs the flatten, both filters and both derived columns; the transform's CPU therefore runs inside the shared ClickHouse where the cgroup sampler cannot see it. — This arm's efficiency comparison leans on ClickHouse's own ProfileEvents (system.query_log), whose MV cost rides on the parent insert and excludes background merges; the arm's client-side CPU and the other arms' are not the same work, and the page must say so. For the same reason the harness refuses to gate this arm's headroom against the direct-insert ingest ceiling — the ceiling was measured without the MV's per-row work — so its records carry headroom-unproven on the ClickHouse axis rather than a share against a figure that does not describe them. · affects cpu, server-side-cpu
• An unbundled Avro converter — Apache Kafka's Connect runtime ships none, so this arm adds io.confluent:kafka-connect-avro-converter 8.3.0 and its dependency closure, distributed from Confluent's Maven repository rather than Maven Central. — Apache-2.0 at the artifact level (POM-verified). Apicurio's Apache-2.0 converter was evaluated and rejected: its documented Confluent compatibility is server-side (adr/0001 scopes it to Apicurio *serving* the Confluent API), and its own serdes are documented only against Apicurio's registry API — no supported client-side path to Redpanda's Confluent-API registry, and no other Apache-2.0 converter on Maven Central speaks that API at all. · affects decode-path, licensing
• clickhouse-kafka-connect derives insert_deduplication_token = topic-partition-minOffset-maxOffset on every schema-path insert, even with exactlyOnce=false. — Inert here — the landing table is ENGINE = Null (no parts, no dedup window) — but it is the same mechanism ddl.sql discloses for the Spate arm, so it is declared rather than discovered. · affects dedup

Tell us we got this wrong

If this system is configured badly here, that is a bug in this benchmark rather than a result about Kafka Connect + clickhouse-kafka-connect, and the pull request that fixes it is the most valuable one this repository can receive. How to send one. The whole configuration is at entrants/kafka-connect.

This system also reports CPU per row, Cores, data plane, Peak memory, data plane, Peak charged memory, Throttled, ClickHouse CPU per row, GC pause p99, ClickHouse CPU, ClickHouse CPU per written row, ClickHouse CPU wait, Bytes inserted, Rows per insert, Rows written server-side, GC pause max, GC pause p99.9, GC pause total, JVM heap committed peak, JVM heap configured, JVM heap live peak. Those figures are on each arm’s own disclosure in the results table.