Apache Iggy connectors
native · connector-runtime · Apache-2.0 · planned · rust
Homepage · Source · Its configuration in this repository
Has this configuration been reviewed upstream?
Not yet. This configuration has not been checked by the people who maintain this system, and until it has, treat its numbers as ours rather than theirs.
Why this is not measured yet
Its connector runtime cannot deliver a decoded Avro payload to any sink. The runtime labels a batch with the decoder's schema rather than the type the decoder produced (core/connectors/runtime/src/sink.rs:362), while the bytes beside it are the decoded payload — JSON, because every converting decoder renders to JSON. The SDK rebuilds the payload from that label (core/connectors/sdk/src/sink.rs:199), so the ClickHouse sink is handed a Payload::Avro holding JSON and drops it: both its body builders match Payload::Json alone (sinks/clickhouse_sink/src/body.rs:44-58 and :76-86). The empty body then returns Ok(()), so nothing surfaces as an error. The same defect hits proto and flatbuffer; only json, text and raw round-trip. No shipped configuration avoids it, and no test in that tree exercises a non-JSON schema through a sink. Its consumer is at-most-once, which this suite compares guarantee-for-guarantee. The runtime hard-codes AutoCommit::When(AutoCommitWhen::PollingMessages) (core/connectors/runtime/src/sink.rs:522), documented upstream as sending "the commit with the poll request itself, before your code sees the batch" (core/sdk/src/clients/consumer.rs:501). A crash between poll and insert loses the batch. No sink or stream config field exposes auto-commit, and upstream's own docs name the crash-safe alternative the runtime does not use. An arm cannot honestly declare [guarantees] delivery = "at-least-once", and the contract has no slot for a system whose only shipped mode is weaker than that. Its fetch path cannot use the envelope. The runtime builds one consumer client for the whole process (core/connectors/runtime/src/stream.rs:68-75) and creates every stream consumer from it, and that connection is lockstep — one request in flight per client (core/sdk/src/tcp/tcp_client.rs:1403). So one poll is outstanding at a time however many [[streams]] entries are configured, and a 32-CPU envelope cannot be spent. Unlike the two above this is a legitimate architectural result rather than a defect, and would be worth publishing on its own — but not as one asterisk among three. Taken together an arm would measure our own patched build, running a weaker delivery guarantee than every other arm, bounded by a connection limit no tuning reaches. Rule 7 is the test: sending that configuration upstream and asking "did we handicap you?" has no good answer. Revisit when the runtime propagates the payload's schema and exposes offset-commit semantics, and when the Spate connectors (spate-etl/spate#358, #359) exist to compare against.
Its arms
No arms are implemented yet.
Where it has been measured
No published measurement carries this system yet.
Tell us we got this wrong
If this system is configured badly here, that is a bug in this benchmark rather than a result about Apache Iggy connectors, and the pull request that fixes it is the most valuable one this repository can receive. How to send one. The whole configuration is at entrants/iggy.