ClickPipes kafka exactly once#6457
Open
genzgd wants to merge 5 commits into
Open
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
4 Skipped Deployments
|
genzgd
changed the title
dhtclk
reviewed
Jun 29, 2026
| Each insert block covers a contiguous range of offsets and carries a deterministic [deduplication token](/guides/developer/deduplicating-inserts-on-retries) of the form `topic:partition:firstOffset-lastOffset`. On replay, ClickPipes reproduces the same offset range and therefore the same token, so ClickHouse rejects the duplicate. Because the token depends only on the offset range, a replay is deduplicated even when the rebuilt block isn't byte-for-byte identical. | ||
|
|
||
| :::note Deduplication window | ||
| Token deduplication is bounded by the target table's [`replicated_deduplication_window`](/operations/settings/merge-tree-settings#replicated_deduplication_window) (the most recent 1,000 insert blocks by default) and [`replicated_deduplication_window_seconds`](/operations/settings/merge-tree-settings#replicated_deduplication_window_seconds). ClickHouse recognizes a replayed block as a duplicate only while its token is still within both bounds. High-throughput pipes can churn through the block-count window quickly, so we recommend raising `replicated_deduplication_window` on the target table to cover your worst-case replay delay (the time window defaults to 7 days, which is usually generous enough). Data replayed after its token has left the window is inserted again, so exactly-once isn't guaranteed in that case. |
Collaborator
There was a problem hiding this comment.
Wanted to verify, looks like replicated_deduplication_window defaults to 10,000 as of 25.9, and replicated_deduplication_window_seconds defaults to 3600 as of 25.10 according to the linked settings docs. This should probably be updated to match.
Contributor
Author
There was a problem hiding this comment.
Good catch, these have definitely changed from our original design. I'll update.
dhtclk
approved these changes
Jun 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Small update to ClickPipes docs for Kafka exactly once, currently in very limited private preview. Not ready to merge.
Checklist