Add cloud clickpipe commands for managing ClickPipes#130
Open
markdawson wants to merge 7 commits into
Open
Conversation
78cfcfb to
66520ab
Compare
|
|
The OpenAPI spec describes several ClickPipe request fields as required for some pipe types and not others, but the schemas have no `required` array so the optionality heuristic infers required for everyone. The live API rejects the resulting empty defaults — cdc mode rejects empty publicationName / replicationSlotName, and database pipes (Postgres/MySQL/BigQuery) reject the destination's table/managedTable/columns/tableDefinition. Flip these to Option / skip-when-empty so callers can omit them, update build_destination to emit a database-pipe-shaped destination when called with an empty table, and record the matching exemptions in spec_coverage_test::OPTIONALITY_EXEMPTIONS. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Provisions a Postgres service and a ClickHouse service in parallel, sets up a publication on the source, creates a ClickPipe with Postgres source and replicationMode=cdc, polls until the pipe is Running, then verifies both the initial snapshot rows and follow-up inserts replicate end-to-end. The cloud-api CleanupRegistry grows ClickPipe tracking so a hung pipe is torn down before its parent service. CI runs the test on the existing nightly cron and on PRs that touch ClickPipe library or CLI source. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Apply the same underscore convention to the clickpipe_run_tags helpers introduced on this branch, plus the existing run_tags / postgres_run_tags helpers, so all suites tag and filter consistently. Service names keep dashes since they are DNS-style identifiers, not tags. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
8bd9994 to
e9715f9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
cloud clickpipecommand group for managing ClickPipes in ClickHouse Cloud, built on top of theclickhouse-cloud-apilibrary client.Commands
list— list ClickPipes on a serviceget— get ClickPipe details (state, scaling, source, destination)create— create a ClickPipe; source type is a nested subcommanddelete,start,stop,resync— lifecycle operations (resync is CDC-only)scale— update scaling configurationsettings get|update— manage per-pipe settingsCreate source types
clickpipe createuses nested subcommands, one per source:object-storage— S3, GCS, Azure Blobkafka— Kafka and Kafka-compatible, with TLS, SCRAM, and MSK IAM authkinesis— Amazon Kinesispostgres,mysql,mongodb— CDC sourcesbigquery— BigQuery sourceOther
--jsonoutput supported for every subcommand