Skip to content

Add cloud clickpipe commands for managing ClickPipes#130

Open
markdawson wants to merge 7 commits into
mainfrom
md/add-clickpipes-support
Open

Add cloud clickpipe commands for managing ClickPipes#130
markdawson wants to merge 7 commits into
mainfrom
md/add-clickpipes-support

Conversation

@markdawson
Copy link
Copy Markdown
Collaborator

@markdawson markdawson commented Apr 21, 2026

Summary

Adds a cloud clickpipe command group for managing ClickPipes in ClickHouse Cloud, built on top of the clickhouse-cloud-api library client.

Commands

  • list — list ClickPipes on a service
  • get — get ClickPipe details (state, scaling, source, destination)
  • create — create a ClickPipe; source type is a nested subcommand
  • delete, start, stop, resync — lifecycle operations (resync is CDC-only)
  • scale — update scaling configuration
  • settings get|update — manage per-pipe settings

Create source types

clickpipe create uses nested subcommands, one per source:

  • object-storage — S3, GCS, Azure Blob
  • kafka — Kafka and Kafka-compatible, with TLS, SCRAM, and MSK IAM auth
  • kinesis — Amazon Kinesis
  • postgres, mysql, mongodb — CDC sources
  • bigquery — BigQuery source

Other

  • --json output supported for every subcommand
  • Strict clap value validation on enum inputs (source type, auth mode, etc.)
  • Unit tests for type serialization, state enum, scaling patch, settings, table mapping parsing, and destination deserialization
  • Scaling, settings, and the postgres source block are treated as optional to match real API responses

@markdawson markdawson temporarily deployed to cloud-integration April 21, 2026 20:17 — with GitHub Actions Inactive
@markdawson markdawson marked this pull request as ready for review April 23, 2026 14:58
@sdairs sdairs force-pushed the md/add-clickpipes-support branch from 78cfcfb to 66520ab Compare May 5, 2026 19:26
@sdairs sdairs temporarily deployed to cloud-integration May 5, 2026 19:26 — with GitHub Actions Inactive
@sdairs sdairs changed the base branch from monorepo-structure to main May 5, 2026 19:26
@CLAassistant
Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ sdairs
❌ markdawson
You have signed the CLA already but the status is still pending? Let us recheck it.

@sdairs sdairs had a problem deploying to cloud-integration May 5, 2026 21:31 — with GitHub Actions Failure
@sdairs sdairs temporarily deployed to cloud-integration May 5, 2026 22:08 — with GitHub Actions Inactive
markdawson and others added 7 commits May 5, 2026 23:25
The OpenAPI spec describes several ClickPipe request fields as required for
some pipe types and not others, but the schemas have no `required` array so
the optionality heuristic infers required for everyone. The live API rejects
the resulting empty defaults — cdc mode rejects empty publicationName /
replicationSlotName, and database pipes (Postgres/MySQL/BigQuery) reject
the destination's table/managedTable/columns/tableDefinition.

Flip these to Option / skip-when-empty so callers can omit them, update
build_destination to emit a database-pipe-shaped destination when called
with an empty table, and record the matching exemptions in
spec_coverage_test::OPTIONALITY_EXEMPTIONS.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Provisions a Postgres service and a ClickHouse service in parallel, sets
up a publication on the source, creates a ClickPipe with Postgres source
and replicationMode=cdc, polls until the pipe is Running, then verifies
both the initial snapshot rows and follow-up inserts replicate end-to-end.

The cloud-api CleanupRegistry grows ClickPipe tracking so a hung pipe is
torn down before its parent service. CI runs the test on the existing
nightly cron and on PRs that touch ClickPipe library or CLI source.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Apply the same underscore convention to the clickpipe_run_tags helpers
introduced on this branch, plus the existing run_tags / postgres_run_tags
helpers, so all suites tag and filter consistently. Service names keep
dashes since they are DNS-style identifiers, not tags.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@sdairs sdairs force-pushed the md/add-clickpipes-support branch from 8bd9994 to e9715f9 Compare May 5, 2026 22:26
@sdairs sdairs temporarily deployed to cloud-integration May 5, 2026 22:28 — with GitHub Actions Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants