Skip to content

[FLINK-40102][mysql]Support batch emitting CreateTableEvents to reduce schema coordination overhead#4466

Open
eric666666 wants to merge 2 commits into
apache:masterfrom
eric666666:pr/FLINK-40102-mysql-emit-create-table-events-in-batch
Open

[FLINK-40102][mysql]Support batch emitting CreateTableEvents to reduce schema coordination overhead#4466
eric666666 wants to merge 2 commits into
apache:masterfrom
eric666666:pr/FLINK-40102-mysql-emit-create-table-events-in-batch

Conversation

@eric666666

Copy link
Copy Markdown
Contributor

PR Description

What is the purpose of the change

Add a new option scan.emit.create-table-events.in-batch.enabled for MySQL CDC pipeline connector. When enabled, all CreateTableEvents are collected and emitted in batch during the snapshot initialization phase, significantly reducing schema coordination overhead in scenarios with thousands of tables.

Brief change log

  • MySqlDataSourceOptions: Add SCAN_EMIT_CREATE_TABLE_EVENTS_IN_BATCH_ENABLED config option (default false)
  • MySqlDataSourceFactory: Read config and pass to MySqlSourceConfigFactory
  • MySqlSourceConfig / MySqlSourceConfigFactory: Add emitCreateTableEventsInBatchEnabled field and accessors
  • MySqlPipelineRecordEmitter: Add shouldBatchEmitCreateTableEvents() helper; batch-emit all cached CreateTableEvents on first record, populate alreadySendCreateTableTables to prevent duplicate emissions

Verifying this change

  • Added MySqlPipelineRecordEmitterTest (6 test cases) covering:
    • Batch emit when enabled
    • Batch emit only once (idempotent)
    • No batch emit when disabled
    • Empty cache no-op
    • Config default propagation
  • Updated MySqlDataSourceFactoryTest.testOptionalOption to verify option registration and propagation

@github-actions github-actions Bot added docs Improvements or additions to documentation mysql-cdc-connector mysql-pipeline-connector labels Jul 9, 2026
@eric666666 eric666666 changed the title [FLINK-40102]MySQL CDC Source: Support batch emitting CreateTableEvents to reduce schema coordination overhead [FLINK-40102][mysql]Support batch emitting CreateTableEvents to reduce schema coordination overhead Jul 9, 2026
Comment on lines +143 to +145
if (shouldEmitAllCreateTableEventsInSnapshotMode
&& shouldBatchEmitCreateTableEvents()) {
// In snapshot mode or batch emit mode, we emit all schemas at once.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC CreateTableEvents are still emitted one-by-one with this option enabled, and lots of schema coordination might block job from starting up.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @yuxiqian. The goal of this PR is specifically the INITIAL-mode case: without it, each snapshot split's low-watermark triggers a per-table CreateTableEvent interleaved with the data stream. Each one makes the downstream SchemaOperator emit a FlushEvent, which flushes whatever data is currently buffered — so with thousands of tables the data stream is interrupted thousands of times, and on Paimon/Iceberg that means thousands of tiny files + throughput loss.

With the option on, all CreateTableEvents are emitted together up front (first record, before any DataChangeEvent). The FlushEvent count is unchanged, but those flushes now hit an empty buffer (no data has flowed yet), so no real data files are produced. Per-table JDBC (SHOW CREATE TABLE) is also collapsed from N connections to one batched fetch.

The downstream per-event coordination you pointed at (one blocking RPC per CreateTableEvent) is a separate concern; happy to address it in a follow-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Improvements or additions to documentation mysql-cdc-connector mysql-pipeline-connector

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants