[FLINK-40102][mysql]Support batch emitting CreateTableEvents to reduce schema coordination overhead#4466
Conversation
| if (shouldEmitAllCreateTableEventsInSnapshotMode | ||
| && shouldBatchEmitCreateTableEvents()) { | ||
| // In snapshot mode or batch emit mode, we emit all schemas at once. |
There was a problem hiding this comment.
IIUC CreateTableEvents are still emitted one-by-one with this option enabled, and lots of schema coordination might block job from starting up.
There was a problem hiding this comment.
Thanks @yuxiqian. The goal of this PR is specifically the INITIAL-mode case: without it, each snapshot split's low-watermark triggers a per-table CreateTableEvent interleaved with the data stream. Each one makes the downstream SchemaOperator emit a FlushEvent, which flushes whatever data is currently buffered — so with thousands of tables the data stream is interrupted thousands of times, and on Paimon/Iceberg that means thousands of tiny files + throughput loss.
With the option on, all CreateTableEvents are emitted together up front (first record, before any DataChangeEvent). The FlushEvent count is unchanged, but those flushes now hit an empty buffer (no data has flowed yet), so no real data files are produced. Per-table JDBC (SHOW CREATE TABLE) is also collapsed from N connections to one batched fetch.
The downstream per-event coordination you pointed at (one blocking RPC per CreateTableEvent) is a separate concern; happy to address it in a follow-up.
PR Description
What is the purpose of the change
Add a new option
scan.emit.create-table-events.in-batch.enabledfor MySQL CDC pipeline connector. When enabled, all CreateTableEvents are collected and emitted in batch during the snapshot initialization phase, significantly reducing schema coordination overhead in scenarios with thousands of tables.Brief change log
MySqlDataSourceOptions: AddSCAN_EMIT_CREATE_TABLE_EVENTS_IN_BATCH_ENABLEDconfig option (defaultfalse)MySqlDataSourceFactory: Read config and pass toMySqlSourceConfigFactoryMySqlSourceConfig/MySqlSourceConfigFactory: AddemitCreateTableEventsInBatchEnabledfield and accessorsMySqlPipelineRecordEmitter: AddshouldBatchEmitCreateTableEvents()helper; batch-emit all cached CreateTableEvents on first record, populatealreadySendCreateTableTablesto prevent duplicate emissionsVerifying this change
MySqlPipelineRecordEmitterTest(6 test cases) covering:MySqlDataSourceFactoryTest.testOptionalOptionto verify option registration and propagation