[FLINK-40007][postgres] Fix snapshot fetch size conf does not take effect#4453
[FLINK-40007][postgres] Fix snapshot fetch size conf does not take effect#4453JNSimba wants to merge 3 commits into
Conversation
|
@JNSimba Please add ut cover this bug . |
….getFetchSize() instead of debezium getSnapshotFetchSize()
|
@Mrart Thanks, added the unit tests; could you please trigger the CI again? |
|
From my perspective, connectorConfig.getSnapshotFetchSize() and scan.snapshot.fetch.size should match. I'm wondering if there might be a bug in the parameter conversion. |
This assumes that |
|
Okay, LGTM. cc @yuxiqian Can you help merge this? |
What is the purpose of the change
Relate pr: #2766
The Postgres incremental source uses the Debezium connector config's snapshot
fetch size (
connectorConfig.getSnapshotFetchSize(), default10240) for thesnapshot split read, instead of the Flink CDC option
scan.snapshot.fetch.size(default
1024). As a result:scan.snapshot.fetch.sizehas no effect for the Postgres source.<=the fetch size, the JDBCserver-side cursor returns the whole chunk in a single batch, loading every
row of the chunk into memory at once. On wide tables (many columns / large
rows) this can exhaust the heap and OOM.
The MySQL source already reads
sourceConfig.getFetchSize()for its snapshotread. This PR makes the Postgres snapshot read task do the same, so that
scan.snapshot.fetch.sizeis honored consistently across connectors.Brief change log
PostgresSnapshotSplitReadTasknow takes thePostgresSourceConfigand usessourceConfig.getFetchSize()for the snapshot select statement, instead ofconnectorConfig.getSnapshotFetchSize().Verifying this change
This change is covered by existing Postgres source tests; the snapshot fetch
size now follows the
scan.snapshot.fetch.sizeoption.Does this pull request potentially affect one of the following parts:
Documentation