Skip to content

[FLINK-39613][runtime] Unquote schema metadata keys#4461

Open
qiuyanjun888 wants to merge 1 commit into
apache:masterfrom
qiuyanjun888:fix/flink-39613-cdc-bugfix
Open

[FLINK-39613][runtime] Unquote schema metadata keys#4461
qiuyanjun888 wants to merge 1 commit into
apache:masterfrom
qiuyanjun888:fix/flink-39613-cdc-bugfix

Conversation

@qiuyanjun888

Copy link
Copy Markdown

Summary

  • Normalize primary-keys and partition-keys entries in SchemaMetadataTransform by stripping MySQL-style outer backticks.
  • Preserve existing unquoted key behavior and unescape doubled backticks inside quoted identifiers.
  • Add a focused regression test for backtick-quoted primary/partition key entries.

Root Cause

projection is parsed through Calcite with the MySQL dialect and therefore resolves backtick-quoted identifiers to their unquoted column names. In contrast, primary-keys and partition-keys were parsed with plain comma splitting and trimming, so entries like `order-id` were kept verbatim and did not match the schema column name order-id.

Fix

  • Apply a small normalization step to primary/partition key entries after trimming.
  • If an entry is enclosed in backticks, remove the outer backticks and decode doubled backticks (`` -> ).

Validation

  • RED: mvn -pl flink-cdc-runtime -am -Dtest=org.apache.flink.cdc.runtime.operators.transform.SchemaMetadataTransformTest#testPrimaryAndPartitionKeysStripBackticks -DfailIfNoTests=false -DskipITs -Drat.skip=true -Dcheckstyle.skip=true -Dspotless.check.skip=true test
  • GREEN: same focused test after the fix.
  • mvn -pl flink-cdc-runtime -am -Dtest=org.apache.flink.cdc.runtime.operators.transform.SchemaMetadataTransformTest -DfailIfNoTests=false -DskipITs -Drat.skip=true -Dcheckstyle.skip=true -Dspotless.check.skip=true test
  • mvn -pl flink-cdc-runtime -am -Dtest=org.apache.flink.cdc.runtime.operators.transform.PreTransformOperatorTest -DfailIfNoTests=false -DskipITs -Drat.skip=true -Dcheckstyle.skip=true -Dspotless.check.skip=true test
  • mvn -pl flink-cdc-runtime -am -DskipTests -DskipITs -Drat.skip=true -Dcheckstyle.skip=true spotless:check
  • mvn -pl flink-cdc-runtime -am -DskipTests -DskipITs -Drat.skip=true -Dspotless.check.skip=true test-compile

Jira

Review

cc @yuxiqian @leonardBang, could you please take a look when you have time?

AI Assistance

This PR was prepared with AI assistance and manually verified with the commands above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant