Skip to content

Resolve env-var placeholders in Flink config.yaml - #369

Merged
ferenc-csaky merged 2 commits into
mainfrom
feat/iceberg-auto-maintenance
Jul 22, 2026
Merged

Resolve env-var placeholders in Flink config.yaml#369
ferenc-csaky merged 2 commits into
mainfrom
feat/iceberg-auto-maintenance

Conversation

@velo

@velo velo commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

BaseRunner.initConfiguration() now runs the loaded config.yaml values through the existing EnvVarResolver (the same ${VAR} mechanism already applied to the SQL script and compiled plan). This lets secrets be injected into Flink config via env vars instead of landing in the plaintext config.yaml ConfigMap.

Motivating use: Iceberg auto-maintenance wires a JDBC lock store whose credentials must not sit in a ConfigMap. cloud-compilation emits flink-maintenance.lock.jdbc.user: ${ICEBERG_LOCK_USER} / ...password: ${ICEBERG_LOCK_PASSWORD} placeholders; the flink pod mounts them from the iceberg-maintainance-locktable secret, and this change resolves them at runtime.

Configuration.toMap() returns a copy, so mutating conf while iterating is safe. Values without ${...} are untouched.

Test

CliRunnerTest.initConfiguration_shouldResolveEnvVarPlaceholdersInConfig — asserts a ${VAR} config value resolves from env and literals pass through. Full CliRunnerTest green (14/14, under Java 17).

Related

  • Pairs with cloud-compilation PR (branch feat/iceberg-auto-maintenance) which emits the config.
  • Wayfinder ticket: DataSQRL/cloud-compilation-wayfinder#13

🤖 Generated with Claude Code

Signed-off-by: Marvin Froeder <marvin@datasqrl.com>
@velo
velo marked this pull request as ready for review July 21, 2026 16:52

@ferenc-csaky ferenc-csaky left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added 2 comments, but the core logic looks good.

Comment thread flink-sql-runner/src/main/java/com/datasqrl/flinkrunner/BaseRunner.java Outdated
Comment thread flink-sql-runner/src/main/java/com/datasqrl/flinkrunner/BaseRunner.java Outdated
Signed-off-by: Marvin Froeder <marvin@datasqrl.com>
@ferenc-csaky
ferenc-csaky enabled auto-merge (squash) July 21, 2026 20:10
@ferenc-csaky
ferenc-csaky disabled auto-merge July 22, 2026 10:32
@ferenc-csaky
ferenc-csaky merged commit a51d807 into main Jul 22, 2026
13 checks passed
@ferenc-csaky
ferenc-csaky deleted the feat/iceberg-auto-maintenance branch July 22, 2026 10:32
@ferenc-csaky ferenc-csaky added the enhancement New feature or request label Jul 22, 2026
@ferenc-csaky ferenc-csaky added this to the 0.11.0 milestone Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants