Repository navigation
Conversation
compute_rule_fingerprint now hashes user_metadata when it is non-empty, so a save that changes only user_metadata is written as a new version instead of being skipped. Rules without user_metadata keep their existing fingerprint. Updates the storage docs and save docstrings, flips the tests that asserted the old behaviour, and adds Delta and Lakebase integration tests for a metadata-only overwrite. In DQX Studio, replaces the test that required draft and published runs to share a fingerprint. Resolves databrickslabs#1476 Relates to databrickslabs#1686 Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
compute_rule_fingerprintnow includesuser_metadata. Before this change, a save that changed onlyuser_metadatamatched the stored fingerprint and was skipped. The checks table kept the old values and the caller got no error (#1476).user_metadatais hashed only when it is non-empty, so rules without it keep their current fingerprint. Key order does not change the hash.message_expris still left out of the fingerprint.Notes for reviewers
user_metadatawas left out on purpose, so that a tag edit such as changing an owner would not create a new version. This PR reverses that. Any change to a rule, including its metadata, is now a new version.user_metadataget a new fingerprint. The firstsave_checksafter upgrading writes a new version even if nothing changed, and results written before the upgrade won't match the newly stored fingerprints. Matching on rule name still works. This should go in the release notes.run_modeandbinding_versionto each check'suser_metadatawhen it submits a run, so draft and published runs of the same rules now get differentrule_set_fingerprintvalues. Nothing in Studio compares them today. This PR replaces the one Studio test that asserted they matched and updates a docstring. Deployed Studio pins core 0.17.0 (dqx_versioninapp/databricks.yml), so it is unaffected until that pin is bumped.save_checks([], mode="overwrite")still does nothing. That part of [BUG]: TableChecksStorageHandler.save() in overwrite mode #1686 will be a separate fix.Two questions for the Studio maintainers before the pin is bumped:
user_metadata.dq_resolved_rules. Rows for rules with metadata were computed the old way, so therule_set_fingerprintin thedq_rules_coreview won't match core until those rows are recomputed. Should that be a one-time backfill?Linked issues
Resolves #1476
Relates to #1686
Tests
Unit tests: updated or removed the tests that asserted the old behaviour, and added tests for a change to metadata alone, key order, and empty or missing metadata.
test_fingerprint_byte_paritypins seven known hashes; the one for the rule withuser_metadatawas re-captured. The other six are unchanged, which confirms that rules without metadata keep their fingerprint.Integration tests: new Delta and Lakebase tests save a rule, change only its
user_metadata, save again in overwrite mode, and check that the new value loads back.Run locally:
make fmt(no changes),make lint,make testandmake app-test(Studio backend). Integration tests on serverless: all oftest_save_and_load_checks_from_table.py(including the new test) andtest_apply_checks_with_rule_fingerprint.py, 33 passed. The new Lakebase test has not been run locally.Documentation and Demos
This pull request and its description were written by Isaac.