Repository navigation
Add data_quality_monitors bundle resource (direct engine) - #7094
Draft
kevinlian2it wants to merge 6 commits into
Draft
kevinlian2it wants to merge 6 commits into
kevinlian2it wants to merge 6 commits into
Conversation
Adds the data-quality/v1 Monitor API as a direct-engine bundle resource, alongside the legacy quality_monitors resource it supersedes. Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
Integration test reportCommit: 06f399e
|
Comment on lines
+15
to
+21
| ignore_remote_changes: | ||
| # Create-time options for the built-in dashboard; quality_monitors treats the | ||
| # same fields as not returned on read. | ||
| - field: data_profiling_config.skip_builtin_dashboard | ||
| reason: input_only | ||
| - field: data_profiling_config.warehouse_id | ||
| reason: input_only |
Member
There was a problem hiding this comment.
why can't these be set on the protos? then they would become generated with spec:input_only?
Author
There was a problem hiding this comment.
Good point: neither field is returned on read, so they should be INPUT_ONLY in the proto. I'll send the proto change. Until it reaches the SDK, the hand-written entries are needed; I'll remove them in the SDK bump that picks up the generated ones.
| // dataQualityMonitorUpdateMask lists every client-settable field; object_type | ||
| // and object_id are the identity and recreate on change. This matches the mask | ||
| // the Terraform provider sends. | ||
| const dataQualityMonitorUpdateMask = "anomaly_detection_config,data_profiling_config" |
Member
There was a problem hiding this comment.
DABs uses update_mask: * - does the API support this?
Author
There was a problem hiding this comment.
Yes. The v1 update goes through ApiUtil.update, which accepts * (Monitor has no IMMUTABLE fields), and we've exercised a * mask against staging. Switched to "*".
Comment on lines
+207
to
+210
| s := m.DataProfilingConfig.Schedule | ||
| if s != nil && s.PauseStatus != dataquality.CronSchedulePauseStatusCronSchedulePauseStatusUnpaused { | ||
| m.DataProfilingConfig.Schedule = nil | ||
| } |
Member
There was a problem hiding this comment.
would be good to cover this with an acceptance test
Author
There was a problem hiding this comment.
Added acceptance/bundle/data_quality_monitor, which validates a dev-mode target (schedule dropped) and a production target (schedule kept).
…ceptance test Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
Commit: Regression test reportSelected 14 changed tests (limit=20, 0 not selected) 659 tests: regression=15 unreleased=0 coverage=644 failing=0
15 more failure details omitted, see the full report in the run. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Adds a
data_quality_monitorsbundle resource for the data-quality/v1 Monitor API (anomaly detection on schemas, data profiling on tables). Direct engine only. The legacyquality_monitorsresource is unchanged; deprecating it is a separate follow-up.bundle/config/resources/data_quality_monitor.goembedsdataquality.Monitor. The ID is<object_type>,<object_id>, the same as the Terraform import ID fordatabricks_data_quality_monitor.bundle/direct/dresources/data_quality_monitor.go: CRUD onw.DataQuality. Update sends the static maskanomaly_detection_config,data_profiling_config, the same mask the Terraform provider uses.configs/data_quality_monitors.yml:object_typeandobject_idare ID fields.data_profiling_config.assets_dirrecreates on change, because the update API ignores it.skip_builtin_dashboardandwarehouse_idignore remote changes, because they are create-time inputs. Output-only fields come from the generated config.quality_monitors:run_asis rejected (the API has none), and presets that pause triggers dropdata_profiling_config.schedule(itspause_statusis read-only). The monitor has no name, so there is no name prefix. It has no permissions and no workspace URL, becauseobject_idis a UUID.Why
Data quality monitoring is moving to GA on the data-quality/v1 API, and
quality_monitorsonly covers the legacy table-monitor API. This resource lets bundles manage v1 monitors, including schema-level anomaly detection, which the legacy resource can't express.Notes for reviewers:
ListMonitoris unimplemented server-side, so lookup variables aren't supported for this resource.snapshot: {}is lost when a monitor is defined in Python.quality_monitorshas the same pre-existing limitation; it's visible in thedata_quality_monitors-supportgolden.Tests
all_test.go,state_load_test.go,resources_test.go, and the mutator tests.apply_target_mode_testasserts the dev-mode schedule drop.resources/data_quality_monitors/basic: schema monitor that references${resources.schemas.x.schema_id}. Covers create, in-place update, no drift, recreate on an identity change, and destroy.data_profiling: table monitor. Covers output-only fields not drifting, in-place schedule update, andassets_dirrecreate.remote-delete: an out-of-band delete is detected.lifecycle: cloud-capable.deployment/bind/data_quality_monitor: bind and unbind, and a malformed ID is rejected.python/data_quality_monitors-support.golangci-lint run ./...reports 0 issues. tidy, whitespace, deadcode, codeowners and lockfile checks pass, and all generators are idempotent. 505 PyDABs tests pass.{experimental: {python: …}, resources: …}instead ofpick(…)).lifecycletest against a real workspace. In particular, whether GET echoesskip_builtin_dashboardandwarehouse_id.This pull request and its description were written by Isaac.