feat(alert): add Azure Monitor Logs Deprecation Alert

[jcantrill]

ref: LOG-9195

Description

backport of #3358

Links

https://redhat.atlassian.net/browse/LOG-9195

cc @xperimental

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c5a06621-3e9b-4f36-a037-855c4e0fdb4f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[qodo-for-rh-openshift]

PR Summary by Qodo

Add Prometheus alert for Azure Monitor Logs (HTTP Data Collector API) deprecation

✨ Enhancement ⚙️ Configuration changes 🕐 10-20 Minutes

AI Description

• Add a Prometheus alert when ClusterLogForwarder uses the deprecated azureMonitor output type.
• Embed retirement date and migration/support links in alert annotations for operator guidance.
• Backport the rule into both the source and bundle PrometheusRule manifests.

Diagram

graph TD
  P["Prometheus"] --> TSDB[("Metrics")]
  TSDB --> R["Collector PrometheusRule"] --> A["Azure deprecation alert"] --> N["Alert notification"]

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Surface as ClusterLogForwarder status/condition (operator-managed)

• ➕ Visible directly on the CR without requiring Prometheus/Alertmanager wiring
• ➕ Can be versioned/validated as part of reconciliation logic
• ➕ Avoids reliance on metric availability/labels
• ➖ Requires controller changes and backport complexity
• ➖ Less integrated with existing alerting workflows used by SRE teams

2. Provide only documentation/release-note guidance (no alert)

• ➕ Zero runtime/noise impact
• ➕ No dependency on metrics/prometheus-rule plumbing
• ➖ High risk of customers missing the deadline until breakage occurs
• ➖ No proactive detection of deprecated configurations in live clusters

Recommendation: Keep the Prometheus alert approach since it matches existing monitoring patterns and provides proactive notice. Before merge, ensure the duplicated severity key in the existing ClusterLogForwarderOutputErrorRate rule is intentional (YAML duplicate keys can be ambiguous; most parsers will take the last value, but it’s easy to ship an unintended severity).

Files changed (2) +38 / -0

Other (2) +38 / -0

collector_monitoring.coreos.com_v1_prometheusrule.yamlAdd Azure Monitor Logs deprecation alert to bundled PrometheusRule +20/-0

Add Azure Monitor Logs deprecation alert to bundled PrometheusRule

• Introduces 'ClusterLogForwarderAzureMonitorLogsDeprecation' which fires when 'log_forwarder_output_type{output="azureMonitor"}' is present, warning about the September 14, 2026 retirement and pointing to migration docs. The file also now contains duplicate 'severity' keys under 'ClusterLogForwarderOutputErrorRate' (critical + info), which should be confirmed as intended.

bundle/manifests/collector_monitoring.coreos.com_v1_prometheusrule.yaml

collector_alerts.yamlAdd Azure Monitor Logs deprecation alert to source PrometheusRule config +18/-0

Add Azure Monitor Logs deprecation alert to source PrometheusRule config

• Adds the same 'ClusterLogForwarderAzureMonitorLogsDeprecation' alert with a condensed message and multi-line summary including Red Hat and Microsoft links. Also introduces a duplicate 'severity' key under 'ClusterLogForwarderOutputErrorRate' (critical + info), which should be validated for correctness.

config/prometheus/collector_alerts.yaml

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jcantrill

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [jcantrill]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[qodo-for-rh-openshift]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Context used

⚠️ Tickets: not configured — ticket URL found in PR but could not be fetched — check ticket provider credentials

✅ Compliance rules (platform): 9 rules

1. Duplicate severity key ✓ Resolved 🐞 Bug ≡ Correctness

Description

The ClusterLogForwarderOutputErrorRate rule defines labels.severity twice (critical and
info). Duplicate YAML keys may be rejected by strict tooling or may result in an unintended final
severity value, changing alert routing/urgency.

Code

config/prometheus/collector_alerts.yaml[R39-40]

        severity: critical
+        severity: info

Relevance

⭐⭐⭐ High

Team fixes alert label/template issues in same PrometheusRule YAML; likely to fix duplicate severity
key too.

PR-#3310
PR-#3237

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Both the source rule and the bundled manifest show severity defined twice under the same labels:
mapping for ClusterLogForwarderOutputErrorRate, which makes the resulting alert severity ambiguous
and potentially invalid depending on YAML validation/parsing.

config/prometheus/collector_alerts.yaml[37-41]
bundle/manifests/collector_monitoring.coreos.com_v1_prometheusrule.yaml[38-42]
config/prometheus/kustomization.yaml[1-5]
config/default/kustomization.yaml[15-27]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`ClusterLogForwarderOutputErrorRate` has two `severity` entries under the same `labels` map (`critical` then `info`). Duplicate YAML keys are invalid/ambiguous and can either fail validation or cause one value to override the other, changing the alert severity unexpectedly.

### Issue Context
This duplication exists in both the source PrometheusRule config and the generated bundle manifest, so fixing only one may reintroduce drift.

### Fix Focus Areas
- config/prometheus/collector_alerts.yaml[38-41]
- bundle/manifests/collector_monitoring.coreos.com_v1_prometheusrule.yaml[38-42]
- config/prometheus/kustomization.yaml[1-5]
- config/default/kustomization.yaml[15-27]

### Expected fix
- Keep exactly one `severity` key for `ClusterLogForwarderOutputErrorRate` (likely `critical`, since it is an error-rate alert).
- Ensure the bundle manifest is regenerated/updated so it no longer contains the duplicate key.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[vparfonov]

/retest-required

[jcantrill]

/hold

[xperimental]

/unhold
/lgtm

[xperimental]

/retest-required

[jcantrill]

/override ci/prow/e2e-target
/override ci/prow/functional-target

[openshift-ci]

@jcantrill: Overrode contexts on behalf of jcantrill: ci/prow/e2e-target, ci/prow/functional-target

Details

In response to this:

/override ci/prow/e2e-target
/override ci/prow/functional-target

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.