testing: Add Test Coverage for Prometheus Metrics

[chinma-yyy]

Problem

We're exposing 5 Prometheus metrics from the controller, but only 1 of them (BootstrapCompleted) has test coverage. This makes it harder to catch regressions when metrics behavior changes.

Current Metrics Status

• ✅ BootstrapCompleted - Has tests
• ❌ RulesTotal - No tests
• ❌ TaintOperations - No tests
• ❌ Failures - No tests
• ❌ EvaluationDuration - No tests (and not even used in code yet)

What Needs to Be Done

1. Add tests for RulesTotal metric

This gauge tracks the total number of active rules. We should test:

• Value increases when a rule is added to the cache
• Value decreases when a rule is removed from the cache

2. Add tests for TaintOperations metric

This counter tracks taint add/remove operations. We should test:

• Counter increments when adding a taint (with labels: rule, operation="add")
• Counter increments when removing a taint (with labels: rule, operation="remove")

3. Add tests for Failures metric

This counter tracks operational failures. We should test:

• Counter increments on evaluation errors (label: reason="EvaluationError")
• Counter increments on taint operation failures (labels: reason="AddTaintError" or reason="RemoveTaintError")

4. Implement and test EvaluationDuration metric

This histogram is defined but never used. We should:

• Add instrumentation to the evaluateRuleForNode() function to record evaluation duration
• Add tests to verify the histogram records evaluations

Acceptance Criteria

• All 5 metrics have unit test coverage
• Tests verify metrics behave correctly in both success and failure scenarios
• All tests pass with make test

[ajaysundark]

cc @rawadhossain could you help take a first look at this? Thanks

[ajaysundark]

cc @AvineshTripathi for guidance

[rawadhossain]

Hi, thanks for raising the issue!
I took a look at the current implementation and noticed a few things.

The description contains a factual error regarding EvaluationDuration. The metric is already instrumented in evaluateRuleForNode() using prometheus.NewTimer(metrics.EvaluationDuration) and ObserveDuration(), so the work here would be adding test coverage rather than implementing the metric.

Also the existing coverage around BootstrapCompleted is broader than the issue description suggests. There are already multiple tests covering the bootstrap completion flow, including a dedicated metrics test that verifies the counter behavior.

The overall request still seems valid since RulesTotal, TaintOperations, Failures, and EvaluationDuration are all actively used in the controller but currently lack dedicated metrics coverage. However, the implementation may be more involved than the issue description implies:

• The existing counterValue() helper only supports counters.
• RulesTotal (Gauge) and EvaluationDuration (Histogram) will likely require additional helpers or a different testing approach.
• The current metrics tests live in the envtest-based controller test suite, so exercising these metrics will likely require integration-style test scenarios that trigger the relevant reconciliation paths.

I think it would be helpful to update the issue description to better reflect the current implementation and testing requirements.

[ajaysundark]

/assign @rawadhossain