apollo_dashboard: alert on the Chainlink oracle guard trips

[asaf-sw]

Surfaces two of the Chainlink client's guards to on-call, on top of the config
switch that makes the client selectable.

• P4 when a feed reading is rejected for falling outside the freshness window.
  This is Ohad's "alert if price isn't fresh from the last day": the feeds
  guarantee an update every 24h, so a rejection means the feed stopped publishing
  or its updated_at is implausible. The counter covers both directions, since
  a timestamp too far in the future is the poisoning case the guard exists to
  reject, an unbounded updated_at otherwise reading as permanently fresh.
• P3 when a rate is rejected for falling outside the configured sanity bounds.
  Higher, because this is the wrong-feed and poisoned-feed detector, and the one
  guard consensus cannot substitute for: validators check only that they agree
  with each other, and every node reads the same chain state, so a plausible but
  wrong price produces unanimous agreement.

The other two guards get no alert. Every guard trip also increments the pair's
error_count, which already carries a paging alert, so invalid-answer and
contract-call trips would page twice for a case that alert covers. These two earn
their own because they name a cause the aggregate cannot.

or vector(0) keeps each query defined before its counter first exists, since
sum over an empty vector returns empty rather than zero and the generated rules
page on their no-data state. That is also why these apply to observers where the
sibling oracle alerts do not: NotApplicable wraps the whole expression in an
is_observer gate that discards the fallback wherever that metric is absent.
Observers read the same feeds, so a trip is environment-wide either way.

The counters stay flat while a feed is served over HTTP, so these alerts are
silent until a feed is switched to Chainlink.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com

[reviewable-StarkWare]

This change is 

[asaf-sw]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• apollo_dashboard: alert on the Chainlink oracle guard trips #14948 👈 (View in Graphite)
• apollo_l1_gas_price,apollo_l1_gas_price_types: Chainlink guard counters and errors #14981 : 1 other dependent PR (#14982 )
• apollo_l1_gas_price_types: currency pair and rate kind #14979
• apollo_batcher: bound in-flight view calls so they cannot exhaust the blocking pool #14972 : 1 other dependent PR (#14942 )
• apollo_batcher: read class definitions through the class manager in view calls #14967
• main-v0.14.3

This stack of pull requests is managed by Graphite. Learn more about stacking.

[matanl-starkware]

@matanl-starkware reviewed 5 files and all commit messages, and made 1 comment.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on asaf-sw).

---

crates/apollo_dashboard/src/alert_scenarios/chainlink_oracle.rs line 44 at r2 (raw file):

        "Chainlink oracle feed outside the freshness window",
        &CHAINLINK_ORACLE_STALE_FEED_COUNT,
        AlertSeverity::WorkingHours,

The severity should be defined per env (by replacer)

Code quote:

AlertSeverity::WorkingHours,

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit e00c341. Configure here.}

[matanl-starkware]

@matanl-starkware made 2 comments.
Reviewable status: 1 of 5 files reviewed, 3 unresolved discussions (waiting on asaf-sw).

---

crates/apollo_dashboard/src/alert_scenarios/chainlink_oracle.rs line 1 at r4 (raw file):

use apollo_l1_gas_price::metrics::{

I'm not sure we need a different file for chainlink alerts.
Let's try to consolidate all Oracle-related alerts into a single file, preferably the same alerts.

Code quote:

use apollo_l1_gas_price::metrics::{

---

crates/apollo_dashboard/src/alert_scenarios/chainlink_oracle.rs line 37 at r4 (raw file):

/// bound. The feeds guarantee an update every 24h, so a rejection means the feed stopped publishing
/// or its `updated_at` is implausible.
pub(crate) fn get_chainlink_oracle_stale_feed_alert() -> Alert {

I should re-review this PR after the metrics are combined (all errors together)

Code quote:

get_chainlink_oracle_stale_feed_alert