Superseding event during retry does not reset retry counter — resource can permanently stall #3479

Bug Report

What did you do?

Operated a reconciler with an InformerEventSource watching a secondary resource. The reconciler throws when the secondary's status is stale (its informer cache hasn't propagated the latest update yet). The retry policy is the default @GradualRetry.

During the retry cycle, the informer delivers a superseding event for the secondary resource (status is now fresh). The superseding event triggers an immediate re-execution (handleRetryOnException line 382–385), but the informer cache is still stale at execution time — so that re-execution also fails.

No maxReconciliationInterval is configured.

What did you expect to see?

The superseding event represents genuinely new external state. When its re-execution fails, the resource should get a fresh retry budget — the new event is a different failure from the original, and the retry counter should reset.

After one more retry (with the cache now fresh), the reconciler should succeed.

What did you see instead? Under which circumstances?

The retry counter from the original failure cycle is never reset. When the superseding event's re-execution fails, handleRetryOnException falls through to nextDelay() (line 396), which returns Optional.empty() because lastAttemptIndex >= maxAttempts — the budget was already consumed by the original cycle.

The resource is permanently stuck. No further reconciliation is attempted.

From the operator logs (real CI failure):

04:38:42  Retry 5 → "another reconciliation will be attempted because a superseding event
           has been received or another retry attempt is pending"
04:38:49  Retry 6 → "no superseding event is present and this is the retry last attempt"
           → "Exhausted retries for scope: ..."

The superseding event was consumed during retry 5, but the informer cache was still stale. By retry 6 the event flag is cleared — the event has been "used up" without ever driving a successful reconciliation.

Root cause

Two things combine:

1. handleRetryOnException does not reset the retry counter when eventPresent is true (line 382–385). It calls submitReconciliationExecution immediately (skipping nextDelay()), but the GenericRetryExecution.lastAttemptIndex is carried forward unchanged.

2. GenericRetryExecution has no reset mechanism. cleanupOnSuccessfulExecution sets the retry to null (line 441), but nothing resets the counter for a new failure cycle triggered by a superseding event.

So when the superseding event's re-execution fails, the next call to nextDelay() sees the counter from the original cycle and reports exhaustion.

Reproducing test

PR #3478 has two @Disabled failing tests in EventProcessorTest:

1. newEventShouldBeRetryableAfterPriorRetryExhaustion — unit-level: a new event after retry exhaustion inherits the exhausted counter and gets zero retry budget.

2. supersedingEventConsumedDuringRetryShouldNotPermanentlyStallReconciliation — uses a real TimerEventSource and CountDownLatch to reproduce the end-to-end scenario:

   • Initial event → fails → timer retry starts (blocked by latch)
   • Superseding event injected while retry is executing
   • Latch released → retry fails → superseding event triggers immediate re-execution → also fails
   • nextDelay() returns empty (exhausted from original cycle) → resource permanently stuck
   • Expects 4 executions (the 4th being a retry with fresh budget that succeeds), but only 3 occur.

Environment

Kubernetes cluster type: vanilla (KinD)

java-operator-sdk version: 5.2.5 (also reproduced at 5.4.0, code path unchanged on main)

Java: 21

Possible Solution

Reset the retry state when a superseding event triggers re-execution. One approach: in handleRetryOnException, when eventPresent is true (line 382), set state.setRetry(null) before calling submitReconciliationExecution. This gives the superseding event's execution a fresh GenericRetryExecution if it fails.

This matches the semantics: a superseding event is new external state, not a continuation of the previous failure. It should get its own retry budget.

Additional context

maxReconciliationInterval masks this bug — if configured, scheduleExecutionForMaxReconciliationInterval fires after exhaustion (line 404) and eventually drives a successful reconciliation. But the default is no interval, and operators that don't configure it are exposed to permanent stalls.