Skip to content

CancelResumeRaceTest.activeIdReleasedBeforeOutcomeWriteTest is flaky (expected 2 but was 3) #477

Description

@devhawk

CancelResumeRaceTest.activeIdReleasedBeforeOutcomeWriteTest failed on one
matrix entry of an otherwise green run, on a branch that does not touch it.

Run: 33219043815 (PR #476), job test / test (25, temurin)

org.opentest4j.AssertionFailedError: expected: <2> but was: <3>
	at dev.dbos.transact.workflow.CancelResumeRaceTest.activeIdReleasedBeforeOutcomeWriteTest(CancelResumeRaceTest.java:170)

Why this looks like flakiness rather than a defect

  • The same commit passed the same suite on the five other matrix entries in
    that run: 17/temurin, 21/temurin, 17/oracle, 21/oracle, 25/oracle. Only
    25/temurin failed, so it is not a JDK 25 language or API problem.
  • The PR is additive constructors plus two read-path changes, and does not
    touch this test, the queue service, or dispatch. The test has not been
    modified since it was introduced in Active lock management #456.
  • The failure is a count being one too high, not a timeout or an assertion
    about correctness of the thing under test.

The assertion being raced (hypothesis)

Line 170 asserts the workflow body ran exactly twice:

assertTrue(impl.secondRunDone.await(15, TimeUnit.SECONDS), ...);
assertEquals("completed", resumedHandle.getResult());
assertEquals(2, impl.runs.get());   // <-- line 170

secondRunDone is counted down by any run after the first, so it tells the
test that a second dispatch happened — not that no third one will. The test
runs with setSpeedupForTest() on the queue service, and reaches this line
while run 1's terminal outcome write is still parked by the JDBC proxy, which
is exactly the window in which the row is dispatchable. If the poller dispatches
once more before the assertion reads the counter, runs is 3.

So the count is bounded by poller timing rather than by a barrier, which would
explain an occasional extra dispatch under different scheduling. I have not
confirmed this by reproducing it — it is the reading that fits the evidence, and
whether a third dispatch in that window is itself acceptable behaviour is the
part worth a maintainer's eye. If it is, the fix is on the test side (assert no
further dispatch, rather than an exact count observed at an arbitrary moment).
If it is not, the flake is pointing at something real in dispatch.

CancelResumeRaceTest.java#L148-L175

First observation as far as I can tell; there is no prior issue for this test.
Filing it so a recurrence has somewhere to land. Compare #474, which tracks a
different flaky test.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions