Skip to content

test(sync): add a multi-device sync integration test harness - #318

Merged
mohamede1945 merged 16 commits into
feat/sync-statusfrom
test/sync-harness
Oct 14, 2026
Merged

mohamede1945 merged 16 commits into
feat/sync-statusfrom
test/sync-harness

Conversation

@mohamede1945

@mohamede1945 mohamede1945 commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #317 (feat/sync-status). Review that one first.

Summary

Adds an end-to-end integration-test harness for sync. Each test drives the public QuranDataService / AuthService of several simulated devices against one fake /v1/sync server. The tests use only the public API and server state, so the same tests will run unchanged against a rewritten sync engine.

Per device: a real in-memory database, the real Ktor/OkHttp client, and a production AppGraph built per device. Only the database and the OIDC auth layer are replaced. The network is faked only at the boundary, with a loopback HTTP FakeSyncServer.

This PR adds the harness plus smoke tests; the scenario suite comes in the next stacked PRs.

What's included

  • FakeSyncServer mirrors the backend's sync rules and side effects:

    • validation (422) before the lastMutationAt head check (409), and atomic POSTs;
    • pagination, with Favorites as the virtual __default__ collection;
    • bookmark and membership cascades;
    • reading-session upsert and move, and reading-bookmark slot collapse;
    • duplicate names → 409 UniqueViolationError;
    • protected collections → 403;
    • htmlStrip, and note UPDATE accepting only body.
  • Errors use the API-gateway envelope by default; the bare backend envelope can be switched on. details.mutationIndex (from the backend PR) is off by default and can be switched on.

  • Fault injection:

    • a lost response after commit, a blocked POST, network down;
    • failing N requests, or rejecting a chosen mutation;
    • web writes between GET and POST or between pages.

    A request log is kept for assertions.

  • The harness (syncScenario { }) gives you devices, a web actor, syncNow() returning a SyncRun (status, skipped, requests), syncUntilQuiescent (rotating order, unchanged head, no skipped runs), convergence checks and a duplicate check after every scenario.

  • knownIssue(id, description, expect) is a strict expected failure:

    • only an AssertionError matching expect counts as the issue reproducing;
    • harness errors, bugs in the fake, and timeouts fail the test;
    • if the block passes, the test fails with "appears fixed; remove knownIssue".
  • Smoke tests:

    • harness paths: first sync, a note and collection round-trip, web changes, repeat syncs as no-ops, a transient 500, restart, an expired session, logout, a lost response, a blocked POST, a head change between pages, pagination, a gate-skipped run, and the bare envelope;
    • one knownIssue (S7: editing a synced note gets 422 forever);
    • tests of the fake server itself against the backend contract.
  • Deleted SynchronizationClientIntegrationTest: 4 @Ignored tests against the live testing API, using a JWT that had expired.

Seams

  • schedulerTimings: SchedulerTimings? = null on SynchronizationClientBuilder.build and SyncEnginePipeline.setup. Both are @HiddenFromObjC, and the generated Shared.h is unchanged. In tests, a run that keeps failing gives up in about 0.3 s instead of about 13 s.
  • One accepted Metro warning ("unused DriverFactory") in the device graph. The JVM DriverFactory is final, and makeDatabase caches one database per process, so each device binds its own in-memory database directly. This goes away when persistence is reworked.

Known differences from the backend

  • Text tag stripping follows sanitize-html's rules but decodes only common named entities. Its KDoc lists the unverified assumptions.
  • Case folding for non-ASCII names uses Kotlin's root locale, not the Postgres collation.
  • Error messages are close to the backend's but not identical.

Tests

  • ./gradlew :sync-pipelines:jvmTest --rerun: 119 tests, 0 failures. The new tests take about 2 s, and were stable across 8 consecutive reruns.
  • ./gradlew allTests --continue: passes on all targets.

Reviewed in two rounds by a separate reviewer; all findings are addressed.

🤖 Generated with Claude Code

Cleanup (rebased 2026-10-14)

mohamede1945 and others added 16 commits October 14, 2026 11:12
SynchronizationClientIntegrationTest only held four @ignore'd tests that
called the testing API with a hard-coded, long-expired access token. They
never ran, and the integration harness that follows replaces them with
tests against a local fake server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sync-pipelines integration harness drives the production graph, but
it needs each simulated device to sync only when the test says so and to
give up on a failing run in milliseconds, not the 30 s, 5 s and ~13 s of
production timings.

- SynchronizationClientImpl.withSchedulerTimings returns a copy of a
  client with other scheduler timings. It is internal; no public API
  changes.
- The sync-pipelines JVM test compilation becomes a friend of the
  syncengine JVM jar (-Xfriend-paths), so the harness can reach that
  seam. Kotlin does not support associating compilations across
  projects. Main code still sees only syncengine's public API.
- sync-pipelines jvmTest gets kotlinx-serialization-json for the fake
  server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FakeSyncServer serves GET and POST /v1/sync on loopback HTTP for the
integration harness, so devices reach it through the production Ktor
client. It fakes the network boundary only, and its rules come from
quran.com-users-backend pre-live (modules/v1/sync and the services it
calls).

- Per-user ResourceLog and head: CREATE resets a row, timestamps are
  monotonic per user, and GET collapses one entry per resource with the
  stored row. The virtual Favorites entry takes a slot on page 1 and
  shifts later pages.
- POST checks run in the backend's order: query, body validation (the
  route middleware), scopes, then the head (409 OutOfSyncError). The
  batch is atomic, collapses reading-bookmark creates per slot, and
  echoes the validated request data.
- Joi rules per resource, including htmlStrip as sanitize-html with no
  allowed tags (tags dropped, text kept, escaped and trimmed), unknown
  keys rejected, and note UPDATE accepting only body.
- Side effects: a new bookmark goes into Favorites, a membership create
  bumps its collection, removing from Favorites deletes an orphaned
  bookmark, BOOKMARK and COLLECTION deletes cascade, sessions upsert by
  verse and a move deletes the session at the target, reading bookmark
  slots upsert, clear as an UPDATE and project slot 1 onto a bookmark,
  duplicate collection names are 409 UniqueViolationError with
  details.name DUPLICATE, and system and protected collections are 403.
- Errors use the API gateway envelope. BE1's details.mutationIndex is
  off by default and switchable.
- Faults: fail or drop requests, outages, network down, lose a response
  after commit (504 or a dropped connection), block a POST, reject a
  chosen mutation, run web writes before a POST or between GET pages,
  malformed responses, vanished resources and a page-size cap.
- A web actor writes with REST semantics, and the request log and
  ServerState support assertions, including a duplicate check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
syncScenario { } runs end-to-end scenarios against FakeSyncServer. Each
SyncDevice is the production AppGraph built with Metro's dynamic graph
factory, so devices are independent of the process-wide graph and of
each other. Tests use only QuranDataService and SyncAuthService.

- A device has its own in-memory SQLDelight database, the real engine
  and Ktor client pointed at the fake server, and FakeAuthRepository at
  the OIDC boundary (sign in to an account, expire the session, rotate
  the token, fail remote logout).
- Short scheduler timings through the internal seam: nothing syncs on
  its own, and a failing run gives up in well under a second.
- syncNow() calls triggerSync(immediately = true) and waits on the
  public syncStatus flow for the run's end; lastRun keeps its requests.
  login(), restart() and awaitSync() wait for the runs they start.
  syncAll and syncUntilQuiescent sync several devices.
- assertConverged compares every device's public flows with the
  server's live data; each scenario ends with a server duplicate check.
- knownIssue("Sx") is a strict expected failure: it fails once the
  issue is fixed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Seven scenarios that pass against today's engine and exercise the
harness end to end: the first sync of an empty account, a note and a
collection with a verse round-tripping between two devices, a web change
reaching a device, repeat syncs being no-ops, a transient 500 retried in
the same sync, and knownIssue("S7") for editing a synced note, which
still fails with 422 on every attempt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Backend PR #1932 (BE1) adds details.mutationIndex to an unexpected 500
ServerError too, not only to typed errors. The fake now does the same
when BE1 mode is on, and a test covers a note range that names a missing
verse (S11).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the friend-path seam with a parameter, as the harness review
asked.

- SynchronizationClientBuilder.build and SyncEnginePipeline.setup take
  schedulerTimings: SchedulerTimings? = null. Both are hidden from
  Objective-C, and null keeps the internal production defaults.
- The harness passes its timings through the internal
  QuranDataServiceSynchronizationClientFactory, so it no longer casts to
  SynchronizationClientImpl or builds a client with production timings
  first.
- Remove SynchronizationClientImpl.withSchedulerTimings and the
  -Xfriend-paths compiler argument. The iOS framework header is
  unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the harness review.

- Real backend 500s (a note range naming a missing verse, leading zeros,
  a reversed range) throw a dedicated BackendCrash and answer 500
  ServerError. Any other exception in the fake, including a failing
  beforePost or betweenPages callback, is recorded in internalErrors so
  the scenario can fail instead of passing it off as a backend error.
- Typed errors always carry details ({} by default), as the backend's
  BaseError does. Only the untyped 500 has none.
- errorEnvelope switches between the API gateway envelope (default) and
  the bare backend body.
- GET honours metadataOnly.
- BOOKMARK and COLLECTION_BOOKMARK read data.mushaf like the backend, so
  a body with only mushafId gets the default group. A legacy mushaf is
  copied onto mushafId in the echo.
- Document that dropConnection with a finite count is not deterministic,
  because OkHttp may retry by itself. The default lost response stays
  the deterministic gateway 504.
- List htmlStrip's unverified assumptions, and drop a needless !!.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…icates

From the harness review.

- syncNow(), login(), awaitSync() and restart() return the SyncRun:
  its status, whether it was skipped, and its requests.
  assertSucceeded(run) fails on a skipped run, so an expired session can
  no longer pass on the previous Succeeded.
- knownIssue(id, description, expect) accepts only an AssertionError
  that expect allows. Other exceptions, harness timeouts and blocks that
  run past a short timeout fail the test as errors.
  syncRejectedWith(422) pins a run that failed for good. Ids are free
  form.
- A run must start within 5 s and end within 15 s, or the harness throws
  HarnessTimeoutException. A run the lifecycle gate skips, which asks
  for no headers, is reported that way instead of hanging. The KDoc
  states the header-based skip detection as a harness assumption.
- syncScenario fails when the fake server hit an internal error. The
  final duplicate check can run inside a known issue (knownDuplicates),
  and it throws AssertionError.
- Convergence compares sorted lists, so duplicates on a device show.
- syncUntilQuiescent rotates the device order each round, and needs a
  round with no skipped run, no POST and no head change.
- restart() builds the graph and subscribes to syncStatus on Main, so
  the stored session's sync cannot finish before the recorder sees it.
- The fake auth layer keeps tokens in the graph's token store, and the
  device database replaces the app's DriverFactory database in a
  binding that takes it. This clears Metro's unused-graph-input
  warnings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Seven scenarios that pass against today's engine:
- restart() resumes the stored session and pushes a pending edit once;
- an expired session gives a skipped run that assertSucceeded rejects;
- logout() clears local data, keeps the server's, and reports a failed
  revocation;
- a lost response (gateway 504) is retried without a duplicate;
- a write during a blocked POST is pushed by the follow-up sync;
- a head change between GET pages is retried;
- pagination reports total and hasMore, and the device reads every page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The device database binding took the app's DriverFactory only to clear
Metro's unused-graph-input warning. A fake factory can't replace it:

- The JVM DriverFactory is a final class with no constructor parameters
  whose makeDriver() always opens jdbc:sqlite:quran.db in the working
  directory.
- The production binding goes through makeDatabase, which caches one
  QuranDatabase for the whole process and ignores later factories.

So the binding no longer takes the factory, its KDoc names both
blockers, and Metro reports DriverFactory as an unused input for device
graphs until persistence offers a factory seam.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- knownIssue requires expect, and the final duplicate check pins its own
  "has duplicates" failure. Its default timeout is 20 s.
- Harness waits use withTimeoutOrNull, so they turn only their own
  timeout into HarnessTimeoutException and let knownIssue's timeout
  through. A run must start within 3 s.
- The fake server ignores I/O errors from clients that went away. On
  close it releases held requests and waits for its handlers before
  scenarios read internalErrors.
- The lost-response test asserts the single POST got the 504, which
  proves the commit happened before it.
- New tests: a run the lifecycle gate skips fails fast with a clear
  message (FakeAuthRepository.holdNextLogout holds a logout after the
  reset began), and the engine reads the bare backend error envelope.
- FakeAuthRepository keeps the token only in the graph's token store.
- A skipped SyncRun has a null status.
- dropConnection is gone; networkDown is the only connection-drop fault.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gn-out

The old engine publishes a run's end (syncSucceeded or syncFailed) inside
the task function, before the Scheduler leaves WaitingForReply. An
IMMEDIATE trigger that lands in that gap is dropped ("Ignoring redundant
trigger: IMMEDIATE"), so a syncNow() right after login() or another
syncNow() could start nothing and time out after 3 s. That is the CI
flake in COL-06, LIFE-07 and CONV-09. syncNow() now repeats the trigger
every 500 ms until a run starts. SyncScheduler remembers an IMMEDIATE
that arrives during a run, so there the first trigger starts the run and
none is repeated.

FakeAuthRepository counts a header denial before AuthService sees the
empty headers and clears the session. So a skipped run could return while
authState still showed Success, which failed "an expired session gives a
skipped run that assertSucceeded rejects". A skipped run now returns once
authState no longer shows a session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop private plan references from the harness's comments: the BE1 name
for the backend's mutationIndex (now cited as
quran.com-users-backend#1932), S-issue tags, and "today's engine". The
syncNow KDoc no longer names one scheduler, since the retrigger guards
against any scheduler that drops an immediate trigger.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remove FakeSyncServerTest's "metadataOnly returns only the head" and the
fake's metadataOnly branch it tested. No engine or scenario sends
metadataOnly, so the branch was dead code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add an "Integration Tests" section to AGENTS.md: where the suite lives,
why test bodies use only the public API and the fake server, how to
write a scenario, how knownIssue works, the fault hooks, keeping
FakeSyncServer in step with the backend when a synced type is added,
and how to run it. The README's command list points to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mohamede1945
mohamede1945 merged commit aaa363a into main Oct 14, 2026
1 check passed
@mohamede1945
mohamede1945 deleted the test/sync-harness branch October 14, 2026 18:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant