Skip to content

feat(sync)!: run sync on the sync-v2 engine and plain tables - #323

Merged
mohamede1945 merged 30 commits into
feat/sync-v2-enginefrom
feat/sync-v2-wiring
Oct 14, 2026
Merged

mohamede1945 merged 30 commits into
feat/sync-v2-enginefrom
feat/sync-v2-wiring

Conversation

@mohamede1945

@mohamede1945 mohamede1945 commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #322 (the sync-v2 engine). This is port step 2 of 5. The review's findings have landed as new commits (see "Review fixes").

Summary

Moves persistence onto sync-v2's plain tables and wires the new engine in behind QuranDataService. The public API is unchanged apart from the deliberate changes listed below. The old syncengine stays compiled for its shared types, but it never runs; it gets deleted in step 4.

  • Persistence:
    • Domain tables are plain data (notes plus note_range, collections, verse-in-collection, reading bookmarks, reading sessions), plus sync_item/sync_meta.
    • Every local write goes through the repository rules (trim, note creation-time nudge, one highlight color per verse, newest-20 sessions). The importer goes through them too.
    • App-facing queries hide archived notes and system collections.
    • Schema version 2: a version-1 database is dropped and recreated. This is pre-release, with no backward compatibility.
  • Wiring: SyncClient runs SyncEngineV2 through a new SyncScheduler.
    • Same triggers as before: 5 s after a local change, 30 s after app start, immediately, and every 30 min.
    • Backoff that honours the engine's retryable and retryAfterMillis.
    • The scheduler never stops syncing after retries (S21).
    • Pulls continue during a push backoff.
    • syncStatus semantics are the same as feat(sync): add immediate sync trigger and sync status #317.
  • Auth:
    • The account id comes from the token's sub, and switching accounts wipes local sync data (S3).
    • A 401 forces one token refresh; if the refresh fails, the session ends (S25).
    • A transient refresh error keeps the session.
  • Engine rule fixes found while wiring:
    • a collection renamed here and deleted remotely comes back with its verses;
    • identical new notes created on two devices are linked, not duplicated.
  • Bugs the ported tests caught:
    • replacing a bookmark's only collection moved its lastUpdated;
    • addReadingSession threw for a visit older than the newest 20.

Review fixes

Each fix is its own commit, with a regression test:

  • 401 refresh storm: after a forced refresh, a 401 again fails the run without retries. There is at most one refresh per run.
  • Transient refresh errors: only a TechnicalFailure caused by a network error or a timeout keeps the session. Anything else ends it, as before.
  • Collection order: newest first by creation time again. Synced collections keep their own clientCreatedAt.
  • syncStatus during a push backoff: a pull-only run keeps the 403 Failed until a run that pushes succeeds.
  • "Sync now": a request during a run now runs right after it, and a request while a retry is pending runs at once.
  • Leaks: QuranDataService.clear() closes the sync client, which stops the scheduler's command loop and closes the HTTP client.
  • Tests:
    • the lifecycle tests' sync-token and account checks are restored;
    • a refused refresh after a 401 ends the session;
    • ERR-06 now covers the catalog's race, an edit made while the GET is held;
    • ERR-03 expects exactly one 504;
    • a TXT-09 variant where the phone's spelling is newer;
    • bookmark dates for memberships that come from the server;
    • the order of reading sessions visited at the same time.
  • Docs: sync-v2's ensureFavorites and prune are documented as production API.

Integration suite

The integration suite goes from 57 knownIssues to 2, with no scenario going from pass to fail. The flip commits remove the pins area by area. The two that remain:

  • S4 (NOTE-12): the public Note model exposes only the first range. The iOS model change is step 5.
  • S14 (IMP-10): logout clears the import history. Step 3.

About 15 scenarios had their premise adapted to the new engine; each change is explained in its commit message. For example, ERR-06's race can no longer happen, because the remote delete is merged before the push.

Public API changes (iOS/Android follow-ups)

  • updateNote(id, body[, timestamp]): the range parameters are removed, because ranges are immutable.
  • updateCollection/deleteCollection throw ReadOnlyCollectionException. updateCollection throws DuplicateCollectionNameException.
  • Collection.isReadOnly is added.
  • PageReadingBookmark.mushafId is added: 5, 6 or 7, the representative mushaf of the server's group.
  • Shared.h:
    • SharedNote_ is renamed to SharedNote and SharedCollection_ to SharedCollection, so iOS renames Note_ to Note;
    • 56 plumbing classes are removed;
    • no sync-v2 types are exposed.
  • Behaviour changes:
    • notes hides archived notes;
    • bookmark ids are "sura:ayah";
    • names starting with system: are rejected;
    • surah and juz reading bookmarks set on the web show as empty.

Tests

  • ./gradlew allTests --continue passes on every module and target:

    • sync-v2: 177 JVM;
    • auth: 139 per target;
    • persistence: 171 JVM, 180 on iOS;
    • sync-pipelines: 296 JVM, 73 on Android host and iOS.
  • :umbrella:linkDebugFrameworkIosSimulatorArm64 and :demo:android:assembleDebug succeed.

  • The integration suite passed 3 runs in a row with --rerun.

  • Persistence tests went from 247 to 171. 155 were dropped:

    • 152 tested removed sync plumbing (pending ops, ACKs, tombstones, replay, write guards);
    • 3 covered seeded highlight collections, which no longer exist.

    The rest were rewritten for the new tables, and new tests cover the new write rules.

Open

  • Held items are silent. An item the server refuses is held, and syncStatus still reports Succeeded. No public API lists held items yet; that is planned together with sync history.

🤖 Generated with Claude Code

Cleanup (rebased 2026-10-14)

  • Removed 34 sync-v2 tests that replay integration scenarios exactly, now that the scenarios run on sync-v2.
  • Merged LIFE-14 with its server-rejection variant.
  • 2 persistence test deletes and 5 merges.
  • Fixed comments that later changes made stale.
  • Fixed the Apple demo (Note_ → Note), which this PR had broken.
  • Added a README Sync section on the architecture: the sync-v2 engine, persistence on plain tables, the wiring, scheduler triggers and backoff, auth, and syncStatus.

mohamede1945 and others added 27 commits October 14, 2026 11:23
- SyncResult.Failed carries its cause (the server's answer or the network
  error), so the app can report the HTTP status and error code.
- sync(account, push = false) fetches and merges only, for pulling while
  pushes back off after a 403 that may be a scope error.
- SyncType.forbidsItems: only collections have items the server refuses
  with 403. For other types a 403 without a mutationIndex is the request's
  scope, so the run stops instead of reverting the edit or holding a create.
- Collections split read-only (no rename or delete) from system (hidden from
  app lists): an older app's protected color collection is read-only but
  still a user's collection.
- syncTypes(database) is the one type list, and SyncEngineV2.wipe(database,
  types) wipes in one transaction for a reset outside the engine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…refresh errors

- AuthRepository.forceRefreshTokens refreshes even when the token has not
  expired locally, for a 401 from the sync server (S25). AuthService exposes
  it as forceRefreshAccessToken: a refused refresh token ends the session,
  which authState reports.
- A refresh that fails on the network, with a timeout or a 5xx no longer
  clears the session; the error is thrown and the next attempt retries (S3).
  Only a refused refresh token, or an unclassified error as before, ends it.
- AuthService.currentAccountId returns the signed-in user's sub, which sync
  stores next to its token to detect an account switch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…w notes

Two merge rules the integration suite needs, with the queries the app's
repositories read:

- Edit beats delete for a collection renamed locally and deleted remotely now
  recreates it as this device has it, verses included: the server's cascade
  of membership deletes is beaten too (COL-14). A collection recreated only
  because a verse was added keeps just the new verses, as before.
- A new local note equal to a new remote note (same clientCreatedAt, ranges
  and body) is linked to it instead of pushed as a duplicate: two devices
  importing the same legacy notes, or an import over notes another device
  already synced (LIFE-05, IMP-03).
- App-facing queries: a note by id, notes not archived (newest first), a body
  edit, a session by id, and every collection a verse is in.
- kotlinx-serialization and the Ktor client are api dependencies: State and
  SyncApi expose them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SyncScheduler decides when the new sync client runs, keeping today's timings:
30 s after app start, 5 s after a local change (a burst runs once), at once
for an immediate trigger, and 30 min after every run. Runs never overlap; a
local change during a run runs again after it, other triggers during a run or
while a retry waits are dropped, as before.

A failed attempt is retried with backoff (200 ms x 2.5^n, up to 5 retries),
unless it isn't retryable or asks for a longer backoff of its own. After the
retries run out the periodic run is still scheduled, so sync keeps trying
without user action (S21). The run reports a final failure itself, through
willRetry, so the report lands before the next run or a logout's reset.

Not wired yet; the next commits run the sync-v2 engine through it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Persistence and the wiring change together, so every commit builds: the old
sync engine's bridge can't outlive the pending columns it reads.

persistence:
- MobileSyncDatabase: QuranDatabase (key-value store, import history) and
  sync-v2's SyncV2Database (plain domain tables, sync_item, sync_meta) on one
  driver. MobileSyncSchema is version 2 with foreign keys on; a version-1
  database (the old pending columns) is dropped and created again, since
  mobile-sync is pre-release with no backward compatibility.
- Repositories write plain rows through one store per type (NoteStore,
  CollectionStore, ReadingSessionStore, ReadingBookmarkStore), which the
  importer uses too: trimmed text, the note clientCreatedAt nudge, one
  highlight color per verse, the newest 20 reading sessions. App-facing
  queries hide archived notes; collection lists still hide system ones.
- Reset wipes domain rows, sync state (token, account, outbox) and import
  history in one transaction and bumps the engine's wipe epoch.
- API (deliberate): updateNote takes only the body (ranges are immutable);
  renaming or deleting a read-only collection throws
  ReadOnlyCollectionException, renaming onto a taken name throws
  DuplicateCollectionNameException; Collection.isReadOnly;
  PageReadingBookmark.mushafId. Ayah bookmark ids are "sura:ayah".
- Removed: the old tables, *SynchronizationRepository, Remote*/LocalSync*
  inputs, PersistenceWriteBoundaryGuard, CollectionName(Key). The *Impl
  classes and PersistenceModule are hidden from Objective-C.
- Tests ported to the new tables; the ones for removed sync columns and
  plumbing are dropped. Two bugs the port found are fixed: replacing a
  bookmark's only saved collection no longer moves its date, and a visit
  older than the newest 20 no longer throws.

sync-pipelines:
- SyncClient runs SyncEngineV2 through SyncScheduler for the signed-in
  account (the token's sub). A 401 forces one token refresh and runs again;
  a refused refresh ends the session (S25). A 403 that may be the token's
  scope stops pushes for its backoff while automatic runs keep pulling; "sync
  now" pushes. Each request is admitted through the session lifecycle gate.
  syncStatus keeps #317's semantics.
- SyncEnginePipeline only holds the repositories; the result receivers and
  SyncLocalModificationDateStore are gone (the token lives in sync_meta).
- Test harness: device databases are MobileSyncDatabase, FakeAuthRepository
  implements forced refresh, and updateNote call sites drop their ranges.
  The integration suite's knownIssues that this fixes are flipped next.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NOTE-02, NOTE-03, NOTE-06, NOTE-09, NOTE-10, NOTE-14, NOTE-17 and NOTE-18,
plus the smoke test's note edit, pass now: their knownIssues (S1, S2, S7, S12,
S13, S15, S17) and NOTE-17/18's known duplicates are gone, assertions kept.

- NOTE-13 is rewritten: ranges are immutable, so updateNote can't move a note
  and device and server agree on its range (S4's sync part).
- NOTE-12 stays pinned on S4 alone: every range is stored, but the public Note
  model shows the first one (the iOS model change is step 5).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
COL-03, COL-04, COL-06, COL-07, COL-09, COL-10, COL-11, COL-12, COL-13,
COL-14, COL-15, COL-16 and COL-17 pass now (S1, S2, S9, S11, S15, S16, S17,
S26). COL-12 and COL-17 now also check the typed errors:
DuplicateCollectionNameException, and ReadOnlyCollectionException for an older
app's protected color collection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BMK-02, BMK-05, BMK-07, BMK-11, BMK-16, BMK-17, BMK-19 (S6, S15, S23) and
HL-07 (S24) pass now; their knownIssues are gone and the assertions kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- RS-08, RS-10, RS-11, RS-12 and the RS-13 variant pass (S1, S5, S8, S19).
  RS-09's tablet visit is now newer than the phone's sessions, so the
  newest-20 prune every device runs keeps it and the test checks S1 alone.
- RB-07 (S18) passes, and RB-06 (S13) checks the page's mushaf (7) through
  the new PageReadingBookmark.mushafId.
- TXT-11 (S13) passes. TXT-09's phone collections are dated before the web's:
  linking a local create to a remote collection of the same name keeps the
  newer spelling, so the test was timing-dependent.
- Fake fix: a reading-bookmark UPDATE that set name to null kept the old name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ERR-05, ERR-07, ERR-08, ERR-10 (S11), ERR-12 (S21) and ERR-19 (S27) pass.
- ERR-06 (both) is rewritten: the engine merges a remote delete before it
  pushes, so S10's race can't happen; they check the row is gone before the
  POST and no 404 follows.
- New: ERR-18 variant, pulls go on while pushes back off after a 403.
- Adapted to the new engine: ERR-03 (one POST, one lost response), ERR-13 (the
  GET is merged before a broken POST), OBS-02 (a permanent 503, since a
  rejected mutation is now held), and the harness paths tests (a 422 with a
  code is held; a write during a blocked POST is pushed in the same run).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- LIFE-09 (S3) and the LIFE-14 variant (S25: one forced refresh) pass.
- CONV-02, CONV-04 and CONV-08 pass (S2, S6, S7, S13).
- IMP-07 (S11) and IMP-09 (S20) pass: the importer goes through the
  repositories, so it prunes to the newest 20 sessions. IMP-08 expects the
  second note's creation time nudged by 1 ms.
- IMP-10 stays pinned on S14 (step 3).
- The S6/S7 pins are no longer used and are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
README and AGENTS.md now describe the app database (MobileSyncDatabase), the
stores that apply the write rules to every local write, the synced types, the
sync client and scheduler timings, and the schema policy (no migrations; an
older database is dropped). SYNCENGINE_ARCHITECTURE.md is marked as the
previous engine's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a forced refresh, a 401 again (a server that rejects every token) used
to come back retryable, so each of the run's retries forced another refresh:
six in a row. Now that second 401 fails the run without retries; the next run
(periodic, or a user's) may refresh once more.

Test: LIFE-14 variant, a server that keeps rejecting tokens gets one refresh
per run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the run

FakeAuthRepository can refuse the next forced refresh, as a revoked refresh
token does. The run then ends without a status of its own (skipped), the
session is cleared, and authState is no longer Success (S25).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h failures

The OIDC library wraps every non-HTTP failure in TechnicalFailure, so keeping
the session for any TechnicalFailure also kept it for errors that will never
succeed (an unreadable response). Now only a TechnicalFailure whose cause
chain holds an IOException or a timeout keeps the session; anything else ends
it, as before.

Tests: a refresh that times out keeps the session; one that fails for another
technical reason ends it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The plain-tables port listed collections in row order. The app-facing order is
back to the old one, created_at DESC (ties by the newer row), and a synced
collection now keeps its own creation time: COLLECTION reads clientCreatedAt
into an ignored createdAt field (written on insert, sent as clientCreatedAt),
where it used to take clientUpdatedAt.

Test: collections are listed newest first by creation time, on the device that
made them and on one that synced them, including a collection from the web.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The repository test still expected row order, which the previous commit
replaced with the app-facing newest-first order. It now checks that order,
with the seeded Favorites (no creation time of its own) last.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… pulls

A pull-only run during a push backoff (a 403 that may be the token's scope)
reported Succeeded, so the 403 vanished from syncStatus while nothing could be
pushed. Such a run now ends as skipped, which keeps the previous status:
Failed stays until a run that pushes succeeds.

Test: the ERR-18 variant asserts the periodic pull keeps the Failed status, and
that the pushing "sync now" turns it into Succeeded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng retry

An immediate trigger (a user's "sync now") during a run, or while a retry
waited, was dropped, so a push the user asked for could wait for the local
change delay, the retry backoff or the periodic run. The scheduler now
remembers one during a run and runs (and pushes) right after it, ahead of a
local change's delay or a retry; one while a retry waits replaces the retry
with an immediate run. triggerSync's KDoc says so.

Tests: an immediate trigger during a run runs right after it; one while a
retry waits runs at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…unt again

The port dropped the lifecycle tests' token assertions with the old token
store. The fixture now runs the real PersistenceResetRepositoryImpl over its
in-memory database (the fake only adds failures and counts), seeds sync_meta
with a token and an account, and checks that every logout and the persisted
reset recovery wipe them, while a logout aborted by a local-auth-clear failure
keeps them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…vice is cleared

QuranDataService.clear() only cancelled the scheduled work: the scheduler's
command loop and scope, and the sync client's HTTP client, outlived the
service. SyncClient is now AutoCloseable; clear() closes it, which stops the
command loop, the timers and any running attempt for good, and closes the HTTP
client. cancelAndJoin after close returns at once.

Tests: close stops the command loop for good (later triggers run nothing);
clearing the service closes its sync client.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ERR-06 rewrite only showed the remote delete is merged before the POST.
The fake server can now hold a GET (ServerFaults.blockGet), and both ERR-06
tests add the catalog's race: the device edits R while the GET carrying R's
delete is held, before the merge commits. Edit beats delete, so R is recreated
with the edit (the note keeps its range; the collection its new name), with no
404. The earlier assertions stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ERR-03 armed a second lost-response fault that the new engine's single POST
never reaches, and only asserted that some response was lost. It now arms one
fault for the POST that carries the verse and the session, and asserts exactly
one 504.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…collection

TXT-09 dates the phone's collections before the web's, so the server keeps
"ÉTUDE". The new variant covers the other side of that rule: the phone's
"étude" is newer, so the link keeps it and renames the server's collection,
on every device.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r tie-break

Coverage the persistence port lost, now at the app level:

- Bookmark dates of verses from the web: a membership's time is its
  clientUpdatedAt, or its clientCreatedAt, or the server's timestamp; the
  bookmark was last updated when its earliest surviving saved membership was
  made. The fake server can now leave client times out of COLLECTION_BOOKMARK
  entries (ServerFaults.omitMembershipTimes), and the web can add a verse with
  explicit client times.
- Reading sessions visited at the same time are listed by verse, on every
  device, the way the prune ranks them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sync-v2 repositories are stand-ins for the app's, but two of their
companion functions are production code: persistence seeds Favorites with
CollectionsRepository.ensureFavorites, and its reading-session store prunes
with ReadingSessionsRepository.prune, the same ranking the sync type's prune
uses. Their KDoc now says so, and the classes say they are test stand-ins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop the S-issue tags the wiring added to auth, the sync client and
scheduler, and the flipped integration scenarios, and rewrite ERR-06's
KDoc without the old-versus-new engine history and the catalog's name
for the race.

Fix comments the wiring made stale:
- ServerFaults.pageSizeCap no longer warns about a client that derives
  its page count from its own limit; sync-v2 pages by hasMore.
- sync-v2's NotesRepository and notes.sq no longer claim every local
  note write goes through NotesRepository; the app and the importer
  write through persistence's NoteStore.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mohamede1945 and others added 3 commits October 14, 2026 11:26
Now that QuranDataService runs on sync-v2 and the scenarios pass end to
end, remove 34 sync-v2 acceptance tests that replay a scenario's steps
and assertions and pin nothing more precise:

- notes: NOTE-01 with CONV-02, TXT-01/03, NOTE-10, NOTE-11, NOTE-15
- collections: CONV-02, LIFE-01, BMK-10, COL-04, BMK-02, BMK-05,
  BMK-03, COL-09, COL-10, COL-14, BMK-16, COL-15, COL-13, COL-17, HL-09
- reading sessions: RS-01, RS-11, RS-13, RS-04, RS-07, IMP-09
- reading bookmarks: RB-02, RB-05, RB-07, ERR-19, RB-04, RB-06
- highlights: HL-01, BMK-06; a Favorites removal from the web: BMK-11
  and BMK-17; a create refused for the scope: ERR-18

Remove the harness's "the engine reads the bare backend error envelope":
SyncErrorParsingTest covers parsing the backend's own shape, and ERR-07
the 422 hold.

Merge LIFE-14 with its server-rejection variant. Since a 401 forces a
refresh, both are the same case; the kept test asserts the first request
got 401, one refresh, and the note landed.

In persistence, remove "a new database has Favorites" (the collections
repository test checks the same row) and the one-line overload test for
addReadingSession, and fold four "already deleted" cases into their
"missing" twins, since hard deletes take the same path. getNotesFlow's
archived check joins getAllNotes'.

Move NOTE-18's KDoc back onto its test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
persistence no longer generates its own Note table class, so the
app-facing Note reaches Swift as Note instead of Note_. The demo still
used Note_ and did not compile; CI does not build it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a Sync section to the README's architecture: the sync-v2 engine,
persistence on plain tables, the sync-pipelines wiring, when the
scheduler runs and backs off, how auth feeds sync (the account from the
token's sub, a forced refresh after a 401), and what syncStatus shows.
Drop the stale claim that imports track sync mutations.

In AGENTS.md, point "adding a synced type" at syncTypes and fold the
development note into the sync-v2 steps, adding the integration fake.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mohamede1945
mohamede1945 merged commit d4a5f26 into main Oct 14, 2026
1 check passed
@mohamede1945
mohamede1945 deleted the feat/sync-v2-wiring branch October 14, 2026 18:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant