feat: add decode layer, anomaly recording and chain-upgrade tracking [redesign(03)] - #348
Draft
prashantasdeveloper wants to merge 14 commits into
Draft
feat: add decode layer, anomaly recording and chain-upgrade tracking [redesign(03)]#348prashantasdeveloper wants to merge 14 commits into
prashantasdeveloper wants to merge 14 commits into
Conversation
Converts silent decode/resolution failures into a queryable defect list. Wires toEnum's Unknown fallbacks and ClaimRevoked's missing-row case into it; the decode layer wires into it in a later commit. An empty table after a full resync is the acceptance signal.
mapChainUpgrade.ts held oldTxVersion/oldSpecVersion as module-level mutable state, which is per-worker rather than per-chain under --workers. Adds a ChainUpgrade entity and drives detection off it, giving a persisted spec->block map later phases use. Also addresses B9: mappingHandlers.ts had the same pattern (lastBlockHash/lastEventIdx/startupHandled) on the hottest path. Folds the per-block dedup state into a block-scoped context object; startupHandled stays a one-shot in-process check since it has nothing to do with block identity.
Defect A11. v8.0.0's identity migration drains ParentDid and removes every ChildDid in a storage migration that emits no events, so the indexer cannot observe the deletion and the rows persist forever. Driven off the persisted ChainUpgrade crossing rather than a module-level flag, so it is deterministic under --workers and across restarts. Still required under a full resync since the rows are created while indexing v5-v7 blocks.
New src/decode/ module. Since Metadata v14 the metadata is self-describing and struct-style events carry field names, so field(event, 'amount') resolves position from the block's own metadata rather than a hardcoded index - immune to field insertion and reordering. Throws FieldNotFound on a missing name rather than returning undefined. mapEvent.ts's existing v14 boundary handling is factored out and reused rather than duplicated.
Pre-7.x tuple events carry no field names, so they need positional decoding keyed by spec range. resolve(module, event, specVersion) throws NoDecoderForSpecVersion on no match and ArityMismatch on a param-count disagreement, both writing an IndexerAnomaly. These shapes are frozen history and the table is written once. Also folds the polymesh_private_dev spec offsets (2_000_000/2_001_000/ 2_002_000), previously repeated inside is7xChain/is7Dot3Chain/is8xChain, into a single normaliseSpecVersion applied before every lookup.
Settlement, asset, identity and external agents handlers now read event parameters by name through decodeEvent() instead of destructuring params by position. Deliberately excludes balances and staking, which Phase 4 rewrites natively on the decode layer - migrating them now would be throwaway work.
Fixture tests per migrated event, plus a metadata-contract test: for each spec version with checked-in metadata (tests/fixtures/event-arity), asserts every registered legacy decoder's declared arity matches the metadata's actual arity. Mechanically detects a chain shape change with no chain running.
Phase 1 already fixed the ordering bug (A13) by switching orderBy to id. This replaces the hand-rolled helper entirely with a thin getAllByFields wrapper over store.getByFields, preserving the id-ordering property across all seven call sites. getPaginatedData stays as a deprecated single-field adapter for mapPolyxTransaction.ts only, which the POLYX ledger phase deletes along with the entity.
getOrCreateAccount is the hottest chain-read path in the indexer, reached twice per asset movement on v8. Adds a per-block resolution cache with negative caching, so an address the chain has no key record for is looked up at most once per block instead of once per reference.
Moves every index db/compat.sql expressed as a plain or composite column index into schema.graphql via @index/@compositeIndexes, leaving only what directives cannot express (expression indexes, generated JSONB columns, the JSONB path index) with a comment on each explaining why. Also fixes docker-entrypoint.sh's startup race: compat.sql ran backgrounded with a self-kill, racing the node's schema creation. It now runs in the foreground after the node starts and its failure stops the container.
Regenerates ModuleIdEnum/EventIdEnum/CallIdEnum from runtime metadata deterministically (alphabetical), and reports new/changed/removed events per spec version plus events that are registered but not handled. Migration generation is intentionally not included (D5: full resync from genesis). Deletes the stale spec_diffs/ directory, which stopped at 5003000.
…apClaim.ts extractHarvesterArgs read event.event.data directly, letting Codec resolve through whichever @polkadot/types-codec module path SubQuery's own SubstrateEvent declares. subql build's stricter resolution disagreed with that path and serializeLikeHarvester's, failing CI even though tsc --noEmit passed locally. Routes through extractArgs's AnyTuple cast instead, matching mapEvent.ts and every other event-argument read in the codebase.
node:fs/node:path import prefix, explicit localeCompare on every alphabetical sort (SonarCloud flags a bare .sort() as locale-unsafe), the two array sorts that were inline in a returned object literal moved to their own statements, and a trailing-whitespace regex replaced with String.trimEnd().
…de/shapes
registerShape(m, e, [{ from: 0, fields: [...] }]) and its to: LAST_Vn variant
were repeated across asset.ts, externalAgents.ts, identity.ts and
settlement.ts often enough that SonarCloud's cross-file duplication detector
matched a 12-line run in externalAgents.ts against settlement.ts. Adds
stable()/discontinuedAt() shorthands to registry.ts and adopts them
everywhere the pattern applies, across all four shape files for internal
consistency, not just the two flagged. No behavioral change — every shape's
from/to/fields is identical; the fixture tests, which iterate
registeredShapes() generically, pass unchanged.
|
F-OBrien
reviewed
Sep 9, 2026
| }; | ||
|
|
||
| for (const pallet of metadata.asLatest.pallets) { | ||
| const section = pallet.name.toString(); |
Contributor
There was a problem hiding this comment.
snapshot.events and snapshot.calls end up keyed by the metadata spelling of the pallet (Asset, ExternalAgents), but every consumer of a section here uses the event.section spelling (asset, externalAgents) — that's what the arity fixtures, CAPTURED_MODULES and project.ts all use. The two never meet.
Consequences:
eventDriftlooks upsnapshot.events[moduleId]with a fixture key, misses, and falls back to{}— so every event in the fixture is reported asremoved. Against mainnet 8000020 that'sadded (0) / removed (81) / reshaped (0): the fixture's entire contents, withaddedandreshapedstructurally unreachable since both derive from the same emptycurrent.arityFixtureForfiltersCAPTURED_MODULESthrough the same lookup, so--writeemits a fixture with"modules": {}. That also makesdecodeContract.test.tspass vacuously for that spec version, which is the check this fixture exists to drive.
The drift report is the part of this script that's meant to catch a reshaped event, and right now it cannot — a real arity change would be indistinguishable from the noise.
Worth noting the two reports disagree with each other in a single run: unhandledEvents lowercases the section before comparing, so it lists asset.AllowanceSpent as present in the runtime while eventDrift calls it removed, off the same snapshot object. The enum reports are unaffected — planEnumUpdates reads module names and event values, never section keys.
Suggested fix — normalise once on the way in, so the metadata spelling never leaves snapshotFromMetadata:
/** A pallet name as `@polkadot/api` reports `event.section`: `ExternalAgents` -> `externalAgents` */
const sectionId = (name: string): string => name[0].toLowerCase() + name.slice(1);
// in the pallet loop:
const section = sectionId(pallet.name.toString());snapshot.modules is unaffected — section.toLowerCase() gives the same fully-lowercased value either way, which is what ModuleIdEnum wants.
With that applied, mainnet 8000020 against the 8000000 fixture goes to added (0) / removed (0) / reshaped (0).
One note on the tests: tests/unit/syncMetadata.test.ts currently keys both the fixture and the snapshot Balances, so the suite is green over the broken script — the producer and consumer conventions are never crossed. Fixing that convention and adding a case built on real metadata (@polkadot/types-support/metadata/static-substrate, already a dependency, so no chain needed) covers it: capture a fixture from a runtime and read it back against that same runtime, and it must show no drift.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 3 — Infrastructure
PR 4 of 10 in the indexer redesign series. Based on
redesign/02-claims(PR #346).Ships no consumer-visible feature. This is the decode layer, anomaly recording, and the
ChainUpgradehook that Phases 4–6 build on — doing it first means the model rewrites in those phases fail loudly rather than silently.What's in this PR
This is the largest PR in the series (11 commits). Please review commit by commit rather than by the combined diff — each is scoped to one concern.
feat: 🎸 add the IndexerAnomaly entity— converts silent corruption into a queryable defect list. An empty table after a full resync is the acceptance signal.feat: 🎸 persist ChainUpgrade and drop module-level upgrade state—mapChainUpgrade.tsheld upgrade-detection state in module-level variables, unsafe under--workers(per-worker rather than per-chain). Also foldsmappingHandlers.ts's per-block dedup state (B9) into a block-scoped context object.feat: 🎸 retire ChildIdentity rows at the v8 boundary(defect A11) — the v8 upgrade deletes every child identity in a storage migration that emits no events, so the indexer retained stale rows forever. Driven off the persistedChainUpgradecrossing.feat: 🎸 decode event fields by name— newsrc/decode/module. Struct-style events (Metadata v14+) resolve parameters by the field name the block's own metadata carries, immune to field insertion/reordering.feat: 🎸 add the legacy tuple decoder table with arity assertions— positional decoding for pre-7.x tuple events, keyed by spec range, withNoDecoderForSpecVersion/ArityMismatchanomalies on mismatch. Also folds thepolymesh_private_devspec offsets into one normalisation.refactor: 💡 migrate surviving handlers onto the decode layer— settlement, asset, identity and external agents. Deliberately excludes balances and staking, which Phase 4 rewrites natively on the decode layer.test: 💍 add fixture and metadata-contract tests for decoded events— per-event fixture tests, plus a metadata-contract test asserting every registered decoder's declared arity matches the checked-in metadata's actual arity for each captured spec version.refactor: 💡 replace getPaginatedData with store.getByFields— rework, not an introduction. Phase 1 (714e1a7) already fixed the ordering bug (A13); this replaces the hand-rolled helper withstore.getByFieldsacross all seven call sites, preservingid-ordering.perf: ⚡ cache account resolution per block— per-block cache with negative caching forgetOrCreateAccount, the hottest chain-read path (reached twice per asset movement on v8).chore: 🤖 consolidate index declarations into schema.graphql— moves every indexdb/compat.sqlcould express as@index/@compositeIndexesinto the schema; only expression indexes, generated JSONB columns and the JSONB path index remain incompat.sql, each commented. Also fixesdocker-entrypoint.sh's startup race (compat.sqlran backgrounded with a self-kill, racing the node's schema creation).feat: 🎸 add scripts/sync-metadata.ts— regenerates the three chain enums from runtime metadata deterministically, and reports new/changed/removed events per spec version plus events that are registered but never handled (~150 sit as[]today). No migration generation (D5: full resync). Deletes the stalespec_diffs/(stopped at 5003000).If review stalls, there is a clean split point after commit 7: commits 1–7 are the decode layer and anomaly plumbing that Phases 4–6 depend on; 8–11 are independent cleanups nothing blocks on.
Consumer impact
None.
IndexerAnomalyandChainUpgradeare additive; index consolidation is transparent (the indexes already existed, they just move to being declared in one place); no existing entity changes shape.Verification
All green — 359 tests across 26 suites. No
db/migrations/*entries (decision D5, full resync from genesis).