Telegram (ask questions / claim the issue here first): https://t.me/+DOylgFv1jyJlNzM0
Labels: enhancement, backend, Stellar Wave, devops, infra
An indexer that's dropping every single event still reports as healthy, because nothing counts failures.
soroban-event-worker.ts:247-253 swallows per-event errors with logger.error and moves on, and there's no eventsProcessed/eventsFailed counter anywhere. /health only reports DB status and indexer lag, and lag stays low because the upsert bumps updatedAt on every poll regardless. Admin metrics only show lastLedger/lagSeconds. So a broken indexer that fails on every event looks fine from the outside.
What the fix has to hold to
- processed/failed counts are tracked and visible
- a spike in failures can surface as something other than "healthy"
Done when
Where to start
backend/src/workers/soroban-event-worker.ts for the counters, backend/src/routes/v1/admin.routes.ts and backend/src/routes/health.routes.ts to expose them. A Prometheus exposition endpoint is out of scope. Small-to-medium; the judgment call is what counts as "degraded".
Labels:
enhancement,backend,Stellar Wave,devops,infraAn indexer that's dropping every single event still reports as healthy, because nothing counts failures.
soroban-event-worker.ts:247-253swallows per-event errors withlogger.errorand moves on, and there's no eventsProcessed/eventsFailed counter anywhere./healthonly reports DB status and indexer lag, and lag stays low because the upsert bumpsupdatedAton every poll regardless. Admin metrics only show lastLedger/lagSeconds. So a broken indexer that fails on every event looks fine from the outside.What the fix has to hold to
Done when
GET /v1/admin/metrics.indexer, with a degraded signal when failures spikeWhere to start
backend/src/workers/soroban-event-worker.tsfor the counters,backend/src/routes/v1/admin.routes.tsandbackend/src/routes/health.routes.tsto expose them. A Prometheus exposition endpoint is out of scope. Small-to-medium; the judgment call is what counts as "degraded".