Skip to content

Expand observability: worker queue depth, store transaction boundaries, virtual-time tracking #3

Description

@coderabbitai

Background

Raised during review of PR #1 (Initial repo structure) by @leynos.

The PR introduces structured HTTP request logging and REST handler error logging, but observability for the deterministic worker, in-memory store, and virtual clock (specified in ADR 0004) is absent. Without this, diagnosing queue stalls, store transaction failures, or virtual-time drift in CI will be difficult.

Required work

  • Worker queue depth: emit a structured log event whenever the queue depth changes (task enqueued, task dequeued, queue drained to idle).
  • Store transaction boundaries: log transaction begin/commit/rollback with duration and affected slice names.
  • Virtual-time advancement: log each advanceClock call with the delta and resulting virtual timestamp.
  • Metrics: expose throughput, latency, and error-rate counters on the /_digitalpuddle admin surface (can be a simple JSON endpoint rather than Prometheus initially).
  • Alerts / thresholds: define documented thresholds for worker queue stall (e.g. queue non-empty for > N virtual seconds without progress) and store transaction failure rate that should trigger a test failure or warning.

References

/cc @leynos

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions