Credence Backend

API and services for the Credence economic trust protocol. Provides health checks, trust score and bond status endpoints (to be wired to Horizon and a reputation engine).

About

This service is part of Credence. It supports:

• Public query API (trust score, bond status, attestations)
• Horizon listener for bond withdrawal events
• Redis-based caching layer
• Configurable lock timeouts – Prevents indefinite waits on locked rows with policy-based timeouts and automatic retry
• Horizon listener / identity state sync – Reconciles DB with on-chain bond state (see Identity state sync).
• Reputation engine (off-chain score from bond data) (future)

Prerequisites

• Node.js 18+
• npm or pnpm
• Redis server (for caching)
• Stellar Horizon server (for blockchain events)
• @stellar/stellar-sdk (Stellar blockchain integration)
• Docker & Docker Compose (for containerised dev)

Setup

npm install
# Set Redis URL in environment
export REDIS_URL=redis://localhost:6379
# Set Horizon URL for blockchain events
export HORIZON_URL=https://horizon-testnet.stellar.org
# Set Stellar network passphrase
export STELLAR_NETWORK_PASSPHRASE="Test SDF Network ; September 2015"
cp .env.example .env
# Edit .env with your actual values

The server fails fast on startup if any required environment variable is missing or invalid. See Environment Variables below.

Run locally

Development (watch mode):

npm run dev

Production:

npm run build
npm start

API runs at http://localhost:3000. The frontend proxies /api to this URL.

---

Docker (recommended for local dev)

The project ships with Dockerfile, docker-compose.yml, and an example env file so you can spin up the full stack (API + PostgreSQL + Redis) in one command.

Quick start

# 1. Create your local env file
cp .env.example .env

# 2. Build and start all services
docker compose up --build

# 3. Verify health
curl http://localhost:3000/api/health
# → {"status":"ok","service":"credence-backend"}

Services

Service	Port	Description
backend	3000	Express / TypeScript API
postgres	5432	PostgreSQL 16
redis	6379	Redis 7

All ports are configurable via .env (see .env.example).

Seeding the database

Drop any .sql files into the init-db/ directory. PostgreSQL will execute them once when the data volume is first created. A placeholder file (init-db/001_schema.sql) is included as a starting point.

To re-run init scripts, remove the volume and restart:

docker compose down -v   # removes data volumes
docker compose up --build

Useful commands

# Stop all services
docker compose down

# Stop and remove volumes (reset DB/Redis data)
docker compose down -v

# View logs
docker compose logs -f backend

# Rebuild only the backend image
docker compose build backend

# Open a psql shell
docker compose exec postgres psql -U credence

Environment variables

All configuration is driven by environment variables. Copy .env.example to .env and adjust as needed. The full reference — every required and optional variable with defaults, validation bounds, and common pitfalls — is in docs/CONFIG_TEMPLATE.md. Key variables:

Variable	Default	Description
PORT	3000	Backend listen port
POSTGRES_USER	credence	PostgreSQL user
POSTGRES_PASSWORD	credence	PostgreSQL password
POSTGRES_DB	credence	PostgreSQL database name
POSTGRES_PORT	5432	Host-exposed PG port
REDIS_PORT	6379	Host-exposed Redis port
DATABASE_URL	(composed)	Full PG connection string
REDIS_URL	(composed)	Full Redis connection URL

---

Scripts

Command	Description
npm run dev	Start with tsx watch
npm run build	Compile TypeScript
npm start	Run compiled dist/
npm run lint	Run ESLint
npm test	Run test suite (vitest)
npm run test:watch	Run tests in watch mode
npm run test:coverage	Run tests with coverage
npm run migrate:create	Create new migration in src/migrations/
npm run migrate:dev	Build and run pending migrations (local)
npm run migrate	Run pending migrations (CI/production)
npm run migrate:down	Rollback last migration
npm run migrate:dry-run	Preview pending migrations without running

API (current)

Method	Path	Description
GET	/api/health	Health check
GET	/api/health/cache	Redis cache health check
GET	/api/trust/:address	Trust score from reputation engine
GET	/api/bond/:address	Bond status
GET	/api/attestations/:address	List attestations for address
POST	/api/attestations	Create attestation
GET	/api/verification/:address	Verification proof (stub)
GET	/api/analytics/summary	Aggregated analytics from materialized view
GET	/api/reports/top-talkers	Top N tenants by request count in last hour

Invalid input returns 400 with { "error": "Validation failed", "details": [{ "path", "message" }] }. See docs/VALIDATION.md.

List endpoints support offset/page and cursor-based pagination. See docs/PAGINATION_CONTRACT.md for cursor format, page-size limits, and ordering guarantees.

Full request/response documentation, cURL examples, and import instructions: docs/api.md

API versioning & stability policy: docs/API_STABILITY.md

OpenAPI spec

docs/openapi.yaml

Render with npx @redocly/cli preview-docs docs/openapi.yaml or paste into editor.swagger.io.

For instructions on how to regenerate the spec after modifying schemas or routes, see docs/OPENAPI.md.

Postman / Insomnia collection

docs/credence.postman_collection.json

Import via File → Import in Postman or Insomnia. See docs/api.md for step-by-step instructions and Newman CLI usage.

Health endpoint (detailed)

The health API reports status per dependency (database, Redis, optional external) without exposing internal details.

• Readiness (GET /api/health or GET /api/health/ready): Returns 200 when all configured critical dependencies (DB, Redis) are up; returns 503 if any critical dependency is down. When DATABASE_URL or REDIS_URL are not set, those dependencies are reported as not_configured and do not cause 503.
• Liveness (GET /api/health/live): Returns 200 when the process is running (no dependency checks). Use for Kubernetes/orchestrator liveness probes.

Response shape (readiness):

{
  "status": "ok",
  "service": "credence-backend",
  "dependencies": {
    "db": { "status": "up" },
    "redis": { "status": "up" }
  }
}

status may be ok, degraded (optional external down), or unhealthy (critical dependency down). Each dependency status is up, down, or not_configured. Optional env: DATABASE_URL, REDIS_URL to enable DB and Redis checks.

Testing

Health endpoints are covered by unit and route tests. Run:

npm test
npm run test:coverage

Scenarios covered: all dependencies up, DB down (503), Redis down (503), both down (503), only external down (200 degraded), liveness always 200, and no dependencies configured (200 ok).

Identity state sync

The identity state sync listener keeps database identity and bond state in sync with on-chain state (reconciliation or full refresh). Use it to correct drift from missed events or for recovery.

• Location: src/listeners/identityStateSync.ts
• Reconciliation by address: sync.reconcileByAddress(address) – fetches current state from the contract, diffs with DB, and updates the store if there is drift.
• Full resync: sync.fullResync() – reconciles all known identities (union of store and contract addresses). Use for recovery or bootstrap.

You supply:

• ContractReader – Fetches current bond/identity state from chain (e.g. Horizon or contract reads). Implement getIdentityState(address) and optionally getAllIdentityAddresses().
• IdentityStateStore – Your persistence layer (e.g. DB). Implement get, set, and getAllAddresses.

State shape is IdentityState: address, bondedAmount, bondStart, bondDuration, active. See src/listeners/types.ts.

Tests cover: no drift (no update), single drift (one address corrected), full resync (multiple drifts), chain missing, store-only addresses, and error handling.

Logging

We rely on structured logging to maintain a consistent schema and protect PII. See docs/LOGGING.md for our policy on reserved keys (request_id, tenant, actor) and redaction rules.

Monitoring

Comprehensive monitoring with Prometheus and Grafana is available.

• docs/OBSERVABILITY.md — operator's index of every Prometheus metric, the Grafana dashboard panel for each, the PromQL behind every alert, and runnable triage queries. Start here if you are operating the service.

• docs/monitoring.md — full setup, instrumentation, and deployment guide for Prometheus + Grafana.

• docs/SLA.md — uptime commitments and per-endpoint SLO/SLI targets for downstream integrators.

• Metrics instrumentation guide

• Grafana dashboard setup

• Prometheus configuration

• Alert rules

• Deployment instructions

Quick start:

# Install metrics dependency
npm install prom-client

# Start monitoring stack
docker-compose up -d

# Access services
# - Prometheus: http://localhost:9090
# - Grafana: http://localhost:3001 (admin/admin)
# - Metrics endpoint: http://localhost:3000/metrics

The Grafana dashboard includes:

• HTTP metrics (request rate, latency, error rate, status codes)
• Infrastructure health (DB, Redis status and check duration)
• Business metrics (reputation calculations, identity verifications, bulk operations)

Deployment: Cutover, Health Gates & Rollback

The service deploys to Kubernetes as a zero-downtime rolling update (k8s/deployment.yaml). See:

• docs/k8s.md — manifests, ConfigMap/Secret keys, and the first-time kubectl apply -k k8s/ quick start.
• docs/deployment-cutover.md — the cutover sequence, exactly what the readiness/liveness/startup probes check (and their timing), and how to detect a bad rollout and trigger kubectl rollout undo.

Backup Strategy (WAL + PITR)

Historical performance benchmarks, latency distributions, and throughput figures across major releases are documented in docs/PERF_BASELINE.md. Use this document to eyeball performance regressions during pre-release testing.

Resilience: Timeouts & Retries

The backend implements a comprehensive timeout and retry strategy for all external service dependencies. Webhook deliveries are now idempotent by default: duplicate retries for the same subscriber/event pair are ignored automatically using a persistent reservation keyed by the subscriber ID and event ID. See docs/timeouts-and-retries.md for:

• Global request budgets and timeout budgets by service type (database, cache, HTTP, Soroban, webhooks)
• Default and per-provider retry policies
• Downstream error classification (NETWORK_ERROR vs TIMEOUT_ERROR vs RPC_ERROR) with typed surfacing
• Environment variable tuning guide
• Operational runbook (symptom → diagnosis → tuning)

Horizon Listener

The service includes a Horizon withdrawal events listener that:

• Monitors Stellar blockchain for withdrawal transactions affecting bonds
• Updates bond states (amount, active status) based on on-chain events
• Creates score history snapshots for significant withdrawals
• Maintains consistency between on-chain and database states
• Handles errors gracefully with automatic retry and recovery

See docs/horizon-listener.md for detailed documentation.

Caching

The service includes a Redis-based caching layer with:

• Connection management - Singleton Redis client with health monitoring
• Namespacing - Automatic key namespacing (e.g., trust:score:0x123)
• TTL support - Set expiration times on cached values
• Health checks - Built-in Redis health monitoring
• Graceful fallback - Continues working when Redis is unavailable

See docs/caching.md for detailed documentation.

Developer SDK

A TypeScript/JavaScript SDK is available at src/sdk/ for programmatic access to the API. See docs/sdk.md for full documentation.

Configuration

The config module (src/config/index.ts) centralizes all environment handling:

• Loads .env files via dotenv for local development
• Validates all environment variables at startup using Zod
• Fails fast with a clear error message listing every invalid or missing variable
• Exports a fully typed Config object consumed by the rest of the application

Usage

import { loadConfig } from "./config/index.js";

const config = loadConfig();
console.log(config.port); // number
console.log(config.db.url); // string
console.log(config.features); // { trustScoring: boolean, bondEvents: boolean }

For testing, use validateConfig() which throws a ConfigValidationError instead of calling process.exit:

import { validateConfig, ConfigValidationError } from "./config/index.js";

try {
  const config = validateConfig({ DB_URL: "bad" });
} catch (err) {
  if (err instanceof ConfigValidationError) {
    console.error(err.issues); // Zod issues array
  }
}

Environment Variables

Variable	Required	Default	Description
PORT	No	3000	Server port (1–65535)
NODE_ENV	No	development	development, production, or test
LOG_LEVEL	No	info	debug, info, warn, or error
DB_URL	Yes	—	PostgreSQL connection URL
REDIS_URL	Yes	—	Redis connection URL
JWT_SECRET	Yes	—	JWT signing secret (≥ 32 chars)
JWT_EXPIRY	No	1h	JWT token lifetime
ENABLE_TRUST_SCORING	No	false	Enable trust scoring feature
ENABLE_BOND_EVENTS	No	false	Enable bond event processing
HORIZON_URL	No	—	Stellar Horizon API URL
CORS_ORIGIN	No	*	Allowed CORS origin
ANALYTICS_REFRESH_CRON	No	*/5 * * * *	Refresh cadence for analytics materialized view
ANALYTICS_STALENESS_SECONDS	No	300	Max acceptable analytics staleness before marked stale
DB_POOL_IDLE_TIMEOUT_MS	No	300000	Milliseconds a pooled connection may stay idle before being closed. Kills idle connections to keep pool counts predictable.
DB_POOL_MAX	No	20	Maximum connections in the primary / replica pools
DB_WORKER_POOL_MAX	No	5	Maximum connections in the background-worker pool
DB_POOL_CONNECTION_TIMEOUT_MS	No	5000	Milliseconds to wait for an available connection
DB_STATEMENT_TIMEOUT_MS	No	30000	Per-statement timeout; kills runaway queries

Analytics materialized views

Analytics endpoints are backed by PostgreSQL materialized views to reduce response latency on aggregate queries.

• View source: analytics_metrics_mv
• Refresh mode: REFRESH MATERIALIZED VIEW CONCURRENTLY
• Default cadence: every 5 minutes (ANALYTICS_REFRESH_CRON)
• Freshness window: 300 seconds (ANALYTICS_STALENESS_SECONDS)

The endpoint response includes staleness metadata:

• asOf: timestamp of snapshot used in the response
• ageSeconds: age of snapshot when served
• fresh: whether snapshot age is within tolerated window
• refreshStatus: ok, stale, or failed_recently

When a refresh fails, the API keeps serving the last successful snapshot and marks the response with degraded freshness metadata.

Database Migrations

The project uses node-pg-migrate for PostgreSQL database migrations with versioning and rollback support.

Prerequisites

• PostgreSQL database
• DATABASE_URL environment variable set (e.g., postgres://user:password@localhost:5432/credence)

Quick Start

Development (recommended):

# Build TypeScript and run all pending migrations
npm run migrate:dev

Production/CI:

# Requires dist/ to be built first
npm run build
npm run migrate

Creating Migrations

Create a new TypeScript migration file:

npm run migrate:create my_migration_name

This creates a timestamped .ts file in src/migrations/.

Migration Workflow

Development:

# Build and run all pending migrations
npm run migrate:dev

# Check which migrations would run (dry run)
npm run migrate:dev -- --dry-run
# Preview pending SQL statements via Admin API
curl -X GET http://localhost:3000/api/admin/migrations/dry-run -H "Authorization: Bearer <ADMIN_API_KEY>"

Production/CI (requires build first):

npm run build
npm run migrate
npm run migrate:down

Rollback:

# Development (builds first)
npm run migrate:dev -- migrate:down

# Production (requires dist/ built)
npm run migrate:down

Migration File Structure

import { MigrationBuilder } from "node-pg-migrate";

export async function up(pgm: MigrationBuilder): Promise<void> {
  // Apply changes (create tables, add columns, etc.)
  pgm.createTable("users", {
    id: "id",
    email: { type: "varchar(255)", notNull: true },
  });
}

export async function down(pgm: MigrationBuilder): Promise<void> {
  // Reverse changes
  pgm.dropTable("users");
}

Environment Variables

Variable	Description	Default
DATABASE_URL	PostgreSQL connection string	Required
MIGRATIONS_TABLE	Table name for tracking migrations	pgmigrations
MIGRATIONS_SCHEMA	Schema for migrations table	public

CI/CD Integration

Run migrations in CI/CD pipelines (requires build first):

# Dockerfile or CI script
npm ci
npm run build
DATABASE_URL=postgres://... npm run migrate
npm start

Initial Schema

The first migration (src/migrations/001_initial_schema.ts) creates:

• identities - Identity and bond state
• attestations - Attestation records
• reputation_scores - Cached reputation scores

After running npm run build, migrations are executed from dist/migrations/.

Best Practices

1. Always test both up() and down() before committing
2. Keep migrations idempotent - safe to run multiple times
3. Use transactions - enabled by default for atomicity
4. Don't modify existing migrations after they've been applied
5. Create new migrations for schema changes
6. Back up production database before running migrations

Tech

• Node.js
• TypeScript
• Express
• PostgreSQL (with migrations via node-pg-migrate)
• Prometheus (metrics)
• Grafana (visualization)
• Redis (caching layer)
• @stellar/stellar-sdk (Stellar blockchain integration)
• Vitest (testing)
• Zod (env validation)
• dotenv (.env file support)

Extend with additional Horizon event ingestion when implementing the full architecture.

Stellar/Soroban Integration

• Adapter implementation: src/clients/soroban.ts
• Integration notes: docs/stellar-integration.md
• Tests: src/clients/soroban.test.ts

Graceful Shutdown

On SIGTERM or SIGINT, the Credence Backend API executes an ordered graceful shutdown sequence:

1. Stops accepting new HTTP connections and allows in-flight requests to drain (server.close()).
2. Closes WebSocket subscription server connections gracefully.
3. Stops event consumers and background schedulers.
4. Closes database connection pools (primary, worker, replica) cleanly (pool.end()).
5. Disconnects from Redis connection.

The grace period is configurable via SHUTDOWN_GRACE_PERIOD_MS (default: 30,000 ms). For more details, see docs/graceful-shutdown.md.

Graceful Degradation

During maintenance or database upgrades, operators can gracefully degrade the service to a read-only state. This is done on a per-request basis by passing the X-Read-Only header set to true or 1.

When active, any state-mutating requests (POST, PUT, PATCH, DELETE) are cleanly rejected with a 503 Service Unavailable response, while read-only requests (GET, HEAD, OPTIONS) are permitted to proceed.

For more details, see docs/graceful-degrade.md.

Replay-Safe Handlers & Side-Effects

To prevent duplicate side-effects (e.g., duplicate webhooks or notifications) when failed inbound events are replayed or retried, the system implements a context-aware replay-safe handler wrapper and a side-effect execution helper.

For details on configuration and usage, see docs/REPLAY_SAFE_HANDLERS.md.

• Replay & Inspection Guide (Operator)

Observability & Logging

For observability, request tracing, metrics, and structured logging guidelines:

• Structured Logging Policy: See docs/LOGGING.md for logs, formats, and conventions.
• Request Tracing & Metrics: See docs/observability.md for request tracing, PII redaction rules, and the req.log request-scoped logger.

Security

For security policies, reporting, and architecture documentation:

• Security Policy & Vulnerability Reporting: See SECURITY.md for details on supported versions and how to report a vulnerability.
• Security Architecture: See docs/security.md for details on the API key scope model, encrypted evidence storage, rate limiting, and dependency scanning SLAs.
• Canonical JWT Claims Reference: See docs/JWT_CLAIMS.md for standard, custom, and impersonation JWT claims, headers, and consumer middleware.
• Rate Limiting Support & Operations: See docs/rate-limiting.md for details on default tier rate limits, environment configuration, troubleshooting, and support FAQs.
• Evidence Upload Security: See docs/evidence-upload-security.md for file upload security configurations, size/count limits, and magic number validations.
• Secret-Rotation Posture: See docs/SECRETS.md for where secrets live, rotation cadence, and blast radius of each credential type.
• Incoming Webhook Security & Posture: See docs/WEBHOOK_RECEIVE.md for signature verification, 5-minute replay window tolerance, and CIDR allowed origins.

Testing

For a full walkthrough — prerequisites, pg-mem vs testcontainers, running migrations, all test commands, the chaos suite, and troubleshooting — see docs/CONTRIBUTING-TESTING.md.

Quick reference:

pnpm test                  # all tests (testcontainers auto-provisions Postgres)
pnpm run test:coverage     # with coverage (40% global threshold)
pnpm run coverage:audit    # audit-sensitive coverage (disputes, governance, evidence)
pnpm run test:chaos        # chaos suite (requires docker-compose.test.yml up)

Rate Limiting

#rate-limiting

API requests are limited per-tier using a token-bucket algorithm. See docs/RATE_LIMITING_DESIGN.md for tier sizes, burst allowance, and reset windows.