Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 58 additions & 0 deletions docs/architecture/scheduler-jitter.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# Scheduler Jitter / Anti-Thundering-Herd

## Problem
When multiple offering distribution ticks are scheduled at the same wall-clock time, all workers fire simultaneously, causing:
- Database connection pool exhaustion
- Rate-limit contention on external APIs
- Uneven system load with burst/starve pattern

## Solution: Jittered Scheduling

### Implementation
```typescript
interface SchedulerConfig {
/** Base cron expression for the tick */
cronExpression: string
/** Maximum random delay in milliseconds (default: 5000) */
maxJitterMs: number
/** Minimum spacing between ticks in ms to avoid herd (default: 1000) */
minSpacingMs: number
}

class JitteredScheduler {
private offeringTicks: Map<string, ScheduledTick>
private jitterProvider: () => number // injectable for testing

constructor(config: SchedulerConfig) {
this.jitterProvider = () => Math.floor(Math.random() * config.maxJitterMs)
}

async scheduleTick(offeringId: string, handler: () => Promise<void>): Promise<void> {
const jitter = this.jitterProvider()
setTimeout(async () => {
await handler()
}, jitter)
}

async scheduleMultiOffering(offeringIds: string[], handler: (id: string) => Promise<void>): Promise<void> {
// Stagger ticks to avoid thundering herd
for (let i = 0; i < offeringIds.length; i++) {
const stagger = i * this.config.minSpacingMs
const jitter = this.jitterProvider()
setTimeout(async () => {
await handler(offeringIds[i])
}, stagger + jitter)
}
}
}
```

### Benefits
- **Reduced DB contention**: Staggered writes avoid lock contention
- **Predictable load**: Even CPU/memory utilization across tick windows
- **Resilience**: Jitter prevents cascading failures from simultaneous retries

### Testing
- **Deterministic jitter**: Inject fixed jitter provider for reproducible tests
- **Concurrency stress**: Test with 100+ offerings at same cron tick
- **Recovery**: Verify jittered retry on failure
17 changes: 17 additions & 0 deletions docs/testing/scheduler-jitter-test-plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Scheduler Jitter Test Plan

## Unit Tests
- [ ] `JitteredScheduler.scheduleTick()` respects maxJitterMs bound
- [ ] `JitteredScheduler.scheduleMultiOffering()` respects minSpacingMs between ticks
- [ ] Jitter provider is injectable for deterministic testing
- [ ] Negative jitter values are clamped to 0
- [ ] Zero maxJitterMs produces immediate execution (no delay)

## Integration Tests
- [ ] 50 simultaneous offering ticks with 5s maxJitter complete within maxJitterMs + tolerance
- [ ] No two ticks fire within minSpacingMs of each other
- [ ] Database connection pool never exceeds max during jittered batch

## Stress Tests
- [ ] 200 offerings at same cron tick complete without connection exhaustion
- [ ] Jittered scheduler recovers from handler failure without affecting other ticks
Loading