test(0293): M3 load-test evidence, 500 req/s row, the box ceiling - #326
Merged
stkrolikiewicz merged 11 commits intoSep 18, 2026
Merged
stkrolikiewicz merged 11 commits into
stkrolikiewicz merged 11 commits into
Conversation
Regime 3 held after the quota fix: 29,968 requests, 0 errors, gateway p95 74 ms. k6 on the laptop saw p95 464 ms; the difference is the client's network, so the in-region run in 0293 supplies the reportable number. Four open items closed with pointers.
…ence 500 and 1000 req/s are informational; the only M3 bar is AC 5, p95 < 100 ms at 100 req/s, reported as p95. The cache argument is stated against production's real hit rate (~4 % over 2026-09-04 → 09-16).
Regime 3 from the laptop, bracketed by two cache controls: k6 p95 127 ms against 464 the day before, gateway ~80 — the earlier tail was the client network. Not an evidence run (stale pool, backfill running). A ~4 % slow mode of ~+60 ms sits between Lambda and ClickHouse; the database is not it.
…pool in-chain /price serves only assets with a 1m candle in the last 24 h, so an asset can leave the pool after setup() probed it. On 2026-09-18 that was 67 of 30,001 requests: http_req_failed crossed 0.1 % and k6 exited 99 with zero server errors. A 404 is now counted in aged_out (own bar, rate<0.01) and excluded from the body checks; http_req_failed means 5xx/429/transport. gen_pool.mjs replaces the README one-liner so the wide regime rebuilds its pool in the same command chain — a 20 h old pool had lost 38 %.
… claim It starts 2026-09-21 (est. 22–28 days). The every-other-minute slow queries are therefore not the backfill, and the days before Monday are the last window to measure the box without it.
…de pool 2026-09-18 10:40–10:57 UTC, laptop, plan i12bsj at 150 req/s. AC scenario (17 assets) p95 49.0 ms; wide pool (3,463 assets, regenerated in-chain) p95 129.9 ms, aged_out 0.06 %; controls before/after p95 64.9 / 51.3 ms. All four runs exit 0, zero failed requests.
…ove 100 req/s VARIANTS multiplies the cache keys through min_volume_usd, which the handler applies in memory after the same ClickHouse query — the pool alone (~3,400 assets) cannot defeat a 10 s TTL at 500 or 1000 req/s. The warm-up climbs from 100 to RATE so the shared box meets the load gradually. Flat and identical to before at RATE=100, VARIANTS=1.
…d requests 2026-09-18 11:49–11:55 UTC: 149,880 requests at 500.00/s, 0 failed, aged_out 0.03 %, 14,056 cache keys (4 variants), ~2.6 % hits at the head of main. Gateway p95 87–95 ms, ClickHouse median 6 ms, box load1 ~11 (peak 19.7) of 24 cores, explorer ingestion unaffected.
iterationInTest restarts at 0 in main and replayed variant 0, which the warm-up's last pass had cached seconds earlier — 3,833 gateway hits at the head of the 500 req/s run.
… all four runs The ramp to 1000 req/s found the ceiling: ClickHouse collapsed ~30 s after the ramp reached ~900–1000 req/s (load1 247 on 24 cores, no exceptions), aborted after 1.5 min. All 14,865 failed requests are Lambda throttles at the temporary 700 reserved-concurrency cap on the API handler. Observer logs: diagnostic, evidence, 500 and 1000 runs — ClickHouse, explorer ingestion, box load and gateway errors every 30 s.
…eiling AC scenario p95 49.0 ms at 100 req/s (98.3 % hits, plan named) — Tranche 3 AC 5 met on the scenario it names; miss-only p95 129.9 ms from Poland, 45–90 at the gateway. 500 req/s of misses held with zero failures. The ramp to 1000 found the shared ClickHouse box's ceiling between 500 and ~900 req/s; collateral on the explorer's indexer recorded, including the alarm the first post-run check missed. 0047 gets its first measured data point.
stkrolikiewicz
added a commit
that referenced
this pull request
Sep 18, 2026
PR #326 merged. 7 of 8 criteria met; the in-region client was dropped by decision. AC scenario p95 49.0 ms at 100 req/s; 500 req/s of misses held; the shared ClickHouse box saturates between 500 and ~900 req/s.
stkrolikiewicz
deleted the
test/0293_load-test-ramp-to-1000-rps-from-inside-eu-central-1
branch
September 18, 2026 13:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs/prices-api-load-test-100rps.md): the 2026-09-17 regime-3 re-run after the ClickHouse quota fix, and the 2026-09-18 evidence run — Tranche 3 AC 5 met on the scenario it names (AC scenario p95 49.0 ms at 100 req/s, 98.3 % cache hits, usage plan named), with the miss-only row beside it (p95 129.9 ms from Poland, 45–90 ms at the gateway).production-ingestion-backlog-agepage is recorded, as is the post-run alarm check that missed it.aged_outinstead of failed requests,gen_pool.mjsto rebuild the wide pool in the run's own command chain,VARIANTScache-key multiplier viamin_volume_usd, and a ramped warm-up for rates above 100 req/s; README updated.