diff --git a/docs/runbooks/manual-api-key-tier.md b/docs/runbooks/manual-api-key-tier.md index f1fcc16f..5250fb50 100644 --- a/docs/runbooks/manual-api-key-tier.md +++ b/docs/runbooks/manual-api-key-tier.md @@ -357,9 +357,9 @@ Then delete the row from the table below. Keep this current. One row per **key**, so a customer mid-rotation has two; delete a row when its key is deleted, and the last one when the plan goes. -| Customer | Plan name | Plan ID | Key name | Key ID | Limits | Issued | Issued by | -| ------------ | --------- | ------- | -------- | ------ | ------ | ------ | --------- | -| _(none yet)_ | | | | | | | | +| Customer | Plan name | Plan ID | Key name | Key ID | Limits | Issued | Issued by | +| ------------------------------ | --------------------------------- | -------- | ------------------------------------------------- | ------------ | ------------------------------------- | ---------- | -------------- | +| loadtest (internal, task 0121) | `prices-production-loadtest-plan` | `i12bsj` | `prices-production-loadtest-key-20260819T114230Z` | `lxrwlyhjm7` | 150 req/s, burst 300, 1,000,000/month | 2026-08-19 | stkrolikiewicz | **Key name** is a column because key names are not unique and now carry the issuing instant — during a rotation two rows may share a customer, and the name is what diff --git a/lore/1-tasks/active/0121_TEST_api-load-test-100rps-report.md b/lore/1-tasks/active/0121_TEST_api-load-test-100rps-report.md index 429fdff0..2358678d 100644 --- a/lore/1-tasks/active/0121_TEST_api-load-test-100rps-report.md +++ b/lore/1-tasks/active/0121_TEST_api-load-test-100rps-report.md @@ -25,6 +25,35 @@ history: note: > Promoted to active — starting the load-test pass. Uses the 20-asset list fixed by [[0120]] (PR #226) for the spread scenario. + - date: 2026-08-20 + status: active + who: stkrolikiewicz + note: > + Script and plan done; the run itself still needs a BE window. Plan + `prices-production-loadtest-plan` (`i12bsj`, 150 req/s / burst 300 / + 1M mies.) + key issued and **registered** in + `docs/runbooks/manual-api-key-tier.md` — note it sits under the stage + ceiling (200/400), so unlike soroban-block-explorer's harness we need + no infra flag to lift a throttle. `price_load.js` extended rather than + rewritten: warmup phase excluded from thresholds (Lambda cold starts + would otherwise pollute p95 over 30k samples), asset pool from 0120's + list, `X-Request-Id` for a CH `log_comment` join, non-200 counted as + failure, and `dropped_iterations` as a threshold so a run that failed + to sustain the rate cannot be reported as one that did. + **Three measured findings shaping the method:** (1) there is NO + `X-Cache` header, so hit/miss percentiles cannot be tagged per + request — and the gateway cache key is the PATH ONLY, so pool size is + the only lever on hit rate; over 300 s an asset misses at most 30 + times, making 1/20/1000+ assets the ~0 %/2 %/100 % miss regimes. + Report each number with its regime. (2) Canonical USDC answers 404 + ([[0178]]), which puts a permanent 5 % floor under the error rate with + the 20-asset pool — the AC's 0.1 % would be unreachable for a reason + that has nothing to do with load. `setup()` now probes the pool, drops + unservable assets and names them, so the exclusion lands in the report + instead of vanishing. (3) Smoke run against prod (3 req/s, 19 assets, + mostly cache misses): p95 **83 ms**, zero failures — an early signal + that the 200 ms AC is comfortable, though not a substitute for the + 5-minute run. --- # Load test — 100 req/s on `GET /assets/{id}/price` diff --git a/packages/prices-api/loadtest/README.md b/packages/prices-api/loadtest/README.md index c2e8d74c..aa9fae59 100644 --- a/packages/prices-api/loadtest/README.md +++ b/packages/prices-api/loadtest/README.md @@ -21,17 +21,19 @@ k6 run packages/prices-api/loadtest/price_load.js \ Deploy-gated — requires the stack deployed (Phase 4 CDK) and a price row written by the ingest/current-prices path. -> **The key matters, and no key that exists today will do.** Task 0157 caps the +> **The key matters, and the default plan will not do.** Task 0157 caps the > CDK-managed plan (`pricing-api-free-production`) at **1 req/s with a 100 000/month > quota**. Running this script against a key on that plan does not measure the > system — it measures our own throttle, and reports a configuration artefact as > an SLO result. > -> A 100 req/s run for 5 minutes is 30 000 requests, so even a generous monthly -> quota is a real constraint if the test is repeated. Provision a dedicated -> throttle-and-quota-headroom plan for the run (see -> `docs/runbooks/manual-api-key-tier.md`), and state in the report which plan the -> key was on. Tracked in task 0121. +> **Provisioned for this task (0121):** plan `prices-production-loadtest-plan` +> (`i12bsj`) at **150 req/s, burst 300, 1 M/month**, key +> `prices-production-loadtest-key-20260819T114230Z` — see the registry table in +> `docs/runbooks/manual-api-key-tier.md`. The stage ceiling +> (`apiGatewayThrottleRate: 200`, burst 400) sits above the plan, so 100 req/s +> needs no CDK change. **The report must name the plan the key was on.** +> Wind the plan down after the milestone run; it is drift until then. ## Approximate run — local server @@ -63,10 +65,68 @@ k6 run packages/prices-api/loadtest/price_load.js \ | var | default | meaning | |-----|---------|---------| | `BASE_URL` | `http://localhost:8080` | API base (include `/` for the gateway) | -| `ASSET` | `native` | asset identifier under test | +| `ASSET` | (unset) | pin a single asset — the cache-dominated regime | +| `ASSETS` | 0120's 20-asset list | path to a JSON id pool (ignored when `ASSET` is set) | | `API_KEY` | (none) | sent as `x-api-key` (required against the gateway) | | `RATE` | `100` | requests/second | -| `DURATION` | `5m` | sustained duration | +| `WARMUP` | `30s` | low-rate phase before the measured window, excluded from thresholds | +| `DURATION` | `5m` | sustained measured duration | | `VUS` / `MAX_VUS` | `50` / `200` | pre-allocated / max virtual users | Smoke first with `-e RATE=20 -e DURATION=20s` before the full 5-minute run. + +## Which regime to run, and why it matters more than the knobs + +The gateway caches `/price` for 10 s **keyed on the path only**, so the pool size +is the only lever on the hit rate — no query parameter busts it. Over 300 s an +asset can miss at most 30 times, which fixes the arithmetic: + +| pool | run with | max misses of 30 k | the p95 is really measuring | +|------|----------|--------------------|------------------------------| +| 1 asset | `-e ASSET=native` | ~30 (0.1 %) | the gateway cache | +| 20 assets (default) | *(nothing — it is the default)* | ~600 (2 %) | the AC scenario, still cache-dominated | +| 1000+ assets | `-e ASSETS=/path/pool.json` | 30 000 (100 %) | the real data path | + +There is **no `X-Cache` header** on this API (verified 2026-08-20), so hit and +miss percentiles cannot be tagged per request. Run the regimes separately and +label each number with the regime it came from. Uniform sampling never warms a +hot key the way real traffic would, so the wide pool is a **worst case**, not a +typical one — report it as such rather than inventing a traffic distribution. + +Generate a wide pool by walking the listing (any key works; it is one page per +200 assets): + +```sh +node -e 'const f=async()=>{let c=null,out=[];do{const u=new URL(process.env.BASE_URL+"/v1/assets");u.searchParams.set("limit","200");if(c)u.searchParams.set("cursor",c);const r=await fetch(u,{headers:{"x-api-key":process.env.API_KEY}});const j=await r.json();out.push(...j.data.map(a=>a.contract_address||`${a.asset_code}:${a.issuer_address}`));c=j.cursor;await new Promise(s=>setTimeout(s,1100));}while(c);console.log(JSON.stringify(out));};f()' > /tmp/pool.json +``` + +## Before you run against production + +The read path lands on the ClickHouse box **shared with soroban-block-explorer**, +and their load-test runbook records that our own OHLCV batch is bursty enough to +double their p95 on its own — the same is true in reverse. Two rules, both +borrowed from their harness: + +1. **Check the box is quiet first**, and again after. A contaminated run cannot be + corrected after the fact; discard and re-run it, and name the discarded run in + the report rather than quietly dropping it. +2. **Schedule the window away from our own OHLCV batch**, and tell BE before you + start. 100 req/s of cache misses is real load on infrastructure another team + depends on. + +## Reading the result + +`dropped_iterations` is a threshold, not a statistic: any dropped iteration means +k6 could not keep the offered rate, so the run did **not** sustain 100 req/s and +its p95 is not the AC's number. Re-run with more `MAX_VUS`. + +For the error-rate half, sample size is the whole argument: 30 000 requests with +zero errors puts the 95 % upper bound at 3/30000 = **0.01 %**, a 10× margin under +the 0.1 % AC (rule of three). Say that explicitly — it is the cleanest claim in +the report, and it only holds because all 30 000 samples are on the one endpoint +under test. + +Cold starts are not in k6's output. Read `InitDuration` and `ConcurrentExecutions` +from Lambda, plus API Gateway `Latency` vs `IntegrationLatency`, over the +`phase:main` window — that split is what makes a "where did the p95 go" table +writable. diff --git a/packages/prices-api/loadtest/price_load.js b/packages/prices-api/loadtest/price_load.js index 8065cc9b..14d52195 100644 --- a/packages/prices-api/loadtest/price_load.js +++ b/packages/prices-api/loadtest/price_load.js @@ -1,4 +1,4 @@ -// k6 load test for the §9 SLO: +// k6 load test for the §9 SLO and Tranche 2 AC 2 (task 0121): // 100 req/s sustained for 5 minutes on GET /assets/{id}/price // → p95 latency < 200 ms, error rate < 0.1% // @@ -6,47 +6,153 @@ // — no gateway cache / no Lambda cold start, so a lower bound on prod p95): // // k6 run packages/prices-api/loadtest/price_load.js \ -// -e BASE_URL=https:/// -e API_KEY= -e ASSET=native +// -e BASE_URL=https:/// -e API_KEY= // -// Knobs (env): RATE (req/s, default 100), DURATION (default 5m), ASSET, +// Knobs (env): RATE (req/s, default 100), DURATION (default 5m), WARMUP +// (default 30s), ASSET (single-asset mode), ASSETS (path to an id pool), // API_KEY, VUS, MAX_VUS. k6 exits non-zero if a threshold is breached. +// +// ── Why the asset pool decides what you are measuring ─────────────────────── +// The gateway caches /price for 10 s keyed on the PATH ONLY (`addGet(price, +// [PATH_ID])` in api-gateway-stack.ts) — no query parameter can bust it, so the +// number of DISTINCT assets in the pool is the only lever on the hit rate. +// Over a 300 s run each asset can miss at most 300/10 = 30 times, so: +// +// pool size | max misses | of 30k requests | what the p95 mostly measures +// ----------|--------------|-----------------|------------------------------ +// 1 (hot) | 30 | 0.1 % | the gateway cache +// 20 (spread)| 600 | 2 % | the AC scenario, cache-dominated +// 1000+(wide)| ≥30,000 | 100 % | the real data path (CH round trip) +// +// There is no `X-Cache` header on this API (verified 2026-08-20), so hit/miss +// percentiles cannot be tagged per request — run the regimes separately and +// report which one each number came from. Uniform sampling across the pool is +// the conservative choice: it never warms a hot key the way real traffic would, +// so `wide` is a worst case rather than a typical one. Say so in the report +// instead of inventing a traffic distribution. +// +// ── Warmup ───────────────────────────────────────────────────────────────── +// A step-start at 100 req/s pays Lambda cold starts in the first seconds, which +// visibly pollutes p95 over 30k samples. The run therefore opens with a low-rate +// warmup phase, tagged `phase:warmup` and EXCLUDED from every threshold. Cite +// the `phase:main` window; read cold-start incidence from Lambda's InitDuration +// in CloudWatch for the same window. import http from 'k6/http'; import { check } from 'k6'; +import exec from 'k6/execution'; const BASE_URL = __ENV.BASE_URL || 'http://localhost:8080'; -const ASSET = __ENV.ASSET || 'native'; const API_KEY = __ENV.API_KEY || ''; const RATE = Number(__ENV.RATE || 100); +const WARMUP = __ENV.WARMUP || '30s'; +const DURATION = __ENV.DURATION || '5m'; + +// Asset pool. ASSET pins a single id (cache-dominated); otherwise the pool is +// read from a JSON file — the 20-asset conformance list by default, so 0121 and +// 0120 measure the same assets. +function loadPool(path) { + const raw = JSON.parse(open(path)); + const list = Array.isArray(raw) ? raw : raw.assets; + return list.map((a) => (typeof a === 'string' ? a : a.id)); +} +const POOL = __ENV.ASSET + ? [__ENV.ASSET] + : loadPool(__ENV.ASSETS || '../../../tools/scripts/conformance-assets.json'); export const options = { scenarios: { - price: { + // Excluded from thresholds — its job is to have containers already warm. + warmup: { + executor: 'constant-arrival-rate', + rate: Math.max(1, Math.round(RATE / 10)), + timeUnit: '1s', + duration: WARMUP, + preAllocatedVUs: Number(__ENV.VUS || 50), + maxVUs: Number(__ENV.MAX_VUS || 200), + tags: { phase: 'warmup' }, + }, + main: { executor: 'constant-arrival-rate', rate: RATE, timeUnit: '1s', - duration: __ENV.DURATION || '5m', + duration: DURATION, + startTime: WARMUP, preAllocatedVUs: Number(__ENV.VUS || 50), maxVUs: Number(__ENV.MAX_VUS || 200), + tags: { phase: 'main' }, }, }, thresholds: { - // p95 < 200 ms. - http_req_duration: ['p(95)<200'], - // error rate < 0.1%. - http_req_failed: ['rate<0.001'], + 'http_req_duration{phase:main}': ['p(95)<200'], + 'http_req_failed{phase:main}': ['rate<0.001'], + // Dropped iterations mean k6 could not keep the offered rate — the run did + // NOT sustain 100 req/s and its p95 is not the AC's number. + dropped_iterations: ['count<1'], }, }; const PARAMS = { - headers: API_KEY ? { 'x-api-key': API_KEY } : {}, + headers: { + ...(API_KEY ? { 'x-api-key': API_KEY } : {}), + // Managed WAF rulesets 403 a missing User-Agent; k6 sends one, this pins it. + 'User-Agent': 'stellar-prices-api-loadtest/0121 (k6)', + }, tags: { endpoint: 'price' }, + // Anything other than 200 is a failure. The default (status < 400) would let + // a 204 or a challenge served as 2xx pass silently. + responseCallback: http.expectedStatuses(200), }; -export default function () { - const res = http.get(`${BASE_URL}/v1/assets/${ASSET}/price`, PARAMS); +// Probe the pool once and keep only assets the API can actually serve. An asset +// with no current-price row answers 404 forever, so leaving one in the pool puts +// a floor under the error rate that has nothing to do with load: one dead asset +// in twenty is a permanent 5 %, against an AC of 0.1 %. Measured 2026-08-20 — +// canonical USDC is exactly this case (task 0178), so the default pool needs the +// probe to be usable at all. Whatever it drops is printed: put the list in the +// report rather than letting it vanish. +export function setup() { + const live = []; + const dropped = []; + for (const asset of POOL) { + const res = http.get(`${BASE_URL}/v1/assets/${encodeURIComponent(asset)}/price`, { + ...PARAMS, + tags: { phase: 'probe' }, + }); + (res.status === 200 ? live : dropped).push(res.status === 200 ? asset : `${asset} → ${res.status}`); + } + if (dropped.length) { + console.warn(`pool: dropped ${dropped.length}/${POOL.length} unservable asset(s): ${dropped.join(', ')}`); + } + if (!live.length) { + exec.test.abort('pool: no asset answered 200 — nothing to measure'); + } + console.log(`pool: ${live.length} asset(s) under test`); + return { pool: live }; +} + +export default function (data) { + const pool = data.pool; + const asset = pool[exec.scenario.iterationInTest % pool.length]; + const params = { + ...PARAMS, + headers: { + ...PARAMS.headers, + // Stamped into ClickHouse system.query_log.log_comment when the API runs + // with request-id logging, so a slow request can be joined to its query. + 'X-Request-Id': `lt0121-${exec.scenario.iterationInTest}-${__VU}`, + }, + }; + const res = http.get(`${BASE_URL}/v1/assets/${encodeURIComponent(asset)}/price`, params); check(res, { 'status is 200': (r) => r.status === 200, - 'has price_usd': (r) => r.json('price_usd') !== undefined, + // A body-read failure must not pass as a slow 200. + 'body parses with price_usd': (r) => { + try { + return typeof r.json('price_usd') === 'string'; + } catch (_) { + return false; + } + }, }); }