test(0121): k6 script and load-test plan for the 100 req/s AC

docs/runbooks/manual-api-key-tier.md

@@ -357,9 +357,9 @@ Then delete the row from the table below.
 Keep this current. One row per **key**, so a customer mid-rotation has two;
 delete a row when its key is deleted, and the last one when the plan goes.
 
-| Customer     | Plan name | Plan ID | Key name | Key ID | Limits | Issued | Issued by |
-| ------------ | --------- | ------- | -------- | ------ | ------ | ------ | --------- |
-| _(none yet)_ |           |         |          |        |        |        |           |
+| Customer                       | Plan name                         | Plan ID  | Key name                                          | Key ID       | Limits                                | Issued     | Issued by      |
+| ------------------------------ | --------------------------------- | -------- | ------------------------------------------------- | ------------ | ------------------------------------- | ---------- | -------------- |
+| loadtest (internal, task 0121) | `prices-production-loadtest-plan` | `i12bsj` | `prices-production-loadtest-key-20260819T114230Z` | `lxrwlyhjm7` | 150 req/s, burst 300, 1,000,000/month | 2026-08-19 | stkrolikiewicz |
 
 **Key name** is a column because key names are not unique and now carry the issuing
 instant — during a rotation two rows may share a customer, and the name is what

lore/1-tasks/active/0121_TEST_api-load-test-100rps-report.md

@@ -25,6 +25,35 @@ history:
     note: >
       Promoted to active — starting the load-test pass. Uses the 20-asset
       list fixed by [[0120]] (PR #226) for the spread scenario.
+  - date: 2026-08-20
+    status: active
+    who: stkrolikiewicz
+    note: >
+      Script and plan done; the run itself still needs a BE window. Plan
+      `prices-production-loadtest-plan` (`i12bsj`, 150 req/s / burst 300 /
+      1M mies.) + key issued and **registered** in
+      `docs/runbooks/manual-api-key-tier.md` — note it sits under the stage
+      ceiling (200/400), so unlike soroban-block-explorer's harness we need
+      no infra flag to lift a throttle. `price_load.js` extended rather than
+      rewritten: warmup phase excluded from thresholds (Lambda cold starts
+      would otherwise pollute p95 over 30k samples), asset pool from 0120's
+      list, `X-Request-Id` for a CH `log_comment` join, non-200 counted as
+      failure, and `dropped_iterations` as a threshold so a run that failed
+      to sustain the rate cannot be reported as one that did.
+      **Three measured findings shaping the method:** (1) there is NO
+      `X-Cache` header, so hit/miss percentiles cannot be tagged per
+      request — and the gateway cache key is the PATH ONLY, so pool size is
+      the only lever on hit rate; over 300 s an asset misses at most 30
+      times, making 1/20/1000+ assets the ~0 %/2 %/100 % miss regimes.
+      Report each number with its regime. (2) Canonical USDC answers 404
+      ([[0178]]), which puts a permanent 5 % floor under the error rate with
+      the 20-asset pool — the AC's 0.1 % would be unreachable for a reason
+      that has nothing to do with load. `setup()` now probes the pool, drops
+      unservable assets and names them, so the exclusion lands in the report
+      instead of vanishing. (3) Smoke run against prod (3 req/s, 19 assets,
+      mostly cache misses): p95 **83 ms**, zero failures — an early signal
+      that the 200 ms AC is comfortable, though not a substitute for the
+      5-minute run.
 ---
 
 # Load test — 100 req/s on `GET /assets/{id}/price`

packages/prices-api/loadtest/README.md

@@ -21,17 +21,19 @@ k6 run packages/prices-api/loadtest/price_load.js \
 Deploy-gated — requires the stack deployed (Phase 4 CDK) and a price row written
 by the ingest/current-prices path.
 
-> **The key matters, and no key that exists today will do.** Task 0157 caps the
+> **The key matters, and the default plan will not do.** Task 0157 caps the
 > CDK-managed plan (`pricing-api-free-production`) at **1 req/s with a 100 000/month
 > quota**. Running this script against a key on that plan does not measure the
 > system — it measures our own throttle, and reports a configuration artefact as
 > an SLO result.
 >
-> A 100 req/s run for 5 minutes is 30 000 requests, so even a generous monthly
-> quota is a real constraint if the test is repeated. Provision a dedicated
-> throttle-and-quota-headroom plan for the run (see
-> `docs/runbooks/manual-api-key-tier.md`), and state in the report which plan the
-> key was on. Tracked in task 0121.
+> **Provisioned for this task (0121):** plan `prices-production-loadtest-plan`
+> (`i12bsj`) at **150 req/s, burst 300, 1 M/month**, key
+> `prices-production-loadtest-key-20260819T114230Z` — see the registry table in
+> `docs/runbooks/manual-api-key-tier.md`. The stage ceiling
+> (`apiGatewayThrottleRate: 200`, burst 400) sits above the plan, so 100 req/s
+> needs no CDK change. **The report must name the plan the key was on.**
+> Wind the plan down after the milestone run; it is drift until then.
 
 ## Approximate run — local server
 
@@ -63,10 +65,68 @@ k6 run packages/prices-api/loadtest/price_load.js \
 | var | default | meaning |
 |-----|---------|---------|
 | `BASE_URL` | `http://localhost:8080` | API base (include `/<stage>` for the gateway) |
-| `ASSET` | `native` | asset identifier under test |
+| `ASSET` | (unset) | pin a single asset — the cache-dominated regime |
+| `ASSETS` | 0120's 20-asset list | path to a JSON id pool (ignored when `ASSET` is set) |
 | `API_KEY` | (none) | sent as `x-api-key` (required against the gateway) |
 | `RATE` | `100` | requests/second |
-| `DURATION` | `5m` | sustained duration |
+| `WARMUP` | `30s` | low-rate phase before the measured window, excluded from thresholds |
+| `DURATION` | `5m` | sustained measured duration |
 | `VUS` / `MAX_VUS` | `50` / `200` | pre-allocated / max virtual users |
 
 Smoke first with `-e RATE=20 -e DURATION=20s` before the full 5-minute run.
+
+## Which regime to run, and why it matters more than the knobs
+
+The gateway caches `/price` for 10 s **keyed on the path only**, so the pool size
+is the only lever on the hit rate — no query parameter busts it. Over 300 s an
+asset can miss at most 30 times, which fixes the arithmetic:
+
+| pool | run with | max misses of 30 k | the p95 is really measuring |
+|------|----------|--------------------|------------------------------|
+| 1 asset | `-e ASSET=native` | ~30 (0.1 %) | the gateway cache |
+| 20 assets (default) | *(nothing — it is the default)* | ~600 (2 %) | the AC scenario, still cache-dominated |
+| 1000+ assets | `-e ASSETS=/path/pool.json` | 30 000 (100 %) | the real data path |
+
+There is **no `X-Cache` header** on this API (verified 2026-08-20), so hit and
+miss percentiles cannot be tagged per request. Run the regimes separately and
+label each number with the regime it came from. Uniform sampling never warms a
+hot key the way real traffic would, so the wide pool is a **worst case**, not a
+typical one — report it as such rather than inventing a traffic distribution.
+
+Generate a wide pool by walking the listing (any key works; it is one page per
+200 assets):
+
+```sh
+node -e 'const f=async()=>{let c=null,out=[];do{const u=new URL(process.env.BASE_URL+"/v1/assets");u.searchParams.set("limit","200");if(c)u.searchParams.set("cursor",c);const r=await fetch(u,{headers:{"x-api-key":process.env.API_KEY}});const j=await r.json();out.push(...j.data.map(a=>a.contract_address||`${a.asset_code}:${a.issuer_address}`));c=j.cursor;await new Promise(s=>setTimeout(s,1100));}while(c);console.log(JSON.stringify(out));};f()' > /tmp/pool.json
+```
+
+## Before you run against production
+
+The read path lands on the ClickHouse box **shared with soroban-block-explorer**,
+and their load-test runbook records that our own OHLCV batch is bursty enough to
+double their p95 on its own — the same is true in reverse. Two rules, both
+borrowed from their harness:
+
+1. **Check the box is quiet first**, and again after. A contaminated run cannot be
+   corrected after the fact; discard and re-run it, and name the discarded run in
+   the report rather than quietly dropping it.
+2. **Schedule the window away from our own OHLCV batch**, and tell BE before you
+   start. 100 req/s of cache misses is real load on infrastructure another team
+   depends on.
+
+## Reading the result
+
+`dropped_iterations` is a threshold, not a statistic: any dropped iteration means
+k6 could not keep the offered rate, so the run did **not** sustain 100 req/s and
+its p95 is not the AC's number. Re-run with more `MAX_VUS`.
+
+For the error-rate half, sample size is the whole argument: 30 000 requests with
+zero errors puts the 95 % upper bound at 3/30000 = **0.01 %**, a 10× margin under
+the 0.1 % AC (rule of three). Say that explicitly — it is the cleanest claim in
+the report, and it only holds because all 30 000 samples are on the one endpoint
+under test.
+
+Cold starts are not in k6's output. Read `InitDuration` and `ConcurrentExecutions`
+from Lambda, plus API Gateway `Latency` vs `IntegrationLatency`, over the
+`phase:main` window — that split is what makes a "where did the p95 go" table
+writable.

packages/prices-api/loadtest/price_load.js

@@ -1,52 +1,158 @@
-// k6 load test for the §9 SLO:
+// k6 load test for the §9 SLO and Tranche 2 AC 2 (task 0121):
 //   100 req/s sustained for 5 minutes on GET /assets/{id}/price
 //   → p95 latency < 200 ms, error rate < 0.1%
 //
 // Run against a deployed stage (authoritative) or the local server (approximate
 // — no gateway cache / no Lambda cold start, so a lower bound on prod p95):
 //
 //   k6 run packages/prices-api/loadtest/price_load.js \
-//     -e BASE_URL=https://<api>/<stage> -e API_KEY=<key> -e ASSET=native
+//     -e BASE_URL=https://<api>/<stage> -e API_KEY=<key>
 //
-// Knobs (env): RATE (req/s, default 100), DURATION (default 5m), ASSET,
+// Knobs (env): RATE (req/s, default 100), DURATION (default 5m), WARMUP
+// (default 30s), ASSET (single-asset mode), ASSETS (path to an id pool),
 // API_KEY, VUS, MAX_VUS. k6 exits non-zero if a threshold is breached.
+//
+// ── Why the asset pool decides what you are measuring ───────────────────────
+// The gateway caches /price for 10 s keyed on the PATH ONLY (`addGet(price,
+// [PATH_ID])` in api-gateway-stack.ts) — no query parameter can bust it, so the
+// number of DISTINCT assets in the pool is the only lever on the hit rate.
+// Over a 300 s run each asset can miss at most 300/10 = 30 times, so:
+//
+//   pool size |  max misses  | of 30k requests | what the p95 mostly measures
+//   ----------|--------------|-----------------|------------------------------
+//   1  (hot)  |          30  |          0.1 %  | the gateway cache
+//   20 (spread)|        600  |          2 %    | the AC scenario, cache-dominated
+//   1000+(wide)|     ≥30,000 |        100 %    | the real data path (CH round trip)
+//
+// There is no `X-Cache` header on this API (verified 2026-08-20), so hit/miss
+// percentiles cannot be tagged per request — run the regimes separately and
+// report which one each number came from. Uniform sampling across the pool is
+// the conservative choice: it never warms a hot key the way real traffic would,
+// so `wide` is a worst case rather than a typical one. Say so in the report
+// instead of inventing a traffic distribution.
+//
+// ── Warmup ─────────────────────────────────────────────────────────────────
+// A step-start at 100 req/s pays Lambda cold starts in the first seconds, which
+// visibly pollutes p95 over 30k samples. The run therefore opens with a low-rate
+// warmup phase, tagged `phase:warmup` and EXCLUDED from every threshold. Cite
+// the `phase:main` window; read cold-start incidence from Lambda's InitDuration
+// in CloudWatch for the same window.
 
 import http from 'k6/http';
 import { check } from 'k6';
+import exec from 'k6/execution';
 
 const BASE_URL = __ENV.BASE_URL || 'http://localhost:8080';
-const ASSET = __ENV.ASSET || 'native';
 const API_KEY = __ENV.API_KEY || '';
 const RATE = Number(__ENV.RATE || 100);
+const WARMUP = __ENV.WARMUP || '30s';
+const DURATION = __ENV.DURATION || '5m';
+
+// Asset pool. ASSET pins a single id (cache-dominated); otherwise the pool is
+// read from a JSON file — the 20-asset conformance list by default, so 0121 and
+// 0120 measure the same assets.
+function loadPool(path) {
+  const raw = JSON.parse(open(path));
+  const list = Array.isArray(raw) ? raw : raw.assets;
+  return list.map((a) => (typeof a === 'string' ? a : a.id));
+}
+const POOL = __ENV.ASSET
+  ? [__ENV.ASSET]
+  : loadPool(__ENV.ASSETS || '../../../tools/scripts/conformance-assets.json');
 
 export const options = {
   scenarios: {
-    price: {
+    // Excluded from thresholds — its job is to have containers already warm.
+    warmup: {
+      executor: 'constant-arrival-rate',
+      rate: Math.max(1, Math.round(RATE / 10)),
+      timeUnit: '1s',
+      duration: WARMUP,
+      preAllocatedVUs: Number(__ENV.VUS || 50),
+      maxVUs: Number(__ENV.MAX_VUS || 200),
+      tags: { phase: 'warmup' },
+    },
+    main: {
       executor: 'constant-arrival-rate',
       rate: RATE,
       timeUnit: '1s',
-      duration: __ENV.DURATION || '5m',
+      duration: DURATION,
+      startTime: WARMUP,
       preAllocatedVUs: Number(__ENV.VUS || 50),
       maxVUs: Number(__ENV.MAX_VUS || 200),
+      tags: { phase: 'main' },
     },
   },
   thresholds: {
-    // p95 < 200 ms.
-    http_req_duration: ['p(95)<200'],
-    // error rate < 0.1%.
-    http_req_failed: ['rate<0.001'],
+    'http_req_duration{phase:main}': ['p(95)<200'],
+    'http_req_failed{phase:main}': ['rate<0.001'],
+    // Dropped iterations mean k6 could not keep the offered rate — the run did
+    // NOT sustain 100 req/s and its p95 is not the AC's number.
+    dropped_iterations: ['count<1'],
   },
 };
 
 const PARAMS = {
-  headers: API_KEY ? { 'x-api-key': API_KEY } : {},
+  headers: {
+    ...(API_KEY ? { 'x-api-key': API_KEY } : {}),
+    // Managed WAF rulesets 403 a missing User-Agent; k6 sends one, this pins it.
+    'User-Agent': 'stellar-prices-api-loadtest/0121 (k6)',
+  },
   tags: { endpoint: 'price' },
+  // Anything other than 200 is a failure. The default (status < 400) would let
+  // a 204 or a challenge served as 2xx pass silently.
+  responseCallback: http.expectedStatuses(200),
 };
 
-export default function () {
-  const res = http.get(`${BASE_URL}/v1/assets/${ASSET}/price`, PARAMS);
+// Probe the pool once and keep only assets the API can actually serve. An asset
+// with no current-price row answers 404 forever, so leaving one in the pool puts
+// a floor under the error rate that has nothing to do with load: one dead asset
+// in twenty is a permanent 5 %, against an AC of 0.1 %. Measured 2026-08-20 —
+// canonical USDC is exactly this case (task 0178), so the default pool needs the
+// probe to be usable at all. Whatever it drops is printed: put the list in the
+// report rather than letting it vanish.
+export function setup() {
+  const live = [];
+  const dropped = [];
+  for (const asset of POOL) {
+    const res = http.get(`${BASE_URL}/v1/assets/${encodeURIComponent(asset)}/price`, {
+      ...PARAMS,
+      tags: { phase: 'probe' },
+    });
+    (res.status === 200 ? live : dropped).push(res.status === 200 ? asset : `${asset} → ${res.status}`);
+  }
+  if (dropped.length) {
+    console.warn(`pool: dropped ${dropped.length}/${POOL.length} unservable asset(s): ${dropped.join(', ')}`);
+  }
+  if (!live.length) {
+    exec.test.abort('pool: no asset answered 200 — nothing to measure');
+  }
+  console.log(`pool: ${live.length} asset(s) under test`);
+  return { pool: live };
+}
+
+export default function (data) {
+  const pool = data.pool;
+  const asset = pool[exec.scenario.iterationInTest % pool.length];
+  const params = {
+    ...PARAMS,
+    headers: {
+      ...PARAMS.headers,
+      // Stamped into ClickHouse system.query_log.log_comment when the API runs
+      // with request-id logging, so a slow request can be joined to its query.
+      'X-Request-Id': `lt0121-${exec.scenario.iterationInTest}-${__VU}`,
+    },
+  };
+  const res = http.get(`${BASE_URL}/v1/assets/${encodeURIComponent(asset)}/price`, params);
   check(res, {
     'status is 200': (r) => r.status === 200,
-    'has price_usd': (r) => r.json('price_usd') !== undefined,
+    // A body-read failure must not pass as a slow 200.
+    'body parses with price_usd': (r) => {
+      try {
+        return typeof r.json('price_usd') === 'string';
+      } catch (_) {
+        return false;
+      }
+    },
   });
 }