Skip to content

feat(0119): reject every invalid input with a 400 envelope - #217

Merged
stkrolikiewicz merged 9 commits into
developfrom
feat/0119_api-input-validation-hardening
Aug 18, 2026
Merged

feat(0119): reject every invalid input with a 400 envelope#217
stkrolikiewicz merged 9 commits into
developfrom
feat/0119_api-input-validation-hardening

Conversation

@stkrolikiewicz

Copy link
Copy Markdown
Collaborator

Summary

  • Custom ValidatedQuery/ValidatedJson/ValidatedPath extractors route every axum rejection through the standard ErrorEnvelope — the old text/plain 400/415/422 bodies are gone; 400s now carry Cache-Control: no-store, and POST /prices/batch gets an explicit 16 KB body limit
  • Query params are typed serde enums (the six stringly parse_* fns deleted): unknown tokens 400 with the valid values enumerated, search is capped at 1–12 ASCII alphanumerics, and the same enums publish into OpenAPI (contract-tested — the test caught dangling $refs utoipa-axum does not auto-register)
  • Pagination cursor: deny_unknown_fields, 256-char token cap, and payload type-checked against the active sort — closes a client-reachable 500 (corrupt v reaching ClickHouse toFloat64)
  • OHLCV: real date parsing (chrono, already in the build graph) with validated epochs bound via toDateTime(?), plus an explicit window rule (start < end, ≤5000 points at the chosen granularity) replacing silent truncation; the timeframe window anchors to end when only end is given
  • 43 new CH-less negative tests drive the real router with AppState::without_ch() (panics on any CH access, proving validation precedes every query) and run in the existing CI cargo test step; prices-api added to the clippy -D warnings gate; all 21 live-CH integration tests pass against the prod-pinned 26.3.10.60

Task: lore/1-tasks/active/0119_FEATURE_api-input-validation-hardening.md (param → rule → error-code table + recorded policies)

Axum's own Query/Json/Path rejections answered in text/plain with 415/422
for body failures, bypassing the ErrorEnvelope contract. New wrapper
extractors (ValidatedQuery/ValidatedJson/ValidatedPath) map every
rejection to 400 + envelope; new invalid_body code; bad_request now
carries Cache-Control: no-store; /prices/batch gets an explicit 16 KB
body limit so an oversized body is refused before parsing. CH-less
negative tests prove each rejection fires before any ClickHouse call.
… fns

The six param enums (SortCol, Order, TypeFilter, Timeframe, Granularity,
BaseCurrency) now derive Deserialize + ToSchema with explicit per-variant
renames, so params deserialize straight into typed forms: an unknown token
fails serde with a message enumerating the valid values (400 envelope via
ValidatedQuery), and the same enums publish into OpenAPI. Deletes the six
stringly parse() fns. Case policy: exact documented tokens; base_currency
keeps lowercase usd/xlm aliases (historically case-insensitive). New:
search capped at 1-12 ASCII alphanumeric (SEP-11 alphanum12 prefix).
Cursor: deny_unknown_fields (foreign lookalikes 400), 256-char token cap,
and valid_for(sort) — a numeric sort requires a finite-parseable v, which
previously reached ClickHouse toFloat64() and threw a 500 from client
input; code sorts require an asset-code shape.

OHLCV: parse_time (chrono; already in the build graph transitively)
replaces the shape-only valid_iso8601, rejecting impossible dates that CH
would have interpreted freely; the validated epoch (not the raw string)
is bound via toDateTime(?) so exactly one interpretation of the window
exists. New window rule before any DB call: start < end, and
ceil(span/granularity) <= OHLCV_MAX_POINTS with a message naming the
count — replacing the silent newest-5000 truncation. The timeframe
window anchors to end when only end is given (?end=..&timeframe=7d is
the 7d window ending there); timeframe=all starts at Stellar genesis.
…s-api

Registers the six query-param enums in ApiDoc components — utoipa-axum
collects body schemas from routes but a params-tuple $ref is NOT
auto-registered, so the served document carried dangling refs (caught by
the new enum-publication contract test). BatchRequest.assets publishes
minItems/maxItems mirroring MAX_BATCH; search publishes its 1-12 length
bounds. Task file gains the AC 1 param -> rule -> error-code table and
the recorded policies (unknown params ignored, exact-token case with the
base_currency lowercase exception, cursor sort-binding limitation,
CH-in-CI deliberately deferred to 0120/0122). CI clippy line now
includes prices-api.
…eframe cliff

Local-CH smoke caught the one positive-path change 0119 makes:
timeframe=all&granularity=1h is now an up-front 400 (genesis to now is
~95k hourly buckets against the 5000 cap), where the merge test leaned
on the old silent newest-5000 truncation. Narrowed with explicit
start/end around the seeded candles — merge + backfill_note coverage
unchanged. All 21 integration tests pass against the prod-pinned CH
26.3.10.60, confirming the toDateTime epoch binds. Task notes record
the consumer-visible change and the ~2029-06 cliff where bare
timeframe=all at 1d itself crosses 5000 buckets.
develop gained AppConfig.portal_enabled (task 0183, PR #207) while this
branch was in flight; the new tests/common harness constructs AppConfig
literally and did not compile against the merge. Off in tests — the
portal gate is irrelevant to the validation suite.
…e_time

/code-review (high) on the branch surfaced four correctness bugs and a
set of contract gaps; all fixed:

- sort=code cursors: the charset/non-empty rule rejected tokens the API
  itself issues (Soroban rows carry empty codes; lossy-decoded on-chain
  garbage exists), dead-ending pagination. String sorts now cap length
  only — no 500 risk exists on that path (plain string bind).
- end-anchored windows could derive a negative eff_start and bind it
  into toDateTime(?) unvalidated (CH throw -> 500). Clamped to epoch 0.
- fencepost: inclusive SQL bounds hold span/gran + 1 bucket starts, so
  an aligned 5000-bucket span was passed through and silently trimmed
  by LIMIT. Point count now +1.
- parse_time regressions vs the old pipeline: minute-precision offsets
  (11:30Z, 11:30+02:00) and colonless +0200 now parse; millis decided
  by digit count (13+) as documented, so 12-digit values reject.
- one window interpretation for real: the validated eff_start always
  binds into SQL; the parallel now() - INTERVAL path (calendar-aware
  1 YEAR vs 365d seconds() — divergent on leap years) is deleted along
  with Timeframe::interval() and OhlcvArgs.since_interval.
- spec honesty: base_currency documents the lowercase legacy aliases,
  search documents empty-as-absent, start/end document epoch forms;
  maxItems asserted against MAX_BATCH (const, not literal); search cap
  derives from identity::MAX_CODE_LEN.
- task notes record the full blast radius of the 5000-bucket 400s (not
  just timeframe=all), the UTC pinning of naive datetimes, and the
  end-anchoring flip for far-future ?end.
@karczuRF

Copy link
Copy Markdown
Collaborator

Review notes — input-validation hardening

Reviewed origin/develop...origin/feat/0119_api-input-validation-hardening (23 files, +1183/−219). CI green on all three jobs.

The plumbing looks correct: the ValidatedQuery/ValidatedJson/ValidatedPath extractors, the enum-ised param parsers, the cursor deny_unknown_fields + valid_for type check, and the epoch-clamping of start/end all hold up — no unsound SQL bind, no missing await, no dropped error path, no reachable panic.

Everything below is the same shape instead: a validation rule stricter than the data model it guards, so requests that worked before now return 400.


1. ?search rejects asset-code prefixes the API itself can address — packages/prices-api/src/assets/handlers.rs:199

The rule requires every byte of search to be is_ascii_alphanumeric(). But classic asset codes are 4/12 raw bytes off the ledger, decoded with String::from_utf8_lossy(...) in packages/prices-ingest-core/src/canonical.rs:27, and AssetIdentifier::parse (identity.rs:53) imposes no charset restriction on the code half of CODE:ISSUER.

This PR's own common/cursor.rs:22-26 comment argues the opposite for the same data:

the DB legitimately holds empty codes (Soroban rows) and lossy-decoded on-chain garbage, so the only safe rule here is a length cap — any charset restriction would 400 a cursor the API itself just issued.

An asset whose on-chain code contains ., -, a space or a replacement char is fetchable via GET /v1/assets/{code}:{issuer} and appears in GET /v1/assets, but ?search=<its prefix> now 400s instead of returning it. A length cap alone — matching the cursor decision — is the consistent rule.

2. start == end is rejected although the SQL bounds are inclusive — handlers.rs:367

if eff_start >= eff_end 400s a zero-span window, but the very next block reasons that a zero-span window is legitimate: points = ceil(span/g) + 1 deliberately counts one bucket for span == 0 ("the SQL bounds are inclusive on both ends").

?start=2026-06-15&end=2026-06-15&granularity=1d — the natural way to ask for one day's candle — returned that candle before this PR (>= midnight AND <= midnight) and now returns 400 start must be before end. The guard wants to be eff_start > eff_end.

3. Granularity still defaults from timeframe on an explicit window, so plain ?start= 400s — handlers.rs:322

granularity falls back to timeframe.default_granularity(), and timeframe falls back to H2415m, regardless of whether start/end were supplied.

GET /v1/assets/native/ohlcv?start=2020-01-01 (no timeframe, no granularity) computes ~231,000 points at 15m and returns 400 telling the caller to pick a coarser granularity — for a request that carried no granularity and no timeframe at all. Before this PR it returned the newest 5000 15m candles. When start/end are explicit, the auto-granularity should come from the actual window width, not from a timeframe the caller never asked for.

4. +HH:MM offsets are unreachable over HTTP, and replacen guarantees the failure — handlers.rs:528

parse_time's unit test asserts "2026-06-15T11:30:00+0200" and "2026-06-15T11:30+02:00" parse. In a real query string + decodes to a space, so the handler receives 2026-06-15T11:30:00 02:00; s.contains(' ') is then true and replacen(' ', "T", 1) rewrites it to 2026-06-15T11:30:00T02:00, which every parser below rejects → 400 invalid start.

So curl '.../ohlcv?start=2026-06-15T11:30:00+02:00' (or any client that does not percent-encode +) is refused for a value the docstring and OpenAPI description advertise as accepted. The unit tests pass the string straight to parse_time, so they can't catch this, and none of the 43 new router-level tests cover an offset form.

5. Issued cursors are not URL-safe, so echoing next_cursor verbatim can 400 — common/cursor.rs:54

encode uses base64::engine::general_purpose::STANDARD, which emits +, / and =. A client that appends the next_cursor from the response body to ?cursor= without percent-encoding turns + into a space, decode fails, and pagination dies with 400 invalid cursor mid-walk. tests/list.rs:98-105 works around exactly this (.replace('+', "%2B")), which is evidence the shape is already known. URL_SAFE_NO_PAD on both encode and decode removes the footgun; the 256-char cap and deny_unknown_fields added here don't address it.

6. The "start must be before end" message names a parameter the client never sent — handlers.rs:360

With ?start=2099-01-01 and no end, eff_end is now, the guard trips, and the caller is told start must be before end for a param they never supplied. tests/ohlcv.rs::ohlcv_future_start_without_end_is_400 locks in the status but not the message. A distinct "start is in the future" saves a support round-trip.

7. timeframe=all at its own default granularity will start returning 400 around 2029-06 — handlers.rs:374

Genesis (1_443_571_200) → now at 1d is ~3,976 points today and grows 365/yr; it crosses OHLCV_MAX_POINTS = 5000 in mid-2029, at which point the bare GET .../ohlcv?timeframe=all — the documented default for that timeframe — hard-400s with no client change. The task file records this deliberately ("Time bomb, on purpose"), so this is for visibility rather than as an oversight: defaulting All to 1w in Timeframe::default_granularity removes the cliff at no cost.


Checked and cleared

  • Bind ordering in list_assets/ohlcv still matches placeholder order after the since_interval removal; toDateTime(?) receives an i64 clamped to 0..=4_102_444_800, inside ClickHouse DateTime range — the parseDateTimeBestEffort → 500 path is genuinely closed.
  • points can never under-count actual buckets (ceil(span/g)+1 >= floor(end/g)−ceil(start/g)+1 for every aligned grain, and 1M at 30 d over-counts), so silent truncation can't slip back through the check.
  • Cursor::valid_for(true) requiring a finite f64 does close the toFloat64 500; u32 on id rejects negatives in serde.
  • DefaultBodyLimit::max on the batch router survives .nest("/v1", …) and the outer portal/auth layers (inner layer inserts the extension last, closest to the handler).
  • chrono adds no new packages to Cargo.lock beyond the prices-api dep edge, so clock was already resolved — the "compiles nothing new" claim holds.

…RL realities

All seven review points addressed:

1. search: length-only cap (64B, shared MAX_STRING_PAYLOAD_LEN with the
   cursor) — the charset rule made lossy-decoded but listed asset codes
   unsearchable, contradicting this branch's own cursor reasoning.
2. start == end allowed (inclusive SQL bounds = one bucket); guard is
   now strictly greater-than.
3. granularity omitted on an explicit start/end window derives from the
   window span (finest fitting 5000 points), not from a timeframe the
   caller never sent — ?start=2020-01-01 alone is answerable again.
4. parse_time undoes query-string percent-decoding of '+': the space
   separator is fixed positionally (byte 10) and a trailing ' HH:MM' /
   ' HHMM' after a time is recovered as a +offset; raw-in-URL offsets
   work over HTTP, not only in unit tests.
5. cursors are minted URL_SAFE_NO_PAD (STANDARD still decoded for
   in-flight tokens) — next_cursor echoed verbatim survives a query
   string.
6. future-only start now says 'start is in the future' instead of
   blaming an end the client never sent.
7. the 2029 timeframe=all cliff is defused as a side effect of (3):
   all's default granularity is span-derived, so it self-coarsens
   (1d today, 1w post-2029) instead of hard-400ing.

Spec + validation table updated to match; 21 live-CH integration tests
green (cursor walk exercises the URL-safe tokens end-to-end).
@stkrolikiewicz
stkrolikiewicz merged commit 36ff62a into develop Aug 18, 2026
3 checks passed
stkrolikiewicz added a commit that referenced this pull request Aug 18, 2026
…low-ups

All 8 ACs checked off. Completion notes record the two hardening rounds
(8-angle /code-review + okarcz's PR #217 review), the emerged decisions
(length-only asset-code rules, URL-safe cursors, '+'-decoding recovery,
span-derived auto-granularity defusing the 2029 all-timeframe cliff) and
the one intentionally modified integration test. 0206 owns the deferred
remainders: cursor {sort,order} binding, ValidatedPath<AssetIdentifier>,
negative-test assert helper, parse_time dedup.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants