perf(pypangraph): validate loaded graphs with jsonschema-rs

[ivan-aksamentov]

⚠️ AI-generated contribution. This pull request was implemented by an AI agent. Review the code, tests, and benchmark methodology before merging.

Alternative PRs (mutually exclusive, merge one):

• -> jsonschema-rs #201 (you are here): swap the validator, keep dicts; smallest change
• msgspec #202: typed decode; fastest
• pydantic #203: typed models; most common library

Problem

Pangraph.from_json validates every loaded graph against the JSON schema generated from the Rust types before building the object. On real graphs that validation, not parsing, dominates load time. Issue #200 reports the slowdown.

On the benchmark graph below, parsing the JSON takes about 40 ms while the pure-Python jsonschema pass takes about 2 seconds: the validator walks every node, edit and position in interpreted Python.

This change

Validate with jsonschema-rs, a Rust-backed validator for the same schema, compiled once at import and reused for every load. Nothing else moves: the parsed value stays a dict, the collections and the whole public API are untouched, and the set of accepted and rejected graphs is identical. The load becomes bounded by decompression and parsing instead of validation.

Why this is the one to merge

• Largest win per line changed. The load goes from about 2 seconds to about 47 ms (a 42x end-to-end speedup) by changing the validator and its dependency. No new model layer, no refactor.
• Zero API or behaviour change. Pangraph, its collections, and every downstream consumer keep working against dicts exactly as before. The only observable difference is speed.
• Lowest review and regression risk. The diff is confined to the loader plus dependency declarations. There is nothing to get subtly wrong in a hand-built type model, and no new failure modes in downstream code.
• Same validation, same errors. jsonschema-rs enforces the identical schema (required fields, types, the strand enum, non-negative integers), so a graph that loaded before still loads, and a graph that was rejected before is still rejected.

If the goal is to remove the bottleneck now with the least surface area, this is the strongest choice.

Benchmark

Fixture: packages/pypangraph/tests/data/staph.json.gz (664 blocks, 6817 nodes, 15 paths; 1.81 MB compressed, 9.72 MB decoded). Median of 7 runs on one machine in the project Python container, measured by packages/pypangraph/benchmarks/bench_load. Correctness parity was verified for every engine: each accepts the valid graph and rejects missing-field, wrong-type, negative-value and bad-strand mutations.

Full load (parse and validate together):

loader	parse + validate	speedup
baseline (json + jsonschema)	1989 ms	1x
jsonschema-rs (this PR)	47 ms	42x
msgspec	17 ms	117x
pydantic	92 ms	22x

Per-phase breakdown. Decompression and JSON parsing are shared; this PR changes only the validation step.

Shared (unchanged by this PR):

phase	ms
gzip decompress	23
json parse	37

Validation step (what this PR changes):

validator	ms
jsonschema (baseline)	1951
jsonschema-rs (this PR)	10

Methodology notes:

• Precompiling the pure-Python jsonschema validator does not help; the cost is the interpreted traversal, not validator construction.
• fastjsonschema was rejected: it errors on the schema's format: uint annotation.
• format: uint is a decorative annotation; the non-negative range is enforced by minimum: 0. No engine asserts on the format string.

Conclusion: validation is about 98% of load time, and every candidate removes it. jsonschema-rs gives the largest single-step reduction (about 200x on the validation step) while keeping the loader and its data model unchanged.

The three alternatives

All three PRs branch from feat/merge and rewrite the same loader; they cannot be combined.

• jsonschema-rs (this PR). Swap the validator, keep dicts. One-line loader change, no downstream impact. Fastest to review, biggest win for the smallest change.
• msgspec. Decode JSON bytes straight into typed structs; validation is part of decoding. Fastest overall (beats parsing into a dict) and gives typed internals, at the cost of a small typed-model layer and reading the models in the collections.
• pydantic. Parse and validate into typed models with model_validate_json. Typed internals from the most widely used validation library, about 22x, same small model layer.

Pick jsonschema-rs for the minimal, lowest-risk fix; pick a typed option if typed internals are worth a model layer.

Work items

• Validate with a jsonschema-rs validator compiled once at import, in pypangraph/class_graph.py.
• Declare jsonschema-rs in pyproject.toml, requirements.txt, and the Python container.
• Add parameterized tests locking in the accept/reject contract.
• Add benchmarks/bench_load and a graph-loading doc.

Verify

./dev/docker/python bash -c 'pip install -e packages/pypangraph pytest && cd packages/pypangraph && python3 -m pytest -q'
./dev/docker/python bash -c 'cd packages/pypangraph && python3 benchmarks/bench_load'

[ivan-aksamentov]

⚠️ AI-generated analysis. Produced by an AI agent. Verify the version facts, figures, and links before acting on it.

Heads up: CI is red here, and it's not something we can pin our way out of.

What's happening:

• The only failing job is Python 3.9. Every graph-load test fails at _VALIDATOR.validate() [src] with int too big to convert.

• Our block and node IDs are full-range u64 hashes. In the staph fixture the largest is 18445259059398967581 (just under u64::MAX), and about half of the 14298 IDs sit above i64::MAX.

• jsonschema-rs 0.35.0 (2025-11-16) is the release that first accepts integers that big, and it is the same release that dropped Python 3.9 [changelog] [issue]:

  Support for arbitrary-precision numbers, including large integers and high-precision decimals that exceed standard floating-point limits.

  Support for Python 3.8 & 3.9.

  (the second line is under the release's "Removed" section.)

• The newest wheel pip can install on 3.9 is jsonschema-rs 0.34.0 (2025-11-14), released two days earlier, which still reads integers as i64 and fails on half our IDs.

• So no jsonschema-rs version both runs on 3.9 and accepts our IDs. msgspec and pydantic do not hit this (they take u64 directly), which is why those two stay green on 3.9.

Two ways forward:

• Set requires-python >=3.10 on this branch and drop 3.9 from its CI matrix. Turns CI green, but leaves this option with a narrower Python range than the other two.

  Python 3.9 market share data (click to expand)

  • Python 3.9 has been end-of-life since 2025-10-31 [doc], so no more upstream security fixes.
  • 3.9 download share: 0.8% for biopython [doc] (the honest proxy for our users), 9.6% for numpy [doc] (mostly CI and servers).
  • The distros still defaulting to system Python 3.9 are RHEL/Rocky/Alma 9, Amazon Linux 2023 and Debian 11. Nobody runs scientific Python on the system interpreter there though, it is conda/venv, and RHEL 9 and AL2023 both ship 3.11 as well.

• Go with msgspec (perf(pypangraph): decode graphs into typed msgspec models #202) or pydantic (perf(pypangraph): parse graphs into typed pydantic models #203) instead. They keep 3.9 and are faster anyway.

Given msgspec and pydantic avoid this entirely and are faster, I would lean toward one of them and keep this branch only if the minimal-diff angle is what we care about most.