ROBOKOP Visualizations

Figures for the ROBOKOP knowledge-graph advisory board slides and other figures I have created, consolidated from scripts previously scattered and refactored to run end-to-end, minimize manual steps, and use the API when appropriate.

Design

• Prefer the live Automat API over raw data. Most figures need only aggregate counts (per node category, per predicate, per source), which the Automat API serves directly via /metadata, /graph-metadata, and /schema -- no multi-GB download required. common/automat_client.py wraps these calls.
• Raw data only where the API can't help. A couple of figures need something the API doesn't expose (e.g. the aggregator-tier provenance chain for the source-flow Sankey). Those do a one-time local scan of nodes.jsonl/edges.jsonl and cache the result, tagged by the live graph_version so it's never ambiguous which release a figure reflects.
• Some figures fundamentally can't be API-sourced. Figures 8/9 need ORION's build-time pre/post-normalization QC data, which only exists transiently during a KG build -- Automat only serves the final merged graph. Those figures document this clearly and ship a real sample dataset rather than pretending they're fully self-contained.
• One folder per figure, each with its own generate.py and README. Nothing writes outside its own output/.

Setup

uv sync

For figures that need local raw data (see that figure's README), place or symlink nodes.jsonl/edges.jsonl under data/raw/, or set ROBOKOP_NODES_PATH / ROBOKOP_EDGES_PATH. See config.py.

Figures

#	Figure	Status	Required files (see figure README for full detail)
01	Source → aggregator → ROBOKOP Sankey	Working	data/raw/edges.jsonl once, then data/<version>/source_breakdown.json
02	Node/edge count trend bars (version-over-version)	Working; synthetic fallback until a 2nd real round exists	data/<version>/schema_summary.json for ≥2 real versions, or the synthetic example folder
03	Gain/loss by source (version-over-version)	Working	figure 01's source_breakdown.json for two versions
04	Category-change Sankey (version-over-version)	Working; synthetic fallback until a 2nd real round exists	data/<version>/node_categories.tsv.gz for ≥2 real versions, or the synthetic example folder
05	Chord diagram (type-level + ID-prefix-level)	Working	none for type-level (live /schema); data/<version>/prefix_pairs.json for ID-prefix-level
06	KG structure network, by source	Working	data/raw/nodes.jsonl + edges.jsonl once, then data/<version>/kg_meta_*.jsonl
07	KG structure network, by node type	Working	same kg_meta_*.jsonl as figure 06
08	Source connectivity heatmap (ORION pre/post-normalization)	Working, sample data only	counts.txt/pairs.txt from the ORION build pipeline (not the live API -- see its README)
09	Source network diagram (ORION pre/post-normalization)	Working, sample data only	same counts.txt/pairs.txt as figure 08
10	Node-type connectivity heatmap (per ORION source)	Working; optional pre-vs-post panel (synthetic)	none for single-panel (live per-source /schema); synthetic example folder for the comparison panel
11	Pre-merge vs. post-merge node-type network	Working, faithful port of the original's hand-tuned design (see its README for exactly what's identical vs. adapted)	kg_meta_*.jsonl (post-merge side) + live per-source /schema (pre-merge side)
12	Single-gene neighborhood	Working	none -- pure live API, generalized beyond CFTR

Raw-data scans (data_prep/)

Four scripts need a local KGX dump; everything else is pure API:

Script	Needs	Produces	Used by
build_source_breakdown.py	data/raw/edges.jsonl	data/<version>/source_breakdown.json	figures 01, 03
build_node_categories.py	data/raw/nodes.jsonl	data/<version>/node_categories.tsv.gz	figure 04
build_kg_structure.py	data/raw/nodes.jsonl + edges.jsonl	data/<version>/kg_meta_{nodes,within_edges,cross_edges}.jsonl	figures 06, 07
build_prefix_pairs.py	data/raw/edges.jsonl	data/<version>/prefix_pairs.json	figure 05 (ID-prefix panel)

Two more scripts need no raw data at all, just the live API:

Script	Produces	Used by
snapshot_schema_summary.py	data/<version>/schema_summary.json (true unique node-prefix + predicate counts)	figure 02
build_example_previous_version.py	data/example_previous_version/ -- jittered, clearly-labeled synthetic data	figures 02, 04 fallback
build_example_pre_normalization.py	data/example_pre_normalization/ -- jittered, clearly-labeled synthetic data	figure 10 pre-vs-post panel

All raw-scan scripts cross-check their totals against the live API and warn if a local file doesn't match the currently deployed graph. Run them once per advisory-board round -- the small derived outputs (not the raw dumps) are committed to the repo, since some of them (like the current 6fe13d850fdbf89c baseline) can never be regenerated once a graph version is no longer live. data/example_previous_version/ is the one exception -- it's fully regenerable, so it's gitignored, not committed. build_kg_structure.py is the heaviest scan (two full passes, 10-20+ minutes depending on machine load) since it needs both raw files.

On synthetic fallback data

Figures 02 and 04 both need two time points to show a trend, and only one real advisory-board round has been captured with correct methodology so far. Rather than either block those figures entirely or silently reuse an earlier round's data that was counted a different (less correct) way, build_example_previous_version.py jitters the real current data into an explicitly-labeled stand-in. Every figure built from it says so clearly -- [SYNTHETIC] in chart legends/titles and a warning printed to the console -- and both figures automatically prefer real data the moment a second real round exists.

Figure 10's pre-vs-post comparison panel uses the same pattern for a different reason: real pre-normalization data doesn't just not-exist-yet like figures 02/04's second round -- it can never exist after the fact, since Automat only serves each source's already-normalized graph. build_example_pre_normalization.py jitters the real live post-normalization structure into a plausible pre-normalization stand-in, modeled on what the original heatmap captions in the source material actually described (normalization only ever increased connectivity). Same unmissable labeling convention, same "clearly synthetic, never presented as real" rule.