feat: add NCATS GARD rare-disease registry as a biolink:Disease source

[SkyeAv]

Adds GARD — the NCATS Genetic and Rare Diseases rare-disease registry — as a biolink:Disease source in the existing disease (diseasephenotype) pipeline. GARD is a flat CSV of rare-disease terms (CURIE, preferred label, pipe-separated synonyms); it carries no cross-references of its own, so it contributes identifiers and labels/synonyms only — no concord — and every term is typed biolink:Disease.

Ingest

• New handler: src/datahandlers/gard.py downloads the Salesforce ContentVersion CSV (pull_gard) and parses it to per-prefix labels/synonyms files (pull_gard_labels_and_synonyms). The DisplayName is emitted as a label and as an exact synonym; each pipe-separated Synonyms value becomes its own OIO:hasExactSynonym row (matching the Orphanet/DOID convention). The URL column is not ingested: Babel handlers emit only labels/synonyms/taxa/descriptions and there is no per-identifier URL attribute file for it to go in.
• Download: get_gard / get_gard_labels_and_synonyms rules in src/snakefiles/datacollect.snakefile; disease_gard_ids (an awk transform of the labels file, mirroring DOID/Orphanet) in src/snakefiles/diseasephenotype.snakefile. The download URL is configured as gard_download_url in config.yaml (a query-string Salesforce link with no stable filename, fetched directly rather than via pull_via_urllib) and passed to the rule as a params value, so repointing it retriggers the download instead of reusing a stale CSV.
• Wiring: GARD added to disease_ids and disease_labelsandsynonyms; GARD joins the existing Disease.txt (no new compendium file), so the util.py aggregators and report_tables.py need no change.

Design (load-bearing)

• Local-id form — unpadded. The registry publishes zero-padded seven-digit ids (GARD:0006038 "Chikungunya fever"), but DOID — which does cross-reference GARD — emits the unpadded form for 2,164 of its 2,187 distinct GARD xrefs (GARD:6038, from DOID:0050012 "chikungunya"), the other 23 padded. Babel standardizes on the unpadded form: normalize_gard_curie() strips leading zeros when parsing the registry CSV and in doid.build_xrefs(), so DOID's padded outliers meet the rest. Without this, 1,886 rare diseases would carry two identifiers in two separate cliques (the registry singleton and DOID's) and no DOID GARD xref would ever pick up a registry label.
• extra_prefixes=[GARD]: GARD is registered neither in the Biolink Model's disease id_prefixes nor in its prefix map (verified against the pinned biolink_version), so write_compendium would silently drop every GARD CURIE — both the ~16k registry terms and the 2,186 that reach disease cliques through DOID's concord. The disease build passes extra_prefixes=[GARD] at the write_compendium call site in build_compendium (the documented escape hatch). Two network-marked tests in tests/datahandlers/test_gard.py lock this in: one asserts GARD is absent from disease id_prefixes (and flips to prompt dropping the line once registered), the other asserts NodeFactory.create_node(..., extra_prefixes=[GARD]) for biolink:Disease does not raise (mirroring test_all_override_target_types_are_writable). The impact report's section-4 samples are flagged "NOT emitted — prefix not registered", which is exactly this precondition made visible.
• No GARD concord, but cliques still merge. GARD asserts no xrefs, so there is no GARD concord file. The merging happens in the other direction: DOID's existing xrefs pull 1,886 registry terms into the DOID/MONDO cliques for the same disease. (DOID also asserts 300 GARD ids the current registry no longer publishes; those stay in their DOID clique without a label, like any other xref target Babel does not ingest.)
• Ingest guards, not log lines. A broken distribution must not produce a green build with zero rare diseases in it, so the failure modes raise rather than log: pull_gard rejects a response whose Content-Type is not a CSV (an expired ContentVersion link serves an HTML error page with HTTP 200, which urllib does not raise on), and the parser raises if the ID/DisplayName headers are missing or if no term parses at all. A scan of the published CSV found no DisplayName/Synonyms value with an embedded tab or newline and no row with an empty DisplayName.

Impact report

Regenerated and committed at docs/sources/GARD/impact-report.md (+ impact-report/ detail CSVs), in synthetic mode against a complete local disease intermediate set (all 10 disease_ids files and all 8 disease_concords), with GARD's own intermediates and the DOID concord rebuilt from the current DOID release so the report reflects the unpadded ids. Results:

• 16,214 identifiers added, all GARD: / biolink:Disease.
• 14,328 new single-identifier cliques (a 3.25% increase; 440,990 → 455,318 total cliques).
• 1,886 GARD identifiers land in 1,644 existing cliques, all via DOID's pre-existing xrefs — GARD's ids file promotes CURIEs already present in those cliques to first-class typed identifiers and gives them a label. 0 merges, and no clique gains a structurally new identifier, so no existing clique is restructured.
• 0 cross-reference rows contributed (GARD has no concord).

Docs

• docs/sources/GARD/README.md (new) + an entry in the docs/sources/ index + the committed impact-report.md.
• Also adds [tool.ruff.format] exclude = ["*.md"] to pyproject.toml: origin/main lacks this CI fix and ruff 0.16.0 began formatting Python code fences inside Markdown (overlapping rumdl), which would fail this PR's "Check Python formatting with ruff" job. It is a prerequisite for this PR's CI and is documented inline.

Testing

• uv run pytest tests/datahandlers/test_gard.py tests/datahandlers/test_doid.py --network --no-cov -q → 12 passed
• uv run pytest -m unit --no-cov -q → 412 passed, 107 deselected
• uv run ruff check / ruff format --check / snakefmt --check --compact-diff . / rumdl check . → all clean
• uv run snakemake -c 1 -n get_gard get_gard_labels_and_synonyms disease_gard_ids → 3 jobs resolve in the correct chain
• uv run source-impact-report --source GARD → report written (14,328 new cliques, 1,644 modified, 0 merges)

TODO before / after merge

• Register GARD in the Biolink Model. File an issue at https://github.com/biolink/biolink-model/issues and follow it with a PR adding GARD to the valid id prefixes for Disease (https://biolink.github.io/biolink-model/Disease/#valid-id-prefixes), and to the Biolink prefix map so GARD CURIEs resolve. Once it lands and biolink_version is bumped: drop extra_prefixes=[GARD] in src/createcompendia/diseasephenotype.py, invert test_gard_not_in_biolink_disease_id_prefixes, and update docs/sources/GARD/README.md. This is the same escape hatch GTDB is on (PR feat: add GTDB as a biolink:OrganismTaxon source in the taxon pipeline #978).

Questions for the reviewer

• extra_prefixes vs. Biolink-first. This PR ships the extra_prefixes=[GARD] escape hatch so the ~16k rare-disease terms actually surface in Disease.txt now (rather than being silently dropped), with Biolink registration tracked as the TODO above — is that the preferred path, or should GARD wait on upstream Biolink registration before being kept?
• Retired GARD ids. 300 GARD ids that DOID asserts are absent from the current registry, so they enter Disease.txt label-less alongside their DOID clique. Keeping them means data still citing a retired GARD id normalizes correctly; the alternative is filtering DOID's GARD xrefs against the ingested registry. This PR keeps them — say the word if you'd rather filter.

[SkyeAv]

Update: no-URL exclusion + source-impact report

Per the GARD URL field: a row with no URL has no public rarediseases.info.nih.gov page and is a candidate/provisional term — not a real rare disease — so it is now excluded from the ingest. This drops ~9,949 of ~16,214 rows, keeping 6,265 real rare-disease terms.

• The filter is the single if not (row.get("URL") or "").strip(): continue line in src/datahandlers/gard.py; because disease_gard_ids derives ids from the labels file via awk, the exclusion cascades into the ids file and the compendium automatically. The parse logs a kept/skipped/no-URL/empty-name summary so a future NCATS format change can't silently zero the output.
• A source-impact report (synthetic mode) is now generated and committed at docs/sources/GARD/impact-report.md (+ impact-report/ detail CSVs). Baseline disease intermediates came from the 2025dec11 published snapshot (MP's concord, absent from that snapshot, was rebuilt locally from UberGraph). Result: 6,265 new single-identifier cliques (241,269 → 247,534), 0 merges / 0 modified / 0 xrefs — purely additive, no existing clique disturbed.
• The report's section-4 samples are flagged "NOT emitted — prefix not registered in Biolink Model for biolink:Disease"; that flag is why the build passes extra_prefixes=[GARD] (see the PR body's Design section). Registering GARD upstream removes both the flag and the need for the escape hatch.

[SkyeAv]

Reverted: the URL-field filter was a false positive

A missing GARD URL does not mean a row isn't a real rare disease — a term may simply lack a GARD page. Per maintainer feedback: GARD:0001049 "Cerebral arteriopathy ... type 1" (CADASIL) has a URL, while GARD:0024558 "Cerebral arteriopathy with subcortical infarcts and leukoencephalopathy" (also CADASIL) does not — same disease family, so excluding the no-URL one would drop a real rare disease.

• The if not (row.get("URL") or "").strip(): continue filter in src/datahandlers/gard.py is removed; all 16,214 GARD terms are ingested (no URL gating). The URL column is back to "reference only, not ingested" (the CURIE resolves via the Biolink prefix map).
• The source-impact report is regenerated: 16,214 new cliques (241,269 → 257,483, +6.72%), 0 merges / 0 modified / 0 xrefs — still purely additive.
• Tests updated: the verbatim fixture (one row with a URL+synonyms, one without either) now asserts both are kept.

[SkyeAv]

@gaurav this branch successfully rebuilt the Disease Compendia at /projects/babel/runs/goetzs/NIH-GARD-1.18 on HT1 with the GARD ontology. Do you want me to put anything from that here or is a path to the sucessful output enough to merge this?

[Copilot]

Pull request overview

Adds GARD as a label-and-synonym-only disease source in Babel’s disease pipeline.

Changes:

• Adds GARD download, parsing, configuration, and pipeline wiring.
• Preserves GARD identifiers through the Biolink prefix escape hatch.
• Adds tests, documentation, and a source-impact report.

Reviewed changes

Copilot reviewed 14 out of 15 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
config.yaml	Configures the GARD download and disease inputs.
pyproject.toml	Excludes Markdown from Ruff formatting.
src/prefixes.py	Defines the GARD prefix.
src/datahandlers/gard.py	Downloads and parses GARD data.
src/createcompendia/diseasephenotype.py	Allows GARD during compendium writing.
src/snakefiles/datacollect.snakefile	Adds GARD collection rules.
src/snakefiles/diseasephenotype.snakefile	Generates typed GARD identifiers.
tests/data/gard_sample.csv	Provides representative GARD records.
tests/datahandlers/test_gard.py	Tests parsing and prefix handling.
docs/sources/README.md	Adds GARD to the source index.
docs/sources/GARD/README.md	Documents GARD ingestion.
docs/sources/GARD/impact-report.md	Summarizes source impact.
docs/sources/GARD/impact-report/new-cliques.csv	Lists newly introduced cliques.
docs/sources/GARD/impact-report/modified-cliques.csv	Records modified-clique results.
docs/sources/GARD/impact-report/new-xrefs.tsv	Records cross-reference results.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.