Add CATMAID pass-through for the hosted instances - #95
Merged
Conversation
New catmaid_client module: a curated read-only registry of ~49 CATMAID
API commands runnable against any VFB-hosted instance (registry and
anonymous tokens fetched from virtualflybrain.org/data/EM/catmaid.json).
Commands that take skeleton ids accept CATMAID skids, VFB ids or a
mixed list; VFB ids are converted through the KB's
database_cross_reference xrefs (site map derived live from the Site
link_base URLs, static fallback), and CATMAID neuron ids are bridged
from skids via neurons/from-models since the KB does not store them.
Results default to a VFB envelope carrying id_map/unmatched plus a
reverse_map of skid-shaped result keys back to VFB ids; raw=True (or
?raw=true) returns the untouched CATMAID response, sharing one cache
entry with the wrapped view.
ha_api gains GET /catmaid, /catmaid/{instance} and
/catmaid/{instance}/{command}, riding the shared dispatch machinery;
the path allowlist learns prefix routes, and _dispatch_to_pool grows an
opt-in client_error_types so worker-side ValueErrors surface as 400s
rather than 500s (only the /catmaid handlers set it).
Only the FAFB, FANC and L1EM projects have skid xrefs in the KB, so the
other instances are skid-only; a VFB id passed there errors clearly.
swc grows aligned=true (with optional template=), serving the template- registered volume.swc from the VFB image store instead of CATMAID's EM-space skeleton. A skid input is reverse-mapped to its VFB record first; registrations come from the KB's in_register_with folders, and a neuron registered to several templates asks the caller to pick rather than guessing. catmaid.json now caches for the whole run by default (TTL 0); a positive VFBQUERY_CATMAID_CONFIG_TTL restores periodic refresh for long-lived servers.
aligned=True plus a separate template= read as one boolean choice, but VFB will soon register nearly every adult neuron to two spaces (JRC2018U and the planned unified brain+VNC template), so the option now takes a value: a template short_form or label picks the space, vfb/true means 'the VFB copy' and works only while a single registration exists (it errors listing the choices otherwise), and original/catmaid/omitted is the CATMAID EM-space skeleton. The separate template= parameter is gone before anyone depends on it.
larva1099 is simply not yet loaded into VFB, so its xrefs will arrive with the data; abd1.5, iav-robo, iav-tnt and l3vnc host mutant specimens, which sit outside VFB's wildtype-based data model, so no xref import is planned for them.
One row for the original EM-space skeleton (CATMAID) plus one per VFB template registration, each HEAD-checked for an actual volume.swc in the image store and carrying the aligned= value that fetches it through the swc command. A skid with no VFB record lists the original only, with a note. First local (VFBquery-answered) registry command; the registry gains a 'local' flag for it.
The test-examples job wraps every vfb.*(...) call in print(), so 'fafb = vfb.catmaid(...)' became 'fafb = print(...)' and every later fafb call exploded on None. Import catmaid directly for the instantiation line — the rewrite leaves it alone — and the block runs clean under the exact CI sed (verified locally against live services).
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
New
catmaid_clientmodule exposing the VFB-hosted CATMAID instances (registry and anonymous read-only tokens fetched from https://virtualflybrain.org/data/EM/catmaid.json, cached for the whole run) through a curated, read-only command registry — skeletons, SWC export, connectivity, connectivity matrix, annotations, connectors, labels, nodes, stats and landmarks. Anywhere a command takes skeleton ids you can pass CATMAID skids, VFB ids or a mixed list: VFB ids are converted through the KB'sdatabase_cross_referencexrefs (site map derived live from Sitelink_baseURLs, static fallback), and CATMAID neuron ids are bridged from skids vianeurons/from-models. Results default to a VFB envelope withid_map/unmatched/reverse_map;raw=True(or?raw=true) returns the untouched CATMAID response.swcadditionally takesaligned=naming a template space (short_form or label, e.g.aligned=JRC2018Unisex), serving VFB's template-registeredvolume.swcfrom the image store instead of CATMAID's EM-space skeleton — a skid input is reverse-mapped to its VFB record first.aligned=vfbmeans "the VFB copy" and works while a single registration exists; once neurons are registered to several spaces (e.g. a unified brain+VNC template) it errors listing the choices.aligned=originalor omitted is the CATMAID original. A companionswc_alignmentscommand lists the spaces an SWC is available in for one neuron — the original plus each VFB registration, HEAD-checked for an actualvolume.swcand carrying thealigned=value that fetches it.The HA API gains
GET /catmaid,GET /catmaid/{instance}andGET /catmaid/{instance}/{command}?ids=…[&project=][&raw=true]on the shared dispatch machinery. The path allowlist learns prefix routes, and_dispatch_to_poolgrows an opt-inclient_error_typesso caller mistakes come back as 400s (only the /catmaid handlers set it).Why
Third-party tools and VFBchat need CATMAID data addressed by VFB ids without every client reimplementing token handling, id conversion and the skid↔neuron-id dance.
How to test
pytest src/test/test_catmaid_passthrough.py(unit +integration-marked live tests), orpython -m vfbquery.ha_apiandcurl 'localhost:8080/catmaid/fafb/neuron_names?ids=VFB_001011rj,10603863'.Follow-ups left
larva1099 is skid-only until its neurons are loaded into VFB; abd1.5, iav-robo, iav-tnt and l3vnc host mutant specimens outside VFB's wildtype-based data model, so they are expected to stay skid-only. NBLAST compare and point/pointcloud reads were left out of the registry pending a use case.