A provenance-tracking interactive analysis environment that retrospectively produces reproducible artifacts (ISA-Tab, workflows, RO-Crate).
The central abstraction is a Ledger — a persistent, append-only log of everything the user does during an analysis session. Every tool run, LLM interaction, custom script execution, and data transformation is recorded. At any point, the user can "crystallize" the ledger into formal artifacts:
- ISA-Tab for data/metadata
- Nextflow / Galaxy .ga for the workflow
- RO-Crate bundling everything
The user never thinks in terms of workflows — they just work. BioLedger remembers.
Every action in a session creates a LedgerEntry — an immutable, timestamped record stored in a local SQLite database. Entries are created automatically; you don't have to do anything special to get provenance.
| When | Entry Kind | What's Captured |
|---|---|---|
| You run a bioinformatics tool | tool_run |
Container image, exact command, input/output files (with SHA-256 hashes), parameters, exit code, duration |
| You run a custom script | script_run |
Same as tool_run, plus the script itself is saved as a file reference |
| You import data files | data_import |
File paths, checksums, sizes, format metadata |
| The LLM is called | llm_call |
Model name, prompt summary, full message history, token count, tool calls made |
| ISA-Tab metadata is generated | metadata_gen |
Generated files, ontology terms used |
| You add a note | user_note |
Free-form text annotation |
Each entry also carries:
parent_id— links entries into a DAG (directed acyclic graph) so BioLedger knows that tool B's input came from tool A's outputfiles— every input, output, log, and script file is recorded with its path, SHA-256 hash, size, and roletool_spec_snapshot— a frozen copy of the tool spec at the time of execution, so the record is self-contained even if the spec changes later
When you crystallize a session, BioLedger walks this DAG to produce a workflow. When you package it, all referenced files, the workflow, and the full ledger are bundled into an RO-Crate.
Chat messages (the back-and-forth with the LLM) are stored separately from ledger entries — they drive the conversation context but aren't part of the provenance graph.
The examples/hello_bioledger/ directory contains a complete end-to-end walkthrough: load an ISA-Tab dataset, import a custom tool, run it in a session, crystallize a workflow, and package everything into an RO-Crate.
# Import the example tool
bioledger tool import examples/hello_bioledger/line_counter.bioledger.yaml
# Create a session
bioledger session new --name "hello world"
# Start the interactive session
bioledger resume <session_id>Inside the session:
you> load examples/hello_bioledger/data/
assistant> Loaded dataset "Sample metadata analysis" — 3 samples, 2 organisms
you> run line_counter on s_study.txt
assistant> line_counter completed. Outputs: [summary.json]
you> quit
# Generate a Nextflow workflow from the session
bioledger crystallize <session_id>
# Bundle into an RO-Crate
bioledger package <session_id>ISA-Tab to reproducible package without writing workflow syntax. See the full walkthrough for details on what BioLedger records at each step.
- Tool discovery — search the Galaxy Tool Shed, nf-core modules, and Snakemake wrappers; run discovered tools via their native engines; import BioLedger YAML specs
- Building tool specs — author BioLedger tool specs manually for Docker, Conda, Nextflow, and Galaxy executors
- CSV to ISA-Tab — start from a plain samplesheet and convert to structured metadata
- Variant calling on L. donovani — use the public library repos to discover tools and data, then run a full pipeline end-to-end
- Fan-out scatter — run a tool across a collection of files using Snakemake or Nextflow (requires
bioledger[snakemake]orbioledger[nextflow]; Galaxy fan-out is not yet supported)
BioLedger uses LLM providers (OpenAI by default) for interactive analysis, tool parsing, and ontology lookup. Set your API key as an environment variable or in a .env file in your working directory:
# OpenAI (default)
export OPENAI_API_KEY="sk-..."
# Or use Anthropic
export ANTHROPIC_API_KEY="sk-ant-..."
# Or use Google Gemini
export GOOGLE_API_KEY="..."Or copy the example .env file and fill in your keys (BioLedger loads it automatically):
cp .env.example .env
# Edit .env with your API keyThe default model is openai:gpt-4o. To use a different provider, set both the API key and the model:
# Example: switch to Google Gemini
GOOGLE_API_KEY=...
BIOLEDGER_DEFAULT_MODEL=google-gla:gemini-2.5-flashTool execution runs inside Docker containers. Install Docker if you plan to run bioinformatics tools through BioLedger. For HPC and shared-cluster environments where Docker isn't available, install a workflow engine extra (bioledger[snakemake] or bioledger[nextflow]) — engines handle conda/Apptainer/Singularity natively.
# Core only (Docker-based single-tool execution)
pip install -e .
# With all optional dependencies
pip install -e ".[cli,isaforge,toolforge,analysis,crateforge,dev]"
# With workflow engine support for fan-out operations
pip install -e ".[snakemake]" # Snakemake (pure Python)
pip install -e ".[nextflow]" # Nextflow (requires Java)
pip install -e ".[galaxy]" # Galaxy (planemo + bioblend)
pip install -e ".[snakemake,nextflow,galaxy]" # All three enginesSchema dependencies: BioLedger depends on
bioledger-toolspec-schemaandbioledger-isatab-schema, which live in sibling repositories. If you seeModuleNotFoundError, install them first in editable mode:pip install -e ../bioledger-toolspec-schema pip install -e ../bioledger-isatab-schema
Or with conda:
conda create -n bioledger python=3.11 -y
conda activate bioledger
pip install -e ".[cli,isaforge,toolforge,analysis,crateforge,dev]"# Create a new analysis session
bioledger session new --name "RNA-seq analysis"
# Resume an interactive session
bioledger resume <session_id>
# List sessions
bioledger session listBioLedger is organized into forges — specialized modules that handle different aspects of the analysis lifecycle:
| Forge | Purpose |
|---|---|
| ISAForge | ISA-Tab metadata generation and validation |
| ToolForge | Tool spec management, tool discovery (Tool Shed + nf-core search), spec authoring |
| AnalysisForge | Interactive analysis, tool execution, workflow crystallization |
| CrateForge | RO-Crate packaging with full provenance |
| Engines | Multi-engine execution adapters (Snakemake, Nextflow, Galaxy) |
ToolForge manages bioinformatics tool specifications. Import BioLedger YAML specs, search the Galaxy Tool Shed and nf-core modules for new tools, validate specs, and search the local store.
# Search the Galaxy Tool Shed and nf-core modules
bioledger tool search "fastqc"
bioledger tool search "bwa" --source toolshed
bioledger tool search "samtools" --source nfcore
bioledger tool search "alignment" --source all
# Import a BioLedger YAML spec
bioledger tool import specs/samtools_sort.bioledger.yaml
# Validate a spec (--strict treats warnings as errors)
bioledger tool validate ~/.bioledger/tools/fastqc.bioledger.yaml
bioledger tool validate specs/my_tool.bioledger.yaml --strict
# List all tools in the local store
bioledger tool list
# Search by name
bioledger tool list --search "fastqc"
# Show tool details
bioledger tool show fastqcBioLedger can search external tool repositories — the Galaxy Tool Shed, nf-core modules, and Snakemake wrappers — for tools that aren't installed locally. Use bioledger tool search (sources: toolshed, nfcore, all) to find tools by keyword. In an interactive session, the agent's search_toolshed, search_nfcore, and search_snakemake_wrappers tools cover all three repositories — including Snakemake wrappers, which are not exposed through the CLI flag.
you> search the tool shed for fastqc
assistant> Tool Shed results:
fastqc: FastQC — Read Quality reports (repo: iuc/fastqc)
...
you> run the fastqc tool from the tool shed on the raw reads
assistant> Submitted discovered tool fastqc as job abc123.
Source: toolshed:iuc/fastqc. Type 'status' to check progress.
Discovered tools are downloaded and run via their native workflow engine — the original tool XML, Nextflow module, or Snakemake wrapper is used directly, not translated to a BioLedger spec. Galaxy Tool Shed tools run via planemo (local Galaxy); nf-core modules run via Nextflow; Snakemake wrappers run via Snakemake.
Pin a specific upstream revision with @<revision> on the discovered-source string — e.g. nfcore:bowtie2/align@abc123 or snakemake:bio/fastqc@v7.2.0. The resolved revision is recorded on the ledger entry (discovered_source / discovered_revision) so packaged workflows reproduce the same upstream code.
Tool specs are stored as YAML in ~/.bioledger/tools/ and use a two-layer model:
- ExecutionSpec — container, command template, inputs, outputs, parameters
- InterfaceSpec (optional) — UI hints, conditionals, repeat blocks
Authoring a new tool? See the spec reference: src/bioledger/toolspec/README.md — the canonical format guide lives in bioledger-toolspec-schema/docs/GUIDE.md, and the Pydantic models in that package are the source of truth.
AnalysisForge powers the interactive chat loop. Load a dataset (CSV samplesheet or ISA-Tab directory), get LLM-powered tool suggestions, run tools in Docker containers, and review outputs — all tracked in the ledger.
# Start or resume an interactive session
bioledger resume <session_id>Inside the session, use load with a CSV samplesheet or ISA-Tab directory:
you> load samples.csv
assistant> Loaded dataset "Samples" — 3 samples, 2 organisms (Mus musculus, Homo sapiens)
you> load /path/to/isa-tab/
assistant> Loaded dataset "my_rnaseq" (3 samples, formats: fastq, txt).
Suggested workflow:
1. Quality control (FastQC)
2. Trimming (Trimmomatic)
3. Alignment (HISAT2)
you> run fastqc on the raw reads
assistant> Suggested: fastqc
Params: {threads: 4}
Run this tool? [y/N]: y
assistant> fastqc completed. Outputs: [sample1_fastqc.html, sample1_fastqc.zip]
you> review
TOOL [abc123] tool_run: fastqc outputs=[sample1_fastqc.html, ...]
DATA [def456] data_import: my_rnaseq
you> quit
Convert a session's tool runs into a reproducible Nextflow workflow:
# Generate Nextflow DSL2
bioledger crystallize <session_id>
# Include only specific entries
bioledger crystallize <session_id> --entry abc123 --entry def456ISAForge handles ISA-Tab metadata — loading existing ISA-Tab archives, building new investigation/study/assay structures, and validating them against ontology terms via the OLS4 API.
Package a session (or selected entries) into a self-contained RO-Crate:
# Package entire session
bioledger package <session_id>
# Package specific entries to a custom directory
bioledger package <session_id> --entry abc123 --entry def456 --output-dir ./my_crateThe resulting crate includes:
ro-crate-metadata.json— full provenance graphworkflow.nf— crystallized Nextflow workflow (single-engine sessions)ledger.json— raw ledger entries- All input/output data files referenced by included entries
For multi-engine sessions (entries from 2+ distinct executors, e.g. Docker + Snakemake), the crate uses a composite workflow entity as mainEntity:
- Each engine's native artifact (
workflow.nf,*.smk) is included as a supplementary file - A synthesized composite workflow entity references them via
hasPart - The dependency graph from the ledger is embedded in the composite entity's description
BioLedger supports three workflow engines for fan-out (scatter) operations and native workflow toolspecs. Each engine is an optional pip extra — the core package works with Docker only. Note: fan-out/scatter is currently supported via Snakemake and Nextflow only; Galaxy is supported for single-tool runs and Tool Shed discovery but not yet for fan-out.
| Engine | Install Extra | Requires |
|---|---|---|
| Snakemake | bioledger[snakemake] |
Python (pure) |
| Nextflow | bioledger[nextflow] |
Java/JVM |
| Galaxy | bioledger[galaxy] |
planemo (local) or bioblend (remote) |
# Install one or more engine extras
pip install -e ".[snakemake]"
pip install -e ".[snakemake,nextflow,galaxy]"When a tool spec has a collection input (e.g. bam: ToolInput(collection=True)), BioLedger can scatter the tool across a collection of files using a workflow engine:
you> run samtools_sort on all the bam files in data/
assistant> Suggested: samtools_sort (via snakemake)
Scattering over: data/*.bam
Run this tool? [y/N]: y
assistant> Submitted fan-out via snakemake as job abc123.
Type 'status' to check progress.
The agent detects collection-eligible inputs, selects an available engine, generates a parameterized workflow (Snakefile / Nextflow DSL2 / Galaxy .ga), and submits it. All paths are parameterized via config — no hardcoded paths in generated workflows.
When no engine is specified, BioLedger auto-detects in preference order: Nextflow → Snakemake → Galaxy. If none are installed, the error message includes install instructions.
When using Galaxy, the adapter queries the Galaxy instance's tool panel before submission. If the tool isn't available, it errors with a clear message — no Docker fallback.
bioledger session new --name "My analysis" --description "RNA-seq of mouse liver"
bioledger session list # active sessions
bioledger session list --all # include archived
bioledger session show <session_id> # details + recent chat
bioledger session rename <session_id> "Better name"
bioledger session describe <session_id> "Updated description"
bioledger session archive <session_id> # soft-deleteBioLedger reads configuration from environment variables prefixed with BIOLEDGER_:
| Variable | Default | Description |
|---|---|---|
OPENAI_API_KEY |
— | OpenAI API key (or use another provider's key) |
ANTHROPIC_API_KEY |
— | Anthropic API key |
GOOGLE_API_KEY |
— | Google AI API key |
BIOLEDGER_DEFAULT_MODEL |
openai:gpt-4o |
LLM model in provider:model format (also switches primary task models) |
BIOLEDGER_HOME_DIR |
~/.bioledger |
Data directory for sessions, tools, cache |
Setting BIOLEDGER_DEFAULT_MODEL switches the provider for all primary tasks. Utility tasks (parsing, fixing, review) stay on openai:gpt-4o-mini by default. Providers: openai, anthropic, google-gla, google-vertex, groq, ollama. See .env.example for examples.
pip install -e ".[dev]"
pytest tests/ -v # 897 tests
ruff check src/ tests/
mypy src/bioledgerMIT