Prove what your agent read. An open-source, local-first web scraper where every capture is a signed, offline-verifiable Ed25519 provenance manifest.
Bluefield turns any URL into clean Markdown or structured JSON, tracks how a page changes over time, and signs every capture with an offline-verifiable Ed25519 provenance manifest. Months later, with no network and no trust in us, anyone can prove exactly what a URL served and when.
Repository: https://github.com/danielvhofmann/bluefield
$ npx @bluefields/cli scrape https://example.comThat captures the page into a content-addressed store on your own disk and
signs it with a local key. No server, no account, no hosted database, no
telemetry — grep the source. Install it for keeps with
npm install -g @bluefields/cli (then bf --help), or build from source:
git clone https://github.com/danielvhofmann/bluefield && cd bluefield && pnpm install && pnpm -r build.
Give an assistant like Claude web access as native tools. The
@bluefields/mcp-server exposes scrape, map, crawl,
search, extract, and an offline verify — so your agent can not just
read the web, but prove what it read.
Add this to claude_desktop_config.json (on macOS:
~/Library/Application Support/Claude/claude_desktop_config.json). The same
block works in Claude Desktop, Claude Code, Cursor, and VS Code:
{
"mcpServers": {
"bluefields": {
"command": "npx",
"args": ["-y", "@bluefields/mcp-server"],
"env": {
"BLUEFIELDS_API_KEY": "bf_your_key_here"
}
}
}
}The MCP server is a thin client to the hosted Bluefield API — a free,
hard-capped reference demo, not a commercial product — so its data-plane
tools need a BLUEFIELDS_API_KEY. The one exception is verify, which runs
entirely offline against a pinned public key: no key, no network, no trust
in the server. See packages/mcp/README.md for the
tool list and configuration.
The whole point is that a signed capture is fail-closed: flip a single byte
of the stored page and bf verify catches it — offline, with a non-zero exit.
The animated cast was recorded before bf verify gained its capture method line (2026-07-26); the transcript below is a verbatim run of the current build.
Static transcript (a verbatim run, 2026-07-26)
$ bash examples/tamper-demo.sh
· throwaway store: /var/folders/…/T/bf-tamper-demo.aVQcKP (your ~/.bluefield is untouched)
1) Capture https://example.com/
→ this is the ONLY step that needs network; everything below is offline.
captured https://example.com/
snapshot 2a909528-8a32-430b-8780-aba17348b55b
sha256 ff67a9d764d6a2367a187734e697f6a53217db9a21c101d410a113ca871a299d
signed yes changed: yes
2) Verify the untouched capture — expect ✓ VERIFIED (content hash + signature)
✓ VERIFIED 2a909528
content hash matches : yes
signature valid : yes
keyId : ur6Z7L3GVCzaRX2EIVxG15lCIwHgkm5KlPRiOY8L6io
capture method : http-v1 — no JavaScript executed; 559 B
3) Tamper: flip ONE byte of the stored page on disk
object: objects/ff/ff67a9d764d6a2367a187734e697f6a53217db9a21c101d410a113ca871a299d
byte 0: 60 → 28 (file length unchanged — a silent edit)
4) Verify again — expect ✗ FAILED with a non-zero exit (fail-closed)
✗ FAILED 2a909528
content hash matches : NO
signature valid : yes
keyId : ur6Z7L3GVCzaRX2EIVxG15lCIwHgkm5KlPRiOY8L6io
capture method : http-v1 — no JavaScript executed; 559 B
content hash mismatch;
✓ Tamper detected: one flipped byte → 'content hash mismatch' → verify exited non-zero.
The signed hash proved the bytes changed, entirely offline. That is the point.bf crawl captures a whole site — every page through the same signed path. So
the proof scales: crawl a site, flip one byte of one stored page, re-verify
everything, and verification pinpoints exactly which page was touched
(examples/site-proof-demo.sh, a real run):
$ bf crawl https://bluefields-data.com --max-pages 6 --depth 2
crawl done — job 582fb1d0-…
captured 6 blocked 0 skipped 0 failed 0
# …flip ONE byte of ONE stored page, then re-verify ALL six:
✓ PASS 77da7a4d https://bluefields-data.com/about
✓ PASS bc986259 https://bluefields-data.com/pricing
✓ PASS adc7a39f https://bluefields-data.com/integrations
✓ PASS de0c2f76 https://bluefields-data.com/playground
✓ PASS 6b2d2caa https://bluefields-data.com/products
✗ FAIL 550acd5d https://bluefields-data.com/
✓ 6 pages, 6 signatures. One byte changed. The proof finds it.Crawling is robots-respecting by design (no flag to disable it), paced per
host, https-only, and resumable — Ctrl-C pauses, --resume continues.
Bluefield is not the fastest or the most battle-tested scraper. It is the one that lets you prove what you captured. Here is the honest tradeoff:
| Bluefield | Firecrawl | Crawl4AI | Jina Reader | |
|---|---|---|---|---|
| Open source | ✓ | ✓ | ✓ | partial |
| Signed, offline-verifiable provenance | ✓ | ✗ | ✗ | ✗ |
| Local-first (no account, runs on your disk) | ✓ | ✗ (hosted) | ✓ | ✗ (hosted) |
| Site crawl (every page a signed snapshot) | ✓ | ✓ (unsigned) | ✓ (unsigned) | ✗ |
| Sitemap map / URL discovery | ✓ | ✓ | ✓ | ✗ |
| MCP server | ✓ | ✓ | ✓ | ✗ |
| Watch + diff a page over time | ✓ | ✗ | ✗ | ✗ |
| Search endpoint | ✗ | ✓ | ✗ | ✓ |
| Coverage on hard / anti-bot sites | roughly a tie | mature, strong | fair | fair |
| Tail latency on the hardest targets | fair | best | fair | good |
| Price | free (self-host) | paid tiers | free | cheapest hosted |
| Community maturity | new | large | large | medium |
Where the others genuinely win: Firecrawl has more mature coverage, better tail latency on the hardest targets, and a much larger community. Crawl4AI has a large community too. Jina is the cheapest hosted reader. If those are what you need, use them — we would rather be honest than lose your trust.
Where Bluefield wins: provenance you can verify offline, a local-first design that never phones home, a first-class MCP server, and built-in watch + diff. If "prove what your agent read" matters, this is the point of the project.
Every signed capture conforms to the Bluefield Provenance Manifest (BPM) — a small, permanent, offline-verifiable record: an Ed25519 signature over a JCS-canonical manifest binding the content hash, the URL, the fetch time, and the robots posture.
BPM is a proposed specification, not an adopted or ratified standard. The
format, signature scheme, and verification procedure are written up in
packages/attest/spec/SPEC.md, with conformance
test vectors under packages/attest/spec/test-vectors/. Feedback and
independent verifier implementations are welcome.
The CLI runs the same audited engine (fetcher / extractor / differ /
attest) as the hosted Bluefield demo, wired to an embedded database (PGlite)
and a filesystem object store — so nothing leaves your machine beyond fetching
the page itself; verify, log, diff, and export are fully offline.
# Capture a page (needs network; https-only, blocks localhost/file:// by design)
$ bf scrape https://example.com/
# Prove it — recomputes the hash and checks the signature, fully offline
$ bf verify 3f9a2b
# Diff, history, and a portable proof bundle anyone can re-verify
$ bf diff https://example.com/
$ bf log https://example.com/
$ bf export 3f9a2b --out ./proof # writes <id>.content + <id>.bundle.json| Package | What it is | License |
|---|---|---|
@bluefields/cli |
The bf command-line tool + local library |
AGPL-3.0 |
@bluefields/fetcher |
Tiered HTML capture (HTTP → headless browser), SSRF-guarded | AGPL-3.0 |
@bluefields/extractor |
HTML/PDF → clean Markdown / structured JSON | AGPL-3.0 |
@bluefields/differ |
Classifies and diffs successive captures | AGPL-3.0 |
@bluefields/attest |
The BPM reference implementation + spec + verifier | Apache-2.0 |
@bluefields/primitives |
Canonicalization, SSRF guard, robots.txt, env, logging | Apache-2.0 |
@bluefields/db |
Drizzle schema definitions (table shapes only) | Apache-2.0 |
@bluefields/sdk |
TypeScript client + offline attestation verifier | Apache-2.0 |
bluefields (Python) |
Python client SDK | Apache-2.0 |
@bluefields/mcp-server |
MCP server exposing scrape/verify tools to AI agents | Apache-2.0 |
Bluefield is dual-licensed by layer:
- Engine + headline tool — AGPL-3.0-or-later:
fetcher,extractor,differ,cli. If you run a modified version as a network service, the AGPL requires you to offer your users its source. - Spec, verifier, SDKs, primitives, MCP, schema — Apache-2.0:
attest(BPM spec + reference verifier),primitives,db,sdk-js,sdk-py,mcp. Permissively licensed so that verifying a Bluefield attestation, or building on the spec, carries no copyleft obligation.
Full texts: LICENSE-APACHE and LICENSE-AGPL.
Every source file carries an SPDX header naming its license.
Self-hosting Bluefield means running the bf CLI (or importing the library)
with your own keys — no Bluefield account, no hosted API. See
SELFHOST.md.
We would rather you trust us because the claims are checkable than because the copy is loud.
- What Bluefield is best at: provenance. Every result is a signed, offline-verifiable record — you can prove what your agent read, months later, with no network and no trust in us. If "prove what you scraped" matters, this is the point of the project.
- Coverage and speed: plain HTTP fetching handles most static sites with zero vendor keys. JS-heavy and anti-bot ("fortress") sites need a proxy and an unlocker you bring, and some will still return a challenge. We do not claim to beat commercial scraping APIs on raw reachability or tail latency on the hardest targets — mature hosted services (Firecrawl and friends) invest heavily there, and on those axes they often win.
- Extraction quality: good on articles, docs, and product pages; weaker on dense listing, forum, and some structured pages. It is an honest engine, not a magic one.
- No telemetry. The CLI phones home to nothing. This is a design property, not a promise — the source is here to check.
Requirements: Node 22+, pnpm 11+.
pnpm install # install + generate the lockfile
pnpm -r build # build every package to dist/
pnpm -r test # run the full test suite
pnpm typecheck # type-check every package
pnpm lint # biomeSee CONTRIBUTING.md.
The Python SDK (packages/sdk-py) is not part of the pnpm workspace; build and
test it with your Python toolchain (pip install -e packages/sdk-py[dev] then
pytest).