llmorch

One instruction becomes a runnable project, written by models from several different vendors at once.

You say build a notes app with tags. llmorch splits that into files, hands each one to whichever model is best placed to write it, has a model from a different vendor review what the first one wrote, checks that the finished files agree with each other, then starts the thing and drives it over HTTP to find out whether it actually runs. What lands on disk is a folder you can open and use, beside a report saying which model wrote what and what the evidence for it is.

Splitting work across vendors is easy to say and awkward to do. This project is mostly about the two things that make it awkward.

The models cannot talk to each other. Each one writes a single file, alone, with no sight of the others and no memory between them. So before any of them start, a planner writes an interface contract — the routes, the data shapes, the page names, the runtime it all has to work under — and every model is handed it verbatim. That contract is the only coordination there is, and it is the reason a frontend written by one vendor works against a backend written by another. Everything else in here exists to check that the contract was kept.

Free tiers are rationed along several axes at once, with different numbers per provider, days that end at different midnights, and limits that are sometimes shared across every model on an account. Get that arithmetic wrong and you do not get a slightly worse result — you get a 429 in the middle of work you have already partly paid for. So every request in the system, the planning request included, goes through admission control before it is sent.

Nothing here needs an API key to try. llmorch run "build a notes app" runs the whole pipeline against a mock provider, offline, and writes a folder at the end of it.

Using it

Quick start

Windows — installs it, puts it on PATH, and leaves a Desktop shortcut:

powershell -ExecutionPolicy Bypass -File .\setup.ps1

Then, anywhere:

llmorch          # set up keys and which models to use, in a browser
llmorch start    # then talk to it

Nothing needs a key to try: llmorch run "build a notes app" exercises the whole pipeline against a mock provider with no network at all.

Every way in

llmorch                                          # the setup page, in a browser
llmorch start                                    # a session, using what it saved
./llmorch                                        # ...from a checkout, no PATH needed
llmorch "build a notes app"                      # ...with the first thing said
llmorch run "build a notes app"                  # mock provider, no network
llmorch run --live --providers all "build a CLI that converts CSV to markdown"
llmorch run --smoke "build a notes app"          # ...and then run what it wrote
llmorch chat                                     # a session, not one shot
llmorch ask "what serves /api/items?"            # a question about what it built
llmorch resume <run_id>                          # after a quota wall

What happens when you ask for something

One instruction becomes a folder you can run. Every step is either free or is a request that had to pass admission control first:

1. Plan — one model turns the sentence into a set of nodes (one file each) and an interface contract: the routes, pages, data shapes and runtime every node is held to. One request, at HIGH priority, and cached by task text so repeating a build costs nothing.
2. Assign — free. Each node goes to a model by fitness, remaining quota and an even split, or by rotation if you asked for that. A job you pinned goes where you pinned it. (A bidding round can inform this at one request per model, and is skipped when there are more models than nodes — spending fifteen requests to place six is not worth what it buys.)
3. Execute — one request per node, run concurrently. A failure fails over to a different vendor, because failure modes correlate within one. A reply cut off mid-file grows the budget and retries rather than blaming the model.
4. Verify — free first: does it parse, is it a stub, did it stop early. Then a second vendor reviews it, which is one more request and is advisory.
5. Check the set — free. Eight checks read the finished files as a group: promised pages exist, referenced assets were written, the frontend calls only declared routes, the backend serves every one, modules agree on names and signatures.
6. Run it — optional, and the only step that executes what a model wrote. --smoke starts the project and drives the contract against it over HTTP.
7. Write it down — runs/<id>/report.md, and the live dashboard.

The contract in step 1 is the whole trick. It is the only thing every model sees, and it is why a frontend written by one vendor works against a backend written by another without the two ever exchanging a message.

Set it up in a browser, then talk to it in a terminal

llmorch on its own opens a local page: paste your keys, tick which models this account may use, and choose whether runs are live, reviewed, and started after they are built. It saves to settings.json beside the ledger, and llmorch start runs on that — so the flags are chosen once rather than restated every time. The command line still overrides them (llmorch start --mock), because a stored preference should never be the reason you cannot do something once.

It has a tab per mode — Chat, AI agent, Crew — each holding what that mode needs and nothing else: who answers questions and whether an answer may quote a file you named; which single model writes everything, and the warning that review cannot run when there is only one; review depth, node budget, how many run at once, and whether the work is divided by fitness or by rotation. Choosing a mode there is its own control rather than a side effect of opening the tab, so looking at what a mode would do never changes which one you get.

Choosing who does what

Its Who does what tab is where you say which model does which job — planner, chat and questions, backend, frontend, styling, docs, glue, reviewer. Anything left on Automatic is assigned the way it always was: by fitness, remaining quota and an even split, which is usually better than a person's guess. A pin binds the assignment and nothing else — failover still runs its whole ladder, because a model that has tripped its circuit breaker is not the one you meant to insist on, and a pinned reviewer that shares the author's vendor is skipped for that file rather than allowed to review its own family.

That page is the only part of the system that accepts a write, so unlike the read-only dashboard it carries a token, refuses a non-loopback Host, and will only write the key variables the manifest actually declares. Keys go one way: it can tell you a key is set and has no endpoint that gives one back. The token is kept for the machine rather than minted per launch, so the link keeps working — a fresh secret each time turned every bookmark into a dead end, and a stable one stops a cross-origin post exactly as well.

A session, not one shot

chat keeps the conversation: the first instruction builds a project, and each one after it is planned as a change to what already exists, so "now add tags" rewrites the two files that need it rather than the six that do not. What a later turn remembers is the instructions, the interface contract, and one summary per file — never the file contents, because a conversation that pasted its artifacts back into the planner would grow every prompt with the project instead of with the request. Sessions are saved after every turn; --continue picks the last one back up. A session is named after the first thing you ask it for — 20260907-165836-notes-app — and the slug alone is enough to name it again: llmorch ask --session notes-app "...". The timestamp stays in front because it is what makes these sort chronologically, which is how "the most recent session" is worked out.

Three kinds of session

A session asks what kind of session it is, before anything is said:

How should this session work?

  1  Chat        questions only — nothing gets built
  2  One agent   a single model plans and writes every file
  3  A crew      several models split the work and review each other  (default)

None of the three is a separate implementation. Chat is the question lane as a standing choice, one agent is every job pinned to one model, and a crew is what the system does anyway — which is why each can be described honestly, including what it gives up: pin every job to one model and cross-vendor review stops happening, because there is no second vendor left to ask.

A mode is a default, not a lock — /build still builds inside a chat session. --mode chat|agent|crew skips the question, the setup page can answer it once for good, and a resumed session keeps the mode it was opened in. A non-interactive stdin is never prompted.

Telling an instruction from a question

Not every line is an instruction. "what does the server do?" is a question and "looks good" is neither, and a session that planned both spent a request to be told there was nothing to plan. Which lane a line belongs in is a property of the line, so it is decided before anything is spent: a question is answered from what the session already knows, an acknowledgement costs nothing at all, and anything the classifier does not recognise is an instruction — the old behaviour, which is what makes it safe for the rules to stay small. /ask and /build settle the residue by naming the lane outright.

An answer is one request and writes nothing. It is grounded in the same memory a plan is made against — the instructions, the contract, one summary per file, and who wrote it — plus the contents of any file the question names, so the prompt still grows with the request rather than with the project. llmorch ask reaches the same lane from a shell prompt, without opening a session.

Running what it built

--smoke starts the generated project, drives the contract's pages and routes against it over HTTP, and reports what came back. How to start it is part of the contract the planner emits, so it is stated rather than guessed:

"launch": {"command": ["node", "server.js"], "port": 3000, "ready_path": "/"}

The command is checked before it runs: the program must be a known interpreter, and every path in it must resolve inside the output folder, through the same containment check that writing those files used. A command that fails either test is refused by name — never quietly replaced with a guess.

A project with dependencies gets --smoke-install, which runs a lockfile-pinned install first — npm ci --ignore-scripts and its pnpm and yarn equivalents. The recipe is chosen by which lockfile the build produced, not by anything a model said, and package install scripts stay disabled. Without that flag, a project needing an install is skipped with the command that would fix it, rather than being started into a folder where it can only fail. It is off by default and has to be asked for: every other step in the system treats model output as untrusted data, and this one hands it the interpreter. What it buys is the only evidence in a run that did not come from reading the code — a project whose files all parse, all pass review, and all agree with each other can still serve every page from the wrong directory, and nothing static will say so.

Checking that the pieces fit each other

Before any of that, eight deterministic checks read the finished artifacts as a set rather than one at a time — the pages the contract promised exist, the assets and modules they reference were written, the frontend calls only declared routes, the backend mentions every declared route, and the modules agree with each other on names and signatures. They cost no requests and they catch the failure a split build makes likely and a single author never would: every file impeccable against its own spec, and the project broken because two models agreed with the spec and not with each other.

Each run leaves runs/<run_id>/report.md next to the folder it produced — verdict first, then which model wrote which file, what the quota bought, how evenly the work landed, and what the checks and the smoke run found. The artifacts stay on disk indefinitely and look equally plausible either way; the evidence about them should not be the one part that lives in scrollback.

Watching a run

The tables are in the browser, not the terminal. A run's numbers change while it is happening, and a terminal can only show the state they were in when it ended — which is the least interesting moment. So a run publishes what it is doing to its own directory as it goes, and llmorch dashboard shows it live: which node each model has right now, how many attempts it is on, tokens as each one lands, and what is left of the day. The terminal keeps a verdict and a URL. --tables puts the full output back for anyone without a browser open.

Two "today" figures appear on that page and they disagree on purpose. The run panel's comes from the vendor's own rate-limit headers and is what admission control believes; the quota table's is this machine's ledger. Each says which it is, because two numbers under one name is how a reader ends up trusting neither.

Other commands

Supporting commands: doctor --probe (verify wire names before depending on them), discover (ask a key which models it can reach, spending no tokens), quota, ledger, dashboard (read-only, loopback only).

--providers narrows the roster itself, not just the client registry: a model left enabled that nothing can reach is a node assigned to nobody. The same narrowing covers a provider that is enabled in models.yaml with no key in the environment, which is the identical fault through a quieter door.

The library underneath

The quota half is a library in its own right, and the part of this repository with value outside it: llmorch.quota does the rationing arithmetic above and has no idea the orchestrator exists.

from llmorch.quota import Governor, Ticket, model_spec, provider_spec, quota_manifest

manifest = quota_manifest(
    providers=[
        provider_spec(
            "groq",
            base_url="https://api.groq.com/openai/v1",
            api_key_env="GROQ_API_KEY",
            rpm=30, tpm=8000, rpd=1000,   # as the provider's docs state them
            reserve_requests=50,          # held back for critical retries
        ),
    ],
    models=[
        model_spec(
            "groq/gpt-oss-120b",
            provider="groq",
            wire_name="openai/gpt-oss-120b",
            context=131072,
            max_output=4096,
        ),
    ],
)

governor = Governor(manifest)

verdict = governor.try_acquire("groq/gpt-oss-120b", 400, 1200)
if isinstance(verdict, Ticket):
    ...                                    # call the provider, then:
    # governor.commit(verdict, response.usage)
else:
    print(verdict.verdict.value, verdict.reason)

No YAML, no role taxonomy, no dependency on the orchestrator.

The distinction the whole thing turns on

Every refusal is not the same refusal:

Verdict	Meaning	What a caller should do
GRANTED	proceed	send the request
WAIT	a per-minute window is full	sleep retry_after_s, then retry
EXHAUSTED_TODAY	daily cap reached	try another provider; resume after its local midnight
UNSERVABLE	larger than the per-minute ceiling	never send it at this size — split it
COST_BLOCKED	would spend real money unauthorised	nothing, until a budget is set

UNSERVABLE is not WAIT. A request bigger than a provider's per-minute token ceiling will not fit however long you wait, and treating it as a wait condition hangs the caller forever. Against an 8,000 TPM provider this case is routine, not exotic.

WAIT is not EXHAUSTED_TODAY. A window that clears in nine seconds is not a model being unavailable for the day, and conflating them writes off a healthy provider until midnight. Both distinctions were learned by getting them wrong.

What else is in the library

Piece	What it does
llmorch.quota.Governor	Admission control across RPM / TPM / RPD / TPD, account- and model-scoped, with a reserve for critical work
llmorch.quota.LedgerStore	Append-only SQLite record of every call, stamped with the provider's own day key
llmorch.quota.restore_governor	Replays today's ledger at startup, so a fresh process does not think it holds the whole daily allowance
llmorch.quota.TokenEstimator	Character-based estimation, self-correcting per provider from real usage
llmorch.providers.OpenAICompatProvider	Dependency-free client for any OpenAI-shaped endpoint
llmorch.providers.parse_rate_limit_headers	Turns x-ratelimit-* headers — and prose like "retry in 12.4s" — into a snapshot

Three properties worth knowing, because they are easy to get wrong and all three cost a live 429 to discover:

• Reserve on the estimate, reconcile on commit. Without it, concurrent fan-out races: several callers each read a counter none of them has yet incremented.
• Monotonic clock for sliding windows, wall clock for day boundaries, never mixed. Wall time for a per-minute window lets a clock adjustment grant free quota; monotonic time for a daily counter never resets.
• Response headers override local counters. Local counting is inference; headers are fact, and they cost nothing to read. A stated wait also outranks any keyword in the body — if the server says come back in twelve seconds, waiting works, whatever else the error text mentions.

No dependencies you did not ask for

PyYAML, pydantic and tzdata — the last only because Windows ships no timezone database, and a provider whose day ends at Pacific midnight cannot be tracked without one.

Deliberately not an HTTP library and not a vendor SDK. The provider client is stdlib urllib on a worker thread, because every SDK ships its own retry and rate-limit machinery, which would sit underneath the governor and silently retry requests it never admitted. Admission control only works if every call goes through it.

What is in this repository

Path	What lives there
src/llmorch/quota/	The library. Admission control (governor.py), the append-only usage ledger (store.py), sliding windows and day keys (windows.py), self-correcting token estimation (estimator.py). Imports nothing from the orchestrator, and a test asserts it.
src/llmorch/providers/	One dependency-free client for any OpenAI-shaped endpoint, plus rate-limit header parsing and a deterministic mock that can inject each failure a real free tier produces.
src/llmorch/registry/	models.yaml loaded and validated — every role chain must span two vendors, every paid provider must declare pricing, every model must fit its provider's per-request ceiling.
src/llmorch/negotiate/	Deciding who does what: task decomposition, revision, bidding, the assignment reconciler, question-vs-instruction classification, and answering a question.
src/llmorch/engine/	Running it: the scheduler, the worker with its failover ladder, cross-vendor review, the two verification tiers, cross-artifact contract checks, materialisation, checkpoints, the smoke run, and live progress.
src/llmorch/configure/	The setup page — the only server here that accepts a write.
src/llmorch/dashboard/	The read-only window on quota, spend and the run in flight.
src/llmorch/report/	Terminal renderers and report.md. Recompute nothing, so the file and the screen cannot disagree.
tests/	765 of them, no network, no keys. conftest.py unsets every provider key so the suite tests the code and not the machine.
models.yaml	The roster: 15 models across 3 vendors, every wire name confirmed with a live call.
docs/original-plan.md	The plan this was built from.
HANDOFF.md	Current state, what is next, and the invariants not to break — each one learned by getting it wrong against a live API.

Licence

MIT. See LICENSE.

Current state, what is next, and the invariants not to break are in HANDOFF.md. The original 45k plan is in docs/original-plan.md.

Getting it

You need Python 3.11 or newer and git. No API key is required to try it.

1. Get the code

git clone https://github.com/Pranay4040/llmorch.git
cd llmorch

No git? On the repository page, use the green Code button → Download ZIP, unzip it, and cd into the folder.

2. Install it

Windows — one command, and you never touch a path again:

powershell -ExecutionPolicy Bypass -File .\setup.ps1

It creates the virtualenv, installs llmorch, puts it on your PATH so llmorch works from any directory, and leaves an llmorch shortcut on your Desktop that opens its own window.

macOS and Linux:

python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

That puts an llmorch executable in the environment's bin directory, which answers to a bare llmorch only once that directory is on PATH — activating the virtualenv does it. Without activating, the repository root carries a shim that needs neither: ./llmorch, or llmorch.cmd on Windows.

3. Check it works — no key, no network

python -m pytest -q                     # 766 tests, all offline
llmorch run "build a notes app"         # a whole build against the mock provider

The second one plans, executes, writes a folder and checks the result without touching the network. If that works, the install is sound.

4. Get a key

Groq is the one to start with — it is free and allows a thousand requests a day:

1. Sign up at console.groq.com.

2. Create an API key.

3. Run llmorch and paste it into the Keys tab, or put it in .env yourself:

   GROQ_API_KEY=your-key-here
   

.env is gitignored, and so is anything else matching .env*. Copy .env.example if you want the comments explaining each provider's free tier.

Gemini (ai.google.dev) and OpenRouter (openrouter.ai) are also free and already declared in models.yaml. Any two vendors is enough for cross-vendor review to work.

5. Confirm the key reaches something

llmorch doctor --probe --providers groq

One trivial call per model. It proves the key, the wire names and the limits — a model list is not an entitlement, and this is the only thing that tells them apart.

6. Use it

llmorch          # choose your models and how runs behave, in a browser
llmorch start    # then talk to it

Continuous integration

CI runs the suite on Linux (3.11 and 3.13) and Windows (3.12), then does a full offline demo run with --smoke — plan, execute against the mock provider, write the folder, and start what it wrote. It needs no secrets and never spends quota. Windows is in the matrix on purpose: several of the invariants this project holds were faults that appear only there.