Upgrade to Cartesi Machine v0.21

[GCdePaula]

This upgrades Dave’s complete state-transition stack to Cartesi Machine v0.21.0 and solidity-step v0.15.0. It also closes snapshot-durability, cross-implementation testing, and build/release-tooling gaps exposed while qualifying that upgrade.

The compatibility target is same-version equivalence. Cartesi Machine v0.20 and v0.21 roots are expected to differ, so the contracts, node, Lua client, machine images, and proof producers must move together.

The Honeypot fixture pins the exact commit from cartesi/honeypot#37, replacing Dave’s local compatibility patches. The upstream PR does not need to merge before this SHA-pinned build can be used.

What changes

Cartesi Machine and state-transition compatibility

• Updates the emulator from v0.20.0 to v0.21.0 and solidity-step from v0.14.0 to v0.15.0.
• Updates the Rust bindings, Lua client, proof producers, machine images, and CI/build paths together.
• Sources primitive commitment geometry from the release-owned constants instead of maintaining Dave-specific aliases.
• Replaces the RISC-V and CMIO helper chain with one CartesiStateTransition that directly composes the upstream transition libraries. The tournament-facing IStateTransition.transitionState interface remains stable.
• Covers accepted and rejected inputs, halt, exception, unexpected manual yield, mcycle overflow, uarch limits, reset, padding, and input openings.
• Keeps Dave’s existing Ruler collector for this release. Adoption of the new cm_collect_* APIs remains a separate migration described in docs/plans/collect-hashes-migration.md.

Durable snapshot publication

• Executes normal input advances in publication batches controlled by --snapshot-gap-inputs, with a default of 64.
• An open partial tail waits for more inputs. A sealed epoch publishes its final shorter batch before rolling.
• Keeps a closed transient checkpoint as the rollback point within a batch and a mutable shared clone for execution. A rejected input discards the poisoned clone and resumes from the checkpoint.
• Publishes a retained boundary in durability order:
  1. verify and close the stored machine;
  2. sync its backing files;
  3. rename it without replacement into the content-addressed store;
  4. atomically register the snapshot and all window roots in the database.
• A crash may leave an unreferenced content-addressed directory, but cannot leave a database row pointing to an undurable machine. Recovery may replay at most one complete batch.
• Dispute-time densification may still publish additional proof-positioning boundaries.

Build and artifact lifecycle

• Makes the Cartesi Machine provider choice explicit:
  • a set, absolute LIBCARTESI_PATH selects an external installation;
  • an unset variable selects the prepared source checkout;
  • Cargo build scripts perform no downloads.
• Moves verified source preparation into just machine::setup.
• Uses semantic input stamps and receipts for generated bindings, machine images, and the devnet bundle.
• Keeps Justfiles as the public task graph while subsystem scripts own acquisition, diagnostics, cleanup, binding generation, and other procedural work.
• Splits diagnostics into base, E2E, and aggregate doctor scopes with a shared status contract: healthy, diagnosed setup issue, or checker failure.
• Keeps the release corpus outside ordinary setup because it is a large, release-specific conformance fixture.

Cross-implementation evidence

• Runs all 35 released v0.21 computation-hash cases through the release CLI.
• Compares Dave’s collector with the 17 released mcycle answers.
• Expands Solidity/FFI coverage across coordinate boundaries, terminal states, real composed witnesses, and mutations at the DA, CMIO, ustep, and reset seams.
• Adds focused per-PR E2E coverage and a manually dispatched full battery. The full battery remains manual until hosted-runner cost, disk bounds, and signal have been measured.

Compatibility and deployment notes

This is a fresh deployment generation:

• CartesiStateTransition changes from a two-address constructor to no constructor arguments.
• The old RISC-V and CMIO composition helpers are removed.
• Tournament and state-transition bytecode changes, which changes dependent CREATE2 deployment identities.
• Existing disputes cannot mix v0.20 and v0.21 components.
• Existing v0.20 node state directories are deliberately refused because their stored configuration pins the old emulator version and template hash. Operators must rebuild state from chain data using the v0.21 image.

The factory ABI changes introduced by #274 are already part of this PR’s base and are not introduced here.

Suggested review order

1. Machine/STF compatibility: version pins, generated constants, direct Solidity composition, and proof producers.
2. Snapshot invariants: batch scheduling, transient rollback checkpoints, durable publication order, and crash recovery.
3. Cross-implementation evidence and gas policy: corpus tests, Solidity/FFI coverage, and calibration records.
4. Build, doctor, artifact, and CI restructuring.

Deliberately deferred

• Adoption of cm_collect_*, pending resolution of the characterized RX_REJECTED at imcyclemax discrepancy.
• Reduction from three tournament levels to two; that remains a separately measured change.
• Scheduling the full E2E battery. It stays manual until representative hosted runs establish its cost, storage requirements, and actionable signal.
• Multi-process ownership of one state directory and broader recovery machinery.
• Migration of existing v0.20 node state.
• A permanent legacy/new collector dual path.

[guidanoli]

Solidity part looks good to me.

[stephenctw]

LGTM! Just some doc suggestions and minor nits.

--snapshot-gap-inputs help

Help text still sounds like “keep every Nth snapshot,” but an open epoch now waits until a full gap (default 64) before executing. 1 processes every input; seal still flushes a short tail. Worth saying that in the clap help so people don’t think the node is caught up.

Nit

gap < 1 panics in set_snapshot_gap_inputs. Clap validation would be nicer.

latest_boundary_in

Dispute skips vanished snapshot dirs; the runner retries forever on the newest row. Intended (“operator restores files”), or should resume fall back like dispute positioning?