diff --git a/app-sequencer/advanced/divergence.md b/app-sequencer/advanced/divergence.md new file mode 100644 index 000000000..7daab10d7 --- /dev/null +++ b/app-sequencer/advanced/divergence.md @@ -0,0 +1,112 @@ +--- +title: "Divergence and the content-identity check" +sidebar_label: "Divergence detection" +description: "How accepted base-layer batch content is compared with local sealed content, what the check cannot prove, and why divergence stops the sequencer." +--- + +Canonical divergence occurs when a batch accepted at a given nonce does not match the valid batch the sequencer sealed for that nonce, or when no matching valid local batch exists. + +This condition invalidates the assumption that the local batch tree mirrors canonical history. The sequencer records the mismatch and stops before the accepted frontier, snapshot lifecycle, or recovery logic can advance from an incorrect identity. + +## What the check compares + +When the sequencer seals a batch, it encodes the exact payload the submitter will broadcast and stores its Keccak-256 hash with the batch record. The payload hash is written in the same database operation that seals the batch and is protected from later modification. + +When base-layer synchronization encounters a batch that the scheduler acceptance predicate considers fully accepted, the sequencer: + +1. reads the accepted batch nonce and payload from the safe input stream; +2. finds the valid closed local batch carrying that nonce; +3. hashes the landed payload bytes; +4. compares the landed hash with the seal-time local hash. + +![The sequencer hashes a batch when it is sealed and hashes the accepted payload observed in the safe base-layer view. A matching local batch with equal hashes advances the accepted frontier. A missing local batch is foreign, while different hashes are a mismatch; either failure records persistent divergence and freezes the frontier.](../images/divergence-check.jpg) + +Content-equal copies are accepted. The identity of the physical base-layer transaction does not matter because identical batch bytes have the same scheduler effect. + +## Divergence classifications + +The implementation records two kinds of content-identity violation: + +| Kind | Meaning | +| ---------- | ---------------------------------------------------------------------------------------------------------------- | +| `foreign` | The scheduler accepted a nonce for which no valid closed local batch exists | +| `mismatch` | A valid closed local batch exists at the nonce, but its sealed payload hash differs from the landed payload hash | + +Possible causes include a delayed transaction from an abandoned branch, use of the submitter key outside the sequencer, database corruption, or a defect that changed sealed or submitted content. + +## When the comparison runs + +The comparison runs only after the off-chain scheduler predicate accepts a batch from the base-layer safe input stream. It does not compare payloads that are: + +- undecodable as batches; +- stale; +- carrying the wrong batch nonce; +- below the batch-tree anchor after checkpoint recovery. + +The first three cases have no scheduler effect, so their content cannot change canonical application state. Inputs below a recovery anchor belong to trusted history already folded into the checkpoint. The rebuilt local tree begins at the anchor and intentionally contains no earlier batch rows. + +## Detection timing + +Detection begins when the landing appears in the configured RPC endpoint's safe view and the input reader synchronizes that range. It therefore follows the base layer's safe-head delay and the input reader's polling cadence. + +There is no fixed detection duration that applies to every supported chain or provider. The batch submitter's configurable confirmation depth also does not control this check. The authoritative trigger is inclusion in the safe input view used to construct the accepted frontier. + +Soft confirmations issued before detection may already depend on a local state that canonical execution will not reproduce. This residual window is part of the optimistic design. + +## Atomic marker and frozen frontier + +The divergence marker is inserted in the same database transaction as the safe-head synchronization that detects the violation. The accepted landing is not added to `safe_accepted_batches`. + +Once the marker exists: + +- accepted-frontier population returns without scanning further; +- snapshot promotion cannot advance through the mismatched landing; +- the danger check reports canonical divergence before every staleness condition; +- startup refuses automatic recovery; +- the process exits with terminal code `30`. + +The marker persists across restarts. A supervisor that ignores code `30` will repeatedly start a process that immediately refuses the same state. + +## Why automatic recovery is unsafe + +Preemptive recovery assumes that every accepted nonce identifies the matching local batch. It uses that accepted frontier to select a cascade point and preserve the valid prefix. + +A content-identity violation disproves that assumption. Cascading from the local tree could preserve state that was never canonical or discard the wrong branch. The sequencer therefore refuses to derive a repair from the record whose identity is in question. + +The supported remedy is to preserve evidence, stop the old deployment, and rebuild a fresh data directory using [Cockroach recovery](../recovery/cockroach.md). + +## What the check does not prove + +The content-identity check proves that accepted batch bytes match the bytes sealed locally for the same nonce. It does not compare resulting application state. + +Matching bytes can still produce different state if: + +- the application is nondeterministic; +- the Cartesi machine and live prediction run different application versions; +- the scheduler implementations disagree about execution; +- direct inputs are applied differently outside the compared batch payload. + +The independent watchdog addresses this broader class by replaying canonical base-layer inputs and comparing resulting state bytes with the sequencer's finalized snapshot. See [Monitoring and watchdog operation](../operations/monitoring.md). + +## Prevention and residual risk + +Wallet-nonce resolution reduces the chance that an abandoned submission later claims a relevant slot, but it does not remove the need for content comparison. A checkpoint rebuild has the additional limitation that its fresh database lacks the original wallet-nonce watermark. See [Flush unresolved wallet nonces](../operations/orchestration.md#flush-unresolved-wallet-nonces) and [Cockroach recovery](../recovery/cockroach.md#resolve-the-submitter-wallet). + +## Operator response + +When canonical divergence is reported: + +1. stop automatic restarts and prevent use of the submitter key; +2. preserve the database, logs, RPC observations, and base-layer transaction evidence; +3. identify whether the marker is `foreign` or `mismatch`; +4. investigate key use, delayed transactions, software versions, and storage integrity; +5. fix the underlying cause before rebuilding; +6. rebuild from a verified checkpoint and validate the result with the watchdog. + +Removing the marker from the database is not a repair. It would allow frontier processing to continue without restoring the rejected identity assumption. + +## Next steps + +- Review the acceptance algorithm in [Scheduler semantics](./scheduler-semantics.md). +- Review the freeze and wallet-nonce properties in [Invariants](./invariants.md). +- Prepare terminal-failure handling with [Failure modes](../recovery/failure-modes.md). diff --git a/app-sequencer/advanced/formal-verification.md b/app-sequencer/advanced/formal-verification.md new file mode 100644 index 000000000..6ffd814ba --- /dev/null +++ b/app-sequencer/advanced/formal-verification.md @@ -0,0 +1,191 @@ +--- +title: "Formal verification" +sidebar_label: "Formal verification" +description: "What the bounded TLA+ model proves about preemptive recovery, its configured state space, and the behavior outside its scope." +--- + +The sequencer repository contains a TLA+ specification of the slot-level preemptive recovery design. TLC explores every reachable state within the configured bounds and checks that declared safety invariants remain true. + +This provides stronger evidence than a set of hand-selected examples for the behavior represented by the model. It does not prove the complete sequencer implementation correct. + +## Why model the recovery protocol + +Recovery depends on interleavings between: + +- local batch creation and submission; +- wallet-nonce assignment; +- base-layer inclusion or replacement by a flush transaction; +- movement from included to safe; +- scheduler acceptance or rejection; +- suffix invalidation and replacement branching. + +Many failures require an unusual order, such as an abandoned transaction winning a nonce slot after recovery has started. A model checker systematically explores those orders within a finite configuration and returns a concrete trace if one reaches an invalid state. + +## Model state + +The specification represents: + +- a valid batch spine with `Gold`, `Silver`, `Bronze`, `Pending`, and open `Tip` states; +- invalidated branches; +- the current safe block; +- the submitter's next wallet nonce; +- the next wallet-nonce slot processed by the base layer; +- included base-layer entries; +- the scheduler cursor and next expected batch nonce; +- submitted batches detached from the valid spine by recovery. + +A genesis sentinel provides an initial accepted ancestor in the model. The implementation handles the first-batch edge structurally and does not submit a sentinel batch. + +## Modeled transitions + +TLC explores combinations of these actions: + +| Action | Meaning | +| ------------------ | ---------------------------------------------------------- | +| `AdvanceTip` | Close the open batch and append a new tip | +| `SubmitBatch` | Assign wallet nonces to every unsubmitted pending batch | +| `L1IncludeSpine` | Let a valid submitted batch win its wallet-nonce slot | +| `L1SkipSpine` | Let a flush no-op win and displace a valid submitted batch | +| `L1IncludeDead` | Let a previously invalidated batch win its unresolved slot | +| `L1SkipDead` | Let a flush no-op displace an invalidated batch | +| `AdvanceSafeBlock` | Move included batches into the safe view | +| `SchedulerStep` | Process a safe entry and accept it or reject it | +| `SchedulerSkip` | Advance over a wallet-nonce slot consumed by a no-op | +| `Resolve` | Invalidate a stale suffix and create a replacement tip | + +At a contested wallet-nonce slot, the model allows either the batch or the flush transaction to win. This captures the adversarial outcome recovery must tolerate. + +![The bounded recovery model moves from an open batch to a closed and submitted batch, lets either the batch or a flush transaction win the base-layer nonce slot, advances an included batch to the safe scheduler view, and either accepts it or resolves an invalid suffix by creating a replacement branch.](../images/recovery-model-transitions.jpg) + +## Checked safety invariants + +The configured invariant `Inv` combines the following properties: + +| Invariant | Property checked | +| ----------------------- | ------------------------------------------------------------------------------ | +| `TypeOK` | The spine remains non-empty and the model's counters remain natural numbers | +| `BatchNoncesContiguous` | Non-tip batches on the valid spine carry contiguous nonces | +| `InvalidOnlyOnGold` | Invalid branches attach only to accepted `Gold` ancestors | +| `ZombieSafety` | The scheduler's expected nonce equals the length of the accepted `Gold` prefix | +| `L1WNonceUnique` | No two included base-layer entries occupy the same wallet nonce | +| `L1BeforeCursor` | Every included entry is below the next unprocessed base-layer wallet slot | +| `SchedulerBehindL1` | The scheduler cursor never advances beyond base-layer slot processing | +| `DeadNotYetIncluded` | Detached submitted batches retain only unresolved wallet-nonce slots | + +`ZombieSafety` is the central recovery property. It states that late, displaced, or invalidated submissions never make the scheduler accept more or fewer batches than the valid `Gold` prefix represents. + +## Safety is not liveness + +The model checks invariants in every reachable state. These are safety claims: specified bad states are not reached. + +It does not establish that: + +- the base layer eventually includes a transaction; +- the safe head eventually advances; +- a flush eventually completes; +- recovery always terminates; +- the sequencer resumes serving users within a time bound. + +Those are liveness and operational claims. They depend on the base layer, provider availability, supervisor behavior, configuration, and implementation tests. + +## Configured bounds + +The committed TLC configuration uses: + +| Constant | Value | +| ----------------- | ----: | +| `MaxBatchIndex` | 5 | +| `MaxSafeBlock` | 5 | +| `MAX_WAIT_BLOCKS` | 2 | +| `MaxWalletNonce` | 8 | + +The implementation's staleness constant is `1200`, while the model uses `2`. This reduction preserves the transition from fresh to stale but does not reproduce production timing or scale. + +Wallet nonces need a separate bound because repeated displacement and resubmission can keep generating new values. Increasing any bound can expand the state space sharply. + +The recovery design notes record a completed exploration of approximately 157 million states with no invariant violations for the committed model and configuration. That result applies only to the specification version and bounds that produced it. + +## What the model does not cover + +The TLA+ model intentionally excludes major parts of the deployed system: + +- the danger threshold and preemptive margin; +- wall-clock estimation during an RPC outage; +- the complete runtime sequence of stop, restart, flush, wait, synchronize, and resume; +- process crashes and SQLite or filesystem transaction boundaries; +- the implementation's danger-threshold invalidation of an open tip; +- the implementation's direct cascade of a pending batch displaced by a flush; +- direct-input queuing, censorship backstop, frames, transaction fees, and application execution; +- content-identity divergence detection; +- snapshots, checkpoint recovery, feeds, and HTTP behavior; +- signatures, chain identity, RPC completeness checks, and cryptographic assumptions. + +Two differences deserve particular attention: + +1. The model resolves an aging open tip at `MAX_WAIT_BLOCKS`; the implementation can replace it earlier at the configured danger threshold. +2. The model does not directly represent the implementation path that cascades a pending frontier batch after a flush no-op displaced it. + +The safety of those implementation paths is supported by separate reasoning and tests. TLC has not explored them as equivalent actions. + +## Relationship between model and code + +The specification is maintained manually. There is no refinement proof, trace conformance check, verified compiler, or generated implementation connecting it to the Rust code. + +A successful TLC run means: + +- the model satisfies its declared invariants within the selected bounds; +- TLC found no represented execution that violates those properties. + +It does not mean: + +- the Rust implementation exactly matches every modeled transition; +- the model contains every relevant failure; +- larger bounds cannot reveal a counterexample; +- the surrounding application and infrastructure are correct. + +Treat model checking as one layer of evidence alongside implementation review, unit tests, end-to-end recovery scenarios, and production monitoring. + +## Run the model checker + +The specification, configuration, and task file are located at: + +```text +sequencer/docs/recovery/ + preemptive.tla + preemptive.cfg + justfile +``` + +With TLC installed and available as `tlc`, run: + +```bash +cd sequencer/docs/recovery +just check-preemptive +``` + +The task executes: + +```bash +tlc -workers auto -deadlock preemptive.tla +``` + +You can point the task at another TLC executable with the `TLC` environment variable. Start with the committed bounds. Record the specification revision, configuration, TLC version, worker count, state count, runtime, and result for every verification run. + +## Interpreting a counterexample + +When TLC reports an invariant violation: + +1. identify the first transition after which the property becomes false; +2. determine whether the trace represents permitted production behavior; +3. check whether the specification, implementation, or invariant is wrong; +4. convert the trace into a focused implementation test when applicable; +5. rerun the original and corrected models with the same bounds; +6. increase relevant bounds to look for a larger related counterexample. + +A counterexample can reveal an incorrect design or an inaccurate model. It should not be dismissed solely because production code is structured differently. + +## Next steps + +- Review the modeled recovery path in [Preemptive recovery](../recovery/preemptive.md). +- Compare the checked properties with [Cross-module invariants](./invariants.md). +- Review unmodeled ordering behavior in [Scheduler semantics](./scheduler-semantics.md). diff --git a/app-sequencer/advanced/invariants.md b/app-sequencer/advanced/invariants.md new file mode 100644 index 000000000..0a56a2434 --- /dev/null +++ b/app-sequencer/advanced/invariants.md @@ -0,0 +1,183 @@ +--- +title: "Cross-module invariants" +sidebar_label: "Invariants" +description: "The properties that connect scheduler agreement, ordering, recovery, snapshots, wallet nonces, and checkpoint anchoring across the sequencer." +--- + +An invariant is a property that must hold in every legitimate execution, including restart, catch-up, and recovery. + +This page focuses on cross-module invariants. These are the properties whose definition, enforcement, and consumers live in different parts of the system. They deserve explicit documentation because a change can look correct within one component while breaking an assumption elsewhere. + +## Failure policy + +The sequencer uses a fail-loud policy for impossible internal states: + +1. **Reject the operation or stop the process.** Do not continue from a state that violates an internal contract. +2. **Do not invent a fallback result.** A neighboring component's output is not recomputed with a second algorithm to create an alternate path. +3. **Do not hide missing or contradictory state.** Required rows, snapshot references, nonces, and identities fail explicitly when absent or inconsistent. + +Availability can be restored after a visible stop. A silently externalized inconsistency can become a signed batch, incorrect confirmation, or misleading feed event and may require a full checkpoint rebuild. + +## Invalid input is not an invariant violation + +Untrusted callers can legitimately produce malformed signatures, application rejections, low-fee transactions, and arbitrary direct-input payloads. These cases are reachable by design and therefore have deterministic handling rules. + +A transaction-level rejection does not stop an accepted batch. The scheduler skips the affected transaction and continues. Treating caller-controlled invalid input as an impossible state would allow a public client to crash the sequencer. + +The distinction is: + +- **invalid external input** follows a defined rejection or skip path; +- **an impossible internal state** returns an error, violates a database constraint, or stops the process. + +## Enforcement mechanisms + +The invariants are maintained through several layers: + +| Mechanism | Examples | +| ------------------------------- | ---------------------------------------------------------------------------------------------- | +| Shared implementation | Checkpoint replay drives the canonical scheduler fold directly | +| SQLite transactions | Batch sealing and pending-snapshot insertion commit together | +| SQLite constraints and triggers | Batch nonce continuity, write-once hashes, and anchor immutability | +| Write ordering | Snapshot data is synchronized before its database reference is committed | +| Runtime checks | Chain identity, safe-head monotonicity, and content identity | +| Persisted failure markers | Canonical divergence freezes frontier progress across restart | +| Tests and model checking | Multi-round recovery, nonce-zero recovery, snapshot crash cases, and bounded slot-level safety | + +No single mechanism covers the whole system. Some of the most important agreement properties still rely on code review and tests. + +## Scheduler and ordering invariants + +### I1. Scheduler acceptance agrees across implementations + +The canonical scheduler, accepted-frontier predicate, and inclusion lane must make compatible decisions about ordering and batch acceptance. + +The checkpoint fold uses the canonical scheduler implementation directly. The accepted-frontier predicate is narrower and omits structural frame checks because it processes the sequencer's own sealed batches. The inclusion lane remains a live prediction whose agreement is maintained through shared types, tests, and review, not through a complete equivalence proof. + +If this property fails, the sequencer can confirm state the canonical machine will not reproduce. + +### I2. Drained direct inputs belong to the new frame + +When the safe frontier advances, newly covered direct inputs are sequenced into the frame carrying the new safe block. Canonical execution therefore observes: + +```text +direct inputs through safe block S +then transactions validated for frame S +``` + +Assigning those direct inputs to the earlier frame would make the live application evaluate transactions against a different state from the scheduler. + +### I3. Frame safe blocks never decrease along the valid path + +New frames begin at the current safe frontier, and safe-head persistence rejects backward movement. This supports the scheduler's structural checks and ensures that the first frame is the oldest frame for staleness testing. + +### I4. Tip-only recovery has no dangerous closed batch ahead of it + +The danger check evaluates closed batches before the open tip. Combined with non-decreasing safe blocks, an open-tip recovery decision means no non-accepted closed batch crossed the observed danger threshold first. + +This allows startup to replace the open tip without a wallet-nonce flush. The tip has no base-layer transaction to resolve. + +## Recovery and snapshot invariants + +### I5. Pending-snapshot cleanup is scoped to the invalidated suffix + +Recovery deletes pending snapshot references only at or after the cascade pivot, in the same transaction that invalidates the suffix and opens the replacement tip. + +Deleting a wider range could remove a pending snapshot for a batch that remains valid, causing promotion to fail later. Deleting a narrower range could let catch-up load state from an invalidated branch. + +### I6. A committed promotion includes the matching drain advance + +Snapshot promotion and safe-input drain advancement commit in one transaction. A restart cannot observe a promoted snapshot while still attempting to process the input that caused that promotion. + +### I7. A committed batch close has a pending snapshot + +The batch close, next-tip creation, and pending-snapshot row commit together. Snapshot files are created and synchronized before that transaction. + +Promotion can therefore require the pending row instead of handling a missing row as an ordinary condition. + +### I8. Runtime startup has loadable state and one valid open tip + +Plain setup registers the genesis finalized snapshot before writing its completion marker. Checkpoint recovery registers the reconstructed finalized snapshot and batch-tree anchor before completing. `run` refuses an incomplete setup, requires the finalized snapshot, and ensures that a valid open tip exists before starting the inclusion lane. + +The lane can follow one unconditional load-and-replay path instead of supporting an empty-state fallback. + +## Identity, nonce, and cursor invariants + +### I9. An accepted nonce identifies the matching local batch content + +For each fully accepted landing at or above the batch-tree anchor, the landed payload hash must match the seal-time hash of the valid closed local batch at the same nonce. + +A missing local batch or different hash records canonical divergence and freezes the frontier. See [Divergence and the content-identity check](./divergence.md). + +### I10. Feed offset zero means replay from genesis + +Valid sequenced-feed rows begin at offset 1 and are append-only. Offset 0 is reserved as the sentinel meaning no row has been consumed. + +Catch-up and subscription code can use one comparison, `offset > cursor`, without confusing a real transaction with the genesis position. + +### I11. The sequencer's batch inputs are recorded but never executed as direct inputs + +Safe inputs from the batch-submitter address participate in ordering and cursor advancement. They must not be passed to the application as direct inputs or emitted to feed consumers as application transactions. + +Sender checks enforce this at catch-up replay, live execution, and feed delivery. Those consumers must remain synchronized. + +### I12. Safe-head timestamps represent genuine progress + +The input reader advances the persisted safe head only after observing a higher safe block, apart from recording the initial observation, and records synchronization time with that committed progress. Repeated reads of the same head do not refresh the progress timestamp. + +The stale-view and wall-clock danger checks rely on this timestamp. Refreshing it without progress would hide an outage. + +### I13. A referenced snapshot dump exists and is complete + +Snapshot creation writes and synchronizes the dump before inserting its database row. Cleanup removes the row before deleting the filesystem directory. Startup sweeps unreferenced directories and resets stale leases. + +This ordering prevents the normal creation and cleanup paths from leaving a committed snapshot reference to an incomplete dump, subject to the storage durability assumptions described in [Data, snapshots, and backups](../operations/data-and-state.md#durability-and-crash-guarantees). + +### I14. The wallet-nonce watermark covers every broadcast nonce + +Before the sequencer broadcasts a batch or flush transaction at wallet nonce `W`, it durably raises the stored watermark to at least W. Recovery completes only after the safe nonce has passed that watermark and the pending nonce is no greater than the safe nonce. + +This prevents a locally forgotten transaction from surviving outside the flushed range while an intact database is available. A fresh checkpoint-recovery database lacks the original watermark, which is documented as a separate residual risk in [Cockroach recovery](../recovery/cockroach.md#resolve-the-submitter-wallet). + +### I15. A divergence marker freezes the accepted frontier + +The marker is written atomically with the synchronization that detects a foreign or mismatched accepted batch. Frontier population returns immediately whenever the marker exists, and the danger check gives divergence higher priority than every recovery condition. + +This prevents normal recovery, batch promotion, or restart from advancing through known-divergent history. + +### I16. The valid batch tree has one parentless root at the deployment anchor + +A genesis deployment uses anchor nonce 0. A checkpoint-recovered deployment uses the replay result N'. The parentless valid root must carry exactly that anchor, and every child carries its parent's nonce plus one. + +Database triggers limit the tree to one valid parentless root and enforce nonce continuity. The anchor becomes immutable after setup completes. This lets a recovered deployment resume at N' without creating fake historical batch rows. + +## Invariants, assumptions, and guarantees + +An invariant enforced by the implementation is not automatically an end-to-end guarantee. Some properties depend on environmental assumptions: + +- the RPC endpoint provides an honest and internally consistent safe view; +- the base layer and `InputBox` follow their contracts; +- the host clock and configured block time are suitable for outage estimation; +- the application is deterministic across the live and canonical environments; +- storage honors synchronized writes; +- operator-supplied checkpoint state and metadata are genuine. + +If an environmental assumption is encoded as an unconditional invariant, normal behavior can trigger a false failure. For example, wall-clock time can move backward, so elapsed-time code uses a guarded calculation instead of asserting monotonic system time. + +## Review checklist for invariant changes + +When changing an enforcement point: + +1. identify every reader that depends on the property; +2. verify crash behavior before and after each commit boundary; +3. test restart, replay, and recovery paths, not only steady state; +4. preserve the distinction between invalid input and impossible state; +5. confirm that errors remain visible and do not create a second source of truth; +6. update the repository's detailed invariant register with new enforcement and dependency locations. + +Names and module boundaries can change. Treat this page as a map of relationships, not as a stable internal API. + +## Next steps + +- Read the canonical rules in [Scheduler semantics](./scheduler-semantics.md). +- Review terminal disagreement handling in [Divergence and the content-identity check](./divergence.md). +- Understand the model-checked subset in [Formal verification](./formal-verification.md). diff --git a/app-sequencer/advanced/scheduler-semantics.md b/app-sequencer/advanced/scheduler-semantics.md new file mode 100644 index 000000000..e8619f829 --- /dev/null +++ b/app-sequencer/advanced/scheduler-semantics.md @@ -0,0 +1,146 @@ +--- +title: "Scheduler semantics" +sidebar_label: "Scheduler semantics" +description: "The scheduler's deterministic input-ordering algorithm, batch acceptance gates, direct-input backstop, and nonce effects." +--- + +The scheduler defines the canonical order in which direct inputs and sequenced transactions affect application state. The Cartesi machine runs this algorithm, while the sequencer predicts the same outcome before base-layer settlement. + +Agreement between those paths is essential. A difference can make a soft-confirmed result disagree with canonical execution. See [Divergence and the content-identity check](./divergence.md). + +## Input stream and sender classification + +The scheduler processes `InputBox` inputs in base-layer order. Each input provides: + +- an authenticated sender; +- an inclusion block; +- an opaque payload. + +Classification uses only the sender address: + +| Sender | Classification | Initial action | +| ---------------------------------- | -------------- | ----------------------------------------------- | +| Configured batch-submitter address | Batch | Decode and evaluate the batch acceptance gates | +| Any other address | Direct input | Add the input to the waiting direct-input queue | + +The payload does not contain a trusted tag that can override this classification. Application-specific decoding occurs after the scheduler has selected the path. + +## Processing algorithm + +For each `InputBox` input, in order, the scheduler performs: + +```text +1. Execute every overdue direct input using this input's inclusion block. +2. Classify the new input by sender. +3. If it is direct, enqueue it. +4. If it is a batch, evaluate the batch acceptance gates. +``` + +The overdue-input check runs before classification. A malformed, stale, or wrong-nonce batch still advances the block reference used by the censorship backstop. + +## Direct-input censorship backstop + +A waiting direct input becomes overdue when: + +```text +current input block - direct input block >= 1200 +``` + +The scheduler executes all overdue direct inputs in queue order before processing the new input. Each executed input is removed from the queue, preventing a second execution. + +The value `1200` is the shared protocol constant `MAX_WAIT_BLOCKS`. It is also used by batch staleness, linking the maximum delay for direct inputs to the maximum age of a sequenced batch. + +The backstop is event-driven. Time passing or blocks being produced does not invoke the scheduler by itself. If an application receives no new inputs, an overdue direct input remains queued until another input arrives. Any sender can trigger evaluation by adding an application input, so progress does not depend on the sequencer returning. + +See [Direct inputs vs sequenced transactions](../concepts/direct-vs-sequenced.md). + +## Batch acceptance gates + +A payload from the batch-submitter address passes through these gates in order. The first failed gate determines the result. + +| Order | Gate | Result when the gate fails | Batch nonce consumed? | +| ----- | ------------------------------------- | -------------------------- | --------------------- | +| 1 | Decode the payload as a batch | Reject as undecodable | No | +| 2 | Match the next expected batch nonce | Reject with wrong nonce | No | +| 3 | Check whether the batch has no frames | Accept as an empty no-op | **Yes** | +| 4 | Validate frame structure | Reject as malformed | No | +| 5 | Check batch staleness | Skip as stale | No | +| 6 | Execute every frame | Accept and execute | **Yes** | + +Only an accepted batch advances the next expected batch nonce. An empty batch is accepted because it has no frame state to validate or age to measure. + +## Frame structure requirements + +A non-empty batch is structurally valid only when: + +1. every frame's safe block is no greater than the batch's inclusion block; +2. frame safe blocks are non-decreasing within the batch. + +The first rule prevents a frame from claiming knowledge of base-layer activity that had not occurred when the batch was included. The second ensures that the sequencer's claimed base-layer view moves only forward. + +These requirements also justify testing staleness against the first frame. Because safe blocks are non-decreasing, the first frame has the oldest safe block and therefore the greatest age. + +## Batch staleness + +For a non-empty batch, the scheduler evaluates: + +```text +batch inclusion block - first frame safe block >= 1200 +``` + +When the expression is true, the batch is skipped as stale. No application state changes, and the expected batch nonce does not advance. + +The unchanged nonce makes later numbered batches ineligible until a valid replacement uses the expected value. [Staleness and the danger zone](../concepts/staleness.md) gives the complete numbering example, and [Preemptive recovery](../recovery/preemptive.md) explains repair. + +## Frame execution order + +For every frame in an accepted batch, the scheduler performs two steps: + +1. **Drain covered direct inputs.** Execute queued direct inputs whose inclusion blocks are at or before the frame's safe block. +2. **Execute sequenced transactions.** Evaluate the frame's transactions on the resulting application state. + +This order gives the safe block its meaning. A frame claiming safe block 500 promises that every direct input through block 500 is reflected before that frame's transactions run. + +The sequencer follows the same drain-first order locally when accepting a transaction. This allows it to evaluate the transaction against the state the canonical scheduler is expected to reproduce. + +## Transaction-level outcomes + +Each transaction in an accepted frame enters one validation and execution path. The transaction is skipped without changing application state when: + +- its signature cannot identify a sender; +- its maximum fee is below the frame's fee price; +- the application rejects it. + +A skipped transaction does not reject the frame or batch. Execution continues with the next transaction, and the accepted batch still consumes its nonce. + +These are defined responses to caller-controlled input, not invariant violations. The scheduler and sequencer must produce the same result for each case. + +## Where the semantics are implemented + +The acceptance behavior appears in three forms: + +| Component | Responsibility | +| ------------------------------ | ----------------------------------------------------------------------------------- | +| Canonical `Scheduler` fold | Executes the complete algorithm in the Cartesi machine and during checkpoint replay | +| Off-chain acceptance predicate | Determines which safe base-layer batches advance the accepted frontier | +| Inclusion lane prediction | Applies direct inputs and transactions locally before settlement | + +The checkpoint recovery fold drives the canonical scheduler implementation directly, avoiding a parallel implementation of the algorithm. + +The off-chain acceptance predicate has a narrower role. It checks sender, decoding, nonce, and staleness, but omits the two structural frame checks. The predicate processes the sequencer's own sealed submissions, which are assumed well formed under the codebase's self-trust model. A malformed self-submission is treated as a sequencer defect, not as normal hostile input. + +Agreement across these components is maintained by shared code where possible, tests, and review. There is no complete mechanical proof that the live prediction and canonical fold are equivalent for every application. + +## Worked ordering example + +The following example combines two queued deposits with a two-frame batch. Frame 1 covers direct inputs through block 103, while frame 2 advances the safe block to 108. + +![The base-layer stream contains deposit A at block 100, deposit B at block 106, and an accepted two-frame batch. The scheduler drains deposit A before frame 1, executes transactions X and Y, drains deposit B before frame 2, and then executes transaction Z.](../images/scheduler-ordering-example.jpg) + +Deposit B remains queued during frame 1 because block 106 is above safe block 103. Frame 2 advances the claimed view to 108 and drains it before transaction Z. + +## Next steps + +- Learn how batches encode these rules in [Batches, frames, and the safe block](../concepts/batches-frames-safe-block.md). +- Review cross-module dependencies in [Invariants](./invariants.md). +- Understand disagreement handling in [Divergence and the content-identity check](./divergence.md). diff --git a/app-sequencer/advanced/threat-model.md b/app-sequencer/advanced/threat-model.md new file mode 100644 index 000000000..bc5a38cbb --- /dev/null +++ b/app-sequencer/advanced/threat-model.md @@ -0,0 +1,192 @@ +--- +title: "Threat model" +sidebar_label: "Threat model" +description: "The assets, trust boundaries, adversarial behavior, environmental assumptions, security controls, and residual risks of the app-specific sequencer." +--- + +The threat model defines the behavior the sequencer is designed to withstand and the assumptions operators must preserve. A correctness failure is security-relevant because it can change application state, mislead clients, or affect user assets even when no attacker directly steals a key. + +## Protected assets + +The design protects: + +- **Canonical application-state integrity.** Base-layer replay must produce the deterministic state defined by scheduler ordering. +- **Soft-confirmation honesty.** Clients must be able to distinguish provisional acceptance from base-layer settlement and detect recovery invalidation. +- **User operations and direct inputs.** Valid inputs must not be silently lost, duplicated, reordered outside the protocol, or attributed to the wrong sender. +- **Batch-submitter identity and key.** Only the authorized deployment should submit batches from the configured account. +- **Recovery integrity.** Batch nonces, wallet nonces, checkpoints, and accepted-frontier state must not resume from an incorrect position. +- **Feed and snapshot consistency.** Consumer offsets and served application state must correspond to the sequencer state they claim to represent. + +Availability is important but subordinate to state integrity. The sequencer stops when continued operation could externalize an internal contradiction. + +## Actors and trust boundaries + +| Actor or component | Trust level | Security assumptions and capabilities | +| ---------------------------------- | ------------------------------- | -------------------------------------------------------------------------------------------------------------- | +| Base-layer consensus and safe view | Trusted protocol dependency | Provides ordered blocks and the semantics represented by the RPC `safe` tag | +| `InputBox` contract | Trusted | Authenticates the original sender and assigns application inputs in order | +| Configured RPC endpoint | Trusted, fail-stop | May become unavailable, but must not fabricate a consistent false chain, safe head, log set, or contract state | +| Mempool and block builders | Adversarial | May delay, reorder, drop, retain privately, replace, or selectively include transactions | +| Public transaction clients | Untrusted | May submit malformed signatures, replayed payloads, invalid nonces, low fees, and application-specific attacks | +| Direct-input senders | Untrusted | May submit arbitrary application payloads through the `InputBox` | +| WebSocket and snapshot consumers | Untrusted readers | Cannot directly mutate sequencer state but may create load and receive data exposed by the shared listener | +| Operator configuration | Trusted | Defines chain identity, endpoints, timing, addresses, storage, and process policy | +| Submitter key and host secrets | Trusted and confidential | Must remain inaccessible to public clients and unrelated workloads | +| Local host and storage | Trusted | Must enforce access control and preserve synchronized writes | +| Sequencer and application code | Trusted for correctness | Determinism and protocol agreement are preconditions supported by review, tests, and monitoring | +| External gateway and supervisor | Trusted operator infrastructure | Enforces exposure policy, rate limits, TLS, restart behavior, and terminal-exit handling | + +## Adversarial base-layer transaction handling + +The mempool and block builders may delay, reorder, replace, retain, or later publish a transaction that one provider no longer reports. The design therefore requires two independent properties: uncertain submitter nonce slots are resolved before a provisional branch is replaced, and accepted batch content matches the local sealed content for the same batch nonce. + +[Flush unresolved wallet nonces](../operations/orchestration.md#flush-unresolved-wallet-nonces) defines the first control. [Divergence and the content-identity check](./divergence.md) defines the second and explains why an application-state watchdog remains necessary. + +## Untrusted transaction clients + +`POST /tx` is a public trust boundary. The sequencer validates: + +- request size and encoding; +- the EIP-712 signature and recovered sender; +- the deployment-specific signing domain; +- the sender's application nonce; +- the maximum fee against the current price; +- application-specific transaction rules. + +Invalid requests receive a defined HTTP error before entering a batch. The canonical scheduler repeats signature, fee, and application validation when executing batch content because base-layer data cannot be trusted solely because it appears to come from the submitter address. + +Infrastructure must still rate-limit the route. Validation does not make unlimited parsing, signature recovery, or application simulation free. + +## Untrusted direct-input senders + +The `InputBox` authenticates a direct input's sender, but the payload remains untrusted. The application must validate method identifiers, token addresses, lengths, ranges, permissions, and state transitions. + +Direct inputs cannot be rejected by the public sequencer API because they bypass it. The canonical application implementation must handle every possible direct payload deterministically without panicking or consuming unbounded resources. + +## RPC trust and consistency + +The code accepts one configured RPC URL shared across reading, submission, flushing, and recovery. The model assumes responses form one truthful and internally consistent safe view. + +The implementation adds several checks: + +- chain ID is verified before initial setup and before keyed writes; +- the input reader verifies per-application input indexes are contiguous; +- a contract input-count query at the scanned safe block confirms the log range is complete; +- safe-head movement is persisted monotonically; +- recovery refuses to cascade if its post-flush synchronization is behind the block where the flush observed nonce resolution. + +These controls detect wrong-chain configuration and many incomplete or lagging responses. They do not make a Byzantine RPC safe. A provider that fabricates logs and matching contract state consistently is outside the model. + +The checks do not make a Byzantine RPC safe. A provider that fabricates mutually consistent logs and contract state is outside the model. [Production security](../operations/security.md#use-one-consistent-rpc-source) defines the deployment controls for preserving this assumption. + +## Base-layer reorganization boundary + +The input reader follows the RPC endpoint's safe view, not the latest head. Reorganizations that occur before data enters that view are absorbed by the base layer and are not persisted as canonical sequencer inputs. + +The design assumes previously safe data will not be reorganized away beyond the base layer's promised semantics. A deep reorganization that invalidates an already processed safe block falls outside the normal recovery model. + +The lifecycle state called a finalized snapshot inherits this same boundary. It indicates promotion from the safe view and does not claim absolute irreversibility. + +## Self-trust and fail-loud behavior + +The sequencer treats its own code and the application as correct protocol participants. Automatic recovery targets crashes, outages, delayed transactions, and provisional batch failure. It does not attempt to repair a malformed self-submission or nondeterministic application by guessing the intended state. + +Near-free internal checks remain valuable. Types, database constraints, triggers, assertions, content hashes, and persisted markers stop the process when an impossible state is observed. This is fail-loud enforcement, not a graceful fallback. + +See [Cross-module invariants](./invariants.md). + +## Environmental assumptions + +### Deterministic application execution + +The live application, canonical Cartesi machine, checkpoint loader, and watchdog must interpret the same inputs identically. Application state serialization used for watchdog comparison must also be deterministic. + +Deploy compatible application and sequencer versions together. A byte-identical batch is not enough if two runtimes execute it differently. + +### Block-time estimation + +When the provider is unavailable, the sequencer estimates missed blocks using: + +```text +elapsed wall-clock seconds / configured seconds per block +``` + +This assumes the configured average is suitable for the target chain and the host clock does not drift significantly. The estimate is used only as a conservative refusal signal. Startup does not modify recovery state solely from estimated danger; it waits for a usable base-layer view. + +### Single active submitter + +Only one keyed writer may use the deployment's data directory and submitter account at a time. [Process supervision and recovery operations](../operations/orchestration.md#prevent-overlapping-sequencer-instances) defines the excluded combinations. + +### Durable local storage + +The crash model assumes storage preserves acknowledged synchronized writes. [Data, snapshots, and backups](../operations/data-and-state.md#durability-and-crash-guarantees) defines the database and filesystem contract. + +### Trusted checkpoint archives + +Checkpoint recovery trusts the archived application state and next batch nonce. [Snapshots and checkpoints](../recovery/snapshots.md#validate-checkpoint-archives) defines archive validation and its limits. + +## In-scope failures and attacks + +The design explicitly considers: + +- temporary and prolonged RPC outages; +- process crashes at arbitrary runtime points; +- delayed, reordered, dropped, replaced, and resurfacing base-layer transactions; +- base-layer movement before the configured safe view; +- malformed, replayed, incorrectly signed, low-fee, and application-invalid public transactions; +- arbitrary direct-input payloads from authenticated base-layer senders; +- wrong-chain RPC configuration; +- incomplete log ranges and inconsistent post-flush synchronization; +- stale batches, nonce poisoning, and repeated recovery rounds; +- foreign or mismatched accepted batch content; +- local database loss handled through a trusted checkpoint. + +## Out-of-scope conditions + +The Rust sequencer does not itself provide complete protection against: + +- denial of service, rate limiting, and unbounded client traffic; +- a Byzantine RPC endpoint that lies consistently; +- a compromised `InputBox` or base-layer consensus protocol; +- host compromise, submitter-key theft, or malicious operator configuration; +- secret encryption and secrets-manager policy; +- dependency or build-system supply-chain compromise; +- application or sequencer defects as adversarially exploitable behavior; +- storage hardware that loses acknowledged durable writes; +- deep reorganizations of previously safe base-layer data. + +These conditions require infrastructure controls, contract assurance, supply-chain security, code review, or incident-specific remediation. + +## Residual risks + +Important risks remain even when the stated assumptions hold: + +- soft confirmations can be revoked during preemptive recovery; +- divergence is detected only after the relevant landing reaches the safe view; +- checkpoint recovery from a fresh database lacks the old wallet-nonce watermark; +- the content-identity check compares batch bytes, not application state; +- snapshot and WebSocket routes have no built-in authentication on the current shared listener; +- model checking covers a bounded subset of recovery and has no mechanical link to the code; +- public validation can still consume resources before rejecting a request. + +These are operational design constraints, not reasons to ignore the controls. Document them in client behavior, monitoring, key management, and incident runbooks. + +## Security review questions + +For each change, ask: + +1. Which actor supplies every input to the changed path? +2. Can untrusted data reach a database write, signed transaction, application transition, feed event, or process-control decision? +3. Does the change assume the mempool forgets transactions permanently? +4. Does it rely on a safe head or wallet nonce observed from a different RPC view? +5. Could a crash occur after an external effect but before the corresponding durable record? +6. Does a new fallback hide an invariant violation or create a second source of truth? +7. Can a correctness failure affect assets or confirmations even without direct exploitation? +8. Which recovery, snapshot, divergence, and restart tests demonstrate the intended boundary? + +## Next steps + +- Apply operational controls from [Production security](../operations/security.md). +- Review failure handling in [Failure modes](../recovery/failure-modes.md). +- Understand the supporting properties in [Cross-module invariants](./invariants.md). +- Review verification scope in [Formal verification](./formal-verification.md). diff --git a/app-sequencer/api-reference/api.md b/app-sequencer/api-reference/api.md new file mode 100644 index 000000000..1d2d961d7 --- /dev/null +++ b/app-sequencer/api-reference/api.md @@ -0,0 +1,154 @@ +--- +title: "HTTP and WebSocket API" +sidebar_label: "HTTP and WebSocket API" +description: "Every endpoint the sequencer exposes, with request and response shapes, error codes, and which endpoints are public." +--- + +The sequencer exposes a small public surface for applications, plus a set of internal endpoints for operators. + +By default it listens on `127.0.0.1:3000`, which is local only. See [Configure, set up, and run the sequencer](../operations/setup-and-running.md). + +## Public endpoints + +### POST /tx + +Submit a signed transaction. Explained in [Submitting operations](../usage/submitting-operations.md). + +Request: + +```json +{ + "message": { + "nonce": 0, + "max_fee": 1, + "data": "0x..." + }, + "signature": "0x...", + "sender": "0x..." +} +``` + +| Field | Type | Notes | +| ----------------- | ---------- | --------------------------------------------------- | +| `message.nonce` | `uint32` | Per sender, starting at 0 | +| `message.max_fee` | `uint16` | Fee exponent, base 129/128 | +| `message.data` | hex string | Application payload | +| `signature` | hex string | EIP-712 signature, exactly 65 bytes | +| `sender` | hex string | Must match the address recovered from the signature | + +Success, HTTP `200`: + +```json +{ "ok": true, "sender": "0x...", "nonce": 0 } +``` + +Errors carry a `code` field. Branch on the code, not the message. + +| Status | Code | Cause | +| ------ | -------------------- | --------------------------------------------------------------------------------------------------------------------------- | +| `400` | `BAD_REQUEST` | Malformed request, or an application payload larger than the application's declared maximum. Note this is `400`, not `413`. | +| `400` | `INVALID_SIGNATURE` | Signature invalid, or `sender` does not match the recovered signer | +| `413` | `PAYLOAD_TOO_LARGE` | The raw JSON body exceeds 4 KiB | +| `422` | `EXECUTION_REJECTED` | The application refused the transaction | +| `429` | `OVERLOADED` | Ordering queue full, message `queue full`. Retry. | +| `503` | `UNAVAILABLE` | Shutting down or not ready | +| `500` | `INTERNAL_ERROR` | Unexpected failure | + +### GET /ws/subscribe + +``` +GET /ws/subscribe?from_offset= +``` + +WebSocket stream of sequenced transactions in execution order. Explained in [Reading the sequenced feed](../usage/reading-the-feed.md). + +`from_offset` is optional, default `0`, and is **exclusive**: delivery starts at the first transaction after it. + +Offsets are database row ids. They start at `1`, ascend, and are not guaranteed contiguous. A client resumes by storing the last offset it received, never by incrementing a counter. + +Messages are JSON text frames. Binary fields are hex encoded with a `0x` prefix. + +```json +{ + "kind": "user_op", + "offset": 10, + "sender": "0x...", + "fee": 1, + "data": "0x..." +} +``` + +```json +{ + "kind": "direct_input", + "offset": 11, + "sender": "0x...", + "block_number": 123, + "payload": "0x..." +} +``` + +There are no other message kinds. In particular there is no message signalling that an earlier transaction was invalidated. + +Limits: + +| Limit | Value | Behaviour when exceeded | +| ---------------------- | ------------- | ------------------------------------------------------------------------------------------ | +| Concurrent subscribers | 64 | Further connections get HTTP `429` with `OVERLOADED`, before the WebSocket upgrade | +| Catch-up window | 50,000 events | Socket is upgraded then immediately closed, code `1008`, reason `catch-up window exceeded` | + +## Batch wire format + +An integrator or auditor reading the base layer directly needs to understand the batch structure posted by the sequencer and decoded by the machine. + +A batch is **SSZ encoded**, and posted as the raw encoding with no wrapper or tag. There is nothing in the payload saying what it is: classification is by sender address alone, so an input from the sequencer's address is decoded as a batch and anything else is a direct input. See [Scheduler semantics](../advanced/scheduler-semantics.md). + +``` +Batch + nonce uint64 + frames list of Frame + +Frame + user_ops list of WireUserOp + safe_block uint64 + fee_price uint16 fee exponent, base 129/128 + +WireUserOp + nonce uint32 + fee uint16 the sender's max fee, same encoding + ... signature, sender and payload +``` + +Keep these wire-format details distinct from the sequencer's local representation: + +- **A batch carries no parent reference.** Its `nonce` identifies its position, and the machine expects the next number. The parent links in [The batch tree](../concepts/batch-tree.md) exist only in the sequencer's local record and are absent from the posted data. +- **`safe_block` and `fee_price` are per frame, not per batch.** A batch can carry several frames with advancing safe blocks, which is the mechanism described in [Batches, frames, and the safe block](../concepts/batches-frames-safe-block.md). + +Use the field names and ordering in `sequencer-core` for the deployed release. The encoding is consensus-critical, and this page provides only a summary. + +## Health endpoints + +| Endpoint | Checks performed | +| -------------- | -------------------------------------------------------------- | +| `GET /livez` | The process is alive | +| `GET /readyz` | That it is not shutting down and the inclusion lane is running | +| `GET /healthz` | The same readiness signal, returning a small status body | + +## Operator endpoints + +:::warning Internal only +These serve application state to an operator's own watchdog and indexers. They have **no authentication** and must not be exposed publicly. Keep them behind network controls. +::: + +| Endpoint | Returns | +| -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `GET /finalized_state/inclusion_block` | Cheap JSON for detecting progress: `{ "inclusion_block": , "l2_tx_index": }`. `404` if no finalized snapshot exists yet. | +| `GET /finalized_state` | The settled state file, as `application/octet-stream`. Headers `X-Inclusion-Block`, `X-L2-Tx-Index`, and `ETag: "block-"`. Send `If-None-Match` to get a `304` when unchanged. | +| `GET /latest_snapshot` | The most recent snapshot, pending if there is one, otherwise finalized. Intended for an indexer that fetches state and then subscribes from `X-L2-Tx-Index`. | + +Both streaming endpoints hold the snapshot open for the life of the response, including when a client disconnects early. + +## Next steps + +- For the signing domain, see [EIP-712 domain](./eip712.md). +- For settings and exit codes, see [Reference](/app-sequencer/reference). diff --git a/app-sequencer/api-reference/constants-and-exit-codes.md b/app-sequencer/api-reference/constants-and-exit-codes.md new file mode 100644 index 000000000..3675fb923 --- /dev/null +++ b/app-sequencer/api-reference/constants-and-exit-codes.md @@ -0,0 +1,65 @@ +--- +title: "Constants and exit codes" +sidebar_label: "Constants and exit codes" +description: "Fixed protocol values, the settings that relate to them, API limits, and the process exit codes a supervisor must handle." +--- + +Two kinds of fixed value an integrator or operator needs to look up: the constants the protocol is built on, and the codes the process exits with. + +## Protocol constants + +Some values are part of the protocol and cannot be changed by an operator. Others are local settings. The difference matters: a protocol constant is compiled into the application's machine, so both sides agree on it by construction. + +### Fixed by the protocol + +| Constant | Value | Meaning | +| ---------------------- | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Staleness deadline | `1200` blocks | A batch that reaches the base layer this long after the block it names is skipped. The same figure bounds how long a direct input can be delayed. Roughly 4 hours where blocks are twelve seconds apart. | +| EIP-712 domain name | `CartesiAppSequencer` | See [EIP-712 domain](./eip712.md) | +| EIP-712 domain version | `1` | | +| Signature length | 65 bytes | | +| Fee encoding base | `129/128` | Fees are exponents. An exponent `n` means `(129/128)^n` smallest units. About 0.78 percent per step. | + +The staleness deadline is compiled into the machine and cannot be changed by an operator. A sequencer therefore cannot extend the time it holds a direct input. + +### Set by the operator + +These are local tuning, and each relates to the deadline above. + +| Setting | Default | Meaning | +| ---------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------- | +| `CARTESI_SEQUENCER_PREEMPTIVE_MARGIN_BLOCKS` | `300` | How far before the deadline the sequencer stops to avoid losing work. Must be below the deadline. Validated at startup. | +| `CARTESI_SEQUENCER_L1_READ_STALE_AFTER_BLOCKS` | `600` | When the sequencer's view of the base layer is too old to trust. Must be below the danger threshold. | +| `CARTESI_SEQUENCER_SECONDS_PER_BLOCK` | `12` | Assumed block time, used to reason about elapsed time during an outage. | + +The defaults assume a chain with twelve second blocks. On a chain with a different block time these are the first things to revisit, because what they really express is a duration. + +### Limits on the API + +| Limit | Value | +| -------------------- | ------------- | +| Feed subscribers | 64 | +| Feed catch-up window | 50,000 events | + +## Exit codes + +The sequencer exits with a code that tells a supervisor what to do next. Reacting to these correctly is what makes automated operation safe: some exits should be retried immediately, and some must not be. + +| Code | Meaning | +| ----- | ------- | +| `0` | Clean shutdown | +| `1` | Unclassified runtime, worker, storage, or I/O failure | +| `2` | Invalid command or configuration | +| `10` | Restart with startup recovery expected | +| `20` | Transient refusal | +| `30` | Terminal condition requiring operator investigation | +| `40` | Setup requires explicit checkpoint recovery | +| `101` | Rust panic | + +The code is the stable process-control signal. [Process supervision and recovery operations](../operations/orchestration.md#exit-codes-and-required-actions) defines backoff, alerting, startup windows, and the commands an operator should run. [Cockroach recovery](../recovery/cockroach.md) defines the procedure associated with code `40`. + +## Next steps + +- For the endpoints and their limits, see [HTTP and WebSocket API](./api.md). +- For settings, see [Configure, set up, and run the sequencer](../operations/setup-and-running.md). +- For how a supervisor should react to an exit, see [Process supervision and recovery operations](../operations/orchestration.md). diff --git a/app-sequencer/api-reference/eip712.md b/app-sequencer/api-reference/eip712.md new file mode 100644 index 000000000..52cb9b6fd --- /dev/null +++ b/app-sequencer/api-reference/eip712.md @@ -0,0 +1,47 @@ +--- +title: "EIP-712 domain" +sidebar_label: "EIP-712 domain" +description: "The exact typed-data domain and struct used to sign transactions for the sequencer." +--- + +Transactions sent to the sequencer are signed as EIP-712 typed data, so a wallet can show a user what they are approving. Every field below must match exactly, because a domain mismatch produces a signature that recovers to the wrong address and is rejected. + +## Domain + +| Field | Value | +| ------------------- | --------------------------------------------------------- | +| `name` | `CartesiAppSequencer` | +| `version` | `1` | +| `chainId` | The chain id of the base layer the application settles on | +| `verifyingContract` | The application's address on the base layer | + +`name` and `version` are fixed for every deployment. `chainId` and `verifyingContract` are the values the sequencer was set up with. + +Pinning both means a signature is valid for exactly one application on one chain, so a transaction cannot be replayed against another deployment. + +## Type + +```solidity +struct UserOp { + uint32 nonce; + uint16 max_fee; + bytes data; +} +``` + +| Field | Meaning | +| --------- | --------------------------------------------------------------------------------- | +| `nonce` | Per-sender counter, starting at 0 | +| `max_fee` | Fee exponent, base 129/128. See [Fees and data availability](../concepts/fees.md) | +| `data` | The application payload | + +Standard EIP-712 encoding hashes `data` before including it in the typed message. + +## Signature + +The signature is 65 bytes. The `sender` field sent alongside it must match the address recovered from it, or the submission is rejected. + +## Next steps + +- To submit a signed transaction, see [Submitting operations](../usage/submitting-operations.md). +- For the endpoint that accepts a signed transaction, see the [HTTP and WebSocket API](./api.md). diff --git a/app-sequencer/concepts/batch-tree.md b/app-sequencer/concepts/batch-tree.md new file mode 100644 index 000000000..1723e7e9d --- /dev/null +++ b/app-sequencer/concepts/batch-tree.md @@ -0,0 +1,42 @@ +--- +title: "The batch tree" +sidebar_label: "The batch tree" +description: "How the sequencer records alternative batch histories and uses an anchor to identify where local history begins." +--- + +The sequencer keeps its own record of every batch it has built and how they relate. In normal running that record is a straight line, one batch after another. It is a tree because failure sometimes makes it branch. + +## Where the batch tree exists + +A batch **as posted to the base layer** carries only its number and its frames. It contains no parent field. The machine needs only the numbered order because that order defines the relationship between batches. See [HTTP and WebSocket API](../api-reference/api.md) for the exact structure. + +The parent links described on this page exist only in the **sequencer's own record**. They are how it tracks what it built and what it has abandoned. Nothing about this tree is visible to the application or to anyone reading the base layer. + +## How recovery creates branches + +A batch can reach the base layer too late and be skipped, taking every batch behind it out of contention. See [Staleness and the danger zone](./staleness.md). + +When that happens, the work built on the doomed batches is no longer viable, but the sequencer cannot just delete it, because until the base layer has settled it does not yet know for certain what happened. What it does instead is mark the doomed line as invalid and start a fresh line from the last batch that is still good. + +This creates a branch with one parent, an abandoned path, and a live path. The tree preserves both histories, allowing recovery to invalidate old work without deleting its record. + +![A batch-tree anchor leads to the last accepted parent, where history branches. The abandoned provisional branch remains in local history as invalid, while a replacement branch begins from the accepted parent and becomes the current valid tip.](../images/batch-tree-recovery.png) + +## Where batch history begins + +Every deployment's record starts somewhere, and that starting number is the **anchor**. + +For a fresh deployment, the anchor is zero because no prior history exists. After a checkpoint rebuild, the anchor is the number at which the rebuilt local history begins. The database contains no earlier batches. See [Cockroach recovery](../recovery/cockroach.md). + +The anchor is what stops a rebuilt deployment from believing it should begin at zero, which would collide with everything already on the base layer. Its record has exactly one starting point, and everything else descends from it. + +## How the tree supports soft confirmations + +Mostly it does not. The tree is the sequencer's internal bookkeeping, and an application never sees it. + +It matters indirectly, in one way. The tree is how the sequencer can tell the difference between work that is settled, work that is still a prediction, and work that has been abandoned. That distinction is what a soft confirmation ultimately rests on. See [Soft confirmations](./soft-confirmations.md). + +## Related concepts + +- For what invalidates a line, read [Staleness and the danger zone](./staleness.md). +- For how a doomed line is cleared, read [Preemptive recovery](../recovery/preemptive.md). diff --git a/app-sequencer/concepts/batches-frames-safe-block.md b/app-sequencer/concepts/batches-frames-safe-block.md new file mode 100644 index 000000000..0e45a142e --- /dev/null +++ b/app-sequencer/concepts/batches-frames-safe-block.md @@ -0,0 +1,82 @@ +--- +title: "Batches, frames, and the safe block" +sidebar_label: "Batches, frames, and the safe block" +description: "How the sequencer packages transactions, what a frame carries, and how the safe block keeps the sequencer and the machine in one order." +--- + +The sequencer does not post transactions to the base layer one at a time. It packages them, and the shape of that package is what lets the machine reconstruct the same order without trusting the sequencer. + +## The structure of a batch + +A batch contains transactions, frames, and a safe block reference. + +- A **batch** is what the sequencer posts to the base layer. It carries a number, its nonce, and a list of frames. +- A **frame** is a group of transactions inside a batch. It carries a **safe block** and a fee price alongside its transactions. +- The **safe block** is a base-layer block number, and it is the instruction that ties the two sides together. + +## How the safe block determines execution order + +A frame's safe block is a statement by the sequencer: _I have accounted for everything that arrived directly on the base layer up to this block._ + +The machine does not take that on trust. It acts on it. Before running a frame's transactions, it runs every direct input recorded at or before that frame's safe block. The sequencer's claim and the machine's behaviour are the same rule, so both end up interleaving the two sources of transactions identically. + +This is why the sequencer can answer immediately and still be right. It applies the same rule locally when it accepts a transaction, so the state it judges that transaction against already contains the direct inputs that will run ahead of it. + +## Why batches contain multiple frames + +A batch could have carried a single safe block for everything in it. Frames exist so the number can move forward part way through. + +While the sequencer is filling a batch, the base layer keeps producing blocks and direct inputs keep arriving. Frames let one batch record that some transactions were evaluated at block 95 and later transactions at block 100. The batch can therefore remain open while the sequencer's base-layer view advances. + +![A sequencer batch with nonce 42 contains two frames. Each frame records its own safe block, status, and transactions, while references to the base-layer chain show how the safe block advances as the batch remains open.](../images/batch-frame-interleaving.png) + +``` +Batch, nonce 42 + frame 1 safe block 95 transactions A, B, C + frame 2 safe block 100 transaction D +``` + +Run by the machine, that becomes: + +``` +run direct inputs recorded up to block 95 +run A, B, C +run direct inputs recorded up to block 100 +run D +``` + +## Rules for a valid batch + +The machine checks a batch before running it, and rejects one that is not well formed. + +- **A frame cannot claim the future.** Every frame's safe block must be at or before the block the batch itself was recorded in. A batch cannot claim to have accounted for base-layer activity that had not happened when it was posted. +- **Safe blocks cannot go backwards.** Across the frames of a batch they must be non-decreasing. The sequencer's view of the base layer only moves forward, so a batch that goes backwards is malformed. +- **Batches arrive in numbered order.** Each batch carries the next expected nonce. One that carries the wrong number is rejected. +- **A batch must not be too old.** Measured from its **first** frame's safe block, a batch that reaches the base layer 1200 blocks or more after that block is skipped as stale. + +## How a stale batch affects later batches + +A batch that is accepted consumes its nonce, and the machine then expects the next number. A batch that is **skipped as stale does not consume its nonce**. The machine is still waiting for that same number. + +Every later batch then carries a number the machine does not expect, so each is rejected in turn. No application state is corrupted and no machine state needs to be reversed because those batches never take effect. The resulting loss covers everything from the missed batch onward. + +It is also what makes repair possible. Because the nonce never advanced, a rebuilt batch can reuse it and be accepted as though the skipped one had never been sent. See [Soft confirmations](./soft-confirmations.md) for what this means for a user. + +## How transaction-level failures are handled + +Once the machine begins executing a valid batch, a problem with one transaction does not stop the remaining transactions. The machine skips only the affected transaction when: + +- its signature cannot be used to identify the sender; +- the application rejects it; or +- its maximum fee is lower than the price set for that part of the batch. + +A skipped transaction makes no change to the application state. The machine continues with the rest of the batch, and the batch still advances the expected batch number. + +Transactions submitted through the public API normally fail before reaching this stage. The API checks the signature first and returns `400` if it is invalid. The sequencer then checks the transaction against the application and the current price, returning `422` if either check fails. Only a transaction that passes these checks is stored in a batch. See [Submitting operations](../usage/submitting-operations.md). + +The machine still needs its own rule because it cannot assume that every batch was constructed correctly. If unexpected transaction data reaches the base layer, the machine must produce a predictable result. Skipping only the invalid transaction allows the remaining valid transactions to continue without invalidating the entire batch. + +## Related concepts + +- To see where this sits in the whole system, read [Architecture at a glance](../foundations/architecture.md). +- To see how the two ways in differ, read [Direct inputs vs sequenced transactions](./direct-vs-sequenced.md). diff --git a/app-sequencer/concepts/direct-vs-sequenced.md b/app-sequencer/concepts/direct-vs-sequenced.md new file mode 100644 index 000000000..1940ec561 --- /dev/null +++ b/app-sequencer/concepts/direct-vs-sequenced.md @@ -0,0 +1,50 @@ +--- +title: "Direct inputs vs sequenced transactions" +sidebar_label: "Direct inputs vs sequenced transactions" +description: "The two ways a transaction reaches a Cartesi application, how the machine tells them apart, and why one of them cannot be censored." +--- + +There are two ways into an application that runs a sequencer: + +- A **sequenced transaction** goes to the sequencer, which orders it and posts it inside a batch. +- A **direct input** goes straight to the InputBox contract on the base layer, skipping the sequencer completely. + +Both end up in the same application, in one agreed order. They differ in who carries them, how fast they run, who pays, and whether anyone can stop them. + +## How inputs are classified + +Both direct inputs and sequenced transactions arrive at the same InputBox contract with no flag or label in the payload. The application distinguishes them by the **sender's address**. Anything sent by the sequencer's address is treated as a batch, while every other sender produces a direct input. Payload contents cannot alter this classification. + +## Direct and sequenced inputs compared + +| | Sequenced transaction | Direct input | +| ------------------------- | ----------------------------------- | ------------------------------------------------------------------ | +| Sent to | The sequencer | The InputBox contract | +| Answer arrives | Immediately, as a soft confirmation | When the base layer records it | +| Who pays base-layer costs | The sequencer, which posts batches | The user, who pays their own gas | +| Can delivery be refused | Yes, the sequencer may decline it | No, delivery is guaranteed | +| Can it be delayed | Yes, by the sequencer | Only up to a fixed limit | +| Typical use | Ordinary application activity | Deposits, and reaching the application when the sequencer will not | + +## When direct inputs run + +A direct input does not run the moment it is recorded. It still waits, for the sequencer to decide when it is safe to be picked up, by way of the safe block it puts in each frame. + +The frame's safe block instructs the application to run everything recorded up to the given safe block number. See [Batches, frames, and the safe block](./batches-frames-safe-block.md). + +Direct inputs still get executed in the order they were recorded on the base layer with the oldest first, however they are executed in short bursts decided by the safe block. + +## Why deposits arrive as direct inputs + +Deposits come from a portal contract on the base layer, which posts to the InputBox itself. They are direct inputs by construction, not by choice, which is a useful property: it means funds can always reach an application. + +## Designing the direct-input path + +An application has to be designed to handle both direct inputs and sequenced transactions. The application receives both inputs and decides what each is allowed to do. + +That decision sets how much of the application still works without its sequencer. If the direct route only accepts deposits, users can move funds in while the sequencer is unavailable, but everything else waits for it to come back. + +## Related concepts + +- To see the whole path, read [Architecture at a glance](../foundations/architecture.md). +- For the guarantees and limitations of a soft confirmation, read [Soft confirmations](./soft-confirmations.md). diff --git a/app-sequencer/concepts/execution-order.md b/app-sequencer/concepts/execution-order.md new file mode 100644 index 000000000..fce6f8833 --- /dev/null +++ b/app-sequencer/concepts/execution-order.md @@ -0,0 +1,62 @@ +--- +title: "Deterministic execution order" +sidebar_label: "Deterministic execution order" +description: "How the sequencer and the machine reach the same order, and what happens when they do not." +--- + +The sequencer's value rests on one claim: the application runs transactions in the order reported by the sequencer. This page explains how the system maintains that agreement and what happens if it fails. + +## How the ordering logic stays consistent + +The Cartesi machine produces the final execution order. To provide fast confirmations, the live sequencer must predict that order before a batch reaches the base layer. + +Three parts of the implementation contribute to this process: + +| Component | Purpose | +| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | +| **Scheduler** | Runs inside the Cartesi machine and determines the final execution order. Recovery uses the same scheduler code to rebuild application state. | +| **Batch acceptance check** | Predicts whether the scheduler will accept a batch based on its sender, number, and timing. | +| **Inclusion lane** | Builds the live local order by processing direct inputs and user transactions in the sequence the scheduler is expected to reproduce. | + +![Inside the Cartesi machine, the scheduler classifies InputBox inputs, accepts valid batches, drains direct inputs, and executes frame transactions before the application computes canonical state. Before settlement, the host inclusion lane predicts execution order and transaction outcomes, while the batch acceptance check predicts the accepted frontier.](../images/execution-agreement.jpg) + +The scheduler is the authority. The other two components predict its decisions so the sequencer can respond without waiting for base-layer settlement. + +These components share the same protocol definitions, but they are separate implementation paths. The batch acceptance check also performs a narrower task than the scheduler and therefore omits two structural checks that the scheduler applies. + +Tests and code review are used to keep all three paths consistent. The implementation does not automatically guarantee that they will always agree. + +## Rules that determine execution order + +The order is decided by three things, applied in the same way on both sides. + +**The base layer decides arrival.** Everything reaches the application through the InputBox contract, and the order it records is not up for debate. Neither side chooses it. + +**The sender decides the kind.** Anything sent by the sequencer's address is a batch. Everything else is a direct input. Classification is by who sent it, never by anything in the payload, so it cannot be spoofed. + +**Each frame decides the interleaving.** A frame's safe block means: run every direct input recorded up to this block, then run this frame's transactions in the order they appear. Both sides apply that literally. + +The result is that neither side is trusting the other's conclusion. Both are computing the same function over the same input. + +## Why the sequencer can answer early + +The sequencer applies these rules to its own view before it answers. When it accepts a transaction, it has already drained the direct inputs that will run ahead of it, so the state it judges the transaction against is the state the application will have. + +The immediate answer predicts the machine's decision using the sequencer's current view of the inputs. Agreement depends on the three implementations described above remaining aligned. + +## When a transaction is skipped + +Being in an accepted batch does not guarantee that an individual transaction takes effect. A transaction that fails canonical validation is skipped without changing state or preventing later transactions in the batch from being considered. [Scheduler semantics](../advanced/scheduler-semantics.md#transaction-level-outcomes) defines the exact outcomes. + +## How the sequencer detects disagreement + +Sameness by construction is a strong argument, but the system does not rely on the argument alone. + +The sequencer compares accepted base-layer batch content with the local batch stored for the same position. If they differ, it freezes the accepted frontier and stops because later provisional results may depend on the wrong history. + +Detection occurs only after the relevant base-layer observation is safe enough to trust. [Divergence detection and response](../advanced/divergence.md) explains the comparison, timing boundary, and operator response. + +## Related concepts + +- For how the packaging works, read [Batches, frames, and the safe block](./batches-frames-safe-block.md). +- For the guarantees and limitations of the early answer, read [Soft confirmations](./soft-confirmations.md). diff --git a/app-sequencer/concepts/fees.md b/app-sequencer/concepts/fees.md new file mode 100644 index 000000000..de69bd142 --- /dev/null +++ b/app-sequencer/concepts/fees.md @@ -0,0 +1,81 @@ +--- +title: "Fees and data availability" +sidebar_label: "Fees and data availability" +description: "Why transactions carry a fee, how the exponent encoding works, and what the sequencer is paying for." +--- + +Posting batches to the base layer costs the operator real money, and the cost grows with the amount of data posted. Application fees can recover this expense from users. + +## What the fee is for + +The sequencer pays base-layer gas every time it records a batch. The larger the batch, the more it pays. Without a fee, a user submitting large or frequent transactions costs the operator more than a user who barely uses the application, and nothing balances that. + +The fee is charged in the application's own terms. The sequencer does not define what the token is or how balances work, because that is the application's business. What the protocol provides is a way to express a price and a limit, and a rule for comparing them. + +## The exponent encoding + +The protocol represents every fee as a 16 bit**exponent**. This includes the maximum fee signed by a user and the price assigned to a frame. The corresponding amount is calculated as: + +```text +fee amount = floor((129 / 128)ⁿ) +``` + +In this formula: + +- `n` is the exponent stored in the transaction or frame; +- `129 / 128`, or `1.0078125`, is the fixed base; and +- the result is expressed in the smallest unit of the application's token. + +Each increase of `1` in the exponent raises the unrounded amount by approximately 0.78 percent. The protocol rounds the final result down to a whole token unit, so nearby exponents can produce the same amount when the values are small. + +| Exponent | Decoded amount | +| -------: | -------------: | +| `0` | `1` unit | +| `90` | `2` units | +| `256` | `7` units | +| `1060` | `3824` units | + +Exponent `0` represents the minimum fee of one unit. The encoding has no special value for a zero fee. + +This representation provides three protocol benefits: + +- **Compact values.** Each fee occupies two bytes on the wire while supporting a wide range of token denominations and prices. +- **Log-space policy calculations.** Fee adjustments can use exponent addition, reducing the need for multiplication in policy and storage calculations. +- **Deterministic conversion.** The implementation uses exact integer arithmetic and a precomputed table. It uses no floating-point operations, so the sequencer and machine can produce identical results. + +## Maximum fee and frame price + +**`max_fee`** travels with a transaction. It is the most the sender is willing to pay, set by whoever signs. + +**The frame price** is set by the sequencer for each frame, and is fixed once that frame is closed. The next frame samples a fresh recommended price, so the price can move as conditions change, but never underneath transactions already placed. + +**The comparison happens at submission, not later.** When a transaction arrives, the sequencer checks it against the price of the frame currently open, and a transaction that does not meet it is rejected there and then with HTTP `422`. Only transactions that pass are stored and acknowledged. + +An accepted transaction **cannot** later become underpriced. A frame's price remains fixed for that frame's lifetime, and the transaction has already passed the fee check. A higher price in a later frame does not affect it. + +So `max_fee` fails fast, not late. If it is too low you find out in the response. + +## How clients choose a fee + +Clients currently have no public endpoint from which to discover the recommended fee. + +The sequencer sets a frame's price from its own policy, but **it does not expose that price**. +A bid below the current price returns HTTP `422` immediately. The client learns about the rejection on its first attempt, and the transaction is never stored in a batch. + +The practical approaches are: + +- **Start from the deployment's baseline.** The default policy derives a recommended fee exponent of **1060**, so a bid of `1` is rejected. Obtain the deployment's current baseline from its operator. +- **Bid comfortably above it.** The encoding is exponential, so a modest bump in the exponent is a large bump in the amount. +- **Retry on `422`.** The typed rejection allows a client to raise its bid and resubmit after a failed attempt. +- **Publish a default.** For an application whose sequencer you run, the policy is yours, and a sensible client default can ship alongside the application. + +## How batch size affects fees + +Batch size is the other half of the same problem. Larger batches spread the fixed cost of a base-layer transaction across more work, but cost more to post and take longer to fill. + +The same policy determines the target batch size and recommended fee, so the two values move together. + +## Related concepts + +- To set a fee on a transaction, see [Submitting operations](../usage/submitting-operations.md). +- For how frames are formed, see [Batches, frames, and the safe block](./batches-frames-safe-block.md). diff --git a/app-sequencer/concepts/soft-confirmations.md b/app-sequencer/concepts/soft-confirmations.md new file mode 100644 index 000000000..f37ad540f --- /dev/null +++ b/app-sequencer/concepts/soft-confirmations.md @@ -0,0 +1,86 @@ +--- +title: "Soft confirmations" +sidebar_label: "Soft confirmations" +description: "What a soft confirmation promises, when it can be undone, how long the exposure lasts, and how a frontend should treat it." +--- + +A **soft confirmation** is the sequencer's immediate answer to a user. It notifies the user that a transaction has been validated, executed against current state, and durably placed in the sequencer's current ordering. It arrives as soon as the sequencer has processed the transaction, long before anything reaches the base layer. + +It is a prediction. A reliable one under normal condition, but a prediction nevertheless. + +## What a Soft Confirmation Means + +A soft confirmation says the sequencer has accepted the transaction, executed it, and placed it in the ordering it intends to post. + +It does **not** hand back a position. The response carries the sender and the nonce it accepted, not an offset, a batch, or a frame. A position exists only once the transaction appears in the feed, and a client that needs one reads it there. + +The prediction is trustworthy because the sequencer is not guessing. It applies the same ordering rules the machine will apply later, so under normal running the position it reports is the position the application ends up using. See [Architecture at a glance](../foundations/architecture.md) for how the two sides stay aligned. + +## Transaction lifecycle and settlement timing + +While the speed of execution of a transaction via the sequencer is described as "Fast" below is the complete path it takes: + +![A transaction moves from submission to a soft confirmation, waits in an open batch, is sealed and posted to the base layer, becomes recorded, appears in the safe view, and finally settles. The open batch can remain open for up to two hours.](../images/soft-confirmation-lifecycle.png) + +| Stage | What it means | Typical time from submission | +| ---------------------------- | ---------------------------------- | --------------------------------------- | +| **Submitted** | The sequencer has it | Immediate | +| **Soft confirmed** | Accepted, ordered, and answered | Immediate | +| **Sealed into a batch** | The batch it belongs to has closed | **Anything up to 2 hours** | +| **Posted to the base layer** | The batch has been submitted | Seconds after sealing | +| **Recorded** | It is in a block | One block, about 12 seconds on Ethereum | +| **Settled** | Deep enough to be irreversible | About 13 minutes after recording | + +**A batch closes on one of two conditions: it reaches its size target, or it has been open too long.** The second is a wall clock limit, and it defaults to **2 hours**. + +What that means in practice depends entirely on traffic: + +- **A busy application** fills batches by size, so they close often and the sealed row is small. +- **A quiet application** does not, so a transaction can sit in an open batch for up to 2 hours before it is even posted despite being soft confirmed the whole time. + +So a soft confirmation is immediate, and settlement can still be hours away. Both statements are true, and an interface that treats "confirmed" as "nearly settled" will be wrong on a quiet application. + +## Limits of a soft confirmation + +A soft confirmation is **not** settlement. It does not mean the transaction has reached the base layer, and it does not mean the result can never change. + +The gap matters because a user acts on it. Someone who is shown a completed trade or a successful move has been told something that is, at that instant, still a prediction. + +## When a Soft Confirmation Can Be Invalidated + +A soft confirmation is invalidated when its transaction does not become part of canonical execution. The main liveness case is a batch that reaches the base layer after the protocol deadline and is skipped by the scheduler. + +[Staleness and the danger zone](./staleness.md) explains the deadline, why stale batches are skipped, and how one missed batch affects the sequence that follows it. + +## How Invalidation Affects Later Batches + +Because batches use consecutive numbers, invalidation can affect a suffix of provisional history, not just one batch. Recovery removes that suffix and resumes from the accepted canonical frontier. The staleness page gives a worked example of this numbering effect. + +## How Long Transactions Remain at Risk + +The risk begins when the sequencer issues a soft confirmation. It ends only after the transaction's batch has reached the base layer and the application can confirm its outcome from a sufficiently settled view of that layer. There is no single fixed duration for this process. + +Before submission, a transaction may remain in an open batch for up to the configured batch limit, which defaults to two hours. The batch must then be submitted, recorded, and allowed to settle. On Ethereum, reaching the settled view used by the sequencer typically takes about two epochs, or roughly 13 minutes after the batch is recorded. Submission delays or base-layer disruption can extend the total time. + +The sequencer monitors this progress and stops issuing new confirmations when it detects that its current view is no longer safe. Detection is not immediate because the relevant base-layer events must first become settled enough to trust. Transactions confirmed before the problem becomes visible may therefore still be affected. + +The 13 minute period is the approximate observation delay after base-layer recording, not the complete lifetime of a soft confirmation. For a quiet application, the full period of risk can include up to two hours of waiting for the batch to close, followed by submission and settlement time. + +## How Clients Observe Invalidation + +The ordered feed does not send rollback messages. A transaction removed during recovery is absent from a later replay, but a client that already received it gets no live retraction. [Reading the sequenced feed](../usage/reading-the-feed.md) explains cursor storage, replay, and reconciliation. + +## How clients should handle soft confirmations + +- **Show a soft confirmation as fast, not as final.** Distinguish it in the interface from something that has settled on the base layer. A user should be able to tell the difference between "accepted" and "settled". +- **Treat feed delivery as provisional.** Appearance in the feed means the transaction belongs to the current valid local ordering. It does not prove base-layer acceptance or settlement. +- **Reconcile outstanding transactions.** The feed does not announce rollbacks. A client must track its outstanding transactions and compare them against later reads or safe application state to detect a lapse. +- **Size the caution to the stakes.** Adding ceremony everywhere throws away the point of the sequencer, so scale it instead: + - *Low value or easily repeated*, such as a move in a game or a post: act on the fast answer. + - *Meaningful but recoverable*, such as a transfer inside the application: act on it, but mark it as not yet settled. + - *Expensive or irreversible*, such as anything paying out or crossing a boundary: wait for settlement. + +## Next steps + +- To see how the two sides stay aligned, read [Architecture at a glance](../foundations/architecture.md). +- To see what the sequencer is and is not trusted for, read [Trust model and guarantees](../foundations/trust-model.md). diff --git a/app-sequencer/concepts/staleness.md b/app-sequencer/concepts/staleness.md new file mode 100644 index 000000000..8d51774b0 --- /dev/null +++ b/app-sequencer/concepts/staleness.md @@ -0,0 +1,54 @@ +--- +title: "Staleness and the danger zone" +sidebar_label: "Staleness and the danger zone" +description: "Why a batch that arrives too late is skipped, how that spreads to later batches, and why the sequencer steps back before it happens." +--- + +A batch is **stale** when it reaches the base layer too long after the point in time it claims to describe. The machine skips a stale batch entirely. This page explains why that rule exists and how it affects later batches. + +## When a batch becomes stale + +Every batch names a base-layer block in its first frame. This is the point through which it has accounted for direct inputs. The batch's age is the distance from that block to its recorded inclusion block. + +If that age reaches **1200 blocks**, roughly 4 hours where blocks are twelve seconds apart, the batch is stale and is skipped. + +## Why stale batches are skipped + +It seems harsh to discard work that is otherwise valid, but the alternative is worse. + +A batch says "run every direct input up to block N, then run these transactions." If that batch is accepted long after block N, then direct inputs that arrived in the meantime are pushed behind transactions that were decided without any knowledge of them. A sequencer that had fallen far behind, or one that wanted to hold direct inputs back, could keep doing this indefinitely. + +The deadline closes that off. A sequencer cannot keep write priority while ignoring what is arriving directly, because its work stops being accepted once it falls far enough behind. The same block limit makes an overdue direct input eligible for forced execution when the scheduler processes a later application input. See [Direct and sequenced inputs](./direct-vs-sequenced.md). + +## How staleness affects later batches + +Batches carry consecutive numbers so the machine can identify the next batch it should execute. Suppose the machine has accepted batch `41`. It now expects batch `42`. + +If batch `42` arrives on time and passes every check, the machine executes it and moves on to batch `43`. If batch `42` arrives stale, the machine skips all of its transactions and continues to expect batch `42`. Skipping the batch does not consume its number. + +Any batches already created after it now have numbers that are too high: + +![The staleness timeline moves from the normal operating range through the default danger threshold at 900 blocks to the deadline at 1200 blocks. If batch 42 arrives stale, batches 43 and 44 are rejected because the machine still expects 42. A replacement batch 42 restores progress.](../images/staleness-cascade.png) + +This is why one stale batch affects every later batch in the same sequence. The later batches may be recent and otherwise valid, but the machine cannot execute them while it is still waiting for batch `42`. + +No application state needs to be reversed because none of these batches take effect. Recovery can then build a replacement batch numbered `42`. Once the machine accepts that replacement, it advances to `43` and the sequence can continue. See [Batches, frames, and the safe block](./batches-frames-safe-block.md). + +## How the danger zone prevents stale batches + +Waiting for a batch to become stale would detect the problem only after that batch and all later batches were already lost. + +The sequencer tracks how close each unsettled batch is to the deadline. When a batch enters the configured safety margin, the sequencer treats it as a danger signal and stops issuing confirmations that may not survive. + +That margin is configurable, and defaults to 300 blocks, roughly an hour. The point of the margin is runway: an operator gets time to notice and act before the deadline arrives, instead of discovering the problem after it is too late. + +There is a second signal for the case where the sequencer's view of the base layer has frozen. A stalled connection can make everything look fine, because nothing appears to be aging when no new blocks are being seen. So the sequencer also checks its view against the clock, and distrusts a view that has stopped advancing. Without that, it could keep issuing doomed confirmations throughout an outage. + +## What happens when danger is detected + +The sequencer exits when it detects this condition. [Preemptive recovery](../recovery/preemptive.md) explains the next steps, and [Soft confirmations](./soft-confirmations.md) explains how users are affected. + +## Related concepts + +- For what a stale batch means to a user, read [Soft confirmations](./soft-confirmations.md). +- For the recovery that follows, read [Preemptive recovery](../recovery/preemptive.md). diff --git a/app-sequencer/foundations/architecture.md b/app-sequencer/foundations/architecture.md new file mode 100644 index 000000000..50d767b7d --- /dev/null +++ b/app-sequencer/foundations/architecture.md @@ -0,0 +1,106 @@ +--- +title: "Architecture at a glance" +sidebar_label: "Architecture at a glance" +description: "How sequencer prediction, base-layer recording, and canonical execution work together from transaction submission to settlement." +--- + +The App Sequencer adds a fast prediction layer to a Cartesi application without replacing its canonical execution path. + +- The **sequencer** runs as an ordinary service on the application operator's infrastructure. It validates, orders, and executes transactions against a provisional view of application state. +- The **scheduler and application** run inside the deterministic Cartesi machine. The scheduler derives the authoritative order from base-layer inputs, and the application computes state from that order. +- The **base layer** records application inputs and settles commitments to the machine's result. + +The sequencer can respond before base-layer processing because it predicts what the scheduler will compute. The prediction remains a soft confirmation until the corresponding transaction is accepted through the canonical path. + +## Sequencer prediction and canonical execution + +![The App Sequencer runs off-chain with its HTTP API, inclusion lane, batch submitter, input reader, and local database. It exchanges transactions and soft confirmations with users, posts batches to the InputBox, and reads recorded inputs. The Cartesi machine reads the InputBox and runs the scheduler before the application logic. Direct transactions reach the InputBox without passing through the sequencer.](../images/architecture.png) + +The diagram has three system boundaries: + +1. **Sequencer infrastructure.** The application operator runs the API, transaction-ordering loop, database, base-layer reader, and batch submitter. +2. **Base-layer contracts.** The `InputBox` records application inputs. The application contract, portals, and other Cartesi Rollups contracts retain their existing roles. +3. **Cartesi machine.** The scheduler interprets the recorded inputs, and the application executes the resulting order deterministically. + +The sequencer cannot change the order that the scheduler derives after inputs are recorded. Its responsibility is to maintain a provisional order that the scheduler is expected to reproduce later. + +## System components and responsibilities + +### Sequencer service + +| Component | Responsibility | +| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | +| HTTP API | Accepts signed user transactions and returns success or a structured rejection | +| Inclusion lane | Applies protocol and application validation, drains relevant direct inputs, executes accepted transactions, and stores them in a single deterministic order | +| Ordered feed | Publishes the sequencer's current valid transaction history to clients and indexers | +| Batch submitter | Posts completed batches to the base-layer `InputBox` using the configured submitter account | +| Input reader | Reads safe base-layer inputs so the sequencer can account for direct inputs and observe submitted batches | +| Local database | Stores deployment identity, ordered transactions, batches, snapshots, submission state, and recovery metadata | + +### Base-layer contracts + +| Component | Responsibility | +| -------------------- | ------------------------------------------------------------------------------------- | +| `InputBox` | Records inputs sent to the application, their senders, and their base-layer positions | +| Application contract | Identifies the application and its data-availability configuration | +| Portals | Transfer assets and submit the corresponding deposit inputs to the `InputBox` | + +The `InputBox` records arrival order. It does not determine the complete application execution order because batches contain their own ordered transactions and frames govern when direct inputs run. + +### Cartesi machine + +| Component | Responsibility | +| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | +| Scheduler | Classifies recorded inputs, validates batches, combines direct inputs with sequenced transactions, and produces the authoritative execution order | +| Application | Validates application-specific behavior and computes state and outputs from that order | + +The Cartesi machine is deterministic and reproducible. Its result can be committed to the base layer and challenged through the fraud-proof system. [Trust model and guarantees](./trust-model.md) explains the assumptions around each boundary. + +## Transaction lifecycle from submission to settlement + +1. **Submission.** A user signs an EIP-712 transaction and sends it to the sequencer's HTTP API. +2. **Validation.** The sequencer verifies the request and signature, checks the current fee, and applies the application's validation rules. +3. **Predicted execution.** The inclusion lane applies any direct inputs that should execute first, then executes and durably stores the accepted transaction. +4. **Soft confirmation.** The API returns success. This confirms the sequencer's current prediction, not base-layer settlement. +5. **Batch construction.** The transaction is stored in the open frame and batch. The batch closes according to the configured size or time policy. +6. **Base-layer submission.** The batch submitter sends the completed batch to the `InputBox`. +7. **Canonical ordering.** The scheduler reads the base-layer input, validates the batch, drains the direct inputs covered by each frame, and executes the listed transactions. +8. **Application settlement.** The application computes its result, and the rollup settles a commitment to that state through the base layer. + +The first four stages normally complete without waiting for base-layer settlement. The remaining stages depend on batch policy, base-layer inclusion, and the settlement level required by the client. + +| Observation | Source of the result | Meaning | +| --------------------------- | ------------------------- | --------------------------------------------------------------------------- | +| HTTP success | Sequencer | The transaction was accepted, executed, and stored in the provisional order | +| Feed entry | Sequencer | The transaction has a provisional ordering position | +| Canonical application state | Scheduler and application | The transaction affected the result derived from base-layer inputs | +| Sufficiently settled state | Rollup and base layer | The client has reached its required settlement threshold | + +## How frames align sequenced and direct inputs + +A batch can remain open while new direct inputs arrive. Frames let the sequencer record how far it has accounted for those inputs at different points in the batch. The sequencer and scheduler use each frame's safe block to combine direct inputs and sequenced transactions consistently. + +[Batches, frames, and the safe block](../concepts/batches-frames-safe-block.md) explains the structure. [Deterministic execution order](../concepts/execution-order.md) explains the complete ordering rules. + +## The direct-input path + +A user or contract can submit an input to the `InputBox` without using the sequencer. Deposits normally follow this route through portal contracts. The available actions depend on the application. + +[Direct and sequenced inputs](../concepts/direct-vs-sequenced.md) explains how the scheduler classifies both paths and compares their behavior. + +## Application and client integration responsibilities + +An application team integrates and operates three pieces: + +- **Application logic.** The application must implement deterministic validation, user-transaction execution, direct-input execution, and snapshot behavior required by the sequencer library. +- **Application-specific sequencer binary.** A small executable combines the application implementation with the sequencer runtime and selects the deployment-specific configuration. +- **Client or frontend.** The client signs and submits transactions, consumes the ordered feed, stores its cursor, distinguishes soft confirmations from settlement, and reconciles unsettled activity against canonical application state. + +The App Sequencer provides the runtime and Rust client components, but application logic, deployment configuration, operational ownership, and user-facing confirmation behavior remain application responsibilities. + +See [Application integration](../usage/integration.md), [Application requirements](../usage/application-requirements.md), and [Consuming the sequenced transaction feed](../usage/reading-the-feed.md). + +## Next steps + +- To examine the security and operational assumptions, read [Trust model and guarantees](./trust-model.md). +- To try the complete client flow, follow the [Quickstart](../usage/quickstart.md). diff --git a/app-sequencer/foundations/glossary.md b/app-sequencer/foundations/glossary.md new file mode 100644 index 000000000..2a5f7e9aa --- /dev/null +++ b/app-sequencer/foundations/glossary.md @@ -0,0 +1,61 @@ +--- +title: "App Sequencer glossary" +sidebar_label: "App Sequencer glossary" +description: "Key terms used throughout the App Sequencer documentation." +--- + +This glossary provides concise definitions for App Sequencer terminology. Follow the links for the complete behavior, guarantees, and implementation guidance behind each term. + +**Anchor.** The batch number at which a deployment's local record begins. It is zero for a new deployment or the resume number for a deployment rebuilt from a checkpoint. See [Understanding the batch tree](../concepts/batch-tree.md). + +**App-specific sequencer.** A sequencer dedicated to one application deployment. It uses that application's validation and execution logic to provide a provisional transaction order and soft confirmations. See [App-specific sequencing](../overview/app-specific-sequencing.md). + +**Base layer.** The blockchain that records application inputs and supports settlement of the rollup's state commitments, usually Ethereum. + +**Batch.** A numbered package of frames that the sequencer submits to the base-layer `InputBox`. See [Batches, frames, and the safe block](../concepts/batches-frames-safe-block.md). + +**Batch submitter.** Either the sequencer worker that posts completed batches or the dedicated base-layer account used for those submissions. The surrounding context should identify which meaning applies. + +**Batch tree.** The sequencer's record of batches it has built. It forms a straight line during normal operation and branches when recovery abandons an unsettled suffix and creates a replacement sequence. See [Understanding the batch tree](../concepts/batch-tree.md). + +**Canonical state.** The application state computed from the execution order derived inside the Cartesi machine from base-layer inputs. + +**Checkpoint.** An archived finalized snapshot and its recovery metadata. It provides a trusted starting point for rebuilding a deployment after local state is lost or no longer trusted. See [Snapshots and checkpoints](../recovery/snapshots.md). + +**Cockroach recovery.** The procedure for rebuilding a deployment from a checkpoint and base-layer history. See [Cockroach recovery](../recovery/cockroach.md). + +**Danger zone.** The configurable safety margin before the staleness deadline. The sequencer stops when an unsettled batch enters this range so recovery can begin before the batch becomes stale. See [Staleness and the danger zone](../concepts/staleness.md). + +**Direct input.** An application input recorded through the base-layer `InputBox` without being included in a sequencer batch. The scheduler classifies an input as direct when its sender is not the configured batch-submitter address. See [Direct inputs vs sequenced transactions](../concepts/direct-vs-sequenced.md). + +**Divergence.** A fault in which a batch observed in canonical execution differs from the batch the sequencer sealed for the same position. The sequencer records the fault and stops. See [Divergence handling](../advanced/divergence.md). + +**Feed.** The ordered stream of the sequencer's current valid transactions. It is provisional, can publish before base-layer acceptance, and does not send rollback messages after recovery. See [Consuming the sequenced transaction feed](../usage/reading-the-feed.md). + +**Frame.** An ordered section inside a batch. It carries a safe block, a frame fee, and zero or more sequenced transactions. + +**Inclusion lane.** The sequencer's single ordering loop. It drains relevant direct inputs, validates and executes user transactions, stores accepted transactions, and manages open frames and batches. + +**InputBox.** The base-layer contract that records inputs for a Cartesi application, including each input's sender and position. The scheduler interprets this record to derive the complete execution order. + +**Offset.** An ascending identifier attached to each message in the sequenced feed. Offsets begin at 1 and may contain gaps, so a client resumes from its last processed offset instead of counting messages. + +**Preemptive recovery.** The routine recovery path triggered before an unsettled batch reaches the staleness deadline. It stops unsafe confirmation, identifies the canonical frontier, invalidates the affected provisional suffix, and resumes from the expected batch number. See [Preemptive recovery](../recovery/preemptive.md). + +**Safe block.** A base-layer block number carried by a frame. Before executing the frame's sequenced transactions, the scheduler executes pending direct inputs recorded at or before this block. + +**Scheduler.** The component inside the application's Cartesi machine that derives the authoritative execution order from base-layer inputs. The sequencer predicts its result by following matching protocol rules. + +**Sequenced transaction.** A signed user transaction accepted by the sequencer and placed in a batch, as distinct from a direct input. + +**Sequencing.** Deciding the order in which application transactions execute. + +**Settled.** Accepted through the canonical rollup path and observed at the settlement threshold required by the client. The exact threshold depends on the chain and the risk of the action. + +**Snapshot.** A durable copy of application state at a known transaction offset. A pending snapshot depends on unsettled batches; a finalized snapshot has been promoted after the corresponding canonical progress is observed. + +**Soft confirmation.** The sequencer's immediate response after it accepts, executes, and durably stores a transaction in its provisional ordering. It is not proof of base-layer acceptance or settlement. See [Soft confirmations](../concepts/soft-confirmations.md). + +**Stale batch.** A non-empty batch whose base-layer inclusion block is at least `MAX_WAIT_BLOCKS` after the safe block in its first frame. The scheduler skips it without consuming its batch number. See [Staleness and the danger zone](../concepts/staleness.md). + +**Watchdog.** A process that compares sequencer snapshots with independently reproduced canonical application state and reports mismatches. See [Monitoring the sequencer](../operations/monitoring.md). diff --git a/app-sequencer/foundations/trust-model.md b/app-sequencer/foundations/trust-model.md new file mode 100644 index 000000000..9464f7dcf --- /dev/null +++ b/app-sequencer/foundations/trust-model.md @@ -0,0 +1,90 @@ +--- +title: "Trust model and guarantees" +sidebar_label: "Trust model and guarantees" +description: "Which App Sequencer properties are enforced by the protocol, which depend on the operator, and how failures affect users." +--- + +The App Sequencer gives one operator control over the application's fast transaction path. The protocol limits that control, but it does not remove the need to trust the operator for availability, transaction selection, and fair provisional ordering. + +This page separates protocol-enforced properties from operational assumptions. It also explains what clients must expect when the provisional order and canonical application state do not agree. + +## Protocol-enforced limitations + +These limits hold regardless of the sequencer operator's conduct: + +- **The sequencer cannot forge a valid user transaction.** Each sequenced transaction carries an EIP-712 signature that is checked against its sender and deployment-specific domain. +- **The sequencer does not determine canonical state.** The scheduler derives execution order inside the Cartesi machine, and the application computes state from that order. +- **The sequencer cannot rewrite settled canonical history.** Recovery changes the sequencer's unsettled local history. It does not change application state that has already settled through the canonical path. +- **The sequencing service does not custody user assets.** Assets remain governed by the application's contracts, portals, and application logic. +- **The sequencer cannot permanently remove the base-layer input route.** Users and contracts can submit directly to the `InputBox`, although the application determines which actions that path supports. + +These protections do not make the sequencer economically neutral. Transaction selection and ordering can affect prices, trades, liquidations, games, and other order-sensitive behavior. + +## Sequencer powers and trust assumptions + +The operator controls the provisional fast path and can: + +- **refuse or delay transactions;** +- **choose the order among accepted transactions,** including placing its own transaction first; +- **observe submitted transactions before they appear on the base layer;** +- **change configurable fee and batching policy;** +- **make the fast path unavailable by stopping or misoperating the service;** +- **issue a soft confirmation that is later invalidated** if its batch does not become part of canonical execution. + +The protocol does not enforce first come, first served ordering. Applications in which ordering carries economic value must therefore treat the operator as trusted for fairness or add application-level protections against harmful ordering. + +## Protocol protections and recovery paths + +### Signed user transactions + +Deployment-bound signatures prevent the sequencer from inventing a user's authorization. They do not prevent it from excluding or reordering authorized transactions. [EIP-712 domain](../api-reference/eip712.md) defines the exact signed fields and deployment binding. + +### Canonical recomputation + +The scheduler reads the base-layer `InputBox` history and derives the execution order inside the Cartesi machine. The application then computes its own state from that order. + +A provisional order announced by the sequencer cannot, by itself, force the machine to accept that order. The batch must pass the scheduler's identity, numbering, structure, timing, and transaction checks. + +### Direct-input availability + +Users and contracts can reach the application through the base-layer `InputBox` without the sequencer. This protects access, but not speed, cost, or feature equivalence. The application defines which actions the direct route supports. [Direct and sequenced inputs](../concepts/direct-vs-sequenced.md) explains the behavior and scheduler backstop. + +### Fraud-proof settlement + +Adding the sequencer does not replace the rollup's dispute mechanism. Commitments to application execution remain subject to the same [fraud-proof system](/fraud-proofs) and its underlying assumptions. + +## How the system responds to divergence + +If accepted base-layer batch content differs from the matching local sealed batch, the sequencer records a terminal divergence and stops. Detection follows the configured safe view, so provisional results issued before detection may already depend on the wrong history. [Divergence detection and response](../advanced/divergence.md) defines the comparison, timing, and operator procedure. + +## Trust assumptions by participant + +| Participant or component | Relied on for | Not relied on for | +| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- | +| Application operator | Service availability, funding batch submission, transaction selection, and fair provisional ordering | Creating valid user signatures or directly defining canonical state | +| Application implementation | Deterministic validation and execution, correct fee and nonce handling, direct-input behavior, and reproducible snapshots | Base-layer consensus or transaction inclusion | +| Sequencer implementation | Predicting scheduler behavior correctly, preserving durable ordering, and stopping safely on detected faults | Canonical authority over settled application state | +| Base layer | Consensus, ordered transaction recording, and the settlement properties assumed by the rollup | Fair inclusion or timely inclusion of the sequencer's submissions | +| Cartesi Rollups contracts, including the `InputBox` | Correct contract behavior, sender authentication, and recording application inputs | Deriving the application's complete execution order | +| Base-layer node used by the sequencer | A consistent and truthful view of the chain, with failures treated as unavailable data | Byzantine fault tolerance at the sequencer's RPC boundary | +| Block builders | No cooperative behavior is assumed | They may delay, reorder, or omit batch submissions and direct inputs | +| Users submitting transactions | No trusted behavior is assumed | Requests are decoded, bounded, signed, and validated before execution | + +## Guarantee summary + +| Property | Classification | +| --------------------------------------- | ------------------------------------------------------------------ | +| User authorization | Enforced through signature verification and application validation | +| Canonical execution order | Derived by the scheduler from base-layer inputs | +| Canonical application state | Computed deterministically inside the Cartesi machine | +| Fair ordering on the fast path | Depends on the operator or application-level controls | +| Fast API availability | Depends on sequencer infrastructure and its dependencies | +| Survival of a soft confirmation | Conditional on its batch becoming part of canonical execution | +| Available actions through direct inputs | Defined by the application | +| Base-layer inclusion time | Depends on the chain, fee market, and block builders | + +## Next steps + +- To see where each trusted component sits, read [Architecture at a glance](./architecture.md). +- To design client behavior around provisional results, read [Soft confirmations](../concepts/soft-confirmations.md). +- To understand failure recovery, read [Staleness and the danger zone](../concepts/staleness.md) and [Preemptive recovery](../recovery/preemptive.md). diff --git a/app-sequencer/images/architecture.png b/app-sequencer/images/architecture.png new file mode 100644 index 000000000..394179130 Binary files /dev/null and b/app-sequencer/images/architecture.png differ diff --git a/app-sequencer/images/batch-frame-interleaving.png b/app-sequencer/images/batch-frame-interleaving.png new file mode 100644 index 000000000..dc9ea00cd Binary files /dev/null and b/app-sequencer/images/batch-frame-interleaving.png differ diff --git a/app-sequencer/images/batch-tree-recovery.png b/app-sequencer/images/batch-tree-recovery.png new file mode 100644 index 000000000..617c4a300 Binary files /dev/null and b/app-sequencer/images/batch-tree-recovery.png differ diff --git a/app-sequencer/images/divergence-check.jpg b/app-sequencer/images/divergence-check.jpg new file mode 100644 index 000000000..df6e701c3 Binary files /dev/null and b/app-sequencer/images/divergence-check.jpg differ diff --git a/app-sequencer/images/execution-agreement.jpg b/app-sequencer/images/execution-agreement.jpg new file mode 100644 index 000000000..bb4a9a19d Binary files /dev/null and b/app-sequencer/images/execution-agreement.jpg differ diff --git a/app-sequencer/images/integration-duality.png b/app-sequencer/images/integration-duality.png new file mode 100644 index 000000000..dce9a82af Binary files /dev/null and b/app-sequencer/images/integration-duality.png differ diff --git a/app-sequencer/images/preemptive-recovery.jpg b/app-sequencer/images/preemptive-recovery.jpg new file mode 100644 index 000000000..eb2907e7b Binary files /dev/null and b/app-sequencer/images/preemptive-recovery.jpg differ diff --git a/app-sequencer/images/recovery-model-transitions.jpg b/app-sequencer/images/recovery-model-transitions.jpg new file mode 100644 index 000000000..9fb452072 Binary files /dev/null and b/app-sequencer/images/recovery-model-transitions.jpg differ diff --git a/app-sequencer/images/scheduler-ordering-example.jpg b/app-sequencer/images/scheduler-ordering-example.jpg new file mode 100644 index 000000000..59b632fb5 Binary files /dev/null and b/app-sequencer/images/scheduler-ordering-example.jpg differ diff --git a/app-sequencer/images/snapshot-lifecycle.jpg b/app-sequencer/images/snapshot-lifecycle.jpg new file mode 100644 index 000000000..9b828f7e9 Binary files /dev/null and b/app-sequencer/images/snapshot-lifecycle.jpg differ diff --git a/app-sequencer/images/soft-confirmation-lifecycle.png b/app-sequencer/images/soft-confirmation-lifecycle.png new file mode 100644 index 000000000..0ee3ee992 Binary files /dev/null and b/app-sequencer/images/soft-confirmation-lifecycle.png differ diff --git a/app-sequencer/images/staleness-cascade.png b/app-sequencer/images/staleness-cascade.png new file mode 100644 index 000000000..74b8923be Binary files /dev/null and b/app-sequencer/images/staleness-cascade.png differ diff --git a/app-sequencer/images/submission-timeout-recovery.jpg b/app-sequencer/images/submission-timeout-recovery.jpg new file mode 100644 index 000000000..2ea496b21 Binary files /dev/null and b/app-sequencer/images/submission-timeout-recovery.jpg differ diff --git a/app-sequencer/images/transaction-paths.png b/app-sequencer/images/transaction-paths.png new file mode 100644 index 000000000..54499af9d Binary files /dev/null and b/app-sequencer/images/transaction-paths.png differ diff --git a/app-sequencer/images/watchdog-comparison.png b/app-sequencer/images/watchdog-comparison.png new file mode 100644 index 000000000..56db49aa6 Binary files /dev/null and b/app-sequencer/images/watchdog-comparison.png differ diff --git a/app-sequencer/operations/data-and-state.md b/app-sequencer/operations/data-and-state.md new file mode 100644 index 000000000..9e2c597c5 --- /dev/null +++ b/app-sequencer/operations/data-and-state.md @@ -0,0 +1,104 @@ +--- +title: "Data, snapshots, and backups" +sidebar_label: "Data and backups" +description: "What the sequencer stores, how it protects committed state, and how to back up a deployment and archive recovery checkpoints." +--- + +The sequencer stores its complete local history under `CARTESI_SEQUENCER_DATA_DIR`, which defaults to the relative path `sequencer-data`. + +Use an explicit path on durable storage in production. Losing the directory removes the local batch history, snapshots, and wallet-nonce watermark needed for an ordinary restart and preemptive recovery. + +## Data directory layout + +A configured data directory contains: + +```text +/ + sequencer.db + sequencer.db-wal present while SQLite uses its write-ahead log + sequencer.db-shm present while SQLite uses shared memory + dumps/ + / + info.toml + state application-defined snapshot data +``` + +The contents under `state` are defined by the application's snapshot implementation. It can be a file or a larger application-specific structure even though the reference application uses a state file. + +Dump directory names are opaque. Use their metadata and the database references to determine their lifecycle instead of inferring meaning from a directory name. + +## State stored by the sequencer + +The SQLite database records: + +- the pinned chain, application, `InputBox`, genesis block, and batch-submitter identities; +- whether setup completed; +- the open and sealed batch tree, including invalidated branches; +- frames, user operations, direct inputs, and the ordered feed; +- the latest observed base-layer safe head and accepted-batch frontier; +- pending and finalized snapshot references; +- the highest submitter wallet nonce covered before broadcast; +- recovery and divergence state. + +The dump directories hold application state at selected transaction offsets. The database links each usable dump to its pending or finalized lifecycle state. + +## Durability and crash guarantees + +Production writer connections use SQLite WAL mode with `synchronous=FULL`. Each committed database transaction is synchronized before the sequencer externalizes work that depends on it. This supports two important guarantees: + +- an HTTP success response follows the commit that stores the accepted operation; +- the submitter wallet-nonce watermark is committed before a corresponding base-layer transaction is broadcast. + +Snapshot creation crosses the filesystem and database. The sequencer first creates and synchronizes the dump and its `info.toml`, then commits the database row that references it. A failed database commit can leave an unreferenced directory, but it cannot leave a committed row pointing to an incomplete dump through the normal creation path. + +Startup repairs interrupted snapshot housekeeping before loading application state. [Snapshots and checkpoints](../recovery/snapshots.md#how-snapshots-are-created-and-promoted) describes the creation, promotion, and cleanup sequence. These guarantees also depend on the application's snapshot methods honoring their durability contract and on storage that preserves acknowledged writes. + +## Snapshot lifecycle and automatic cleanup + +The live data directory retains the snapshots needed for current operation and garbage-collects superseded dumps. It is not a historical checkpoint archive. + +[Snapshots and checkpoints](../recovery/snapshots.md) is the authoritative guide to pending and finalized lifecycle states, promotion, retention, and recovery suitability. + +## Back up the live deployment + +The safest complete backup is a coordinated copy while the sequencer is stopped: + +1. stop `run` and wait for the process to exit; +2. prevent another instance or maintenance command from starting; +3. copy the entire data directory, including SQLite sidecar files and `dumps/`; +4. verify the copy and record the deployment identity and backup time; +5. restart the sequencer from the original directory. + +Do not copy only `sequencer.db` while the process is running. Committed pages may still reside in the WAL, and snapshot files can change lifecycle while the copy is in progress. + +If downtime is unacceptable, use a backup procedure that coordinates a SQLite-consistent database snapshot with the referenced dump directories. A generic recursive live filesystem copy does not provide that coordination. + +A complete backup is useful for preserving the current deployment. Restoring it later is safe only if the deployment has not produced additional base-layer activity since that point. + +## Preserve recovery checkpoints + +Checkpoint recovery uses a complete promoted dump directory, not a live data-directory backup or an HTTP snapshot response. Archive the application state and its original `info.toml` together before live cleanup removes them. + +Follow the single coordinated procedure in [Archive a complete recovery checkpoint](../recovery/snapshots.md#archive-a-complete-recovery-checkpoint). It also defines archive validation, retention, and the warning about `/finalized_state` responses. + +## Restore boundaries + +Do not restore an old data-directory backup over a deployment that continued submitting batches. Its local history and wallet-nonce watermark can lag the base layer and cause identity, setup, or divergence failures. + +When the original directory is lost or no longer trusted, rebuild a fresh directory from an archived checkpoint using [Cockroach recovery](../recovery/cockroach.md). + +Do not edit `info.toml`, synthesize a checkpoint, or combine application state and metadata from different dumps. Recovery trusts these inputs and cannot reconstruct a missing next batch nonce safely. + +## Capacity planning + +Monitor both database and snapshot storage. Transaction, frame, batch, direct-input, and feed records accumulate with activity. Snapshot garbage collection limits superseded dumps, but the current pending and finalized application snapshots can still be large. + +Alert before the volume approaches exhaustion. A full disk can prevent transaction commits, batch closure, snapshot creation, or recovery metadata updates and can stop the process. + +Also monitor inode availability when an application's dump format creates many files. + +## Next steps + +- Configure the directory with [Configure, set up, and run the sequencer](./setup-and-running.md). +- Archive valid recovery inputs using [Snapshots and checkpoints](../recovery/snapshots.md). +- Rebuild a lost deployment with [Cockroach recovery](../recovery/cockroach.md). diff --git a/app-sequencer/operations/monitoring.md b/app-sequencer/operations/monitoring.md new file mode 100644 index 000000000..8a6c7edc9 --- /dev/null +++ b/app-sequencer/operations/monitoring.md @@ -0,0 +1,153 @@ +--- +title: "Monitoring and watchdog operation" +sidebar_label: "Monitoring and watchdog" +description: "How to monitor sequencer health and progress, run the independent watchdog, and alert on conditions the health probes cannot detect." +--- + +Reliable operation requires two kinds of observation: + +- **service monitoring** checks whether the sequencer, storage, base-layer connection, and submitter are functioning; +- **independent verification** checks whether the sequencer's promoted application state matches canonical execution inside a Cartesi machine. + +The health endpoints cover only a small part of the first category. The watchdog provides the second. + +## Why independent verification matters + +The sequencer predicts canonical execution using its own code, storage, and base-layer view. It also performs an internal content-identity check when it observes accepted batches. Those checks are important, but they cannot provide complete independence from the system being checked. + +The watchdog starts from an independently managed Cartesi machine snapshot, reads application inputs from the base layer, advances canonical execution, and compares the resulting state bytes with the sequencer's promoted state at the same inclusion block. + +A mismatch is a critical correctness event. Stop transaction traffic, preserve both sequencer and watchdog state, and investigate before restarting the deployment. + +## How the watchdog compares state + +Each watchdog tick: + +1. loads the canonical checkpoint named by its `head.json`; +2. reads `GET /finalized_state/inclusion_block` from the sequencer; +3. exits successfully without a full comparison if the promoted block has not advanced; +4. fetches the relevant `InputAdded` logs from the base layer; +5. advances the canonical Cartesi machine through those inputs; +6. obtains the machine's application-state bytes; +7. downloads `GET /finalized_state` from the sequencer and compares the bytes; +8. writes a new watchdog checkpoint only after a successful comparison. + +![The watchdog reads base-layer inputs independently, advances its own Cartesi machine to the sequencer's finalized inclusion block, and compares canonical machine bytes with the sequencer's finalized bytes. A match promotes the watchdog checkpoint, while a mismatch stops processing and raises an alert.](../images/watchdog-comparison.png) + +The watchdog uses its own persistent state directory and base-layer replay. Do not place that state inside the sequencer data directory or treat the sequencer's state as the watchdog's source of truth. + +## Initialize the watchdog + +Initialize the watchdog once with a Cartesi machine snapshot and block that match the deployment's current promoted state: + +```bash +sequencer-watchdog init +``` + +For a long-running deployment, do not assume block `0`. Supply a bootstrap snapshot representing the same block reported by the sequencer's finalized-state endpoint, or reuse the watchdog state directory from the previous deployment of the same monitor. + +The watchdog stores stable deployment configuration, `head.json`, status metrics, and its selected Cartesi machine checkpoint under `CARTESI_WATCHDOG_STATE_DIR`. Keep this directory durable. + +Use watchdog and canonical-machine artifacts built for the same application, chain configuration, and release as the sequencer. A mismatched machine image produces a state mismatch even when the sequencer is operating correctly. + +## Schedule watchdog ticks + +Run one comparison cycle with: + +```bash +sequencer-watchdog tick +``` + +`tick` is not a daemon. It performs one cycle and exits. Schedule it with a systemd timer, cron, or a Kubernetes CronJob. + +The wrapper takes a non-blocking kernel `flock` on its state directory. Also configure the external scheduler to prevent overlapping ticks, such as `concurrencyPolicy: Forbid` for a Kubernetes CronJob. + +Choose a cadence that bounds how long a mismatch can remain undetected while respecting base-layer and Cartesi machine costs. Alert when the last successful tick becomes older than that bound. + +## Interpret watchdog results + +| Exit code | Meaning | Operator action | +| --------: | ------------------------------------------------------------------------- | -------------------------------------------------- | +| `0` | Comparison succeeded, or the promoted block was unchanged | Record success | +| `1` | Transient RPC, network, Cartesi machine, or sequencer error after retries | Retry on the next schedule and alert if persistent | +| `2` | Deterministic state mismatch or inclusion-block regression | Stop and alert immediately | + +Each completed tick atomically writes a Prometheus textfile to: + +```text +$CARTESI_WATCHDOG_STATE_DIR/status.prom +``` + +Override the path with `CARTESI_WATCHDOG_METRICS_FILE`. The file exposes: + +- `cartesi_watchdog_status{chain,app_address,state="ok|warning|failed"}`; +- `cartesi_watchdog_divergence_info{chain,app_address,kind}` when a deterministic failure occurs. + +Alert when the active status is `failed`, when `warning` persists, or when the metrics file stops receiving completed tick results. + +## Use the health endpoints correctly + +Use `/livez` for process liveness and `/readyz` for traffic routing. `/healthz` exposes the same readiness condition in a JSON body. The [API reference](../api-reference/api.md#health-endpoints) defines the exact checks. + +A successful probe does not establish RPC freshness, batch-submitter progress, account balance, storage health, or canonical agreement. Monitor those dependencies separately and do not treat `200` as evidence that batches are reaching the base layer. + +The finalized-state inclusion-block endpoint is a progress cursor for promoted snapshots. Lack of movement is not automatically a failure because an idle application may produce no new closed and accepted batch. + +## Monitor operational dependencies + +Monitor at least: + +### Base-layer view + +- RPC availability and request latency; +- observed safe-head block and timestamp; +- distance between the observed safe head and an independent chain view; +- repeated long-range log-query partitioning or failures; +- wrong-chain detection. + +### Batch submission + +- age of the oldest unsettled batch; +- distance to the danger threshold and staleness deadline; +- submitter transaction inclusion and replacement retries; +- batch-submitter account balance; +- wallet-nonce gaps or a pending nonce that stops advancing. + +### API and feed + +- request latency and throughput; +- rates of `429`, `500`, and `503` responses; +- connected feed subscribers and rejected connections; +- feed-consumer lag and catch-up-window failures. + +### Storage and process + +- data-volume bytes and inodes available; +- SQLite, snapshot, and garbage-collection failures in logs; +- restart count and exit-code distribution; +- time spent in startup or recovery; +- last successful complete backup and archived checkpoint. + +## Recommended alerts + +Page an operator immediately for: + +- sequencer exit code `30`, `40`, or `101`; +- watchdog exit code `2` or `state="failed"`; +- canonical divergence logs; +- a wrong-chain RPC response; +- disk space approaching exhaustion; +- a batch approaching the staleness deadline; +- a depleted submitter account. + +Use warning alerts for persistent exit code `1`, `10`, or `20`, repeated provider failures, growing API overload, stale watchdog ticks, and unexpected lack of progress when the application has known traffic. + +## Protect monitoring endpoints + +Keep health and snapshot routes on the internal operator network. [Separate public and internal routes](./security.md#separate-public-and-internal-routes) defines the route policy and access controls. + +## Next steps + +- Handle process exits using [Process supervision and recovery operations](./orchestration.md). +- Review every route in [HTTP and WebSocket API](../api-reference/api.md). +- Prepare incident response using [Failure modes](../recovery/failure-modes.md). diff --git a/app-sequencer/operations/orchestration.md b/app-sequencer/operations/orchestration.md new file mode 100644 index 000000000..917aeed3a --- /dev/null +++ b/app-sequencer/operations/orchestration.md @@ -0,0 +1,146 @@ +--- +title: "Process supervision and recovery operations" +sidebar_label: "Supervision and recovery" +description: "How to supervise the sequencer, interpret its exit codes, prevent overlapping writers, and resolve submitter wallet nonces." +--- + +The sequencer expects a process supervisor. It deliberately exits when some recovery paths need a clean restart with no active workers, and its exit code tells the supervisor whether to retry or wait for an operator. + +The supervisor must preserve the data directory, prevent overlapping instances, capture logs, and apply the correct retry policy to each exit code. + +## Supervisor responsibilities + +A production supervisor should: + +- run `setup` as a separate initialization step; +- start one `run` process for each sequencer deployment; +- mount the same durable data directory on every restart; +- allow enough startup time for base-layer synchronization and recovery; +- stop the outgoing process before starting a replacement; +- retain logs and the exit code from every process generation; +- use bounded backoff for retryable failures; +- stop and alert on operator-directed exit codes. + +## Exit codes and required actions + +The supervisor should group process exits by required action: + +| Codes | Supervisor policy | +|---|---| +| `0` | Keep the process stopped unless deployment policy explicitly starts it again | +| `1`, `10`, `20` | Capture logs and restart with bounded backoff. Allow a longer startup window for code `10`, which can trigger recovery | +| `2` | Stop until the command or configuration is corrected | +| `30`, `40` | Stop and alert. Code `40` requires the documented checkpoint-recovery procedure | +| `101` | Capture the panic, alert, and use a controlled restart policy while investigating | + +[Constants and exit codes](../api-reference/constants-and-exit-codes.md#exit-codes) is the authoritative definition of every code. [Cockroach recovery](../recovery/cockroach.md) covers the operator procedure required by code `40`. + +## Restart and backoff policy + +Use bounded exponential backoff for codes `1`, `10`, and `20`. Reset the retry counter only after the process has remained healthy for an operationally meaningful period. + +Alert on: + +- repeated code `1`, because an unclassified failure may indicate a software, storage, or persistent provider problem; +- repeated code `10`, because recovery should not trigger continuously; +- prolonged code `20`, because the dependency is not recovering; +- every code `30`, `40`, or `101`. + +Do not use a short fixed startup deadline. A recovery boot can take many minutes while it resolves wallet nonces, waits for the safe head, synchronizes base-layer inputs, and rebuilds application state. + +## Prevent restart loops + +Many container platforms restart every nonzero exit automatically. That policy is unsafe for codes `30` and `40` because neither can be fixed by launching the same command again. + +Use an entrypoint or supervisor that records the child exit code and branches explicitly. Codes `30` and `40` must leave the workload stopped. Code `2` also requires corrected configuration before another attempt. + +Do not infer behavior by parsing error-message text. The exit-code mapping is the stable orchestration contract. + +## Initialization and startup order + +For a new deployment: + +1. provision durable storage; +2. run plain `setup` with the deployment identity and submitter address; +3. mount the submitter key only where keyed commands require it; +4. start `run` under the supervisor; +5. wait for `/readyz` before routing transaction traffic; +6. start the watchdog after its independent state has been initialized. + +A completed plain setup is idempotent, so an initialization job may run again against the same completed data directory. Do not apply that rule to `setup --recovery`, which is a one-shot rebuild command for a fresh directory. + +## Prevent overlapping sequencer instances + +The sequencer has no leader election. Run exactly one `run` process for each deployment and data directory. + +Two instances can: + +- compete for the same submitter wallet nonces; +- produce conflicting local orders; +- contend for the same SQLite database; +- invalidate assumptions used by startup recovery. + +Use a deployment strategy that fully terminates the current process before starting its replacement. Avoid rolling updates that briefly run two replicas. + +The same exclusion applies to keyed maintenance. Never run `flush-mempool` or `setup --recovery` while `run` is active for the same submitter account. + +## Flush unresolved wallet nonces + +The batch-submitter account uses consecutive Ethereum transaction nonces. A broadcast transaction can disappear from the local node's mempool and still survive elsewhere, then land after the operator assumed it was gone. + +`flush-mempool` resolves every nonce slot the deployment may have used. It reads the pinned submitter address and the persisted wallet-nonce watermark from the data directory, sends zero-value self-transfers for unresolved slots, and waits until: + +```text +pending nonce <= safe nonce +and +safe nonce >= persisted watermark + 1 +``` + +The replacement transaction and the original batch transaction compete for the same nonce. Either can win. The command succeeds only after every covered slot has an outcome visible at the RPC's safe level. + +This is not absolute chain irreversibility, and the command does not repair application state or invalidate batches. It only removes uncertainty from the submitter account's nonce range. + +Run it with the sequencer stopped: + +```bash +CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT=https://your-node.example \ +CARTESI_SEQUENCER_AUTH_PRIVATE_KEY_FILE=/run/secrets/submitter-key \ +CARTESI_SEQUENCER_DATA_DIR=/var/lib/cartesi-sequencer \ + flush-mempool +``` + +The command requires: + +- a data directory with completed setup; +- the original wallet-nonce watermark stored in that directory; +- a key matching the pinned submitter address; +- an RPC serving the pinned chain; +- enough base-layer funds for replacement transactions. + +It raises the watermark before broadcasting and waits for the safe nonce to cover that durable boundary. If a transaction watch times out, it rechecks the nonce state and retries. + +## When to run `flush-mempool` + +Use the standalone command when: + +- the submitter account has a wedged nonce range; +- the deployment is being decommissioned and all nonce slots must be resolved; +- an operator runbook explicitly directs you to use it. + +Do not run it after every abrupt shutdown. Normal startup scans the base layer and the submitter retries pending batches. Preemptive recovery and `setup --recovery` also perform their own flush when required. + +Because the command sends replacement transactions, it can cause an original batch transaction to lose its nonce. Run it only when the corresponding operational procedure is prepared to reconcile the canonical result. + +## Preserve state across restarts + +Mount the configured data directory on durable storage that survives process and node replacement. + +Do not restore an old data-directory copy over a deployment that continued running. That local history can conflict with later base-layer activity. Use the documented checkpoint recovery procedure instead. + +See [Data, snapshots, and backups](./data-and-state.md) for its contents, backup procedure, and restore boundaries. + +## Next steps + +- Configure the process using [Configure, set up, and run the sequencer](./setup-and-running.md). +- Define alerts with [Monitoring and watchdog operation](./monitoring.md). +- Follow [Preemptive recovery](../recovery/preemptive.md) and [Cockroach recovery](../recovery/cockroach.md) for the two recovery paths. diff --git a/app-sequencer/operations/security.md b/app-sequencer/operations/security.md new file mode 100644 index 000000000..62f77e07b --- /dev/null +++ b/app-sequencer/operations/security.md @@ -0,0 +1,127 @@ +--- +title: "Production security" +sidebar_label: "Production security" +description: "Protect the submitter account, control public and internal routes, secure the base-layer connection, and prevent concurrent writers." +--- + +Production hardening focuses on four boundaries: the batch-submitter key, the HTTP listener, the base-layer RPC connection, and exclusive control of the deployment's state and wallet nonces. + +The sequencer validates signed user transactions and protects several protocol boundaries itself. Authentication, TLS termination, traffic filtering, secret distribution, and instance coordination belong to the surrounding infrastructure. + +## Protect the batch-submitter key + +The batch-submitter key authorizes base-layer transactions from the address identified as the sequencer. Its holder can consume that account's nonces and submit inputs that the scheduler attempts to decode as batches. + +The key is online because `run` signs batch submissions and recovery may sign replacement transactions. Store it in a secret manager and expose it to the process through a read-only file: + +```bash +CARTESI_SEQUENCER_AUTH_PRIVATE_KEY_FILE=/run/secrets/submitter-key +``` + +The sequencer reads the first line, trims it, derives the address, and checks that it matches the submitter identity pinned during setup. + +Avoid `CARTESI_SEQUENCER_AUTH_PRIVATE_KEY` in production. Environment values can appear in process inspection, container configuration, diagnostic output, and shell history. + +Exactly one key source must be configured for: + +- `run`; +- `flush-mempool`; +- `setup --recovery`. + +Plain `setup` rejects a signing key and accepts only the public submitter address. + +Restrict the key file to the sequencer identity, mount it read-only, prevent it from entering logs or backups, and define a rotation procedure that accounts for the submitter identity pinned in storage and in canonical scheduler configuration. + +## Use a dedicated submitter account + +Use one batch-submitter account for one sequencer deployment. Do not share it with unrelated scripts, applications, or operator transactions. + +The sequencer assigns consecutive Ethereum wallet nonces and persists the highest nonce it intends to use before broadcasting. Uncoordinated activity from another process can consume a nonce, block later transactions, or introduce an input the local batch history does not expect. + +Keep the account funded and monitor its balance. Batch submission and mempool flushing both spend base-layer gas. A depleted account can prevent submission until unsettled batches approach the staleness deadline. + +## Control public API exposure + +The sequencer does not authenticate HTTP or WebSocket clients. + +`POST /tx` requires a valid user signature, which prevents transaction forgery. It does not prevent request floods, signature-recovery work, application-validation load, or pressure on the inclusion queue. + +`GET /ws/subscribe` also has no client authentication. Its fixed subscriber and catch-up limits do not replace perimeter protection. See the [WebSocket API reference](../api-reference/api.md#get-wssubscribe) for their exact values. + +Place a gateway or reverse proxy in front of the sequencer to provide: + +- TLS termination; +- network and identity-based access controls where required; +- request-rate and connection-rate limiting; +- client quotas; +- observability and abuse detection; +- request-body limits no larger than the sequencer's own 4 KiB limit for `POST /tx`. + +The default listener is `127.0.0.1:3000`, which is local to the host. Changing `CARTESI_SEQUENCER_HTTP_ADDR` to a non-loopback interface is an explicit exposure decision. + +## Separate public and internal routes + +The current runtime merges public, feed, health, and snapshot routes onto one HTTP listener. It does not offer separate bind addresses or ports for ingress and operator endpoints. + +Treat the route groups differently at the gateway: + +| Route | Intended audience | +| ---------------------------------------------------------- | ------------------------------------ | +| `POST /tx` | Application clients | +| `GET /ws/subscribe` | Authorized frontends and indexers | +| `GET /livez`, `/readyz`, `/healthz` | Orchestrator and internal monitoring | +| `GET /finalized_state`, `/finalized_state/inclusion_block` | Watchdog and trusted operators | +| `GET /latest_snapshot` | Trusted indexers and operators | + +The snapshot routes stream application state and have no authentication. Deny them at the public ingress. A single-listener deployment can still enforce separation through path-based proxy rules and network policy. + +Health endpoints reveal little data, but they are not required by public clients and should remain on the internal route tier. + +## Secure the base-layer RPC connection + +The sequencer refuses plaintext RPC to a non-loopback host by default. Use an `https://` endpoint whenever traffic leaves the host. + +For a Docker, Kubernetes, or private VPC network where plaintext transport is intentionally protected by the network boundary, set: + +```bash +CARTESI_SEQUENCER_ALLOW_INSECURE_RPC=true +``` + +This option only permits the transport. It does not authenticate the RPC, validate its operational quality, or protect traffic after it leaves the trusted network. + +Apply the setting to every command that uses the RPC. Review it whenever the endpoint changes so a private-network exception is not accidentally reused with a public provider. + +Keyed write paths verify the live RPC chain identifier against the deployment's pinned chain before signing. A mismatch is terminal. Protect RPC credentials separately because they can grant paid capacity or expose application traffic even when they cannot sign submitter transactions. + +## Use one consistent RPC source + +The design assumes the configured node can fail but returns a consistent, truthful chain view when it responds. + +Avoid generic load-balanced pools in which requests can reach replicas with different safe heads, log indexes, mempools, or even chain configurations. This is especially important because setup detection, batch submission, recovery flushing, and input synchronization rely on related observations across calls. + +If availability infrastructure sits in front of the node, it must preserve a consistent view and must not silently fail over to a different chain. Monitor the RPC's chain identifier, safe-head lag, log completeness, and error rate. + +## Enforce single-instance operation + +The deployment has no leader election or distributed writer coordination. Enforce one active keyed writer for each submitter account and data directory. [Prevent overlapping sequencer instances](./orchestration.md#prevent-overlapping-sequencer-instances) defines the excluded command combinations and safe update behavior. + +## Production hardening checklist + +- Store the submitter key in a read-only secret file. +- Use a dedicated, funded submitter account. +- Run exactly one keyed writer for the deployment. +- Keep the sequencer listener on an internal network. +- Terminate TLS and enforce path-based access at a gateway. +- Apply rate and connection limits to public API routes. +- Block snapshot and health routes from the public internet. +- Use one consistent HTTPS RPC source. +- Mount the data directory on durable storage with restricted permissions. +- Monitor disk capacity, submitter balance, exit codes, batch age, and watchdog status. +- Archive complete checkpoint directories outside the live data volume. +- Rehearse process restart and checkpoint recovery before production launch. + +## Next steps + +- Configure the runtime using [Configure, set up, and run the sequencer](./setup-and-running.md). +- Supervise keyed processes with [Process supervision and recovery operations](./orchestration.md). +- Review the broader assumptions in [Trust model and guarantees](../foundations/trust-model.md). diff --git a/app-sequencer/operations/setup-and-running.md b/app-sequencer/operations/setup-and-running.md new file mode 100644 index 000000000..1b9f3f539 --- /dev/null +++ b/app-sequencer/operations/setup-and-running.md @@ -0,0 +1,163 @@ +--- +title: "Configure, set up, and run the sequencer" +sidebar_label: "Configure and run" +description: "Configure an application-specific sequencer, initialize its data directory, start its workers, and verify readiness." +--- + +The sequencer has three commands with different responsibilities: + +- **`setup`** initializes a data directory and pins it to one deployment. +- **`run`** starts the API and background workers from an initialized directory. +- **`flush-mempool`** resolves uncertain transaction nonces for the batch-submitter account during operator-directed maintenance. + +Plain `setup` is read-only on the base layer and uses the submitter address without its private key. The keyed commands are `run`, `flush-mempool`, and `setup --recovery`. + +## Configuration model + +Every command-line option can also be supplied through a `CARTESI_SEQUENCER_*` environment variable. Command-line values take precedence when both forms are present. + +The executable only accepts settings used by the selected command. Deployment identity is provided to `setup` and then stored in the data directory. Later commands read it from storage instead of accepting another chain identifier or application address. + +Set `CARTESI_SEQUENCER_DATA_DIR` explicitly in production. Its default, `sequencer-data`, is relative to the process working directory and can resolve to ephemeral storage in a container. + +## Settings required by each command + +| Setting | `setup` | `run` | `flush-mempool` | Default | +| --------------------------------------------------------------------------------- | --------------------------------------------------- | -------------------- | -------------------- | ---------------- | +| `CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT` | Required | Required | Required | None | +| `CARTESI_SEQUENCER_DATA_DIR` | Optional | Optional | Optional | `sequencer-data` | +| `CARTESI_SEQUENCER_BLOCKCHAIN_ID` | Required | Read from storage | Read from storage | None | +| `CARTESI_SEQUENCER_APP_ADDRESS` | Required | Read from storage | Read from storage | None | +| `CARTESI_SEQUENCER_BATCH_SUBMITTER_ADDRESS` | Required | Read from storage | Read from storage | None | +| `CARTESI_SEQUENCER_AUTH_PRIVATE_KEY_FILE` or `CARTESI_SEQUENCER_AUTH_PRIVATE_KEY` | Rejected by plain setup; required by recovery setup | Exactly one required | Exactly one required | None | +| `CARTESI_SEQUENCER_ALLOW_INSECURE_RPC` | Optional | Optional | Optional | `false` | +| `CARTESI_SEQUENCER_SECONDS_PER_BLOCK` | Optional | Optional | Optional | `12` | + +Prefer `CARTESI_SEQUENCER_AUTH_PRIVATE_KEY_FILE`. The file's first line must contain the hexadecimal private key. The sequencer derives its address and rejects it if it does not match the submitter address pinned during setup. + +## Initialize a new deployment + +Run plain `setup` once for a new data directory: + +```bash +CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT=https://your-node.example \ +CARTESI_SEQUENCER_BLOCKCHAIN_ID=1 \ +CARTESI_SEQUENCER_APP_ADDRESS=0xYourApplicationAddress \ +CARTESI_SEQUENCER_BATCH_SUBMITTER_ADDRESS=0xYourSubmitterAddress \ +CARTESI_SEQUENCER_DATA_DIR=/var/lib/cartesi-sequencer \ + setup +``` + +Plain setup performs these tasks: + +1. validates the timing configuration and creates the data and dump directories; +2. verifies the RPC chain identifier; +3. discovers the application's `InputBox` and its genesis block; +4. pins the chain, application, `InputBox`, and submitter identities; +5. synchronizes base-layer inputs through the current safe head; +6. registers the application's genesis state as the initial finalized snapshot; +7. writes the setup-complete marker. + +It sends no base-layer transaction and requires no signing key. A successful plain setup is idempotent: running it again against the completed directory returns without changing the deployment. + +If setup detects batch-submitter activity that a new deployment cannot account for, it exits with code `40`. The remedy is an explicit checkpoint rebuild with `setup --recovery`, not another plain setup attempt. Recovery setup requires a fresh data directory, a finalized checkpoint, its inclusion block, and the submitter key. See [Cockroach recovery](../recovery/cockroach.md). + +## Start an initialized deployment + +Start the sequencer with the same data directory and the matching submitter key: + +```bash +CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT=https://your-node.example \ +CARTESI_SEQUENCER_AUTH_PRIVATE_KEY_FILE=/run/secrets/submitter-key \ +CARTESI_SEQUENCER_DATA_DIR=/var/lib/cartesi-sequencer \ + run +``` + +`run` refuses to start unless setup completed and the data directory contains a deployment identity and finalized snapshot. It reads the chain identifier, application address, `InputBox`, genesis block, and submitter address from storage. + +At startup it validates the supplied key against the pinned submitter address. It also checks the RPC chain when the node is reachable. A warm start can continue temporarily from its pinned identity if the initial chain-identifier query fails, but the input reader verifies the chain again on its first successful connection. + +After startup, the process runs: + +- the HTTP and WebSocket server; +- the inclusion lane that orders and executes accepted transactions; +- the base-layer input reader; +- the batch submitter; +- the danger detector; +- snapshot promotion and cleanup. + +## Configure batching and submission + +The following `run` settings control when batches close and how the submitter polls and observes them: + +| Setting | Default | Effect | +| --------------------------------------------------------- | ------: | ------------------------------------------------------------------------------------------------------------------------ | +| `CARTESI_SEQUENCER_MAX_BATCH_OPEN_SECONDS` | `7200` | Forces an open batch to close after two hours even when it has not reached its size target | +| `CARTESI_SEQUENCER_BATCH_SUBMITTER_IDLE_POLL_INTERVAL_MS` | `5000` | Sets the delay before an idle or transiently failing submitter checks again | +| `CARTESI_SEQUENCER_BATCH_SUBMITTER_CONFIRMATION_DEPTH` | `2` | Waits for the inclusion confirmation plus two additional confirmations before the submitter considers its watch complete | + +A shorter batch-open limit reduces the time a transaction can wait before submission on a quiet application. It also creates smaller batches and can increase base-layer cost per transaction. + +The confirmation-depth setting controls the submitter's transaction watcher. Canonical snapshot promotion and input processing still follow the base-layer safe head observed by the input reader. + +## Configure protocol timing + +Three settings control when the sequencer distrusts its base-layer view or stops before a batch becomes stale: + +| Setting | Default | Effect | +| ---------------------------------------------- | ------: | ---------------------------------------------------------------------------------------------------- | +| `CARTESI_SEQUENCER_PREEMPTIVE_MARGIN_BLOCKS` | `300` | Reserves this many blocks of recovery runway before the 1,200-block staleness deadline | +| `CARTESI_SEQUENCER_L1_READ_STALE_AFTER_BLOCKS` | `600` | Rejects a base-layer safe view whose timestamp is this many assumed blocks old | +| `CARTESI_SEQUENCER_SECONDS_PER_BLOCK` | `12` | Converts wall-clock delay into estimated missed blocks and sets several polling and timeout cadences | + +The danger threshold is: + +```text +1200 - CARTESI_SEQUENCER_PREEMPTIVE_MARGIN_BLOCKS +``` + +The preemptive margin must be greater than zero and lower than `1200`. The read-staleness value must be greater than zero and strictly lower than the resulting danger threshold. Startup rejects an invalid combination. + +These settings are shared by `setup` and `run` so the initial sync and live process use the same timing model. `flush-mempool` uses only `SECONDS_PER_BLOCK` to pace confirmation watches and safe-head polling. + +The defaults assume approximately twelve-second blocks. Review all three together before deploying on a chain with different timing. + +## Configure base-layer RPC behavior + +`CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT` should identify one consistent base-layer node. The threat model assumes the node may fail but does not return a deliberately false view. Avoid a load-balanced endpoint whose replicas can disagree about chain identity, safe-head position, or available logs. + +Remote RPC endpoints must use HTTPS. Plain HTTP is accepted automatically for loopback hosts. To use HTTP on a trusted private network, set: + +```bash +CARTESI_SEQUENCER_ALLOW_INSECURE_RPC=true +``` + +This setting permits plaintext transport. Configure it on every command that connects to the base layer. [Production security](./security.md#secure-the-base-layer-rpc-connection) covers authentication, consistency, and deployment controls for the RPC boundary. + +`CARTESI_SEQUENCER_LONG_BLOCK_RANGE_ERROR_CODES` is a comma-separated list of provider error codes that cause a failed `eth_getLogs` request to be split into smaller block ranges. The defaults are: + +```text +-32005,-32600,-32602,-32616 +``` + +Only matching RPC errors trigger range splitting. A transport timeout or another error code is returned through the normal provider error path. + +## Verify startup and readiness + +The API listens on `127.0.0.1:3000` by default. Change it with `CARTESI_SEQUENCER_HTTP_ADDR` when the service must accept connections from another interface. + +Use these probes: + +```bash +curl --fail http://127.0.0.1:3000/livez +curl --fail http://127.0.0.1:3000/readyz +curl --fail http://127.0.0.1:3000/healthz +``` + +Use `/livez` for process liveness and `/readyz` for traffic routing. `/healthz` returns the readiness condition as JSON. These probes cover runtime availability, not dependency progress or canonical correctness. See the exact response semantics in [HTTP and WebSocket API](../api-reference/api.md#health-endpoints) and the operational limits in [Monitoring and watchdog operation](./monitoring.md#use-the-health-endpoints-correctly). + +## Next steps + +- Configure the supervisor using [Process supervision and recovery operations](./orchestration.md). +- Protect the deployment using [Production security](./security.md). +- Plan durable storage and backups with [Data, snapshots, and backups](./data-and-state.md). diff --git a/app-sequencer/overview/app-specific-sequencing.md b/app-sequencer/overview/app-specific-sequencing.md new file mode 100644 index 000000000..10d528966 --- /dev/null +++ b/app-sequencer/overview/app-specific-sequencing.md @@ -0,0 +1,91 @@ +--- +title: "App-specific sequencing" +sidebar_label: "App-specific sequencing" +description: "What an app-specific sequencer does, why an application might use one, and how its early ordering relates to canonical execution." +--- + +An **app-specific sequencer** is a service dedicated to one Cartesi application. It receives signed transactions, decides their provisional order, executes them against its current view of application state, and returns a fast response before those transactions settle through the base layer. + +A Cartesi application runs its logic inside a [Cartesi machine](/cartesi-machine). Its inputs are recorded through contracts on the base layer, while commitments to the resulting state are settled there. If these concepts are new to you, begin with [Cartesi Rollups](/cartesi-rollups/1.5/). + +The App Sequencer adds a fast transaction path to this design. It does not replace the base layer, the application's Cartesi machine, or the mechanisms that determine canonical state. + +## Why use an app-specific sequencer + +A rollup application needs a deterministic transaction order because changing the order can change the result. It also needs to decide how quickly users learn whether their transactions were accepted. + +An application can rely on one of three broad approaches: + +| Approach | Response time | Capacity and control | Base-layer cost | +| ------------------------------------------ | -------------------------------------------- | -------------------------------------------------------------- | -------------------------------------------------------------------------------------- | +| Send each input directly to the base layer | Tied to base-layer processing and settlement | No sequencer to operate | Each sender pays for an individual input | +| Use shared sequencing infrastructure | Fast, depending on the service | Capacity and policy are shared with other applications | Costs may be shared across applications | +| Run an app-specific sequencer | Fast soft confirmations | Capacity and operating policy are dedicated to one application | The operator funds batch submissions and may recover the cost through application fees | + +An app-specific sequencer offers four main benefits: + +- **Dedicated capacity.** Other applications do not compete for space inside the sequencer, although every submitted batch still competes for space on the base layer. +- **Fast feedback.** The sequencer returns a soft confirmation after it accepts, executes, and durably stores a transaction in its current ordering. +- **Application-level policy.** The operator controls settings such as batch sizing and fee policy. Protocol ordering rules remain fixed so that configuration cannot silently change canonical execution. +- **Base-layer verifiability.** The application still derives its canonical result from inputs recorded on the base layer. + +## How transaction ordering affects application state + +Sequencing answers a basic question: **which transaction executes first?** + +Every participant must eventually use the same answer. For example, a withdrawal can succeed or fail depending on whether a deposit or another withdrawal executes before it. If different components use different orders, they calculate different application states. + +Without a sequencer, the base layer records individual inputs and determines their recorded positions. Users must wait for base-layer progress before treating the result as sufficiently settled. + +The App Sequencer provides an earlier answer. It orders and executes a signed transaction off-chain as soon as it can safely accept it. This improves responsiveness, but the result remains provisional until the corresponding batch is accepted through the canonical path. + +## One sequencer for each application + +An app-specific sequencer serves exactly one application deployment. Its signing domain, application contract, batch-submitter identity, local state, and transaction validation are tied to that deployment. + +This matches the structure of Cartesi applications. Each application has its own logic, contracts, and state. The application team integrates the sequencer library with that logic and operates the resulting application-specific binary. + +Dedicated sequencing provides isolation and control, but it also makes the application team responsible for operating the service and funding batch submissions. + +## Where the sequencer fits in the transaction path + +![A user can send a signed transaction through the App Sequencer and receive a soft confirmation before the sequencer posts a batch to Ethereum. The user can also submit a direct input to Ethereum without using the sequencer. Both paths reach the scheduler and application logic inside the Cartesi machine.](../images/transaction-paths.png) + +An input can reach the application through either path: + +- A **sequenced transaction** goes through the sequencer and is later posted in a batch. +- A **direct input** goes straight to the base-layer `InputBox`. Deposits normally use this path because portal contracts submit them directly. + +Both sources become part of the application's canonical input history. [Direct and sequenced inputs](../concepts/direct-vs-sequenced.md) compares their behavior, costs, and availability. + +## How the sequencer predicts the final order + +The sequencer maintains a fast, provisional view of order and application state. The scheduler inside the Cartesi machine later derives the authoritative order from base-layer inputs. Both apply matching protocol rules, which allows the sequencer to predict the scheduler's result during normal operation. + +[Architecture at a glance](../foundations/architecture.md) follows this path from submission to settlement. [Batches, frames, and the safe block](../concepts/batches-frames-safe-block.md) explains the data structure, and [Deterministic execution order](../concepts/execution-order.md) explains how both input paths are combined. + +### When the predicted order can change + +An HTTP success response and a feed entry both describe the sequencer's provisional view. That view can change if a batch fails to join canonical execution. [Soft confirmations](../concepts/soft-confirmations.md) explains the client contract, while [Staleness and the danger zone](../concepts/staleness.md) explains the main liveness condition that can invalidate a sequence of batches. + +## Guarantees and limitations + +The App Sequencer provides fast transaction acceptance and a prediction of execution order. It does not provide base-layer settlement. + +The protocol limits the sequencer in several important ways: + +- It cannot create a valid user transaction without the user's signature. +- It cannot directly determine canonical application state. The application computes that state from the order derived inside the Cartesi machine. +- It cannot rewrite history that has already settled through the canonical path. +- It does not hold user assets as part of sequencing. + +The operator still has meaningful power over the fast path. It can refuse, delay, or reorder transactions that users submit to it, and it sees those transactions before they reach the base layer. These choices can have financial consequences in an order-sensitive application. + +Users can submit direct inputs without the sequencer, but the actions available through that route depend on the application. Direct inputs are also slower and require the sender to pay the base-layer transaction cost. + +[Trust model and guarantees](../foundations/trust-model.md) describes these boundaries in detail. + +## Next steps + +- To decide whether the operational and trust tradeoffs suit your application, read [When to use the App Sequencer](./when-to-use.md). +- To see how the system components fit together, read [Architecture at a glance](../foundations/architecture.md). diff --git a/app-sequencer/overview/when-to-use.md b/app-sequencer/overview/when-to-use.md new file mode 100644 index 000000000..112ae4e15 --- /dev/null +++ b/app-sequencer/overview/when-to-use.md @@ -0,0 +1,88 @@ +--- +title: "When to use the App Sequencer" +sidebar_label: "When to use it" +description: "How to decide whether fast soft confirmations, dedicated sequencing, and their operational tradeoffs suit an application." +--- + +The App Sequencer is most useful when fast feedback materially improves an application and the team is prepared to operate a dedicated service. It adds responsiveness, but it also introduces provisional results and centralized control over the fast transaction path. + +## Applications that benefit from a sequencer + +Consider the App Sequencer when several of these conditions apply: + +| Application characteristic | Likely benefit | Reason | +| --------------------------------------------------------- | -------------- | ----------------------------------------------------------------------- | +| Users expect an immediate response | High | A soft confirmation arrives before base-layer settlement | +| Many actions happen within application state | High | Transactions can be grouped into batches instead of posted individually | +| The application has steady or bursty traffic | High | Batches can spread submission costs across multiple transactions | +| The team needs control over capacity and fees | High | The sequencer is dedicated to one application | +| Transaction order affects user experience | Mixed | Fast ordering helps, but the operator becomes trusted for fairness | +| Most activity consists of deposits or other direct inputs | Low | Those inputs bypass the sequencer | +| Users already tolerate base-layer settlement time | Low | A faster provisional state may add little value | + +Games, trading interfaces, collaborative applications, and social applications can benefit when users need to continue interacting without waiting for every action to settle. + +The strongest fit is an application that can safely distinguish between an action that is **accepted** and one that is **settled**. + +## When a sequencer may add little value + +The App Sequencer may not be a good fit when: + +- users are comfortable waiting for the base layer; +- most inputs must already be submitted directly through base-layer contracts; +- the application cannot represent provisional results or recover from their invalidation; +- the team cannot maintain a stateful, availability-sensitive service; +- centralized transaction ordering creates an unacceptable fairness or regulatory risk. + +An application should also consider what remains possible without the sequencer. The direct-input route is always available at the protocol level, but the application decides which actions direct inputs can perform. If direct inputs only support deposits, an unavailable sequencer can leave other application actions waiting until service returns. + +## Operational requirements + +Running an app-specific sequencer requires: + +- an application-specific binary that integrates the sequencer library with the application's execution logic; +- persistent local storage for ordering, batches, snapshots, and recovery state; +- a dedicated, funded base-layer account for submitting batches; +- reliable access to one consistent base-layer node; +- monitoring for API availability, base-layer progress, batch submission, staleness risk, and divergence; +- backup and recovery procedures for application snapshots and sequencer state. + +The sequencer pays the base-layer cost of each batch. The application can charge sequencer fees to recover that cost, but it must choose a fee and batch policy suited to its traffic. Quiet applications may leave transactions in an open batch longer, while busy applications tend to close batches by size. + +[Configure, set up, and run the sequencer](../operations/setup-and-running.md) and [Fees and data availability](../concepts/fees.md) cover these responsibilities in detail. + +## Trust and user experience tradeoffs + +Adopting the App Sequencer means accepting two important tradeoffs. + +### The operator controls the fast-path order + +The sequencer can refuse or delay a submitted transaction. It can also choose the order among transactions it accepts because the protocol does not enforce first come, first served behavior. + +This is especially important for trading and other order-sensitive applications. The sequencer cannot forge a user's signature or dictate canonical application state, but its ordering decisions can still create or redistribute economic value. + +Users can bypass the sequencer by submitting a direct input, subject to the actions supported by the application. This route is slower and requires the user to pay the base-layer cost. + +### Soft confirmations can be invalidated + +A soft confirmation reports the sequencer's current prediction before settlement. If its batch is not accepted through the canonical path, recovery can remove the transaction from the valid provisional history. + +The interface must communicate the difference between accepted and settled. It also needs a way to reconcile outstanding transactions because the live feed does not publish rollback messages. See [Soft confirmations](../concepts/soft-confirmations.md). + +## A practical decision rule + +Use the App Sequencer when fast feedback materially improves the product and the team can: + +1. operate and monitor the service reliably; +2. fund and manage batch submission; +3. accept centralized control over provisional ordering; +4. show users which results remain unsettled; +5. reconcile or recover provisional state when a batch is invalidated. + +If one of these conditions is unacceptable, direct base-layer inputs or another sequencing model may be a better fit. + +## Next steps + +- To understand the system boundaries, read [Architecture at a glance](../foundations/architecture.md). +- To evaluate its guarantees, read [Trust model and guarantees](../foundations/trust-model.md). +- To begin integrating it, follow the [Quickstart](../usage/quickstart.md). diff --git a/app-sequencer/recovery/cockroach.md b/app-sequencer/recovery/cockroach.md new file mode 100644 index 000000000..b951d5608 --- /dev/null +++ b/app-sequencer/recovery/cockroach.md @@ -0,0 +1,178 @@ +--- +title: "Cockroach recovery" +sidebar_label: "Cockroach recovery" +description: "How to rebuild a sequencer from a trusted checkpoint and base-layer history when the local database is lost or cannot be trusted." +--- + +Cockroach recovery is the repository's name for rebuilding a deployment from a trusted checkpoint after its local database is lost or cannot be trusted. + +The procedure does not repair the old database. It creates a fresh data directory, loads application state from the checkpoint, and reconstructs everything that happened after that checkpoint from the base layer. + +This is an explicit, operator-run procedure using `setup --recovery`. It is separate from the automatic [preemptive recovery](./preemptive.md) used for an intact database with a failing provisional batch suffix. + +## When to use checkpoint recovery + +Use this procedure when: + +- the data directory is lost or corrupted beyond use; +- canonical divergence causes the sequencer to exit with code `30`; +- a fresh setup detects previous batch-submitter activity and exits with code `40`; +- an incident makes the local batch history untrustworthy. + +Do not use it for an ordinary crash, a temporary provider outage, or routine danger-zone recovery. Those cases retain a usable database and follow the normal startup path. + +## Required recovery inputs + +Prepare all of the following before wiping or replacing any deployment data: + +| Input | Requirement | +| -------------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| Checkpoint directory | A complete archived non-genesis finalized dump produced by this sequencer, containing `state` and `info.toml` | +| Checkpoint block `B` | The `promoted_inclusion_block` stored in that checkpoint's `info.toml` | +| Deployment identity | The same chain, application address, and batch-submitter address used by the original deployment | +| Submitter key | The key matching the configured batch-submitter address | +| Submitter funds | Enough base-layer funds to submit every required flush transaction | +| RPC endpoint | A consistent endpoint that provides safe-block reads, historical logs, state queries, fee estimation, and transaction submission | +| Compatible binary | A release that can load the checkpoint's application state and `info.toml` format | +| Fresh data directory | An empty replacement directory that no running sequencer can access | + +Recovery trusts the checkpoint's application state and next batch nonce. Keep the entire checkpoint directory together and do not edit its metadata. + +## Run the recovery command + +Stop the current sequencer and prevent any other instance from using the submitter key. Then run: + +```bash +CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT=https://your-node.example \ +CARTESI_SEQUENCER_BLOCKCHAIN_ID= \ +CARTESI_SEQUENCER_APP_ADDRESS= \ +CARTESI_SEQUENCER_BATCH_SUBMITTER_ADDRESS= \ +CARTESI_SEQUENCER_AUTH_PRIVATE_KEY_FILE=/run/secrets/submitter-key \ + setup --recovery \ + --checkpoint-dump-dir /path/to/archived/checkpoint \ + --checkpoint-block +``` + +Run the command against a freshly prepared data directory. Recovery setup is a one-shot operation and refuses to run over a completed setup. + +The command-line checkpoint block must exactly match the `promoted_inclusion_block` recorded in the archived `info.toml`. The current implementation requires the value as an argument but does not compare it with that metadata field. Treat the archived value as authoritative and copy it exactly. + +## Recovery data model + +The procedure uses six values: + +| Symbol | Meaning | Source | +| ------ | -------------------------------------------------------------- | ------------------------------------ | +| **S** | Trusted application state at the checkpoint | The checkpoint's `state` subtree | +| **A** | Last safe block whose direct inputs are already reflected in S | Read from the application state | +| **B** | Base-layer block where the checkpoint was promoted | `info.toml` and `--checkpoint-block` | +| **N** | Next batch nonce recorded at the checkpoint | `info.toml` | +| **C** | Safe block where all submitter wallet nonces are resolved | Returned by the flush | +| **N'** | Next batch nonce after replaying accepted batches through C | Computed during recovery | + +The checkpoint must satisfy `A < B`. This leaves a well-defined range of direct inputs that arrived before checkpoint promotion but were not yet represented in the saved application state. + +## How reconstruction works + +### Discover and pin the deployment + +Setup verifies the configured chain ID, discovers the application's `InputBox` and genesis block, and pins the deployment identity in the new database. It refuses a checkpoint block earlier than the `InputBox` genesis block. + +### Synchronize base-layer inputs + +The input reader loads direct inputs and batch submissions through the current safe head. During recovery setup, accepted-frontier construction is deferred until the rebuilt batch-tree anchor is available. + +### Load the checkpoint + +Recovery reads: + +- the application state **S** from `state`; +- the next batch nonce **N** from `info.toml`; +- the last executed safe block **A** from the application state; +- the operator-supplied checkpoint block **B**. + +The command stops if the checkpoint cannot be parsed, the application cannot load its state, or `A >= B`. + +### Resolve the submitter wallet + +Recovery invokes the [wallet-flush mechanism](../operations/orchestration.md#flush-unresolved-wallet-nonces). The flush returns **C**, the safe block where nonce resolution was observed. Recovery synchronizes again and refuses to continue if the refreshed view is behind C. + +An intact runtime database stores the highest wallet nonce the deployment has broadcast. A freshly rebuilt database does not have that watermark. During checkpoint recovery, the flush therefore covers the nonce range visible through the configured provider but cannot prove coverage of a transaction that the provider has forgotten while another network participant still retains it. Use a consistent, well-connected RPC source, preserve the old data directory when it is available, and keep the content-identity alert active after recovery. A previously unseen transaction that later creates a content mismatch causes the sequencer to stop instead of continuing from a false frontier. + +### Reconstruct the missing interval + +Starting from S and N, recovery processes two non-overlapping ranges: + +- **Seed range `(A, B]`:** direct inputs that existed before checkpoint promotion but were not yet executed in S. +- **Replay range `(B, C]`:** all application inputs, including direct inputs and batch submissions, after the checkpoint through the flush boundary. + +The scheduler rules classify and apply the replay stream. Accepted batches advance the next expected batch nonce; stale batches, wrong-nonce batches, and flush transactions do not. + +### Persist the rebuilt deployment + +Recovery writes the reconstructed application state as the new finalized snapshot at block C. It stores the resulting next batch nonce N' as the batch-tree anchor, places the feed cursor after inputs already represented in the rebuilt state, and marks setup complete last. + +The next `run` opens its first local batch at N'. Base-layer inputs after C remain pending and are processed normally during catch-up. + +## Worked example + +Assume the archived checkpoint contains: + +- `B = 1,000,000`, the promotion block; +- `N = 500`, so the next batch expected at the checkpoint is batch 500; +- `A = 999,950`, the last safe block reflected in the application state. + +Two direct inputs at blocks 999,970 and 999,990 fall in `(A, B]`. They are seeded before replay. After the checkpoint, batches 500, 501, and 502 reach the base layer along with more direct inputs. + +The wallet flush completes at `C = 1,000,100`. Recovery then: + +1. loads the checkpoint state at N = 500; +2. seeds the two direct inputs in `(999,950, 1,000,000]`; +3. replays all application inputs in `(1,000,000, 1,000,100]`; +4. advances the batch nonce for each batch accepted by the scheduler; +5. produces N' = 503; +6. saves the rebuilt state at C and anchors the new local tree at 503. + +Normal operation resumes by creating batch 503. Inputs after C remain available for the ordinary input reader and are not silently skipped. + +## Why the resume nonce must be exact + +The scheduler expects one specific batch nonce. In the example, batches through 502 were accepted, so the first unused nonce is 503. + +- A resume nonce that is too low collides with a nonce already accepted by the scheduler and causes a visible refusal. +- A resume nonce that is too high creates a gap. No accepted batch occupies the missing nonce, so later batches cannot advance the scheduler. + +Recovery therefore reads N from the sequencer-produced checkpoint and computes N' by replay. Do not guess either value or assemble `info.toml` manually. + +## Trust boundary and validation limits + +Recovery validates the checkpoint format, application loading, deployment identity, chain ID, block ranges, and the post-flush safe view. It does not independently prove that the checkpoint state and next batch nonce are historically correct. + +Replaying from genesis would provide that proof, but it would remove the main benefit of checkpoint recovery. Operational safety therefore depends on archiving genuine promoted checkpoints, protecting them from modification, recording their provenance, and testing the restore procedure. + +See [Snapshots and checkpoints](./snapshots.md) for the archive requirements. + +## Interrupted recovery + +Recovery setup writes its completion marker last. If the command stops before that marker is written, the directory is not considered ready for `run`. + +Do not resume from a partially rebuilt directory. Preserve logs for diagnosis, remove the incomplete recovery directory, create a fresh one, and run the command again from the same verified checkpoint. The wallet flush and base-layer replay are designed to be repeated, but partial local recovery state is not accepted as a continuation point. + +## Validate the rebuilt deployment + +Before reopening public traffic: + +1. confirm that recovery setup exits successfully; +2. start `run` with the original deployment identity and submitter key; +3. verify `/livez`, `/readyz`, and the finalized snapshot endpoint; +4. confirm that new transactions can be accepted and batches can reach the base layer; +5. run the independent watchdog comparison; +6. reconcile clients and indexers from a known feed offset. + +Preserve the old data directory and incident evidence until the rebuilt deployment has been independently verified. + +## Next steps + +- Create valid archives using [Snapshots and checkpoints](./snapshots.md). +- Review automatic repair in [Preemptive recovery](./preemptive.md). +- Prepare supervision and startup policy with [Process supervision and recovery operations](../operations/orchestration.md). diff --git a/app-sequencer/recovery/failure-modes.md b/app-sequencer/recovery/failure-modes.md new file mode 100644 index 000000000..27426e032 --- /dev/null +++ b/app-sequencer/recovery/failure-modes.md @@ -0,0 +1,82 @@ +--- +title: "Failure modes" +sidebar_label: "Failure modes" +description: "How the sequencer responds to base-layer outages, crashes, stale batches, divergence, and loss of local state." +--- + +The sequencer distinguishes failures that delay progress from failures that make its local state unsafe to use. + +- A **liveness failure**, such as a temporary RPC outage, is retried or handled through a controlled restart. +- A **recoverable batch failure** causes the sequencer to abandon provisional work and continue from the last accepted batch. +- A **correctness failure** stops the deployment until an operator rebuilds or repairs it. + +This distinction determines whether the supervisor should restart the process, wait for an external dependency, or stop and alert an operator. + +## Failure response summary + +| Condition | Sequencer response | Operator response | +| ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- | +| Temporary base-layer RPC failure | The input reader and batch submitter retry | Monitor the outage and allow retries | +| Base-layer view becomes too old to trust | The process exits with code `20` and refuses startup until a usable view is available | Restart with backoff and investigate if it persists | +| Open batch enters the danger zone | The process exits with code `10`; startup replaces the open batch without flushing | Restart and allow startup recovery to finish | +| Closed batch enters the danger zone | The process exits with code `10`; startup flushes unresolved wallet nonces, synchronizes, and invalidates the affected suffix | Restart and allow additional recovery time | +| Local and base-layer batch content diverge | The process exits with code `30` | Stop automatic restarts and rebuild from a checkpoint | +| Fresh setup detects earlier submitter activity | Setup exits with code `40` | Run `setup --recovery` with a trusted checkpoint | +| Unclassified process, storage, or worker error | The process exits with code `1` | Restart with backoff, then investigate repeated failures | +| Local data is lost or cannot be trusted | Normal startup is not possible | Rebuild from a checkpoint | + +See [Exit codes and supervisor policy](../operations/orchestration.md#exit-codes-and-required-actions) for the complete process-control contract. + +## Base-layer outages + +The input reader and batch submitter retry temporary RPC failures, and the API may continue while the saved base-layer view remains usable. If that view becomes too old to trust, the process stops. Startup refuses with code `20` until it can obtain enough direct evidence to choose a safe path. + +See [Staleness and the danger zone](../concepts/staleness.md) for the timing rule and [Preemptive recovery](./preemptive.md#startup-recovery-decision) for the startup decision. + +## Process crashes and abrupt shutdowns + +An ordinary crash does not require checkpoint recovery. Committed database state and referenced snapshots are recovered during startup, while interrupted housekeeping is repaired before workers begin. Restart with backoff and investigate repeated worker or storage failures. See [Durability and crash guarantees](../operations/data-and-state.md#durability-and-crash-guarantees). + +## Extended downtime + +During a long shutdown, provisional batches continue aging. Startup synchronizes before accepting work, then selects ordinary startup, open-batch replacement, flush-and-cascade recovery, or refusal. See [Startup recovery decision](./preemptive.md#startup-recovery-decision). + +## Dropped, delayed, and resurfacing transactions + +A transaction that disappears from one provider can survive elsewhere and appear later. Recovery therefore resolves uncertain submitter nonce slots before abandoning submitted batches. See [Flush unresolved wallet nonces](../operations/orchestration.md#flush-unresolved-wallet-nonces). + +## Batches that arrive too late + +A stale batch and the provisional suffix after it do not take effect. Preemptive recovery resumes from the accepted frontier, while affected clients must reconcile their provisional results. See [Staleness and the danger zone](../concepts/staleness.md) and [Preemptive recovery](./preemptive.md). + +## Canonical divergence + +Canonical divergence is terminal because accepted base-layer content differs from the local batch stored for the same position. Stop automatic restarts, preserve evidence, and follow [Divergence detection and response](../advanced/divergence.md). + +## Loss or corruption of local state + +If local state is lost or untrusted, normal startup and preemptive recovery cannot establish a safe frontier. Use the restore boundaries in [Data, snapshots, and backups](../operations/data-and-state.md#restore-boundaries), then follow [Cockroach recovery](./cockroach.md) when reconstruction is required. + +## Direct inputs during an outage + +Users can still submit deposits and other direct inputs while the sequencer is offline because those inputs bypass the sequencer. + +The scheduler checks the direct-input delay rule when another input is processed. If no new input arrives, a queued direct input can remain pending beyond its normal delay. Any later application input advances processing and can release the earlier one. A user who needs progress can submit another direct input without waiting for the sequencer to return. + +See [Direct inputs vs sequenced transactions](../concepts/direct-vs-sequenced.md). + +## Failures outside automatic recovery + +Automatic recovery does not correct every source of failure: + +- **Application or sequencer defects.** Detected invariant violations stop the process because signing more batches could expand the incident. +- **Nondeterministic application behavior.** A state mismatch requires investigation and checkpoint recovery after the cause is fixed. +- **Compromised submitter keys.** Rotate or replace the deployment according to the incident plan. Recovery cannot make a stolen key trustworthy. +- **Traffic floods and API abuse.** Rate limits, request filtering, and denial-of-service protection belong at the external gateway. +- **Storage hardware failures.** Database transactions cannot compensate for a device that acknowledges writes and later loses them. + +## Next steps + +- Learn how automatic batch repair works in [Preemptive recovery](./preemptive.md). +- Prepare checkpoint-based reconstruction with [Snapshots and checkpoints](./snapshots.md). +- Configure restart behavior in [Process supervision and recovery operations](../operations/orchestration.md). diff --git a/app-sequencer/recovery/preemptive.md b/app-sequencer/recovery/preemptive.md new file mode 100644 index 000000000..557e78751 --- /dev/null +++ b/app-sequencer/recovery/preemptive.md @@ -0,0 +1,129 @@ +--- +title: "Preemptive recovery" +sidebar_label: "Preemptive recovery" +description: "How the sequencer detects batches approaching the staleness limit, selects a safe startup action, and resumes from the accepted frontier." +--- + +Preemptive recovery protects the sequencer from batches that may reach the base layer too late. It starts before the scheduler's staleness limit, while the sequencer can still determine which work remains usable. + +This is the routine recovery path for an intact local database. It does not rebuild a lost or untrusted deployment. + +## Why recovery begins before the deadline + +The sequencer acts before an unsettled batch reaches the protocol's staleness deadline. This preserves time to stop cleanly, resolve uncertain submissions, and construct a replacement from the accepted frontier. + +The configured threshold is: + +```text +danger threshold = maximum wait blocks - preemptive margin +``` + +[Staleness and the danger zone](../concepts/staleness.md) defines the deadline and batch-number cascade. [Configure protocol timing](../operations/setup-and-running.md#configure-protocol-timing) lists the defaults and validation rules. + +## Runtime detection and controlled restart + +A dedicated danger detector checks the local database every two seconds. It reads the most recent safe-head observation and the valid batch path. It does not modify the database or contact the base layer. + +The detector checks for: + +- accepted base-layer content that differs from the matching local batch; +- a base-layer view that has become too old to trust; +- a closed batch that has crossed the danger threshold; +- an open batch that has crossed the danger threshold; +- a batch whose estimated age has crossed the threshold while the observed safe head is stalled. + +When any condition is detected, the runtime stops all workers and exits. The exit code tells the supervisor whether to restart for recovery, retry after a transient refusal, or stop for operator action. + +Recovery does not mutate the batch tree inside the running process. A fresh process performs the decision before the API, input lane, and batch submitter start, which ensures that no other writer is active during recovery. + +## Startup recovery decision + +Startup first tries to synchronize the base-layer safe head. It then runs the same danger check against the updated or previously persisted view. + +| Startup result | Condition | Action | +| -------------------------- | ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | +| **Proceed** | No danger is present | Start normally without recovery writes | +| **Recover the open batch** | Only the open batch is in danger | Invalidate it and open a fresh batch without flushing | +| **Flush and cascade** | A closed batch beyond the accepted frontier is in danger | Flush unresolved wallet nonces, synchronize again, invalidate the provisional suffix, and open a fresh batch | +| **Refuse** | The base-layer view is stale, danger exists only in the wall-clock estimate, or canonical content diverged | Exit without modifying the batch tree | + +![At startup, the sequencer synchronizes the safe head and checks for danger before workers start. It can proceed normally, replace an aging open batch, flush unresolved nonces and cascade a closed suffix, or refuse recovery when the view is unsafe or divergent.](../images/preemptive-recovery.jpg) + +The refusal cases have different operational outcomes: + +- a stale view or estimated-only danger exits with code `20`; retry after the provider can supply a fresh safe view; +- canonical divergence exits with code `30`; stop automatic restarts and rebuild from a checkpoint. + +## Recovering an aging open batch + +An open batch has not been submitted and has no submitter wallet nonce. There is no hidden base-layer transaction whose outcome must be resolved. + +Startup therefore performs one database transaction that: + +1. confirms that the open batch still crosses the danger threshold; +2. invalidates that batch; +3. clears any pending snapshot references in the invalidated range, although an open batch normally has none; +4. opens a new batch from the last valid parent. + +The replacement reuses the batch nonce expected by the scheduler. No mempool flush is required. + +## Recovering closed batches + +A closed batch may already exist in one or more transaction pools, even when the local provider no longer reports it. Recovery must resolve those transactions before deciding which branch to keep. + +### 1. Flush unresolved wallet nonces + +Recovery invokes the shared wallet-flush mechanism and waits for every covered submitter nonce slot to have an outcome at the base-layer safe level. [Flush unresolved wallet nonces](../operations/orchestration.md#flush-unresolved-wallet-nonces) defines the watermark, replacement transactions, and completion conditions. + +### 2. Synchronize the accepted frontier + +The flush returns the safe block where nonce resolution was observed. The input reader synchronizes again, and recovery refuses to continue if the resulting view is behind that block. + +This second synchronization determines which batches the scheduler accepted after all relevant transaction outcomes were settled. + +### 3. Invalidate the provisional suffix + +Everything after the last accepted batch is invalidated in one database transaction. This includes batches that became stale, batches rejected after a stale predecessor, and batches displaced by flush transactions. + +Pending snapshots associated with the invalidated branch are cleared so startup cannot load application state from abandoned work. + +### 4. Open the replacement batch + +The sequencer opens a new batch from the accepted frontier. It uses the next batch nonce expected by the scheduler and includes any safe direct inputs that have not yet been drained. + +The machine sees an ordinary next batch. The abandoned local branch remains recorded as invalid history but cannot affect the valid path. + +## User-visible effects + +Transactions contained only in invalidated batches do not take effect. Any soft confirmations issued for them must be treated as revoked. + +Direct inputs are not lost. They remain part of the base-layer input stream and are drained into the replacement path if they were not already included in accepted state. + +Clients should reconcile their local state with the sequenced feed after reconnecting. See [Limits of a soft confirmation](../concepts/soft-confirmations.md#limits-of-a-soft-confirmation) and [Reading the sequenced feed](../usage/reading-the-feed.md). + +## Distinguishing recovery from failure + +Recoverable danger, a temporarily unusable base-layer view, and canonical divergence produce different exit codes and restart policies. [Process supervision and recovery operations](../operations/orchestration.md#exit-codes-and-required-actions) defines the required supervisor behavior. + +## Recovery boundaries + +Preemptive recovery depends on a trustworthy local database. It repairs the provisional end of the batch tree, but it cannot recover from: + +- a lost or irreparably corrupted data directory; +- canonical divergence; +- a compromised checkpoint or application implementation; +- an incorrect deployment identity. + +Use [Cockroach recovery](./cockroach.md) when the local record cannot be used. + +## Verification coverage + +The recovery model is checked with TLA+ for batch acceptance, staleness, nonce resolution, suffix invalidation, and replacement branching. The implementation also includes unit and end-to-end tests for open-batch recovery, closed-batch recovery, provider outages, delayed transactions, nonce-zero recovery, snapshot cleanup, and repeated recovery rounds. + +See [Formal verification](../advanced/formal-verification.md) for the scope and limitations of those guarantees. + +## Next steps + +- Review operator responses in [Failure modes](./failure-modes.md). +- Prepare the rebuild path in [Cockroach recovery](./cockroach.md). +- Configure timing values in [Configure, set up, and run the sequencer](../operations/setup-and-running.md#configure-protocol-timing). diff --git a/app-sequencer/recovery/snapshots.md b/app-sequencer/recovery/snapshots.md new file mode 100644 index 000000000..e8c8beb82 --- /dev/null +++ b/app-sequencer/recovery/snapshots.md @@ -0,0 +1,134 @@ +--- +title: "Snapshots and checkpoints" +sidebar_label: "Snapshots and checkpoints" +description: "How snapshots are created and promoted, what the snapshot endpoints return, and how to archive a complete recovery checkpoint." +--- + +A snapshot is a durable copy of application state at a known position in the sequenced transaction stream. Snapshots support three distinct tasks: + +- restarting the inclusion lane without replaying the entire local history; +- initializing an indexer before it follows the live feed; +- rebuilding a deployment from an archived recovery checkpoint. + +The same snapshot data participates in each task, but the required lifecycle state and metadata are different. + +## Snapshot lifecycle + +The sequencer maintains pending and finalized snapshot references in SQLite while storing application dumps on the filesystem. + +| Lifecycle state | Created when | What it represents | Appropriate use | +| --------------- | ------------------------------------------------------------------------ | -------------------------------------------------- | -------------------------------------------------------------------------------------- | +| **Pending** | A batch closes | Application state after provisional sequenced work | Local catch-up and the latest-state endpoint | +| **Finalized** | The matching batch is observed accepted through the base-layer safe view | The newest promoted application state | Watchdog comparison, settled indexer initialization, and non-genesis recovery archives | + +`Finalized` is the implementation's name for the promoted lifecycle state. It means the batch was accepted in the base-layer safe view used by the sequencer. It does not claim stronger irreversibility than that base-layer observation. + +Plain `setup` also creates a genesis snapshot directly in the finalized lifecycle state so every normal startup has an application state to load. + +![The snapshot lifecycle begins when a batch closes and the sequencer creates and synchronizes a dump. The dump becomes a pending snapshot, is promoted after the batch is accepted in the safe view, and can then supply application bytes through HTTP or a complete recovery checkpoint containing the state subtree and info.toml.](../images/snapshot-lifecycle.jpg) + +## How snapshots are created and promoted + +When the inclusion lane closes a batch, it performs the following sequence: + +1. creates a new dump directory; +2. writes `info.toml` and the application's state under `state`; +3. synchronizes the dump to storage; +4. seals the batch and records the pending snapshot in one database transaction. + +This ordering prevents a committed database row from referring to an incomplete dump through the normal creation path. A failed database transaction may leave an unreferenced directory, which startup cleanup removes. + +As the safe input frontier advances, the sequencer observes accepted batch submissions. It promotes the newest applicable pending snapshot and advances the direct-input drain in the same database transaction. The atomic update prevents a crash from promoting a snapshot without advancing the input position that justified it. + +## How startup selects a snapshot + +The inclusion lane loads the newest pending snapshot when one exists. Otherwise, it loads the finalized snapshot. The same database record supplies both the dump path and the sequenced-feed offset, preventing application state and replay position from being mixed. + +Loading a pending snapshot is safe for normal restart because preemptive recovery runs first. If startup invalidates a provisional branch, it clears pending snapshots associated with that branch before the inclusion lane starts. + +This internal restart behavior does not make pending snapshots suitable for checkpoint recovery. A rebuild needs a promoted checkpoint whose batch nonce is grounded in the accepted base-layer history. + +## Snapshot directory format + +Each dump is a directory with two top-level entries: + +```text +dumps// + state/ + + info.toml +``` + +The `state` entry is an application-owned subtree. An application may represent it as a file or as a directory containing several files. Operators must treat it as opaque and archive the complete subtree. + +The sequencer owns `info.toml`. It contains: + +| Field | Purpose | +| -------------------------- | ------------------------------------------------------------------------- | +| `format_version` | Identifies the supported checkpoint metadata format | +| `next_batch_nonce` | Batch nonce from which recovery should resume before replay | +| `l2_tx_index` | Sequenced-feed position represented by the snapshot | +| `promoted_inclusion_block` | Base-layer block where the snapshot entered the finalized lifecycle state | + +The promotion block is absent while a snapshot is pending. It is stamped into `info.toml` when the snapshot is promoted and restored from the authoritative database row at startup if a crash interrupted that filesystem update. + +## Snapshot HTTP endpoints + +The HTTP API can stream the latest promoted state for watchdogs, the latest available state for indexers, and a lightweight promoted-position cursor. Streaming responses lease their dump so garbage collection cannot remove it during transfer. + +[Operator endpoints](../api-reference/api.md#operator-endpoints) defines the exact routes, headers, and cache behavior. Keep these endpoints internal according to [Production security](../operations/security.md#separate-public-and-internal-routes). + +:::danger Snapshot responses are not recovery checkpoints +The streaming endpoints return only the application's state bytes. They do not return `info.toml` or package the complete dump directory. + +An HTTP snapshot can initialize a compatible reader, but it cannot supply the batch nonce and metadata required by `setup --recovery`. +::: + +## Why checkpoint recovery requires a finalized snapshot + +[Cockroach recovery](./cockroach.md) treats the checkpoint state and its recorded next batch nonce as trusted inputs. A pending snapshot describes an outcome the sequencer expected before the corresponding batch was accepted. That branch can still be invalidated by preemptive recovery. + +A promoted checkpoint binds its state to a batch observed accepted through the base-layer safe view and records the promotion block needed to replay the later interval. Use only a complete finalized dump produced by the sequencer. Do not use a pending dump, an HTTP state response, or metadata assembled by hand. + +## Archive a complete recovery checkpoint + +The sequencer garbage-collects dumps that are no longer referenced. A production deployment therefore needs a separate checkpoint archive. + +Use this coordinated procedure: + +1. query `GET /finalized_state/inclusion_block` and record the returned inclusion block; +2. stop the sequencer, or use another mechanism that prevents snapshot promotion and garbage collection during the copy; +3. locate the referenced dump whose `info.toml` contains the matching `promoted_inclusion_block`; +4. copy the complete dump directory, including all contents under `state` and the original `info.toml`; +5. parse `info.toml` and confirm its promotion block matches the value observed before the copy; +6. record the chain ID, application address, batch-submitter address, sequencer release, application release, archive time, and checkpoint block; +7. store the archive on independent, access-controlled storage and verify its integrity. + +Keep several checkpoint generations. An incident may affect the newest snapshot, so the recovery plan should allow selection of an older known-good checkpoint. + +See [Back up the live deployment](../operations/data-and-state.md#back-up-the-live-deployment) for complete data-directory backup guidance. + +## Validate checkpoint archives + +A successful copy is not enough to prove that an archive can restore a deployment. Regularly test that: + +- `info.toml` uses a format supported by the recovery binary; +- the application can load the archived state subtree; +- `next_batch_nonce`, `l2_tx_index`, and `promoted_inclusion_block` are present and plausible; +- the recorded deployment identity matches the intended environment; +- `setup --recovery` can rebuild a non-production data directory from the archive; +- the watchdog confirms the rebuilt state after replay. + +Protect checkpoints from unauthorized modification. Recovery does not include an independent historical proof of the state or next batch nonce, so a corrupted but loadable checkpoint can produce an incorrect rebuild. + +## Snapshot retention and cleanup + +The sequencer retains dumps referenced by pending or finalized snapshot rows and protects active HTTP streams with leases. Superseded, unreferenced, and unleased dumps are eligible for garbage collection after promotion and during startup cleanup. + +This automatic cleanup controls local disk use but does not maintain a historical archive. Monitor the data volume, archive checkpoints before they are superseded, and keep recovery copies outside the live data directory. + +## Next steps + +- Rebuild a deployment with [Cockroach recovery](./cockroach.md). +- Secure the snapshot routes using [Production security](../operations/security.md#separate-public-and-internal-routes). +- Initialize an indexer with [Reading the sequenced feed](../usage/reading-the-feed.md). diff --git a/app-sequencer/troubleshooting.md b/app-sequencer/troubleshooting.md new file mode 100644 index 000000000..dbdb9f0e9 --- /dev/null +++ b/app-sequencer/troubleshooting.md @@ -0,0 +1,63 @@ +--- +title: "Troubleshooting" +sidebar_label: "Troubleshooting" +description: "Common questions and common errors, with what to do about them." +--- + +## Common questions + +**Is a soft confirmation final?** +No. Treat it as a provisional result. See [Soft confirmations](./concepts/soft-confirmations.md). + +**Can the sequencer steal funds?** +Sequencing does not give the service custody of application assets or authority over canonical state. See [Trust model and guarantees](./foundations/trust-model.md). + +**Can it censor a user?** +It can refuse the fast path. Users can still submit supported actions through the application's direct-input path. See [Direct and sequenced inputs](./concepts/direct-vs-sequenced.md). + +**Can the operator front-run users?** +Yes, in the sense that nothing in the protocol forces first come, first served, and the operator sees transactions before anyone else. An application where ordering carries value should treat the operator as trusted for fairness. + +**Do deposits still work if the sequencer is down?** +They can still reach the `InputBox` as direct inputs. Canonical execution may wait for another recorded input to trigger the scheduler. See [Direct inputs during an outage](./recovery/failure-modes.md#direct-inputs-during-an-outage). + +**How do I find out my next nonce?** +Track it in the client. There is no endpoint that reports it. The response to a successful submission echoes the nonce that was accepted. + +**Why did my transaction disappear after being accepted?** +Its batch was invalidated after a recovery. Fee and application checks happen at submission, so an accepted transaction was not rejected for those reasons later. + +## Errors when submitting + +**`429` with code `OVERLOADED`.** The queue is full. Retry with bounded backoff. + +**`400` with an invalid signature.** Usually the EIP-712 domain does not match. Check the chain id, and that `verifyingContract` is the application's address. Also check that `sender` matches the key that signed. + +**`413`.** The complete request body exceeds the ingress limit. Reduce it before retrying. + +**`422` with code `EXECUTION_REJECTED`.** Correct the maximum fee, nonce, or application condition, then sign again. See [Submitting transactions](./usage/submitting-operations.md#handle-submission-errors). + +## Errors on the feed + +**Closed immediately with code `1008`, `catch-up window exceeded`.** Initialize from a snapshot and subscribe from its offset. See [Recover after a long absence](./usage/reading-the-feed.md#recover-after-a-long-absence). + +**Cannot connect because the server reports overload.** The subscriber limit has been reached. Use a small number of durable indexers instead of connecting every client directly. + +**A transaction never appears.** Reconcile its status because its provisional history may have been invalidated. See [Soft confirmations](./concepts/soft-confirmations.md). + +## Errors when running the sequencer + +**`remote RPC must use https`.** The endpoint is plain HTTP to a non-loopback host. Use `https://`, or set `CARTESI_SEQUENCER_ALLOW_INSECURE_RPC=true` on **every** command that dials the base layer. + +**Refuses to start, setup not complete.** It is looking at an unprepared directory. Check `CARTESI_SEQUENCER_DATA_DIR`, remembering the default is a relative path. + +**Exit code 30.** Terminal. Do not restart in a loop. See [Constants and exit codes](./api-reference/constants-and-exit-codes.md). + +**Restarts repeatedly.** Usually a base-layer connection that is unreachable or lagging, so the sequencer keeps stepping back from the deadline. Fix the connection. + +**Batches stop being posted.** Check the submitter account has funds. + +## Next steps + +- For what can fail and how it is handled, see [Failure modes](./recovery/failure-modes.md). +- For terms, see the [App Sequencer glossary](./foundations/glossary.md). diff --git a/app-sequencer/tutorials/build-wallet-sequencer.md b/app-sequencer/tutorials/build-wallet-sequencer.md new file mode 100644 index 000000000..1dd90fd34 --- /dev/null +++ b/app-sequencer/tutorials/build-wallet-sequencer.md @@ -0,0 +1,459 @@ +--- +title: "Build an ERC-20 wallet with the App Sequencer" +sidebar_label: "Build a sequenced wallet" +description: "Build a Cartesi ERC-20 wallet from source, connect it to the App Sequencer, run it locally, and submit signed operations." +--- + +import WorkspaceCargo from './snippets/_wallet-workspace-cargo.md'; +import CoreCargo from './snippets/_wallet-core-cargo.md'; +import CoreLib from './snippets/_wallet-core-lib.md'; +import WalletMethod from './snippets/_wallet-method.md'; +import WalletApplication from './snippets/_wallet-application.md'; +import WalletSequencer from './snippets/_wallet-sequencer.md'; +import WalletCanonical from './snippets/_wallet-canonical.md'; +import WalletMachine from './snippets/_wallet-machine.md'; +import WalletClient from './snippets/_wallet-client.md'; + +In this tutorial, you will build a Cartesi wallet and its application-specific sequencer from an empty directory. You will write the shared application logic, create the host sequencer and canonical Cartesi Machine programs, package the machine, and submit a deposit, transfer, and withdrawal on a local network. + +The tutorial uses the sequencer [`v0.1.0-alpha.9` release](https://github.com/cartesi/sequencer/releases/tag/v0.1.0-alpha.9). + +By the end, the project will support two input paths: + +- ERC-20 deposits enter through the base layer and reach the wallet as direct inputs. +- Transfers and withdrawals are signed by users and sent to the sequencer for fast ordering. + + +## Understand the project you will build + +The application is divided into three Rust crates and one client: + +| Component | Runs in | Responsibility | +| --- | --- | --- | +| `app-core` | Both execution paths | Implements wallet state, deposits, transfers, withdrawals, fees, nonces, and snapshots | +| `app-sequencer` | Host system | Accepts signed user operations, predicts execution, forms batches, and submits them to the base layer | +| `canonical-app` | Cartesi Machine | Applies direct inputs and sequencer batches in the canonical order | +| `client` | User system | Encodes and signs wallet operations and reads the ordered feed | + +Keeping the application logic in `app-core` ensures that the host sequencer and Cartesi Machine execute the same rules. This follows the structure recommended in [Application integration](../usage/integration.md#recommended-project-structure). + +## Prerequisites + +Install these tools before continuing: + +- Cartesi CLI `2.0.0-alpha.35`; +- Docker with the Buildx plugin; +- Rust `1.95.0` or later, and Cargo; +- [`cross`](https://github.com/cross-rs/cross) for the RISC-V build, installed with `cargo install cross --git https://github.com/cross-rs/cross`; +- Foundry, for the `cast` command used to mint test tokens; +- Node.js 20 or later and npm; +- `jq` and `curl`. + +Check the main commands: + +```bash +cartesi --version +docker buildx version +rustc --version +cargo --version +cross --version +cast --version +node --version +npm --version +jq --version +``` + +Outside a Cargo project, `cross --version` prints two warning lines about missing package metadata and falling back to the host cargo. That is expected and does not mean the install failed. + +Docker must be running before you build the canonical application or start the local Cartesi environment. + +## Step 1: create the project structure + +Create an empty project and the directories for each component: + +```bash +mkdir wallet-sequencer-tutorial +cd wallet-sequencer-tutorial +export PROJECT_ROOT=$PWD + +mkdir -p app-core/src +mkdir -p app-sequencer/src +mkdir -p canonical-app/src +mkdir -p client +mkdir -p machine/out +``` + +The final repository structure would look like the below directory structure so ensure to follow the subsequent steps correctly: + +```text +wallet-sequencer-tutorial/ +├── app-core/ +│ ├── src/ +│ │ ├── lib.rs +│ │ ├── method.rs +│ │ └── wallet.rs +│ └── Cargo.toml +├── app-sequencer/ +│ ├── src/main.rs +│ └── Cargo.toml +├── canonical-app/ +│ ├── src/main.rs +│ └── Cargo.toml +├── client/ +│ ├── feed.mjs +│ ├── package.json +│ └── wallet-client.mjs +├── machine/ +│ ├── out/ +│ └── Dockerfile +├── Cargo.toml +├── Cross.toml +└── cartesi.toml +``` + +Later steps use `$PROJECT_ROOT` to return here from other terminals. Export it again in each new terminal you open. + +The `machine/out` directory will receive the compiled RISC-V binary. The `sequencer-data` directory will be created later when you initialize the sequencer. + +## Step 2: configure the Rust workspace + +Create the root `Cargo.toml` and copy this configuration into it: + + + +The three `sequencer` dependencies point to the same Git tag. Keeping them on one release prevents the runtime, shared protocol types, and canonical scheduler from drifting apart. + +The `types` and `trolley` revisions match the versions selected by that sequencer release. They provide the portal payload types and Cartesi Machine I/O used by this application. + +## Step 3: define the wallet operations + +Create `app-core/Cargo.toml`: + + + +Create `app-core/src/lib.rs` to expose the types needed by the two executables: + + + +Create `app-core/src/method.rs`: + + + +The `Method` enum uses Simple Serialize, or SSZ. The union selector is `0` for `Withdrawal` and `1` for `Transfer`. Both operations carry a 256-bit amount, and a transfer also carries a 20-byte recipient address. + +The maximum method size is therefore one selector byte, 32 amount bytes, and 20 address bytes. + +## Step 4: implement the shared wallet state + +Create `app-core/src/wallet.rs`: + + + +This file implements the complete `Application` contract required by the sequencer: + +- `validate_user_op` checks the sender's next nonce and ability to pay the frame fee. +- `execute_valid_user_op` charges the fee, advances the nonce, and applies a transfer or withdrawal. +- `execute_direct_input` recognizes ERC-20 portal deposits and credits the depositor. +- the progress methods track the last safe block and number of executed inputs. +- the dump methods save and restore crash-recovery state. +- `canonical_snapshot_bytes` sorts addresses before serialization so the same logical state always produces the same bytes. + +The fixed addresses come from the local environment created by the Cartesi CLI. The batch submitter is Anvil account 9. Its address is used both to classify canonical batches and to receive this tutorial wallet's fees. + +:::caution Method execution in this example +The application validates the nonce and fee balance before inclusion. It still charges the fee and advances the nonce if a method is malformed or if the remaining balance cannot cover the requested transfer or withdrawal. Such an operation produces no method output. Production applications should define and test their failure policy explicitly. +::: + +## Step 5: create the host sequencer + +Create `app-sequencer/Cargo.toml` and `app-sequencer/src/main.rs`: + + + +The executable is small because the released `sequencer` crate supplies command parsing, setup, storage, the HTTP and WebSocket API, batching, submission, and recovery. The closure passed to `run_main` creates the application's genesis state during `setup`. + +When compiled, this program provides three commands: + +```text +app-sequencer setup +app-sequencer run +app-sequencer flush-mempool +``` + +This tutorial uses `setup` and `run`. + +## Step 6: create the canonical application + +Create `canonical-app/Cargo.toml` and `canonical-app/src/main.rs`: + + + +`run_scheduler_forever` reads inputs inside the Cartesi Machine. Inputs sent by `BATCH_SUBMITTER_ADDRESS` are decoded as sequencer batches. Other inputs, including portal deposits, enter the direct-input queue. + +The same `WalletApp` type runs here and in the host sequencer. The shared type is the main protection against different execution rules on the two paths. + +## Step 7: configure the Cartesi Machine build + +Create `Cross.toml`, `cartesi.toml`, and `machine/Dockerfile`: + + + +The `cross` image and runtime image are pinned to the same versions used by the sequencer release. `cross` compiles `wallet-canonical` for RISC-V. The Dockerfile then places that binary and `cartesi-init` in the root file system that the Cartesi CLI turns into a Cartesi Machine. + +## Step 8: create the client and feed subscriber + +Create `client/package.json`, `client/wallet-client.mjs`, and `client/feed.mjs`: + + + +Install the JavaScript dependencies: + +```bash +cd client +npm install +cd .. +``` + +The wallet client performs four tasks: + +1. encodes the selected wallet method with SSZ; +2. builds a `UserOp` containing the sender's nonce, maximum fee, and method bytes; +3. signs the operation with the sequencer's EIP-712 domain; +4. sends the signed request to `POST /tx`. + +The feed subscriber connects to `/ws/subscribe` with offset `0`. It first receives stored events whose offsets are greater than zero, then continues with new events. + +## Step 9: build every component + +Check the host crates first: + +```bash +cargo check -p app-core -p app-sequencer +cargo build --release -p app-sequencer +``` + +Compile the canonical program for the Cartesi Machine: + +```bash +SOURCE_DATE_EPOCH=0 \ +CARGO_PROFILE_RELEASE_STRIP=symbols \ +cross build \ + --package wallet-canonical \ + --target riscv64gc-unknown-linux-musl \ + --release + +cp \ + target/riscv64gc-unknown-linux-musl/release/wallet-canonical \ + machine/out/dapp +``` + +Package the binary as a Cartesi Machine: + +```bash +cartesi build +``` + +The first build downloads the Rust dependencies, cross-compilation image, Cartesi SDK, guest tools, and RISC-V runtime image. Later builds reuse Docker and Cargo caches. + +## Step 10: start the local Cartesi environment + +Run the Cartesi application from the project root: + +```bash +cartesi run --block-time 1 --default-block safe +``` + +Keep this terminal open. When startup finishes, the CLI prints the local application URL, Anvil RPC URL, machine hash, and deployed application address. + +The commands below assume the default port and project name. Open a second terminal and return to the project root: + +```bash +export PROJECT_ROOT=/path/to/wallet-sequencer-tutorial +cd "$PROJECT_ROOT" + +export L1_RPC=http://127.0.0.1:6751/anvil +export APP_ADDRESS=$(cartesi address-book --json | jq -r .Application) +export SEQUENCER_URL=http://127.0.0.1:3000 +export CHAIN_ID=31337 +export MAX_FEE=2000 + +echo "$APP_ADDRESS" +``` + +Every later terminal reuses `SEQUENCER_URL`, so if you change the sequencer's port in Step 12, change it here and export the same value everywhere. + +If you passed a custom port or project name to `cartesi run`, pass the matching values to `cartesi address-book` and update `L1_RPC`. + +## Step 11: initialize the sequencer + +The sequencer stores deployment identity, application snapshots, batches, and feed offsets in its data directory. Initialize that directory once: + +```bash +CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT="$L1_RPC" \ +CARTESI_SEQUENCER_BLOCKCHAIN_ID="$CHAIN_ID" \ +CARTESI_SEQUENCER_APP_ADDRESS="$APP_ADDRESS" \ +CARTESI_SEQUENCER_BATCH_SUBMITTER_ADDRESS=0xa0Ee7A142d267C1f36714E4a8F75612F20a79720 \ +CARTESI_SEQUENCER_DATA_DIR=./sequencer-data \ +CARTESI_SEQUENCER_FEE_ORACLE_FIXED_LOG_GAS_PRICE=0 \ +CARTESI_SEQUENCER_SECONDS_PER_BLOCK=1 \ + ./target/release/app-sequencer setup +``` + +The fixed fee oracle is required because chain ID `31337` has no public-network fee oracle preset. Setup fails on an unknown chain without it. + +A fee is an exponent, not a token amount, so `0` here does not mean transactions are free. The sequencer adds fixed per-operation terms to the configured gas price, which gives every frame in this tutorial a price of `1356`. Decoded, that is `38276` of the wallet's smallest units, and it is what each operation is charged. [Fees and data availability](../concepts/fees.md) explains the encoding. + +Plain `setup` uses the submitter address but does not require its private key or send a transaction. + +## Step 12: start the sequencer + +Run the sequencer with the private key for Anvil account 9: + +```bash +CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT="$L1_RPC" \ +CARTESI_SEQUENCER_AUTH_PRIVATE_KEY=0x2a871d0798f97d79848a013d4936a73bf4cc922c825d33c1cf7073dff6d409c6 \ +CARTESI_SEQUENCER_DATA_DIR=./sequencer-data \ +CARTESI_SEQUENCER_SECONDS_PER_BLOCK=1 \ +CARTESI_SEQUENCER_MAX_BATCH_OPEN_SECONDS=5 \ +CARTESI_SEQUENCER_BATCH_SUBMITTER_IDLE_POLL_INTERVAL_MS=500 \ +CARTESI_SEQUENCER_BATCH_SUBMITTER_CONFIRMATION_DEPTH=0 \ + ./target/release/app-sequencer run +``` + +:::warning Development key +This is a public Anvil test key. Never fund it or use it on a public network. Use a protected key file for an operated deployment. +::: + +The shorter batch interval keeps the local exercise moving. Wait until the sequencer reports that it is ready, then verify it from another terminal: + +```bash +curl --fail "$SEQUENCER_URL/readyz" +``` + +The sequencer listens on `127.0.0.1:3000` by default. If that port is already in use, the process exits with `Address already in use`. Add `CARTESI_SEQUENCER_HTTP_ADDR=127.0.0.1:` to the command above and export a matching `SEQUENCER_URL` in every terminal. + +## Step 13: watch the ordered feed + +Open another terminal, enter the client directory, and start the subscriber: + +```bash +cd "$PROJECT_ROOT/client" +export SEQUENCER_URL=http://127.0.0.1:3000 +npm run feed +``` + +Leave this process running. It will print the direct input and the two signed operations used in the following steps. + +## Step 14: deposit test tokens + +Return to a terminal at the project root. The local test token is deployed with no supply, so mint some for Anvil account 0 first: + +```bash +cd "$PROJECT_ROOT" +export TEST_TOKEN=$(cartesi address-book --json | jq -r .TestToken) + +cast send "$TEST_TOKEN" "mint(uint256)" 1000000000000000000000 \ + --rpc-url "$L1_RPC" \ + --private-key 0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80 +``` + +The token's symbol is `FUN`, and `mint` credits the account that sends the transaction. Now deposit one token into the wallet for that account: + +```bash +cartesi deposit erc20 1 --token "$TEST_TOKEN" +``` + +The Cartesi CLI approves the local ERC-20 portal and submits the deposit for the current application. Without `--token`, the command prompts for the token address with the correct value already filled in. + +The environment runs with `--default-block safe`, so the sequencer sees the deposit only once the safe head passes the block that recorded it. Expect to wait up to a minute or two. + +The feed subscriber then prints a `direct_input` event. Its `sender` is the ERC-20 portal, and its payload contains the token address, account 0 address, amount, and execution-layer data. + +One token has 18 decimal places in this environment, so the wallet credits account 0 with `1000000000000000000` units. + +## Step 15: transfer tokens through the sequencer + +Use Anvil account 0 as Alice and account 1 as Bob: + +```bash +cd "$PROJECT_ROOT/client" + +export SEQUENCER_URL=http://127.0.0.1:3000 +export CHAIN_ID=31337 +export MAX_FEE=2000 +export APP_ADDRESS=$(cd .. && cartesi address-book --json | jq -r .Application) + +export ALICE_PRIVATE_KEY=0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80 +export BOB_ADDRESS=0x70997970C51812dc3A010C7d01b50e0d17dc79C8 +export WALLET_PRIVATE_KEY="$ALICE_PRIVATE_KEY" + +npm run wallet -- \ + transfer \ + "$BOB_ADDRESS" \ + 400000000000000000 \ + 0 +``` + +Alice's first nonce is `0`. The client transfers `0.4` token and prints a response similar to: + +```text +Application payload: 0x01... +Soft confirmation: { ok: true, sender: '0xf39F...', nonce: 0 } +``` + +The feed prints the operation as a `user_op`. Its `data` starts with `0x01`, the SSZ selector for a transfer. + +The successful response is a soft confirmation. It means the sequencer validated, executed, and stored the operation in its provisional order. It does not mean that the operation has settled on the base layer. + +`MAX_FEE` is the highest fee exponent the operation will accept. The sequencer rejects a submission with HTTP `422` and code `EXECUTION_REJECTED` when it falls below the frame's price, which is `1356` here, so `2000` leaves room. An accepted operation is charged the frame price, not its own maximum. + +## Step 16: request a withdrawal + +Bob received `0.4` token and has not sent an operation, so his next nonce is also `0`. Select Bob's key and request a withdrawal of `0.1` token: + +```bash +export BOB_PRIVATE_KEY=0x59c6995e998f97a5a0044966f0945389dc9e86dae88c7a8412f4603b6b78690d +export WALLET_PRIVATE_KEY="$BOB_PRIVATE_KEY" + +npm run wallet -- \ + withdraw \ + 100000000000000000 \ + 0 +``` + +The sequencer returns another soft confirmation. The feed shows a `user_op` whose data starts with `0x00`, the withdrawal selector. + +During application execution, the wallet deducts the frame fee and withdrawal amount from Bob's wallet balance. The canonical application emits an ERC-20 transfer voucher for the requested amount. Executing that voucher on the base layer is a separate settlement step and is outside this tutorial. + +## Step 17: inspect the sequencer state + +The five-second batch interval closes the open batch shortly after the operations are submitted. Fetch the most recent wallet snapshot: + +```bash +sleep 6 +curl --fail --silent "$SEQUENCER_URL/latest_snapshot" | jq +``` + +The snapshot contains the sorted balances, per-sender nonces, executed input count, and last executed safe block. Alice and Bob each have nonce `1`. Their balances also reflect the transfer, withdrawal, and fees charged by the wallet. + +`/latest_snapshot` is an operator endpoint. It is useful for this local inspection, but it has no authentication and must not be exposed publicly. + +## What you built + +The running system now demonstrates the complete integration path: + +1. The portal deposit entered through the base layer as a direct input. +2. Alice signed a transfer and received a soft confirmation from the sequencer. +3. Bob signed a withdrawal and received a soft confirmation. +4. The WebSocket feed delivered all three inputs in execution order. +5. The host sequencer formed and submitted batches while the Cartesi Machine applied the canonical scheduling rules. + +The shared `app-core` crate made both execution paths use the same wallet rules and snapshot format. The batch submitter address was also kept consistent in the canonical scheduler, sequencer setup, and runtime key. + +## Next steps + +- Read [Application requirements](../usage/application-requirements.md) before replacing the tutorial wallet with production application logic. +- Use [Application integration](../usage/integration.md) to adapt the three-crate structure to an existing project. +- Follow [Submitting operations](../usage/submitting-operations.md) for client retry, nonce, and fee handling. +- Follow [Reading the sequenced feed](../usage/reading-the-feed.md) to add durable cursor storage and reconnection. +- Review [Soft confirmations](../concepts/soft-confirmations.md) before presenting sequencer responses as application outcomes. diff --git a/app-sequencer/tutorials/snippets/_wallet-application.md b/app-sequencer/tutorials/snippets/_wallet-application.md new file mode 100644 index 000000000..dbd456666 --- /dev/null +++ b/app-sequencer/tutorials/snippets/_wallet-application.md @@ -0,0 +1,348 @@ +```rust title="app-core/src/wallet.rs" +use std::collections::HashMap; +use std::io::Write; +use std::path::{Path, PathBuf}; + +use alloy_primitives::{Address, U256, address}; +use alloy_sol_types::{SolCall, sol}; +use serde::{Deserialize, Serialize}; +use ssz::Decode; +use tracing::{error, warn}; +use types::{Erc20Deposit, Erc20Transfer}; + +use crate::method::{MAX_METHOD_PAYLOAD_BYTES, Method}; +use sequencer_core::application::{ + AppError, AppOutput, AppOutputs, Application, InvalidReason, +}; +use sequencer_core::l2_tx::{DirectInput, ValidUserOp}; +use sequencer_core::user_op::UserOp; + +pub const DEVNET_ERC20_PORTAL_ADDRESS: Address = + address!("0x22E57511C30CcE6CDaa742E13CE3b774fDC663b1"); +pub const DEVNET_TEST_TOKEN_ADDRESS: Address = + address!("0x88A2120B7068E78692C8fd12E751d610B6377E4d"); +pub const BATCH_SUBMITTER_ADDRESS: Address = + address!("0xa0Ee7A142d267C1f36714E4a8F75612F20a79720"); + +sol! { + function DepositNotice( + address token, + address sender, + uint256 amount + ) external; + function TransferNotice( + address sender, + address recipient, + uint256 amount + ) external; +} + +#[derive(Debug, Clone, Copy, Serialize, Deserialize)] +pub struct WalletConfig { + pub erc20_portal: Address, + pub token: Address, + pub fee_recipient: Address, +} + +impl WalletConfig { + pub const fn devnet() -> Self { + Self { + erc20_portal: DEVNET_ERC20_PORTAL_ADDRESS, + token: DEVNET_TEST_TOKEN_ADDRESS, + fee_recipient: BATCH_SUBMITTER_ADDRESS, + } + } +} + +#[derive(Debug, Clone)] +pub struct WalletApp { + config: WalletConfig, + balances: HashMap, + nonces: HashMap, + executed_input_count: u64, + last_executed_safe_block: u64, +} + +#[derive(Serialize, Deserialize)] +struct BalanceEntry { + address: Address, + balance: U256, +} + +#[derive(Serialize, Deserialize)] +struct NonceEntry { + address: Address, + nonce: u32, +} + +#[derive(Serialize, Deserialize)] +struct WalletSnapshot { + config: WalletConfig, + balances: Vec, + nonces: Vec, + executed_input_count: u64, + last_executed_safe_block: u64, +} + +impl WalletApp { + pub fn new(config: WalletConfig) -> Self { + Self { + config, + balances: HashMap::new(), + nonces: HashMap::new(), + executed_input_count: 0, + last_executed_safe_block: 0, + } + } + + fn balance_of(&self, address: Address) -> U256 { + self.balances + .get(&address) + .copied() + .unwrap_or(U256::ZERO) + } + + fn nonce_of(&self, address: Address) -> u32 { + self.nonces.get(&address).copied().unwrap_or(0) + } + + fn credit(&mut self, address: Address, amount: U256) { + self.balances + .insert(address, self.balance_of(address) + amount); + } + + fn debit(&mut self, address: Address, amount: U256) -> bool { + let balance = self.balance_of(address); + if balance < amount { + return false; + } + self.balances.insert(address, balance - amount); + true + } + + fn snapshot_bytes(&self) -> Result, AppError> { + let mut balances = self + .balances + .iter() + .map(|(address, balance)| BalanceEntry { + address: *address, + balance: *balance, + }) + .collect::>(); + balances.sort_unstable_by_key(|entry| entry.address.into_array()); + + let mut nonces = self + .nonces + .iter() + .map(|(address, nonce)| NonceEntry { + address: *address, + nonce: *nonce, + }) + .collect::>(); + nonces.sort_unstable_by_key(|entry| entry.address.into_array()); + + serde_json::to_vec(&WalletSnapshot { + config: self.config, + balances, + nonces, + executed_input_count: self.executed_input_count, + last_executed_safe_block: self.last_executed_safe_block, + }) + .map_err(|error| AppError::Internal { + reason: format!("snapshot encoding failed: {error}"), + }) + } + + fn from_snapshot(bytes: &[u8]) -> Result { + let snapshot: WalletSnapshot = serde_json::from_slice(bytes) + .map_err(|error| AppError::Internal { + reason: format!("snapshot decoding failed: {error}"), + })?; + + Ok(Self { + config: snapshot.config, + balances: snapshot + .balances + .into_iter() + .map(|entry| (entry.address, entry.balance)) + .collect(), + nonces: snapshot + .nonces + .into_iter() + .map(|entry| (entry.address, entry.nonce)) + .collect(), + executed_input_count: snapshot.executed_input_count, + last_executed_safe_block: snapshot.last_executed_safe_block, + }) + } +} + +impl Application for WalletApp { + const MAX_METHOD_PAYLOAD_BYTES: usize = MAX_METHOD_PAYLOAD_BYTES; + + fn validate_user_op( + &self, + sender: Address, + user_op: &UserOp, + current_fee: u16, + ) -> Result<(), InvalidReason> { + let expected = self.nonce_of(sender); + if user_op.nonce != expected { + return Err(InvalidReason::InvalidNonce { + expected, + got: user_op.nonce, + }); + } + + let required = sequencer_core::fee::fee_to_linear(current_fee); + let available = self.balance_of(sender); + if available < required { + return Err(InvalidReason::InsufficientFeeBalance { + required, + available, + }); + } + Ok(()) + } + + fn execute_valid_user_op( + &mut self, + user_op: &ValidUserOp, + safe_block: u64, + ) -> Result { + let sender = user_op.sender; + let fee = sequencer_core::fee::fee_to_linear(user_op.fee); + if !self.debit(sender, fee) { + return Err(AppError::Internal { + reason: "validated operation cannot pay its fee".to_string(), + }); + } + + self.credit(self.config.fee_recipient, fee); + self.nonces.insert(sender, self.nonce_of(sender) + 1); + + let mut outputs = Vec::new(); + match Method::from_ssz_bytes(&user_op.data) { + Ok(Method::Transfer(transfer)) + if self.debit(sender, transfer.amount) => + { + self.credit(transfer.to, transfer.amount); + outputs.push(AppOutput::Notice( + TransferNoticeCall { + sender, + recipient: transfer.to, + amount: transfer.amount, + } + .abi_encode(), + )); + } + Ok(Method::Withdrawal(withdrawal)) + if self.debit(sender, withdrawal.amount) => + { + outputs.push(AppOutput::Voucher { + destination: self.config.token, + value: U256::ZERO, + payload: Erc20Transfer { + recipient: sender, + amount: withdrawal.amount, + } + .abi_encode(), + }); + } + _ => {} + } + + self.executed_input_count = + self.executed_input_count.saturating_add(1); + self.last_executed_safe_block = + self.last_executed_safe_block.max(safe_block); + Ok(outputs) + } + + fn execute_direct_input( + &mut self, + input: &DirectInput, + ) -> Result { + let mut outputs = Vec::new(); + if input.sender == self.config.erc20_portal { + match Erc20Deposit::decode(&input.payload) { + Ok(deposit) if deposit.token == self.config.token => { + self.credit(deposit.sender, deposit.value); + outputs.push(AppOutput::Notice( + DepositNoticeCall { + token: deposit.token, + sender: deposit.sender, + amount: deposit.value, + } + .abi_encode(), + )); + } + Ok(deposit) => { + warn!( + token = %deposit.token, + "ignoring unsupported token" + ); + } + Err(error) => { + error!(%error, "ignoring malformed portal deposit"); + } + } + } + + self.executed_input_count = + self.executed_input_count.saturating_add(1); + self.last_executed_safe_block = self + .last_executed_safe_block + .max(input.block_number); + Ok(outputs) + } + + fn last_executed_safe_block(&self) -> u64 { + self.last_executed_safe_block + } + + fn executed_input_count(&self) -> u64 { + self.executed_input_count + } + + fn from_dump(prefix: &Path) -> Result { + Self::from_snapshot(&std::fs::read( + Self::state_file_in_dump(prefix), + )?) + } + + fn create_dump(&self, prefix: &Path) -> Result<(), AppError> { + std::fs::create_dir(prefix)?; + let mut state = + std::fs::File::create(Self::state_file_in_dump(prefix))?; + state.write_all(&self.snapshot_bytes()?)?; + state.sync_all()?; + std::fs::File::open(prefix)?.sync_all()?; + if let Some(parent) = prefix.parent() { + std::fs::File::open(parent)?.sync_all()?; + } + Ok(()) + } + + fn delete_dump(prefix: &Path) -> Result<(), AppError> { + std::fs::remove_dir_all(prefix)?; + Ok(()) + } + + fn state_file_in_dump(prefix: &Path) -> PathBuf { + prefix.join("state.json") + } + + fn canonical_snapshot_bytes(&self) -> Result, AppError> { + self.snapshot_bytes() + } + + fn export_state(&self) -> Result { + String::from_utf8(self.snapshot_bytes()?).map_err(|error| { + AppError::Internal { + reason: format!("state export is not UTF-8: {error}"), + } + }) + } +} +``` diff --git a/app-sequencer/tutorials/snippets/_wallet-canonical.md b/app-sequencer/tutorials/snippets/_wallet-canonical.md new file mode 100644 index 000000000..8264c9e12 --- /dev/null +++ b/app-sequencer/tutorials/snippets/_wallet-canonical.md @@ -0,0 +1,33 @@ +```toml title="canonical-app/Cargo.toml" +[package] +name = "wallet-canonical" +version.workspace = true +edition.workspace = true +license.workspace = true + +[dependencies] +app-core.workspace = true +sequencer-canonical.workspace = true +trolley.workspace = true +``` + +```rust title="canonical-app/src/main.rs" +use app_core::{ + BATCH_SUBMITTER_ADDRESS, WalletApp, WalletConfig, +}; +use sequencer_canonical::{ + SchedulerConfig, run_scheduler_forever, +}; +use trolley::cmt::RollupCmt; + +fn main() { + let rollup = RollupCmt::try_new() + .expect("failed to initialize rollup I/O"); + + run_scheduler_forever( + rollup, + WalletApp::new(WalletConfig::devnet()), + SchedulerConfig::new(BATCH_SUBMITTER_ADDRESS), + ); +} +``` diff --git a/app-sequencer/tutorials/snippets/_wallet-client.md b/app-sequencer/tutorials/snippets/_wallet-client.md new file mode 100644 index 000000000..788a827ec --- /dev/null +++ b/app-sequencer/tutorials/snippets/_wallet-client.md @@ -0,0 +1,155 @@ +```json title="client/package.json" +{ + "name": "wallet-sequencer-client", + "private": true, + "type": "module", + "scripts": { + "wallet": "node wallet-client.mjs", + "feed": "node feed.mjs" + }, + "dependencies": { + "ethers": "^6.15.0", + "ws": "^8.18.0" + } +} +``` + +```js title="client/wallet-client.mjs" +import { + Wallet, + concat, + getAddress, + getBytes, +} from "ethers"; + +function required(name) { + const value = process.env[name]; + if (!value) throw new Error(`${name} is required`); + return value; +} + +function uint256LittleEndian(value) { + const bigEndian = value.toString(16).padStart(64, "0"); + return Uint8Array.from(Buffer.from(bigEndian, "hex")).reverse(); +} + +function encodeTransfer(recipient, amount) { + return concat([ + "0x01", + uint256LittleEndian(amount), + getBytes(getAddress(recipient)), + ]); +} + +function encodeWithdrawal(amount) { + return concat(["0x00", uint256LittleEndian(amount)]); +} + +async function submitUserOperation(privateKey, nonce, data) { + const wallet = new Wallet(privateKey); + const message = { + nonce, + max_fee: Number(process.env.MAX_FEE ?? 2000), + data, + }; + + const domain = { + name: "CartesiAppSequencer", + version: "1", + chainId: Number(required("CHAIN_ID")), + verifyingContract: required("APP_ADDRESS"), + }; + + const types = { + UserOp: [ + { name: "nonce", type: "uint32" }, + { name: "max_fee", type: "uint16" }, + { name: "data", type: "bytes" }, + ], + }; + + const signature = await wallet.signTypedData( + domain, + types, + message, + ); + const response = await fetch(`${required("SEQUENCER_URL")}/tx`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + message, + signature, + sender: wallet.address, + }), + }); + + const body = await response.text(); + if (!response.ok) { + throw new Error( + `sequencer returned HTTP ${response.status}: ${body}`, + ); + } + + console.log("Application payload:", data); + console.log("Soft confirmation:", JSON.parse(body)); +} + +const [action, ...args] = process.argv.slice(2); +const privateKey = required("WALLET_PRIVATE_KEY"); + +switch (action) { + case "transfer": { + const [recipient, amount, nonce] = args; + if (!recipient || !amount || nonce === undefined) { + throw new Error( + "usage: transfer ", + ); + } + await submitUserOperation( + privateKey, + Number(nonce), + encodeTransfer(recipient, BigInt(amount)), + ); + break; + } + case "withdraw": { + const [amount, nonce] = args; + if (!amount || nonce === undefined) { + throw new Error("usage: withdraw "); + } + await submitUserOperation( + privateKey, + Number(nonce), + encodeWithdrawal(BigInt(amount)), + ); + break; + } + default: + throw new Error("choose one action: transfer or withdraw"); +} +``` + +```js title="client/feed.mjs" +import WebSocket from "ws"; + +const sequencerUrl = process.env.SEQUENCER_URL; +if (!sequencerUrl) throw new Error("SEQUENCER_URL is required"); + +const feedUrl = + `${sequencerUrl.replace(/^http/, "ws")}` + + "/ws/subscribe?from_offset=0"; +const socket = new WebSocket(feedUrl); + +socket.on("open", () => { + console.log(`Subscribed to ${feedUrl}`); +}); +socket.on("message", (data) => { + console.log(JSON.stringify(JSON.parse(data.toString()), null, 2)); +}); +socket.on("close", (code, reason) => { + console.log(`Feed closed with code ${code}: ${reason.toString()}`); +}); +socket.on("error", (error) => { + console.error("Feed error:", error); +}); +``` diff --git a/app-sequencer/tutorials/snippets/_wallet-core-cargo.md b/app-sequencer/tutorials/snippets/_wallet-core-cargo.md new file mode 100644 index 000000000..8ff50523b --- /dev/null +++ b/app-sequencer/tutorials/snippets/_wallet-core-cargo.md @@ -0,0 +1,18 @@ +```toml title="app-core/Cargo.toml" +[package] +name = "app-core" +version.workspace = true +edition.workspace = true +license.workspace = true + +[dependencies] +sequencer-core.workspace = true +alloy-primitives.workspace = true +alloy-sol-types.workspace = true +ssz.workspace = true +ssz_derive.workspace = true +serde.workspace = true +serde_json.workspace = true +tracing.workspace = true +types.workspace = true +``` diff --git a/app-sequencer/tutorials/snippets/_wallet-core-lib.md b/app-sequencer/tutorials/snippets/_wallet-core-lib.md new file mode 100644 index 000000000..38f6fdde8 --- /dev/null +++ b/app-sequencer/tutorials/snippets/_wallet-core-lib.md @@ -0,0 +1,10 @@ +```rust title="app-core/src/lib.rs" +mod method; +mod wallet; + +pub use method::{Method, Transfer, Withdrawal}; +pub use wallet::{ + BATCH_SUBMITTER_ADDRESS, DEVNET_ERC20_PORTAL_ADDRESS, + DEVNET_TEST_TOKEN_ADDRESS, WalletApp, WalletConfig, +}; +``` diff --git a/app-sequencer/tutorials/snippets/_wallet-machine.md b/app-sequencer/tutorials/snippets/_wallet-machine.md new file mode 100644 index 000000000..7d2c5e41b --- /dev/null +++ b/app-sequencer/tutorials/snippets/_wallet-machine.md @@ -0,0 +1,53 @@ +```toml title="Cross.toml" +[target.riscv64gc-unknown-linux-musl] +image = "ghcr.io/cross-rs/riscv64gc-unknown-linux-musl@sha256:f5a375283c54578efcc6e61c78ea7661392c2e9b2e108afe89938d2f7b8b489d" +``` + +```toml title="cartesi.toml" +sdk = "cartesi/sdk:0.12.0-alpha.41" + +[machine] +ram_length = "128Mi" +entrypoint = "/dapp/dapp" +use_docker_workdir = true + +[drives.root] +builder = "docker" +dockerfile = "machine/Dockerfile" +format = "ext2" +``` + +```dockerfile title="machine/Dockerfile" +# Stage 1 obtains cartesi-init from the machine guest tools package. +FROM riscv64/debian:stable-slim AS extractor + +ARG MACHINE_GUEST_TOOLS_VERSION=0.17.2 +ARG TOOLS_SHA512="4af9911a5a76738d526bfc2b5462cf96c9dee98ec8b23f3ca91ac4849d5761765f471b5e2e8779809bc4a26d2799f8e744622864fa549ada5941e21d999ff4be" + +ADD https://github.com/cartesi/machine-guest-tools/releases/download/v${MACHINE_GUEST_TOOLS_VERSION}/machine-guest-tools_riscv64.deb /tmp/tools.deb + +RUN echo "${TOOLS_SHA512} /tmp/tools.deb" | sha512sum -c - \ + && dpkg -x /tmp/tools.deb /tmp/out + +# Stage 2 creates the root file system used by the Cartesi Machine. +FROM riscv64/alpine@sha256:372839ff152f938e12282226fb5f9ddaef72f9662dcadbf9dd0de5ce287c694e + +ARG ALPINE_MAIN_REPOSITORY=https://dl-cdn.alpinelinux.org/alpine/v3.22/main +ARG LIBGCC_VERSION=14.2.0-r6 + +RUN apk add --no-cache \ + --repository="${ALPINE_MAIN_REPOSITORY}" \ + "libgcc=${LIBGCC_VERSION}" + +COPY --from=extractor --chmod=755 \ + /tmp/out/usr/sbin/cartesi-init \ + /usr/sbin/cartesi-init + +RUN adduser -h /dapp -D dapp +ENV PATH="/dapp:${PATH}" +WORKDIR /dapp +COPY --chown=dapp:dapp --chmod=755 machine/out/dapp . + +USER dapp +ENTRYPOINT ["dapp"] +``` diff --git a/app-sequencer/tutorials/snippets/_wallet-method.md b/app-sequencer/tutorials/snippets/_wallet-method.md new file mode 100644 index 000000000..63dcf12bc --- /dev/null +++ b/app-sequencer/tutorials/snippets/_wallet-method.md @@ -0,0 +1,24 @@ +```rust title="app-core/src/method.rs" +use alloy_primitives::{Address, U256}; +use ssz_derive::{Decode, Encode}; + +pub const MAX_METHOD_PAYLOAD_BYTES: usize = 1 + 32 + 20; + +#[derive(Debug, Clone, PartialEq, Eq, Encode, Decode)] +#[ssz(enum_behaviour = "union")] +pub enum Method { + Withdrawal(Withdrawal), + Transfer(Transfer), +} + +#[derive(Debug, Clone, PartialEq, Eq, Encode, Decode)] +pub struct Withdrawal { + pub amount: U256, +} + +#[derive(Debug, Clone, PartialEq, Eq, Encode, Decode)] +pub struct Transfer { + pub amount: U256, + pub to: Address, +} +``` diff --git a/app-sequencer/tutorials/snippets/_wallet-sequencer.md b/app-sequencer/tutorials/snippets/_wallet-sequencer.md new file mode 100644 index 000000000..bb7d04439 --- /dev/null +++ b/app-sequencer/tutorials/snippets/_wallet-sequencer.md @@ -0,0 +1,30 @@ +```toml title="app-sequencer/Cargo.toml" +[package] +name = "app-sequencer" +version.workspace = true +edition.workspace = true +license.workspace = true + +[dependencies] +app-core.workspace = true +sequencer.workspace = true +tokio.workspace = true +tracing-subscriber.workspace = true +``` + +```rust title="app-sequencer/src/main.rs" +use app_core::{WalletApp, WalletConfig}; +use tracing_subscriber::EnvFilter; + +#[tokio::main] +async fn main() -> std::process::ExitCode { + tracing_subscriber::fmt() + .with_env_filter( + EnvFilter::try_from_default_env() + .unwrap_or_else(|_| EnvFilter::new("info")), + ) + .init(); + + sequencer::run_main(|| WalletApp::new(WalletConfig::devnet())).await +} +``` diff --git a/app-sequencer/tutorials/snippets/_wallet-workspace-cargo.md b/app-sequencer/tutorials/snippets/_wallet-workspace-cargo.md new file mode 100644 index 000000000..c3da02d14 --- /dev/null +++ b/app-sequencer/tutorials/snippets/_wallet-workspace-cargo.md @@ -0,0 +1,27 @@ +```toml title="Cargo.toml" +[workspace] +resolver = "2" +members = ["app-core", "app-sequencer", "canonical-app"] + +[workspace.package] +version = "0.1.0" +edition = "2024" +license = "Apache-2.0" + +[workspace.dependencies] +app-core = { path = "app-core" } +sequencer = { git = "https://github.com/cartesi/sequencer", tag = "v0.1.0-alpha.9" } +sequencer-core = { git = "https://github.com/cartesi/sequencer", tag = "v0.1.0-alpha.9" } +sequencer-canonical = { package = "canonical-app", git = "https://github.com/cartesi/sequencer", tag = "v0.1.0-alpha.9" } +alloy-primitives = { version = "1.6", features = ["serde", "k256"] } +alloy-sol-types = "1.6" +ssz = { package = "ethereum_ssz", version = "0.10" } +ssz_derive = { package = "ethereum_ssz_derive", version = "0.10" } +serde = { version = "1", features = ["derive"] } +serde_json = "1" +tokio = { version = "1.53", features = ["macros", "rt-multi-thread"] } +tracing = "0.1" +tracing-subscriber = { version = "0.3", features = ["env-filter"] } +types = { version = "0.1", git = "https://github.com/GCdePaula/cartesi-tools-rs", rev = "ed14b98ecfe9796dc3ca7c9b96bfdbf0ef9baf22" } +trolley = { version = "0.1", git = "https://github.com/GCdePaula/cartesi-tools-rs", rev = "ed14b98ecfe9796dc3ca7c9b96bfdbf0ef9baf22" } +``` diff --git a/app-sequencer/usage/application-requirements.md b/app-sequencer/usage/application-requirements.md new file mode 100644 index 000000000..cc9975772 --- /dev/null +++ b/app-sequencer/usage/application-requirements.md @@ -0,0 +1,212 @@ +--- +title: "Application integration requirements" +sidebar_label: "Application requirements" +description: "The execution, progress, persistence, and determinism contracts an application must satisfy to work safely with the app-specific sequencer." +--- + +An application integrates with the sequencer by implementing `sequencer_core::application::Application`. The off-chain sequencer uses this implementation to predict application state, and the canonical scheduler uses it inside the Cartesi machine to compute the authoritative result. + +Both execution paths must produce the same state and outputs for the same ordered inputs. The interface therefore defines more than application methods. It also defines progress tracking, recovery dumps, canonical state bytes, and failure behavior. + +This page describes the application code shared by the two execution paths. The application-specific sequencer binary is covered in [Integrating the sequencer with a Cartesi application](./integration.md). + +## Complete interface overview + +The required and optional parts of `Application` are grouped below. + +| Area | Item | Required | Purpose | +| ------------------------ | -------------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------- | +| Payload bound | `MAX_METHOD_PAYLOAD_BYTES` | Yes | Limits the encoded application payload accepted in `UserOp.data` and supplies the sequencer's batch-sizing calculation | +| Validation | `validate_user_op` | Yes | Checks application-level acceptance rules without changing state | +| User operation execution | `execute_valid_user_op` | Yes | Applies an operation that passed the protocol and application checks | +| Direct input execution | `execute_direct_input` | Yes | Applies an input recorded directly on the base layer | +| Progress | `last_executed_safe_block` | Yes | Reports the highest base-layer block covered by executed inputs | +| Progress | `executed_input_count` | Yes | Reports how many user operations and direct inputs have executed | +| Persistence | `create_dump` | Yes | Writes a complete and durable recovery dump | +| Persistence | `from_dump` | Yes | Reconstructs equivalent application state from a dump | +| Persistence | `delete_dump` | Yes | Removes a dump that the sequencer no longer needs | +| Persistence | `state_file_in_dump` | Yes | Locates the canonical state file inside a dump | +| State comparison | `canonical_snapshot_bytes` | Conditional | Returns deterministic state bytes for machine inspection and watchdog comparison | +| Diagnostics | `export_state` | No | Returns human-readable JSON for debugging | + +`canonical_snapshot_bytes` and `export_state` have default implementations that return an error. Implement canonical bytes when the deployment serves machine state through inspect requests or uses the watchdog comparison. + +## User operation validation and execution + +Every user operation must pass through `validate_and_execute_user_op`. This shared function is used by the off-chain inclusion path and the canonical scheduler, giving both paths the same execution sequence: + +```text +1. Check user_op.max_fee against the current frame fee +2. Call app.validate_user_op(...) +3. Build a ValidUserOp with the committed frame fee +4. Call app.execute_valid_user_op(...) +``` + +Application code should call the shared function in tests and custom execution paths. Calling `execute_valid_user_op` directly bypasses the protocol fee guard and can create behavior that the canonical scheduler will not reproduce. + +### Keep validation read-only + +`validate_user_op` receives: + +- the recovered sender address; +- the original `UserOp`, including its nonce, offered `max_fee`, and application payload; +- the fee exponent of the current frame. + +It must inspect state without changing it. Validation can run in contexts where a mutation would be applied twice or at a different point during replay. Side effects in validation can therefore make live execution, restart replay, and canonical execution disagree. + +The protocol checks `max_fee >= current_fee` before application validation. The application still uses `current_fee` when it needs to verify that the sender can pay the resulting fee from application state. + +The current rejection vocabulary contains: + +| Reason | Meaning | +| ------------------------ | -------------------------------------------------------------------- | +| `InvalidNonce` | The operation nonce does not match the application's expected nonce | +| `InvalidMaxFee` | The sender's offered fee is below the current frame fee | +| `InsufficientFeeBalance` | Application state shows that the sender cannot pay the committed fee | + +A validation rejection changes no state, produces no output, and is not placed in the ordered transaction stream. + +### Execute an accepted operation deterministically + +`execute_valid_user_op` receives a `ValidUserOp` containing the sender, the committed frame fee, and the application payload. It also receives the frame's `safe_block`. + +The method must: + +- apply the application transition exactly once; +- charge or account for the committed fee according to the application design; +- update application replay protection, such as the sender nonce; +- increment `executed_input_count`; +- set the safe-block clock to `max(previous_clock, safe_block)`; +- return deterministic notices and vouchers as `AppOutput` values. + +The valid operation no longer contains the submitted nonce or offered `max_fee`. Any checks that depend on those fields belong in `validate_user_op`. Execution uses the frame fee selected by the protocol. + +An operation may be included while producing no outputs. The CMA wallet uses this behavior when a decoded action cannot be completed after its protocol-level acceptance: it charges the data-availability fee, consumes the nonce, and returns an empty output list. Applications must define this behavior carefully because an included no-op differs from a validation rejection. + +Notices and vouchers computed off-chain are predictions. The corresponding outputs become authoritative when the canonical machine executes the recorded batch. + +## Direct input handling + +`execute_direct_input` has no default implementation. Every application must define how inputs that did not enter through `POST /tx` affect its state. + +The method receives: + +- the base-layer sender; +- the base-layer inclusion block; +- the raw payload. + +The canonical scheduler treats every recorded input from an address other than the configured batch submitter as a direct input. The application must then authenticate and decode the input according to its own rules. For example, the CMA wallet credits a deposit only when the sender is its configured ERC-20 portal and the payload names its supported token. + +For every executed direct input, the application must increment `executed_input_count` and update its safe-block clock with `input.block_number`. An ignored or unsupported direct input still counts as executed once the application has processed it. + +The actions supported through this method determine what users can do while the sequencer is unavailable. See [Direct inputs vs sequenced transactions](../concepts/direct-vs-sequenced.md). + +## Progress tracking + +### Safe-block clock + +`last_executed_safe_block` returns the greatest block covered by any input executed by the current application state: + +```text +user operation: max(clock, frame.safe_block) +direct input: max(clock, input.block_number) +``` + +It returns `0` before any input executes. The value is part of logical application state and must survive cloning and dump restoration. + +Recovery uses this clock to determine which base-layer inputs are already reflected in a checkpoint. Reporting a value that is too high can skip required inputs. Reporting one that is too low can execute an input again. + +### Executed input count + +`executed_input_count` counts user operations and direct inputs that the application executed. It is primarily a diagnostic agreement check used to compare live and replayed application instances. + +Persist the count in every dump and restore it exactly. Do not derive it from balances, nonces, or database row numbers because those values can represent different histories. + +## Recovery dump contract + +The sequencer creates application dumps at batch boundaries, restores them during startup and recovery, and deletes superseded dumps. A dump may contain several application-specific files. + +### Creating a dump + +`create_dump(prefix)` receives a path that does not yet exist. The implementation creates that directory and writes every value that can influence future execution, including: + +- application databases or state bytes; +- sender nonces and other replay protection; +- application configuration that changes execution; +- `last_executed_safe_block`; +- `executed_input_count`; +- metadata required to decode or reconstruct the main state. + +When the method returns `Ok`, the dump must survive an immediate kernel crash. On POSIX systems, this requires synchronizing each file, the dump directory, and its parent directory before returning. The sequencer writes the SQLite row that references the dump only after `create_dump` succeeds. + +### Restoring and deleting dumps + +`from_dump(prefix)` must reconstruct state equivalent to the state that created the dump. Equivalence includes future behavior, progress values, and canonical state bytes, not only visible balances. + +`delete_dump(prefix)` removes a previously created dump when the sequencer's garbage collection marks it as superseded. The implementation should limit deletion to the supplied dump path. + +### Identifying canonical state + +`state_file_in_dump(prefix)` is a pure path function. It must return one file inside the dump without loading application state. The bytes in that file must match the canonical machine's inspected state for the same logical history. + +`canonical_snapshot_bytes()` returns the in-memory form of that same canonical representation. Keeping both paths byte-identical allows the watchdog to compare predicted and canonical state without application-specific conversion. The CMA wallet uses its ledger records image for both values and stores other recovery data, such as nonces and progress, in a separate metadata file. + +`export_state()` can expose convenient JSON for debugging, but the sequencer does not use that JSON to restore state. + +## Determinism across execution environments + +Application behavior must depend only on the ordered input and current application state. Avoid consensus-path behavior based on: + +- wall-clock time; +- random values; +- floating-point calculations; +- unordered collection iteration; +- thread scheduling; +- host-specific file layout or environment state; +- platform-specific numeric or serialization behavior. + +Use the `safe_block`, direct-input block number, and EIP-712 domain supplied by the protocol when execution needs chain context. + +Applications with target-specific storage must preserve the same logical and canonical byte representation on the host and in the Cartesi machine. The CMA integration uses a host buffer for sequencer prediction and a machine drive for canonical execution, then verifies that both produce byte-identical records. + +## Error and replay behavior + +User-caused refusal belongs in deterministic validation or in a clearly defined included outcome. `AppError::Internal` and I/O errors represent failures from which the sequencer cannot safely continue. + +An internal execution error stops the off-chain inclusion lane. Reserve it for invariant violations and infrastructure failures. Do not use it as a general response to malformed application data or insufficient business-level balance. + +Any input that succeeds during live execution must succeed with the same result during replay. A transaction that reads unpersisted configuration, current time, or external mutable state can pass live and fail after restart, preventing the sequencer from recovering. + +## Runtime type requirements + +The `Application` trait requires `Send`. The `sequencer::run_main` entry point adds these bounds: + +```rust +Application + Clone + Sync + 'static +``` + +`Clone` must produce an independent instance with equivalent logical state. A shallow clone of a mutable database handle or foreign pointer may violate that requirement. Applications that wrap non-Rust state must provide safe synchronization and ownership across clones. + +The genesis constructor is intentionally outside the trait. The application-specific binary passes it to `run_main`, and the closure is invoked only by `setup`. Normal `run` startup restores state through `from_dump`. + +## Verification checklist + +Before deploying an application integration, test that: + +- validation produces no state changes; +- rejected operations leave nonces, balances, progress, and outputs unchanged; +- valid operations and direct inputs update both progress values correctly; +- application payloads at the declared size limit are accepted and larger payloads are rejected at ingress; +- dumps restore all logical state and canonical bytes exactly; +- a cloned application has equivalent state without unsafe shared mutation; +- replaying persisted inputs reproduces live state and outputs; +- the canonical scheduler and off-chain prediction produce byte-identical state; +- the host build and machine build use compatible encodings and arithmetic. + +The CMA integration demonstrates these checks in `cma-app-core/tests/application.rs` and `cma-canonical-app/tests/duality.rs`. + +## Next steps + +- Follow the full build sequence in [Integrating the sequencer with a Cartesi application](./integration.md). +- Review recovery state in [Snapshots and checkpoints](../recovery/snapshots.md). +- Study ordering agreement in [Deterministic execution order](../concepts/execution-order.md). diff --git a/app-sequencer/usage/integration.md b/app-sequencer/usage/integration.md new file mode 100644 index 000000000..5002d5c99 --- /dev/null +++ b/app-sequencer/usage/integration.md @@ -0,0 +1,186 @@ +--- +title: "Integrating the sequencer with a Cartesi application" +sidebar_label: "Application integration" +description: "How to connect application logic to the off-chain sequencer, include the canonical scheduler in the Cartesi machine, and verify that both execution paths agree." +--- + +Adding the app-specific sequencer produces two programs that use the same application logic: + +| Component | Runs in | Responsibility | +| --------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------- | +| Application core | Both programs | Validates operations, changes application state, handles direct inputs, and serializes state | +| Sequencer binary | Off-chain host | Accepts signed operations, predicts their results, creates batches, and submits those batches to the base layer | +| Canonical application | Cartesi machine | Reads recorded inputs, applies the authoritative scheduling rules, and emits the application's notices and vouchers | + +The sequencer accelerates an existing application. It does not replace the Cartesi machine or change where settlement occurs. + +## Recommended project structure + +Keep the application logic in a library that both programs can compile. A practical workspace looks like this: + +```text +my-application/ +├── app-core/ # Application trait implementation and domain logic +├── app-sequencer/ # Small off-chain binary using sequencer::run_main +└── canonical-app/ # Cartesi machine entry point using the canonical scheduler +``` + +`app-core` depends on `sequencer-core`, which contains the shared protocol types and the `Application` trait. `app-sequencer` depends on the higher-level `sequencer` crate. `canonical-app` depends on `sequencer-core` and the rollup I/O library used to read inputs and emit outputs inside the machine. + +This separation is important when application dependencies need different host and RISC-V implementations. The CMA wallet, for example, uses the same ledger API in both environments, with a host-backed buffer for prediction and a persistent machine drive for canonical execution. + +![The shared app-core library supplies the same application logic to the host app-sequencer and the canonical application. Each executable adds the responsibilities of its own execution environment.](../images/integration-duality.png) + +## Step 1: implement the shared application core + +Implement `sequencer_core::application::Application` on the state type shared by the host sequencer and canonical machine. The interface covers payload limits, validation and execution, direct inputs, progress tracking, durable dumps, and canonical state bytes. + +[Application integration requirements](./application-requirements.md) is the authoritative method contract and verification checklist. Complete that contract before wiring either executable. At this stage, the integration-specific goal is to keep the implementation in a dependency that can compile for both the host and the Cartesi machine, with target-specific storage hidden behind the same logical interface. + +## Step 2: build the application-specific sequencer + +The sequencer repository provides a library, so each application builds its own executable. The binary is intentionally small because command parsing, setup, recovery, HTTP services, and batch submission are supplied by the `sequencer` crate. + +A typical crate declares these dependencies: + +```toml +[dependencies] +app-core = { path = "../app-core" } +sequencer = { path = "../sequencer/sequencer" } +tokio = { version = "1.35", features = ["macros", "rt-multi-thread"] } +tracing-subscriber = { version = "0.3", features = ["env-filter"] } +``` + +Adjust the paths or version declarations to match the sequencer release used by your project. The entry point then supplies the application constructor: + +```rust +use app_core::{MyApp, MyAppConfig}; +use tracing_subscriber::EnvFilter; + +#[tokio::main] +async fn main() -> std::process::ExitCode { + tracing_subscriber::fmt() + .with_env_filter( + EnvFilter::try_from_default_env() + .unwrap_or_else(|_| EnvFilter::new("info")), + ) + .init(); + + sequencer::run_main(|| { + MyApp::genesis(MyAppConfig::from_env()) + .expect("failed to initialize application state") + }) + .await +} +``` + +The constructor closure runs during `setup`, when the genesis dump is created. A normal `run` restores the application from a dump. Configuration that affects execution must therefore be stored in the dump, or checked against the stored deployment configuration, instead of depending only on the current process environment. + +The resulting executable supports `setup`, `run`, and `flush-mempool`. The sequencer repository's `examples/wallet-sequencer` crate is the smallest reference for this composition. + +## Step 3: include the canonical scheduler in the machine + +The Cartesi machine must process batches and direct inputs according to the canonical scheduler. Its entry point needs to: + +1. initialize the rollup I/O connection; +2. create the same application state type used by the off-chain sequencer; +3. create `SchedulerConfig` with the batch submitter's address; +4. pass each advance-state request to `Scheduler::process_input` with its sender, inclusion block, EIP-712 domain, and payload; +5. emit every returned notice and voucher through the rollup I/O connection; +6. serve canonical state bytes for supported inspect requests. + +The reference I/O loop is in `examples/canonical-app/src/scheduler/mod.rs` in the sequencer repository. With that loop available to the application, the machine entry point has this shape: + +```rust +use app_core::{MyApp, MyAppConfig}; +use canonical_app::{run_scheduler_forever, SchedulerConfig}; +use trolley::cmt::RollupCmt; + +fn main() { + let rollup = RollupCmt::try_new() + .expect("failed to initialize rollup I/O"); + let app = MyApp::genesis(MyAppConfig::from_env()) + .expect("failed to initialize application state"); + + run_scheduler_forever( + rollup, + app, + SchedulerConfig::new(BATCH_SUBMITTER_ADDRESS), + ); +} +``` + +Here, `canonical_app` represents the application's I/O shell based on the reference example. `run_scheduler_forever` is not exported by `sequencer-core` itself. + +### Keep the batch identity consistent + +The canonical scheduler classifies an input by its base-layer sender: + +- an input from `SchedulerConfig.sequencer_address` is decoded as a sequencer batch; +- an input from any other address is queued as a direct input. + +The address passed to `SchedulerConfig::new` must match both `CARTESI_SEQUENCER_BATCH_SUBMITTER_ADDRESS` used during `setup` and the address derived from the private key used during `run`. A mismatch causes valid batches to be handled as direct inputs. + +Do not confuse the batch submitter with an application-level fee recipient. They may use the same address, but they serve different purposes. The CMA wallet keeps them as separate configuration values. + +### Flush machine-backed state when required + +Applications that keep canonical state on a persistent machine drive must ensure writes reach that drive before the rollup yields and a machine snapshot is taken. The CMA wallet adapts the reference I/O loop to synchronize its accounts drive before each request boundary. An in-memory application does not need this extra step. + +Compile the canonical program for the machine target, package it with its runtime dependencies, and build a new machine image. Because the scheduler becomes part of that image, its template hash changes. Integrating the sequencer into an existing deployment therefore requires deployment of the new image. + +## Step 4: prove that both paths agree + +After the application-level checks in the [requirements checklist](./application-requirements.md#verification-checklist) pass, test the assembled integration at three boundaries: + +1. **Scheduler agreement:** give direct inputs and batches to `Scheduler`, replay the same operations through the prediction path, and require byte-identical canonical state. +2. **Machine execution:** boot the built Cartesi machine image, send a direct input followed by a covering batch, and verify the notices and vouchers produced inside the machine. +3. **End-to-end synchronization:** run the sequencer and rollups node against the same base layer, submit an operation through the sequencer, and compare the predicted state with the machine's finalized state. + +The CMA integration contains examples of each important layer: + +| Path | What it demonstrates | +| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | +| `sequencer-integration/cma-app-core` | A real application adapter, deterministic execution, target-specific state backing, and complete dumps | +| `sequencer-integration/cma-sequencer` | The thin `run_main` composition | +| `sequencer-integration/cma-canonical-app` | The canonical scheduler loop and machine-drive synchronization | +| `sequencer-integration/cma-canonical-app/tests/duality.rs` | Byte-for-byte agreement between canonical scheduling and off-chain prediction | +| `sequencer-integration/cma-machine-test` | Execution of direct inputs and a batch in the built machine image | + +Application-specific choices in that demo, including its ERC-20 portal format, SSZ method union, libcma ledger, and `/dev/pmem1` drive, are examples and are not sequencer protocol requirements. + +## Step 5: set up and run the service + +After building the application-specific binary, initialize one data directory for each deployment with `setup`, then start it with `run` and the matching submitter key. Follow [Configure, set up, and run the sequencer](../operations/setup-and-running.md) for the complete commands and configuration. Use [Process supervision and recovery operations](../operations/orchestration.md) when preparing the production process. + +## Step 6: add the client fast path + +Clients that use the sequencer must: + +1. encode the application's method payload; +2. sign a `UserOp` using the deployment's EIP-712 domain; +3. submit it to `POST /tx`; +4. treat the successful response as a soft confirmation; +5. consume the ordered WebSocket feed and reconcile later status changes. + +The original direct-input route remains available. The application decides which actions that route supports through `execute_direct_input`. + +See [Submitting operations](./submitting-operations.md), [Reading the sequenced feed](./reading-the-feed.md), and [Soft confirmations](../concepts/soft-confirmations.md) before updating production clients. + +## Integration checklist + +- The same deterministic application logic compiles for the host and the Cartesi machine. +- The `Application` implementation covers execution, progress, persistence, and canonical state bytes. +- Dumps restore every value that can affect future execution and are durable when created. +- The application satisfies the `Clone`, `Send`, `Sync`, and `'static` bounds required by `run_main`. +- The canonical machine runs the scheduler before application execution. +- The batch submitter address is identical in setup, runtime key configuration, and `SchedulerConfig`. +- Persistent machine state is synchronized before snapshots when the storage design requires it. +- Agreement tests compare canonical bytes, not only high-level balances or outputs. +- The new machine image and its template hash are deployed before the sequencer is opened to clients. + +## Next steps + +- Run the local transaction flow in the [Quickstart](./quickstart.md). +- Send a signed transaction with [Submitting transactions](./submitting-operations.md). +- Re-check any interface rule in [Application requirements](./application-requirements.md). diff --git a/app-sequencer/usage/quickstart.md b/app-sequencer/usage/quickstart.md new file mode 100644 index 000000000..0df3da197 --- /dev/null +++ b/app-sequencer/usage/quickstart.md @@ -0,0 +1,210 @@ +--- +title: "Quickstart" +sidebar_label: "Quickstart" +description: "Initialize and run an application-specific sequencer, submit a signed transaction, and find it in the ordered feed." +--- + +This guide covers the shortest complete client loop: initialize a sequencer for a deployed application, start it, submit one application transaction, and read that transaction from the ordered feed. + +## Prerequisites + +You need: + +- a local base-layer node, usually Anvil, available at `http://127.0.0.1:8545`; +- a deployed Cartesi application contract whose data-availability configuration points to an `InputBox`; +- an application-specific sequencer binary built with that application's `Application` implementation; +- a funded base-layer account dedicated to submitting batches; +- a user account that can pass the application's nonce, fee-balance, and method validation; +- Node.js with `ethers` installed, plus `curl` and `websocat`. + +The sequencer is a library that each application builds into its own executable. If your application does not have one yet, follow [Application integration](./integration.md). The sequencer repository's `examples/wallet-sequencer` crate is a reference binary, but its wallet state and method encoding must still match the application deployment you use. + +:::note Application-specific values are required +This guide uses placeholders for the application address, keys, and `METHOD_DATA`. A successful submission requires a payload your application can decode and state that satisfies its validation rules. For a wallet, that usually means the sender has deposited enough application funds to cover the action and its fee. +::: + +## Step 1: initialize the sequencer data directory + +`setup` creates the initial application snapshot and records the deployment identity in the data directory. It reads from the base layer but sends no transaction, so it needs the batch submitter's address and never its private key. + +```bash +export APP_ADDRESS=0xYourApplicationAddress +export SUBMITTER_ADDRESS=0xYourBatchSubmitterAddress +export SEQUENCER_DATA_DIR=./sequencer-data + +export CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT=http://127.0.0.1:8545 +export CARTESI_SEQUENCER_BLOCKCHAIN_ID=31337 +export CARTESI_SEQUENCER_APP_ADDRESS=$APP_ADDRESS +export CARTESI_SEQUENCER_BATCH_SUBMITTER_ADDRESS=$SUBMITTER_ADDRESS +export CARTESI_SEQUENCER_DATA_DIR=$SEQUENCER_DATA_DIR + +./app-sequencer setup +``` + +Replace `./app-sequencer` with your executable. If you are working inside the sequencer repository, keep the exported configuration and run the reference binary with: + +```bash +cargo run -p wallet-sequencer --bin wallet-sequencer-devnet -- setup +``` + +The `wallet-sequencer-devnet` binary selects the reference wallet's local development configuration. The standard `wallet-sequencer` binary uses its non-local configuration. + +The application contract must be deployed before this step. During setup, the sequencer verifies the chain identifier and discovers the application's `InputBox` through the contract's data-availability configuration. + +`setup` is idempotent for an already prepared data directory. Keep the directory because `run` reads the pinned identity and genesis state from it. + +## Step 2: start the sequencer + +Put the batch-submitter private key in a file readable only by the current user. The key must derive the address supplied during setup. + +```bash +install -m 600 /dev/null /tmp/batch-submitter.key +``` + +Open `/tmp/batch-submitter.key` in an editor, place the hexadecimal private key on its first line, and start the sequencer: + +```bash +CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT=http://127.0.0.1:8545 \ +CARTESI_SEQUENCER_AUTH_PRIVATE_KEY_FILE=/tmp/batch-submitter.key \ +CARTESI_SEQUENCER_DATA_DIR=$SEQUENCER_DATA_DIR \ + ./app-sequencer run +``` + +Leave this process running. `run` obtains the chain identifier, application address, and batch-submitter address from the data directory. A key for a different address causes startup to fail. + +The API listens on `127.0.0.1:3000` by default. In another terminal, verify readiness: + +```bash +curl --fail http://127.0.0.1:3000/readyz +``` + +A loopback RPC endpoint may use plaintext HTTP. Remote RPC endpoints require HTTPS unless the operator explicitly allows HTTP on a trusted private network. See [Configure, set up, and run the sequencer](../operations/setup-and-running.md). + +## Step 3: sign and submit a transaction + +The user signs this EIP-712 type: + +```solidity +struct UserOp { + uint32 nonce; + uint16 max_fee; + bytes data; +} +``` + +Create `sign.mjs`: + +```js +import { readFileSync } from "node:fs"; +import { Wallet } from "ethers"; + +function required(name) { + const value = process.env[name]; + if (!value) throw new Error(`${name} is required`); + return value; +} + +const wallet = new Wallet( + readFileSync(required("USER_PRIVATE_KEY_FILE"), "utf8").trim(), +); + +const domain = { + name: "CartesiAppSequencer", + version: "1", + chainId: Number(required("CHAIN_ID")), + verifyingContract: required("APP_ADDRESS"), +}; + +const types = { + UserOp: [ + { name: "nonce", type: "uint32" }, + { name: "max_fee", type: "uint16" }, + { name: "data", type: "bytes" }, + ], +}; + +const message = { + nonce: Number(required("USER_NONCE")), + max_fee: Number(required("MAX_FEE")), + data: required("METHOD_DATA"), +}; + +const signature = await wallet.signTypedData(domain, types, message); +console.log(JSON.stringify({ message, signature, sender: wallet.address })); +``` + +Store the user's key in another protected file: + +```bash +install -m 600 /dev/null /tmp/user.key +``` + +Add the user's private key to the first line of `/tmp/user.key`. Then supply the deployment and application-specific values and post the signed request: + +```bash +CHAIN_ID=31337 \ +APP_ADDRESS=$APP_ADDRESS \ +USER_PRIVATE_KEY_FILE=/tmp/user.key \ +USER_NONCE=0 \ +MAX_FEE=1100 \ +METHOD_DATA=0xYourApplicationPayload \ + node sign.mjs | \ + curl --fail-with-body \ + --request POST http://127.0.0.1:3000/tx \ + --header 'content-type: application/json' \ + --data @- +``` + +The nonce starts at `0` for a sender with no accepted user operations. `METHOD_DATA` must be the hexadecimal encoding expected by your application. + +`max_fee` is a fee exponent. The unmodified policy starts at exponent `1060`, so `1100` clears that baseline. A deployment can use a different current price, and there is no public fee-discovery endpoint. Obtain the expected baseline from the operator or handle an `EXECUTION_REJECTED` response by correcting the fee and signing again. + +A successful request returns: + +```json +{ + "ok": true, + "sender": "0x...", + "nonce": 0 +} +``` + +The server sends this response only after it validates, executes, and durably stores the operation in its current order. A rejected request returns a non-`200` status with a stable error `code`. See [Submitting transactions](./submitting-operations.md) for the complete error model. + +## Step 4: read the transaction from the feed + +Subscribe from offset `0`: + +```bash +websocat 'ws://127.0.0.1:3000/ws/subscribe?from_offset=0' +``` + +The feed replays its current valid ordering and then waits for new messages. Find the `user_op` with the sender and application payload used above: + +```json +{ + "kind": "user_op", + "offset": 1, + "sender": "0x...", + "fee": 1060, + "data": "0x..." +} +``` + +The displayed `fee` is the committed frame price, so it can be lower than the submitted `max_fee`. The offset may also be greater than `1` if other user operations or direct inputs were ordered first. + +The feed message does not include the nonce or signature. Applications that need reliable transaction matching should include their own request identifier in the application payload. See [Consuming the sequenced transaction feed](./reading-the-feed.md) for cursor and reconnection handling. + +## Understand the confirmation status + +The `POST /tx` success response and the feed entry describe the sequencer's current prediction. They do not show that the operation's batch has reached the base layer. + +Treat the response as a soft confirmation. The operation can still be invalidated if its batch fails to reach the base layer within the protocol deadline. Feed delivery adds an ordering cursor but no additional settlement guarantee. + +For valuable or irreversible actions, verify the outcome from sufficiently settled canonical state. See [Soft confirmations](../concepts/soft-confirmations.md). + +## Next steps + +- Learn the complete request and retry behavior in [Submitting transactions](./submitting-operations.md). +- Build a reliable feed consumer with [Consuming the sequenced transaction feed](./reading-the-feed.md). +- Prepare a production process using [Configure, set up, and run the sequencer](../operations/setup-and-running.md). diff --git a/app-sequencer/usage/reading-the-feed.md b/app-sequencer/usage/reading-the-feed.md new file mode 100644 index 000000000..b9bbbd272 --- /dev/null +++ b/app-sequencer/usage/reading-the-feed.md @@ -0,0 +1,201 @@ +--- +title: "Reading the sequenced transaction feed" +sidebar_label: "Reading the sequenced feed" +description: "How to consume the ordered WebSocket feed, store a reliable resume cursor, recover after disconnection, and account for optimistic delivery." +--- + +The sequenced transaction feed is a database-backed WebSocket stream of the inputs in the sequencer's current execution order. It includes accepted user operations and direct inputs when they enter that order. + +Use the feed to maintain an index, update a provisional application view, or observe activity without repeatedly scanning the base layer. Do not use it as proof that a transaction has settled. + +## Connect to the feed + +Open a WebSocket connection to: + +```text +GET /ws/subscribe?from_offset= +``` + +For a local sequencer, the full URL is: + +```bash +websocat 'ws://127.0.0.1:3000/ws/subscribe?from_offset=0' +``` + +`from_offset` is optional and defaults to `0`. It is an exclusive cursor, so the server sends messages whose offset is greater than the supplied value. Use `0` for the earliest available history, or the last committed offset when resuming. + +Replay and live delivery use the same connection. The server first reads existing rows in ascending offset order, then waits for additional rows. It does not send a separate message when replay has caught up with live activity. + +Messages are JSON text frames. Byte fields are hexadecimal strings with a `0x` prefix. + +## Understand the two message types + +### User operation + +A transaction accepted through `POST /tx` appears as: + +```json +{ + "kind": "user_op", + "offset": 10, + "sender": "0x...", + "fee": 1060, + "data": "0x..." +} +``` + +| Field | Meaning | +| -------- | ----------------------------------------------------------------- | +| `kind` | Always `user_op` for an operation submitted through the sequencer | +| `offset` | Resume cursor assigned by the sequencer's database | +| `sender` | Address recovered from the operation's EIP-712 signature | +| `fee` | Fee exponent committed for the frame that contains the operation | +| `data` | Application-specific method payload | + +`fee` is the price assigned to the operation when it was ordered. The sender's offered `max_fee` is a separate value and is absent from the feed. + +The message does not contain the operation's nonce, signature, offered `max_fee`, batch number, frame number, safe block, outputs, or execution result. If a client needs to match a feed message to a submission unambiguously, include an application-level identifier in `data`. Matching only by `sender` and `data` can be ambiguous when a sender submits the same payload more than once. + +### Direct input + +An input that reached the application through the base layer appears as: + +```json +{ + "kind": "direct_input", + "offset": 11, + "sender": "0x...", + "block_number": 123, + "payload": "0x..." +} +``` + +| Field | Meaning | +| -------------- | -------------------------------------------------- | +| `kind` | Always `direct_input` for a base-layer input | +| `offset` | Resume cursor assigned by the sequencer's database | +| `sender` | Base-layer sender recorded for the input | +| `block_number` | Base-layer block that included the input | +| `payload` | Raw input payload passed to the application | + +A direct input appears when the sequencer places it into the application execution order. Its `block_number` records where it arrived on the base layer. + +Inputs sent by the configured batch submitter are filtered from `direct_input` delivery. Those inputs carry encoded sequencer batches, whose user operations already appear individually as `user_op` messages. + +## Treat the offset as an opaque cursor + +Offsets begin at `1` and increase with the underlying database rows. They are not guaranteed to be consecutive. + +Gaps can occur because invalidated rows are excluded from later reads and batch-submitter inputs are filtered before WebSocket delivery. A sequence such as `40`, `41`, `45` does not mean the client lost messages. + +For every successfully applied message: + +1. read its `offset`; +2. apply the message to the local view; +3. store that exact offset as the new cursor. + +Never calculate a cursor with `lastOffset + 1`. On reconnection, pass the exact last offset that was fully processed: + +```text +GET /ws/subscribe?from_offset=45 +``` + +The next message may have any offset greater than `45`. + +## Process messages without losing progress + +An indexer should update its materialized view and resume cursor in one local database transaction: + +```text +cursor = load_stored_cursor() + +loop: + connect to /ws/subscribe?from_offset=cursor + + for each message: + begin local database transaction + + if message.offset <= cursor: + skip it + else: + apply message to provisional view + store message.offset as cursor + + commit local database transaction + + if disconnected: + reconnect using the stored cursor +``` + +This ordering avoids two common failures: + +- Storing the cursor before applying the message can lose that message if the process stops between the two writes. +- Applying the message before storing the cursor can apply it twice after a crash unless both changes are atomic or the handler is idempotent. + +Process messages serially. Starting asynchronous work for several messages at once can commit a later offset before an earlier message finishes, which breaks the execution order the feed provides. + +## Recover from disconnections + +For an ordinary network interruption, reconnect with the last committed offset. The database-backed replay covers messages written while the client was offline, then the connection continues with live delivery. + +Use retry delay and backoff when the server is unavailable. A graceful sequencer shutdown closes active subscriptions, and reaching the subscriber limit rejects a new WebSocket handshake with HTTP `429` and error code `OVERLOADED`. + +### Recover after a long absence + +One connection can replay at most 50,000 deliverable events by default. If the requested cursor is further behind, the server completes the WebSocket upgrade and immediately closes the socket with: + +| Property | Value | +| ---------- | -------------------------- | +| Close code | `1008` | +| Reason | `catch-up window exceeded` | + +An operator-managed indexer recovers by using a snapshot as its new starting point: + +1. request `GET /latest_snapshot` on the operator's internal network; +2. read the snapshot bytes and the `X-L2-Tx-Index` response header; +3. replace the provisional local state and cursor together; +4. subscribe with `from_offset` set to the header value. + +The snapshot contains application state through that offset. The exclusive subscription then supplies every later feed message. + +The snapshot routes are internal operator endpoints. Apply the access controls described in [Sequencer security](../operations/security.md#separate-public-and-internal-routes). + +## Know what the feed confirms + +The feed reports the sequencer's current provisional ordering. It supplies input identity, payload, and a resume cursor, but it does not report batch position, application outputs, base-layer acceptance, settlement, or a later invalidation. + +A fresh replay excludes invalidated batches. A client that already received an affected message gets no rollback notification and will not detect the change by resuming from its latest cursor. + +Use one of these reconciliation strategies when invalidation matters to the product: + +- rebuild the provisional view from a newer `/latest_snapshot` and resume from its offset; +- compare important outcomes with the canonical application state; +- wait for sufficiently settled base-layer state before allowing an irreversible action. + +Feed delivery and the `POST /tx` response are both optimistic results from the same sequencer. Seeing a submitted operation on the feed does not turn its soft confirmation into a final confirmation. See [Soft confirmations](../concepts/soft-confirmations.md). + +## Choose between the feed and a snapshot + +The two interfaces solve different problems: + +| Need | Recommended source | +| ----------------------------------- | ------------------------------------------------------- | +| Maintain an ordered activity index | WebSocket feed | +| Continue after a short interruption | Feed replay from the stored offset | +| Initialize a stateful indexer | Latest snapshot, followed by the feed | +| Display a current predicted balance | Application state derived from a snapshot or an indexer | +| Establish a settled result | Canonical settled state and the base layer | + +The CMA wallet demo polls `/latest_snapshot` because its interface needs current ledger balances. It does not reconstruct the wallet ledger from WebSocket messages. A production indexer can load the same kind of state snapshot once, then use the feed to keep its materialized view current. + +## Capacity and connection behavior + +The server limits subscriber count, catch-up events, and inbound frame size. The endpoint responds to WebSocket pings, while other inbound data is ignored because delivery is server to client. See [`GET /ws/subscribe`](../api-reference/api.md#get-wssubscribe) for the exact limits and close behavior. + +Run a small number of durable indexers against the sequencer and let user-facing applications read from those indexers. Connecting every browser directly can exhaust the subscriber limit and gives each browser the burden of replay, persistence, and rollback reconciliation. + +## Next steps + +- To create the user operations that appear in the feed, see [Submitting transactions](./submitting-operations.md). +- For the exact endpoint contract and close behavior, see [HTTP and WebSocket API](../api-reference/api.md). +- To initialize an indexer from application state, see [Snapshots and checkpoints](../recovery/snapshots.md). diff --git a/app-sequencer/usage/submitting-operations.md b/app-sequencer/usage/submitting-operations.md new file mode 100644 index 000000000..43963326e --- /dev/null +++ b/app-sequencer/usage/submitting-operations.md @@ -0,0 +1,121 @@ +--- +title: "Submitting transactions" +sidebar_label: "Submitting operations" +description: "How to construct, sign, submit, retry, and interpret an application transaction sent to the sequencer." +--- + +A client sends a user operation to the sequencer as EIP-712 typed data. The sequencer verifies the signature, applies the protocol and application checks, executes the operation against its predicted state, and stores accepted operations in order. + +## Submit a transaction with POST /tx + +Send a JSON request to: + +```text +POST /tx +Content-Type: application/json +``` + +The request contains the sender's next nonce, maximum fee, application payload, EIP-712 signature, and expected signer: + +```json +{ + "message": { + "nonce": 0, + "max_fee": 1100, + "data": "0x..." + }, + "signature": "0x...", + "sender": "0x..." +} +``` + +Use `0x`-prefixed hexadecimal values. The application defines the payload encoding and its maximum decoded size. See [`POST /tx`](../api-reference/api.md#post-tx) for the exact field types, limits, and response schema. + +## Sign the transaction with EIP-712 + +Sign the exact nonce, maximum fee, and payload sent in the request. The domain binds the signature to the deployment's chain and application contract, so a signature for another deployment is rejected. + +[EIP-712 domain](../api-reference/eip712.md) defines the exact domain and `UserOp` type. The [Quickstart](./quickstart.md#step-3-sign-and-submit-a-transaction) contains a complete ethers example. + +## Interpret the soft confirmation + +HTTP `200` returns: + +```json +{ + "ok": true, + "sender": "0x...", + "nonce": 0 +} +``` + +Before returning `200`, the sequencer has validated, executed, and committed the operation to its provisional ordering. The response is therefore stronger than queue admission, but it remains a soft confirmation. + +The response contains no feed offset, application output, batch number, frame number, or settlement status. Read the sequenced feed to discover its current ordering position. Feed appearance does not add a settlement guarantee because both responses come from the same off-chain sequencer. + +## Manage the sender nonce + +Nonces are maintained separately for each sender. A sender's first accepted operation uses nonce `0`; each accepted operation advances the expected value by one. + +Clients should: + +- store the next nonce for each sender; +- serialize submissions made by the same sender; +- advance the local nonce only after an HTTP `200` response; +- leave the nonce unchanged after a definite rejection; +- reconcile application state before deciding what to do after an ambiguous network failure. + +The public API does not provide a nonce lookup endpoint. Applications can expose their own read model, or clients can maintain the value from their accepted submission history. + +Concurrent requests with the same sender nonce race against one another. At most one can match the application's expected nonce after the earlier accepted operation advances it. + +### Handle an ambiguous timeout + +The endpoint waits for the storage commit before replying. A connection can still fail after the commit and before the client receives the response. In that case, the client cannot tell from the transport error whether the operation was accepted. + +Do not automatically advance the nonce or sign a different operation with the same nonce. First reconcile using the application's state, an application-level transaction identifier, or a corrected replay of the sequenced feed. Retrying the identical signed request is safe from double execution because the nonce prevents a second inclusion, but the retry may return `EXECUTION_REJECTED` if the first attempt already succeeded. + +![A signed submission can end in definite success, definite rejection, or an ambiguous transport failure. After an ambiguous failure, the client reconciles application state or the feed, treats confirmed evidence as acceptance, or retries the identical request and reconciles again if the outcome remains unknown.](../images/submission-timeout-recovery.jpg) + +## Set the maximum fee + +`max_fee` is a fee exponent, not a token amount. The sequencer rejects a value below the current frame price, while an accepted operation pays the committed frame price, which can be lower than the submitted maximum. + +The current recommended value is not exposed through a public endpoint. Obtain a baseline from the operator. If the value is too low, update it, sign the changed message again, and resubmit with the same nonce. [Fees and data availability](../concepts/fees.md) defines the exponent and charging model. + +## Handle submission errors + +Use the stable error `code` and HTTP status for control flow, not the human-readable message. Handle errors by outcome: + +- Correct definite request, signature, payload, fee, nonce, or application rejections before retrying. These responses do not consume the nonce. +- Retry overload and temporary unavailability with bounded backoff. +- Stop automatic retries and alert the operator for an internal error. +- Reconcile transport failures and lost responses because the operation may already have committed. + +The [`POST /tx` reference](../api-reference/api.md#post-tx) lists every status and error code. + +## Using the Rust client + +The `sequencer-rust-client` crate wraps submission and WebSocket connection setup: + +```rust +use sequencer_rust_client::SequencerClient; + +let client = SequencerClient::new("http://127.0.0.1:3000")?; +let response = client.submit_tx(&request).await?; +let stream = client.subscribe(from_offset).await?; +``` + +`SequencerClient::new` uses a three-second HTTP timeout. Use `new_with_timeout` or `with_request_timeout` to change it. + +`submit_tx` decodes a successful response and reports non-`200` responses as `SubmitRejected::Http`. Use `submit_tx_with_status` when the caller needs the raw HTTP status and response body for its own error-code handling. + +The current client constructor accepts `http://` endpoints. Deployments that terminate TLS at a gateway need to connect through an appropriate internal HTTP endpoint or use another client until HTTPS endpoint support is added. + +The Rust client opens the feed but leaves WebSocket message decoding, cursor persistence, reconnection, and catch-up recovery to the caller. See [Consuming the sequenced transaction feed](./reading-the-feed.md). + +## Next steps + +- Run the complete local flow in the [Quickstart](./quickstart.md). +- Consume accepted operations using [Consuming the sequenced transaction feed](./reading-the-feed.md). +- Check all public response shapes in [HTTP and WebSocket API](../api-reference/api.md). diff --git a/docusaurus.config.js b/docusaurus.config.js index b3edf0a81..a72b36cc2 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -209,6 +209,12 @@ const config = { activeBaseRegex: "^/fraud-proofs", position: "left", }, + { + label: "App Sequencer", + to: "/app-sequencer", + activeBaseRegex: "^/app-sequencer", + position: "left", + }, { type: "search", className: "navbar-search-custom", @@ -397,6 +403,14 @@ const config = { from: '/cartesi-rollups', // the old/base route to: '/cartesi-rollups/1.5/', // the new route to redirect to }, + { + from: "/app-sequencer/operations/configuration", + to: "/app-sequencer/operations/setup-and-running/", + }, + { + from: "/app-sequencer/operations/flush-mempool", + to: "/app-sequencer/operations/orchestration/", + }, ], }, ], @@ -452,6 +466,18 @@ const config = { docItemComponent: "@theme/ApiItem", }, ], + [ + "@docusaurus/plugin-content-docs", + { + id: "app-sequencer", + path: "app-sequencer", + routeBasePath: "app-sequencer", + sidebarPath: require.resolve("./sidebarsAppSequencer.js"), + editUrl: "https://github.com/cartesi/docs/tree/develop", + showLastUpdateTime: true, + docItemComponent: "@theme/ApiItem", + }, + ], ], themes: [ "docusaurus-theme-openapi-docs", diff --git a/get-started/index.mdx b/get-started/index.mdx index a27c3154c..fab12fc28 100644 --- a/get-started/index.mdx +++ b/get-started/index.mdx @@ -46,6 +46,17 @@ import DocCard from '@theme/DocCard'; +
+
+ +
+
+ ## Join the Cartesi Community diff --git a/sidebarsAppSequencer.js b/sidebarsAppSequencer.js new file mode 100644 index 000000000..e2b85e5ae --- /dev/null +++ b/sidebarsAppSequencer.js @@ -0,0 +1,175 @@ +module.exports = { + appSequencerSidebar: [ + { + type: 'category', + label: 'Overview', + collapsed: false, + link: { + type: 'generated-index', + title: 'Overview', + description: + 'Start here: what the App Sequencer is, and whether your application needs it.', + slug: '/', + }, + items: [ + { type: 'doc', id: 'overview/app-specific-sequencing', label: 'App-specific sequencing' }, + { type: 'doc', id: 'overview/when-to-use', label: 'When to use it' }, + ], + }, + { + type: 'category', + label: 'Foundations', + collapsed: false, + link: { + type: 'generated-index', + title: 'Foundations', + description: + 'The mental model: how the sequencer is put together, what it can and cannot do, and the vocabulary used throughout.', + slug: '/foundations', + }, + items: [ + { type: 'doc', id: 'foundations/architecture', label: 'Architecture at a glance' }, + { type: 'doc', id: 'foundations/trust-model', label: 'Trust model and guarantees' }, + { type: 'doc', id: 'foundations/glossary', label: 'App Sequencer glossary' }, + ], + }, + { + type: 'category', + label: 'Core Concepts', + collapsed: true, + link: { + type: 'generated-index', + title: 'Core Concepts', + description: + 'The mechanisms in detail: how operations become batches, reach L1, and stay in one agreed order.', + slug: '/concepts', + }, + items: [ + { type: 'doc', id: 'concepts/soft-confirmations', label: 'Soft confirmations' }, + { type: 'doc', id: 'concepts/batches-frames-safe-block', label: 'Batches, frames, and the safe block' }, + { type: 'doc', id: 'concepts/batch-tree', label: 'The batch tree' }, + { type: 'doc', id: 'concepts/direct-vs-sequenced', label: 'Direct inputs vs sequenced transactions' }, + { type: 'doc', id: 'concepts/staleness', label: 'Staleness and the danger zone' }, + { type: 'doc', id: 'concepts/fees', label: 'Fees and data availability' }, + { type: 'doc', id: 'concepts/execution-order', label: 'Deterministic execution order' }, + ], + }, + { + type: 'category', + label: 'API Reference', + collapsed: true, + link: { + type: 'generated-index', + title: 'API Reference', + description: + 'The interfaces a client talks to: the HTTP and WebSocket API, the typed-data domain used to sign transactions, and the fixed values and codes around them.', + slug: '/api-reference', + }, + items: [ + { type: 'doc', id: 'api-reference/api', label: 'HTTP and WebSocket API' }, + { type: 'doc', id: 'api-reference/eip712', label: 'EIP-712 domain' }, + { type: 'doc', id: 'api-reference/constants-and-exit-codes', label: 'Constants and exit codes' }, + ], + }, + { + type: 'category', + label: 'Usage Guide', + collapsed: true, + link: { + type: 'generated-index', + title: 'Usage Guide', + description: + 'Build against the sequencer: meet the requirements an application has to satisfy, stand one up, submit transactions, and read the ordered feed.', + slug: '/usage', + }, + items: [ + { type: 'doc', id: 'usage/quickstart', label: 'Quickstart' }, + { type: 'doc', id: 'usage/application-requirements', label: 'Application requirements' }, + { type: 'doc', id: 'usage/integration', label: 'Application integration' }, + { type: 'doc', id: 'usage/submitting-operations', label: 'Submitting operations' }, + { type: 'doc', id: 'usage/reading-the-feed', label: 'Reading the sequenced feed' }, + ], + }, + { + type: 'category', + label: 'Tutorials', + collapsed: true, + link: { + type: 'generated-index', + title: 'Tutorials', + description: + 'Build complete examples with the App Sequencer, from a local development environment to signed operations and feed consumption.', + slug: '/tutorials', + }, + items: [ + { type: 'doc', id: 'tutorials/build-wallet-sequencer', label: 'Build a sequenced wallet' }, + ], + }, + { + type: 'category', + label: 'Deployment & Operations', + collapsed: true, + link: { + type: 'generated-index', + title: 'Deployment & Operations', + description: + 'Run a sequencer in production: set it up, configure it, secure it, and monitor it.', + slug: '/operations', + }, + items: [ + { type: 'doc', id: 'operations/setup-and-running', label: 'Configure and run' }, + { type: 'doc', id: 'operations/orchestration', label: 'Supervision and recovery' }, + { type: 'doc', id: 'operations/data-and-state', label: 'Data and backups' }, + { type: 'doc', id: 'operations/monitoring', label: 'Monitoring and watchdog' }, + { type: 'doc', id: 'operations/security', label: 'Production security' }, + ], + }, + { + type: 'category', + label: 'Recovery and Resilience', + collapsed: true, + link: { + type: 'generated-index', + title: 'Recovery and Resilience', + description: + 'How the sequencer survives failures: what can go wrong, the two repair paths, and the checkpoints a rebuild depends on.', + slug: '/recovery', + }, + items: [ + { type: 'doc', id: 'recovery/failure-modes', label: 'Failure modes' }, + { type: 'doc', id: 'recovery/preemptive', label: 'Preemptive recovery' }, + { type: 'doc', id: 'recovery/cockroach', label: 'Cockroach recovery' }, + { type: 'doc', id: 'recovery/snapshots', label: 'Snapshots and checkpoints' }, + ], + }, + { + type: 'category', + label: 'Protocol and Internals', + collapsed: true, + link: { + type: 'generated-index', + title: 'Protocol and Internals', + description: + 'The deep design: the rules the sequencer mirrors on-chain, its invariants, and its formal guarantees.', + slug: '/advanced', + }, + items: [ + { type: 'doc', id: 'advanced/scheduler-semantics', label: 'Scheduler semantics' }, + { type: 'doc', id: 'advanced/divergence', label: 'Divergence detection' }, + { type: 'doc', id: 'advanced/invariants', label: 'Invariants' }, + { type: 'doc', id: 'advanced/threat-model', label: 'Threat model' }, + { type: 'doc', id: 'advanced/formal-verification', label: 'Formal verification' }, + ], + }, + { + type: 'doc', + id: 'troubleshooting', + label: 'Troubleshooting', + }, + { + type: 'link', + label: 'GitHub', + href: 'https://github.com/cartesi/sequencer', + }, + ], +};