diff --git a/Cargo.lock b/Cargo.lock index 2f5a4574e..751ad9fc0 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -814,6 +814,16 @@ dependencies = [ "windows-sys 0.59.0", ] +[[package]] +name = "fs4" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e72ed92b67c146290f88e9c89d60ca163ea417a446f61ffd7b72df3e7f1dfd5" +dependencies = [ + "rustix", + "windows-sys 0.61.2", +] + [[package]] name = "futures-core" version = "0.3.31" @@ -1573,6 +1583,7 @@ dependencies = [ "clap_mangen", "digest 0.11.3", "fluent-bundle", + "fs4", "glob", "googletest", "hashbrown 0.17.1", diff --git a/Cargo.toml b/Cargo.toml index bd5128e6f..6b6d607cd 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -99,6 +99,7 @@ serde-saphyr = "0.0.6" minijinja = { version = "2.12.0", features = ["loader"] } cap-primitives = "3.4.4" cap-std = { version = "3.4.4", features = ["fs_utf8"] } +fs4 = "1.1.0" camino = "1.2.0" semver = { version = "1", features = ["serde"] } anyhow = "1" diff --git a/docs/adr-011-use-ninja-dyndep-for-serial-dependency-ordering.md b/docs/adr-011-use-ninja-dyndep-for-serial-dependency-ordering.md new file mode 100644 index 000000000..afa9ea0ea --- /dev/null +++ b/docs/adr-011-use-ninja-dyndep-for-serial-dependency-ordering.md @@ -0,0 +1,133 @@ +# Architecture decision record (ADR): Use Ninja dyndep for serial dependency ordering + +## Status + +Accepted. + +## Date + +2026-08-11 + +## Context and problem statement + +Manifest authors need an explicit way to run the direct `deps` of an action or +target in declaration order. The existing Ninja dependency classes preserve +freshness and scheduling constraints, but ordinary implicit dependencies are +all visible to the scheduler and may run concurrently. + +The implementation must retain a single Ninja invocation so a shared dependency +runs once, propagate a failing early dependency to stop later work through the +annotated path, and leave unrelated branches available for normal concurrent +scheduling. It must also leave the Intermediate Representation (IR) +backend-agnostic and make generated Ninja output executable in every command +path. + +## Decision + +**Y-statement:** In the context of declaration-ordered direct dependencies, +and facing the forces of shared-work reuse, failure short-circuiting, +unrelated-branch concurrency, backend-neutral IR, and executable generated +output, we decided to use staged Ninja dyndep sidecars for +`dependency_order: serial`, accepting the Ninja 1.10 floor, reserved generated +state beneath `.netsuke`, and a path-scoped ordering guarantee. + +In the context of a `dependency_order: serial` manifest `deps` list, Netsuke +will use staged Ninja dyndep sidecars to reveal one direct dependency at a time +and will materialize those sidecars atomically beneath `.netsuke/dyndep`. The +runner materializes every sidecar file before Ninja starts; no Ninja edge +produces sidecar content. + +`dependency_order` is a closed `parallel`/`serial` enum on the shared action +and target AST shape. It is copied to `BuildEdge`, where it remains a logical +graph annotation. Only the Ninja generator lowers a serial list containing two +or more direct dependencies into synthetic phony gates beneath +`.netsuke/serial` and content-addressed dyndep sidecars beneath +`.netsuke/dyndep`. + +The gate edge associated with the next sidecar depends on the preceding gate, +so each later direct dependency remains unavailable until earlier work +succeeds. + +The main generated build file declares `ninja_required_version = 1.10` only +when staged serial lowering is present. The generator exposes a complete bundle +containing main-file text and every required sidecar. String-only generation +rejects a graph requiring sidecars instead of returning an incomplete file. + +The serial guarantee is deliberately path-scoped: each direct dependency in the +annotated list becomes schedulable only after its predecessor succeeds. A later +dependency independently reachable through another requested path remains free +to run through that other path. + +## Rationale + +- **One scheduler preserves shared work.** The generated gates stay inside one + Ninja invocation, so Ninja continues to deduplicate a repeated or diamond + dependency. +- **Dyndep controls visibility.** A later real dependency is absent from the + relevant graph path until its sidecar is revealed, unlike an order-only edge + whose transitive inputs are already visible to Ninja. +- **The IR remains portable.** Gates, sidecar paths, and Ninja version syntax + are backend mechanics rather than manifest graph concepts. +- **Bundle ownership prevents incomplete output.** Treating sidecars as part of + the generated artefact makes every runner path materialize them before Ninja + loads the main file. +- **Content addressing makes state reusable.** Existing matching sidecars are + safely reused; mismatching content is corruption and is reported rather than + overwritten. + +## Consequences + +- Serial lists with zero or one dependency use ordinary Ninja lowering; no + relative order needs enforcing and no dyndep version floor is emitted. +- User graph paths in outputs, inputs, implicit dependencies, and order-only + dependencies cannot use `.netsuke/serial` or `.netsuke/dyndep`, because those + names are reserved generated state. +- `build`, `clean`, and `generate` each materialize sidecars relative to the + effective Ninja working directory. `clean` may leave the immutable, + content-addressed sidecars in place. +- `src/ninja_gen/dyndep.rs` owns staging and naming. The command-boundary + module `src/runner/dyndep_publication.rs` opens the effective capability and + orchestrates publication and retention; `src/runner/process/dyndep_files.rs` + owns atomic sidecar writes and verification, while + `src/runner/process/dyndep_retention.rs` owns the lease and cleanup. Neither + side of the boundary may broaden the path-scoped guarantee with a global + scheduler. +- Tests must continue to use real Ninja for ordered starts, failure + short-circuiting, shared-work reuse, and unrelated-branch concurrency. + +## Alternatives considered + +### Order-only phony gate chain + +Rejected. Ninja eagerly schedules already-visible transitive inputs, so an +order-only chain can order gate completion without preventing later real +dependencies from starting early. + +### Ninja pool with depth one + +Rejected. A pool provides mutual exclusion, not declaration order, and would +serialize unrelated work outside the annotated dependency list. + +### Recursive Ninja or Netsuke invocation per dependency + +Rejected. Separate child schedulers lose the enclosing build's memoization and +can execute a shared dependency more than once. + +### A Netsuke-owned global scheduler + +Rejected for this feature. It would change the execution architecture and +global reachability semantics rather than implement the requested scoped +manifest policy. It requires a separately approved design. + +## Implementation references + +- Manifest and IR contract: [`src/ast/mod.rs`](../src/ast/mod.rs), + [`src/ir/graph.rs`](../src/ir/graph.rs), and + [`src/ir/from_manifest.rs`](../src/ir/from_manifest.rs) +- Ninja bundle generation: + [`src/ninja_gen/dyndep.rs`](../src/ninja_gen/dyndep.rs) +- Atomic sidecar materialization: + [`src/runner/process/dyndep_files.rs`](../src/runner/process/dyndep_files.rs) +- User contract: [user's guide](users-guide.md#run-direct-dependencies-serially) +- Implementation history: + [issue #552 ExecPlan](execplans/issue-552-support-serial-dependency-ordering-for-actions-and-targets.md) diff --git a/docs/adr-012-bound-dyndep-sidecar-retention.md b/docs/adr-012-bound-dyndep-sidecar-retention.md new file mode 100644 index 000000000..4d870d2e1 --- /dev/null +++ b/docs/adr-012-bound-dyndep-sidecar-retention.md @@ -0,0 +1,107 @@ +# Architecture decision record (ADR): Bound dyndep sidecar retention + +## Status + +Accepted. + +## Date + +2026-08-15 + +## Context and problem statement + +Serial dependency ordering uses immutable, content-addressed Ninja dyndep +sidecars beneath `.netsuke/dyndep`. A changed manifest therefore produces new +`.dd` files without changing or overwriting files that an existing generated +Ninja manifest references. Without a cleanup policy, obsolete sidecars would +accumulate indefinitely. Cleanup must also avoid removing a sidecar that a +concurrent Netsuke command is still consuming. + +The policy must preserve every sidecar in the current generated bundle, +remove stale temporary files left by interrupted atomic writes, and bound +obsolete storage deterministically. It must define the failure boundary for +`clean` and make the compatibility consequence of retaining an old +`generate --output` manifest explicit. + +## Decision + +Netsuke will retain immutable, content-addressed dyndep sidecars. Each +sidecar-capable `build`, `generate`, or `clean` command materializes every +sidecar in its current bundle before writing or invoking the generated Ninja +file. Publication and cleanup use a capability-scoped, exclusive lease for +the `.netsuke/dyndep` directory. The lease remains held through Ninja +consumption for `build` and `clean`, or through generated-output consumption +for `generate`. + +While that lease is held, Netsuke removes stale `.tmp` files and applies the +following deterministic policy to obsolete `.dd` files: + +- every sidecar in the current bundle is retained; +- at most 32 obsolete `.dd` files are retained; and +- at most 1 MiB of obsolete `.dd` bytes is retained. + +Obsolete files are considered in deterministic path order. A sidecar's +content is never changed in place. `build` and `generate` prune after +materialization. `clean` prunes only after `ninja -t clean` succeeds; a failed +clean does not prune sidecars. + +## Rationale + +- **Content addressing preserves active bundles.** A matching sidecar can be + reused and a mismatching file is corruption, not permission to overwrite + it. +- **The lease protects consumption.** Publication, temporary-file cleanup, + and pruning share one directory lease, so cleanup cannot remove files while + another serial command is using its bundle. +- **Fixed budgets are predictable.** File-count and byte limits provide a + bounded cache without relying on filesystem timestamps or an age-based + policy. +- **`clean` keeps failure evidence.** Deferring cleanup until successful + `ninja -t clean` avoids deleting historical state when the requested clean + did not complete. + +## Consequences + +An old arbitrary manifest written by `generate --output` may lose referenced +sidecars after a later Netsuke command applies retention. Such a manifest +must be regenerated before use when its sidecars have been pruned. Generated +manifests should therefore be treated as command outputs paired with the +current sidecar cache, not as permanently self-contained artefacts. + +The policy does not use sidecar age, and it does not make sidecars mutable. +There is no guarantee that an obsolete sidecar remains available merely +because its manifest was generated successfully in an earlier command. + +## Alternatives considered + +### Retain sidecars by age + +Rejected. Wall-clock age is not deterministic and does not bound storage. + +### Keep every content-addressed sidecar + +Rejected. Immutable files would accumulate without bound as manifests change. + +### Mutate or overwrite existing sidecars + +Rejected. A content-addressed path must continue to identify one byte +sequence, and overwriting it could change the graph seen by an existing +manifest. + +### Prune without a directory lease + +Rejected. Publication and cleanup could race with a command that is consuming +the current bundle, removing a required sidecar between materialization and +Ninja or output consumption. + +## Implementation references + +- Runner publication boundary: + [`src/runner/dyndep_publication.rs`](../src/runner/dyndep_publication.rs) +- Atomic sidecar materialization: + [`src/runner/process/dyndep_files.rs`](../src/runner/process/dyndep_files.rs) +- Retention and lease implementation: + [retention implementation](../src/runner/process/dyndep_retention.rs) +- User contract: [user's guide](users-guide.md#run-direct-dependencies-serially) +- Serial dyndep architecture: + [ADR-011](adr-011-use-ninja-dyndep-for-serial-dependency-ordering.md) diff --git a/docs/contents.md b/docs/contents.md index 9261ccb42..898d8c5d9 100644 --- a/docs/contents.md +++ b/docs/contents.md @@ -57,14 +57,20 @@ operator, user, and contributor references are easier to find. - [adr-010-scope-glob-capability-to-literal-prefix.md](adr-010-scope-glob-capability-to-literal-prefix.md): Glob capability-scoping decision record, opening the metadata capability at a pattern's literal directory prefix instead of an ambient root. +- [adr-011-use-ninja-dyndep-for-serial-dependency-ordering.md](adr-011-use-ninja-dyndep-for-serial-dependency-ordering.md): + Serial `deps` ordering decision record, covering staged Ninja dyndep bundles, + their scoped execution guarantee, and generated-state ownership. +- [ADR-012](adr-012-bound-dyndep-sidecar-retention.md): + Deterministic retention, lease, and failure-boundary policy for generated + dyndep sidecars. ## User and operator guides - [quickstart.md](quickstart.md): First-run walkthrough for building with Netsuke. - [v0-1-0-migration-guide.md](v0-1-0-migration-guide.md): Migration notes for - the v0.1.0 child-environment API additions and glob behaviour, and the - stability caveat that covers them. + the v0.1.0 child-environment API, glob behaviour, and serial-dependency + additions, plus the stability caveat that covers them. - [users-guide.md](users-guide.md): End-user reference for authoring and running Netsuke manifests, including executable discovery and `command_available` branch selection. diff --git a/docs/developers-guide.md b/docs/developers-guide.md index 856bec9b8..90fb9c1ab 100644 --- a/docs/developers-guide.md +++ b/docs/developers-guide.md @@ -285,7 +285,7 @@ The lowering stages have deliberately separate responsibilities: `$out` tokens are resolved per entry; tokens inside backticks are preserved. The resulting action contains ordinary command text and no Ninja placeholders. -- `src/ninja_gen.rs` emits a scalar command unchanged. For a list, it puts +- `src/ninja_gen/mod.rs` emits a scalar command unchanged. For a list, it puts each entry in a brace group and joins the groups with `&&`. Each group uses `eval` with a shell-quoted entry payload. This keeps an inline comment or a trailing control operator such as `&` inside the entry from consuming the @@ -598,19 +598,19 @@ no narrower — that covers the enumerated integration-test crates. The `test_support` crate uses capability-backed fixture helpers and remains linted by Whitaker under its own narrow policy. -`test_support` is a workspace member, but the root Whitaker invocation selects -only the `netsuke-build` package (the Cargo package name behind the `netsuke` -targets; see ADR-007) and disables Dylint dependency checks. It therefore -compiles `test_support` as a dependency without applying the root -`dylint.toml`. Its one sanctioned ambient boundary is configured per crate. -Workspace membership makes Dylint discover the root configuration even when -launched from `test_support/`, so the scoped recipe supplies the contents of -`test_support/dylint.toml` explicitly through `DYLINT_TOML`. The second pass -also uses `--package test_support` and `--no-deps`, because running from a -member directory alone would otherwise check the parent workspace. That -configuration names only `test_support::fs` in `excluded_paths`. The root -`excluded_crates` must not contain `test_support`: every other module in the -crate remains subject to the filesystem policy. +The root Whitaker invocation selects only the `netsuke-build` package (the +Cargo package name behind the `netsuke` targets; see ADR-007) and disables +Dylint dependency checks. It supplies the root `dylint.toml` contents +explicitly through `DYLINT_TOML`, so every invocation receives the same +capability-boundary policy regardless of how Dylint resolves the current +crate. `test_support` is a workspace member with one sanctioned ambient +boundary configured per crate. Its second, scoped invocation supplies +`test_support/dylint.toml` through `DYLINT_TOML`, and uses `--package +test_support` and `--no-deps`, because running from a member directory alone +would otherwise check the parent workspace. That configuration names only +`test_support::fs` in `excluded_paths`. The root `excluded_crates` must not +contain `test_support`: every other module in the crate remains subject to the +filesystem policy. Permanent exceptions belong in `dylint.toml`, scoped as narrowly as the lint allows. Do not use Rust `#[allow]` or `#[expect]` for `no_std_fs_operations`: @@ -1606,13 +1606,89 @@ unit tests where a small fixed set of cases must all be verified. manifest `deps` into `BuildEdge.implicit_deps`, and manifest `order_only_deps` into `BuildEdge.order_only_deps`. Keep those classes separate: recipe interpolation (`$in` and `{{ ins }}`) receives only `BuildEdge.inputs`, while -`src/ninja_gen.rs` renders implicit deps with Ninja's single-pipe separator. +`src/ninja_gen/mod.rs` renders implicit deps with Ninja's single-pipe separator. + +`ast::DependencyOrder` is the closed manifest enum responsible for YAML and +Serde. `src/ir/from_manifest.rs` explicitly converts it to the +serialization-free `ir::DependencyOrder` stored in +`BuildEdge::dependency_order`; both types have matching `Parallel` and `Serial` +variants, and `parallel` remains the default. The ordering policy applies only +to a manifest `deps` list; never infer it from the number or shape of graph +edges, and do not apply it to inputs or order-only dependencies. `src/ir/cycle.rs::CycleDetector::visit` traverses `inputs` and `implicit_deps` when detecting cycles. It intentionally does not traverse `order_only_deps`, because order-only dependencies express scheduling order rather than rebuild freshness. +### Serial dependency bundles + +`src/ninja_gen/dyndep.rs` owns the Ninja-specific lowering for a serial list +with more than one dependency. It produces a `GeneratedNinja` bundle: the main +build-file text plus immutable, content-addressed `GeneratedDyndep` sidecars. +The generated phony gates live under `.netsuke/serial`; sidecars live under +`.netsuke/dyndep`. Those are reserved graph namespaces, validated before +generation. User graph paths in outputs, inputs, implicit dependencies, and +order-only dependencies cannot use either namespace. A string-only generator +must return `DyndepFilesRequired` for a graph that needs sidecars rather than +returning an incomplete build file. + +Generated bundles need Ninja 1.10 or newer only when a serial direct-dependency +list has at least two items and therefore needs staged ordering; parallel lists +and serial lists with zero or one item retain the existing Ninja requirement. + +Each gate reveals one real dependency through a pre-materialized Ninja dyndep +file. The gate edge associated with the next sidecar depends on the preceding +gate, which keeps later direct dependencies unavailable to the scheduler until +earlier work succeeds. The runner materializes every sidecar file before Ninja +starts; no Ninja edge produces sidecar content. This is not an order-only +chain or a Ninja pool: both leave the real dependencies visible to Ninja too +early. Preserve one top-level Ninja invocation so shared nodes keep Ninja's +normal execute-once memoization. + +`GeneratedNinja` is the query-command boundary: generation may construct and +return it, but it must not publish any filesystem state. +`src/runner/dyndep_publication.rs` owns the `materialize_dyndep_bundle` +command, which every `build`, `clean`, and `generate` boundary must call before +writing or invoking the main file. That command opens the effective +working-directory capability and injects it into +`src/runner/process/dyndep_files.rs`, which owns atomic sidecar writes and +content verification. The materializer may only use that injected `Dir`; it +must not inspect CLI state or reopen ambient authority. It verifies existing +content, then uses a same-directory temporary file plus atomic rename. Keep +generated sidecars content-addressed and idempotent; corruption is an error, +not a reason to overwrite an unknown file. + +`src/runner/process/dyndep_retention.rs` owns the publication lease and +retention cleanup. The command-boundary module invokes it after materialization +or successful clean while retaining the lease through bundle consumption. + +`DyndepPublicationLease` also coordinates retention. Sidecar-capable `build`, +`generate`, and `clean` commands hold the capability-scoped exclusive +`.netsuke/dyndep` directory lease through Ninja or generated-output +consumption. While the lease is held, stale `.tmp` files are removed and +obsolete `.dd` files are retained in deterministic path order up to 32 files +and 1 MiB. The current bundle is always retained. `build` and `generate` +prune after materialization; `clean` prunes only after successful +`ninja -t clean`, never after a failed clean. Do not introduce age-based +cleanup or mutate an existing content-addressed sidecar. See +[ADR-012](adr-012-bound-dyndep-sidecar-retention.md) for the durable policy. + +`src/runner/dyndep_generation_telemetry.rs` owns runner-boundary generation +telemetry, and `src/runner/process/dyndep_telemetry.rs` owns publication +telemetry. They may wrap their respective boundaries with bounded +outcome-and-duration metrics and spans. Do not put manifest paths, action IDs, +sidecar names, or content in those fields; `src/ninja_gen` generation and +rendering must remain telemetry-free so their query responsibilities stay +explicit. + +The intended serial guarantee is path-scoped. A later dependency that is +independently reachable elsewhere in the requested graph may start via that +other path. Do not broaden the implementation with a global lock, pool, or +new scheduler without an approved design change. See +[ADR-011](adr-011-use-ninja-dyndep-for-serial-dependency-ordering.md) for the +durable decision and its alternatives. + ### Recipe placeholder ownership `src/ir/cmd_interpolate.rs` owns the private `INS_TOKEN` and `OUTS_TOKEN` diff --git a/docs/execplans/issue-552-support-serial-dependency-ordering-for-actions-and-targets.md b/docs/execplans/issue-552-support-serial-dependency-ordering-for-actions-and-targets.md new file mode 100644 index 000000000..0a2fac1f9 --- /dev/null +++ b/docs/execplans/issue-552-support-serial-dependency-ordering-for-actions-and-targets.md @@ -0,0 +1,1485 @@ +# Issue 552: Support serial dependency ordering with Ninja dyndep + +This ExecPlan is a living document. Keep `Progress`, `Surprises & discoveries`, +`Decision log`, and `Outcomes & retrospective` current as implementation +proceeds. + +Status: **Complete — coordinated, bounded dyndep retention passes deterministic +and focused validation.** + +Issue: [#552](https://github.com/leynos/netsuke/issues/552) + +## Purpose and big picture + +Netsuke currently treats every dependency list as an unordered graph. Users who +want an aggregate action such as `all` to run formatting, linting, tests, and +spelling in declaration order must encode orchestration outside the manifest. +The desired syntax is a closed `dependency_order` field on actions and targets: + +```yaml +actions: + - name: all + command: ":" + dependency_order: serial + deps: + - check-fmt + - lint + - test + - spelling +``` + +After this change, omitting the field continues to mean `parallel`. A serial +list starts each dependency only after the preceding dependency has completed +successfully, stops before later entries after a failure, and continues to use +one Ninja scheduler so shared work is built at most once per invocation. +Unrelated graph branches remain eligible to run concurrently. + +The observable implementation uses Ninja's dynamic dependency feature, +`dyndep`, introduced in Ninja 1.10. Netsuke emits a staged chain of phony gate +edges. Each stage's dyndep file reveals exactly one real dependency, while the +next dyndep file is not available to Ninja until the preceding gate succeeds. +This differs materially from an order-only chain around already-visible +dependencies: Ninja cannot schedule a dependency before its dyndep statement +has revealed it. + +The implementation is complete when schema, IR, Ninja generation, dyndep-file +materialization, documentation, and regression tests satisfy every acceptance +criterion in issue #552 and all repository gates pass. + +## Constraints + +- Use the declarative field `dependency_order`, with a closed enum containing + `parallel` and `serial`. Do not add a bare Boolean `serial` flag. +- Default to `parallel` so existing manifests and generated Ninja remain + unchanged. +- Apply the field only to a target or action's `deps` list. It must not reorder + `sources`, `order_only_deps`, the dependencies of another node, or the global + worker pool. +- Preserve the user's `deps` declaration order through parsing, IR lowering, + and Ninja generation. +- Keep one top-level Ninja invocation. Do not implement this feature through + nested `netsuke build` or nested `ninja` commands. +- Use Ninja dyndep syntax version 1 and require Ninja 1.10 or newer only when a + generated build uses serial dependency ordering. +- Preserve shared-dependency memoization within the encompassing Ninja build. +- Avoid Ninja pools: depth-one pools provide mutual exclusion, not a + declaration-order guarantee, and would broaden the serialization scope. +- Treat generated dyndep files as part of the generated Ninja artefact. Never + print a main Ninja file that silently relies on sidecars Netsuke has not + materialized. +- Keep generated paths deterministic, content-addressed where appropriate, + relative to the effective Ninja working directory, and isolated beneath the + existing `.netsuke` state namespace. +- Use `cap_std`, `cap_std::fs_utf8`, and `camino` for production file access; + do not introduce `std::fs` or `std::path` production code. +- Materialize sidecars atomically and idempotently so concurrent Netsuke + processes cannot observe partial dyndep content. +- Use the existing capability filesystem for all path access. The retention + lease may use the narrowly scoped `fs4` lock operation only because the + workspace capability APIs do not provide a cross-process advisory lock; + direct use of the pinned toolchain's `std::fs::File::lock` would bypass the + capability-filesystem policy. Record its exact version and maintenance basis. +- No production Rust source file may exceed 400 lines. In particular, + `src/ninja_gen/mod.rs` and `src/runner/process/mod.rs` are already near the + limit, so new responsibilities belong in focused submodules. +- Every new module must begin with a `//!` module-level comment, and every new + public API must have Rustdoc with a useful example. +- Follow Red–Green–Refactor. Capture the expected failing focused test before + making it pass, but never commit a state whose required gates fail. +- Update user, design, developer, repository-layout, roadmap, and architectural + decision documentation as described below. +- Run repository gates sequentially, using the shared Cargo cache. Do not run + formatting, linting, or test gates in parallel. +- Do not implement this draft until the user explicitly approves it. + +## Tolerances + +- **Scope:** if correct implementation requires a Netsuke-owned scheduler, + global graph serialization, or a manifest redesign beyond `dependency_order`, + stop and obtain approval. Those are architectural scope changes, not + implementation details. +- **Dependencies:** `fs4` is the sole added crate, declared as `fs4 = "1.1.0"` + in `Cargo.toml` and resolved to 1.1.0 in `Cargo.lock`. The lockfile records + its crates.io source and checksum; the published package metadata identifies + the upstream repository at and describes the + cross-platform file-lock implementation. This is the maintenance basis + available from the repository's dependency records; no stronger support or + release-cadence claim is inferred. Existing `cap_std::fs_utf8` APIs open the + lock file through the effective directory capability, but provide no + cross-process advisory-lock operation. Although the pinned toolchain's + `std::fs::File::lock` could provide the operation, direct `std::fs` use would + bypass the capability-filesystem policy. `fs4` operates only on that + already-opened handle, so it does not expand path authority. +- **Generated-output compatibility:** parallel manifests should retain + byte-for-byte generated Ninja output. If unavoidable output churn appears, + isolate it, explain it, and obtain approval before refreshing broad snapshots. +- **Public API compatibility:** existing callers of `ninja_gen::generate` and + `generate_into` must continue to work for ordinary graphs. A graph containing + serial ordering may require the new bundle API; the string-only APIs must + return a specific error instead of returning an incomplete build file. +- **Performance:** serial lowering may add one gate and one small dyndep file + per dependency. It must remain linear in the number of serial dependencies + and must not traverse unrelated subgraphs repeatedly. +- **State:** preserve every sidecar required by the current bundle, then bound + obsolete `.netsuke/dyndep` storage by fixed file-count and byte budgets. + Publication and cleanup share one capability-scoped per-directory lease until + the current command has finished consuming its bundle. +- **Platform support:** sidecar generation must be shell-independent and use + Rust filesystem APIs so Windows, macOS, and Linux release builds share the + same behaviour. +- **Compatibility boundary:** a later serial dependency that is also directly + requested or independently reachable from another requested top-level branch + may become visible through that other branch and execute early. If acceptance + requires suppressing that independent reachability, stop: static Ninja cannot + both keep ordering local and globally delay the shared node. +- **Time:** there is no deadline tolerance. Prefer a correct, reviewable design + and complete evidence over a rushed implementation. + +## Risks + +- **Later dependencies can leak through another requested branch.** Dyndep + delays only the graph path it controls. Ninja unifies nodes globally, so an + independent path to a later dependency can expose it before the serial gate. + Mitigation: document the boundary, test that a genuinely unrelated branch + remains concurrent, and stop for a scheduler-level redesign if stronger + semantics are required. +- **Incomplete generated artefacts.** A main Ninja file that references absent + dyndep sidecars fails during Ninja graph loading. Mitigation: introduce a + bundle type, route every CLI execution and generation path through it, and + make string-only generation reject serial graphs. +- **Races while writing sidecars.** Two builds may generate the same content at + once. Mitigation: write a unique temporary file in `.netsuke/dyndep`, flush + it, atomically rename it, and treat an already-present matching digest as + success. +- **Synthetic output collisions.** User targets could name a path chosen for a + gate or dyndep file. Mitigation: reserve `.netsuke/serial` and + `.netsuke/dyndep` for this feature, reject exact or prefix collisions with a + localized error, and document the reservation. +- **Freshness propagation can be lost.** Depending only on the final gate may + not make every real dependency contribute to the aggregate target's dirty + state. Mitigation: list every gate as an implicit dependency of the annotated + edge and add repeat-build tests that mutate each dependency in turn. +- **Escaping errors can corrupt dyndep syntax.** Ninja paths have escaping + rules distinct from YAML and shell quoting. Mitigation: reuse the generator's + existing path-rendering machinery, test spaces and Ninja metacharacters, and + avoid manually concatenating unescaped paths. +- **Action outputs are synthetic.** Actions and ordinary targets share the AST + `Target` shape but lower differently. Mitigation: exercise both forms at AST, + IR, snapshot, behavioural, and real-Ninja levels. +- **Ninja version failures may be obscure.** Older Ninja releases do not + support dyndep. Mitigation: emit `ninja_required_version = 1.10` when the + feature is present and document the resulting minimum version. +- **Source-file size pressure.** Adding logic directly to near-limit modules + would violate repository policy. Mitigation: establish focused dyndep + generation and materialization modules before adding the implementation. +- **Persistent cache growth.** Content-addressed names keep stale sidecars + harmless but let repeated manifest changes consume unbounded storage. + Mitigation: retain the current bundle plus a deterministic historical budget + while a publication lease protects active sidecars. + +## Progress + +- [x] (2026-08-10 11:59Z) Inspected issue #552, current AST-to-IR-to-Ninja + lowering, runner generation paths, behavioural fixtures, snapshots, and + real-Ninja integration tests. +- [x] (2026-08-10 11:59Z) Falsified the proposed order-only phony gate chain: + real dependencies remain transitively visible and start in parallel. +- [x] (2026-08-10 11:59Z) Falsified recursive per-dependency Ninja execution: + separate child schedulers rebuild a shared dependency more than once. +- [x] (2026-08-10 11:59Z) Validated a minimal dyndep chain with real Ninja: + declaration order, failure short-circuiting, shared work reuse, and unrelated + branch concurrency behaved as required. +- [x] (2026-08-10 11:59Z) Drafted this self-contained implementation plan. +- [x] (2026-08-10) User approved the plan and its compatibility boundary via the + implementation request; the developer also received the repository gate list. +- [x] (2026-08-10) Added schema and IR regressions: omission defaults to + `parallel`, explicit `parallel`/`serial` parse on targets and actions, an + unknown value such as `sequential` is rejected, and declaration order and the + `DependencyOrder` survive lowering for both targets and actions. +- [x] (2026-08-10) Implemented the AST and IR representation: the manifest AST + owns the Serde-enabled `DependencyOrder::{Parallel, Serial}` syntax, while + the IR owns a serialization-free domain enum with the same variants. + `from_manifest` converts explicitly between them and preserves dependency + declaration order. Updated every direct `BuildEdge` literal, doctest, and + fixture to use the domain type. `cargo check --all-targets` and 739 library + tests plus the touched integration tests passed. +- [x] (2026-08-10) Implemented deterministic Ninja bundle and dyndep lowering: + the new `src/ninja_gen/dyndep.rs` submodule adds `GeneratedNinja` (main text + plus content-addressed `GeneratedDyndep` sidecars) and `generate_bundle`. + Serial multi-dependency edges lower into one phony gate and sidecar per + dependency; the version floor is emitted only when gates exist; sidecars are + content-addressed beneath `.netsuke/dyndep`; and string-only generation + returns `NinjaGenError::DyndepFilesRequired` without writing partial output. + Added reserved-namespace collision errors and localization keys across all 35 + catalogues. Unit, integration, and doctests pass. +- [x] (2026-08-10) Implemented atomic dyndep sidecar materialization in + every CLI path. `src/runner/process/dyndep_files.rs` materializes the bundle + sidecars beneath `.netsuke/dyndep` relative to the effective Ninja working + directory, using capability-scoped writes, a same-directory `create_new` + temporary file, and an atomic rename. Existing content is verified and + reused; corruption and concurrent-writer outcomes are covered. + `generate_ninja` now routes every build, clean, and generate invocation + through `generate_bundle` plus materialization before invoking Ninja. Added + runtime tests driving real Ninja: strict declaration order, failure + short-circuiting, and materializer idempotence/corruption paths. Verified + end-to-end with a real serial manifest and real Ninja 1.11.1 (order observed: + fmt, lint, test, all). Full suite: 1930 tests pass. +- [x] (2026-08-11 19:43Z) Re-read this plan, the issue, current implementation + commits, the decision-record convention, and every documentation target named + in Stage 6 before beginning the documentation milestone. Confirmed that the + serial implementation uses the planned AST-to-IR-to-bundle-to-materializer + flow without a scheduler or new dependency. +- [x] (2026-08-11) Documented the manifest syntax and user-visible execution + contract in `docs/users-guide.md`, including the default, serial scope, + failure handling, shared-work behaviour, independent-reachability boundary, + Ninja 1.10 floor, generated sidecars, and reserved paths. Updated the + design, developer, repository-layout, contents, and roadmap documents and + added ADR-011 for the durable backend decision. +- [x] (2026-08-12) Ran the full deterministic suite. Formatting, type checking, + linting, Markdown linting, and Mermaid validation passed; the documentation + example loader rejected the new YAML fence because it lacked a + `tested-example` marker. Made the sample a complete manifest, registered its + stable identifier in the executable-documentation tests, and will rerun the + complete suite before committing. +- [x] (2026-08-12) Re-ran the complete deterministic suite after the + executable-example correction. `make check-fmt`, `make typecheck`, + `make lint`, `make test`, `make markdownlint`, and `make nixie` passed. + `make test` reported 1,939 passed tests, one skipped test, and passing + doctests. The canonical command-specific logs use the current branch suffix + beneath `/tmp`. +- [x] (2026-08-12) Committed the documentation and executable-example coverage + as `7153538` (`Document serial dependency ordering (#552)`). +- [x] (2026-08-12) Ran `coderabbit review --agent` on the committed milestone. + It completed successfully with zero actionable findings. The review log uses + the current branch suffix beneath `/tmp` and ends in `-2.out`. +- [x] (2026-08-12) Refactored the duplicated no-staging assertions in + `src/ninja_gen/dyndep_tests.rs` into the private + `assert_edge_produces_no_staging` helper without merging the distinct + parallel and one-element-serial tests. Committed as `df68f82`. The focused + dyndep module, `make check-fmt`, `make typecheck`, `make lint`, and + `make test` passed. +- [x] (2026-08-12) Linearized `write_atomic` in + `src/runner/process/dyndep_files.rs` by extracting temporary-file creation, + collision verification, write-and-sync, and rename-race helpers. Added the + matching-final-sidecar temporary-name-collision regression and committed the + change as `04e2369`. `make check-fmt`, `make typecheck`, `make lint`, and + `make test` all passed; the full test suite reported 1,940 passed tests, one + skipped test, and passing doctests. +- [x] (2026-08-12) Split `generate_bundle` edge processing into private + `render_edges`, `render_edge`, serial-edge, and display-edge helpers while + preserving sorted de-duplication, MissingAction construction, staged-gate + lowering, and emitted text. Committed as `ce47d61`. The focused dyndep tests + and `make check-fmt`, `make typecheck`, `make lint`, and `make test` passed + with no generated-output or snapshot changes. +- [x] (2026-08-12) Added the empty-sidecar fast path to + `materialize_dyndep_files` before capability opening and state-directory + creation, with a focused test proving it does not create `.netsuke/dyndep`. + The focused materializer suite passed. The first full lint run exposed that + the root Whitaker recipe did not pass the existing `dylint.toml` policy to + Dylint; its documented build-script and ambient-path exclusions therefore + appeared as false positives. The recipe now supplies `DYLINT_TOML` explicitly + for the root pass, matching the existing `test_support` pass. `make + check-fmt`, `make typecheck`, `make lint`, `make test`, `make markdownlint`, + and `make nixie` all passed afterwards. +- [x] (2026-08-12) Replaced the deterministic temporary-sidecar suffix with a + per-process, monotonic attempt name and retry `create_new` collisions. Keep + every name below the final sidecar parent so the capability-scoped rename + stays atomic, and pass the created path through rename-race cleanup rather + than regenerating it. Cover stale files, generated-name distinction, and an + existing matching final sidecar with another temporary file. The focused + suite passed 7 tests. The first full lint run identified a five-argument + helper and then a by-value context; grouping and borrowing the context + resolved both without changing the atomic protocol. `make check-fmt`, `make + typecheck`, `make lint`, `make test` (1,943 passed, 1 skipped, doctests + passed), `make markdownlint`, and `make nixie` all passed. CodeRabbit found + no concerns. +- [x] Committed each green logical change and recorded final evidence here. +- [x] (2026-08-12) Review follow-up: added a real-Ninja regression with two + serial consumers of one shared dependency and an unrelated branch. The test + proves the shared output executes once and uses a marker handshake to prove + that unrelated work progresses concurrently. Added the missing v0.1.0 + migration guidance and removed developer-specific workspace metadata from + this plan. All deterministic gates passed and CodeRabbit returned no + findings for this milestone. +- [x] (2026-08-12) Separated effect-free `GeneratedNinja` production from the + runner's explicit, capability-injected publication command. The command + handlers now consume the bundle after publication, removing the generated + main-string copy. Serial rendering uses a dependency view instead of cloning + `BuildEdge`, and materialization borrows sidecar paths. Added bounded spans, + outcome counters, and duration histograms at bundle generation, serial + lowering, and sidecar publication boundaries. The focused dyndep suite and + runtime Ninja suite passed; all deterministic gates passed (1,944 tests, one + skipped, and passing doctests), then CodeRabbit reported zero findings. +- [x] (2026-08-12) Verified the latest review findings against the current + branch. Stale findings were skipped because the absolute path is already + removed, the shared-work and unrelated-concurrency runtime test already + exists, and the generation/publication and clone issues are resolved. The + confirmed minimal fixes cover docs and locales, the render policy assertion, + reserved-path matrix and prefix boundary, bundle equivalence and shared gate + predicate/`write!`, pipe rejection, fixture-link and header cleanup, IR + dependency-order assertions, integration helpers, Ninja-not-found handling, + and materializer cleanup, bounded retries, directory scans, and localized + errors. Focused evidence: 21 dyndep tests, one render test, and 29 touched + integration/runtime tests passed. +- [x] (2026-08-12) Completed the review-fix validation. `make check-fmt`, + `make typecheck`, `make lint`, `make test`, `make markdownlint`, and `make + nixie` passed; the full suite reported 1,952 passed tests, one skipped test, + and passing doctests. CodeRabbit completed with zero findings. +- [x] (2026-08-12) Validated the second review remediation: control-character + path rejection, dependency-order module split, concurrent runtime-order + coverage, schema and migration links, and targeted locale corrections are + implemented. All 52 focused tests passed. `make check-fmt`, `make typecheck`, + `make lint`, `make test`, `make markdownlint`, and `make nixie` passed; the + full suite reported 1,970 passed tests, one skipped test, and passing + doctests. The subsequent independent CodeRabbit review reported zero + findings. +- [x] (2026-08-12) Closed the remaining serial-dyndep review gaps without + changing scheduling semantics. Public-CLI tests now prove runner-owned + sidecar publication, declaration order, and failure short-circuiting at + `-j 3`; generation telemetry lives at the runner boundary; four bounded + serial-lowering properties cover staging, order, repetition, content-address + invariants, and determinism; AST and IR dependency-order types are distinct; + and existing-sidecar verification is bounded at 16 MiB. All focused suites + passed. The six full deterministic gates passed with 1,979 tests, one skip, + and passing doctests, followed by a zero-finding CodeRabbit review. +- [x] (2026-08-14) Re-verified the documentation and locale review findings + against the current branch. Applied only the still-valid caption, bundle + signature, migration wording, Finnish state-expression, Polish path-label, + and developer-guide boundary corrections; the table contents and runtime + behaviour remain unchanged. +- [x] (2026-08-14) Refactored bounded sidecar verification into named open, + size-check, bounded-read, and outcome helpers. Added public-CLI coverage for + serial `generate` publication plus Ninja loading and serial `clean` + publication before Ninja dispatch. The 9 focused materializer tests and 4 + serial CLI tests passed. `make check-fmt`, `make typecheck`, `make lint`, + `make test`, `make markdownlint`, and `make nixie` passed; the full suite + reported 1,981 passed tests, one skipped test, and passing doctests. The + subsequent CodeRabbit review completed with zero actionable findings. +- [x] (2026-08-14) Added an accessible sequence diagram to the design document + showing runner-owned bundle generation, dyndep sidecar materialization, and + Ninja invocation. `make markdownlint` and `make nixie` passed. +- [x] (2026-08-14) Closed the latest review findings by escaping every graph + path in main Ninja edges and defaults, replacing the global temporary-name + sequence with a private operation-scoped source, and pinning publication + telemetry, growth detection, and bounded collision retries with focused + tests. Corrected the design flow and Ninja loading probe. The focused suites + passed 13 materializer, 49 generator, and 4 CLI tests. `make check-fmt`, + `make typecheck`, `make lint`, `make test`, `make markdownlint`, and `make + nixie` passed; the full suite reported 2,003 passed tests, one skipped test, + and passing doctests. CodeRabbit reviewed the committed milestone and reported + zero actionable findings across 75 changed files. +- [x] (2026-08-14) Rebased onto `origin/main`, preserving main's glob + capability documentation alongside the serial-ordering contract. Renumbered + the serial dyndep record to ADR-011 because main now owns ADR-010. `make + check-fmt`, `make test`, `make typecheck`, `make lint`, `make markdownlint`, + and `make nixie` passed; the test suite reported 2,068 passed tests, one + skipped test, and passing doctests. Lease-protected publication remains. +- [x] (2026-08-14) Re-verified the serial-dyndep sequence diagram against the + runner contract. The capability-scoped publication boundary is + `materialize_dyndep_bundle(cli, bundle)`; it opens the effective directory + before delegating sidecar writes, so the diagram now shows that boundary. +- [x] (2026-08-14) Removed the stale `RUNNER_IO_DYNDEP_RACE` key and its locale + entries after confirming that no runtime path references it. Concurrent + publication continues to use verified content outcomes and the live read, + rename, corruption, and bounded temporary-collision errors. +- [x] (2026-08-15) Re-verified the latest documentation review findings against + the current implementation. Corrected the reserved-namespace wording in the + ADR and developer guide to cover outputs, inputs, implicit dependencies, and + order-only dependencies. Corrected migration guidance so `generate` is + described as materializing sidecars while writing its manifest and `build` + and `clean` as materializing them before invoking Ninja. The + `docs/netsuke-design.md` gate description remains unchanged because it + already matches the current runner boundary and command-specific flow; that + review suggestion was stale. +- [x] (2026-08-15) Replaced the earlier no-pruning policy with a coordinated, + deterministic historical budget: the current bundle is preserved; at most 32 + obsolete sidecars and 1 MiB of obsolete sidecar bytes remain; and the + capability-scoped directory lease is held through the serial command's Ninja + or output-consumption boundary. Publication occurs before cleanup, while + `clean` cleans up only after successful `ninja -t clean`. +- [x] (2026-08-15) Added focused materialization coverage for retention count + and byte budgets, current sidecars, stale temporary cleanup, localized cleanup + failures, and bounded telemetry. Added public CLI coverage for repeated + generate, successful clean cleanup, failed-clean preservation, and latest + generated-manifest Ninja loading. The focused materializer (14 tests), + retention (6 tests), and serial CLI (7 tests) suites pass. +- [x] (2026-08-15) Validated the retention milestone: `make check-fmt`, + `make typecheck`, `make lint`, `make test`, `make markdownlint`, and + `make nixie` passed. The focused dyndep materializer (14), retention (6), + and serial CLI (7) suites also passed. The lock implementation uses `fs4` + only on a file already opened through the effective-directory capability, + satisfying the capability-filesystem lint without an exemption. +- [x] (2026-08-15) Re-verified the latest documentation review correction + against the generated-bundle and Ninja-version implementation: staged serial + ordering applies only to serial direct-dependency lists with at least two + items, so only those bundles require Ninja 1.10 or newer; parallel and + zero/one-item serial cases retain the existing requirement. The retention + warning was stale because bounded, lease-coordinated retention already + exists. +- [x] (2026-08-15) Replaced retention's unbounded historical-sidecar + collection and global sort with a lease-protected multi-pass lexical scan. + It retains at most the policy budget, deletes non-fitting candidates during + selection, and performs a final bounded-set sweep. The focused 1,000-file + regression proves that the current sidecar and the first fitting stale paths + survive while all other stale paths are reclaimed. +- [x] (2026-08-15) Simplified the retention telemetry assertion by moving the + typed snapshot traversals into private helpers in a focused nested test + module. This preserves the exact success-label, counter-kind, and reclaimed + total contract while keeping each test module below the 400-line limit. +- [x] (2026-08-15) Extracted the repeated retention-publication fixture from + the separate file-count and byte-budget tests. Both contracts retain their + original assertions while the shared setup remains fallible and + capability-scoped. +- [x] (2026-08-15) Refined bounded retention candidate selection so each + multi-pass scan carries only one lexicographic path and its byte length. The + 1,000-sidecar regression now proves that a sufficient budget retains exactly + the first two obsolete paths while preserving the current bundle sidecar. +- [x] (2026-08-15) Re-verified the reviewed code, runtime, and locale findings + against the current implementation and fixed the valid issues. The + regression covers 1,000 non-fitting sidecars. +- [x] (2026-08-16) Re-verified the documentation and locale requests against + the live module tree and runner ownership boundaries. Updated the AST and + Ninja module paths, clarified publication, atomic-write, and retention + ownership, and corrected the six requested catalogue messages without + changing Fluent placeholders. No supplied finding was stale. + +## Surprises and discoveries + +- (2026-08-12) The first serial-lowering property run correctly shrank to two + repeated dependencies, but exposed a test-oracle error rather than a + generator defect: the preceding gate constrains the next sidecar-producing + phony edge, not the gate edge that consumes that sidecar. The property now + asserts the actual staging relationship. The existing named repeated- + dependency unit test already pins the shrunk case, so no generated regression + seed was retained. +- (2026-08-12) The first full second-review test run found that the shared + shell-quoting BDD fixture still contained a newline-bearing output path. The + new generator validation correctly rejected the whole graph, preventing two + otherwise unrelated quoting scenarios from observing generated content. The + fixture now uses its existing apostrophe path as the edge case; IR-level + command interpolation coverage remains separate from the generator's stricter + Ninja path contract. +- (2026-08-15) A retention warning from the latest review was stale: obsolete + sidecars already have bounded count and byte budgets, and materialization, + cleanup, and command consumption already share a capability-scoped lease. +- (2026-08-15) The retention request was verified stale and contradictory to + accepted ADR-012 bounded automatic retention: the current bundle is + protected, while historical output may need regeneration. +- (2026-08-16) The requested sidecar ownership wording was valid but needed + one boundary split: `runner/dyndep_publication.rs` owns command-level + publication and retention orchestration, `runner/process/dyndep_files.rs` + owns atomic writes and verification, and `runner/process/dyndep_retention.rs` + owns the lease and cleanup. Generation remains an effect-free + `ninja_gen` query. +- (2026-08-11) The prior materializer commit accidentally left surplus blank + lines at EOF in each changed Fluent catalogue. `git show --check` reports + them even though the current worktree is clean. Remove only those trailing + blank lines in a preparatory cleanup before the next full validation run. +- (2026-08-11) The first fresh full-gate run stopped before review: typecheck + found an unused runtime-test helper, and Clippy found `expect` calls in + fallible bundle formatting plus three small idiom violations in the + materializer. `check-fmt`, Markdown linting, and Mermaid validation passed. + The correction remains within the approved implementation and needs no new + dependency or architecture. + +- (2026-08-10) Re-validated the staged dyndep chain with real Ninja 1.11.1: + declaration order holds when all sidecars are pre-materialized; a later + sidecar is revealed only after the preceding gate; failure of an early real + dependency stops later stages from being scheduled; and unrelated branches + remain available. The chain requires no generator recipe and no nested Ninja + process. +- (2026-08-10) Ninja path escaping in build/dyndep documents uses `$` as the + escape character. Spaces, `$`, `:`, `|`, and similar metacharacters in target + or dependency paths must use Ninja's dollar escape: a dollar sign followed by + a space for a literal space, `$$` for a literal dollar, `$:` for colon, and + `$|` for pipe. Unescaped spaces split a token into multiple paths. The + generator therefore needs a dedicated Ninja path-escape helper distinct from + the existing shell-script escaping. + +- (2026-08-10) Ninja resolves every path named in a build file — including a + `dyndep =` value and every path inside the referenced dyndep document — + relative to Ninja's process working directory, which is the `-C` directory + when one is supplied. The directory containing the main build file does not + affect path resolution. Confirmed with Ninja 1.11.1: with the main build file + in an OS temp directory and `-C` set to the user's project directory, a + sidecar written beneath `project/.netsuke/dyndep/` is located, loaded, and + its revealed dependency built correctly. The runner therefore needs no + architectural change; the plan's existing `.netsuke` navigation already + matches Ninja's model. +- A dyndep document updates the edge by naming the edge's *outputs*, not the + dyndep file itself. The first failed probes named the sidecar path in the + sidecar's `build` statement, which Ninja rejects with + `not mentioned in its dyndep file`. The accepted form is + `build : dyndep | `. + +- An order-only dependency on a phony gate orders only the gate itself. Ninja + eagerly schedules all already-visible transitive inputs, so the real recipes + behind later gates still start concurrently. This invalidates the original + phony-chain proposal even though the gate commands appear ordered. +- Giving every stage a real command that recursively invokes Ninja provides + ordering and failure propagation, but it breaks the shared-dependency + requirement because each child Ninja process owns a separate build memo. +- Dyndep changes the decisive property: the next real dependency is absent + from Ninja's graph until the preceding gate completes and makes the next + dyndep file available. +- Static, pre-materialized dyndep files can still be revealed in stages. A + phony edge may name each existing sidecar as its output and depend on the + previous gate. No generator recipe or nested process is required. +- Ninja's `rspfile_content` binding cannot conveniently encode the required + multiline dyndep document. A literal `\n` remains literal and produces an + invalid file. Netsuke therefore needs to materialize sidecars itself. +- `src/ninja_gen/mod.rs` was 400 lines before the module split, and + `src/runner/process/mod.rs` is close to that limit. The implementation must + be modular rather than appended to those files. +- Netsuke already owns the `.netsuke` workspace-state namespace through its + fetch cache, so `.netsuke/dyndep` does not introduce a second state root. +- (2026-08-12) The initial real-Ninja tests established order and failure but + did not encode the plan's independently observed shared-work and unrelated + concurrency behaviour. The review correctly treats both as separate runtime + contracts, so the follow-up test uses a bounded marker handshake rather than + a timing-only assertion. +- (2026-08-12) `GeneratedNinja` already had a consuming `into_parts` API, so + command handlers can move the generated main string only after sidecar + publication succeeds. An edge-display dependency view similarly removes the + need to clone a complete `BuildEdge` during serial lowering. + +## Decision log + +- **Decision:** retain the existing AST, IR, Ninja generation, runner, and + localization implementation commits as the reviewed functional baseline, then + repair only their trailing-catalogue-whitespace defect before full gates. + **Rationale:** the defect does not alter Fluent messages or behaviour, but a + clean diff is required before the first post-implementation review. **Date:** + 2026-08-11. +- **Decision:** propagate `fmt::Error` through the existing + `NinjaGenError::Format` conversion instead of asserting that `String` + formatting cannot fail. **Rationale:** this keeps the generator's public + error contract intact and satisfies the repository's no-`expect` policy + without adding an abstraction. **Date:** 2026-08-11. +- **Decision:** keep `generate_bundle` as a read-only query and move all + sidecar publication to explicit runner command boundaries. **Rationale:** + generation must be usable without a filesystem effect; accepting a + capability-scoped directory at the materializer makes the publication + authority explicit and testable. **Date:** 2026-08-12. +- **Decision:** keep generation telemetry in the runner boundary's + `src/runner/dyndep_generation_telemetry.rs` and publication telemetry in + `src/runner/process/dyndep_telemetry.rs`; keep `src/ninja_gen` generation and + rendering telemetry-free. **Rationale:** the separate wrappers make query + and command policy explicit while restricting fields to bounded counts and + outcome categories. **Date:** 2026-08-12. +- **Decision:** reject an existing dyndep sidecar larger than 16 MiB before + allocating verification storage. **Rationale:** generated sidecars are small + Ninja fragments, so this conservative ceiling supports large manifests while + preventing an untrusted existing file from causing unbounded memory use. A + metadata-sized buffer and one-byte growth probe also bound reads if the file + changes during verification. **Date:** 2026-08-12. +- **Decision:** replace unbounded historical sidecar retention with a fixed + deterministic budget of 32 obsolete `.dd` files and 1 MiB of obsolete `.dd` + bytes, protected by one advisory lease opened through the effective directory + capability. **Rationale:** immutable digest paths prevent corruption but not + storage growth across changed manifests. The lease protects publication, + cleanup, and current-bundle consumption; its deliberate cost is that serial + commands in one working directory wait for each other. **Date:** 2026-08-15. +- **Decision:** choose retained obsolete sidecars with a multi-pass, + lexicographic directory scan instead of collecting and sorting every stale + entry. **Rationale:** the fixed retention policy bounds the retained set, so + using only that set plus one cursor prevents a large historical cache from + increasing process memory while preserving deterministic path order. + **Date:** 2026-08-15. +- **Decision:** preserve exact lexicographic greedy byte-budget semantics with + bounded multi-pass scans; an arbitrary one-pass `read_dir` scan cannot + implement that policy without changing semantics. Each pass removes + candidates larger than the remaining budget, limiting scans to the fixed + file-cap selection passes plus final cleanup. **Date:** 2026-08-15. + +- **Decision:** use staged Ninja dyndep files rather than an order-only gate + chain, a pool, or recursive builds. **Rationale:** it is the only evaluated + design that keeps a single scheduler, prevents later dependencies from + becoming schedulable through the annotated path, propagates failure, and + leaves unrelated work unconstrained. **Date:** 2026-08-10. +- **Decision:** give the manifest AST and domain IR distinct + `DependencyOrder::{Parallel, Serial}` enums, converting explicitly while + lowering each `BuildEdge`. **Rationale:** the closed AST enum owns YAML and + Serde policy, while the IR enum remains backend-agnostic and free of syntax + responsibilities; both avoid inferring scheduling policy from graph shape. + **Date:** 2026-08-12. +- **Decision:** apply `dependency_order` only to `Target::deps`. + **Rationale:** actions already use the target shape, while sources and + order-only dependencies have distinct freshness semantics not covered by the + issue. **Date:** 2026-08-10. +- **Decision:** keep documentation terminology aligned with the implementation + boundary: `ast::DependencyOrder` owns manifest serialization, while + `ir::DependencyOrder` is the serialization-free domain type explicitly + produced during lowering. **Rationale:** this makes the developer guide and + design diagram describe the fallible `GeneratedNinja` bundle contract without + changing the serial dependency contract. **Date:** 2026-08-14. +- **Decision:** preserve ordinary dependency nodes in IR and perform + Ninja-specific staged lowering in the Ninja generator. **Rationale:** dyndep + gates are a backend mechanism, not a user graph concept; keeping them out of + IR preserves cycle diagnostics and other backends' view of the manifest. + **Date:** 2026-08-10. +- **Decision:** introduce a generated bundle containing the main Ninja text and + zero or more dyndep sidecars. **Rationale:** string-only generation cannot + represent the complete executable artefact. A bundle makes omission of + required sidecars difficult. **Date:** 2026-08-10. +- **Decision:** store immutable, content-addressed sidecars beneath + `.netsuke/dyndep`, and gates beneath `.netsuke/serial`. **Rationale:** + deterministic names make generation reproducible, reuse safe, and stale cache + entries harmless. **Date:** 2026-08-10. +- **Decision:** renumber the serial dyndep record from ADR-010 to ADR-011 after + rebasing. **Rationale:** main introduced the glob capability decision as + ADR-010 after this branch diverged; preserving both decisions under distinct + numbers keeps the repository index and cross-references unambiguous. **Date:** + 2026-08-14. +- **Decision:** generate temporary sidecar candidates from a private, + operation-scoped nonce and local retry sequence. This source is owned only by + dyndep publication and is injectable solely for focused collision tests; + `create_new` remains the authority for each candidate. **Rationale:** this + preserves same-directory atomic writes and bounded collision retries without + process-global mutable state. **Date:** 2026-08-14. +- **Decision:** validate the whole graph before rendering, then use one shared + infallible escaping helper for every user-controlled path in main edges and + default statements. **Rationale:** all supported Ninja metacharacters must be + encoded consistently, while unsupported controls retain the existing + localized validation error. **Date:** 2026-08-14. +- **Decision:** generate no dyndep chain for zero- or one-element serial lists. + **Rationale:** no relative ordering exists to enforce, so ordinary lowering + is equivalent and avoids unnecessary generated state. **Date:** 2026-08-10. +- **Decision:** list all generated gates, in order, as implicit dependencies of + the annotated edge. **Rationale:** every real dependency must continue to + participate in dirty checking; relying only on the final gate obscures that + invariant. **Date:** 2026-08-10. +- **Decision:** document independent reachability as a semantic boundary rather + than globally constraining shared nodes. **Rationale:** global delay would + serialize unrelated branches and violate the scoped-behaviour acceptance + criterion. Stronger semantics require a Netsuke scheduler and explicit + approval. **Date:** 2026-08-10. +- **Decision:** record the architecture in a new ADR before calling the feature + complete. **Rationale:** the Ninja version floor, generated sidecars, state + namespace, and public generator contract are durable choices that are costly + to reverse. **Date:** 2026-08-10. + +## Outcomes and retrospective + +The implementation has delivered the planned closed schema enum, backend-only +staged dyndep lowering, complete generated bundle, capability-scoped atomic +sidecar materialization, and the user-facing and maintainer documentation. The +documentation makes the intentionally limited path-scoped execution guarantee +explicit rather than implying global serialization. The complete deterministic +suite passed after the executable documentation sample was registered, and +CodeRabbit found no actionable concerns on `7153538`. The feature is ready for +the draft pull request and normal reviewer evaluation. + +## Context and orientation + +The relevant pipeline is deliberately small: + +```plaintext +Netsukefile YAML + -> src/ast/mod.rs Target + -> src/ir/from_manifest.rs process_targets + -> src/ir/graph.rs BuildEdge + -> src/ninja_gen/mod.rs generated Ninja artefact + -> src/runner/* materialization and Ninja invocation +``` + +In this plan, a *real dependency* is the action or target named by a manifest +`deps` entry. A *gate* is a synthetic phony Ninja output representing one +position in a serial list. A *dyndep sidecar* is a small Ninja-syntax document +that adds one real dependency to one gate after Ninja has loaded the main build +file. A *bundle* is the main build-file text plus every sidecar required to +execute it. + +The syntax and graph-loading constraints used below follow the official +[Ninja dyndep reference](https://ninja-build.org/manual.html#ref_dyndep). In +particular, the main edge names its dyndep file as an input and each sidecar +contains the version header plus a one-to-one update for that edge. + +`src/ast/mod.rs` defines `Target`, which is shared by ordinary targets and actions. +Its `deps` vector is already ordered by YAML declaration. Add a serde-backed +enum here rather than representing ordering as a string or Boolean. + +`src/ir/from_manifest.rs::process_targets` currently transfers `Target::deps` to +`BuildEdge::implicit_deps`. Keep the vector unchanged and copy the new enum to +the edge. Existing cycle detection continues to inspect the real dependency +graph rather than generated gates. + +`src/ir/graph.rs` defines `BuildEdge`. The field belongs here because +generation must know whether the edge's implicit dependencies are ordered. Many +tests and Rustdoc examples construct `BuildEdge` directly; update every literal +mechanically and default it to parallel. + +`src/ninja_gen/mod.rs` currently renders a graph to one string. Extract dyndep +identifier, sidecar, and staged-edge construction into +`src/ninja_gen/dyndep.rs`. Keep the top-level module responsible for ordinary +rendering and selecting the staged representation. + +`src/runner/mod.rs` and `src/runner/process/mod.rs` connect generation to +`build`, `clean`, and `generate`. Add sidecar materialization in a new focused +module such as `src/runner/process/dyndep_files.rs`; do not let a caller invoke +Ninja with a serial main file until its bundle is materialized. + +The principal existing tests are: + +- `tests/ast_tests/parsing.rs` and `tests/ast_tests/actions.rs` for manifest + syntax; +- `tests/ir_from_manifest_tests.rs` for dependency lowering; +- `tests/ninja_snapshot_tests.rs` for stable generated Ninja; +- `tests/ninja_gen_integration_tests.rs` for real Ninja execution; +- `tests/features/ninja.feature` and `tests/bdd/steps/ninja.rs` for externally + described generation behaviour; and +- `test_support/src/ninja_gen.rs` for shared generator fixtures. + +The current real-Ninja no-op test intentionally runs Ninja once to populate +`.ninja_log` before asserting that a second invocation is a no-op. Preserve +that pattern in serial freshness tests so the result reflects Ninja's normal +incremental state rather than a cold build. + +## Proposed generated form + +For a target `all` whose serial dependencies are `check-fmt`, `lint`, and +`test`, generate deterministic paths represented schematically below. The +actual identifiers use stable digests and escaped Ninja paths. + +```ninja +ninja_required_version = 1.10 + +build .netsuke/dyndep/.dd: phony +build .netsuke/serial//000: phony || .netsuke/dyndep/.dd + dyndep = .netsuke/dyndep/.dd + +build .netsuke/dyndep/.dd: phony .netsuke/serial//000 +build .netsuke/serial//001: phony || .netsuke/dyndep/.dd + dyndep = .netsuke/dyndep/.dd + +build .netsuke/dyndep/.dd: phony .netsuke/serial//001 +build .netsuke/serial//002: phony || .netsuke/dyndep/.dd + dyndep = .netsuke/dyndep/.dd + +build all: | .netsuke/serial//000 $ + .netsuke/serial//001 .netsuke/serial//002 +``` + +The first sidecar contains: + +```ninja +ninja_dyndep_version = 1 +build .netsuke/serial//000: dyndep | check-fmt +``` + +The later sidecars have the same shape for `lint` and `test`. Ninja can load +the first sidecar immediately and therefore schedule `check-fmt`. The second +sidecar's phony-producing edge depends on the first gate, so `lint` remains +unknown through this path until `check-fmt` succeeds. If `check-fmt` fails, the +first gate never completes, the second sidecar never becomes available, and +later dependencies are not scheduled through the serial list. + +Each gate is a phony alias of exactly one real dependency. Repeated or diamond +dependencies still name the same real Ninja node, so the single Ninja scheduler +executes that node at most once. + +## Plan of work + +### Stage 1: Establish red behavioural contracts + +Add parser tests for targets and actions covering omitted ordering, explicit +`parallel`, explicit `serial`, and rejection of an unknown value such as +`sequential`. Add IR tests showing that dependency order and the original +`implicit_deps` sequence survive lowering. + +Add generator tests that describe a complete bundle rather than only the main +string. The first red assertion should require: + +- `ninja_required_version = 1.10` only for a multi-dependency serial edge; +- one deterministic gate and sidecar per dependency; +- the gate edge associated with each later sidecar to depend explicitly on the + previous gate; +- one dyndep statement per gate with the matching real dependency; +- every gate to appear on the annotated edge in declaration order; and +- ordinary parallel snapshots to remain unchanged. + +Add a Gherkin scenario and fixture at `tests/data/dependency_order_serial.yml`. +The scenario should compile a target and an action to IR, generate a bundle, +and inspect the ordered dependency names revealed by its sidecars. The feature +text should describe user behaviour, not implementation internals beyond the +fact that valid staged dyndep output is generated. + +Capture the failing focused commands and their failure messages in the +`Progress` section. Then implement enough of stages 2–4 to make the tests green +before committing. + +### Stage 2: Add the AST and IR contract + +In `src/ast/mod.rs`, add the closed enum and target field. The intended public +shape is: + +```rust +#[derive(Clone, Copy, Debug, Default, Eq, PartialEq, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum DependencyOrder { + #[default] + Parallel, + Serial, +} + +pub struct Target { + // Existing fields remain in their current order. + #[serde(default)] + pub dependency_order: DependencyOrder, +} +``` + +Adjust derives to match the surrounding AST types and add Rustdoc examples +showing the default and serial forms. Do not add this field to `Rule`. + +In `src/ir/graph.rs`, define a domain-only `DependencyOrder` with no Serde +derives or attributes and add it to `BuildEdge`. In `src/ir/from_manifest.rs`, +convert the parsed AST value explicitly while leaving `implicit_deps` in source +order. Update every direct `BuildEdge` construction, fixture, and doctest to use +the IR type explicitly or use a shared test constructor where one already +exists. Do not introduce a new general-purpose builder solely to conceal +updates. + +Run the parser and IR tests. Also run existing cycle tests to prove the new +field does not change graph validation. + +### Stage 3: Generate a complete Ninja bundle + +Before extracting any helper, repeat the repository sweep for equivalent +bundle, digest, path-escape, and sidecar abstractions. Record the ownership and +reuse boundary in `docs/developers-guide.md`: the new types belong to Ninja +generation and may be consumed by runner/output adapters, but must not become +generic manifest or filesystem abstractions. + +Add types along these lines, refining names to fit existing conventions: + +```rust +pub struct GeneratedNinja { + build_file: String, + dyndep_files: Vec, +} + +pub struct GeneratedDyndep { + relative_path: Utf8PathBuf, + content: String, +} + +pub fn generate_bundle(graph: &BuildGraph) -> Result; +``` + +Expose read-only accessors or consuming methods needed by the CLI. Keep fields +private if callers do not need to construct invalid bundles. Document that all +paths are relative to the effective Ninja working directory. + +Keep `generate` and `generate_into` source-compatible for graphs without serial +ordering. If called with a multi-dependency serial edge, return a localized +`NinjaGenError::DyndepFilesRequired` directing the caller to `generate_bundle`; +never return main text that cannot run alone. Perform this check before writing +any bytes so `generate_into` cannot leave partial output in its caller's writer. + +Implement the staged lowering in `src/ninja_gen/dyndep.rs`: + +1. Iterate serial `implicit_deps` in their stored order. +2. Derive a stable parent identity from the annotated edge's canonical output + identity and an explicit schema/version tag. +3. Derive a gate path from that identity and a zero-padded position. +4. Render the sidecar content using the generator's existing Ninja path + escaping. +5. Derive the `.dd` filename from a cryptographic digest of the complete + sidecar bytes and a format-version tag. +6. Emit a phony sidecar edge. Starting with the second stage, make it explicitly + depend on the prior gate. +7. Emit a phony gate edge with the sidecar as an order-only input and its + `dyndep` binding. +8. Replace the annotated edge's direct implicit dependencies with all gate + outputs in the same order. + +Deduplicate identical sidecar content in the bundle by relative path, but do +not collapse gate positions. This preserves the declaration sequence while +allowing the real Ninja node to remain shared. + +Return a localized collision error when a user output occupies the reserved +`.netsuke/serial` or `.netsuke/dyndep` namespace. Add unit coverage for the +error and documentation for the reservation. + +Do not emit the Ninja version binding or any generated files for parallel, +empty serial, or one-element serial lists. Add tests for all three cases. + +### Stage 4: Materialize dyndep files atomically + +Extend the internal generated-content wrapper used by +`src/runner/process/mod.rs` so it carries `GeneratedNinja`, while preserving +the current main-text access required by stdout and JSON output. + +Create `src/runner/process/dyndep_files.rs`. Its single responsibility is to +materialize the bundle's sidecars under the effective Ninja working directory. +Its algorithm should be: + +1. Open the effective working directory through the existing capability-based + filesystem seam. Honour CLI `-C`; otherwise use the current directory. +2. Create `.netsuke/dyndep` if it is absent. +3. If a final content-addressed file exists, read it and verify its content. + Matching content is success; mismatched content is a corruption error. +4. Otherwise create a unique same-directory temporary file with `create_new`, + write all bytes, flush them, and rename it atomically to the final path. +5. If another process wins the rename race, verify the winning file and treat + matching content as success. +6. Clean up only the temporary file owned by this attempt. Never truncate or + replace an existing final sidecar in place. + +Use a narrow injected filesystem seam only if required for deterministic unit +tests; first reuse the runner's existing capability abstractions. Test initial +creation, idempotent reuse, corrupt-content rejection, nested-directory +creation, and the competing-writer outcome without mutating process-wide +environment variables. + +Route all executable and export paths through bundle materialization: + +- `netsuke build` materializes before writing or invoking the main Ninja file; +- `netsuke clean` does the same because loading a serial build file requires + its dyndep inputs even when cleaning; +- `netsuke generate --output ` materializes relative to the effective + Ninja working directory and writes the main file; +- `netsuke generate --stdout` and JSON output also materialize sidecars, then + return only the main file text in their existing output field. + +Document the side effect of stdout/JSON generation. If analysis shows that an +output path outside the working directory makes relative sidecars ambiguous, +retain the working-directory rule rather than inferring a new base from the +output filename. + +### Stage 5: Prove runtime semantics with real Ninja + +Create a focused integration-test module, splitting existing files if needed to +remain below 400 lines. Use actual Ninja processes and filesystem markers, not +assertions over textual edge order alone. + +Add these cases: + +- **Declaration order:** dependency one writes marker `one`; dependency two + first requires `one`, then writes `two`; dependency three requires `two`. The + aggregate succeeds only if recipes start in order. +- **Shared dependency:** the first and second serial entries both depend on a + real `common` node that appends one log entry. Assert exactly one entry for + `common` in one Ninja invocation. +- **Literal repeated entry:** use the same dependency twice in one serial list + and prove its recipe runs once while both gate stages complete. +- **Failure short-circuit:** make the first dependency fail deliberately and + assert no marker or log line exists for later dependencies. +- **Default parallel behaviour:** with `dependency_order` omitted, have two + dependencies create start markers and wait with a bounded timeout for the + peer marker. They succeed only if Ninja may run them concurrently. +- **Scoped serialization:** make the first serial dependency and an unrelated + requested branch meet at a bounded barrier. Assert both start concurrently, + while the second serial dependency begins only after the first finishes. +- **Incremental freshness:** perform a cold build, then a no-op build. Mutate + each real serial dependency in turn and assert the aggregate rebuilds. Finish + with another true no-op invocation after `.ninja_log` has been populated. +- **Path escaping:** use dependency and target paths containing spaces and a + Ninja metacharacter, and prove both main and dyndep files load correctly. + +Avoid timing-only assertions. Markers and bounded handshakes should establish +happens-before and concurrency. A timeout is only a deadlock guard and should +be generous enough for loaded CI workers. + +The scoped test intentionally uses an unrelated branch. Do not add a test that +claims a later dependency independently exposed by another branch will remain +hidden; that is outside the stated compatibility boundary. + +### Stage 6: Document the feature and its architecture + +Create `docs/adr-011-use-ninja-dyndep-for-serial-dependency-ordering.md` using +the repository ADR format. Verify the next ADR number immediately before +creation. The ADR must include a Y-statement and record: + +- the rejected order-only gate, pool, and recursive-build alternatives; +- the Ninja 1.10 version floor for serial builds; +- the generated bundle and `.netsuke` namespace; +- the single-scheduler shared-dependency property; +- the independent-reachability boundary; and +- consequences for generated-output consumers and cache cleanup. + +Mark the ADR `Proposed` while implementation is in progress and `Accepted` only +after behaviour and gates pass. Add it to `docs/contents.md`. + +Update: + +- `docs/users-guide.md` with action and target syntax, defaulting, ordered + execution, failure, shared work, scope, Ninja version, generated state, and + the independent-reachability boundary; +- `docs/netsuke-design.md` with the AST/IR policy, generated bundle, staged + dyndep graph, and why gates remain backend-only; +- `docs/developers-guide.md` with bundle ownership, materialization invariants, + reserved paths, and focused-test guidance; +- `docs/repository-layout.md` with the new generator/runner submodules and + `.netsuke/dyndep` state; +- `docs/roadmap.md` with issue #552 status, marking it complete only after all + evidence is present; and +- relevant Rustdoc on `DependencyOrder`, bundle APIs, errors, and the + materializer. + +Use en-GB-oxendict prose, 80-column paragraphs, attributed code fences, and the +documentation style guide. Run `make fmt` after documentation edits and inspect +the diff so unrelated mechanical reflow is not included. + +### Stage 7: Refactor, validate, and commit + +Once the feature is green, review the changed code and its neighbours for +duplication, long functions, excessive parameters, complex conditionals, and +feature envy. Any non-essential refactor belongs in a separate subsequent +commit and must pass the same gates. Do not broaden the feature commit merely +to tidy unrelated code. + +Use small green commits. A suitable sequence is: + +1. add the AST/IR contract and its tests; +2. add Ninja bundle generation, materialization, and semantic regressions; +3. document the syntax and architectural decision; and +4. apply a separate focused refactor only if post-commit review justifies one. + +Before every commit, run the relevant focused tests and gates. Before declaring +the work complete, run all project gates sequentially through the repository's +gate-running workflow: + +```bash +make check-fmt +make typecheck +make lint +make test +make markdownlint +make nixie +``` + +Capture each command with `tee` to a branch-specific file under `/tmp`, inspect +the complete log on failure, and record the final result and log path in +`Progress`. Do not substitute a narrower command for a named project gate. + +## Concrete implementation steps + +All commands run from the repository root: + +```plaintext + +``` + +First confirm branch and cleanliness: + +```bash +git branch --show-current +git status --short +``` + +Locate all construction and generation seams before editing: + +```bash +rg -n 'Target \{|BuildEdge \{|generate_into|generate\(' src tests test_support +rg -n 'NinjaContent|handle_build|handle_clean|handle_generate' src tests +rg -n '\.netsuke|cap_std|fs_utf8|rename' src tests +``` + +Run the existing focused baseline tests before adding red cases: + +```bash +cargo nextest run --test ast_tests --test ir_from_manifest_tests +cargo nextest run --test ninja_snapshot_tests --test ninja_gen_integration_tests +cargo nextest run --test bdd -- ninja +``` + +Use the exact test-binary and filter names discovered by `cargo nextest list` +if the last BDD filter is not accepted. Record any pre-existing failure before +editing and do not attribute it to this work. + +After adding each red test group, run only that group, record the expected +failure, implement the smallest corresponding production slice, and rerun until +green. Example commands, to be adjusted to the final test names, are: + +```bash +cargo nextest run --test ast_tests dependency_order +cargo nextest run --test ir_from_manifest_tests dependency_order +cargo nextest run --test ninja_snapshot_tests serial_dependency +cargo nextest run --test ninja_gen_integration_tests serial_dependency +cargo nextest run --test bdd serial_dependencies +``` + +After changes to Rust or Markdown, format once and inspect the resulting diff: + +```bash +make fmt +git status --short +git diff --check +git diff --stat +``` + +Then run the complete sequential gates listed in stage 7. Use the +commit-message skill to prepare each commit message in imperative mood with a +wrapped body. Do not push or open a pull request unless separately requested. + +## Validation and acceptance + +Acceptance is evidence-based. The following must all be true: + +- A target and an action both accept `dependency_order: serial`. +- `parallel` is accepted explicitly, omission defaults to it, and any unknown + enum value produces a localized manifest error. +- Serial `deps` retain declaration order from YAML through IR and every staged + dyndep sidecar. +- A real-Ninja test proves ordered start, not merely ordered gate completion. +- A real-Ninja test proves later dependencies do not start after an earlier + failure. +- Shared and repeated real dependencies execute once in one top-level Ninja + invocation. +- A real-Ninja barrier test proves omitted ordering remains parallel. +- A real-Ninja barrier test proves an unrelated graph branch remains parallel + with the active serial stage. +- Sources and order-only dependencies remain governed by their existing + semantics. +- Rebuild and no-op tests prove every serial dependency still participates in + aggregate freshness. +- Serial generation produces a complete bundle, uses valid dyndep syntax, and + declares Ninja 1.10 as the required version. +- Parallel generation produces no dyndep files and preserves existing snapshot + output. +- Sidecar writes are deterministic, idempotent, atomic, capability-oriented, + and covered for corruption and race outcomes. +- CLI build, clean, file output, stdout output, and JSON output never expose or + execute an unmaterialized serial bundle. +- User and internal documentation state the exact guarantees and the + independent-reachability boundary. +- No source file exceeds 400 lines, no lint is suppressed for convenience, and + every repository gate passes. + +The behavioural feature should contain a scenario equivalent to: + +```gherkin +Scenario: Serial dependencies preserve their declaration order + Given a manifest with a serial target depending on check-fmt, lint, and test + When the manifest is compiled and its Ninja bundle is generated + Then the target dependency order is serial + And the dyndep stages reveal check-fmt, lint, and test in that order +``` + +The runtime tests, rather than this textual scenario alone, are authoritative +for concurrency, failure, and shared-execution semantics. + +## Idempotence and recovery + +Generation is deterministic: identical graph input produces identical main +text, sidecar paths, and sidecar bytes. Repeating materialization reuses an +existing sidecar only after verifying its content. Content-addressed filenames +mean abandoned older files do not affect the new graph. + +If Netsuke is interrupted before rename, only its uniquely named temporary file +may remain. A later run may ignore or remove that temporary file and safely +retry. Never use a broad recursive deletion to recover. If a final digest path +contains mismatched bytes, report corruption with the exact relative path and +require the user to remove that single cache file before retrying. + +`netsuke clean` cleans build outputs through Ninja but may leave immutable +dyndep cache entries. Document manual recovery as removal of the narrow +`.netsuke/dyndep` directory only; never suggest deleting the entire workspace or +`.netsuke` root. + +If an implementation experiment shows that the dyndep sequence does not meet +one of the validated invariants, preserve the failing fixture, update +`Surprises & discoveries`, revert only the uncommitted experiment, and return +to the last green commit. Do not compensate with a pool or nested invocation. + +## Artefacts and notes + +During implementation, retain concise evidence in this document: + +- the first failing assertion for each red test group; +- one representative generated main-file fragment and sidecar; +- the observed marker/log order from the real-Ninja ordering test; +- proof that the shared dependency log contains one entry; +- proof that failure leaves later markers absent; +- proof that the unrelated branch crosses the concurrency barrier; and +- final gate commands, log paths, and commit identifiers. + +Do not paste complete build logs or broad snapshots into the plan. Store logs +under `/tmp` and summarize the decisive lines here. + +## Interfaces and dependencies + +The intended interfaces at completion are: + +```rust +pub enum DependencyOrder { + Parallel, + Serial, +} + +pub struct BuildEdge { + // Existing fields. + pub dependency_order: DependencyOrder, +} + +pub struct GeneratedNinja { /* private fields */ } + +pub struct GeneratedDyndep { /* private fields */ } + +pub fn generate_bundle(graph: &BuildGraph) -> Result; +``` + +`GeneratedNinja` must provide the main build text and a read-only or consuming +view of its sidecars. `GeneratedDyndep` must expose only a relative UTF-8 path +and immutable content. The runner materializer consumes those values but does +not decide graph structure or naming. + +`NinjaGenError` gains localized variants for requesting string-only output from +a serial graph and for reserved-output collisions. The materializer gains a +typed or contextual error for directory creation, temporary writes, rename +races, and digest-path corruption. Preserve domain errors within libraries and +convert to `eyre` only at the application boundary, following existing runner +conventions. + +The retention lease uses the sole external addition, `fs4` 1.1.0, resolved and +checksum-pinned in `Cargo.lock`. The existing `cap_std::fs_utf8` facilities +provide the capability-scoped open and filesystem operations, but neither they +nor the capability policy permit direct use of `std::fs::File::lock` for this +operation. The retention source therefore converts only the capability-opened +file to a standard handle for `fs4::FileExt`; the lock crate does not open paths +or widen the authority. Its published package metadata identifies the upstream +repository at and describes the cross-platform +file-lock implementation; this is the maintenance basis available here, not a +claim about an unverified release cadence. + +## Revision note + +2026-08-10: Initial draft. It replaces the disproven order-only phony-gate +proposal with a staged dyndep bundle, adds atomic sidecar materialization, and +records the independent-reachability limit that must be approved with the +implementation approach. + +2026-08-11: Updated the live status after reconciling the committed +implementation with the plan. Added the documentation-preparation evidence and +the narrow trailing-catalogue-whitespace cleanup required before the first full +validation and review. This does not change the remaining implementation scope. + +2026-08-11: Completed the user and maintainer documentation milestone after +review feedback identified that the implementation-only plan was insufficient +for issue #552 acceptance. ADR-011 records the staged-dyndep decision and the +user guide now states the syntax, guarantees, limitations, version floor, and +generated-state behaviour. Final gates and independent review remain before +completion. + +2026-08-12: The first full documentation gate run exposed the repository's +executable-fence contract. The serial-syntax sample is now a valid standalone +manifest with an explicit marker and an entry in the documentation-example +registry, so its syntax cannot drift without the normal test suite detecting +it. + +2026-08-12: The full suite passed after the executable-example correction, and +the independent CodeRabbit review of `7153538` returned no actionable concerns. + +2026-08-14: Re-verified the remaining documentation and locale findings against +the current implementation. Updated only the six requested documentation +surfaces, including the `src/ninja_gen/mod.rs` path and AST-to-IR conversion +description; no Rust or test files were changed. +This completes the implementation plan; the remaining work is only to refresh +the existing draft pull request with the completed documentation milestone. + +2026-08-14: Reduced bounded sidecar verification complexity without changing +its limit or concurrent-growth detection. Public-CLI tests now cover serial +sidecar publication for `generate` and `clean`, including loading generated +output with Ninja. Focused and full deterministic validation passed; the +post-milestone independent review reported zero actionable findings. + +2026-08-14: Added the runner-to-generator-to-materializer sequence to the +design document with a visible figure caption and Mermaid accessibility title +and description. This documents the existing boundary without changing it. + +2026-08-14: Reopened the completed plan for the latest verified review round. +The implementation now scopes temporary-name state to one publication attempt, +routes main-edge and default paths through the validated Ninja escaper, and +adds focused evidence for publication telemetry, concurrent growth, and retry +exhaustion. The corrected command-flow diagram distinguishes `generate` from +the Ninja-invoking `build` and `clean` boundaries. All focused and full gates +passed, and the independent CodeRabbit review reported no concerns. + +2026-08-12: Applied the requested test-only assertion-helper extraction after +plan completion. It changes neither production code nor feature semantics and +retains each separately named zero-staging behaviour test. + +2026-08-12: Refactored the atomic dyndep sidecar writer without changing its +protocol. The temporary-name collision path still accepts only matching final +content, while corruption, a missing final sidecar, write failures, and rename +failures retain their existing localization and error behaviour. + +2026-08-12: Reduced `generate_bundle` complexity by separating stable edge +selection from individual-edge rendering. The new helpers retain the exact +serial staging and ordinary display paths, so generated Ninja text, sidecar +content, and MissingAction errors remain unchanged. + +2026-08-12: Added an empty-sidecar materialization fast path. The first full +gate run exposed pre-existing ambient filesystem findings because the root +Whitaker recipe omitted its `DYLINT_TOML` input. Passing the existing policy to +that invocation restores its intentionally narrow exclusions without weakening +the capability lint; the full deterministic suite passed afterwards. + +2026-08-12: Temporary sidecar names are private to the runner materializer and +must never be reused by callers. A process identifier and monotonic sequence +keep concurrent write attempts distinct; `create_new` remains the final +authority and retries a stale collision. The helper may be used only to create +or inspect sidecars relative to the final dyndep path, preserving same-directory +atomic rename semantics. + +2026-08-12: `RenameFailureContext` is a private, single-use grouping for the +exact temporary path, final path, and expected content after one attempted +rename. Only `rename_temp_file` constructs it and only `handle_rename_failure` +consumes it; it must not become a runner-wide filesystem abstraction. +The first lint pass rejected a by-value context parameter, so the failure +handler borrows the private context instead; this does not alter the cleanup +path or error ownership. + +2026-08-12: PR review follow-up reopens the plan for two missing observable +contracts and a query-command boundary repair. The new real-Ninja regression +proves single execution for shared serial work and concurrent progress for an +unrelated branch. Bundle generation remains effect-free; runner command +boundaries will open and inject the filesystem capability for sidecar +publication. The remaining revision adds bounded outcome and duration +telemetry and removes unnecessary ownership copies. + +2026-08-12: Completed the review remediation in two green commits. Real Ninja +now proves execute-once shared work and unrelated-branch concurrency; the +generator returns an effect-free bundle that command-boundary publication +materializes through an injected directory capability; and bounded telemetry +covers bundle generation, serial lowering, and sidecar materialization. The +final deterministic gate run passed all six checks (1,944 tests, one skipped, +and passing doctests), and CodeRabbit reported zero findings before publication. + +2026-08-12: Verified the documentation review findings against the current +implementation. The plan contains no machine-specific absolute path, so no +path replacement was needed; the test-results sentence was corrected, and the +ADR, design sketch, and user's guide now align with the `BuildEdge` policy and +staged serial-ordering contract. + +2026-08-12: A second review pass verified every reported schema, link, +punctuation, locale, control-character, and test-structure issue against the +current branch. The minimal remediation adds the missing manifest-schema field, +corrects the requested catalogue wording, rejects Ninja path controls across +all edge fields and default targets, moves dependency-order lowering tests into +a dedicated module, and exercises serial runtime order with concurrent jobs and +observable marker preconditions. No finding in this pass was stale. + +2026-08-12: Separated the manifest syntax enum from the domain IR policy. The +Serde-enabled AST type still owns the lowercase YAML spelling and parallel +default; `from_manifest` now converts both variants explicitly into a distinct, +serialization-free IR enum. Focused AST and lowering tests retain omission and +variant coverage, and a compile-fail Rustdoc check guards the IR boundary. + +2026-08-12: Bounded existing-sidecar verification at 16 MiB. Publication now +checks metadata before allocation, reads through a metadata-sized limit, and +uses a one-byte probe so concurrent growth cannot make the read unbounded. An +oversized sidecar fails through a dedicated localized category present in all +35 catalogues; the focused materialization suite passes all nine tests. + +2026-08-12: Restored runner-boundary proof with real `netsuke -j 3 build` +processes, while retaining the direct Ninja tests as focused lowering evidence. +Moved all generation timing, spans, registration, counters, and histograms out +of `src/ninja_gen` and into the runner-owned `generate_ninja` boundary. The +generator is again a pure, fallible graph-to-bundle transformation and the +serial contract remains one scheduler with shared-work reuse, ordered failure +short-circuiting, and concurrency for unrelated branches. + +2026-08-12: Clarified the telemetry ownership decision: generation telemetry is +runner-owned by `src/runner/dyndep_generation_telemetry.rs`, publication +telemetry is owned by `src/runner/process/dyndep_telemetry.rs`, and Ninja +generation/rendering remains telemetry-free. + +2026-08-14: Rebased the completed feature onto `origin/main`. Conflict +resolution retained main's glob capability and migration guidance alongside +the serial dependency documentation. Because main now assigns ADR-010 to glob +capability scoping, the serial dyndep decision and its references move to +ADR-011. All code, test, documentation, and diagram gates passed after removing +one duplicate blank line introduced by the additive documentation merge. +`origin/main` advanced once during the first gate run, so the branch replayed +cleanly onto the new tip and passed the complete gate set again. + +2026-08-15: The review identified that immutable content-addressed sidecars +were safe but unbounded across changed manifests. ADR-012 now records the +replacement policy: current sidecars are never pruned, while deterministic +retention limits obsolete sidecars to 32 files and 1 MiB. A capability-scoped +advisory lease covers materialization, cleanup, and the command's Ninja or +generated-output consumption. The user guide explains that an old arbitrary +`generate --output` manifest can require regeneration after later retention. + +2026-08-15: Full validation passed after replacing the initially rejected +standard-library file-lock call with `fs4` on the same capability-opened file +handle. The new crate adds only a cross-platform advisory-lock implementation; +it neither opens paths nor expands the locking authority. The final evidence +includes all deterministic gates plus 14 materialization, 6 retention, and 7 +public serial-CLI focused tests. + +2026-08-15: Verified the remaining documentation and locale outcomes: the ADR +date punctuation is template-compliant and remains unchanged; the four +retention translations, users' guide sidecar location, and repository-layout +telemetry attribution are corrected. The Polish rename request was invalid, +and the Dutch rename request was already resolved; neither was changed. + +2026-08-16: Rebased the branch cleanly onto `origin/main` and verified the +latest review round. Bundle rendering now validates actions through the same +typed recipe boundary as string generation. Retention now uses one +lease-protected directory traversal and stores no more than the fixed file +budget while selecting deterministic path candidates; lease contention is +observed before the existing blocking acquisition. Review-focused fixtures, +target recipe placeholder coverage, public retention-budget reuse, and the +requested documentation and catalogue corrections are complete. The focused +retention, generator, and serial-CLI suites plus `make check-fmt`, `make test`, +`make typecheck`, `make lint`, `make markdownlint`, and `make nixie` passed. + +2026-08-16: Extracted per-directory-entry retention handling after a valid +CodeScene complexity finding. The traversal now only opens the directory, +initializes its bounded pass state, and delegates each entry; the private +handler retains the existing lock/current-path exclusions, stale temporary-file +cleanup, obsolete-sidecar selection, and localized error contexts. Focused +retention tests and all repository code gates passed. + +2026-08-16: Re-verified the requested documentation and locale corrections +against the current code. The split AST and Ninja module paths are now used in +the live ADR, developer guide, design document, and this plan. Sidecar +ownership names the publication boundary, atomic materializer, and retention +module separately. Hungarian, Russian, and Vietnamese rename messages now say +rename explicitly; the operation-required and generated-Ninja wording is +neutral and precise in the requested catalogues. No supplied finding was +stale. + +2026-08-16: Replaced retention's enumeration-order-dependent replacement with +bounded lexical reselection over the retained paths plus one candidate. This +preserves the greedy count and byte policy without storing the directory's full +contents; a mixed-size, deliberately nonlexical regression covers multiple +evictions. Retention now records aggregate duration, and temporary-name +collisions record fixed retry or exhausted outcomes without exposing paths. + +2026-08-17: Corrected the dependency record after the retention implementation. +`fs4` is the sole external addition and resolves to 1.1.0 in `Cargo.lock`. +The existing capability filesystem opens the lock file but supplies no +cross-process advisory-lock operation. Although the pinned toolchain exposes +`std::fs::File::lock`, direct `std::fs` use would violate the +capability-filesystem policy. `fs4` is therefore applied only to the +capability-opened handle; +the lockfile source/checksum and published package metadata provide the +available maintenance basis without asserting an unverified release cadence. + +2026-08-17: Rebased onto the current `origin/main` and re-verified the next +review round as code-level hypotheses. All supplied findings remained valid: +the materialization telemetry assertion now requires the complete outcome set, +the empty-lease no-op is covered, and dyndep test modules share one +capability-directory fixture. Hindi and Chinese rename diagnostics now name the +operation explicitly. The follow-up gate run covers the rebased command +dispatcher as well as these focused regressions before review is requested. + +2026-08-17: Re-verified the latest documentation, localization, and help-output +review findings before making minimal corrections. The plan's manifest example +now uses executable action-list syntax. User and migration guidance specify the +reserved generated namespaces and conditional regeneration boundary. The design +already described staged serial lowering, so that duplicate finding required no +change. Catalogue corrections retain Fluent placeholders and name rename or +operation-neutral behaviour where requested; Vietnamese rename wording was +already correct. The help example now confirms the documented `hello.txt` +target rather than accepting any non-empty target catalogue. + +2026-08-18: Moved the HTTP 429 fallback for refreshing the shared spelling +dictionary to a separate maintenance branch. It is unrelated to serial +dependency ordering, so this feature branch retains only the review corrections +to its locales and manifest-render tests. diff --git a/docs/netsuke-design.md b/docs/netsuke-design.md index 6f035e61f..410457fc7 100644 --- a/docs/netsuke-design.md +++ b/docs/netsuke-design.md @@ -222,6 +222,7 @@ erDiagram Recipe recipe StringOrList sources StringOrList deps + string dependency_order StringOrList order_only_deps map vars string description @@ -382,6 +383,10 @@ are specified. any of these dependencies will trigger a rebuild of the current target, but `deps` do not appear in `ins` or Ninja `$in`. +- `dependency_order`: An optional `parallel` or `serial` policy for the direct + `deps` list on an action or target. It defaults to `parallel`; `serial` + preserves declaration order without changing the freshness class of `deps`. + - `order_only_deps`: An optional list of prerequisite target names or paths that must be built before this target, but whose modification does not trigger a rebuild of this target. This maps directly to Ninja's order-only @@ -404,6 +409,7 @@ The cleaner model is: - `sources` contribute to `ins` / `$in`. - `deps` affect ordering and rebuild decisions, but do not appear in `ins`. - `order_only_deps` affect ordering only. +- `dependency_order` changes only the scheduling policy for direct `deps`. - `vars`: An optional mapping of local variables. These variables override any global variables defined in the top-level `vars` section for the scope of @@ -768,6 +774,9 @@ pub struct Target { #[serde(default)] pub deps: StringOrList, + #[serde(default)] + pub dependency_order: DependencyOrder, + #[serde(default)] pub order_only_deps: StringOrList, @@ -865,6 +874,7 @@ let ast = NetsukeManifest { }, sources: StringOrList::Empty, deps: StringOrList::Empty, + dependency_order: DependencyOrder::Parallel, order_only_deps: StringOrList::Empty, vars: HashMap::new(), description: None, @@ -1900,6 +1910,9 @@ pub struct BuildEdge { /// Maps to Ninja's '|' syntax and remains separate from `$in` / `{{ ins }}`. pub implicit_deps: Vec, + /// Ordering policy applied to `implicit_deps` from a manifest `deps` list. + pub dependency_order: DependencyOrder, + /// Outputs explicitly generated by the command. pub explicit_outputs: Vec, @@ -1939,6 +1952,7 @@ classDiagram +String action_id +Vec inputs +Vec implicit_deps + +DependencyOrder dependency_order +Vec explicit_outputs +Vec implicit_outputs +Vec order_only_deps @@ -1953,7 +1967,7 @@ classDiagram Exec } class ninja_gen { - +generate(graph: &BuildGraph) String + +generate_bundle(graph: &BuildGraph) Result } BuildGraph "1" o-- "many" Action : actions BuildGraph "1" o-- "many" BuildEdge : targets @@ -2035,7 +2049,7 @@ This transformation involves several steps: traversal are logged, collected, and returned alongside any cycle to aid diagnostics. -### 5.4 Ninja File Synthesis (`ninja_gen.rs`) +### 5.4 Ninja File Synthesis (`ninja_gen/mod.rs`) The final step is to synthesize the `build.ninja` file from the `BuildGraph` IR. This process is a straightforward, mechanical translation from the IR data @@ -2097,6 +2111,69 @@ structures to the Ninja file syntax. build my_app: link foo.o bar.o | lib_dependency.a ``` + A `BuildEdge` whose `dependency_order` is `serial` and has more than one + implicit dependency is an exception to this direct rendering. The generator + lowers it into staged phony gates, with one content-addressed Ninja dyndep + sidecar per dependency. A gate can reveal exactly one real dependency; the + gate edge associated with the next sidecar depends on the preceding gate. + This makes each later dependency unavailable to Ninja until the previous one + succeeds, while preserving one Ninja scheduler and its shared-work + memoization. The runner materializes every sidecar file before Ninja starts; + no Ninja edge produces sidecar content. + + The generated result is a bundle, not merely a string: generation is an + effect-free query that returns the main Ninja text and its + `.netsuke/dyndep` sidecars. Each runner command then materializes those + sidecars through an injected effective-working-directory capability before + it writes or runs the main file. The main file declares + `ninja_required_version = 1.10` only when it contains such staged serial + ordering. `.netsuke/serial` and `.netsuke/dyndep` are reserved for generated + state. `serial` applies only to direct implicit dependencies; it does not + delay an independently reachable node elsewhere in the graph. + +Figure: Runner-owned serial dyndep bundle generation and execution. + +```mermaid +sequenceDiagram + accTitle: Runner-owned serial dependency generation and execution + accDescr { + The runner generates a Ninja bundle and materializes its dyndep sidecars. + Generate writes the manifest without invoking Ninja. Build invokes Ninja + for execution, and clean invokes Ninja in clean tool mode. + } + actor User + participant Runner as runner.generate_ninja + participant NinjaGen as ninja_gen.generate_bundle + participant Dyndep as runner.materialize_dyndep_bundle + participant Ninja + + User->>Runner: netsuke build / clean / generate + Runner->>NinjaGen: generate_bundle(graph) + NinjaGen-->>Runner: GeneratedNinja (build_file, dyndep_files) + Runner->>Dyndep: materialize_dyndep_bundle(cli, bundle) + Dyndep-->>Runner: dyndep sidecars materialized + alt generate + Runner-->>User: write generated Ninja manifest without invoking Ninja + else build + Runner->>Ninja: invoke with bundle.build_file() + Ninja-->>User: serial deps run in order, parallel elsewhere + else clean + Runner->>Ninja: invoke with bundle.build_file() in clean tool mode + Ninja-->>User: clean completed + end +``` + +The runner holds a capability-scoped exclusive lease on the dyndep directory +from sidecar materialization through Ninja consumption or generated-output +consumption. While the lease is held, stale `.tmp` files are removed and +retention preserves the current bundle plus at most 32 obsolete `.dd` files +and 1 MiB of obsolete `.dd` bytes. `build` and `generate` prune after +materialization; `clean` prunes only after successful `ninja -t clean` and not +on failure. Sidecars remain immutable and content-addressed. Consequently, +an older arbitrary `generate --output` manifest may lose its sidecars after a +later command and must be regenerated. See +[ADR-012](adr-012-bound-dyndep-sidecar-retention.md) for this policy. + 4\. **Write Defaults:** Finally, write the `default` statement, listing all paths from `graph.default_targets`. @@ -2132,6 +2209,11 @@ representation portable. optional key-value pairs or flags, keeping the generator easy to scan. - Integration tests snapshot the generated Ninja file with `insta` and execute the Ninja binary to validate structure and no-op behaviour. + Serial-ordering tests additionally use real Ninja to prove declaration + order, failure short-circuiting, shared-work reuse, and unrelated-branch + concurrency. [ADR-011](adr-011-use-ninja-dyndep-for-serial-dependency-ordering.md) + records why staged dyndep is used instead of order-only gates, pools, or + recursive Ninja invocations. ## Section 6: Process Management and Secure Execution @@ -3101,12 +3183,12 @@ goal. 1. Implement the initial `clap` CLI structure for the `build` command. 2. Implement the YAML parser using `serde_saphyr` and the AST data - structures (`ast.rs`). + structures (`ast/mod.rs`). 3. Implement the AST-to-IR transformation logic, including basic validation like checking for rule existence. - 4. Implement the IR-to-Ninja file generator (`ninja_gen.rs`). + 4. Implement the IR-to-Ninja file generator (`ninja_gen/mod.rs`). 5. Implement the `std::process::Command` logic to invoke `ninja`. diff --git a/docs/repository-layout.md b/docs/repository-layout.md index 4c90317f5..622f851da 100644 --- a/docs/repository-layout.md +++ b/docs/repository-layout.md @@ -29,6 +29,7 @@ output and some leaf files so the long-lived structure remains visible. │ ├── ir/ │ ├── localization/ │ ├── manifest/ +│ ├── ninja_gen/ │ ├── runner/ │ ├── snapshots/ │ └── stdlib/ @@ -85,8 +86,11 @@ output and some leaf files so the long-lived structure remains visible. support. - `src/manifest/`: Manifest parsing, expansion, rendering, diagnostics, and manifest-specific tests. -- `src/runner/`: Process execution, path handling, runner errors, and runtime - command orchestration. +- `src/ninja_gen/`: Ninja rendering and staged-dyndep bundle generation. +- `src/runner/`: Process execution, path handling, runner errors, command + orchestration, capability-injected dyndep publication, and bounded + generation and publication telemetry, including + `dyndep_generation_telemetry.rs` and `process/dyndep_telemetry.rs`. - `src/snapshots/`: Checked-in `insta` snapshots for source-level snapshot tests. - `src/stdlib/`: Netsuke standard library modules exposed to manifest @@ -134,3 +138,15 @@ Place feature files in `tests/features/` unless the behaviour depends on Unix-specific platform contracts, in which case use `tests/features_unix/`. Place generated or approved snapshot files under the existing `src/snapshots/` or `tests/snapshots/` hierarchy that matches the test owner. + +Netsuke runtime state belongs under `.netsuke/` in the effective working +directory, never in the repository layout itself. In particular, +`.netsuke/dyndep` contains immutable content-addressed sidecars for serial +dependencies and `.netsuke/serial` is a reserved generated-gate namespace; +manifest outputs must not claim either path. Sidecar-capable commands retain +the current bundle, at most 32 obsolete `.dd` files, and 1 MiB of obsolete +`.dd` bytes. They remove stale `.tmp` files while holding the exclusive +directory lease; `clean` prunes only after successful `ninja -t clean`. +An older arbitrary `generate --output` manifest may therefore need +regeneration after a later command. See +[ADR-012](adr-012-bound-dyndep-sidecar-retention.md). diff --git a/docs/roadmap.md b/docs/roadmap.md index fafa7f7b5..d51b8310e 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -175,6 +175,11 @@ and agents. ordering and rebuild decisions without appearing in recipe arguments. - [x] Align cycle detection, generated Ninja output, and user-facing dependency documentation. + - [x] Add `dependency_order: serial` for direct action and target `deps`. + Staged Ninja dyndep lowering preserves declaration order, failure + short-circuiting, shared-work reuse, and unrelated-branch concurrency; + [ADR-011](adr-011-use-ninja-dyndep-for-serial-dependency-ordering.md) + records the path-scoped guarantee and generated-state contract. - [x] 3.14.4. Add `command_available(name, **kwargs)` as a non-throwing executable probe. Depends on archived task `3.5.1`. See [executable discovery](netsuke-design.md#executable-discovery-filter-which). diff --git a/docs/users-guide.md b/docs/users-guide.md index 42955b857..60fbfde1f 100644 --- a/docs/users-guide.md +++ b/docs/users-guide.md @@ -386,6 +386,9 @@ A target supports these fields: recipe arguments. Declare them on each target; reusable rules reject `deps`. The planned rule-level `deps_from` contract is not implemented in v0.1.0-beta1. +- `dependency_order`: scheduling policy for the `deps` list. `parallel` is the + default; `serial` runs a list with more than one dependency in declaration + order. - `order_only_deps`: ordering dependencies. Their changes do not rebuild the dependent target. - `vars`: values that override global variables for this target. The `env` @@ -408,6 +411,79 @@ v0.1.0-beta1. Cycle detection follows `sources` and `deps`. Order-only dependencies enforce ordering but do not participate in cycle detection. +### Run direct dependencies serially + +Actions and targets both accept `dependency_order`. Omit it, or set it to +`parallel`, to retain Ninja's ordinary concurrent scheduling. Set it to +`serial` when the direct `deps` list is an ordered workflow: + + + +```yaml +netsuke_version: "1.0.0" + +actions: + - name: check-fmt + command: "echo checking format" + - name: lint + command: "echo linting" + - name: test + command: "echo testing" + - name: all + command: ":" + dependency_order: serial + deps: + - check-fmt + - lint + - test + +targets: + - name: release-notes + command: "echo preparing release notes" + - name: release + command: "./package-release" + dependency_order: serial + deps: + - check-fmt + - test + - release-notes +``` + +For a serial list, Netsuke starts each direct dependency only after the +preceding one succeeds. If an earlier dependency fails, later dependencies in +that list do not start through the serial path. Repeated or shared dependencies +are still owned by the one Ninja invocation and execute at most once. + +Serial ordering applies only to the direct `deps` list. It does not serialize +`sources`, `order_only_deps`, or unrelated work. An independently requested or +otherwise reachable later dependency can still start through that separate +path; use a dedicated aggregate action when the whole workflow must share the +same ordered entry point. + +Netsuke uses Ninja's `dyndep` support for serial lists with two or more +dependencies, and generated builds containing staged serial ordering require +Ninja 1.10 or newer. +`netsuke generate`, `build`, and `clean` materialize the generated sidecars +under `.netsuke/dyndep` in the effective working directory before writing or +invoking the generated Ninja file. The sidecars are immutable and +content-addressed. Each sidecar-capable command retains the current bundle, +then at most 32 obsolete `.dd` files and 1 MiB of obsolete `.dd` bytes. Stale +`.tmp` files are removed while the exclusive sidecar-directory lease is held. +`build` and `generate` prune after materialization; `clean` prunes only after +successful `ninja -t clean`, and does not prune when clean fails. + +An older arbitrary manifest written with `generate --output` may lose its +referenced sidecars after a later command. Regenerate that manifest before +using it if retention has removed any of its sidecars. +The paths `.netsuke/serial` and `.netsuke/dyndep` must not occur in any user +graph path, including outputs, inputs, implicit dependencies, and order-only +dependencies; they are reserved for Netsuke-generated gates and sidecars. + +When migrating an existing manifest, see the +[v0.1.0 migration guide](v0-1-0-migration-guide.md#opting-into-serial-dependency-ordering) +for the opt-in syntax, Ninja version requirement, and generated-state +reservation. + ## Use Jinja safely Jinja expressions are allowed in renderable string fields, including variables, @@ -853,7 +929,12 @@ textual outline and a `