diff --git a/docs/contents.md b/docs/contents.md index 06c131482..7d6cba2d6 100644 --- a/docs/contents.md +++ b/docs/contents.md @@ -18,6 +18,11 @@ operator, user, and contributor references are easier to find. - [git-change-detection-helpers-design.md](git-change-detection-helpers-design.md): Git change-detection and glob-matching contracts and verification guidance for maintainers, reviewers, and manifest authors. +- [UX and semantic design](netsuke-test-framework-ux-design.md): Test + dialect, mocking model, and command surface for the Netsukefile testing + framework. +- [Technical design](netsuke-test-framework-technical-design.md): + Implementation architecture for the Netsukefile testing framework. - [roadmap.md](roadmap.md): Phased implementation plan and tracked delivery work. - [archive/roadmap-completed-foundations.md](archive/roadmap-completed-foundations.md): @@ -32,6 +37,9 @@ operator, user, and contributor references are easier to find. - [rfcs/0001-structured-command-blocks.md](rfcs/0001-structured-command-blocks.md): Proposed structured command blocks, shell-free argv templates, typed Jinja interpolation, stream routing, and pipeline semantics. +- [rfcs/0007-netsukefile-testing-framework.md](rfcs/0007-netsukefile-testing-framework.md): + Proposed Netsukefile testing framework: the `netsuke test` command, the YAML + test dialect, and its mocking model. ## Decision records diff --git a/docs/netsuke-test-framework-technical-design.md b/docs/netsuke-test-framework-technical-design.md new file mode 100644 index 000000000..02227741a --- /dev/null +++ b/docs/netsuke-test-framework-technical-design.md @@ -0,0 +1,976 @@ +# Netsuke test framework technical design + +## Front matter + +- **Status:** Draft. +- **Scope:** The implementation architecture of the Netsukefile testing + framework: pipeline integration, injection seams, the test-suite parser, the + mock engine, the fixture engine, command-line integration, and the + verification obligations the implementation must discharge. The user-facing + dialect is normative in the companion + [UX and semantic design](netsuke-test-framework-ux-design.md); this document + does not restate its semantics except where implementation detail depends on + them. +- **Primary audience:** Netsuke developers implementing `netsuke test`, and + reviewers assessing the architecture. +- **Governing documents:** + [RFC 0007](rfcs/0007-netsukefile-testing-framework.md); [netsuke-design.md](netsuke-design.md) + for the compiler pipeline; [ADR-008](adr-008-environment-seam-taxonomy.md) + for environment seams; + [ADR-010](adr-010-scope-glob-capability-to-literal-prefix.md) for filesystem + capability scoping. Accepted ADRs take precedence over this document. + +## 1. Constraints + +Non-negotiable constraints the rest of the document assumes: + +- **C1 — one compiler.** `netsuke test` calls the same public library + functions as the build path: the manifest loader, `BuildGraph` lowering, and + `ninja_gen::generate`. The test runner adds overlays; it never re-implements + manifest semantics. +- **C2 — no ambient mutation.** The runner must not mutate the process + environment, the project root, or global state. All test I/O happens inside + per-case `cap-std` sandboxes; all environment values flow through injected + readers per ADR-008. +- **C3 — no execution.** No action in the first version spawns Ninja, build + commands, or fixture shell commands. The stdlib command helpers are disabled + under test (§5.5). When the deferred `execute` action does arrive it should + drive `NinjaProcessOptions` — the narrow execution type carrying working + directory, job count, and stderr suppression — rather than fabricating a + `Cli`, since that decoupling exists precisely so non-CLI callers can run + Ninja. +- **C4 — deterministic by default.** Clock, network, and environment are + test-controlled. An unmocked impure call is an error, not a silent + passthrough. +- **C5 — localized, stream-pure output.** User-facing strings go through + the Fluent localization layer; `--json` obeys the one-document stream + contract that the wider CLI roadmap mandates. + +## 2. The gap this closes + +Netsukefiles carry logic — `foreach` expansion, `when` conditions, macros, +environment probes, globbing, executable discovery — and nothing verifies it. +An author changes a `when` condition and learns whether it still matches by +running a build and reading the output. That check is slow, answers differently +on different machines, and cannot express the cases that matter most: a target +that must _not_ be generated, or a manifest that must fail with a particular +diagnostic. + +The pipeline is already shaped to close this. Manifest loading is staged and +injectable, `BuildGraph::from_manifest` and `ninja_gen::generate` are public +functions over plain data, and `netsuke help targets` has already established a +restricted load mode. What is missing is a way to drive that pipeline with +substituted seams and assert on what comes out. The architecture below adds +exactly that, and nothing else: no second evaluator, no re-implementation of +manifest semantics. + +## 3. Architecture summary + +The feature adds one new subsystem, `src/testing/`, plus narrow seams in +existing modules. The runner parses test files into a test-suite AST, then +executes each case in a child process that builds a per-case `TestContext` +(sandbox, doubles, environment, clock), drives pipeline actions through the +existing manifest/IR/Ninja code with an overlay-carrying options structure, and +evaluates assertions against structured result views. The parent supervises +those children and renders the report. + +The process boundary exists to make `--timeout` enforceable (§9.1); it is not a +second evaluator. The diagram below shows the flow for one case, with the +parent/child split marked: discovery and parsing happen once in the parent, +case execution happens in a killable child, and the `CaseResult` returns over a +versioned frame protocol for reporting. + +```mermaid +graph TD + subgraph parent["parent process"] + A["netsuke test (CLI)"] --> B[discovery] + B --> C[test-suite parser] + C --> D["scheduler owner
queue, fail-fast, report"] + D --> S["case supervisor
deadline, kill, reap"] + D --> N["report renderer
human / JSON"] + end + S -->|spawn| E + subgraph child["child process (one case)"] + E[TestContext
sandbox, doubles, env, clock] + E --> F[fixture engine] + E --> G[pipeline actions] + G --> H["manifest loader
(with overlays)"] + H --> I[BuildGraph lowering] + I --> J[ninja_gen] + H --> K[mock journal] + G --> L[result views] + L --> M[assertion evaluator] + K --> M + end + M -->|CaseResult frames| D + S -->|timeout: synthesized error| D +``` + +_Figure 1: Test case execution flow across the process boundary. Existing +compiler components (manifest loader, BuildGraph, ninja_gen) are reused +unchanged apart from overlay injection at environment-construction time. The +supervisor enforces the deadline and, on expiry, supplies a synthesized errored +result in place of the child's. Every result returns to the scheduler owner, +which alone holds scheduling state and writes the report._ + +## 4. Pipeline integration + +### 4.1. The pipeline today + +The loader driver `from_str_named` (`src/manifest/mod.rs:109`) runs six stages: +read, `serde_saphyr` parse into a JSON value tree, MiniJinja environment +construction (strict undefined; `env()` and `glob()` registered from an injected +`EnvReader` and the glob expander; stdlib registered according to the selected +`StdlibRegistration`; manifest `vars` exposed as globals), macro registration +plus `expand_foreach` (which also evaluates `when`), `serde_json::from_value` +deserialization into `NetsukeManifest`, and `render_manifest` string rendering. +The fullest-parameterized entry point is `from_path_with_policy_and_env` +(`src/manifest/mod.rs:385`), which already accepts a `NetworkPolicy`, an +`EnvReader`, and a coarse `ManifestLoadStage` callback. + +Two ordering facts drive the design. First, `foreach` and `when` evaluate +against the raw value tree _before_ typed deserialization, so doubles for +`glob`, `env`, and friends must be installed in the environment before +`expand_foreach` runs. Second, manifest macros are registered before expansion +(`register_manifest_macros`, `src/manifest/jinja_macros/mod.rs`), so macro +substitution is an overlay registered _after_ manifest macros and _before_ +expansion. + +### 4.2. The restricted-load precedent + +`netsuke help targets` established the pattern this framework extends. The +loader already selects its standard-library boundary through an enum +(`StdlibRegistration`, `src/manifest/mod.rs:99`) with two variants: +`Full(Box)` for builds, and `ManifestQuery` for side-effect-free +discovery. `src/manifest/query.rs` owns that boundary, and +`register_manifest_query` (`src/stdlib/register.rs:135`) implements it by +registering the pure helpers and replacing `env`, `glob`, `fetch`, `shell`, +`grep`, and `contents` with stubs that raise a located diagnostic naming the +unavailable operation. + +The test runner is a third load mode of exactly this shape, so it extends the +existing enum rather than introducing a parallel mechanism: + +```rust +enum StdlibRegistration { + Full(Box), + ManifestQuery, + Test(Box), // sandbox-rooted; impure helpers refuse +} +``` + +Three consequences follow, each replacing machinery this design would otherwise +have invented: + +- The disabled-helper diagnostic already exists. + `manifest_query_operation_error` (`src/stdlib/register.rs:262`) is the + template for the "unavailable under test" messages in §5.5; the test mode + reuses the mechanism with its own message keys rather than intercepting + MiniJinja's unknown-function error. +- `disabled_env_reader` (`src/manifest/env_reader.rs:79`) already provides + a reader that refuses every lookup. The test reader is that reader with the + case's declared variables layered over it (§5.1). +- `src/manifest/query.rs` is the precedent module for a capability-scoped + non-build load, and the test runner's loader entry belongs beside it rather + than in a new location. + +### 4.3. Loader options + +The loader gains an options-carrying entry point; the existing entry points +become thin wrappers over it with default options. + +```rust +pub struct ManifestLoadOptions<'a> { + pub registration: StdlibRegistration, + pub env_reader: Option, + pub overlays: Option, + pub on_stage: Option<&'a mut dyn FnMut(ManifestLoadStage)>, +} + +/// Test-supplied substitutions applied to the MiniJinja environment +/// after stdlib and manifest-macro registration, before foreach expansion. +pub struct TemplateOverlays { + pub functions: IndexMap, + pub macro_substitutions: IndexMap, +} +``` + +`registration` carries the stdlib boundary rather than a bare `StdlibConfig` +plus a separate `NetworkPolicy`, because §4.2 already binds those together per +mode: the network policy for a test load is a property of +`StdlibRegistration::Test`, not an independently settable knob. This keeps one +place where a load mode's capabilities are decided. `Test` boxes its payload +like `Full`, matching the existing constructors in `parse_with_config.rs` and +`query.rs` and keeping the variants near enough in size that the enum stays +cheap to move. + +The structure is named `TemplateOverlays`, not `EnvOverlays`: in this codebase +"env" means the process environment (ADR-008, `EnvReader`), and these overlays +substitute callables in the MiniJinja _template_ environment. The clock +deliberately does not appear here — it has exactly one owner, `StdlibConfig` +(§5.2). `on_stage` keeps the existing `&mut dyn FnMut` shape from +`from_path_with_policy_and_env`. + +`OverlayCallable` wraps a double's dispatch closure (§7). Registration order +inside environment construction becomes: + +1. `env()` and `glob()` from the effective `EnvReader` and glob expander; +2. stdlib via `register_with_config` (with the test's `StdlibConfig`); +3. manifest `vars` globals; +4. manifest macros; +5. **overlays** — test doubles registered last so they shadow same-named + stdlib functions (MiniJinja `add_function` replaces an existing + registration), plus macro substitutions, which additionally rewrite the + macro-import prelude (§5.4); +6. `expand_foreach`, deserialization, rendering as today. + +The overlay hook is compiled unconditionally: it is an ordinary parameter, not +a test-only `cfg`, because the test runner is a production code path of the +shipped binary. + +### 4.4. Result views + +`NetsukeManifest` already derives `Serialize` (`src/ast/mod.rs:102`), and +`GraphView` (`src/graph_view/`) is an existing deterministic projection of +`BuildGraph` with sorted nodes and edges. The assertion layer builds on both: + +- `result.manifest` — the rendered manifest serialized to a MiniJinja + value. +- `result.graph` — a `TestGraphView` wrapping `GraphView` with the helper + methods the UX design promises (`has_target`, `has_rule`, `target(name)` + field access), exposed as a MiniJinja object. +- `result.ninja` — the string from `ninja_gen::generate` + (`src/ninja_gen/mod.rs:106`), which is already deterministic for snapshot + tests. + +The IR types themselves are not exposed: the views are a stable assertion +surface that can hold shape while internal IR evolves. + +## 5. Injection seams + +Each seam follows the ADR-008 taxonomy; two exist, two are new. + +### 5.1. Environment (existing) + +`EnvReader` (`src/manifest/env_reader.rs:56`) is an +`Arc Result + Send + Sync>`. The runner +builds one from the case's `given.env` map: declared names return their values, +`unset` names and everything else return `EnvReadError::NotPresent`. The host +environment is reachable only through an explicit future opt-in; the default +reader never consults it (C2, C4). + +### 5.2. Clock (new seam) + +`now()` currently calls `OffsetDateTime::now_utc()` directly +(`src/stdlib/time/mod.rs:62`) — a gap relative to ADR-008. The stdlib time +module gains a clock provider in the `EnvReader` shape (an `Arc` closure, +because MiniJinja registration requires `Send + Sync`): + +```rust +pub type ClockProvider = Arc OffsetDateTime + Send + Sync>; +``` + +Production registration wraps `OffsetDateTime::now_utc`; the test runner +supplies a fixed instant parsed from `given.clock.now`. The seam lives in +`StdlibConfig` alongside the existing `path_override` and `home_directory` +knobs — the clock's single owner — and is a prerequisite refactor deliverable +in its own right. + +### 5.3. Network (policy, not transport) + +`fetch()` builds its HTTP agent inline (`src/stdlib/network/mod.rs`), so +transport injection would be invasive. The test runner does not need it: a test +that wants `fetch` results declares a double for the `fetch` function itself, +and overlays register after the stdlib (§4.3), so the double shadows the +refusing stub. The real network code is therefore unreachable under test — +either the overlay answers the call, or the refusing stub raises a diagnostic +telling the author to declare one. No transport seam is built, and no live +agent is ever constructed. + +### 5.4. Macro substitution (new mechanism) + +`substitute("stand_in")` compiles the stand-in macro from the test file through +the same `register_macro` path as manifest macros, then installs a journalling +wrapper that records the call and delegates to the compiled stand-in. Signature +arity is validated when the macro is called, as with ordinary manifest macros; +earlier validation is a possible refinement, not a first-version requirement. + +Installing that wrapper takes more than `add_function`. `register_macro` does +two things per macro: it adds a global function, _and_ it appends +`{% from '